Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Firebird 2.5.1 runs out of memory while restoring database backup [CORE3802] #4145

Closed
firebird-automations opened this issue Mar 29, 2012 · 11 comments

Comments

@firebird-automations
Copy link
Collaborator

Submitted by: Artem Petkevych (artem)

Is related to CORE3433

While restoring table with array fields, Firebird consumes all available memory and restore fails with error "unable to allocate memory from operating system". Same backup can be successfully restored by Firebird 2.1.4. DDL of the table is:

CREATE TABLE RVG (
RYEAR SMALLINT NOT NULL,
INS_ID BIGINT NOT NULL,
S1 DECIMAL(18,2),
S16 DECIMAL(18,2),
R1 DECIMAL(18,2) [1:12],
R4 DECIMAL(18,2) [1:4],
R5 DECIMAL(18,2) [1:4],
R6 DECIMAL(18,2) [1:4],
VK1 COMPUTED BY (R1[1]+R1[2]+R1[3]),
VK2 COMPUTED BY (R1[4]+R1[5]+R1[6]),
VK3 COMPUTED BY (R1[7]+R1[8]+R1[9]),
VK4 COMPUTED BY (R1[10]+R1[11]+R1[12]),
VG COMPUTED BY (R1[1]+R1[2]+R1[3]+R1[4]+R1[5]+R1[6]+R1[7]+R1[8]+R1[9]+R1[10]+R1[11]+R1[12]),
R3 INTEGER [1:4]
);

ALTER TABLE RVG ADD CONSTRAINT PK_RVG PRIMARY KEY (RYEAR, INS_ID);

Table has 5454042 rows. Error occurs after 2560000 records restored.
To reproduce issue one can download backup from this URL: http://www.protec.kiev.ua/~artem/db/rvg.7z

Commits: b153879 c6c1c2c 373ec81

@firebird-automations
Copy link
Collaborator Author

Commented by: @dyemanov

It can be reproduced with 2.5.1 as well, hence I've changed the subject to mention the already released version instead of a snapshot build.

@firebird-automations
Copy link
Collaborator Author

Modified by: @dyemanov

Version: 2.5.1 [ 10333 ]

environment: Windows 7 64-bit, RAM 4 gb, Firebird 2.5.2.26449 32-bit => Windows 7 64-bit, RAM 4 gb, Firebird 2.5.1.26351 32-bit

summary: Firebird 2.5.2 runs out of memory while restoring database backup => Firebird 2.5.1 runs out of memory while restoring database backup

@firebird-automations
Copy link
Collaborator Author

Modified by: @dyemanov

Link: This issue is related to CORE3433 [ CORE3433 ]

@firebird-automations
Copy link
Collaborator Author

Commented by: @dyemanov

This is a regression introduced while implementing CORE3433.

@firebird-automations
Copy link
Collaborator Author

Commented by: @dyemanov

Usage of isc_send() instead of isc_start_and_send() forces the looper to always exit with a non-NULL node, so release_blobs() is never called and thus blobs/arrays bound to the current request are never released, at least until the transaction ends. Hence we have a transaction level memory leak.

The dumb but easiest solution could be to restart the request periodically (e.g. if (records % 1000 == 1)) to keep some kind of a balance between memory usage and performance. Or set some flag if we have blobs/arrays stored right before this record and use that flag to restart the request instead of just sending the record. Or combine these two approaches.

Ideally, the engine should be fixed, but this sounds too risky for point releases.

@firebird-automations
Copy link
Collaborator Author

Modified by: @dyemanov

assignee: Adriano dos Santos Fernandes [ asfernandes ]

@firebird-automations
Copy link
Collaborator Author

Commented by: @asfernandes

That's weird. I tested latest 2.5 in release-mode, with and without CORE3433 change.

Around 2,400,000 records I verified and in both cases top says virtual memory consumption is around 370M.

I can restore it in Linux 64 bits.

Without CORE3433, it takes "3m6.838s" and with it "2m53.671s".

@firebird-automations
Copy link
Collaborator Author

Commented by: @dyemanov

With a 32-bit FB 2.5.2 (release mode) on 64-bit Windows I see VM = 750MB after 1,000,000 records restored, 1.5GB after 2,000,000 records and the OOM error happens at 2,262,000 records. 64-bit FB consumes a bit more memory but unsurprisingly succeeds to restore.

Just for the record, I was restoring using the Services API.

@firebird-automations
Copy link
Collaborator Author

Modified by: @asfernandes

status: Open [ 1 ] => Resolved [ 5 ]

resolution: Fixed [ 1 ]

Fix Version: 2.5.2 [ 10450 ]

@firebird-automations
Copy link
Collaborator Author

Modified by: @pcisar

status: Resolved [ 5 ] => Closed [ 6 ]

@firebird-automations
Copy link
Collaborator Author

Modified by: @pavel-zotov

status: Closed [ 6 ] => Closed [ 6 ]

QA Status: No test

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

2 participants