Issue Details (XML | Word | Printable)

Key: CORE-3802
Type: Bug Bug
Status: Closed Closed
Resolution: Fixed
Priority: Major Major
Assignee: Adriano dos Santos Fernandes
Reporter: Artem Petkevych
Votes: 0
Watchers: 4

If you were logged in you would be able to see more operations.
Firebird Core

Firebird 2.5.1 runs out of memory while restoring database backup

Created: 29/Mar/12 01:42 PM   Updated: 23/Apr/13 01:46 PM
Component/s: Engine
Affects Version/s: 2.5.1, 2.5.2
Fix Version/s: 2.5.2

Environment: Windows 7 64-bit, RAM 4 gb, Firebird 32-bit
Issue Links:

 Description  « Hide
While restoring table with array fields, Firebird consumes all available memory and restore fails with error "unable to allocate memory from operating system". Same backup can be successfully restored by Firebird 2.1.4. DDL of the table is:

    S1 DECIMAL(18,2),
    S16 DECIMAL(18,2),
    R1 DECIMAL(18,2) [1:12],
    R4 DECIMAL(18,2) [1:4],
    R5 DECIMAL(18,2) [1:4],
    R6 DECIMAL(18,2) [1:4],
    VK1 COMPUTED BY (R1[1]+R1[2]+R1[3]),
    VK2 COMPUTED BY (R1[4]+R1[5]+R1[6]),
    VK3 COMPUTED BY (R1[7]+R1[8]+R1[9]),
    VK4 COMPUTED BY (R1[10]+R1[11]+R1[12]),
    VG COMPUTED BY (R1[1]+R1[2]+R1[3]+R1[4]+R1[5]+R1[6]+R1[7]+R1[8]+R1[9]+R1[10]+R1[11]+R1[12]),
    R3 INTEGER [1:4]


Table has 5454042 rows. Error occurs after 2560000 records restored.
To reproduce issue one can download backup from this URL:

 All   Comments   Change History   Subversion Commits      Sort Order: Ascending order - Click to sort in descending order
Dmitry Yemanov added a comment - 30/Mar/12 05:48 AM
It can be reproduced with 2.5.1 as well, hence I've changed the subject to mention the already released version instead of a snapshot build.

Dmitry Yemanov added a comment - 30/Mar/12 07:31 AM
This is a regression introduced while implementing CORE-3433.

Dmitry Yemanov added a comment - 30/Mar/12 07:54 AM
Usage of isc_send() instead of isc_start_and_send() forces the looper to always exit with a non-NULL node, so release_blobs() is never called and thus blobs/arrays bound to the current request are never released, at least until the transaction ends. Hence we have a transaction level memory leak.

The dumb but easiest solution could be to restart the request periodically (e.g. if (records % 1000 == 1)) to keep some kind of a balance between memory usage and performance. Or set some flag if we have blobs/arrays stored right before this record and use that flag to restart the request instead of just sending the record. Or combine these two approaches.

Ideally, the engine should be fixed, but this sounds too risky for point releases.

Adriano dos Santos Fernandes added a comment - 31/Mar/12 01:30 AM
That's weird. I tested latest 2.5 in release-mode, with and without CORE-3433 change.

Around 2,400,000 records I verified and in both cases top says virtual memory consumption is around 370M.

I can restore it in Linux 64 bits.

Without CORE-3433, it takes "3m6.838s" and with it "2m53.671s".

Dmitry Yemanov added a comment - 31/Mar/12 03:56 AM - edited
With a 32-bit FB 2.5.2 (release mode) on 64-bit Windows I see VM = 750MB after 1,000,000 records restored, 1.5GB after 2,000,000 records and the OOM error happens at 2,262,000 records. 64-bit FB consumes a bit more memory but unsurprisingly succeeds to restore.

Just for the record, I was restoring using the Services API.