Issue Details (XML | Word | Printable)

Key: CORE-5702
Type: Bug Bug
Status: Resolved Resolved
Resolution: Fixed
Priority: Critical Critical
Assignee: Vlad Khorsun
Reporter: Mayko Sartor
Votes: 0
Watchers: 7
Operations

If you were logged in you would be able to see more operations.
Firebird Core

Firebird Handle Leak Windows

Created: 08/Jan/18 07:23 PM   Updated: 18/Sep/18 06:48 AM
Component/s: Engine
Affects Version/s: 3.0.2
Fix Version/s: 3.0.4, 4.0 Beta 1

File Attachments: None
Image Attachments:

1. 2018_04_02__FB3_states__32912_and_32939.png
(43 kB)

2. FB_2018_04_29__08_27.png
(147 kB)

3. FB_2018_05_03__07_42.png
(143 kB)

4. FB_2018_05_08__08_05.png
(146 kB)

5. FB_2018_05_13__22_29.png
(164 kB)

6. FB_2018_05_16__07_31.png
(159 kB)

7. FB_2018_05_17__07_00.png
(144 kB)

8. handle_leak.png
(27 kB)

9. handle_leak_events.png
(67 kB)

10. plot_handles.png
(18 kB)

11. plot_handles2.png
(21 kB)

12. plot_handles3.png
(22 kB)

13. plot_handles4.png
(19 kB)
Environment: Windows

QA Status: No test


 Description  « Hide
The firebird running on windows has a leak of handles. Even closing all connections to the database the process firebird.exe does not release the allocated handles. Verified by the proccess explorer it is possible to see that the accumulated handles are of type Event, as shown in the image.

 All   Comments   Change History   Subversion Commits      Sort Order: Ascending order - Click to sort in descending order
Vlad Khorsun added a comment - 08/Jan/18 07:29 PM
Could you provide reproducible example ?

Sean Leyne added a comment - 08/Jan/18 07:58 PM
Are you running SuperServer or SuperClassic mode?

How are the connections being closed?

Mayko Sartor added a comment - 09/Jan/18 02:30 AM
I'm running Superserver. My software use the UniDac componentes to connect to firebird with connection pooling enabled. We set the component to release the connections every 4 minutes. So every open connection execute many querys for 4 minutes and then component closes it.
I tried to reproduce the error through a sample application but I could not. The error does not follow a pattern, simply the handles will accumulate after the days until reaching the limit. I can provide access to a virtual machine on windows azure where the process is running with many accumulated handles.

Mayko Sartor added a comment - 09/Jan/18 02:33 AM
I forgot to say that in superclassic mode the error also occurs.

Vlad Khorsun added a comment - 09/Jan/18 10:35 AM
Could you create full memory dump of process (firebird.exe) and make it available for download ?

Mayko Sartor added a comment - 09/Jan/18 04:22 PM
Link dump:

https://drive.google.com/open?id=1NSOguJyl66bl3Ienlxa7RBDxSzJSd9Au

Generated with Sysinternals ProcDump

Vlad Khorsun added a comment - 09/Jan/18 05:17 PM
This dump contains no useful information.
Is it really *full memory* dump ? It is too small, just 2MB.
Does you set -ma parameter for procdump ?

Mayko Sartor added a comment - 09/Jan/18 05:37 PM

Vlad Khorsun added a comment - 10/Jan/18 04:20 PM
Thanks, last dump is good for analyze.
Anyway, i see no reason for events leakage there.
Afraid, without test case it is close to impossible to find a reason for it.

Or, if you familiar with WinDbg - it could show stack trace when event(s) was created, see

https://blogs.technet.microsoft.com/yongrhee/2011/12/19/how-to-troubleshoot-a-handle-leak/

Vlad Khorsun added a comment - 28/Jan/18 09:55 PM
Is there any news on this issue ?

Christian Geldmann added a comment - 07/Mar/18 04:15 PM
No news?

Vlad Khorsun added a comment - 07/Mar/18 04:41 PM
I have no news

Christian Geldmann added a comment - 08/Mar/18 08:13 AM
Do you know possible reasons for increasing (event) handles in firebird.exe? I had the same problem when I upgraded from FB1.5.6 to FB3.0.1 (without changing anything in my application). Everything works fine except the increasing handles.

Vlad Khorsun added a comment - 08/Mar/18 08:25 AM
Show me the issue and i will investigate and explain it or fix it.
So far there is no sign of a real problem.

Do you use SuperServer ?
What is page cache size ?
How many handles you see ?
How long Firebird instance is running ?
Does handles count increased all the time or stop at some number ?

Christian Geldmann added a comment - 08/Mar/18 08:41 AM

Do you use SuperServer ? => Yes, but also tried SuperClassic

What is page cache size ? => Page Size 4096 / Page Buffers 2048

How many handles you see ? => depending on how long the system was running. Max. was > 80.000

How long Firebird instance is running ? => ca. 4 Month

Does handles count increased all the time or stop at some number ? => increasing all the time

Vlad Khorsun added a comment - 13/Mar/18 05:46 PM
Please, check next snapshot build of v3.0.4

Christian Geldmann added a comment - 16/Mar/18 11:28 AM
From first observations the handles still increase.

Vlad Khorsun added a comment - 16/Mar/18 12:43 PM - edited
Number of handles - much more than number of pages in page cache ?

Christian Geldmann added a comment - 16/Mar/18 04:14 PM
I was too quick with my answer......it seems to be stable now!

Vlad Khorsun added a comment - 17/Mar/18 08:00 AM
Sounds good :) Lets wait one-two days more to finally confirm fix.

Christian Geldmann added a comment - 19/Mar/18 07:43 AM
Well ok, after three days, i see that the handles did increase, but much slower than it did before. Reduced by ca. factor 10.

Sean Leyne added a comment - 19/Mar/18 11:11 AM
Are you saying that the number of handles is still increasing but at a slower rate? Or, the number is stable?

What is the current number?

Christian Geldmann added a comment - 19/Mar/18 12:22 PM
Still increasing at slower rate. Startet last friday with 400 handles, now 900.

Vlad Khorsun added a comment - 19/Mar/18 01:01 PM
Every page cache buffer could consume event handle.
So, so far all is under control, i'd say.

Sean Leyne added a comment - 19/Mar/18 05:10 PM
I could understand a handle for the page cache as a whole, but each page having it's own handle seems odd.

Is there a formula that can be used to determine if the number of handles is too large? (i.e. expected handles <= Cache page + active DB connections + ???)

Otherwise, how would anyone know if they had a problem or not?

Vlad Khorsun added a comment - 19/Mar/18 05:22 PM
> I could understand a handle for the page cache as a whole, but each page having it's own handle seems odd.

Handle *could* be allocated when there is concurrency at this page buffer.

> Is there a formula that can be used to determine if the number of handles is too large? (i.e. expected handles <= Cache page + active DB connections + ???)

No

> Otherwise, how would anyone know if they had a problem or not?

80000 handles for one database with 2048 pages - definitely not good.

Christian Geldmann added a comment - 20/Mar/18 09:56 AM
By the way, is there a possibility to find out if the cache is completely used (e.g. using a page buffer of 2048, and 2048 pages are currently in cache) ?

Vlad Khorsun added a comment - 20/Mar/18 12:22 PM
There is no such possibility, sorry

Vlad Khorsun added a comment - 31/Mar/18 09:09 PM
Christian,

how it is going ? Could we close the ticket ?

Kovalenko Dmitry added a comment - 02/Apr/18 08:52 AM
Seems, leaks was reduced.

See 2018_04_02__FB3_states__32912_and_32939 image.

Through 4 days I may get new state for 32939 build.

Christian Geldmann added a comment - 03/Apr/18 01:11 PM
My application has ca. 3500 handles for firebird.exe after 15 days. Started with 400 handles.
The leak is reduced drastically, but I also like to watch a few more days if it stabilizes now.

Vlad Khorsun added a comment - 05/Apr/18 08:19 AM
Christian,

one more leak was detected (it is rare enough, but anyway) and there was additional patch committed.
Please, test last snapshot build (3.0.4.32942)

Christian Geldmann added a comment - 06/Apr/18 11:58 AM
Ok....thanks! I restarted my app today using 3.0.4.32944. I'll keep you informed.

Vlad Khorsun added a comment - 16/Apr/18 09:35 AM
Christian,

any news after 10 days ? ;)

Christian Geldmann added a comment - 16/Apr/18 02:32 PM
Hi,

I see ca. 3200 handles after running ten days. The handles are still increasing.
No big difference to previous version.

Vlad Khorsun added a comment - 23/Apr/18 08:17 PM
Hi Christian,

how many handles now ?

PS There was one more patch for even more rare leak, btw

Christian Geldmann added a comment - 24/Apr/18 06:42 AM
3700 now. The increase rate is much lower. I will check with the new version now.

Kovalenko Dmitry added a comment - 03/May/18 04:47 AM
For me - now (v3.0.4.32963, SS, x64, Win) all is ok.

Christian Geldmann added a comment - 07/May/18 07:14 AM
For me, handles still increase :-(

Vlad Khorsun added a comment - 07/May/18 07:52 AM
Christian,

how fast it increases ? how many handles now ?

Christian Geldmann added a comment - 07/May/18 09:13 AM
see image

Vlad Khorsun added a comment - 07/May/18 09:23 AM
Firebird 3.0.1 ? Seriously ?

Christian Geldmann added a comment - 07/May/18 09:47 AM
Yes.
3.0.1.32609 (x64) vs. 4.0.0.716 (x64)

Vlad Khorsun added a comment - 07/May/18 10:03 AM
So, you complain that old builds leak the handles ?

Christian Geldmann added a comment - 07/May/18 10:34 AM
?...No. I just did a comparison. If you want, neglect the 3.0.1 plot.

Vlad Khorsun added a comment - 07/May/18 10:46 AM
But what you are compare ?
Let 3.0.1 is a "base line", ok.
But why 4.0.0.716 ??? It is too old build, current is 974.

What i am missing ?

Christian Geldmann added a comment - 07/May/18 07:38 PM
Oops....you're right. My fault. Wrong version installed. I will restart my comparison with the newest snapshot and keep you informed.

Christian Geldmann added a comment - 15/May/18 06:10 AM
Another handles-plot

Vlad Khorsun added a comment - 15/May/18 07:32 AM
The very first fix was not committed into master. Done now. Thanks

Christian Geldmann added a comment - 15/May/18 08:44 AM
Already fixed in 4.0.0.978, or next snapshot tomorrow?

Vlad Khorsun added a comment - 15/May/18 08:51 AM
Next snapshot after 4.0.0.978, i.e. tomorrow

Kovalenko Dmitry added a comment - 16/May/18 05:52 AM
Load test of 3.0.4.32972 (SS, x64, Win10) at system with 128GB

After first iteration - 229 descritions (expected 220)
After second iteration - 230 descriptions

Just for info.

Christian Geldmann added a comment - 17/May/18 06:37 AM
See next plot. Much better now, but there still seem to be an increase of handles.

Vlad Khorsun added a comment - 17/May/18 06:49 AM
Is there a moment when all attachments gone at the last graph ?

Christian Geldmann added a comment - 17/May/18 06:58 AM
I don't know if I understand......what do you mean with "attachments gone"?

Vlad Khorsun added a comment - 17/May/18 07:06 AM
I mean - no attachments.
I.e. moment, when there is no attachments and Firebird able to release resources such as page cache etc

Christian Geldmann added a comment - 17/May/18 07:22 AM
Hmm...no, I don't think so.

Kovalenko Dmitry added a comment - 17/May/18 07:29 AM
231 descriptors (+1 to previous point)

Vlad Khorsun added a comment - 17/May/18 07:36 AM
Christian,

do you expect that Firebird could allocate all required handles\memory at start and never allocate something else ?
I already wrote here that every page cache buffer could consume event handle in some condition.
With page cache of 2048 pages we could expect at least 2300 handles. Could be more but not too much.
Most of them should be released on full disconnect.

Christian Geldmann added a comment - 17/May/18 08:02 AM
Yes I know. I'll continue the test. If it stops at < 3000 handles, everything's ok.

Kovalenko Dmitry added a comment - 18/May/18 06:07 PM
FB 3.0.4.32672, SS, x64, Win10.

I see leak of 1 (one) descriptor after each execution of load test.

Happy debugging :)

Christian Geldmann added a comment - 23/May/18 07:41 AM - edited
For me, at the moment, it seems to be ok (see new plot).
@Vlad: Newest Snapshots of Firebird 3 also contain that bugfix?
              And will there be an official release soon containing the bugfix?

Simonov Denis added a comment - 23/May/18 07:51 AM
Christian Geldmann,
Try last snapshot Firebird-3.0.4.32977
To release an official release, you need to test it.

Vlad Khorsun added a comment - 23/May/18 07:54 AM
Christian,

thanks for testing and for confirmation.
All current and newest snapshots contains this fix, sure.
Can't say when 3.0.4 will be released, sorry.
Usually it happens 2 times a year (for actively supported version).

Christian Geldmann added a comment - 18/Sep/18 06:37 AM
Hi, are there any news regarding the new firebird version (3.0.4) release date?

Dmitry Yemanov added a comment - 18/Sep/18 06:48 AM
It's being prepared these days, so you may expect the release quite soon.