Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Firebird Handle Leak Windows [CORE5702] #5968

Closed
firebird-automations opened this issue Jan 8, 2018 · 78 comments
Closed

Firebird Handle Leak Windows [CORE5702] #5968

firebird-automations opened this issue Jan 8, 2018 · 78 comments

Comments

@firebird-automations
Copy link
Collaborator

Submitted by: Mayko Sartor (maykosartor)

Attachments:
handle_leak_events.png
handle_leak.png
2018_04_02__FB3_states__32912_and_32939.png
FB_2018_04_29__08_27.png
FB_2018_05_03__07_42.png
plot_handles.png
FB_2018_05_08__08_05.png
plot_handles2.png
FB_2018_05_13__22_29.png
FB_2018_05_16__07_31.png
plot_handles3.png
FB_2018_05_17__07_00.png
plot_handles4.png

The firebird running on windows has a leak of handles. Even closing all connections to the database the process firebird.exe does not release the allocated handles. Verified by the proccess explorer it is possible to see that the accumulated handles are of type Event, as shown in the image.

Commits: e5ba540 d5fa522 ad0580e 0cc3d04 0e96bbf 64b6de0 07a9515 4fa1bf4

@firebird-automations
Copy link
Collaborator Author

Modified by: Mayko Sartor (maykosartor)

Attachment: handle_leak_events.png [ 13191 ]

Attachment: handle_leak.png [ 13192 ]

@firebird-automations
Copy link
Collaborator Author

Commented by: @hvlad

Could you provide reproducible example ?

@firebird-automations
Copy link
Collaborator Author

Commented by: Sean Leyne (seanleyne)

Are you running SuperServer or SuperClassic mode?

How are the connections being closed?

@firebird-automations
Copy link
Collaborator Author

Commented by: Mayko Sartor (maykosartor)

I'm running Superserver. My software use the UniDac componentes to connect to firebird with connection pooling enabled. We set the component to release the connections every 4 minutes. So every open connection execute many querys for 4 minutes and then component closes it.
I tried to reproduce the error through a sample application but I could not. The error does not follow a pattern, simply the handles will accumulate after the days until reaching the limit. I can provide access to a virtual machine on windows azure where the process is running with many accumulated handles.

@firebird-automations
Copy link
Collaborator Author

Commented by: Mayko Sartor (maykosartor)

I forgot to say that in superclassic mode the error also occurs.

@firebird-automations
Copy link
Collaborator Author

Commented by: @hvlad

Could you create full memory dump of process (firebird.exe) and make it available for download ?

@firebird-automations
Copy link
Collaborator Author

Commented by: Mayko Sartor (maykosartor)

Link dump:

https://drive.google.com/open?id=1NSOguJyl66bl3Ienlxa7RBDxSzJSd9Au

Generated with Sysinternals ProcDump

@firebird-automations
Copy link
Collaborator Author

Commented by: @hvlad

This dump contains no useful information.
Is it really *full memory* dump ? It is too small, just 2MB.
Does you set -ma parameter for procdump ?

@firebird-automations
Copy link
Collaborator Author

Commented by: Mayko Sartor (maykosartor)

Link with -ma option:

https://drive.google.com/open?id=1bFDohuu8oVRJ7Zs9wEUqXAfz4l3Urnnh

@firebird-automations
Copy link
Collaborator Author

Commented by: @hvlad

Thanks, last dump is good for analyze.
Anyway, i see no reason for events leakage there.
Afraid, without test case it is close to impossible to find a reason for it.

Or, if you familiar with WinDbg - it could show stack trace when event(s) was created, see

https://blogs.technet.microsoft.com/yongrhee/2011/12/19/how-to-troubleshoot-a-handle-leak/

@firebird-automations
Copy link
Collaborator Author

Commented by: @hvlad

Is there any news on this issue ?

@firebird-automations
Copy link
Collaborator Author

Commented by: Christian Geldmann (geldo79)

No news?

@firebird-automations
Copy link
Collaborator Author

Commented by: @hvlad

I have no news

@firebird-automations
Copy link
Collaborator Author

Commented by: Christian Geldmann (geldo79)

Do you know possible reasons for increasing (event) handles in firebird.exe? I had the same problem when I upgraded from FB1.5.6 to FB3.0.1 (without changing anything in my application). Everything works fine except the increasing handles.

@firebird-automations
Copy link
Collaborator Author

Commented by: @hvlad

Show me the issue and i will investigate and explain it or fix it.
So far there is no sign of a real problem.

Do you use SuperServer ?
What is page cache size ?
How many handles you see ?
How long Firebird instance is running ?
Does handles count increased all the time or stop at some number ?

@firebird-automations
Copy link
Collaborator Author

Commented by: Christian Geldmann (geldo79)

Do you use SuperServer ? => Yes, but also tried SuperClassic

What is page cache size ? => Page Size 4096 / Page Buffers 2048

How many handles you see ? => depending on how long the system was running. Max. was > 80.000

How long Firebird instance is running ? => ca. 4 Month

Does handles count increased all the time or stop at some number ? => increasing all the time

@firebird-automations
Copy link
Collaborator Author

Commented by: @hvlad

Please, check next snapshot build of v3.0.4

@firebird-automations
Copy link
Collaborator Author

Modified by: @hvlad

assignee: Vlad Khorsun [ hvlad ]

@firebird-automations
Copy link
Collaborator Author

Commented by: Christian Geldmann (geldo79)

From first observations the handles still increase.

@firebird-automations
Copy link
Collaborator Author

Commented by: @hvlad

Number of handles - much more than number of pages in page cache ?

@firebird-automations
Copy link
Collaborator Author

Commented by: Christian Geldmann (geldo79)

I was too quick with my answer......it seems to be stable now!

@firebird-automations
Copy link
Collaborator Author

Commented by: @hvlad

Sounds good :) Lets wait one-two days more to finally confirm fix.

@firebird-automations
Copy link
Collaborator Author

Commented by: Christian Geldmann (geldo79)

Well ok, after three days, i see that the handles did increase, but much slower than it did before. Reduced by ca. factor 10.

@firebird-automations
Copy link
Collaborator Author

Commented by: Sean Leyne (seanleyne)

Are you saying that the number of handles is still increasing but at a slower rate? Or, the number is stable?

What is the current number?

@firebird-automations
Copy link
Collaborator Author

Commented by: Christian Geldmann (geldo79)

Still increasing at slower rate. Startet last friday with 400 handles, now 900.

@firebird-automations
Copy link
Collaborator Author

Commented by: @hvlad

Every page cache buffer could consume event handle.
So, so far all is under control, i'd say.

@firebird-automations
Copy link
Collaborator Author

Commented by: Sean Leyne (seanleyne)

I could understand a handle for the page cache as a whole, but each page having it's own handle seems odd.

Is there a formula that can be used to determine if the number of handles is too large? (i.e. expected handles <= Cache page + active DB connections + ???)

Otherwise, how would anyone know if they had a problem or not?

@firebird-automations
Copy link
Collaborator Author

Commented by: @hvlad

> I could understand a handle for the page cache as a whole, but each page having it's own handle seems odd.

Handle *could* be allocated when there is concurrency at this page buffer.

> Is there a formula that can be used to determine if the number of handles is too large? (i.e. expected handles <= Cache page + active DB connections + ???)

No

> Otherwise, how would anyone know if they had a problem or not?

80000 handles for one database with 2048 pages - definitely not good.

@firebird-automations
Copy link
Collaborator Author

Commented by: Christian Geldmann (geldo79)

By the way, is there a possibility to find out if the cache is completely used (e.g. using a page buffer of 2048, and 2048 pages are currently in cache) ?

@firebird-automations
Copy link
Collaborator Author

Commented by: @hvlad

There is no such possibility, sorry

@firebird-automations
Copy link
Collaborator Author

Commented by: @hvlad

Next snapshot after 4.0.0.978, i.e. tomorrow

@firebird-automations
Copy link
Collaborator Author

Commented by: @ibprovider

Load test of 3.0.4.32972 (SS, x64, Win10) at system with 128GB

After first iteration - 229 descritions (expected 220)
After second iteration - 230 descriptions

Just for info.

@firebird-automations
Copy link
Collaborator Author

Modified by: @ibprovider

Attachment: FB_2018_05_13__22_29.png [ 13249 ]

Attachment: FB_2018_05_16__07_31.png [ 13250 ]

@firebird-automations
Copy link
Collaborator Author

Commented by: Christian Geldmann (geldo79)

See next plot. Much better now, but there still seem to be an increase of handles.

@firebird-automations
Copy link
Collaborator Author

Modified by: Christian Geldmann (geldo79)

Attachment: plot_handles3.png [ 13251 ]

@firebird-automations
Copy link
Collaborator Author

Commented by: @hvlad

Is there a moment when all attachments gone at the last graph ?

@firebird-automations
Copy link
Collaborator Author

Commented by: Christian Geldmann (geldo79)

I don't know if I understand......what do you mean with "attachments gone"?

@firebird-automations
Copy link
Collaborator Author

Commented by: @hvlad

I mean - no attachments.
I.e. moment, when there is no attachments and Firebird able to release resources such as page cache etc

@firebird-automations
Copy link
Collaborator Author

Commented by: Christian Geldmann (geldo79)

Hmm...no, I don't think so.

@firebird-automations
Copy link
Collaborator Author

Commented by: @ibprovider

231 descriptors (+1 to previous point)

@firebird-automations
Copy link
Collaborator Author

Modified by: @ibprovider

Attachment: FB_2018_05_17__07_00.png [ 13252 ]

@firebird-automations
Copy link
Collaborator Author

Commented by: @hvlad

Christian,

do you expect that Firebird could allocate all required handles\memory at start and never allocate something else ?
I already wrote here that every page cache buffer could consume event handle in some condition.
With page cache of 2048 pages we could expect at least 2300 handles. Could be more but not too much.
Most of them should be released on full disconnect.

@firebird-automations
Copy link
Collaborator Author

Commented by: Christian Geldmann (geldo79)

Yes I know. I'll continue the test. If it stops at < 3000 handles, everything's ok.

@firebird-automations
Copy link
Collaborator Author

Commented by: @ibprovider

FB 3.0.4.32672, SS, x64, Win10.

I see leak of 1 (one) descriptor after each execution of load test.

Happy debugging :)

@firebird-automations
Copy link
Collaborator Author

Commented by: Christian Geldmann (geldo79)

For me, at the moment, it seems to be ok (see new plot).
@vlad: Newest Snapshots of Firebird 3 also contain that bugfix?
And will there be an official release soon containing the bugfix?

@firebird-automations
Copy link
Collaborator Author

Modified by: Christian Geldmann (geldo79)

Attachment: plot_handles4.png [ 13255 ]

@firebird-automations
Copy link
Collaborator Author

Commented by: @sim1984

Christian Geldmann,
Try last snapshot Firebird-3.0.4.32977
To release an official release, you need to test it.

@firebird-automations
Copy link
Collaborator Author

Commented by: @hvlad

Christian,

thanks for testing and for confirmation.
All current and newest snapshots contains this fix, sure.
Can't say when 3.0.4 will be released, sorry.
Usually it happens 2 times a year (for actively supported version).

@firebird-automations
Copy link
Collaborator Author

Modified by: @hvlad

status: Open [ 1 ] => Resolved [ 5 ]

resolution: Fixed [ 1 ]

Fix Version: 4.0 Beta 1 [ 10750 ]

Fix Version: 3.0.4 [ 10863 ]

@firebird-automations
Copy link
Collaborator Author

Commented by: Christian Geldmann (geldo79)

Hi, are there any news regarding the new firebird version (3.0.4) release date?

@firebird-automations
Copy link
Collaborator Author

Commented by: @dyemanov

It's being prepared these days, so you may expect the release quite soon.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

2 participants