New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Severe performance loss and minor memory leak [PYFB58] #75
Comments
Commented by: Dominik Psenner (dpsenner) # -*- coding: utf-8 -*- import sys # Auxiliary method
def test(connection): if __name__ == '__main__': |
Commented by: Dominik Psenner (dpsenner) We found an alternative that could fix the issue without removing the weakrefs. This is the patch: # HG changeset patch _weakref_callback was never called and thus never cleaned up the weakref in the diff -r d36d56a6ad66 -r c71fa20acbc9 fdb/fbcore.py For now we apply this patch to ship a bugfix release. Would you please check and feed back if this fix is safe? |
Commented by: Dominik Psenner (dpsenner) Are there any updates on this issue? |
Commented by: @pcisar Suggested fix was not good and introduced another problems. Proper fix was to make weakref callback work as expected. |
Modified by: @pcisarstatus: Open [ 1 ] => Resolved [ 5 ] resolution: Fixed [ 1 ] Fix Version: 1.4.11 [ 10741 ] |
Modified by: @pcisarstatus: Resolved [ 5 ] => Closed [ 6 ] |
Submitted by: Dominik Psenner (dpsenner)
We implemented a connection pool and due to heavy load connections are practically never freed. We soon noticed significant performance loss when executing millions of queries on a single connection. In our test we observed that executing a random query (select first 1 * from rdb$database) got us these timings:
first 10000 queries: 17 seconds
next 10000 queries: 30 seconds
next 10000 queries: 42 seconds
next 10000 queries: 53 seconds
The transaction is committed after every query. Soon we found out that the transaction class holds references to all its cursors with a weakref:
class Transaction(object):
def __init__:
self._cursors = [] # Weak references to cursors
When creating a cursor the weakref to that cursor is stored in this list and should be freed as soon as the cursor is no longer in use. But unfortunately this is not the case. Only when the transaction is closed also the cursors are freed up. We were able to narrow down the issue to this line:
class Transaction(object):
def cursor(self,connection = None):
[...]
self._cursors.append(weakref.ref(c, _weakref_callback(self.__remove_cursor)))
[...]
This line adds a weakref to the cursor with a weakref callback, but that weakref callback is immediately freed and thus self.__remove_cursor is never invoked. This has the effect that when def __close_cursors(self) is called it has to cycle over millions of weakrefs that point to nowhere. Changing this to:
self._cursors.append(weakref.ref(c, self.__remove_cursor))
solved the performance issue but we do not know if this results in other issues. You might know a better way to solve this. Generally we do not understand why there is the need to keep weakrefs to cursors that have been created. Anyway, this is a blocker for us, please try to work out a fix as soon as you can.
Commits: a7daea5 FirebirdSQL/fbt-repository@b08b0dc
The text was updated successfully, but these errors were encountered: