New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Memory leak when using event_conduit() [PYFB43] #62
Comments
Commented by: Dominik Psenner (dpsenner) The file http://brokenDbEvents.py does reproduce the memory leak and the images EventBlock-X.png show samples of the cyclic references that cannot be resolved by the garbage collector. |
Modified by: Dominik Psenner (dpsenner)Attachment: http://brokenDbEvents.py [ 12620 ] Attachment: EventBlock-0.png [ 12621 ] Attachment: EventBlock-19.png [ 12622 ] |
Modified by: Dominik Psenner (dpsenner)description: A memory leak in the class EventBlock(object), defined in http://dbcore.py at line 1687 most probably caused by the cyclic reference introduced in its constructor at lines 1705 and 1709: 1705: self.__queue.put((ibase.OP_RECORD_AND_REREGISTER,self)) See here for a more detailed description: https://groups.yahoo.com/neo/groups/firebird-python/conversations/messages/742 => A memory leak in the class EventBlock(object), defined in http://dbcore.py at line 1687 most probably caused by the cyclic reference introduced in its constructor at lines 1705 and 1709: 1705: self.__queue.put((ibase.OP_RECORD_AND_REREGISTER,self)) See here for a more detailed description: https://groups.yahoo.com/neo/groups/firebird-python/conversations/messages/742 This may be related to the issue outlined at https://groups.yahoo.com/neo/groups/firebird-python/conversations/messages/742 . Sometimes the event_conduit(events) invoke raises exceptions in the EventBlock object and thus it is impossible to invoke close() on the event_conduit that has been created. The exceptions observed so far are:
File "C:\Python27\lib\site-packages\fdb-1.4.1-py2.7.egg\fdb\http://fbcore.py", line 1601, in event_conduit
File "C:\Python27\lib\site-packages\fdb-1.4.1-py2.7.egg\fdb\http://fbcore.py", line 1601, in event_conduit File "http://dbEvents.py", line 90, in run |
Modified by: Dominik Psenner (dpsenner)description: A memory leak in the class EventBlock(object), defined in http://dbcore.py at line 1687 most probably caused by the cyclic reference introduced in its constructor at lines 1705 and 1709: 1705: self.__queue.put((ibase.OP_RECORD_AND_REREGISTER,self)) See here for a more detailed description: https://groups.yahoo.com/neo/groups/firebird-python/conversations/messages/742 This may be related to the issue outlined at https://groups.yahoo.com/neo/groups/firebird-python/conversations/messages/742 . Sometimes the event_conduit(events) invoke raises exceptions in the EventBlock object and thus it is impossible to invoke close() on the event_conduit that has been created. The exceptions observed so far are:
File "C:\Python27\lib\site-packages\fdb-1.4.1-py2.7.egg\fdb\http://fbcore.py", line 1601, in event_conduit
File "C:\Python27\lib\site-packages\fdb-1.4.1-py2.7.egg\fdb\http://fbcore.py", line 1601, in event_conduit File "http://dbEvents.py", line 90, in run => A memory leak in the class EventBlock(object), defined in http://dbcore.py at line 1687 most probably caused by the cyclic reference introduced in its constructor at lines 1705 and 1709: 1705: self.__queue.put((ibase.OP_RECORD_AND_REREGISTER,self)) See here for a more detailed description: https://groups.yahoo.com/neo/groups/firebird-python/conversations/messages/742 This may be related to the issue outlined at https://groups.yahoo.com/neo/groups/firebird-python/conversations/messages/742 . Sometimes the event_conduit(events) invoke raises exceptions in the EventBlock object and thus it is impossible to invoke close() on the event_conduit that has been created. The exceptions observed so far are:
File "C:\Python27\lib\site-packages\fdb-1.4.1-py2.7.egg\fdb\http://fbcore.py", line 1601, in event_conduit
File "C:\Python27\lib\site-packages\fdb-1.4.1-py2.7.egg\fdb\http://fbcore.py", line 1601, in event_conduit File "http://dbEvents.py", line 90, in run This issue relates also to PYFB41. |
Commented by: @pcisar This is actually not a bug but pitfall in implementation. Developers must catch exceptions thrown in response to errors reported by Firebird and discard now defunct EventConduit. Starting from version 1.5.2 EventConduit doesn't immediately listens to events upon creation, so such exceptions are not thrown by constructor. Instead it's necessary to call begin() method or use newly introduced context manager protocol support for that. |
Modified by: @pcisarstatus: Open [ 1 ] => Resolved [ 5 ] resolution: Fixed [ 1 ] Fix Version: 1.4.2 [ 10641 ] |
Commented by: Dominik Psenner (dpsenner) Great to hear this was resolved so quickly. Do you have a link to the context manager protocol documentation ready at hand? |
Modified by: @pcisarstatus: Resolved [ 5 ] => Closed [ 6 ] |
Commented by: Dominik Psenner (dpsenner) I'm afraid that this issue is not yet resolved with 1.4.2. The EventBlock class still holds a cyclic reference to itself and thus prevents the garbage collector from cleaning it up. To test it, I created n event_conduits and subsequently disposed them: # run this with pdb Then I examined the memory with pdb to determine the instances that are still around and it was exactly the same number I had created: # run this in pdb after the previous one has stopped The objgraph produced with objgraph.show_backrefs() is exactly the same as I have already posted in this issue. |
Commented by: @pcisar Previous fix was incomplete, corrected. |
Modified by: @pcisarstatus: Reopened [ 4 ] => Resolved [ 5 ] resolution: Fixed [ 1 ] Fix Version: 1.4.3 [ 10643 ] Fix Version: 1.4.2 [ 10641 ] => |
Modified by: @pcisarstatus: Resolved [ 5 ] => Closed [ 6 ] |
Commented by: Dominik Psenner (dpsenner) The good thing is, 1.4.3 improved the situation. One cyclic reference has gone away but there's still one cyclic reference left to be broken. I added the following on line 1761 in http://fbcore.py and the memory leak is history:
+ self.__callback = None |
Commented by: Dominik Psenner (dpsenner) For the record, the fix from the last comment works fine only in non-exceptional scenarios. But when one of the exceptions listed in the issue description happen, the EventBlock is still not freed. So far as I understood it, this is mainly because EventBlock's are created and start listening before they were added to the queue. When something fails there before the EventBlock is added to the queue, the EventBlock's close() method is never going to be called and thus it is never garbage collected. |
Submitted by: Dominik Psenner (dpsenner)
Attachments:
brokenDbEvents.py
EventBlock-0.png
EventBlock-19.png
A memory leak in the class EventBlock(object), defined in http://dbcore.py at line 1687 most probably caused by the cyclic reference introduced in its constructor at lines 1705 and 1709:
1705: self.__queue.put((ibase.OP_RECORD_AND_REREGISTER,self))
1709: self.__queue = queue
See here for a more detailed description:
https://groups.yahoo.com/neo/groups/firebird-python/conversations/messages/742
This may be related to the issue outlined at https://groups.yahoo.com/neo/groups/firebird-python/conversations/messages/742 . Sometimes the event_conduit(events) invoke raises exceptions in the EventBlock object and thus it is impossible to invoke close() on the event_conduit that has been created. The exceptions observed so far are:
File "C:\Python27\lib\site-packages\fdb-1.4.1-py2.7.egg\fdb\http://fbcore.py", line 1601, in event_conduit
conduit = EventConduit(self._db_handle,event_names)
File "C:\Python27\lib\site-packages\fdb-1.4.1-py2.7.egg\fdb\http://fbcore.py", line 1811, in __init__
self.__event_blocks.append(EventBlock(self.__queue,db_handle,block_events))
File "C:\Python27\lib\site-packages\fdb-1.4.1-py2.7.egg\fdb\http://fbcore.py", line 1726, in __init__
self.__wait_for_events()
File "C:\Python27\lib\site-packages\fdb-1.4.1-py2.7.egg\fdb\http://fbcore.py", line 1735, in __wait_for_events
"Error while waiting for events:")
DatabaseError: ('Error while waiting for events:\n- SQLCODE: -902\n- Error reading data from the connection.', -902, 335544726)
File "C:\Python27\lib\site-packages\fdb-1.4.1-py2.7.egg\fdb\http://fbcore.py", line 1601, in event_conduit
conduit = EventConduit(self._db_handle,event_names)
File "C:\Python27\lib\site-packages\fdb-1.4.1-py2.7.egg\fdb\http://fbcore.py", line 1811, in __init__
self.__event_blocks.append(EventBlock(self.__queue,db_handle,block_events))
File "C:\Python27\lib\site-packages\fdb-1.4.1-py2.7.egg\fdb\http://fbcore.py", line 1726, in __init__
self.__wait_for_events()
File "C:\Python27\lib\site-packages\fdb-1.4.1-py2.7.egg\fdb\http://fbcore.py", line 1735, in __wait_for_events
"Error while waiting for events:")
DatabaseError: ('Error while waiting for events:\n- SQLCODE: -902\n- Error writing data to the connection.', -902, 335544727)
File "http://dbEvents.py", line 90, in run
event_listener = con.event_conduit(eventNames)
File "C:\Python27\lib\site-packages\fdb-1.4.1-py2.7.egg\fdb\http://fbcore.py", line 1601, in event_conduit
conduit = EventConduit(self._db_handle,event_names)
File "C:\Python27\lib\site-packages\fdb-1.4.1-py2.7.egg\fdb\http://fbcore.py", line 1811, in __init__
self.__event_blocks.append(EventBlock(self.__queue,db_handle,block_events))
File "C:\Python27\lib\site-packages\fdb-1.4.1-py2.7.egg\fdb\http://fbcore.py", line 1726, in __init__
self.__wait_for_events()
File "C:\Python27\lib\site-packages\fdb-1.4.1-py2.7.egg\fdb\http://fbcore.py", line 1735, in __wait_for_events
"Error while waiting for events:")
DatabaseError: ('Error while waiting for events:\n- SQLCODE: 0\n- unknown ISC error 0', 0, 0)
This issue relates also to PYFB41.
Commits: 23f90ec 65160c3 8de82ce fc5dd5a a3d6310 FirebirdSQL/fbt-repository@5462470 FirebirdSQL/fbt-repository@a54cdd7 FirebirdSQL/fbt-repository@1b5c1cf FirebirdSQL/fbt-repository@83703c1 FirebirdSQL/fbt-repository@7ed6220
The text was updated successfully, but these errors were encountered: