Issue Details (XML | Word | Printable)

Key: PYFB-43
Type: Bug Bug
Status: Closed Closed
Resolution: Fixed
Priority: Blocker Blocker
Assignee: Pavel Cisar
Reporter: Dominik Psenner
Votes: 0
Watchers: 0
Operations

If you were logged in you would be able to see more operations.
Firebird driver for Python

Memory leak when using event_conduit()

Created: 07/Nov/14 09:51 AM   Updated: 17/Nov/14 07:38 AM
Component/s: None
Affects Version/s: 1.4.1
Fix Version/s: 1.4.3

File Attachments: 1. File brokenDbEvents.py (2 kB)

Image Attachments:

1. EventBlock-0.png
(72 kB)

2. EventBlock-19.png
(83 kB)
Environment: Any, but was reproduced with WinXP 32bit and Win7 64bit


 Description  « Hide
A memory leak in the class EventBlock(object), defined in dbcore.py at line 1687 most probably caused by the cyclic reference introduced in its constructor at lines 1705 and 1709:

1705: self.__queue.put((ibase.OP_RECORD_AND_REREGISTER,self))
1709: self.__queue = queue

See here for a more detailed description:

https://groups.yahoo.com/neo/groups/firebird-python/conversations/messages/742

This may be related to the issue outlined at https://groups.yahoo.com/neo/groups/firebird-python/conversations/messages/742 . Sometimes the event_conduit(events) invoke raises exceptions in the EventBlock object and thus it is impossible to invoke close() on the event_conduit that has been created. The exceptions observed so far are:

    event_listener = con.event_conduit(eventNames)
  File "C:\Python27\lib\site-packages\fdb-1.4.1-py2.7.egg\fdb\fbcore.py", line 1601, in event_conduit
    conduit = EventConduit(self._db_handle,event_names)
  File "C:\Python27\lib\site-packages\fdb-1.4.1-py2.7.egg\fdb\fbcore.py", line 1811, in __init__
    self.__event_blocks.append(EventBlock(self.__queue,db_handle,block_events))
  File "C:\Python27\lib\site-packages\fdb-1.4.1-py2.7.egg\fdb\fbcore.py", line 1726, in __init__
    self.__wait_for_events()
  File "C:\Python27\lib\site-packages\fdb-1.4.1-py2.7.egg\fdb\fbcore.py", line 1735, in __wait_for_events
    "Error while waiting for events:")
DatabaseError: ('Error while waiting for events:\n- SQLCODE: -902\n- Error reading data from the connection.', -902, 335544726)

    event_listener = con.event_conduit(eventNames)
  File "C:\Python27\lib\site-packages\fdb-1.4.1-py2.7.egg\fdb\fbcore.py", line 1601, in event_conduit
    conduit = EventConduit(self._db_handle,event_names)
  File "C:\Python27\lib\site-packages\fdb-1.4.1-py2.7.egg\fdb\fbcore.py", line 1811, in __init__
    self.__event_blocks.append(EventBlock(self.__queue,db_handle,block_events))
  File "C:\Python27\lib\site-packages\fdb-1.4.1-py2.7.egg\fdb\fbcore.py", line 1726, in __init__
    self.__wait_for_events()
  File "C:\Python27\lib\site-packages\fdb-1.4.1-py2.7.egg\fdb\fbcore.py", line 1735, in __wait_for_events
    "Error while waiting for events:")
DatabaseError: ('Error while waiting for events:\n- SQLCODE: -902\n- Error writing data to the connection.', -902, 335544727)

  File "dbEvents.py", line 90, in run
    event_listener = con.event_conduit(eventNames)
  File "C:\Python27\lib\site-packages\fdb-1.4.1-py2.7.egg\fdb\fbcore.py", line 1601, in event_conduit
    conduit = EventConduit(self._db_handle,event_names)
  File "C:\Python27\lib\site-packages\fdb-1.4.1-py2.7.egg\fdb\fbcore.py", line 1811, in __init__
    self.__event_blocks.append(EventBlock(self.__queue,db_handle,block_events))
  File "C:\Python27\lib\site-packages\fdb-1.4.1-py2.7.egg\fdb\fbcore.py", line 1726, in __init__
    self.__wait_for_events()
  File "C:\Python27\lib\site-packages\fdb-1.4.1-py2.7.egg\fdb\fbcore.py", line 1735, in __wait_for_events
    "Error while waiting for events:")
DatabaseError: ('Error while waiting for events:\n- SQLCODE: 0\n- unknown ISC error 0', 0, 0)

This issue relates also to http://tracker.firebirdsql.org/browse/PYFB-41.

 All   Comments   Change History   Subversion Commits      Sort Order: Ascending order - Click to sort in descending order
Dominik Psenner added a comment - 07/Nov/14 09:56 AM
The file brokenDbEvents.py does reproduce the memory leak and the images EventBlock-X.png show samples of the cyclic references that cannot be resolved by the garbage collector.

Pavel Cisar added a comment - 13/Nov/14 12:30 PM
This is actually not a bug but pitfall in implementation. Developers must catch exceptions thrown in response to errors reported by Firebird and discard now defunct EventConduit. Starting from version 1.5.2 EventConduit doesn't immediately listens to events upon creation, so such exceptions are not thrown by constructor. Instead it's necessary to call begin() method or use newly introduced context manager protocol support for that.

Dominik Psenner added a comment - 13/Nov/14 12:35 PM
Great to hear this was resolved so quickly. Do you have a link to the context manager protocol documentation ready at hand?

Dominik Psenner added a comment - 14/Nov/14 09:55 AM
I'm afraid that this issue is not yet resolved with 1.4.2. The EventBlock class still holds a cyclic reference to itself and thus prevents the garbage collector from cleaning it up. To test it, I created n event_conduits and subsequently disposed them:

# run this with pdb
con = fdb.connect(...)
for i in range(0, 200):
 eventNames = ['test']
 with con.event_conduit(eventNames) as event_listener:
  events = event_listener.wait(timeout)

Then I examined the memory with pdb to determine the instances that are still around and it was exactly the same number I had created:

# run this in pdb after the previous one has stopped
import gc
gc.collect()
import objgraph
len(objgraph.by_type('EventBlock'))
200

The objgraph produced with objgraph.show_backrefs() is exactly the same as I have already posted in this issue.

Pavel Cisar added a comment - 14/Nov/14 10:44 AM
Previous fix was incomplete, corrected.

Dominik Psenner added a comment - 14/Nov/14 12:20 PM
The good thing is, 1.4.3 improved the situation. One cyclic reference has gone away but there's still one cyclic reference left to be broken. I added the following on line 1761 in fbcore.py and the memory leak is history:

     def close(self):
         "Close this block canceling managed events."
         if not self.closed:
             api.isc_cancel_events(self._isc_status,self._db_handle,self.event_id)
+ self.__callback = None
             self.__closed = True
             if db_api_error(self._isc_status):
                 raise exception_from_status(DatabaseError, self._isc_status,
                                             "Error while canceling events:")

Dominik Psenner added a comment - 17/Nov/14 07:38 AM
For the record, the fix from the last comment works fine only in non-exceptional scenarios. But when one of the exceptions listed in the issue description happen, the EventBlock is still not freed.

So far as I understood it, this is mainly because EventBlock's are created and start listening before they were added to the queue. When something fails there before the EventBlock is added to the queue, the EventBlock's close() method is never going to be called and thus it is never garbage collected.