Issue Details (XML | Word | Printable)

Key: JDBC-435
Type: Bug Bug
Status: Closed Closed
Resolution: Fixed
Priority: Major Major
Assignee: Mark Rotteveel
Reporter: Paul Reeves
Votes: 0
Watchers: 0
Operations

If you were logged in you would be able to see more operations.
Jaybird JCA/JDBC Driver

Deadlock in multi-threaded environment

Created: 13/May/16 10:07 AM   Updated: 20/Aug/16 01:13 PM
Component/s: JCA layer, JDBC driver
Affects Version/s: Jaybird 3.0.0
Fix Version/s: Jaybird 2.2.11, Jaybird 3.0.0

File Attachments: None
Image Attachments:

1. deadlock.jpeg
(351 kB)
Issue Links:
Duplicate
 


 Description  « Hide
When closing multiple threaded connections the driver hangs. The problem seems to be related to the test for mc == null in isClosed()

I tried a while ago with 2.2.3 and most recently with the code in the v3.0.0 snapshot. Neither work.

The last version that works for me is 2.1.6.

 All   Comments   Change History   Subversion Commits      Sort Order: Ascending order - Click to sort in descending order
Paul Reeves added a comment - 13/May/16 10:15 AM
Hopefully this screen shot from netbeans will give a better idea of what is happening. If necessary I can provide the app and the database.

Mark Rotteveel added a comment - 13/May/16 12:04 PM
I will investigate this. Having access to the app and database would be very helpful.

As a side note, although JDBC asks that drivers are threadsafe, in general you shouldn't use the same connection from multiple threads simultaneously.

Mark Rotteveel added a comment - 13/May/16 12:22 PM
The problem is not with the null check in isClosed. It is with the fact that the order that executeBatch first acquires a lock on the statement, and later on on the connection, while connection close first acquires a lock on the connection, and then (when closing statements in response to the connection close) on the statement.

Paul Reeves added a comment - 13/May/16 01:07 PM
Just to comment on your side note - no, in the application each thread creates its own connection. With 2.16 I've had up to 100 concurrent threads working without problem.

But good to hear that you may have identified the problem. I can test the fix easily enough when it is ready.

Mark Rotteveel added a comment - 13/May/16 01:42 PM
The screenshot seems to indicate that the connection and statements are shared: Thread 4 and Thread 6 in the screenshot are deadlocked on the monitors of the same connection and statement.

Paul Reeves added a comment - 13/May/16 02:26 PM
But where does that monitor come from?
It is certainly not something in the application code, as far as I am aware. I grepped 'monitor' and found nothing.

Is it something in the JVM? Jaybird?

  

Mark Rotteveel added a comment - 13/May/16 03:22 PM
Monitor is the technical term for what you lock when code does synchronized(object) { .... } (or use a synchronized method) It is an integral part of the object being locked on (so it is a fundamental part of Java).

In this case, it seems to be one and the same connection used by more than one thread.

Paul Reeves added a comment - 13/May/16 04:52 PM
Which poses an interesting question. Is the synchronization problem in my code or is it something in Jaybird?

I just checked with Jaybird 2.1.6 and there is no deadlock. On the other hand I can reproduce this consistently with Jaybird 3.0.0 snapshot running in a debug session in netbeans.

Mark Rotteveel added a comment - 25/May/16 09:38 AM
Paul, could you send me the database + the code (or a download link to it)?

Mark Rotteveel added a comment - 19/Jun/16 09:03 AM
I decided to make the locking a little bit more coarse, with a single lock object for all related objects (eg the statement lock object is the same as the connection lock object, etc). I'm first doing this in the org.firebirdsql.gds.ng.* objects, and will check if I will also need to do something similar with the org.firebirdsql.jdbc.* objects. As connections etc should not really be used from multiple threads at the same time, having these coarser locks should not impact performance for most use cases. It might have an impact on use case where a single connections is shared by multiple threads.

It would still be very helpful to have the code to test with.

Mark Rotteveel added a comment - 19/Jun/16 11:42 AM
In Jaybird 3, I applied a coarser synchronization with - where possible - a single lock object per connection.
 
This should prevent the potential of deadlock as described above, which may occur because in some situations the locks are obtained in order connection->statement, and in others statement->connection (etc).

I've not yet decided if I'm going to backport this to Jaybird 2.2 or not.

Mark Rotteveel added a comment - 19/Jun/16 12:57 PM
Similar change made for 2.2.11.