Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Deadlock in multi-threaded environment [JDBC435] #476

Closed
firebird-automations opened this issue May 13, 2016 · 18 comments
Closed

Deadlock in multi-threaded environment [JDBC435] #476

firebird-automations opened this issue May 13, 2016 · 18 comments

Comments

@firebird-automations
Copy link

Submitted by: @reevespaul

Is duplicated by JDBC434

Attachments:
deadlock.jpeg

When closing multiple threaded connections the driver hangs. The problem seems to be related to the test for mc == null in isClosed()

I tried a while ago with 2.2.3 and most recently with the code in the v3.0.0 snapshot. Neither work.

The last version that works for me is 2.1.6.

Commits: 78d882c a085ae0 7cd423e 4d7f118

@firebird-automations
Copy link
Author

Commented by: @reevespaul

Hopefully this screen shot from netbeans will give a better idea of what is happening. If necessary I can provide the app and the database.

@firebird-automations
Copy link
Author

Modified by: @reevespaul

Attachment: deadlock.jpeg [ 12963 ]

@firebird-automations
Copy link
Author

Modified by: @mrotteveel

Link: This issue is duplicated by JDBC434 [ JDBC434 ]

@firebird-automations
Copy link
Author

Commented by: @mrotteveel

I will investigate this. Having access to the app and database would be very helpful.

As a side note, although JDBC asks that drivers are threadsafe, in general you shouldn't use the same connection from multiple threads simultaneously.

@firebird-automations
Copy link
Author

Commented by: @mrotteveel

The problem is not with the null check in isClosed. It is with the fact that the order that executeBatch first acquires a lock on the statement, and later on on the connection, while connection close first acquires a lock on the connection, and then (when closing statements in response to the connection close) on the statement.

@firebird-automations
Copy link
Author

Commented by: @reevespaul

Just to comment on your side note - no, in the application each thread creates its own connection. With 2.16 I've had up to 100 concurrent threads working without problem.

But good to hear that you may have identified the problem. I can test the fix easily enough when it is ready.

@firebird-automations
Copy link
Author

Commented by: @mrotteveel

The screenshot seems to indicate that the connection and statements are shared: Thread 4 and Thread 6 in the screenshot are deadlocked on the monitors of the same connection and statement.

@firebird-automations
Copy link
Author

Commented by: @reevespaul

But where does that monitor come from?
It is certainly not something in the application code, as far as I am aware. I grepped 'monitor' and found nothing.

Is it something in the JVM? Jaybird?

@firebird-automations
Copy link
Author

Commented by: @mrotteveel

Monitor is the technical term for what you lock when code does synchronized(object) { .... } (or use a synchronized method) It is an integral part of the object being locked on (so it is a fundamental part of Java).

In this case, it seems to be one and the same connection used by more than one thread.

@firebird-automations
Copy link
Author

Commented by: @reevespaul

Which poses an interesting question. Is the synchronization problem in my code or is it something in Jaybird?

I just checked with Jaybird 2.1.6 and there is no deadlock. On the other hand I can reproduce this consistently with Jaybird 3.0.0 snapshot running in a debug session in netbeans.

@firebird-automations
Copy link
Author

Commented by: @mrotteveel

Paul, could you send me the database + the code (or a download link to it)?

@firebird-automations
Copy link
Author

Modified by: @mrotteveel

Fix Version: Jaybird 3.0 [ 10440 ]

@firebird-automations
Copy link
Author

Commented by: @mrotteveel

I decided to make the locking a little bit more coarse, with a single lock object for all related objects (eg the statement lock object is the same as the connection lock object, etc). I'm first doing this in the http://org.firebirdsql.gds.ng.* objects, and will check if I will also need to do something similar with the org.firebirdsql.jdbc.* objects. As connections etc should not really be used from multiple threads at the same time, having these coarser locks should not impact performance for most use cases. It might have an impact on use case where a single connections is shared by multiple threads.

It would still be very helpful to have the code to test with.

@firebird-automations
Copy link
Author

Commented by: @mrotteveel

In Jaybird 3, I applied a coarser synchronization with - where possible - a single lock object per connection.

This should prevent the potential of deadlock as described above, which may occur because in some situations the locks are obtained in order connection->statement, and in others statement->connection (etc).

I've not yet decided if I'm going to backport this to Jaybird 2.2 or not.

@firebird-automations
Copy link
Author

Modified by: @mrotteveel

Fix Version: Jaybird 2.2.11 [ 10751 ]

@firebird-automations
Copy link
Author

Commented by: @mrotteveel

Similar change made for 2.2.11.

@firebird-automations
Copy link
Author

Modified by: @mrotteveel

status: Open [ 1 ] => Resolved [ 5 ]

resolution: Fixed [ 1 ]

@firebird-automations
Copy link
Author

Modified by: @mrotteveel

status: Resolved [ 5 ] => Closed [ 6 ]

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment