Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

AV when process is closing under high load [CORE2064] #2500

Closed
firebird-automations opened this issue Sep 3, 2008 · 8 comments
Closed

AV when process is closing under high load [CORE2064] #2500

firebird-automations opened this issue Sep 3, 2008 · 8 comments

Comments

@firebird-automations
Copy link
Collaborator

Submitted by: @AlexPeshkoff

When LockManager instance is destroyed, blocking action thread deadlocks with main thread. Part of stack BT:

main thread:
#⁠0 0x00002ae1bd3f81bd in sem_wait () from /lib/libpthread.so.0
#⁠1 0x00002ae1bc1856b5 in Firebird::SignalSafeSemaphore::enter (this=0x2aaaaaf173d8)
at ../src/remote/../common/classes/semaphore.h:168
#⁠2 0x00002ae1bc3db6b5 in ~LockManager (this=0x2aaaaaf17338) at ../src/lock/lock.cpp:242
#⁠3 0x00002ae1bc2c10bd in Firebird::RefCounted::release (this=0x2aaaaaf17338)
at ../src/include/../common/classes/RefCounted.h:46
#⁠4 0x00002ae1bc2bd888 in ~RefPtr (this=0x2aaaab180d80) at ../src/include/../common/classes/RefCounted.h:115
#⁠5 0x00002ae1bc2c1aaa in ~Database (this=0x2aaaab180d68) at ../src/jrd/Database.cpp:66
#⁠6 0x00002ae1bc2bc7d4 in Jrd::Database::destroyDb (toDelete=0x2aaaab180d68) at ../src/jrd/../jrd/../jrd/Database.h:263
#⁠7 0x00002ae1bc2b9097 in shutdown_database (dbb=0x2aaaab180d68, release_pools=true) at ../src/jrd/jrd.cpp:5168
#⁠8 0x00002ae1bc2b9ccb in purge_attachment (tdbb=0x7fffeeada250, user_status=0x576e80, attachment=0x2aaaab2a00b0,
force_flag=false) at ../src/jrd/jrd.cpp:5615
********
Here dbb_sync is locked by DatabaseContextHolder : public Database::SyncGuard
********
#⁠9 0x00002ae1bc2afe54 in jrd8_detach_database (user_status=0x576e80, handle=0x2aaaab136ac8) at ../src/jrd/jrd.cpp:2090

blocking thread:
********
Here we try to lock dbb_sync in AST routine
********
#⁠6 0x00002ae1bc1a15b8 in Jrd::Database::Sync::lock (this=0x2aaaab133068, ast=true)
at ../src/lock/../jrd/../jrd/Database.h:98
#⁠7 0x00002ae1bc1c9fc7 in SyncGuard (this=0x41000ed0, dbb=0x2aaaab180d68, ast=true)
at ../src/jrd/../jrd/../jrd/Database.h:150
#⁠8 0x00002ae1bc241e92 in blocking_ast_bdb (ast_object=0x2aaaadbd46d0) at ../src/jrd/cch.cpp:2707
#⁠9 0x00002ae1bc3ddc5a in Jrd::LockManager::blocking_action (this=0x2aaaaaf17338, tdbb=0x0, blocking_owner_offset=20904,
blocked_owner_offset=20904) at ../src/lock/lock.cpp:1293
#⁠10 0x00002ae1bc3ddea9 in Jrd::LockManager::blocking_action_thread (this=0x2aaaaaf17338) at ../src/lock/lock.cpp:1378
#⁠11 0x00002ae1bc3ddcb1 in Jrd::LockManager::blocking_action_thread (arg=0x2aaaaaf17338) at ../src/lock/lock.cpp:1306

Deadlock is resolved by timeout in cleanup semaphore, but this leads to AV (or bad system call - according to races result) in blocking thread.

Commits: 796a2e4

@firebird-automations
Copy link
Collaborator Author

Commented by: @AlexPeshkoff

sorry, unexpectedly pressed something on KB

@firebird-automations
Copy link
Collaborator Author

Modified by: @AlexPeshkoff

assignee: Alexander Peshkov [ alexpeshkoff ]

summary: LockManager does bn => AV when process is closing under high load

description: When LockManager instance is destroyed, blocking action thread deadlocks with main thread. Part of stack BT:

main thread:
#⁠0 0x00002ae1bd3f81bd in sem_wait () from /lib/libpthread.so.0
#⁠1 0x00002ae1bc1856b5 in Firebird::SignalSafeSemaphore::enter (this=0x2aaaaaf173d8)
at ../src/remote/../common/classes/semaphore.h:168
#⁠2 0x00002ae1bc3db6b5 in ~LockManager (this=0x2aaaaaf17338) at ../src/lock/lock.cpp:242
#⁠3 0x00002ae1bc2c10bd in Firebird::RefCounted::release (this=0x2aaaaaf17338)
at ../src/include/../common/classes/RefCounted.h:46
#⁠4 0x00002ae1bc2bd888 in ~RefPtr (this=0x2aaaab180d80) at ../src/include/../common/classes/RefCounted.h:115
#⁠5 0x00002ae1bc2c1aaa in ~Database (this=0x2aaaab180d68) at ../src/jrd/Database.cpp:66
#⁠6 0x00002ae1bc2bc7d4 in Jrd::Database::destroyDb (toDelete=0x2aaaab180d68) at ../src/jrd/../jrd/../jrd/Database.h:263
#⁠7 0x00002ae1bc2b9097 in shutdown_database (dbb=0x2aaaab180d68, release_pools=true) at ../src/jrd/jrd.cpp:5168
#⁠8 0x00002ae1bc2b9ccb in purge_attachment (tdbb=0x7fffeeada250, user_status=0x576e80, attachment=0x2aaaab2a00b0,
force_flag=false) at ../src/jrd/jrd.cpp:5615
********
Here dbb_sync is locked by DatabaseContextHolder : public Database::SyncGuard
********
#⁠9 0x00002ae1bc2afe54 in jrd8_detach_database (user_status=0x576e80, handle=0x2aaaab136ac8) at ../src/jrd/jrd.cpp:2090

blocking thread:
********
Here we try to lock dbb_sync in AST routine
********
#⁠6 0x00002ae1bc1a15b8 in Jrd::Database::Sync::lock (this=0x2aaaab133068, ast=true)
at ../src/lock/../jrd/../jrd/Database.h:98
#⁠7 0x00002ae1bc1c9fc7 in SyncGuard (this=0x41000ed0, dbb=0x2aaaab180d68, ast=true)
at ../src/jrd/../jrd/../jrd/Database.h:150
#⁠8 0x00002ae1bc241e92 in blocking_ast_bdb (ast_object=0x2aaaadbd46d0) at ../src/jrd/cch.cpp:2707
#⁠9 0x00002ae1bc3ddc5a in Jrd::LockManager::blocking_action (this=0x2aaaaaf17338, tdbb=0x0, blocking_owner_offset=20904,
blocked_owner_offset=20904) at ../src/lock/lock.cpp:1293
#⁠10 0x00002ae1bc3ddea9 in Jrd::LockManager::blocking_action_thread (this=0x2aaaaaf17338) at ../src/lock/lock.cpp:1378
#⁠11 0x00002ae1bc3ddcb1 in Jrd::LockManager::blocking_action_thread (arg=0x2aaaaaf17338) at ../src/lock/lock.cpp:1306

Deadlock is resolved by timeout in cleanup semaphore, but this leads to AV (or bad system call - according to races result) in blocking thread.

environment: seems to be OS independent, but reproduced on linux

@firebird-automations
Copy link
Collaborator Author

Commented by: @AlexPeshkoff

I have a fix for an issue - plan to commit tomorrow.

@firebird-automations
Copy link
Collaborator Author

Commented by: @AlexPeshkoff

Enabled AST processing in the end of Database destructor before releasing lock manager.
At the same time added try/catch blocks to blocking thread to try to avoid still not fixed problems in it (if present).

@firebird-automations
Copy link
Collaborator Author

Modified by: @AlexPeshkoff

status: Open [ 1 ] => Resolved [ 5 ]

resolution: Fixed [ 1 ]

Fix Version: 2.5 Beta 1 [ 10251 ]

@firebird-automations
Copy link
Collaborator Author

Modified by: @pcisar

status: Resolved [ 5 ] => Closed [ 6 ]

@firebird-automations
Copy link
Collaborator Author

Modified by: @pavel-zotov

QA Status: No test

@firebird-automations
Copy link
Collaborator Author

Modified by: @pavel-zotov

status: Closed [ 6 ] => Closed [ 6 ]

QA Status: No test => Cannot be tested

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

2 participants