New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Server hang on new attachment right after trace session stop [CORE6027] #6277
Comments
Modified by: @AlexPeshkoffassignee: Alexander Peshkov [ alexpeshkoff ] |
Commented by: @AlexPeshkoff Can you also attach stack trace for core dump of process 5903 (fbtracemgr)? |
Commented by: @artyom-smirnov Trace of core of fbtracemgr: Thread 2 (Thread 0x7fe1b4048700 (LWP 15090)): Thread 1 (Thread 0x7fe1b50fa780 (LWP 15088)): |
Commented by: @AlexPeshkoff Artyom, can you try with this patch? I still see rare segfaults when trace manager is closing but server never hangs for me. |
Modified by: @AlexPeshkoffAttachment: 6027_1.patch [ 13330 ] |
Commented by: @artyom-smirnov With original testcase I do not see server hangs anymore too. |
Commented by: @artyom-smirnov I've noticed, if system wide audit config is defined fbtracemgr crashing much more often. To reproduce enough to set "AuditTraceConfigFile = fbtrace.conf" in firebird.conf and run original reproduce script. Backtrace of coredump: Core was generated by `./bin/fbtracemgr -se service_mgr -user SYSDBA -password ********* -start -c fbt'. |
Commented by: @artyom-smirnov Looks like checking if we shutdowning enough (by aquiring mutex in update_sessions) to prevent this crash: ... |
Commented by: @AlexPeshkoff Yes - an issue will be gone with such fix. But what is worse - robust pthreads mutex does not work as expected, dead process is not detected by the kernel. This can be easily seen when attaching to the hanged server. (gdb) f 1 __owner is LWP numbr of a thread holding mutex, let's take a look at it: (gdb) shell kill -1 8352 No - robust mutex not detected process death. That's great pity for us :( |
Modified by: @AlexPeshkoff |
Commented by: @AlexPeshkoff Hang took place due to munmap() of a region, containing locked shared robust mutex, in fbtracemgr. For details see: |
Modified by: @AlexPeshkoffstatus: Open [ 1 ] => Resolved [ 5 ] resolution: Fixed [ 1 ] Fix Version: 3.0.5 [ 10885 ] Fix Version: 4.0 Beta 2 [ 10888 ] |
Modified by: @pavel-zotovstatus: Resolved [ 5 ] => Resolved [ 5 ] QA Status: No test => Deferred Test Details: Could not reproduce on Windows 8.1 Pro, FB WI-V3.0.4.33054, SS/CS. Test Specifics: [Platform (Windows/Linux) specific] |
Submitted by: @artyom-smirnov
Block progress on CORE6045
Attachments:
6027_1.patch
Test script (https://gist.github.com/artyom-smirnov/076879ff9edff7f2b58147eb5665f0ed):
#!/bin/bash
DB_PATH=/tmp/testdb.fdb
DB=localhost:$DB_PATH
ISQL="/opt/firebird/bin/isql -q -u SYSDBA -p masterkey"
cat << EOF > /tmp/fbtrace_custom.conf
database
{
enabled = true
format = 0
log_connections = true
log_statement_start =true
}
EOF
cat << EOF > /tmp/createdb.sql
create database '$DB';
EOF
cat << EOF > /tmp/q.sql
connect '$DB';
exit 0;
EOF
rm -f $DB_PATH
$ISQL -i /tmp/createdb.sql
while true; do
/opt/firebird/bin/fbtracemgr -se service_mgr -user SYSDBA -password masterkey -start -c /tmp/fbtrace_custom.conf &
rdbtracemgr_pid=$!
sleep 1
kill -TERM $rdbtracemgr_pid
$ISQL -i /tmp/q.sql
done
Run as:
sudo -u firebird ./test.sh
Result will be like this and then server will hang:
Trace session ID 1 started
Trace session ID 2 started
Trace session ID 3 started
Trace session ID 4 started
Trace session ID 5 started
Trace session ID 6 started
Trace session ID 7 started
Trace session ID 8 started
Trace session ID 9 started
Trace session ID 10 started
Trace session ID 11 started
Trace session ID 12 started
Trace session ID 13 started
Trace session ID 14 started
Trace session ID 15 started
Trace session ID 16 started
./test.sh: line 41: 5872 Segmentation fault (core dumped) /opt/firebird/bin/fbtracemgr -se service_mgr -user SYSDBA -password masterkey -start -c /tmp/fbtrace_custom.conf
Trace session ID 17 started
Trace session ID 18 started
./test.sh: line 41: 5903 Segmentation fault (core dumped) /opt/firebird/bin/fbtracemgr -se service_mgr -user SYSDBA -password masterkey -start -c /tmp/fbtrace_custom.conf
Trace session ID 19 started
I/O error during "lseek" operation for file "/tmp/firebird/fb_trace_ggAtlF"
-Error while trying to read from file
-Bad file descriptor
Trace session ID 21 started
I/O error during "lseek" operation for file "/tmp/firebird/fb_trace_ggAtlF"
-Error while trying to read from file
-Bad file descriptor
Trace session ID 23 started
I/O error during "lseek" operation for file "/tmp/firebird/fb_trace_ggAtlF"
-Error while trying to read from file
-Bad file descriptor
I/O error during "lseek" operation for file "/tmp/firebird/fb_trace_ggAtlF"
-Error while trying to read from file
-Bad file descriptor
I/O error during "lseek" operation for file "/tmp/firebird/fb_trace_ggAtlF"
-Error while trying to read from file
-Bad file descriptor
I/O error during "lseek" operation for file "/tmp/firebird/fb_trace_ggAtlF"
-Error while trying to read from file
-Bad file descriptor
I/O error during "lseek" operation for file "/tmp/firebird/fb_trace_ggAtlF"
-Error while trying to read from file
-Bad file descriptor
I/O error during "lseek" operation for file "/tmp/firebird/fb_trace_ggAtlF"
-Error while trying to read from file
-Bad file descriptor
I/O error during "lseek" operation for file "/tmp/firebird/fb_trace_ggAtlF"
-Error while trying to read from file
-Bad file descriptor
Trace session ID 31 started
Backtrace of hanged server:
#0 __pthread_mutex_lock_full (mutex=0x7f8ddf3b6010) at ../nptl/pthread_mutex_lock.c:313
#1 0x00007f8ddc25e315 in Firebird::SharedMemoryBase::mutexLock (this=0x7f8ddbdd3bc0) at /usr/home/firebird/3.0.4/src/common/isc_sync.cpp:3437
#2 0x00007f8ddc1c12d0 in Jrd::ConfigStorage::acquire (this=this@entry=0x7f8ddb8a7230)
at /usr/home/firebird/3.0.4/src/jrd/trace/TraceConfigStorage.cpp:308
#3 0x00007f8ddc1c8215 in Jrd::StorageGuard::StorageGuard (storage=0x7f8ddb8a7230, this=<synthetic pointer>)
at /usr/home/firebird/3.0.4/src/jrd/trace/../../jrd/trace/../../jrd/trace/TraceConfigStorage.h:172
#4 Jrd::TraceManager::update_sessions (this=this@entry=0x7f8ddb4c0070) at /usr/home/firebird/3.0.4/src/jrd/trace/TraceManager.cpp:184
#5 0x00007f8ddbf92ecb in Jrd::TraceManager::needs (this=0x7f8ddb4c0070, e=0) at /usr/home/firebird/3.0.4/src/jrd/../jrd/trace/TraceManager.h:125
#6 0x00007f8ddbf85a1a in Jrd::JProvider::internalAttach (this=<optimized out>, user_status=0x7f8dd9391590,
filename=0x7f8dd939172c "/opt/firebird/security3.fdb", dpb_length=<optimized out>, dpb=<optimized out>, existingId=0x0)
at /usr/home/firebird/3.0.4/src/jrd/jrd.cpp:1874
#7 0x00007f8ddbe82e58 in Firebird::IProviderBaseImpl<Jrd::JProvider, Firebird::CheckStatusWrapper, Firebird::IPluginBaseImpl<Jrd::JProvider, Firebird::CheckStatusWrapper, Firebird::Inherit<Firebird::IReferenceCountedImpl<Jrd::JProvider, Firebird::CheckStatusWrapper, Firebird::Inherit<Firebird::IVersionedImpl<Jrd::JProvider, Firebird::CheckStatusWrapper, Firebird::InheritFirebird::IProvider\ > > > > > >::cloopattachDatabaseDispatcher(Firebird::IProvider*, Firebird::IStatus*, char const*, unsigned int, unsigned char const*) () from /opt/firebird/plugins/libEngine12.so
#8 0x00007f8ddf469df6 in Firebird::IProvider::attachDatabaseFirebird::CheckStatusWrapper\ (dpb=0x7f8dd93917a8 "\001M", dpbLength=34,
fileName=0x7f8dd939172c "/opt/firebird/security3.fdb", status=0x7f8dd9391b00, this=0x7f8ddbbbabe8)
at /usr/home/firebird/3.0.4/src/include/firebird/IdlFbInterfaces.h:2037
#9 Why::Dispatcher::attachOrCreateDatabase (this=0x7f8ddf3ce8f0, status=status@entry=0x7f8dd9391b00, createFlag=createFlag@entry=false,
filename=filename@entry=0x7f8dde7ebe40 "/opt/firebird/security3.fdb", dpbLength=dpbLength@entry=32,
dpb=dpb@entry=0x7f8dd9391e18 "\001\067\001\001\034\006SYSDBAW\022Providers=Engine12") at /usr/home/firebird/3.0.4/src/yvalve/why.cpp:5725
#10 0x00007f8ddf46a4e5 in Why::Dispatcher::attachDatabase (this=<optimized out>, status=status@entry=0x7f8dd9391b00,
filename=filename@entry=0x7f8dde7ebe40 "/opt/firebird/security3.fdb", dpbLength=dpbLength@entry=32,
dpb=dpb@entry=0x7f8dd9391e18 "\001\067\001\001\034\006SYSDBAW\022Providers=Engine12") at /usr/home/firebird/3.0.4/src/yvalve/why.cpp:5640
#11 0x00007f8ddf4223d8 in Firebird::IProviderBaseImpl<Why::Dispatcher, Firebird::CheckStatusWrapper, Firebird::IPluginBaseImpl<Why::Dispatcher, Firebird::CheckStatusWrapper, Firebird::Inherit<Firebird::IReferenceCountedImpl<Why::Dispatcher, Firebird::CheckStatusWrapper, Firebird::Inherit<Firebird::IVersionedImpl<Why::Dispatcher, Firebird::CheckStatusWrapper, Firebird::InheritFirebird::IProvider\ > > > > > >::cloopattachDatabaseDispatcher
(self=<optimized out>, status=0x7f8dd9392148, fileName=0x7f8dde7ebe40 "/opt/firebird/security3.fdb", dpbLength=32,
dpb=0x7f8dd9391e18 "\001\067\001\001\034\006SYSDBAW\022Providers=Engine12")
at /usr/home/firebird/3.0.4/src/include/firebird/IdlFbInterfaces.h:9123
#12 0x000000000044acfa in Firebird::IProvider::attachDatabaseFirebird::CheckStatusWrapper\ (
dpb=0x7f8dd9391e18 "\001\067\001\001\034\006SYSDBAW\022Providers=Engine12", dpbLength=32,
fileName=0x7f8dde7ebe40 "/opt/firebird/security3.fdb", status=0x7f8dd9392140, this=0x7f8ddf3ce8f8)
at /usr/home/firebird/3.0.4/src/include/firebird/IdlFbInterfaces.h:2037
#13 Auth::SrpServer::authenticate (this=0x7f8dde7edee0, status=0x7f8dd9392140, sb=0x7f8ddd724dc8, writerInterface=<optimized out>)
at /usr/home/firebird/3.0.4/src/auth/SecureRemotePassword/server/SrpServer.cpp:159
#14 0x0000000000451b08 in Firebird::IServerBaseImpl<Auth::SrpServer, Firebird::CheckStatusWrapper, Firebird::IAuthImpl<Auth::SrpServer, Firebird::CheckStatusWrapper, Firebird::Inherit<Firebird::IPluginBaseImpl<Auth::SrpServer, Firebird::CheckStatusWrapper, Firebird::Inherit<Firebird::IReferenceCountedImpl<Auth::SrpServer, Firebird::CheckStatusWrapper, Firebird::Inherit<Firebird::IVersionedImpl<Auth::SrpServer, Firebird::CheckStatusWrapper, Firebird::InheritFirebird::IServer\ > > > > > > > >::cloopauthenticateDispatcher(Firebird::IServer*, Firebird::IStatus*, Firebird::IServerBlock*, Firebird::IWriter*) ()
#15 0x00000000004376af in Firebird::IServer::authenticateFirebird::CheckStatusWrapper\ (writerInterface=0x7f8ddd725118, sBlock=0x7f8ddd724dc8,
status=0x7f8dd9392210, this=0x7f8dde7edee8) at /usr/home/firebird/3.0.4/src/include/firebird/IdlFbInterfaces.h:2371
#16 (anonymous namespace)::ServerAuth::authenticate (this=this@entry=0x7f8dde7ef570, send=send@entry=0x7f8ddd7494d8,
flags=flags@entry=ServerAuthBase::USE_COND_ACCEPT) at /usr/home/firebird/3.0.4/src/remote/server/server.cpp:640
--Type <RET> for more, q to quit, c to continue without paging--
#17 0x00000000004453c4 in accept_connection (send=0x7f8ddd7494d8, connect=0x7f8ddd7499b0, port=0x7f8ddd744e40)
at /usr/home/firebird/3.0.4/src/remote/server/server.cpp:1907
#18 process_packet (port=0x7f8ddd744e40, sendL=sendL@entry=0x7f8ddd7494d8, receive=receive@entry=0x7f8ddd7499a8,
result=result@entry=0x7f8dd9392d70) at /usr/home/firebird/3.0.4/src/remote/server/server.cpp:4361
#19 0x0000000000446792 in loopThread () at /usr/home/firebird/3.0.4/src/remote/server/server.cpp:6016
#20 0x00000000004595b5 in (anonymous namespace)::ThreadArgs::run (this=<synthetic pointer>)
at /usr/home/firebird/3.0.4/src/common/ThreadStart.cpp:78
#21 (anonymous namespace)::threadStart (arg=0x7f8dde7ec900) at /usr/home/firebird/3.0.4/src/common/ThreadStart.cpp:94
#22 0x00007f8ddec1c164 in start_thread (arg=<optimized out>) at pthread_create.c:486
#23 0x00007f8ddeb44def in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95
Commits: 13d20b5 39ded8e
====== Test Details ======
Could not reproduce on Windows 8.1 Pro, FB WI-V3.0.4.33054, SS/CS.
The text was updated successfully, but these errors were encountered: