Submitted by: David Hollings (dholling)

Firebird 3.0.5.3320 SS segfaults multiple times a day running on the above environments.
The servers themselves are very busy at times, however, the segfaults seem to occur independent of load - sometimes when the system is mostly idle.

The core files produced are very large (20-50Gb) and a backtrace for all threads fairly massive.

Please let me know if I can assist with any further information or if I should continue attaching back traces.

Thanks
David

Below firebird.log entries for abnormal termination from one server:

server.localdomain Thu Oct 1 07:54:29 2020
/opt/firebird/bin/fbguard: /opt/firebird/bin/firebird terminated abnormally (-1)
--
server.localdomain Thu Oct 1 11:00:02 2020
/opt/firebird/bin/fbguard: /opt/firebird/bin/firebird terminated abnormally (-1)
--
server.localdomain Fri Oct 2 11:07:17 2020
/opt/firebird/bin/fbguard: /opt/firebird/bin/firebird terminated abnormally (-1)
--
server.localdomain Sat Oct 3 15:40:10 2020
/opt/firebird/bin/fbguard: /opt/firebird/bin/firebird terminated abnormally (-1)
--
server.localdomain Tue Oct 6 01:04:02 2020
/opt/firebird/bin/fbguard: /opt/firebird/bin/firebird terminated abnormally (-1)
--
server.localdomain Wed Oct 7 03:02:57 2020
/opt/firebird/bin/fbguard: /opt/firebird/bin/firebird terminated abnormally (-1)
--
server.localdomain Thu Oct 8 09:56:43 2020
/opt/firebird/bin/fbguard: /opt/firebird/bin/firebird terminated abnormally (-1)
--
server.localdomain Thu Oct 8 11:26:11 2020
/opt/firebird/bin/fbguard: /opt/firebird/bin/firebird terminated abnormally (-1)
--
server.localdomain Thu Oct 8 11:53:59 2020
/opt/firebird/bin/fbguard: /opt/firebird/bin/firebird terminated abnormally (-1)
--
server.localdomain Thu Oct 8 19:32:34 2020
/opt/firebird/bin/fbguard: /opt/firebird/bin/firebird terminated abnormally (-1)
--
server.localdomain Thu Oct 8 21:27:07 2020
/opt/firebird/bin/fbguard: /opt/firebird/bin/firebird terminated abnormally (-1)
--
server.localdomain Fri Oct 9 06:44:43 2020
/opt/firebird/bin/fbguard: /opt/firebird/bin/firebird terminated abnormally (-1)
--
server.localdomain Tue Oct 13 12:59:06 2020
/opt/firebird/bin/fbguard: /opt/firebird/bin/firebird terminated abnormally (-1)

Below from a core dump:

[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib64/libthread_db.so.1".
Missing separate debuginfo for /opt/firebird/plugins/libEngine12.so
Missing separate debuginfo for /opt/firebird/lib/libib_util.so
Missing separate debuginfo for /opt/firebird/intl/fbintl
Missing separate debuginfo for /opt/firebird/plugins/libfbtrace.so
Missing separate debuginfo for /opt/firebird/plugins/libLegacy_Auth.so
Missing separate debuginfo for /opt/firebird/UDF/ib_udf.so
Core was generated by `/opt/firebird/bin/firebird'.
Program terminated with signal 11, Segmentation fault.
#⁠0 Firebird::MemoryPool::allocate (this=this@entry=0x7f641369c38000, size=size@entry=288) at /usr/home/firebird/3.0.5/src/common/classes/alloc.cpp:2823
2823 /usr/home/firebird/3.0.5/src/common/classes/alloc.cpp: No such file or directory.
Missing separate debuginfos, use: debuginfo-install glibc-2.17-222.el7.x86_64 libgcc-4.8.5-28.el7_5.1.x86_64 libicu-50.1.2-15.el7.x86_64 libstdc++-4.8.5-28.el7_5.1.x86_64 libtommath-0.42.0-6.el7.x86_64 ncurses-libs-5.9-14.20130511.el7_4.x86_64

And a backtrace of the thread:

Thread 1 (Thread 0x7f62e951c700 (LWP 374339)):
#⁠0 Firebird::MemoryPool::allocate (this=this@entry=0x7f641369c38000, size=size@entry=288) at /usr/home/firebird/3.0.5/src/common/classes/alloc.cpp:2823
#⁠1 0x00000000004777c1 in operator new (pool=..., s=288) at /usr/home/firebird/3.0.5/src/include/../common/classes/alloc.h:303
#⁠2 grow (newCount=4, this=0x7f640ef02700) at /usr/home/firebird/3.0.5/src/include/../common/../common/classes/objects_array.h:248
#⁠3 Firebird::MetadataFromBlr::MetadataFromBlr (this=0x7f640ef026e0, aBlrLength=<optimized out>, aBlr=<optimized out>, aLength=26) at /usr/home/firebird/3.0.5/src/common/classes/InternalMessageBuffer.cpp:79
#⁠4 0x00000000004792ee in Firebird::InternalMessageBuffer::InternalMessageBuffer (this=0x7f62e951b2e0, aBlrLength=26, aBlr=0x7f672d3a03d0 "\005\002\004", aBufferLength=26, aBuffer=<optimized out>) at /usr/home/firebird/3.0.5/src/common/classes/InternalMessageBuffer.cpp:216
#⁠5 0x0000000000442a13 in rem_port::fetch (this=this@entry=0x7f63ea599240, sqldata=sqldata@entry=0x7f67916d4510, sendL=sendL@entry=0x7f67916d3cd8) at /usr/home/firebird/3.0.5/src/remote/server/server.cpp:3571
#⁠6 0x0000000000443812 in process_packet (port=0x7f63ea599240, sendL=sendL@entry=0x7f67916d3cd8, receive=receive@entry=0x7f67916d41a8, result=result@entry=0x7f62e951bd90) at /usr/home/firebird/3.0.5/src/remote/server/server.cpp:4567
#⁠7 0x0000000000446362 in loopThread () at /usr/home/firebird/3.0.5/src/remote/server/server.cpp:6020
#⁠8 0x0000000000458fb5 in run (this=<synthetic pointer>) at /usr/home/firebird/3.0.5/src/common/ThreadStart.cpp:78
#⁠9 (anonymous namespace)::threadStart (arg=0x7f63eaa9e860) at /usr/home/firebird/3.0.5/src/common/ThreadStart.cpp:94
#⁠10 0x00007f6eb411de25 in start_thread () from /lib64/libpthread.so.0
#⁠11 0x00007f6eb3e4b34d in wctype () from /lib64/libc.so.6
#⁠12 0x0000000000000000 in ?? ()

Below a different core file:

[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib64/libthread_db.so.1".
Missing separate debuginfo for /opt/firebird/plugins/libEngine12.so
Missing separate debuginfo for /opt/firebird/lib/libib_util.so
Missing separate debuginfo for /opt/firebird/intl/fbintl
Missing separate debuginfo for /opt/firebird/plugins/libfbtrace.so
Missing separate debuginfo for /opt/firebird/plugins/libLegacy_Auth.so
Missing separate debuginfo for /opt/firebird/UDF/ib_udf.so
Core was generated by `/opt/firebird/bin/firebird'.
Program terminated with signal 11, Segmentation fault.
#⁠0 Firebird::MemPool::releaseBlock (this=0x0, block=0x7f28491a0af1, decrUsage=true) at /usr/home/firebird/3.0.5/src/common/classes/alloc.cpp:2400
2400 /usr/home/firebird/3.0.5/src/common/classes/alloc.cpp: No such file or directory.
Missing separate debuginfos, use: debuginfo-install glibc-2.17-222.el7.x86_64 libgcc-4.8.5-28.el7_5.1.x86_64 libicu-50.1.2-15.el7.x86_64 libstdc++-4.8.5-28.el7_5.1.x86_64 libtommath-0.42.0-6.el7.x86_64 ncurses-libs-5.9-14.20130511.el7_4.x86_64

And the thread:

[Switching to thread 1 (LWP 408831)]
#⁠0 Firebird::MemPool::releaseBlock (this=0x0, block=0x7f28491a0af1, decrUsage=true) at /usr/home/firebird/3.0.5/src/common/classes/alloc.cpp:2400
2400 in /usr/home/firebird/3.0.5/src/common/classes/alloc.cpp
(gdb) bt
#⁠0 Firebird::MemPool::releaseBlock (this=0x0, block=0x7f28491a0af1, decrUsage=true) at /usr/home/firebird/3.0.5/src/common/classes/alloc.cpp:2400
#⁠1 0x000000000047f10b in Firebird::MemPool::releaseMemory (object=<optimized out>, flagExtent=flagExtent@entry=false) at /usr/home/firebird/3.0.5/src/common/classes/alloc.cpp:2383
#⁠2 0x000000000047f2ac in Firebird::MemPool::deallocate (block=<optimized out>) at /usr/home/firebird/3.0.5/src/common/classes/alloc.cpp:2665
#⁠3 0x000000000047f2ba in Firebird::MemPool::globalFree (block=<optimized out>) at /usr/home/firebird/3.0.5/src/common/classes/alloc.cpp:2653
#⁠4 0x000000000047f77a in Firebird::MemoryPool::globalFree (block=<optimized out>) at /usr/home/firebird/3.0.5/src/common/classes/alloc.cpp:2818
#⁠5 0x0000000000477779 in freeData (this=0x7f28491a0bd0) at /usr/home/firebird/3.0.5/src/include/../common/../common/classes/array.h:146
#⁠6 ensureCapacity (preserve=true, newcapacity=43, this=0x7f28491a0bd0) at /usr/home/firebird/3.0.5/src/include/../common/../common/classes/array.h:467
#⁠7 grow (newCount=43, this=0x7f28491a0bd0) at /usr/home/firebird/3.0.5/src/include/../common/../common/classes/array.h:296
#⁠8 grow (newCount=43, this=0x7f28491a0bd0) at /usr/home/firebird/3.0.5/src/include/../common/../common/classes/objects_array.h:246
#⁠9 Firebird::MetadataFromBlr::MetadataFromBlr (this=0x7f28491a0bb0, aBlrLength=<optimized out>, aBlr=<optimized out>, aLength=362) at /usr/home/firebird/3.0.5/src/common/classes/InternalMessageBuffer.cpp:79
#⁠10 0x00000000004792ee in Firebird::InternalMessageBuffer::InternalMessageBuffer (this=0x7f23bd5ef2e0, aBlrLength=206, aBlr=0x7f26d4e64200 "\005\002\004", aBufferLength=362, aBuffer=<optimized out>) at /usr/home/firebird/3.0.5/src/common/classes/InternalMessageBuffer.cpp:216
#⁠11 0x0000000000442a13 in rem_port::fetch (this=this@entry=0x7f286c89a240, sqldata=sqldata@entry=0x7f279c396a90, sendL=sendL@entry=0x7f279c396258) at /usr/home/firebird/3.0.5/src/remote/server/server.cpp:3571
#⁠12 0x0000000000443812 in process_packet (port=0x7f286c89a240, sendL=sendL@entry=0x7f279c396258, receive=receive@entry=0x7f279c396728, result=result@entry=0x7f23bd5efd90) at /usr/home/firebird/3.0.5/src/remote/server/server.cpp:4567
#⁠13 0x0000000000446362 in loopThread () at /usr/home/firebird/3.0.5/src/remote/server/server.cpp:6020
#⁠14 0x0000000000458fb5 in run (this=<synthetic pointer>) at /usr/home/firebird/3.0.5/src/common/ThreadStart.cpp:78
#⁠15 (anonymous namespace)::threadStart (arg=0x7f284de2ffd0) at /usr/home/firebird/3.0.5/src/common/ThreadStart.cpp:94
#⁠16 0x00007f2d7e6b1dd5 in start_thread () from /lib64/libpthread.so.0
#⁠17 0x00007f2d7e3daead in __libc_ifunc_impl_list () from /lib64/libc.so.6
#⁠18 0x0000000000000000 in ?? ()