Issue Details (XML | Word | Printable)

Key: CORE-2306
Type: Bug Bug
Status: Closed Closed
Resolution: Fixed
Priority: Major Major
Assignee: Alexander Peshkov
Reporter: Andreas Kallenbach
Votes: 0
Watchers: 1
Operations

If you were logged in you would be able to see more operations.
Firebird Core

fbserver terminated abnormally when thread start failed

Created: 04/Feb/09 07:37 PM   Updated: 22/Sep/11 12:38 AM
Component/s: Engine
Affects Version/s: 2.1.1
Fix Version/s: 2.5 Beta 2, 2.1.3, 2.0.6

Time Tracking:
Not Specified

Environment:
Fedora release 7 (Moonshine) Redhat Tikanga 5
FirebirdSS-2.1.1.17910-0.nptl.i686.rpm
Dell Poweredge Server 1750, P4-Xeon, 3GB Ram, Direct Attached SCSI RAID10 500GB
Issue Links:
Relate

Planning Status: Unspecified


 Description  « Hide
For the last three weeks our database server has been resetting around 2-3 times a week. I have been doing backup/restores on the weekends to keep potential database corruptions down, but we end up with almost 30 hours of weekly downtime to backup/restore a 45GB database.

Today, we had a server crash and messages contained the following line...

Feb 4 10:08:23 localhost kernel: fbserver[15932]: segfault at 00000058 eip 006c6370 esp bfdc5c44 error 4

firebird.log contained the following line...

localhost.localdomain (Client) Wed Feb 4 10:08:23 2009
        /opt/firebird/bin/fbguard: /opt/firebird/bin/fbserver terminated abnormally (-1)

What can I do to track this issue down and make it stop?

 All   Comments   Work Log   Change History   Version Control   Subversion Commits      Sort Order: Ascending order - Click to sort in descending order
Alexander Peshkov added a comment - 05/Feb/09 06:31 AM
First of all read this.
http://www.ibphoenix.com/admin/menu.nfs?a=ibphoenix&page=ibp_gdb_linux

When you have core file, contact me here. BTW, if you'll be able to provide me remote access to your server (after getting core file), this can seriously help solving your problem.

Andreas Kallenbach added a comment - 05/Feb/09 12:47 PM
Thanks, I have set BugcheckAbort=1 and will follow up once I have a core dump.

Andreas Kallenbach added a comment - 16/Feb/09 12:25 PM
Ok, I do have a core.22213 file that is 3GB in size. A little large to attach here, how else can I send this to you?

Andreas Kallenbach added a comment - 16/Feb/09 01:00 PM
I have downloaded the debug package FirebirdSS-debuginfo-2.1.1.17910-0.nptl.i686.tar.gz and extracted them into the appropriate directory. I assume this is the expected output?

[root@localhost tmp]# gdb /opt/firebird/bin/.debug/fbserver.debug core.22213
GNU gdb Red Hat Linux (6.6-16.fc7rh)
Copyright (C) 2006 Free Software Foundation, Inc.
GDB is free software, covered by the GNU General Public License, and you are
welcome to change it and/or distribute copies of it under certain conditions.
Type "show copying" to see the conditions.
There is absolutely no warranty for GDB. Type "show warranty" for details.
This GDB was configured as "i386-redhat-linux-gnu"...
Using host libthread_db library "/lib/libthread_db.so.1".

warning: core file may not match specified executable file.

warning: Can't read pathname for load map: Input/output error.
Reading symbols from /lib/libdl.so.2...done.
Loaded symbols for /lib/libdl.so.2
Reading symbols from /lib/libncurses.so.5...done.
Loaded symbols for /lib/libncurses.so.5
Reading symbols from /opt/firebird/lib/libicuuc.so.30...Reading symbols from /opt/firebird/lib/.debug/libicuuc.so.30.0.debug...done.
done.
Loaded symbols for /opt/firebird/lib/libicuuc.so.30
Reading symbols from /opt/firebird/lib/libicudata.so.30...Reading symbols from /opt/firebird/lib/.debug/libicudata.so.30.0.debug...done.
done.
Loaded symbols for /opt/firebird/lib/libicudata.so.30
Reading symbols from /opt/firebird/lib/libicui18n.so.30...Reading symbols from /opt/firebird/lib/.debug/libicui18n.so.30.0.debug...done.
done.
Loaded symbols for /opt/firebird/lib/libicui18n.so.30
Reading symbols from /usr/lib/libstdc++.so.5...done.
Loaded symbols for /usr/lib/libstdc++.so.5
Reading symbols from /lib/libm.so.6...done.
Loaded symbols for /lib/libm.so.6
Reading symbols from /lib/libgcc_s.so.1...done.
Loaded symbols for /lib/libgcc_s.so.1
Reading symbols from /lib/libpthread.so.0...done.
Loaded symbols for /lib/libpthread.so.0
Reading symbols from /lib/libc.so.6...done.
Loaded symbols for /lib/libc.so.6
Reading symbols from /lib/ld-linux.so.2...done.
Loaded symbols for /lib/ld-linux.so.2
Reading symbols from /lib/libtinfo.so.5...done.
Loaded symbols for /lib/libtinfo.so.5
Reading symbols from /lib/libnss_files.so.2...done.
Loaded symbols for /lib/libnss_files.so.2
Reading symbols from /opt/firebird/intl/fbintl...Reading symbols from /opt/firebird/intl/.debug/fbintl.debug...done.
done.
Loaded symbols for /opt/firebird/intl/fbintl
Reading symbols from /opt/firebird/UDF/ib_udf.so...
warning: the debug information found in "/opt/firebird/UDF/.debug/ib_udf.so.debug" does not match "/opt/firebird/UDF/ib_udf.so" (CRC mismatch).

done.
Loaded symbols for /opt/firebird/UDF/ib_udf.so
Reading symbols from /opt/firebird/UDF/udflib...done.
Loaded symbols for /opt/firebird/UDF/udflib
Reading symbols from /usr/lib/libgds.so...done.
Loaded symbols for /usr/lib/libgds.so
Core was generated by `/opt/firebird/bin/fbserver'.
Program terminated with signal 11, Segmentation fault.
#0 0x006c6370 in pthread_mutex_lock () from /lib/libpthread.so.0
(gdb)

Andreas Kallenbach added a comment - 16/Feb/09 09:01 PM
Had a second crash of the day with the following output... (same as before)

Program terminated with signal 11, Segmentation fault.
#0 0x006c6370 in pthread_mutex_lock () from /lib/libpthread.so.0

Andreas Kallenbach added a comment - 16/Feb/09 09:04 PM
I don't know if anything can be told by file dates on the libpthread.so.. but it is as follows....

[root@localhost lib]# ls -l libpthread*
-rwxr-xr-x 1 root root 129716 2007-07-08 07:14 libpthread-2.6.so
lrwxrwxrwx 1 root root 17 2007-08-31 10:58 libpthread.so.0 -> libpthread-2.6.so

Alexander Peshkov added a comment - 17/Feb/09 04:48 AM
Andreas, this is almost that output which is needed. Please run in gdb the following command -
(gdb) info thread
You will get something like this:
  3 Thread 1090525520 (LWP 10192) 0x00002ab9be036566
  2 Thread 1082132816 (LWP 10191) 0x00002ab9be0381bd
* 1 Thread 46977265564112 (LWP 10188) 0x00002ab9be1f33eb
For all enumerated threads do the following:
(gdb) t 1
(gdb) bt
(gdb) t 2
(gdb) bt
and so on. That output you should send to me.

What about the size of core file. I'm pretty sure it can be compressed using bzip2 at least 10 times. Certainly, if you'll be able to put it afterwards to any available place (http, ftp, scp, etc.), it will be ideal. Though quite possible that the backtrace (bt) of stacks can already help. Sooner of all we have some OOM problem.

PS. Version of libpthread is OK.

Andreas Kallenbach added a comment - 17/Feb/09 04:48 PM
[root@localhost tmp]# gdb /opt/firebird/bin/.debug/fbserver.debug core.22213
GNU gdb Red Hat Linux (6.6-16.fc7rh)
Copyright (C) 2006 Free Software Foundation, Inc.
GDB is free software, covered by the GNU General Public License, and you are
welcome to change it and/or distribute copies of it under certain conditions.
Type "show copying" to see the conditions.
There is absolutely no warranty for GDB. Type "show warranty" for details.
This GDB was configured as "i386-redhat-linux-gnu"...
Using host libthread_db library "/lib/libthread_db.so.1".

warning: core file may not match specified executable file.

warning: Can't read pathname for load map: Input/output error.
Reading symbols from /lib/libdl.so.2...done.
Loaded symbols for /lib/libdl.so.2
Reading symbols from /lib/libncurses.so.5...done.
Loaded symbols for /lib/libncurses.so.5
Reading symbols from /opt/firebird/lib/libicuuc.so.30...Reading symbols from /opt/firebird/lib/.debug/libicuuc.so.30.0.debug...done.
done.
Loaded symbols for /opt/firebird/lib/libicuuc.so.30
Reading symbols from /opt/firebird/lib/libicudata.so.30...Reading symbols from /opt/firebird/lib/.debug/libicudata.so.30.0.debug...done.
done.
Loaded symbols for /opt/firebird/lib/libicudata.so.30
Reading symbols from /opt/firebird/lib/libicui18n.so.30...Reading symbols from /opt/firebird/lib/.debug/libicui18n.so.30.0.debug...done.
done.
Loaded symbols for /opt/firebird/lib/libicui18n.so.30
Reading symbols from /usr/lib/libstdc++.so.5...done.
Loaded symbols for /usr/lib/libstdc++.so.5
Reading symbols from /lib/libm.so.6...done.
Loaded symbols for /lib/libm.so.6
Reading symbols from /lib/libgcc_s.so.1...done.
Loaded symbols for /lib/libgcc_s.so.1
Reading symbols from /lib/libpthread.so.0...done.
Loaded symbols for /lib/libpthread.so.0
Reading symbols from /lib/libc.so.6...done.
Loaded symbols for /lib/libc.so.6
Reading symbols from /lib/ld-linux.so.2...done.
Loaded symbols for /lib/ld-linux.so.2
Reading symbols from /lib/libtinfo.so.5...done.
Loaded symbols for /lib/libtinfo.so.5
Reading symbols from /lib/libnss_files.so.2...done.
Loaded symbols for /lib/libnss_files.so.2
Reading symbols from /opt/firebird/intl/fbintl...Reading symbols from /opt/firebird/intl/.debug/fbintl.debug...done.
done.
Loaded symbols for /opt/firebird/intl/fbintl
Reading symbols from /opt/firebird/UDF/ib_udf.so...
warning: the debug information found in "/opt/firebird/UDF/.debug/ib_udf.so.debug" does not match "/opt/firebird/UDF/ib_udf.so" (CRC mismatch).

done.
Loaded symbols for /opt/firebird/UDF/ib_udf.so
Reading symbols from /opt/firebird/UDF/udflib...done.
Loaded symbols for /opt/firebird/UDF/udflib
Reading symbols from /usr/lib/libgds.so...done.
Loaded symbols for /usr/lib/libgds.so
Core was generated by `/opt/firebird/bin/fbserver'.
Program terminated with signal 11, Segmentation fault.
#0 0x006c6370 in pthread_mutex_lock () from /lib/libpthread.so.0
(gdb) info thread
  266 process 22216 0xb7f0c402 in __kernel_vsyscall ()
  265 process 22224 0xb7f0c402 in __kernel_vsyscall ()
  264 process 22225 0xb7f0c402 in __kernel_vsyscall ()
  263 process 22226 0xb7f0c402 in __kernel_vsyscall ()
  262 process 22227 0xb7f0c402 in __kernel_vsyscall ()
  261 process 11947 0xb7f0c402 in __kernel_vsyscall ()
  260 process 12664 0xb7f0c402 in __kernel_vsyscall ()
  259 process 13160 0xb7f0c402 in __kernel_vsyscall ()
  258 process 13659 0xb7f0c402 in __kernel_vsyscall ()
  257 process 13660 0xb7f0c402 in __kernel_vsyscall ()
  256 process 13661 0xb7f0c402 in __kernel_vsyscall ()
  255 process 13662 0xb7f0c402 in __kernel_vsyscall ()
  254 process 13663 0xb7f0c402 in __kernel_vsyscall ()
  253 process 13664 0xb7f0c402 in __kernel_vsyscall ()
  252 process 13665 0xb7f0c402 in __kernel_vsyscall ()
  251 process 13666 0xb7f0c402 in __kernel_vsyscall ()
  250 process 13667 0xb7f0c402 in __kernel_vsyscall ()
  249 process 13668 0xb7f0c402 in __kernel_vsyscall ()
  248 process 13669 0xb7f0c402 in __kernel_vsyscall ()
  247 process 13670 0xb7f0c402 in __kernel_vsyscall ()
  246 process 13671 0xb7f0c402 in __kernel_vsyscall ()
  245 process 13672 0xb7f0c402 in __kernel_vsyscall ()
  244 process 13673 0xb7f0c402 in __kernel_vsyscall ()
  243 process 13674 0xb7f0c402 in __kernel_vsyscall ()
  242 process 13675 0xb7f0c402 in __kernel_vsyscall ()
  241 process 13676 0xb7f0c402 in __kernel_vsyscall ()
  240 process 13677 0xb7f0c402 in __kernel_vsyscall ()
  239 process 13678 0xb7f0c402 in __kernel_vsyscall ()
  238 process 13679 0xb7f0c402 in __kernel_vsyscall ()
  237 process 13680 0xb7f0c402 in __kernel_vsyscall ()
  236 process 13681 0xb7f0c402 in __kernel_vsyscall ()
  235 process 13682 0xb7f0c402 in __kernel_vsyscall ()
  234 process 13683 0xb7f0c402 in __kernel_vsyscall ()
  233 process 13684 0xb7f0c402 in __kernel_vsyscall ()
  232 process 13685 0xb7f0c402 in __kernel_vsyscall ()
  231 process 13686 0xb7f0c402 in __kernel_vsyscall ()
  230 process 13687 0xb7f0c402 in __kernel_vsyscall ()
  229 process 13688 0xb7f0c402 in __kernel_vsyscall ()
  228 process 13689 0xb7f0c402 in __kernel_vsyscall ()
  227 process 13690 0xb7f0c402 in __kernel_vsyscall ()
  226 process 13691 0xb7f0c402 in __kernel_vsyscall ()
  225 process 13692 0xb7f0c402 in __kernel_vsyscall ()
  224 process 13693 0xb7f0c402 in __kernel_vsyscall ()
  223 process 13694 0xb7f0c402 in __kernel_vsyscall ()
  222 process 13695 0xb7f0c402 in __kernel_vsyscall ()
---Type <return> to continue, or q <return> to quit---
  221 process 13696 0xb7f0c402 in __kernel_vsyscall ()
  220 process 13697 0xb7f0c402 in __kernel_vsyscall ()
  219 process 13698 0xb7f0c402 in __kernel_vsyscall ()
  218 process 13699 0xb7f0c402 in __kernel_vsyscall ()
  217 process 13700 0xb7f0c402 in __kernel_vsyscall ()
  216 process 13701 0xb7f0c402 in __kernel_vsyscall ()
  215 process 13702 0xb7f0c402 in __kernel_vsyscall ()
  214 process 13703 0xb7f0c402 in __kernel_vsyscall ()
  213 process 13704 0xb7f0c402 in __kernel_vsyscall ()
  212 process 13705 0xb7f0c402 in __kernel_vsyscall ()
  211 process 13706 0xb7f0c402 in __kernel_vsyscall ()
  210 process 13707 0xb7f0c402 in __kernel_vsyscall ()
  209 process 13708 0xb7f0c402 in __kernel_vsyscall ()
  208 process 13709 0xb7f0c402 in __kernel_vsyscall ()
  207 process 13710 0xb7f0c402 in __kernel_vsyscall ()
  206 process 13711 0xb7f0c402 in __kernel_vsyscall ()
  205 process 13712 0xb7f0c402 in __kernel_vsyscall ()
  204 process 13713 0xb7f0c402 in __kernel_vsyscall ()
  203 process 13714 0xb7f0c402 in __kernel_vsyscall ()
  202 process 13715 0xb7f0c402 in __kernel_vsyscall ()
  201 process 13716 0xb7f0c402 in __kernel_vsyscall ()
  200 process 13717 0xb7f0c402 in __kernel_vsyscall ()
  199 process 13718 0xb7f0c402 in __kernel_vsyscall ()
  198 process 13719 0xb7f0c402 in __kernel_vsyscall ()
  197 process 13720 0xb7f0c402 in __kernel_vsyscall ()
  196 process 13721 0xb7f0c402 in __kernel_vsyscall ()
  195 process 13722 0xb7f0c402 in __kernel_vsyscall ()
  194 process 13723 0xb7f0c402 in __kernel_vsyscall ()
  193 process 13724 0xb7f0c402 in __kernel_vsyscall ()
  192 process 13725 0xb7f0c402 in __kernel_vsyscall ()
  191 process 13726 0xb7f0c402 in __kernel_vsyscall ()
  190 process 13727 0xb7f0c402 in __kernel_vsyscall ()
  189 process 13728 0xb7f0c402 in __kernel_vsyscall ()
  188 process 13729 0xb7f0c402 in __kernel_vsyscall ()
  187 process 13730 0xb7f0c402 in __kernel_vsyscall ()
  186 process 13731 0xb7f0c402 in __kernel_vsyscall ()
  185 process 13732 0xb7f0c402 in __kernel_vsyscall ()
  184 process 13733 0xb7f0c402 in __kernel_vsyscall ()
  183 process 13734 0xb7f0c402 in __kernel_vsyscall ()
  182 process 13735 0xb7f0c402 in __kernel_vsyscall ()
  181 process 13736 0xb7f0c402 in __kernel_vsyscall ()
  180 process 13737 0xb7f0c402 in __kernel_vsyscall ()
  179 process 13738 0xb7f0c402 in __kernel_vsyscall ()
  178 process 13739 0xb7f0c402 in __kernel_vsyscall ()
  177 process 13740 0xb7f0c402 in __kernel_vsyscall ()
---Type <return> to continue, or q <return> to quit---
  176 process 13741 0xb7f0c402 in __kernel_vsyscall ()
  175 process 13742 0xb7f0c402 in __kernel_vsyscall ()
  174 process 13743 0xb7f0c402 in __kernel_vsyscall ()
  173 process 13744 0xb7f0c402 in __kernel_vsyscall ()
  172 process 13745 0xb7f0c402 in __kernel_vsyscall ()
  171 process 13746 0xb7f0c402 in __kernel_vsyscall ()
  170 process 13747 0xb7f0c402 in __kernel_vsyscall ()
  169 process 13748 0xb7f0c402 in __kernel_vsyscall ()
  168 process 13749 0xb7f0c402 in __kernel_vsyscall ()
  167 process 13750 0xb7f0c402 in __kernel_vsyscall ()
  166 process 13751 0xb7f0c402 in __kernel_vsyscall ()
  165 process 13752 0xb7f0c402 in __kernel_vsyscall ()
  164 process 13753 0xb7f0c402 in __kernel_vsyscall ()
  163 process 13754 0xb7f0c402 in __kernel_vsyscall ()
  162 process 13755 0xb7f0c402 in __kernel_vsyscall ()
  161 process 13756 0xb7f0c402 in __kernel_vsyscall ()
  160 process 13757 0xb7f0c402 in __kernel_vsyscall ()
  159 process 13758 0xb7f0c402 in __kernel_vsyscall ()
  158 process 13759 0xb7f0c402 in __kernel_vsyscall ()
  157 process 13760 0xb7f0c402 in __kernel_vsyscall ()
  156 process 13761 0xb7f0c402 in __kernel_vsyscall ()
  155 process 13762 0xb7f0c402 in __kernel_vsyscall ()
  154 process 13763 0xb7f0c402 in __kernel_vsyscall ()
  153 process 13764 0xb7f0c402 in __kernel_vsyscall ()
  152 process 13765 0xb7f0c402 in __kernel_vsyscall ()
  151 process 13766 0xb7f0c402 in __kernel_vsyscall ()
  150 process 13767 0xb7f0c402 in __kernel_vsyscall ()
  149 process 13768 0xb7f0c402 in __kernel_vsyscall ()
  148 process 13769 0xb7f0c402 in __kernel_vsyscall ()
  147 process 13770 0xb7f0c402 in __kernel_vsyscall ()
  146 process 13771 0xb7f0c402 in __kernel_vsyscall ()
  145 process 13772 0xb7f0c402 in __kernel_vsyscall ()
  144 process 13773 0xb7f0c402 in __kernel_vsyscall ()
  143 process 13774 0xb7f0c402 in __kernel_vsyscall ()
  142 process 13775 0xb7f0c402 in __kernel_vsyscall ()
  141 process 13776 0xb7f0c402 in __kernel_vsyscall ()
  140 process 13777 0xb7f0c402 in __kernel_vsyscall ()
  139 process 13778 0xb7f0c402 in __kernel_vsyscall ()
  138 process 13779 0xb7f0c402 in __kernel_vsyscall ()
  137 process 13780 0xb7f0c402 in __kernel_vsyscall ()
  136 process 13781 0xb7f0c402 in __kernel_vsyscall ()
  135 process 13782 0xb7f0c402 in __kernel_vsyscall ()
  134 process 13783 0xb7f0c402 in __kernel_vsyscall ()
  133 process 13784 0xb7f0c402 in __kernel_vsyscall ()
  132 process 13785 0xb7f0c402 in __kernel_vsyscall ()
---Type <return> to continue, or q <return> to quit---
  131 process 13786 0xb7f0c402 in __kernel_vsyscall ()
  130 process 13787 0xb7f0c402 in __kernel_vsyscall ()
  129 process 13788 0xb7f0c402 in __kernel_vsyscall ()
  128 process 13789 0xb7f0c402 in __kernel_vsyscall ()
  127 process 13790 0xb7f0c402 in __kernel_vsyscall ()
  126 process 13791 0xb7f0c402 in __kernel_vsyscall ()
  125 process 13792 0xb7f0c402 in __kernel_vsyscall ()
  124 process 13793 0xb7f0c402 in __kernel_vsyscall ()
  123 process 13794 0xb7f0c402 in __kernel_vsyscall ()
  122 process 13795 0xb7f0c402 in __kernel_vsyscall ()
  121 process 13796 0xb7f0c402 in __kernel_vsyscall ()
  120 process 13797 0xb7f0c402 in __kernel_vsyscall ()
  119 process 13798 0xb7f0c402 in __kernel_vsyscall ()
  118 process 13799 0xb7f0c402 in __kernel_vsyscall ()
  117 process 13800 0xb7f0c402 in __kernel_vsyscall ()
  116 process 13801 0xb7f0c402 in __kernel_vsyscall ()
  115 process 13802 0xb7f0c402 in __kernel_vsyscall ()
  114 process 13803 0xb7f0c402 in __kernel_vsyscall ()
  113 process 13804 0xb7f0c402 in __kernel_vsyscall ()
  112 process 13805 0xb7f0c402 in __kernel_vsyscall ()
  111 process 13806 0xb7f0c402 in __kernel_vsyscall ()
  110 process 13807 0xb7f0c402 in __kernel_vsyscall ()
  109 process 13808 0xb7f0c402 in __kernel_vsyscall ()
  108 process 13809 0xb7f0c402 in __kernel_vsyscall ()
  107 process 13810 0xb7f0c402 in __kernel_vsyscall ()
  106 process 13811 0xb7f0c402 in __kernel_vsyscall ()
  105 process 13812 0xb7f0c402 in __kernel_vsyscall ()
  104 process 13813 0xb7f0c402 in __kernel_vsyscall ()
  103 process 13814 0xb7f0c402 in __kernel_vsyscall ()
  102 process 13815 0xb7f0c402 in __kernel_vsyscall ()
  101 process 13816 0xb7f0c402 in __kernel_vsyscall ()
  100 process 13817 0xb7f0c402 in __kernel_vsyscall ()
  99 process 13818 0xb7f0c402 in __kernel_vsyscall ()
  98 process 13819 0xb7f0c402 in __kernel_vsyscall ()
  97 process 13820 0xb7f0c402 in __kernel_vsyscall ()
  96 process 13821 0xb7f0c402 in __kernel_vsyscall ()
  95 process 13822 0xb7f0c402 in __kernel_vsyscall ()
  94 process 13823 0xb7f0c402 in __kernel_vsyscall ()
  93 process 13824 0xb7f0c402 in __kernel_vsyscall ()
  92 process 13825 0xb7f0c402 in __kernel_vsyscall ()
  91 process 13826 0xb7f0c402 in __kernel_vsyscall ()
  90 process 13827 0xb7f0c402 in __kernel_vsyscall ()
  89 process 13828 0xb7f0c402 in __kernel_vsyscall ()
  88 process 13829 0xb7f0c402 in __kernel_vsyscall ()
  87 process 13830 0xb7f0c402 in __kernel_vsyscall ()
---Type <return> to continue, or q <return> to quit---
  86 process 13831 0xb7f0c402 in __kernel_vsyscall ()
  85 process 13832 0xb7f0c402 in __kernel_vsyscall ()
  84 process 13833 0xb7f0c402 in __kernel_vsyscall ()
  83 process 13834 0xb7f0c402 in __kernel_vsyscall ()
  82 process 13835 0xb7f0c402 in __kernel_vsyscall ()
  81 process 13836 0xb7f0c402 in __kernel_vsyscall ()
  80 process 13837 0xb7f0c402 in __kernel_vsyscall ()
  79 process 13838 0xb7f0c402 in __kernel_vsyscall ()
  78 process 13839 0xb7f0c402 in __kernel_vsyscall ()
  77 process 13840 0xb7f0c402 in __kernel_vsyscall ()
  76 process 13841 0xb7f0c402 in __kernel_vsyscall ()
  75 process 13842 0xb7f0c402 in __kernel_vsyscall ()
  74 process 13843 0xb7f0c402 in __kernel_vsyscall ()
  73 process 13844 0xb7f0c402 in __kernel_vsyscall ()
  72 process 13845 0xb7f0c402 in __kernel_vsyscall ()
  71 process 13846 0xb7f0c402 in __kernel_vsyscall ()
  70 process 13847 0xb7f0c402 in __kernel_vsyscall ()
  69 process 13848 0xb7f0c402 in __kernel_vsyscall ()
  68 process 13849 0xb7f0c402 in __kernel_vsyscall ()
  67 process 13850 0xb7f0c402 in __kernel_vsyscall ()
  66 process 13851 0xb7f0c402 in __kernel_vsyscall ()
  65 process 13852 0xb7f0c402 in __kernel_vsyscall ()
  64 process 13853 0xb7f0c402 in __kernel_vsyscall ()
  63 process 13854 0xb7f0c402 in __kernel_vsyscall ()
  62 process 13855 0xb7f0c402 in __kernel_vsyscall ()
  61 process 13856 0xb7f0c402 in __kernel_vsyscall ()
  60 process 13857 0xb7f0c402 in __kernel_vsyscall ()
  59 process 13858 0xb7f0c402 in __kernel_vsyscall ()
  58 process 13859 0xb7f0c402 in __kernel_vsyscall ()
  57 process 13860 0xb7f0c402 in __kernel_vsyscall ()
  56 process 13861 0xb7f0c402 in __kernel_vsyscall ()
  55 process 13862 0xb7f0c402 in __kernel_vsyscall ()
  54 process 13863 0xb7f0c402 in __kernel_vsyscall ()
  53 process 13864 0xb7f0c402 in __kernel_vsyscall ()
  52 process 13865 0xb7f0c402 in __kernel_vsyscall ()
  51 process 13866 0xb7f0c402 in __kernel_vsyscall ()
  50 process 13867 0xb7f0c402 in __kernel_vsyscall ()
  49 process 13868 0xb7f0c402 in __kernel_vsyscall ()
  48 process 13869 0xb7f0c402 in __kernel_vsyscall ()
  47 process 13870 0xb7f0c402 in __kernel_vsyscall ()
  46 process 13871 0xb7f0c402 in __kernel_vsyscall ()
  45 process 13872 0xb7f0c402 in __kernel_vsyscall ()
  44 process 13873 0xb7f0c402 in __kernel_vsyscall ()
  43 process 13874 0xb7f0c402 in __kernel_vsyscall ()
  42 process 13875 0xb7f0c402 in __kernel_vsyscall ()
---Type <return> to continue, or q <return> to quit---
  41 process 13876 0xb7f0c402 in __kernel_vsyscall ()
  40 process 13877 0xb7f0c402 in __kernel_vsyscall ()
  39 process 13878 0xb7f0c402 in __kernel_vsyscall ()
  38 process 13879 0xb7f0c402 in __kernel_vsyscall ()
  37 process 13880 0xb7f0c402 in __kernel_vsyscall ()
  36 process 13881 0xb7f0c402 in __kernel_vsyscall ()
  35 process 13882 0xb7f0c402 in __kernel_vsyscall ()
  34 process 13883 0xb7f0c402 in __kernel_vsyscall ()
  33 process 13884 0xb7f0c402 in __kernel_vsyscall ()
  32 process 13885 0xb7f0c402 in __kernel_vsyscall ()
  31 process 13886 0xb7f0c402 in __kernel_vsyscall ()
  30 process 13887 0xb7f0c402 in __kernel_vsyscall ()
  29 process 13888 0xb7f0c402 in __kernel_vsyscall ()
  28 process 13889 0xb7f0c402 in __kernel_vsyscall ()
  27 process 13890 0xb7f0c402 in __kernel_vsyscall ()
  26 process 13891 0xb7f0c402 in __kernel_vsyscall ()
  25 process 13892 0xb7f0c402 in __kernel_vsyscall ()
  24 process 13893 0xb7f0c402 in __kernel_vsyscall ()
  23 process 13894 0xb7f0c402 in __kernel_vsyscall ()
  22 process 13895 0xb7f0c402 in __kernel_vsyscall ()
  21 process 13896 0xb7f0c402 in __kernel_vsyscall ()
  20 process 13897 0xb7f0c402 in __kernel_vsyscall ()
  19 process 13898 0xb7f0c402 in __kernel_vsyscall ()
  18 process 13899 0xb7f0c402 in __kernel_vsyscall ()
  17 process 13900 0xb7f0c402 in __kernel_vsyscall ()
  16 process 13901 0xb7f0c402 in __kernel_vsyscall ()
  15 process 13902 0xb7f0c402 in __kernel_vsyscall ()
  14 process 13903 0xb7f0c402 in __kernel_vsyscall ()
  13 process 13904 0xb7f0c402 in __kernel_vsyscall ()
  12 process 13905 0xb7f0c402 in __kernel_vsyscall ()
  11 process 13906 0xb7f0c402 in __kernel_vsyscall ()
  10 process 13907 0xb7f0c402 in __kernel_vsyscall ()
  9 process 13908 0xb7f0c402 in __kernel_vsyscall ()
  8 process 13909 0xb7f0c402 in __kernel_vsyscall ()
  7 process 13910 0xb7f0c402 in __kernel_vsyscall ()
  6 process 13911 0xb7f0c402 in __kernel_vsyscall ()
  5 process 13912 0xb7f0c402 in __kernel_vsyscall ()
  4 process 13913 0xb7f0c402 in __kernel_vsyscall ()
  3 process 13914 0xb7f0c402 in __kernel_vsyscall ()
  2 process 13915 0xb7f0c402 in __kernel_vsyscall ()
* 1 process 22213 0x006c6370 in pthread_mutex_lock () from /lib/libpthread.so.0
(gdb) t 1
[Switching to thread 1 (process 22213)]#0 0x006c6370 in pthread_mutex_lock () from /lib/libpthread.so.0
(gdb) bt
#0 0x006c6370 in pthread_mutex_lock () from /lib/libpthread.so.0
#1 0x082f2201 in Firebird::MemoryPool::deallocate (this=0x0, block=0x9799194c) at ../src/common/classes/locks.h:133
#2 0x082f8aad in Firebird::status_exception::release_vector (this=0x56c5) at ../src/common/classes/alloc.h:83
#3 0x082f8b1d in ~status_exception (this=0x9e5dcc0) at ../src/common/fb_exception.cpp:182
#4 0x082f9f5a in ~system_call_failed (this=0x9799194c)
#5 0xb7ca93a8 in strcpy () from /usr/lib/libstdc++.so.5
#6 0x00c4dcfd in _Unwind_DeleteException () from /lib/libgcc_s.so.1
#7 0xb7ca83c0 in __cxa_end_catch () from /usr/lib/libstdc++.so.5
#8 0x080755e3 in gds__thread_start (entrypoint=0x9799194c, arg=0x9799194c, priority=3, flags=1, thd_id=0x0) at ../src/include/fb_exception.h:113
#9 0x08054de3 in SRVR_multi_thread (main_port=0xb7c111cc, flags=2) at ../src/remote/server.cpp:551
#10 0x080541fc in server_main (argc=22213, argv=0xbfb82890) at ../src/remote/inet_server.cpp:448
#11 0x0805d7b9 in main (argc=22213, argv=0x56c5) at ../src/remote/server_stub.cpp:12
(gdb) t 2
[Switching to thread 2 (process 13915)]#0 0xb7f0c402 in __kernel_vsyscall ()
(gdb) bt
#0 0xb7f0c402 in __kernel_vsyscall ()
#1 0x006c848c in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib/libpthread.so.0
#2 0x08070917 in ISC_event_wait (count=1, events=0xbf717ac4, values=0xbf717ac8, micro_seconds=120000000, timeout_handler=0, handler_arg=0xa436e084)
    at ../src/jrd/isc_sync.cpp:671
#3 0x0812f4f0 in latch_bdb (tdbb=0xbf717e70, type=LATCH_exclusive, bdb=0xb797946c, page={pageNum = 2629149, pageSpaceID = 1}, latch_wait=1)
    at ../src/jrd/cch.cpp:5455
#4 0x0812f0a7 in get_buffer (tdbb=0xbf717e70, page={pageNum = 2629149, pageSpaceID = 1}, latch=LATCH_exclusive, latch_wait=1)
    at ../src/jrd/pag.h:173
#5 0x0812ab94 in CCH_fetch_lock (tdbb=0xbf717e70, window=0xbf717ce0, lock_type=6, wait=1, latch_wait=1, page_type=3 '\003') at ../src/jrd/pag.h:173
#6 0x0812a887 in CCH_fetch (tdbb=0xbf717e70, window=0xbf717ce0, lock_type=6, page_type=3 '\003', checksum=1, latch_wait=31968, read_shadow=true)
    at ../src/jrd/cch_proto.h:91
#7 0x081fe0ab in TRA_set_state (tdbb=0xbf717e70, transaction=0xbf717e70, number=821802, state=3) at ../src/jrd/cch_proto.h:71
#8 0x081fddd1 in TRA_rollback (tdbb=0xbf717e70, transaction=0x8e5b45d0) at ../src/jrd/tra.cpp:1299
#9 0x081a3877 in rollback (tdbb=0xbf717e70, next=0x0, status_vector=0xbf718080, retaining_flag=false) at ../src/jrd/jrd.cpp:6313
#10 0x0819d01f in jrd8_rollback_transaction (user_status=0x8e5b45d0, tra_handle=0x8ff901a0) at ../src/jrd/jrd.cpp:3371
#11 0x08083ff7 in isc_rollback_transaction (user_status=0xbf718080, tra_handle=0xac867218) at ../src/jrd/why.cpp:446
#12 0x080556d9 in rem_port::disconnect (this=0x8fc78014, sendL=0x97914708, receiveL=0x979149bc) at ../src/remote/server.cpp:1533
#13 0x0805ba75 in process_packet2 (port=0x8fc78014, sendL=0x97914708, receive=0x979149bc, result=0xbf71834c) at ../src/remote/server.cpp:3471
#14 0x080590d7 in process_packet (port=0x8fc78014, sendL=0xfffffdfc, receive=0xfffffdfc, result=0xbf71834c) at ../src/remote/server.cpp:3363
#15 0x0805c7d3 in loopThread (flags=0x2) at ../src/remote/server.cpp:5381
#16 0x08075770 in threadStart (arg=0xaf12cc60) at ../src/jrd/ThreadData.cpp:272
#17 0x006c444b in start_thread () from /lib/libpthread.so.0
#18 0x0060780e in clone () from /lib/libc.so.6
(gdb) t 3
[Switching to thread 3 (process 13914)]#0 0xb7f0c402 in __kernel_vsyscall ()
(gdb) bt
#0 0xb7f0c402 in __kernel_vsyscall ()
#1 0x006c848c in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib/libpthread.so.0
#2 0x08070917 in ISC_event_wait (count=1, events=0xbed16ac4, values=0xbed16ac8, micro_seconds=120000000, timeout_handler=0, handler_arg=0xb796bc44)
    at ../src/jrd/isc_sync.cpp:671
#3 0x0812f4f0 in latch_bdb (tdbb=0xbed16e70, type=LATCH_exclusive, bdb=0xb797946c, page={pageNum = 2629149, pageSpaceID = 1}, latch_wait=1)
    at ../src/jrd/cch.cpp:5455
#4 0x0812f0a7 in get_buffer (tdbb=0xbed16e70, page={pageNum = 2629149, pageSpaceID = 1}, latch=LATCH_exclusive, latch_wait=1)
    at ../src/jrd/pag.h:173
#5 0x0812ab94 in CCH_fetch_lock (tdbb=0xbed16e70, window=0xbed16ce0, lock_type=6, wait=1, latch_wait=1, page_type=3 '\003') at ../src/jrd/pag.h:173
#6 0x0812a887 in CCH_fetch (tdbb=0xbed16e70, window=0xbed16ce0, lock_type=6, page_type=3 '\003', checksum=1, latch_wait=27872, read_shadow=true)
    at ../src/jrd/cch_proto.h:91
#7 0x081fe0ab in TRA_set_state (tdbb=0xbed16e70, transaction=0xbed16e70, number=821815, state=3) at ../src/jrd/cch_proto.h:71
#8 0x081fddd1 in TRA_rollback (tdbb=0xbed16e70, transaction=0xa817c498) at ../src/jrd/tra.cpp:1299
#9 0x081a3877 in rollback (tdbb=0xbed16e70, next=0x0, status_vector=0xbed17080, retaining_flag=false) at ../src/jrd/jrd.cpp:6313
#10 0x0819d01f in jrd8_rollback_transaction (user_status=0xa817c498, tra_handle=0xabee15f4) at ../src/jrd/jrd.cpp:3371
#11 0x08083ff7 in isc_rollback_transaction (user_status=0xbed17080, tra_handle=0x905f8ecc) at ../src/jrd/why.cpp:446
#12 0x080556d9 in rem_port::disconnect (this=0xaecc1dfc, sendL=0x97914c88, receiveL=0x97914f3c) at ../src/remote/server.cpp:1533
#13 0x0805ba75 in process_packet2 (port=0xaecc1dfc, sendL=0x97914c88, receive=0x97914f3c, result=0xbed1734c) at ../src/remote/server.cpp:3471
#14 0x080590d7 in process_packet (port=0xaecc1dfc, sendL=0xfffffdfc, receive=0xfffffdfc, result=0xbed1734c) at ../src/remote/server.cpp:3363
#15 0x0805c7d3 in loopThread (flags=0x2) at ../src/remote/server.cpp:5381
#16 0x08075770 in threadStart (arg=0xaf12cc60) at ../src/jrd/ThreadData.cpp:272
#17 0x006c444b in start_thread () from /lib/libpthread.so.0
#18 0x0060780e in clone () from /lib/libc.so.6
(gdb) t 4
[Switching to thread 4 (process 13913)]#0 0xb7f0c402 in __kernel_vsyscall ()
(gdb) bt
#0 0xb7f0c402 in __kernel_vsyscall ()
#1 0x006c848c in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib/libpthread.so.0
#2 0x08070917 in ISC_event_wait (count=1, events=0xbe315ac4, values=0xbe315ac8, micro_seconds=120000000, timeout_handler=0, handler_arg=0xb796cb1c)
    at ../src/jrd/isc_sync.cpp:671
#3 0x0812f4f0 in latch_bdb (tdbb=0xbe315e70, type=LATCH_exclusive, bdb=0xb797946c, page={pageNum = 2629149, pageSpaceID = 1}, latch_wait=1)
    at ../src/jrd/cch.cpp:5455
#4 0x0812f0a7 in get_buffer (tdbb=0xbe315e70, page={pageNum = 2629149, pageSpaceID = 1}, latch=LATCH_exclusive, latch_wait=1)
    at ../src/jrd/pag.h:173
#5 0x0812ab94 in CCH_fetch_lock (tdbb=0xbe315e70, window=0xbe315ce0, lock_type=6, wait=1, latch_wait=1, page_type=3 '\003') at ../src/jrd/pag.h:173
#6 0x0812a887 in CCH_fetch (tdbb=0xbe315e70, window=0xbe315ce0, lock_type=6, page_type=3 '\003', checksum=1, latch_wait=23776, read_shadow=true)
    at ../src/jrd/cch_proto.h:91
#7 0x081fe0ab in TRA_set_state (tdbb=0xbe315e70, transaction=0xbe315e70, number=821828, state=3) at ../src/jrd/cch_proto.h:71
#8 0x081fddd1 in TRA_rollback (tdbb=0xbe315e70, transaction=0xaf8b102c) at ../src/jrd/tra.cpp:1299
#9 0x081a3877 in rollback (tdbb=0xbe315e70, next=0x0, status_vector=0xbe316080, retaining_flag=false) at ../src/jrd/jrd.cpp:6313
#10 0x0819d01f in jrd8_rollback_transaction (user_status=0xaf8b102c, tra_handle=0x8e6c4f40) at ../src/jrd/jrd.cpp:3371
#11 0x08083ff7 in isc_rollback_transaction (user_status=0xbe316080, tra_handle=0x93a80708) at ../src/jrd/why.cpp:446
#12 0x080556d9 in rem_port::disconnect (this=0xa8025dfc, sendL=0x97915208, receiveL=0x979154bc) at ../src/remote/server.cpp:1533
#13 0x0805ba75 in process_packet2 (port=0xa8025dfc, sendL=0x97915208, receive=0x979154bc, result=0xbe31634c) at ../src/remote/server.cpp:3471
#14 0x080590d7 in process_packet (port=0xa8025dfc, sendL=0xfffffdfc, receive=0xfffffdfc, result=0xbe31634c) at ../src/remote/server.cpp:3363
#15 0x0805c7d3 in loopThread (flags=0x2) at ../src/remote/server.cpp:5381
#16 0x08075770 in threadStart (arg=0xaf12cc60) at ../src/jrd/ThreadData.cpp:272
#17 0x006c444b in start_thread () from /lib/libpthread.so.0
#18 0x0060780e in clone () from /lib/libc.so.6
(gdb) t 5
[Switching to thread 5 (process 13912)]#0 0xb7f0c402 in __kernel_vsyscall ()
(gdb) bt
#0 0xb7f0c402 in __kernel_vsyscall ()
#1 0x006c848c in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib/libpthread.so.0
#2 0x08070917 in ISC_event_wait (count=1, events=0xbd914ac4, values=0xbd914ac8, micro_seconds=120000000, timeout_handler=0, handler_arg=0xa436eb04)
    at ../src/jrd/isc_sync.cpp:671
#3 0x0812f4f0 in latch_bdb (tdbb=0xbd914e70, type=LATCH_exclusive, bdb=0xb797946c, page={pageNum = 2629149, pageSpaceID = 1}, latch_wait=1)
    at ../src/jrd/cch.cpp:5455
#4 0x0812f0a7 in get_buffer (tdbb=0xbd914e70, page={pageNum = 2629149, pageSpaceID = 1}, latch=LATCH_exclusive, latch_wait=1)
    at ../src/jrd/pag.h:173
#5 0x0812ab94 in CCH_fetch_lock (tdbb=0xbd914e70, window=0xbd914ce0, lock_type=6, wait=1, latch_wait=1, page_type=3 '\003') at ../src/jrd/pag.h:173
#6 0x0812a887 in CCH_fetch (tdbb=0xbd914e70, window=0xbd914ce0, lock_type=6, page_type=3 '\003', checksum=1, latch_wait=19680, read_shadow=true)
    at ../src/jrd/cch_proto.h:91
#7 0x081fe0ab in TRA_set_state (tdbb=0xbd914e70, transaction=0xbd914e70, number=821853, state=3) at ../src/jrd/cch_proto.h:71
#8 0x081fddd1 in TRA_rollback (tdbb=0xbd914e70, transaction=0x8dcc8c3c) at ../src/jrd/tra.cpp:1299
#9 0x081a3877 in rollback (tdbb=0xbd914e70, next=0x0, status_vector=0xbd915080, retaining_flag=false) at ../src/jrd/jrd.cpp:6313
#10 0x0819d01f in jrd8_rollback_transaction (user_status=0x8dcc8c3c, tra_handle=0x8fb9d220) at ../src/jrd/jrd.cpp:3371
#11 0x08083ff7 in isc_rollback_transaction (user_status=0xbd915080, tra_handle=0xabb26754) at ../src/jrd/why.cpp:446
#12 0x080556d9 in rem_port::disconnect (this=0x8f1a9014, sendL=0x97915788, receiveL=0x97915a3c) at ../src/remote/server.cpp:1533
#13 0x0805ba75 in process_packet2 (port=0x8f1a9014, sendL=0x97915788, receive=0x97915a3c, result=0xbd91534c) at ../src/remote/server.cpp:3471
#14 0x080590d7 in process_packet (port=0x8f1a9014, sendL=0xfffffdfc, receive=0xfffffdfc, result=0xbd91534c) at ../src/remote/server.cpp:3363
#15 0x0805c7d3 in loopThread (flags=0x2) at ../src/remote/server.cpp:5381
#16 0x08075770 in threadStart (arg=0xaf12cc60) at ../src/jrd/ThreadData.cpp:272
#17 0x006c444b in start_thread () from /lib/libpthread.so.0
#18 0x0060780e in clone () from /lib/libc.so.6
(gdb) t 266
[Switching to thread 266 (process 22216)]#0 0xb7f0c402 in __kernel_vsyscall ()
(gdb) bt
#0 0xb7f0c402 in __kernel_vsyscall ()
#1 0x006ca21e in sem_wait@GLIBC_2.0 () from /lib/libpthread.so.0
#2 0x08054648 in shutdown_thread (arg=0x0) at ../src/common/classes/semaphore.h:164
#3 0x08075770 in threadStart (arg=0xb7c10c00) at ../src/jrd/ThreadData.cpp:272
#4 0x006c444b in start_thread () from /lib/libpthread.so.0
#5 0x0060780e in clone () from /lib/libc.so.6
(gdb) t 265
[Switching to thread 265 (process 22224)]#0 0xb7f0c402 in __kernel_vsyscall ()
(gdb) bt
#0 0xb7f0c402 in __kernel_vsyscall ()
#1 0x006c848c in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib/libpthread.so.0
#2 0x08070917 in ISC_event_wait (count=1, events=0xb5d4a1d8, values=0xb5d4a1dc, micro_seconds=10000000, timeout_handler=0, handler_arg=0x0)
    at ../src/jrd/isc_sync.cpp:671
#3 0x0812dec6 in cache_writer (arg=0xb7948810) at ../src/jrd/cch.cpp:4371
#4 0x08075770 in threadStart (arg=0xb7c10c00) at ../src/jrd/ThreadData.cpp:272
#5 0x006c444b in start_thread () from /lib/libpthread.so.0
#6 0x0060780e in clone () from /lib/libc.so.6
(gdb) t 264
[Switching to thread 264 (process 22225)]#0 0xb7f0c402 in __kernel_vsyscall ()
(gdb) bt
#0 0xb7f0c402 in __kernel_vsyscall ()
#1 0x006c848c in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib/libpthread.so.0
#2 0x08070917 in ISC_event_wait (count=1, events=0xb5349168, values=0xb534916c, micro_seconds=10000000, timeout_handler=0, handler_arg=0x0)
    at ../src/jrd/isc_sync.cpp:671
#3 0x08214f15 in garbage_collector (arg=0xb7948810) at ../src/jrd/vio.cpp:4103
#4 0x08075770 in threadStart (arg=0xb7c10c00) at ../src/jrd/ThreadData.cpp:272
#5 0x006c444b in start_thread () from /lib/libpthread.so.0
#6 0x0060780e in clone () from /lib/libc.so.6
(gdb) t 263
[Switching to thread 263 (process 22226)]#0 0xb7f0c402 in __kernel_vsyscall ()
(gdb) bt
#0 0xb7f0c402 in __kernel_vsyscall ()
#1 0x006c848c in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib/libpthread.so.0
#2 0x08070917 in ISC_event_wait (count=1, events=0xb29431d8, values=0xb29431dc, micro_seconds=10000000, timeout_handler=0, handler_arg=0x0)
    at ../src/jrd/isc_sync.cpp:671
#3 0x0812dec6 in cache_writer (arg=0xb7958810) at ../src/jrd/cch.cpp:4371
#4 0x08075770 in threadStart (arg=0xb7bfa9a4) at ../src/jrd/ThreadData.cpp:272
#5 0x006c444b in start_thread () from /lib/libpthread.so.0
#6 0x0060780e in clone () from /lib/libc.so.6
(gdb) t 262
[Switching to thread 262 (process 22227)]#0 0xb7f0c402 in __kernel_vsyscall ()
(gdb) bt
#0 0xb7f0c402 in __kernel_vsyscall ()
#1 0x006c848c in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib/libpthread.so.0
#2 0x08070917 in ISC_event_wait (count=1, events=0xb1f42168, values=0xb1f4216c, micro_seconds=10000000, timeout_handler=0, handler_arg=0x0)
    at ../src/jrd/isc_sync.cpp:671
#3 0x08214f15 in garbage_collector (arg=0xb7958810) at ../src/jrd/vio.cpp:4103
#4 0x08075770 in threadStart (arg=0xb7bfa9a4) at ../src/jrd/ThreadData.cpp:272
#5 0x006c444b in start_thread () from /lib/libpthread.so.0
#6 0x0060780e in clone () from /lib/libc.so.6
(gdb) t 261
[Switching to thread 261 (process 11947)]#0 0xb7f0c402 in __kernel_vsyscall ()
(gdb) bt
#0 0xb7f0c402 in __kernel_vsyscall ()
#1 0x006caa0e in __lll_mutex_lock_wait () from /lib/libpthread.so.0
#2 0x006c6883 in _L_mutex_lock_79 () from /lib/libpthread.so.0
#3 0x006c63ad in pthread_mutex_lock () from /lib/libpthread.so.0
#4 0x082236ac in THD_rec_mutex_lock (rec_mutex=0x841dd00) at ../src/common/classes/locks.h:133
#5 0x0819987b in jrd8_detach_database (user_status=0xa7d6d080, handle=0xaf8f6e40) at ../src/jrd/jrd.cpp:2288
#6 0x0807c4b8 in isc_detach_database (user_status=0xa7d6d080, handle=0xadf74914) at ../src/jrd/why.cpp:446
#7 0x0805570d in rem_port::disconnect (this=0xa5d08cf4, sendL=0x9811242c, receiveL=0x981126e0) at ../src/remote/server.cpp:1551
#8 0x0805ba75 in process_packet2 (port=0xa5d08cf4, sendL=0x9811242c, receive=0x981126e0, result=0xa7d6d34c) at ../src/remote/server.cpp:3471
#9 0x080590d7 in process_packet (port=0xa5d08cf4, sendL=0xfffffe00, receive=0xfffffe00, result=0xa7d6d34c) at ../src/remote/server.cpp:3363
#10 0x0805c7d3 in loopThread (flags=0x2) at ../src/remote/server.cpp:5381
#11 0x08075770 in threadStart (arg=0x979904d0) at ../src/jrd/ThreadData.cpp:272
#12 0x006c444b in start_thread () from /lib/libpthread.so.0
#13 0x0060780e in clone () from /lib/libc.so.6
(gdb) t 260
[Switching to thread 260 (process 12664)]#0 0xb7f0c402 in __kernel_vsyscall ()
(gdb) bt
#0 0xb7f0c402 in __kernel_vsyscall ()
#1 0x006caa0e in __lll_mutex_lock_wait () from /lib/libpthread.so.0
#2 0x006c6883 in _L_mutex_lock_79 () from /lib/libpthread.so.0
#3 0x006c63ad in pthread_mutex_lock () from /lib/libpthread.so.0
#4 0x082236ac in THD_rec_mutex_lock (rec_mutex=0x841dd00) at ../src/common/classes/locks.h:133
#5 0x0819987b in jrd8_detach_database (user_status=0x9fb60080, handle=0xadd73288) at ../src/jrd/jrd.cpp:2288
#6 0x0807c4b8 in isc_detach_database (user_status=0x9fb60080, handle=0xae2266a8) at ../src/jrd/why.cpp:446
#7 0x0805570d in rem_port::disconnect (this=0xa4ff4dfc, sendL=0xb627ec80, receiveL=0xb627ef34) at ../src/remote/server.cpp:1551
#8 0x0805ba75 in process_packet2 (port=0xa4ff4dfc, sendL=0xb627ec80, receive=0xb627ef34, result=0x9fb6034c) at ../src/remote/server.cpp:3471
#9 0x080590d7 in process_packet (port=0xa4ff4dfc, sendL=0xfffffe00, receive=0xfffffe00, result=0x9fb6034c) at ../src/remote/server.cpp:3363
#10 0x0805c7d3 in loopThread (flags=0x2) at ../src/remote/server.cpp:5381
#11 0x08075770 in threadStart (arg=0x8fc7c348) at ../src/jrd/ThreadData.cpp:272
#12 0x006c444b in start_thread () from /lib/libpthread.so.0
#13 0x0060780e in clone () from /lib/libc.so.6
(gdb) t 200
[Switching to thread 200 (process 13717)]#0 0xb7f0c402 in __kernel_vsyscall ()
(gdb) bt
#0 0xb7f0c402 in __kernel_vsyscall ()
#1 0x006caa0e in __lll_mutex_lock_wait () from /lib/libpthread.so.0
#2 0x006c6883 in _L_mutex_lock_79 () from /lib/libpthread.so.0
#3 0x006c63ad in pthread_mutex_lock () from /lib/libpthread.so.0
#4 0x082236ac in THD_rec_mutex_lock (rec_mutex=0x841dd00) at ../src/common/classes/locks.h:133
#5 0x0819987b in jrd8_detach_database (user_status=0x770e8080, handle=0xae0c1a78) at ../src/jrd/jrd.cpp:2288
#6 0x0807c4b8 in isc_detach_database (user_status=0x770e8080, handle=0xae0c1734) at ../src/jrd/why.cpp:446
#7 0x0805570d in rem_port::disconnect (this=0xadf77dfc, sendL=0xa4dea200, receiveL=0xa4dea4b4) at ../src/remote/server.cpp:1551
#8 0x0805ba75 in process_packet2 (port=0xadf77dfc, sendL=0xa4dea200, receive=0xa4dea4b4, result=0x770e834c) at ../src/remote/server.cpp:3471
#9 0x080590d7 in process_packet (port=0xadf77dfc, sendL=0xfffffe00, receive=0xfffffe00, result=0x770e834c) at ../src/remote/server.cpp:3363
#10 0x0805c7d3 in loopThread (flags=0x2) at ../src/remote/server.cpp:5381
#11 0x08075770 in threadStart (arg=0x8fb0b28c) at ../src/jrd/ThreadData.cpp:272
#12 0x006c444b in start_thread () from /lib/libpthread.so.0
#13 0x0060780e in clone () from /lib/libc.so.6
(gdb) t 150
[Switching to thread 150 (process 13767)]#0 0xb7f0c402 in __kernel_vsyscall ()
(gdb) bt
#0 0xb7f0c402 in __kernel_vsyscall ()
#1 0x006caa0e in __lll_mutex_lock_wait () from /lib/libpthread.so.0
#2 0x006c6883 in _L_mutex_lock_79 () from /lib/libpthread.so.0
#3 0x006c63ad in pthread_mutex_lock () from /lib/libpthread.so.0
#4 0x082236ac in THD_rec_mutex_lock (rec_mutex=0x841dd00) at ../src/common/classes/locks.h:133
#5 0x0819987b in jrd8_detach_database (user_status=0x57cb6080, handle=0x8e91dab4) at ../src/jrd/jrd.cpp:2288
#6 0x0807c4b8 in isc_detach_database (user_status=0x57cb6080, handle=0x8e1acf58) at ../src/jrd/why.cpp:446
#7 0x0805570d in rem_port::disconnect (this=0xaad06dfc, sendL=0xaa40dd08, receiveL=0xaa40dfbc) at ../src/remote/server.cpp:1551
#8 0x0805ba75 in process_packet2 (port=0xaad06dfc, sendL=0xaa40dd08, receive=0xaa40dfbc, result=0x57cb634c) at ../src/remote/server.cpp:3471
#9 0x080590d7 in process_packet (port=0xaad06dfc, sendL=0xfffffe00, receive=0xfffffe00, result=0x57cb634c) at ../src/remote/server.cpp:3363
#10 0x0805c7d3 in loopThread (flags=0x2) at ../src/remote/server.cpp:5381
#11 0x08075770 in threadStart (arg=0x97992a78) at ../src/jrd/ThreadData.cpp:272
#12 0x006c444b in start_thread () from /lib/libpthread.so.0
#13 0x0060780e in clone () from /lib/libc.so.6
(gdb) t 100
[Switching to thread 100 (process 13817)]#0 0xb7f0c402 in __kernel_vsyscall ()
(gdb) bt
#0 0xb7f0c402 in __kernel_vsyscall ()
#1 0x006caa0e in __lll_mutex_lock_wait () from /lib/libpthread.so.0
#2 0x006c6883 in _L_mutex_lock_79 () from /lib/libpthread.so.0
#3 0x006c63ad in pthread_mutex_lock () from /lib/libpthread.so.0
#4 0x082236ac in THD_rec_mutex_lock (rec_mutex=0x841dd00) at ../src/common/classes/locks.h:133
#5 0x0819987b in jrd8_detach_database (user_status=0x38884080, handle=0xad49ff90) at ../src/jrd/jrd.cpp:2288
#6 0x0807c4b8 in isc_detach_database (user_status=0x38884080, handle=0xaf36fa08) at ../src/jrd/why.cpp:446
#7 0x0805570d in rem_port::disconnect (this=0xb7823f04, sendL=0xa5522888, receiveL=0xa5522b3c) at ../src/remote/server.cpp:1551
#8 0x0805ba75 in process_packet2 (port=0xb7823f04, sendL=0xa5522888, receive=0xa5522b3c, result=0x3888434c) at ../src/remote/server.cpp:3471
#9 0x080590d7 in process_packet (port=0xb7823f04, sendL=0xfffffe00, receive=0xfffffe00, result=0x3888434c) at ../src/remote/server.cpp:3363
#10 0x0805c7d3 in loopThread (flags=0x2) at ../src/remote/server.cpp:5381
#11 0x08075770 in threadStart (arg=0x90588fd4) at ../src/jrd/ThreadData.cpp:272
#12 0x006c444b in start_thread () from /lib/libpthread.so.0
#13 0x0060780e in clone () from /lib/libc.so.6
(gdb) t 50
[Switching to thread 50 (process 13867)]#0 0xb7f0c402 in __kernel_vsyscall ()
(gdb) bt
#0 0xb7f0c402 in __kernel_vsyscall ()
#1 0x006c848c in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib/libpthread.so.0
#2 0x08070917 in ISC_event_wait (count=1, events=0x19451ac4, values=0x19451ac8, micro_seconds=120000000, timeout_handler=0, handler_arg=0xb796e9f4)
    at ../src/jrd/isc_sync.cpp:671
#3 0x0812f4f0 in latch_bdb (tdbb=0x19451e70, type=LATCH_exclusive, bdb=0xb71793ec, page={pageNum = 2629990, pageSpaceID = 1}, latch_wait=1)
    at ../src/jrd/cch.cpp:5455
#4 0x0812f0a7 in get_buffer (tdbb=0x19451e70, page={pageNum = 2629990, pageSpaceID = 1}, latch=LATCH_exclusive, latch_wait=1)
    at ../src/jrd/pag.h:173
#5 0x0812ab94 in CCH_fetch_lock (tdbb=0x19451e70, window=0x19451ce0, lock_type=6, wait=1, latch_wait=1, page_type=3 '\003') at ../src/jrd/pag.h:173
#6 0x0812a887 in CCH_fetch (tdbb=0x19451e70, window=0x19451ce0, lock_type=6, page_type=3 '\003', checksum=1, latch_wait=7392, read_shadow=true)
    at ../src/jrd/cch_proto.h:91
#7 0x081fe0ab in TRA_set_state (tdbb=0x19451e70, transaction=0x19451e70, number=856737, state=3) at ../src/jrd/cch_proto.h:71
#8 0x081fddd1 in TRA_rollback (tdbb=0x19451e70, transaction=0xb62ddab4) at ../src/jrd/tra.cpp:1299
#9 0x081a3877 in rollback (tdbb=0x19451e70, next=0x0, status_vector=0x19452080, retaining_flag=false) at ../src/jrd/jrd.cpp:6313
#10 0x0819d01f in jrd8_rollback_transaction (user_status=0xb62ddab4, tra_handle=0x8e814368) at ../src/jrd/jrd.cpp:3371
#11 0x08083ff7 in isc_rollback_transaction (user_status=0x19452080, tra_handle=0x904a88f8) at ../src/jrd/why.cpp:446
#12 0x080556d9 in rem_port::disconnect (this=0x8fa42f04, sendL=0xab080c00, receiveL=0xab080eb4) at ../src/remote/server.cpp:1533
#13 0x0805ba75 in process_packet2 (port=0x8fa42f04, sendL=0xab080c00, receive=0xab080eb4, result=0x1945234c) at ../src/remote/server.cpp:3471
#14 0x080590d7 in process_packet (port=0x8fa42f04, sendL=0xfffffdfc, receive=0xfffffdfc, result=0x1945234c) at ../src/remote/server.cpp:3363
#15 0x0805c7d3 in loopThread (flags=0x2) at ../src/remote/server.cpp:5381
#16 0x08075770 in threadStart (arg=0xaf12cc60) at ../src/jrd/ThreadData.cpp:272
#17 0x006c444b in start_thread () from /lib/libpthread.so.0
#18 0x0060780e in clone () from /lib/libc.so.6
(gdb) t 25
[Switching to thread 25 (process 13892)]#0 0xb7f0c402 in __kernel_vsyscall ()
(gdb) bt
#0 0xb7f0c402 in __kernel_vsyscall ()
#1 0x006c848c in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib/libpthread.so.0
#2 0x08070917 in ISC_event_wait (count=1, events=0x944cac4, values=0x944cac8, micro_seconds=120000000, timeout_handler=0, handler_arg=0xb796bbd4)
    at ../src/jrd/isc_sync.cpp:671
#3 0x0812f4f0 in latch_bdb (tdbb=0x944ce70, type=LATCH_exclusive, bdb=0xb71793ec, page={pageNum = 2629990, pageSpaceID = 1}, latch_wait=1)
    at ../src/jrd/cch.cpp:5455
#4 0x0812f0a7 in get_buffer (tdbb=0x944ce70, page={pageNum = 2629990, pageSpaceID = 1}, latch=LATCH_exclusive, latch_wait=1)
    at ../src/jrd/pag.h:173
#5 0x0812ab94 in CCH_fetch_lock (tdbb=0x944ce70, window=0x944cce0, lock_type=6, wait=1, latch_wait=1, page_type=3 '\003') at ../src/jrd/pag.h:173
#6 0x0812a887 in CCH_fetch (tdbb=0x944ce70, window=0x944cce0, lock_type=6, page_type=3 '\003', checksum=1, latch_wait=-13088, read_shadow=true)
    at ../src/jrd/cch_proto.h:91
#7 0x081fe0ab in TRA_set_state (tdbb=0x944ce70, transaction=0x944ce70, number=854242, state=3) at ../src/jrd/cch_proto.h:71
#8 0x081fddd1 in TRA_rollback (tdbb=0x944ce70, transaction=0x8e4e4208) at ../src/jrd/tra.cpp:1299
#9 0x081a3877 in rollback (tdbb=0x944ce70, next=0x0, status_vector=0x944d080, retaining_flag=false) at ../src/jrd/jrd.cpp:6313
#10 0x0819d01f in jrd8_rollback_transaction (user_status=0x8e4e4208, tra_handle=0xacd4cacc) at ../src/jrd/jrd.cpp:3371
#11 0x08083ff7 in isc_rollback_transaction (user_status=0x944d080, tra_handle=0x904a8ffc) at ../src/jrd/why.cpp:446
#12 0x080556d9 in rem_port::disconnect (this=0xa83d9cf4, sendL=0xa555bb80, receiveL=0xa555be34) at ../src/remote/server.cpp:1533
#13 0x0805ba75 in process_packet2 (port=0xa83d9cf4, sendL=0xa555bb80, receive=0xa555be34, result=0x944d34c) at ../src/remote/server.cpp:3471
#14 0x080590d7 in process_packet (port=0xa83d9cf4, sendL=0xfffffdfc, receive=0xfffffdfc, result=0x944d34c) at ../src/remote/server.cpp:3363
#15 0x0805c7d3 in loopThread (flags=0x2) at ../src/remote/server.cpp:5381
#16 0x08075770 in threadStart (arg=0xaf12cc60) at ../src/jrd/ThreadData.cpp:272
#17 0x006c444b in start_thread () from /lib/libpthread.so.0
#18 0x0060780e in clone () from /lib/libc.so.6
(gdb)


Andreas Kallenbach added a comment - 17/Feb/09 05:04 PM - edited
There were 266 threads in the core listing and I only enumerated the first and last 5 plus some in between. The core dump can be downloaded at http://www.infinityins.com/core.22213.tar.bz2 it was compressed using....

[root@localhost tmp]# tar -cjf core.22213.tar.bz2 core.22213

[root@localhost tmp]# ls -l core*
-rw------- 1 firebird firebird 3064692736 2009-02-16 08:44 core.22213
-rw------- 1 root root 3064692736 2009-02-17 13:55 core.22213.copy
-rw-r--r-- 1 root root 38228427 2009-02-17 14:00 core.22213.tar.bz2
[root@localhost tmp]#

Andreas Kallenbach added a comment - 18/Feb/09 03:36 PM
Maybe I am grasping at straws here... I'm not versed in Linux programming or its tools, but in the first thread I find this curious...

#4 0x082f9f5a in ~system_call_failed (this=0x9799194c)
#5 0xb7ca93a8 in strcpy () from /usr/lib/libstdc++.so.5
#6 0x00c4dcfd in _Unwind_DeleteException () from /lib/libgcc_s.so.1
#7 0xb7ca83c0 in __cxa_end_catch () from /usr/lib/libstdc++.so.5

libstdc++.so.5 is a dependency that I have always had to install on many of the Fedora/CentOS/Redhat Servers we use before installing firebird*.rpm.

I resolve the dependency by the following....

[root@localhost Desktop]# yum install libstdc++.so.5
Loading "fastestmirror" plugin
Loading mirror speeds from cached hostfile
 * base: holmes.umflint.edu
 * updates: yum.singlehop.com
 * addons: ftp.linux.ncsu.edu
 * extras: pubmirrors.reflected.net
Setting up Install Process
Parsing package install arguments
Resolving Dependencies
--> Running transaction check
---> Package compat-libstdc++-33.i386 0:3.2.3-61 set to be updated
--> Finished Dependency Resolution

Dependencies Resolved

=============================================================================
 Package Arch Version Repository Size
=============================================================================
Installing:
 compat-libstdc++-33 i386 3.2.3-61 base 232 k

Transaction Summary
=============================================================================
Install 1 Package(s)
Update 0 Package(s)
Remove 0 Package(s)

Total download size: 232 k
Is this ok [y/N]: y
Downloading Packages:
(1/1): compat-libstdc++-3 100% |=========================| 232 kB 00:00
Running rpm_check_debug
Running Transaction Test
Finished Transaction Test
Transaction Test Succeeded
Running Transaction
  Installing: compat-libstdc++-33 ######################### [1/1]

Installed: compat-libstdc++-33.i386 0:3.2.3-61
Complete!

Alexander Peshkov added a comment - 19/Feb/09 11:57 AM
Unfortunately I can't work with your core file - a lot of library version mismatches make it unusable. Therefore I'll have to ask you to provide some answers.

I see 2 main problems with provided backtraces. One (which directly caused AV) is an error in gds__thread_start. It will be interesting to try the following in gdb:
t 1
f 4
print *this
This should provide us information about what particular system call failed in gds__thread_start, and what was an error.

But sooner of all failure happened due to too many threads started. May be you should fix limits on your box (don't ask how, this is distro-dependent thing).

What is also strange is that a lot of threads wait for SS_MUTEX lock in detach database code - and this can be a reason for too many threads. Do you have any problems when detaching from server? Can you attach to the server without problems (that mutex blocks attaches too)? It will be very interesting to know, how many of 260 threads are waiting in this place (like thread 100):
#3 0x006c63ad in pthread_mutex_lock () from /lib/libpthread.so.0
 #4 0x082236ac in THD_rec_mutex_lock (rec_mutex=0x841dd00) at ../src/common/classes/locks.h:133
 #5 0x0819987b in jrd8_detach_database (user_status=0x38884080, handle=0xad49ff90) at ../src/jrd/jrd.cpp:2288
(I understand it's not interesting job to count threads in gdb, but...)


Andreas Kallenbach added a comment - 19/Feb/09 02:19 PM
(gdb) t 1
[Switching to thread 1 (process 22213)]#0 0x006c6370 in pthread_mutex_lock ()
   from /lib/libpthread.so.0
(gdb) f 4
#4 0x082f9f5a in ~system_call_failed (this=0x9799194c)
(gdb) print *this
$1 = {<Firebird::status_exception> = {<Firebird::Exception> = {_vptr.Exception = 0x902615ac},
    m_status_vector = {1600414053, 1634038371, 25972, 4, 1049132, -1212125164, 50331657, 553,
      -1751574168, 0, 0, 558925, 1441792, 1734631746, 1277194853, 1936287599, 1935894900,
      1850286124, 11875, 0}, m_strings_permanent = false}, errorCode = 0}
(gdb)

We have not had an issue with attaching/detaching from the server. This particular machine has been in use for almost 2 years and until this started has run very well. It was using 2.0.1 for most of that time until it was upgraded to the current version last month. It was upgraded after these issues started happening. I wanted to see if it would improve with a newer version. I am looking to install this on a newer machine this weekend.

Now, what may be interesting is that before the server hits this error, crashes, and restarts.... users report they can't connect to the database, the cpu shoots up to 100% utilization, stays at 100% for 5 minutes, and then crashes. Near the end of this, attempts to connect to the database start responding with various errors, "Can't find system environment variables, path issues, etc". Other than the times when the database crashes, connections to the database are pretty much instant.

When I have a little time later today.... I will do some more thread enumerating... my human "for" loop skills are not as efficient as the machine.

Andreas Kallenbach added a comment - 18/Mar/09 05:14 PM
I did switch our database to a similar spec'd machine and Operating System to check for hardware issues, but we are still receiving troublesome restarts on the database under load.

Now, I have noticed that the firebird.log contains a number of these errors right before it restarts...

INET/inet_error: read errno = 104

For instance this Monday, we had this error repeat 292 times

localhost.localdomain (Server) Mon Mar 16 18:37:31 2009
        INET/inet_error: read errno = 104

All occurred at 18:37:31 and was promptly followed with...

localhost.localdomain (Client) Mon Mar 16 18:37:31 2009
        /opt/firebird/bin/fbguard: /opt/firebird/bin/fbserver terminated abnormally (-1)

Alexander Peshkov added a comment - 19/Mar/09 06:48 AM
Andreas, I'm sure this is not HW issue. Final error in pthread_mutex_lock() is definitely "too many threads per process" (or may be per user, does not really matter). But the actual reason of your problems is not this, something wrong happens earlier. Looking at your stacks (for all working threads that were shown) I see they all perform simultaneous disconnect. I do not think that all of your clients have simultaneously exited client software:-) I.e. for some reason all network connections of particular process were dropped. Except it I do not see any problems with fbserver at athis moment.

Certainly, exception handling in working thread startup will be fixed.

May be you should try with CS instead of SS?

Alexander Peshkov added a comment - 27/Apr/09 05:43 PM
Changed name after final reasons' analysis

Alexander Peshkov added a comment - 27/Apr/09 05:52 PM
All said about 2415 is true here. fbserver fails due to gcc bug, in next point release it will be fixed.

Alexander Peshkov added a comment - 27/Apr/09 05:54 PM
Segfault can be avoided with older versions if you build them using fresh (> 3.2.X) gcc. Additionally fixed client hang in case when fbserver can't start even first worker thread.