Issue Details (XML | Word | Printable)

Key: CORE-3811
Type: Task Task
Status: Open Open
Priority: Major Major
Assignee: Unassigned
Reporter: Pavel Zotov
Votes: 0
Watchers: 3
Operations

If you were logged in you would be able to see more operations.
Firebird Core

Can not connect to any .fdb after stopped heavy OLTP-test (750 connections)

Created: 09/Apr/12 11:36 AM   Updated: 09/Apr/12 06:05 PM
Component/s: None
Affects Version/s: 2.5.1
Fix Version/s: None

File Attachments: 1. Zip Archive gdb_fb_smp_server.zip (14 kB)



 Description  « Hide
hi all

 ISQL Version: WI-V2.5.1.26351 Firebird 2.5
 Server version:
 Firebird/linux AMD64 (access method), version "LI-V2.5.2.26448 Firebird 2.5"
 Firebird/linux AMD64 (remote server), version "LI-V2.5.2.26448 Firebird 2.5/tcp (fbozzy)/P12"
 Firebird/x86/Windows NT (remote interface), version "WI-V2.5.1.26351 Firebird 2.5/tcp (tlprg)/P12"
 on disk structure version 11.2

Yesterday I run test with 750 connections with small and medium loading of database in each of them.
Today I've stopped this test (first I created some `special` stop-file which serves as signal to each connection script to be canceled - this way kills ~300 session; but after I kill all rest isql`s windows via pskill.exe).

Now the count of threads in fb_smp_server are 4:
bash-3.2$ ps -FLC fb_smp_server
UID PID PPID LWP C NLWP SZ RSS PSR STIME TTY TIME CMD
firebird 5982 5981 5982 0 4 20882 5108 2 13:50 ? 00:00:00 /opt/firebird/bin/fb_smp_server
firebird 5982 5981 5984 0 4 20882 5108 3 13:50 ? 00:00:00 /opt/firebird/bin/fb_smp_server
firebird 5982 5981 5985 0 4 20882 5108 7 13:50 ? 00:00:00 /opt/firebird/bin/fb_smp_server
firebird 5982 5981 5986 0 4 20882 5108 3 13:50 ? 00:00:00 /opt/firebird/bin/fb_smp_server

Unfortunatelly I can not establish new connections to any of databases that are placed on this machine.
The `top` utility for fb_smp_server shows that it is in idle state:

$ top -p $(pgrep fb_smp_server)

top - 15:06:57 up 1 day, 14:37, 7 users, load average: 0.02, 0.01, 0.00
Tasks: 1 total, 0 running, 1 sleeping, 0 stopped, 0 zombie
Cpu0 : 0.0%us, 0.0%sy, 0.0%ni,100.0%id, 0.0%wa, 0.0%hi, 0.0%si, 0.0%st
Cpu1 : 0.0%us, 0.0%sy, 0.0%ni,100.0%id, 0.0%wa, 0.0%hi, 0.0%si, 0.0%st
Cpu2 : 0.0%us, 0.0%sy, 0.0%ni,100.0%id, 0.0%wa, 0.0%hi, 0.0%si, 0.0%st
Cpu3 : 0.0%us, 0.0%sy, 0.0%ni,100.0%id, 0.0%wa, 0.0%hi, 0.0%si, 0.0%st
Cpu4 : 0.0%us, 0.0%sy, 0.0%ni,100.0%id, 0.0%wa, 0.0%hi, 0.0%si, 0.0%st
Cpu5 : 0.0%us, 0.0%sy, 0.0%ni,100.0%id, 0.0%wa, 0.0%hi, 0.0%si, 0.0%st
Cpu6 : 0.0%us, 0.0%sy, 0.0%ni,100.0%id, 0.0%wa, 0.0%hi, 0.0%si, 0.0%st
Cpu7 : 0.0%us, 0.0%sy, 0.0%ni,100.0%id, 0.0%wa, 0.0%hi, 0.0%si, 0.0%st
Mem: 32933152k total, 20914936k used, 12018216k free, 224876k buffers
Swap: 33792716k total, 196k used, 33792520k free, 20130404k cached

  PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
 5982 firebird 18 0 83528 5108 3624 S 0.0 0.0 0:00.41 fb_smp_server

The `iostat` utility also show that there is no any disk activity:

bash-3.2$ iostat -d -m -t 5
Linux 2.6.18-308.el5 (fbozzy) 04/09/2012

Time: 03:08:09 PM
Device: tps MB_read/s MB_wrtn/s MB_read MB_wrtn
sda 195.40 0.66 2.44 91477 339559
sda1 0.00 0.00 0.00 1 0
sda2 187.78 0.01 1.94 1956 270086
sda3 0.00 0.00 0.00 0 0
sda4 0.00 0.00 0.00 0 0
sda5 7.61 0.64 0.50 89518 69472

Time: 03:08:14 PM
Device: tps MB_read/s MB_wrtn/s MB_read MB_wrtn
sda 0.40 0.00 0.01 0 0
sda1 0.00 0.00 0.00 0 0
sda2 0.40 0.00 0.01 0 0
sda3 0.00 0.00 0.00 0 0
sda4 0.00 0.00 0.00 0 0
sda5 0.00 0.00 0.00 0 0

Time: 03:08:19 PM
Device: tps MB_read/s MB_wrtn/s MB_read MB_wrtn
sda 0.00 0.00 0.00 0 0
sda1 0.00 0.00 0.00 0 0
sda2 0.00 0.00 0.00 0 0
sda3 0.00 0.00 0.00 0 0
sda4 0.00 0.00 0.00 0 0
sda5 0.00 0.00 0.00 0 0

Time: 03:08:24 PM
Device: tps MB_read/s MB_wrtn/s MB_read MB_wrtn
sda 1.00 0.00 0.01 0 0
sda1 0.00 0.00 0.00 0 0
sda2 1.00 0.00 0.01 0 0
sda3 0.00 0.00 0.00 0 0
sda4 0.00 0.00 0.00 0 0
sda5 0.00 0.00 0.00 0 0

But any connect to any database (via local or TCP) hangs.
What should be done in such cases ?


 All   Comments   Change History   Subversion Commits      Sort Order: Ascending order - Click to sort in descending order
Pavel Zotov added a comment - 09/Apr/12 05:32 PM
two files in attach:

1) gdb -q -x ... /opt/firebird/bin/fb_smp_server
1) gdb -q -x ... /opt/firebird/bin/.debug/fb_smp_server.debug

Pavel Zotov added a comment - 09/Apr/12 06:05 PM
PS. I see the following output of netstat on this server (may be this will help):

netstat -anep | grep 3050
tcp 0 0 0.0.0.0:3050 0.0.0.0:* LISTEN 54323 382680 5982/fb_smp_server
tcp 157 0 127.0.0.1:3050 127.0.0.1:38899 CLOSE_WAIT 0 0 -
tcp 157 0 127.0.0.1:3050 127.0.0.1:38898 CLOSE_WAIT 0 0