Issue Details (XML | Word | Printable)

Key: CORE-3002
Type: Bug Bug
Status: Open Open
Priority: Major Major
Assignee: Unassigned
Reporter: Karel Rys
Votes: 0
Watchers: 2
Operations

If you were logged in you would be able to see more operations.
Firebird Core

Firebird 2.1.3 64bit superserver - hanging TCP conenctions

Created: 12/May/10 01:17 PM   Updated: 28/Feb/11 03:06 PM
Component/s: Engine
Affects Version/s: 2.1.3
Fix Version/s: None

Environment: Windows 2008 server, 64 bit


 Description  « Hide
At the server side, there are left unclosed attachments.

These conenctions are closed at OS level (netstat -a -n does not list them).
Firebird still lists them in MON$ATTACHMENTS.

In Firebird.conf, I have set up:
  ConnectionTimeout = 300
  DummyPacketInterval = 120
  
Unfortunatelly, this does not help and connections are open, until Firebird is restarted.


 All   Comments   Change History   Subversion Commits      Sort Order: Ascending order - Click to sort in descending order
Dmitry Yemanov added a comment - 12/May/10 03:31 PM
SuperServer or Classic?

Karel Rys added a comment - 12/May/10 03:41 PM
Superserver - sorry, I put this info in subject only.

Dmitry Yemanov added a comment - 12/May/10 03:49 PM
How long are those connections left open? DummyPacketInterval = 120 means two minutes between pings. Also, have you tried to reduce the default TCP keep-alive interval on Windows? Did it help?

Karel Rys added a comment - 12/May/10 04:26 PM
On windows, these connections are not active in the system. Netstat -a -n does not show them at all.

Firebird was restarted 50 hours ago.
Oldest connections I have seen are 43 hours old (yes, almost two days!).
Customer did not allow me to change default TCP keep-alive interval at their server, but as far as I know, this is about 2 hours long, so hanging conenctions are much older...

It seems that Firebird does not try to ping these conenctions, or it does try, but does not remove them, even thought connections are already dead.

Dmitry Yemanov added a comment - 12/May/10 04:44 PM
Does the connection count returned via the server properties (server tray icon or Services / Server Properties in IBExpert) match the one returned with select count(*) from mon$attachments?

Karel Rys added a comment - 12/May/10 04:52 PM
Yes, at this moment, both show 5.

Server Version Info
---------------------------------------------------------------------------
Server Version: WI-V2.1.3.18185 Firebird 2.1
Server Implementation: Firebird/x86-64/Windows NT
Service Version: 2


Configuration Info
---------------------------------------------------------------------------
Base File: C:\Program Files\Firebird\Firebird_2_1\
Lock File: C:\Program Files\Firebird\Firebird_2_1\
Message File: C:\Program Files\Firebird\Firebird_2_1\
Security Database: C:\Program Files\Firebird\Firebird_2_1\security2.fdb



Database Info
---------------------------------------------------------------------------
Number of connections: 5
Number of databases: 1
Databases: D:\FB\CZECHPLANT.FDB

Karel Rys added a comment - 12/May/10 04:56 PM
netstat -a -n

  TCP 192.168.1.2:3050 192.168.1.2:33824 NAVAZANO

(NAVAZANO = connected or something like that)

There are no other connection to TCP 192.168.1.2:3050



Dmitry Yemanov added a comment - 12/May/10 05:41 PM
Does MON$ATTACHMENTS list all of them with value "TCPv4" in the MON$REMOTE_PROTOCOL field?

Karel Rys added a comment - 12/May/10 08:27 PM
Yes, it does (there are more connections to the server at this moment, than before, when I wrote, that there where 5 connections):

MON$ATTACHMENT_ID MON$SERVER_PID MON$STATE MON$ATTACHMENT_NAME MON$USER MON$ROLE MON$REMOTE_PROTOCOL MON$REMOTE_ADDRESS MON$REMOTE_PID MON$CHARACTER_SET_ID MON$TIMESTAMP MON$GARBAGE_COLLECTION MON$REMOTE_PROCESS MON$STAT_ID
2570 19864 1 D:\FB\CZECHPLANT.FDB SYSDBA NONE TCPv4 192.168.1.3 828 51 12.5.2010 21:08:55 1 Q:\plantis\Bin\Plantis.exe 2
2568 19864 1 D:\FB\CZECHPLANT.FDB SYSDBA NONE TCPv4 192.168.1.119 1012 51 12.5.2010 20:17:53 1 Q:\plantis\Bin\Plantis.exe 1153
2566 19864 1 D:\FB\CZECHPLANT.FDB SYSDBA NONE TCPv4 192.168.1.2 21696 51 12.5.2010 18:50:10 1 C:\Program Files (x86)\HK-Software\IBExpert\ibexpert.exe 1184
2485 19864 1 D:\FB\CZECHPLANT.FDB SYSDBA NONE TCPv4 192.168.1.121 2580 51 11.5.2010 9:32:52 1 Q:\plantis\Bin\Plantis.exe 1202
2483 19864 1 D:\FB\CZECHPLANT.FDB SYSDBA NONE TCPv4 192.168.1.3 1860 51 11.5.2010 8:28:21 1 Q:\plantis\Bin\Plantis.exe 1228
2465 19864 0 D:\FB\CZECHPLANT.FDB SYSDBA NONE TCPv4 192.168.1.3 2792 51 10.5.2010 16:36:09 1 Q:\plantis\Bin\Plantis.exe 1292
2446 19864 1 D:\FB\CZECHPLANT.FDB SYSDBA NONE TCPv4 192.168.1.120 3436 51 10.5.2010 14:39:48 1 Q:\plantis\Bin\Plantis.exe 1315

Karel Rys added a comment - 12/May/10 08:31 PM
About Firebirds ability to detect dead connections: I think Firebird sends a ping packet and waits for the answer. May be when there is already error during sending this ping packet, this situation is not handled correctly and server never destroys such a connection?
Most of the problems happen at 192.168.1.3, customer told me now, that this IP address belongs to a terminal server connection. Anyhow, when the TCP connection does not exist at operating system level, it should be destroyed by Firebird too...

Vlad Khorsun added a comment - 12/May/10 08:49 PM
> Firebird sends a ping packet and waits for the answer.

No, it not waits for the answer on dummy packet

> May be when there is already error during sending this ping packet, this situation is not handled correctly and server never destroys such a connection?

All socket errors, returned by the OS :
a) logged into firebird.log
b) force Firebird to close such socket immediately

> Most of the problems happen at 192.168.1.3, customer told me now, that this IP address belongs to a terminal server connection.

Do you have any network software on your server, such as firewall, antivirus, etc ?

> Anyhow, when the TCP connection does not exist at operating system level, it should be destroyed by Firebird too.

Of course. But if OS didn't notified Firebird about broken connection - how it could know about it ?

BTW, how client application owned of "hung" connections was closed - correctly or not ?
And what fbclient version they used ?

Karel Rys added a comment - 12/May/10 09:07 PM
Customer has ESET Mail Security 4 For Microsoft Exchange Server installed on the server, and they probably have windows firewall enabled.

Unfortunatelly, I do not know, how they close the application - I can imagine that sometimes their connection is just broken because of network problems...

FbClient.dll renamed to gds32.dll:
Version 2.1.3.18185 from 16.11.2009.11:56

I think Firebird should recognize a problem, when it tries to send ping packet over a non-existing connection (non-existing at the operating system level). Is it possible to enable debugging messages? I would like to see, what is going on with these connections - logging of each ping packet and a response would help a lot...

In firebird.log, there are messages like this:

SERVER (Server) Mon May 10 10:31:05 2010
INET/inet_error: read errno = 10054


SERVER (Server) Mon May 10 10:48:17 2010
INET/inet_error: read errno = 10054



Evelyne Girard added a comment - 28/Feb/11 03:06 PM
I had the same kind of problem with Firebird 2.1 (17910) 32-bits SuperServer. The server considered some old connections as active even after the client application was closed. It kept those attachments actives forever (several weeks). It didn't matter if I restarted the client computer, the server still considered the attachment active. I have upgraded the server to 2.5 (changed nothing else except the fbclient.dll used by clients of course) and the problem is now fixed.