Issue Details (XML | Word | Printable)

Key: CORE-4856
Type: Bug Bug
Status: Resolved Resolved
Resolution: Fixed
Priority: Major Major
Assignee: Unassigned
Reporter: Artur
Votes: 0
Watchers: 6
Operations

If you were logged in you would be able to see more operations.
Firebird Core

2.5.3.26780 - c0000005 (Access violation) The instruction at 0x%08lx referenced memory at 0x%08lx. The memory could not be %s.

Created: 03/Jul/15 07:10 AM   Updated: 05/Oct/15 11:17 AM
Component/s: Engine
Affects Version/s: 2.5.3 Update 1
Fix Version/s: 2.5.4

File Attachments: 1. File 41304Re40f5be.dmp (50 kB)
2. File 6732R11de2b6.dmp (50 kB)

Environment: Windows 2008R2 SP1, Citrix XenApp 6.5 RP5, Citrix EdgeSight Agent 5.4

QA Status: Cannot be tested


 Description  « Hide
I'm using Firebird DB for Citrix EdgeSight Agent. Unfortunately Firebird server is crashing quite often.

Please see example from dump:


Microsoft (R) Windows Debugger Version 6.2.9200.16384 AMD64
Copyright (c) Microsoft Corporation. All rights reserved.


Loading Dump File [C:\!Support\DMp\4720R6eeafc.dmp]
User Mini Dump File: Only registers, stack and portions of memory are available

Symbol search path is: SRV*c:\localsymbols*http://msdl.microsoft.com/download/symbols
Executable search path is:
Windows 7 Version 7601 (Service Pack 1) MP (2 procs) Free x86 compatible
Product: Server, suite: TerminalServer
Machine Name:
Debug session time: Tue Jun 30 03:02:30.000 2015 (UTC + 2:00)
System Uptime: not available
Process Uptime: 0 days 0:00:11.000
.....................................................
This dump file has an exception of interest stored in it.
The stored exception information can be accessed via .ecxr.
(1270.8d8): Access violation - code c0000005 (first/second chance not available)
eax=00000000 ebx=00000000 ecx=00000024 edx=0000000b esi=000003b8 edi=03dde688
eip=77b8f8b1 esp=03dde640 ebp=03dde6ac iopl=0 nv up ei pl zr na pe nc
cs=0023 ss=002b ds=002b es=002b fs=0053 gs=002b efl=00000246
ntdll!NtWaitForSingleObject+0x15:
77b8f8b1 83c404 add esp,4
0:014> !analyze -v
*******************************************************************************
* *
* Exception Analysis *
* *
*******************************************************************************

*** WARNING: Unable to verify timestamp for csma_ldr32.dll
*** ERROR: Module load completed but symbols could not be loaded for csma_ldr32.dll
*** WARNING: Unable to verify timestamp for msvcr90.dll
*** WARNING: Unable to verify timestamp for rsintcor32.dll
*** ERROR: Module load completed but symbols could not be loaded for rsintcor32.dll
*** WARNING: Unable to verify timestamp for esint32.dll
*** ERROR: Module load completed but symbols could not be loaded for esint32.dll

FAULTING_IP:
fbserver+4332c
0044332c 8b582c mov ebx,dword ptr [eax+2Ch]

EXCEPTION_RECORD: ffffffff -- (.exr 0xffffffffffffffff)
ExceptionAddress: 0044332c (fbserver+0x0004332c)
   ExceptionCode: c0000005 (Access violation)
  ExceptionFlags: 00000000
NumberParameters: 2
   Parameter[0]: 00000000
   Parameter[1]: 0000002c
Attempt to read from address 0000002c

PROCESS_NAME: fbserver.exe

ERROR_CODE: (NTSTATUS) 0xc0000005 - The instruction at 0x%08lx referenced memory at 0x%08lx. The memory could not be %s.

EXCEPTION_CODE: (NTSTATUS) 0xc0000005 - The instruction at 0x%08lx referenced memory at 0x%08lx. The memory could not be %s.

EXCEPTION_PARAMETER1: 00000000

EXCEPTION_PARAMETER2: 0000002c

READ_ADDRESS: 0000002c

FOLLOWUP_IP:
fbserver+4332c
0044332c 8b582c mov ebx,dword ptr [eax+2Ch]

APP: fbserver.exe

FAULTING_THREAD: 000008d8

BUGCHECK_STR: APPLICATION_FAULT_NULL_CLASS_PTR_READ_AFTER_CALL

PRIMARY_PROBLEM_CLASS: NULL_CLASS_PTR_READ_AFTER_CALL

DEFAULT_BUCKET_ID: NULL_CLASS_PTR_READ_AFTER_CALL

IP_ON_STACK:
+0
03ddfce8 70d9 jo 03ddfcc3

FRAME_ONE_INVALID: 1

LAST_CONTROL_TRANSFER: from 03ddfce8 to 0044332c

STACK_TEXT:
WARNING: Stack unwind information not available. Following frames may be wrong.
03ddfcdc 03ddfce8 00ffe4a4 00ffd970 00ffd7d4 fbserver+0x4332c
03ddfce0 00ffe4a4 00ffd970 00ffd7d4 00ffd988 0x3ddfce8
03ddfce8 00ffd7d4 00ffd988 00fe0014 61445352 0xffe4a4
03ddfcec 00ffd988 00fe0014 61445352 662e7274 0xffd7d4
03ddfcf0 00fe0014 61445352 662e7274 00006b62 0xffd988
03ddfcf4 61445352 662e7274 00006b62 03ddfd30 0xfe0014
03ddfcf8 662e7274 00006b62 03ddfd30 0040b0b9 0x61445352
03ddfcfc 00000000 03ddfd30 0040b0b9 00fe0068 0x662e7274


SYMBOL_STACK_INDEX: 0

SYMBOL_NAME: fbserver+4332c

FOLLOWUP_NAME: MachineOwner

MODULE_NAME: fbserver

IMAGE_NAME: fbserver.exe

DEBUG_FLR_IMAGE_TIMESTAMP: 54f8e7fe

STACK_COMMAND: ~14s; .ecxr ; kb

FAILURE_BUCKET_ID: NULL_CLASS_PTR_READ_AFTER_CALL_c0000005_fbserver.exe!Unknown

BUCKET_ID: APPLICATION_FAULT_NULL_CLASS_PTR_READ_AFTER_CALL_fbserver+4332c

WATSON_STAGEONE_URL: http://watson.microsoft.com/StageOne/fbserver_exe/2_5_3_26780/54f8e7fe/fbserver_exe/2_5_3_26780/54f8e7fe/c0000005/0004332c.htm?Retriage=1

Followup: MachineOwner
---------

0:014> lmvm fbserver
start end module name
00400000 007ad000 fbserver T (no symbols)
    Loaded symbol image file: fbserver.exe
    Image path: C:\Program Files (x86)\Citrix\System Monitoring\Agent\Core\Firebird\bin\fbserver.exe
    Image name: fbserver.exe
    Timestamp: Fri Mar 06 00:34:22 2015 (54F8E7FE)
    CheckSum: 003B2D95
    ImageSize: 003AD000
    File version: 2.5.3.26780
    Product version: 2.5.3.26780
    File flags: 2 (Mask 2) Pre-release
    File OS: 40004 NT Win32
    File type: 1.0 App
    File date: 00000000.00000000
    Translations: 0000.04b0 0000.04e4 0409.04b0 0409.04e4


 All   Comments   Change History   Subversion Commits      Sort Order: Ascending order - Click to sort in descending order
Dmitry Yemanov added a comment - 03/Jul/15 07:14 AM
Worth testing with more recent v2.5.4.

Artur added a comment - 23/Sep/15 07:55 AM
Issue seems to be resolved in 2.5.4. After upgrade issue is not existing anymore. (so far tested for few days)

Saravana Kumar added a comment - 30/Sep/15 09:42 PM - edited
I am facing multiple crashes with Firebird 2.5.3 update 1 when we try to do database maintenance from our product. I have attached the crash dump files captured when the issue occurs, we havent seen any such problems with prior versions of Firebird 2.5.

This issue started happening after upgrade to Firebird 2.5.3.

Can you please help me troubleshoot and fix this issue?

Appreciate your help on this.

Thanks,
Saravana

Saravana Kumar added a comment - 01/Oct/15 05:37 AM
The original issue was reported by me, now I have provided fbserver.exe dumps.

Vlad Khorsun added a comment - 01/Oct/15 07:59 AM
Looks like minidump was produced for custom-build binaries:

Image name: fbserver.exe
Timestamp: Fri Mar 06 00:34:22 2015 (54F8E7FE)

Note, official 2.5.3-u1 was released at December 2014

Also, i see that this binaries used msvcr90.dll, i.e. it was build with VC9, while our binaries are built with VC8

Therefore it is useless without debug information.

Also, it is a big doubt that we should investigate this case if it is already fixed in 2.5.4 (as reported above))

Artur added a comment - 01/Oct/15 11:39 AM
I would like to confirm: after 2 weeks testing I didn't have any crashes for EdgeSight Agent.

Solution: Replace FB components with: FB DB 2.5.4.

Tested on 5 servers. 0 crashes. Before: random crashes every day.

Saravana: Check SR70322148 in your systems - I already informed escalation engineer about solution more than week ago (after 3 months waiting for vendor support). EOF

Saravana Kumar added a comment - 01/Oct/15 05:22 PM
Thanks for the suggestion to upgrade to 2.5.4 but am trying to find root cause of the issue. I will try with FB 2.5.4

Artur: I got that information about your upgrade but I hope our Engineer has conveyed that we customize Firebird for EdgeSight.

Artur added a comment - 02/Oct/15 05:56 AM
Saravana: Please update FB to 2.5.4 and realize hotfix as "public". So many your (Citrix) customers are affected.
Private hotfix will push customer to use previous version of EdgeSight Agent - as it's not crashing (with Firebird 2.5.3 and Without update 1 - as you know this version contain security BUG).

Thanks.

P.S. Why did you start to work on this issue so late ? Many customers (Citrix customers) are affected since more than 3 months. Shame..