Issue Details (XML | Word | Printable)

Key: CORE-3207
Type: Bug Bug
Status: Closed Closed
Resolution: Fixed
Priority: Major Major
Assignee: Dmitry Yemanov
Reporter: Mark Kharitonov
Votes: 0
Watchers: 2
Operations

If you were logged in you would be able to see more operations.
Firebird Core

AccessViolationException inside FB_965910463_Class.isc_start_multiple on begin transaction.

Created: 31/Oct/10 02:49 PM   Updated: 04/Feb/11 01:14 PM
Component/s: None
Affects Version/s: 2.5.0
Fix Version/s: 2.5.1, 3.0 Alpha 1

Time Tracking:
Not Specified

File Attachments: 1. File 20101108_203654_Crash_Mode.7z (5.11 MB)

Environment: Windows 7, .NET 4

Planning Status: Unspecified


 Description  « Hide
We use an embedded firebird database through the .NET data provider. We get periodically access violations when beginning a transaction (this may be called in parallel by several threads).

We use the FirebirdSql.Data.FirebirdClient .NET data provider, compiled by us from scratch on .NET 4.

The stack trace follows:
Unhandled Exception: System.AccessViolationException: Attempted to read or write protected memory. This is often an indication that other me
mory is corrupt.
   at FB_965910463_Class.isc_start_multiple(IntPtr[] statusVector, Int32& trHandle, Int16 dbHandleCount, IntPtr tebVectorAddress)
   at FB_965910463_Class.IFbClient.isc_start_multiple(IntPtr[] statusVector, Int32& trHandle, Int16 dbHandleCount, IntPtr tebVectorAddress)
   at FirebirdSql.Data.Client.Native.FesTransaction.BeginTransaction(TransactionParameterBuffer tpb) in c:\dev\3rd_party\Firebird\Firebird.N
ET\Client\Native\FesTransaction.cs:line 180
   at FirebirdSql.Data.Client.Native.FesDatabase.BeginTransaction(TransactionParameterBuffer tpb) in c:\dev\3rd_party\Firebird\Firebird.NET\
Client\Native\FesDatabase.cs:line 333
   at FirebirdSql.Data.FirebirdClient.FbTransaction.BeginTransaction() in c:\dev\3rd_party\Firebird\Firebird.NET\FirebirdClient\FbTransactio
n.cs:line 363
   at FirebirdSql.Data.FirebirdClient.FbCommand.Prepare(Boolean returnsSet) in c:\dev\3rd_party\Firebird\Firebird.NET\FirebirdClient\FbComma
nd.cs:line 1114
   at FirebirdSql.Data.FirebirdClient.FbCommand.ExecuteCommand(CommandBehavior behavior, Boolean returnsSet) in c:\dev\3rd_party\Firebird\Fi
rebird.NET\FirebirdClient\FbCommand.cs:line 1176
   at FirebirdSql.Data.FirebirdClient.FbCommand.ExecuteReader(CommandBehavior behavior) in c:\dev\3rd_party\Firebird\Firebird.NET\FirebirdCl
ient\FbCommand.cs:line 538
   at FirebirdSql.Data.FirebirdClient.FbCommand.ExecuteDbDataReader(CommandBehavior behavior) in c:\dev\3rd_party\Firebird\Firebird.NET\Fire
birdClient\FbCommand.cs:line 620
   at System.Data.Common.DbCommand.System.Data.IDbCommand.ExecuteReader()
   at NHibernate.AdoNet.AbstractBatcher.ExecuteReader(IDbCommand cmd) in c:\Dev\3rd_party\NHibernate\NHibernate\AdoNet\AbstractBatcher.cs:li
ne 235
   at NHibernate.Loader.Loader.GetResultSet(IDbCommand st, Boolean autoDiscoverTypes, Boolean callable, RowSelection selection, ISessionImpl
ementor session) in c:\Dev\3rd_party\NHibernate\NHibernate\Loader\Loader.cs:line 1292
   at NHibernate.Loader.Loader.DoQuery(ISessionImplementor session, QueryParameters queryParameters, Boolean returnProxies) in c:\Dev\3rd_pa
rty\NHibernate\NHibernate\Loader\Loader.cs:line 401
   at NHibernate.Loader.Loader.DoQueryAndInitializeNonLazyCollections(ISessionImplementor session, QueryParameters queryParameters, Boolean
returnProxies) in c:\Dev\3rd_party\NHibernate\NHibernate\Loader\Loader.cs:line 236
   at NHibernate.Loader.Loader.DoList(ISessionImplementor session, QueryParameters queryParameters) in c:\Dev\3rd_party\NHibernate\NHibernat
e\Loader\Loader.cs:line 1644
   at NHibernate.Loader.Loader.ListIgnoreQueryCache(ISessionImplementor session, QueryParameters queryParameters) in c:\Dev\3rd_party\NHiber
nate\NHibernate\Loader\Loader.cs:line 1568
   at NHibernate.Loader.Loader.List(ISessionImplementor session, QueryParameters queryParameters, ISet`1 querySpaces, IType[] resultTypes) i
n c:\Dev\3rd_party\NHibernate\NHibernate\Loader\Loader.cs:line 1562
   at NHibernate.Loader.Criteria.CriteriaLoader.List(ISessionImplementor session) in c:\Dev\3rd_party\NHibernate\NHibernate\Loader\Criteria\
CriteriaLoader.cs:line 73
   at NHibernate.Impl.SessionImpl.List(CriteriaImpl criteria, IList results) in c:\Dev\3rd_party\NHibernate\NHibernate\Impl\SessionImpl.cs:l
ine 1929
   at NHibernate.Impl.CriteriaImpl.List(IList results) in c:\Dev\3rd_party\NHibernate\NHibernate\Impl\CriteriaImpl.cs:line 246
   at NHibernate.Linq.NHibernateExtensions.List(ISession session, String entityName, Expression expr, IList list) in c:\dev\3rd_party\NHiber
nate\NHibernate.Linq\NHibernateExtensions.cs:line 69
   at Shunra.Infra.DAL.NHibernate.StatementScope.GetByExpression[TType](Expression expr, IList list) in c:\dev\windows\Infra\Shunra.Infra.DA
L.NHibernate\StatementScope.cs:line 232
   at Shunra.Infra.DAL.NHibernate.EntityTypeInfoItem.GetByExpression[TType](StatementScope ss, Expression ex, IList list) in c:\dev\windows\
Infra\Shunra.Infra.DAL.NHibernate\DataContext.cs:line 97
   at Shunra.Infra.DAL.NHibernate.StatementScope.GetByExpression(Type type, Expression expr, IList list) in c:\dev\windows\Infra\Shunra.Infr
a.DAL.NHibernate\StatementScope.cs:line 285
   at NetworkCatcher.Entities.Agent.Server.ExploreEndpointCommandFactory.Execute(ExploreEndpointDataCommand cmd) in c:\dev\windows\NC\NC.Ent
ities.Agent.Server\ExploreEndpointCommandFactory.cs:line 31
   at lambda_method(Closure , Object , Object[] )
   at Csla.Reflection.MethodCaller.CallMethod(Object obj, DynamicMethodHandle methodHandle, Object[] parameters) in c:\dev\3rd_party\csla\Re
flection\MethodCaller.cs:line 436
   at Csla.Reflection.MethodCaller.CallMethod(Object obj, String method, Object[] parameters) in c:\dev\3rd_party\csla\Reflection\MethodCall
er.cs:line 349
   at Csla.Server.FactoryDataPortal.InvokeMethod(String factoryTypeName, String methodName, Object e, DataPortalContext context) in c:\dev\3
rd_party\csla\Server\FactoryDataPortal.cs:line 77
   at Csla.Server.FactoryDataPortal.Update(Object obj, DataPortalContext context) in c:\dev\3rd_party\csla\Server\FactoryDataPortal.cs:line
164
   at Csla.Server.DataPortalSelector.Update(Object obj, DataPortalContext context) in c:\dev\3rd_party\csla\Server\DataPortalSelector.cs:lin
e 110
   at Csla.Server.DataPortal.Update(Object obj, DataPortalContext context) in c:\dev\3rd_party\csla\Server\DataPortal.cs:line 287
   at Csla.Server.Hosts.WcfPortal.Update(UpdateRequest request) in c:\dev\3rd_party\csla\Server\Hosts\WcfPortal.cs:line 85
   at Shunra.Common.Csla.WcfPortal.Update(UpdateRequest request) in C:\dev\windows\Common\Shunra.Common\Csla\WcfPortal.cs:line 41
   at SyncInvokeUpdate(Object , Object[] , Object[] )
   at System.ServiceModel.Dispatcher.SyncMethodInvoker.Invoke(Object instance, Object[] inputs, Object[]& outputs)
   at System.ServiceModel.Dispatcher.DispatchOperationRuntime.InvokeBegin(MessageRpc& rpc)
   at System.ServiceModel.Dispatcher.ImmutableDispatchRuntime.ProcessMessage5(MessageRpc& rpc)
   at System.ServiceModel.Dispatcher.ImmutableDispatchRuntime.ProcessMessage41(MessageRpc& rpc)
   at System.ServiceModel.Dispatcher.ImmutableDispatchRuntime.ProcessMessage4(MessageRpc& rpc)
   at System.ServiceModel.Dispatcher.ImmutableDispatchRuntime.ProcessMessage31(MessageRpc& rpc)
   at System.ServiceModel.Dispatcher.ImmutableDispatchRuntime.ProcessMessage3(MessageRpc& rpc)
   at System.ServiceModel.Dispatcher.ImmutableDispatchRuntime.ProcessMessage2(MessageRpc& rpc)
   at System.ServiceModel.Dispatcher.ImmutableDispatchRuntime.ProcessMessage11(MessageRpc& rpc)
   at System.ServiceModel.Dispatcher.ImmutableDispatchRuntime.ProcessMessage1(MessageRpc& rpc)
   at System.ServiceModel.Dispatcher.MessageRpc.Process(Boolean isOperationContextSet)
   at System.ServiceModel.Dispatcher.ChannelHandler.DispatchAndReleasePump(RequestContext request, Boolean cleanThread, OperationContext cur
rentOperationContext)
   at System.ServiceModel.Dispatcher.ChannelHandler.HandleRequest(RequestContext request, OperationContext currentOperationContext)
   at System.ServiceModel.Dispatcher.ChannelHandler.AsyncMessagePump(IAsyncResult result)
   at System.ServiceModel.Dispatcher.ChannelHandler.OnAsyncReceiveComplete(IAsyncResult result)
   at System.Runtime.Fx.AsyncThunk.UnhandledExceptionFrame(IAsyncResult result)
   at System.Runtime.AsyncResult.Complete(Boolean completedSynchronously)
   at System.Runtime.InputQueue`1.AsyncQueueReader.Set(Item item)
   at System.Runtime.InputQueue`1.EnqueueAndDispatch(Item item, Boolean canDispatchOnThisThread)
   at System.Runtime.InputQueue`1.EnqueueAndDispatch(T item, Action dequeuedCallback, Boolean canDispatchOnThisThread)
   at System.ServiceModel.Channels.SingletonChannelAcceptor`3.Enqueue(QueueItemType item, Action dequeuedCallback, Boolean canDispatchOnThis
Thread)
   at System.ServiceModel.Channels.HttpChannelListener.HttpContextReceived(HttpRequestContext context, Action callback)
   at System.ServiceModel.Channels.SharedHttpTransportManager.OnGetContextCore(IAsyncResult result)
   at System.ServiceModel.Channels.SharedHttpTransportManager.OnGetContext(IAsyncResult result)
   at System.Runtime.Fx.AsyncThunk.UnhandledExceptionFrame(IAsyncResult result)
   at System.Net.LazyAsyncResult.Complete(IntPtr userToken)
   at System.Net.LazyAsyncResult.ProtectedInvokeCallback(Object result, IntPtr userToken)
   at System.Net.ListenerAsyncResult.WaitCallback(UInt32 errorCode, UInt32 numBytes, NativeOverlapped* nativeOverlapped)
   at System.Threading._IOCompletionCallback.PerformIOCompletionCallback(UInt32 errorCode, UInt32 numBytes, NativeOverlapped* pOVERLAP)


 All   Comments   Work Log   Change History   Version Control   Subversion Commits      Sort Order: Ascending order - Click to sort in descending order
Dmitry Yemanov added a comment - 31/Oct/10 07:24 PM
A test case would be appreciated.

Mark Kharitonov added a comment - 31/Oct/10 07:33 PM
It is kinda hard to produce a test case. I would gladly, but that requires the time I do not have. Do you have a way to trace fbembed.dll? Or may be there is a special diagnostics build? I could then use it and send you the logs/core dump/whatever.

Dmitry Yemanov added a comment - 31/Oct/10 07:38 PM
Please download and install the PDB (debug symbols) package matching (exactly!) the FB version you use. Then follow the instructions here:
http://www.ibphoenix.com/main.nfs?a=ibphoenix&page=ibp_pdb_win32

Mark Kharitonov added a comment - 01/Nov/10 10:33 AM
Do you know if the previous version is more stable?

Dmitry Yemanov added a comment - 01/Nov/10 10:45 AM
I cannot say anything without knowing the origin of this failure. Prior versions may or may not share the same issue.

Mark Kharitonov added a comment - 01/Nov/10 10:51 AM
Hi.
We have another issue - http://tracker.firebirdsql.org/browse/CORE-3209

This brings me to a question - is embedded database safe with respect to concurrent access, both reads and writes?

Mark Kharitonov added a comment - 08/Nov/10 04:42 PM - edited
I have an ADPlus mini dump of the crash, but compressed with 7z it takes 67M.
You can download it using this link - ftp://customerFTP:FTPC!123@ftp.shunra.com/customer/download/20101108_175423_Crash_Mode.7z

The dump was created by running the application using this ADPlus settings:
ADPlus -crash -mss C:\Users\markk\AppData\Local\Temp\SymbolCache -MiniOnSecond -NoDumpOnFirst -quiet -o C:\Logs -sc

If you have better configuration - please share and I will make sure you get it. Guys, this is a critical issue for us.

Thanks.

Mark Kharitonov added a comment - 08/Nov/10 07:08 PM
I have another crash dump, created using the same parameters. Unfortunately, no heap is available and I am not sure how to configure ADPlus to grab it. Anyway, I am unable to upload the crash dump to our ftp server, in the meantime, if you can provide me with an upload link - I will gladly upload it there. We are talking about 73M.

Thanks.

Dmitry Yemanov added a comment - 08/Nov/10 07:47 PM
I confirm that the crash was possible. The fix has been committed and will be available in the next snapshot build. Follow it here:
http://www.firebirdsql.org/download/snapshot_builds/win/2.5/

Now you should get an exception instead of a crash, but I'm curious what this exception is going to be.

Mark Kharitonov added a comment - 08/Nov/10 07:56 PM
Hi. Several questions:
1. When is the next snapshot build?
2. Is it stable to use in a production environment?
3. Is it something wrong, that we do?

If you do not need the full dump created by ADPlus when the process terminates, but just the mini dump of the AV is good for you, then I can upload just it - 5M. This time crash occurs when updating the database, rather than reading it.

Dmitry Yemanov added a comment - 08/Nov/10 08:18 PM
1. Tomorrow
2. It's what we're going to release as 2.5.1 before this year ends, so the answer is "hopefully yes, although untested"
3. I have no idea yet.

It seems that the exception you should have been getting is bugcheck 266: "next transaction is older than oldest active transaction". This isn't something that should happen in the real life, so something went seriously wrong. Have you validated the database after the crash? Is it healthy?

Mini dump is usually enough, so feel free to provide the new one. I will ask for a full dump, if necessary.

Mark Kharitonov added a comment - 08/Nov/10 08:31 PM
Hi,
I have attached the mini dump.
"hopefully yes, although untested" sounds not very promising... Would you advise taking a previous release of Firebird? Which one can you recommend? We need only the embedded server ability.

Thanks.

Mark Kharitonov added a comment - 08/Nov/10 08:35 PM
About the database corruption.

Can you explain how to avoid it? Because, I have assumed that if we access it only through FireBird embedded database engine, then it will never be corrupt, barring things we do not control, like failing disk sectors, electricity, etc...

Dmitry Yemanov added a comment - 08/Nov/10 08:41 PM
The second mini dump shows the same problem.

Many people use snapshot builds in production. It's the logical choice if you cannot wait for the upcoming maintenance release to appear officially. Alternatively, you may download the original 2.5.0 source code, apply this patch yourself and build the "stable" version for your needs. Or pay for that to be done by someone else.

It's worth checking whether the problem persists when you connect to the standalone FB server instead of the embedded one. Also, you could test this issue against the prior version (2.1.3). Regardless, I'm still interested in the database validation results.

Dmitry Yemanov added a comment - 08/Nov/10 08:46 PM
Regarding the corruption. There are many things you cannot control and they do affect the database validity. A faulty RAM chip is a common source of corruptions, for example. Also, in the case of embedded, it may be your application that is guilty, as the Firebird caches reside in the same address space and are read/write accessible (e.g. accidentally) to your process.

Mark Kharitonov added a comment - 08/Nov/10 08:48 PM
OK. I will try the next build.
Thanks.

Mark Kharitonov added a comment - 08/Nov/10 08:54 PM
On DB corruption. The client is a managed code, we access FB database only through the FB ADO.NET layer.
So it should not be able to reach the FB caches, shouldn't it?

It is a bit difficult to test against standalone FB server...

Mark Kharitonov added a comment - 10/Nov/10 10:22 AM
So, I took the latest build.

How do I know that there is a problem? The process is run from CDB (automatically through Image Execution Flags), so I can monitor the exceptions. What kind of an exception do I have to look for?

Dmitry Yemanov added a comment - 11/Nov/10 07:17 AM
Does it keep crashing or not? As for exceptions, it's up to you to decide what to look for. And once again, have you validated the database using gfix?

Regardless, it's going to turn into the support topic and the tracker is not a good place for that. Choose between firebird-support at YahooGroups (Firebird related) and firebird-net-provider at SourceForge (.NET related).

Mark Kharitonov added a comment - 11/Nov/10 07:30 AM
OK, I will do so.

As for the crash - no it did not crash since then.

Thanks. BTW, if does crash - I return here, OK?

Dmitry Yemanov added a comment - 11/Nov/10 07:42 AM
Sure. I'm closing this ticker as resolved now, but it can be re-opened as soon as you get back with a report that the issue still persists.