Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

AccessViolationException inside FB_965910463_Class.isc_start_multiple on begin transaction. [CORE3207] #3581

Closed
firebird-automations opened this issue Oct 31, 2010 · 30 comments

Comments

@firebird-automations
Copy link
Collaborator

Submitted by: Mark Kharitonov (mark.kharitonov)

Attachments:
20101108_203654_Crash_Mode.7z

We use an embedded firebird database through the .NET data provider. We get periodically access violations when beginning a transaction (this may be called in parallel by several threads).

We use the FirebirdSql.Data.FirebirdClient .NET data provider, compiled by us from scratch on .NET 4.

The stack trace follows:
Unhandled Exception: System.AccessViolationException: Attempted to read or write protected memory. This is often an indication that other me
mory is corrupt.
at FB_965910463_Class.isc_start_multiple(IntPtr[] statusVector, Int32& trHandle, Int16 dbHandleCount, IntPtr tebVectorAddress)
at FB_965910463_Class.IFbClient.isc_start_multiple(IntPtr[] statusVector, Int32& trHandle, Int16 dbHandleCount, IntPtr tebVectorAddress)
at FirebirdSql.Data.Client.Native.FesTransaction.BeginTransaction(TransactionParameterBuffer tpb) in c:\dev\3rd_party\Firebird\Firebird.N
ET\Client\Native\FesTransaction.cs:line 180
at FirebirdSql.Data.Client.Native.FesDatabase.BeginTransaction(TransactionParameterBuffer tpb) in c:\dev\3rd_party\Firebird\http://Firebird.NET\
Client\Native\FesDatabase.cs:line 333
at FirebirdSql.Data.FirebirdClient.FbTransaction.BeginTransaction() in c:\dev\3rd_party\Firebird\http://Firebird.NET\FirebirdClient\FbTransactio
n.cs:line 363
at FirebirdSql.Data.FirebirdClient.FbCommand.Prepare(Boolean returnsSet) in c:\dev\3rd_party\Firebird\http://Firebird.NET\FirebirdClient\FbComma
nd.cs:line 1114
at FirebirdSql.Data.FirebirdClient.FbCommand.ExecuteCommand(CommandBehavior behavior, Boolean returnsSet) in c:\dev\3rd_party\Firebird\Fi
http://rebird.NET\FirebirdClient\FbCommand.cs:line 1176
at FirebirdSql.Data.FirebirdClient.FbCommand.ExecuteReader(CommandBehavior behavior) in c:\dev\3rd_party\Firebird\http://Firebird.NET\FirebirdCl
ient\FbCommand.cs:line 538
at FirebirdSql.Data.FirebirdClient.FbCommand.ExecuteDbDataReader(CommandBehavior behavior) in c:\dev\3rd_party\Firebird\http://Firebird.NET\Fire
birdClient\FbCommand.cs:line 620
at System.Data.Common.DbCommand.System.Data.IDbCommand.ExecuteReader()
at NHibernate.AdoNet.AbstractBatcher.ExecuteReader(IDbCommand cmd) in c:\Dev\3rd_party\NHibernate\NHibernate\AdoNet\AbstractBatcher.cs:li
ne 235
at NHibernate.Loader.Loader.GetResultSet(IDbCommand st, Boolean autoDiscoverTypes, Boolean callable, RowSelection selection, ISessionImpl
ementor session) in c:\Dev\3rd_party\NHibernate\NHibernate\Loader\Loader.cs:line 1292
at NHibernate.Loader.Loader.DoQuery(ISessionImplementor session, QueryParameters queryParameters, Boolean returnProxies) in c:\Dev\3rd_pa
rty\NHibernate\NHibernate\Loader\Loader.cs:line 401
at NHibernate.Loader.Loader.DoQueryAndInitializeNonLazyCollections(ISessionImplementor session, QueryParameters queryParameters, Boolean
returnProxies) in c:\Dev\3rd_party\NHibernate\NHibernate\Loader\Loader.cs:line 236
at NHibernate.Loader.Loader.DoList(ISessionImplementor session, QueryParameters queryParameters) in c:\Dev\3rd_party\NHibernate\NHibernat
e\Loader\Loader.cs:line 1644
at NHibernate.Loader.Loader.ListIgnoreQueryCache(ISessionImplementor session, QueryParameters queryParameters) in c:\Dev\3rd_party\NHiber
nate\NHibernate\Loader\Loader.cs:line 1568
at NHibernate.Loader.Loader.List(ISessionImplementor session, QueryParameters queryParameters, ISet`1 querySpaces, IType[] resultTypes) i
n c:\Dev\3rd_party\NHibernate\NHibernate\Loader\Loader.cs:line 1562
at NHibernate.Loader.Criteria.CriteriaLoader.List(ISessionImplementor session) in c:\Dev\3rd_party\NHibernate\NHibernate\Loader\Criteria\
CriteriaLoader.cs:line 73
at NHibernate.Impl.SessionImpl.List(CriteriaImpl criteria, IList results) in c:\Dev\3rd_party\NHibernate\NHibernate\Impl\SessionImpl.cs:l
ine 1929
at NHibernate.Impl.CriteriaImpl.List(IList results) in c:\Dev\3rd_party\NHibernate\NHibernate\Impl\CriteriaImpl.cs:line 246
at NHibernate.Linq.NHibernateExtensions.List(ISession session, String entityName, Expression expr, IList list) in c:\dev\3rd_party\NHiber
nate\NHibernate.Linq\NHibernateExtensions.cs:line 69
at Shunra.Infra.DAL.NHibernate.StatementScope.GetByExpression[TType](Expression expr, IList list) in c:\dev\windows\Infra\Shunra.Infra.DA
L.NHibernate\StatementScope.cs:line 232
at Shunra.Infra.DAL.NHibernate.EntityTypeInfoItem.GetByExpression[TType](StatementScope ss, Expression ex, IList list) in c:\dev\windows\
Infra\Shunra.Infra.DAL.NHibernate\DataContext.cs:line 97
at Shunra.Infra.DAL.NHibernate.StatementScope.GetByExpression(Type type, Expression expr, IList list) in c:\dev\windows\Infra\Shunra.Infr
a.DAL.NHibernate\StatementScope.cs:line 285
at NetworkCatcher.Entities.Agent.Server.ExploreEndpointCommandFactory.Execute(ExploreEndpointDataCommand cmd) in c:\dev\windows\NC\NC.Ent
ities.Agent.Server\ExploreEndpointCommandFactory.cs:line 31
at lambda_method(Closure , Object , Object[] )
at Csla.Reflection.MethodCaller.CallMethod(Object obj, DynamicMethodHandle methodHandle, Object[] parameters) in c:\dev\3rd_party\csla\Re
flection\MethodCaller.cs:line 436
at Csla.Reflection.MethodCaller.CallMethod(Object obj, String method, Object[] parameters) in c:\dev\3rd_party\csla\Reflection\MethodCall
er.cs:line 349
at Csla.Server.FactoryDataPortal.InvokeMethod(String factoryTypeName, String methodName, Object e, DataPortalContext context) in c:\dev\3
rd_party\csla\Server\FactoryDataPortal.cs:line 77
at Csla.Server.FactoryDataPortal.Update(Object obj, DataPortalContext context) in c:\dev\3rd_party\csla\Server\FactoryDataPortal.cs:line
164
at Csla.Server.DataPortalSelector.Update(Object obj, DataPortalContext context) in c:\dev\3rd_party\csla\Server\DataPortalSelector.cs:lin
e 110
at Csla.Server.DataPortal.Update(Object obj, DataPortalContext context) in c:\dev\3rd_party\csla\Server\DataPortal.cs:line 287
at Csla.Server.Hosts.WcfPortal.Update(UpdateRequest request) in c:\dev\3rd_party\csla\Server\Hosts\WcfPortal.cs:line 85
at Shunra.Common.Csla.WcfPortal.Update(UpdateRequest request) in C:\dev\windows\Common\Shunra.Common\Csla\WcfPortal.cs:line 41
at SyncInvokeUpdate(Object , Object[] , Object[] )
at System.ServiceModel.Dispatcher.SyncMethodInvoker.Invoke(Object instance, Object[] inputs, Object[]& outputs)
at System.ServiceModel.Dispatcher.DispatchOperationRuntime.InvokeBegin(MessageRpc& rpc)
at System.ServiceModel.Dispatcher.ImmutableDispatchRuntime.ProcessMessage5(MessageRpc& rpc)
at System.ServiceModel.Dispatcher.ImmutableDispatchRuntime.ProcessMessage41(MessageRpc& rpc)
at System.ServiceModel.Dispatcher.ImmutableDispatchRuntime.ProcessMessage4(MessageRpc& rpc)
at System.ServiceModel.Dispatcher.ImmutableDispatchRuntime.ProcessMessage31(MessageRpc& rpc)
at System.ServiceModel.Dispatcher.ImmutableDispatchRuntime.ProcessMessage3(MessageRpc& rpc)
at System.ServiceModel.Dispatcher.ImmutableDispatchRuntime.ProcessMessage2(MessageRpc& rpc)
at System.ServiceModel.Dispatcher.ImmutableDispatchRuntime.ProcessMessage11(MessageRpc& rpc)
at System.ServiceModel.Dispatcher.ImmutableDispatchRuntime.ProcessMessage1(MessageRpc& rpc)
at System.ServiceModel.Dispatcher.MessageRpc.Process(Boolean isOperationContextSet)
at System.ServiceModel.Dispatcher.ChannelHandler.DispatchAndReleasePump(RequestContext request, Boolean cleanThread, OperationContext cur
rentOperationContext)
at System.ServiceModel.Dispatcher.ChannelHandler.HandleRequest(RequestContext request, OperationContext currentOperationContext)
at System.ServiceModel.Dispatcher.ChannelHandler.AsyncMessagePump(IAsyncResult result)
at System.ServiceModel.Dispatcher.ChannelHandler.OnAsyncReceiveComplete(IAsyncResult result)
at System.Runtime.Fx.AsyncThunk.UnhandledExceptionFrame(IAsyncResult result)
at System.Runtime.AsyncResult.Complete(Boolean completedSynchronously)
at System.Runtime.InputQueue`1.AsyncQueueReader.Set(Item item)
at System.Runtime.InputQueue`1.EnqueueAndDispatch(Item item, Boolean canDispatchOnThisThread)
at System.Runtime.InputQueue`1.EnqueueAndDispatch(T item, Action dequeuedCallback, Boolean canDispatchOnThisThread)
at System.ServiceModel.Channels.SingletonChannelAcceptor`3.Enqueue(QueueItemType item, Action dequeuedCallback, Boolean canDispatchOnThis
Thread)
at System.ServiceModel.Channels.HttpChannelListener.HttpContextReceived(HttpRequestContext context, Action callback)
at System.ServiceModel.Channels.SharedHttpTransportManager.OnGetContextCore(IAsyncResult result)
at System.ServiceModel.Channels.SharedHttpTransportManager.OnGetContext(IAsyncResult result)
at System.Runtime.Fx.AsyncThunk.UnhandledExceptionFrame(IAsyncResult result)
at System.Net.LazyAsyncResult.Complete(IntPtr userToken)
at System.Net.LazyAsyncResult.ProtectedInvokeCallback(Object result, IntPtr userToken)
at System.Net.ListenerAsyncResult.WaitCallback(UInt32 errorCode, UInt32 numBytes, NativeOverlapped* nativeOverlapped)
at System.Threading._IOCompletionCallback.PerformIOCompletionCallback(UInt32 errorCode, UInt32 numBytes, NativeOverlapped* pOVERLAP)

Commits: a170738 db75644

@firebird-automations
Copy link
Collaborator Author

Commented by: @dyemanov

A test case would be appreciated.

@firebird-automations
Copy link
Collaborator Author

Modified by: @dyemanov

priority: Critical [ 2 ] => Major [ 3 ]

security: Developers [ 10012 ] =>

@firebird-automations
Copy link
Collaborator Author

Commented by: Mark Kharitonov (mark.kharitonov)

It is kinda hard to produce a test case. I would gladly, but that requires the time I do not have. Do you have a way to trace fbembed.dll? Or may be there is a special diagnostics build? I could then use it and send you the logs/core dump/whatever.

@firebird-automations
Copy link
Collaborator Author

Commented by: @dyemanov

Please download and install the PDB (debug symbols) package matching (exactly!) the FB version you use. Then follow the instructions here:
http://www.ibphoenix.com/main.nfs?a=ibphoenix&page=ibp_pdb_win32

@firebird-automations
Copy link
Collaborator Author

Commented by: Mark Kharitonov (mark.kharitonov)

Do you know if the previous version is more stable?

@firebird-automations
Copy link
Collaborator Author

Commented by: @dyemanov

I cannot say anything without knowing the origin of this failure. Prior versions may or may not share the same issue.

@firebird-automations
Copy link
Collaborator Author

Commented by: Mark Kharitonov (mark.kharitonov)

Hi.
We have another issue - CORE3209

This brings me to a question - is embedded database safe with respect to concurrent access, both reads and writes?

@firebird-automations
Copy link
Collaborator Author

Commented by: Mark Kharitonov (mark.kharitonov)

I have an ADPlus mini dump of the crash, but compressed with 7z it takes 67M.
You can download it using this link - ftp://customerFTP:FTPC!123@ftp.shunra.com/customer/download/20101108_175423_Crash_Mode.7z

The dump was created by running the application using this ADPlus settings:
ADPlus -crash -mss C:\Users\markk\AppData\Local\Temp\SymbolCache -MiniOnSecond -NoDumpOnFirst -quiet -o C:\Logs -sc

If you have better configuration - please share and I will make sure you get it. Guys, this is a critical issue for us.

Thanks.

@firebird-automations
Copy link
Collaborator Author

Commented by: Mark Kharitonov (mark.kharitonov)

I have another crash dump, created using the same parameters. Unfortunately, no heap is available and I am not sure how to configure ADPlus to grab it. Anyway, I am unable to upload the crash dump to our ftp server, in the meantime, if you can provide me with an upload link - I will gladly upload it there. We are talking about 73M.

Thanks.

@firebird-automations
Copy link
Collaborator Author

Modified by: @dyemanov

assignee: Dmitry Yemanov [ dimitr ]

@firebird-automations
Copy link
Collaborator Author

Modified by: @dyemanov

status: Open [ 1 ] => In Progress [ 3 ]

@firebird-automations
Copy link
Collaborator Author

Modified by: @dyemanov

status: In Progress [ 3 ] => Open [ 1 ]

@firebird-automations
Copy link
Collaborator Author

Commented by: @dyemanov

I confirm that the crash was possible. The fix has been committed and will be available in the next snapshot build. Follow it here:
http://www.firebirdsql.org/download/snapshot_builds/win/2.5/

Now you should get an exception instead of a crash, but I'm curious what this exception is going to be.

@firebird-automations
Copy link
Collaborator Author

Commented by: Mark Kharitonov (mark.kharitonov)

Hi. Several questions:
1. When is the next snapshot build?
2. Is it stable to use in a production environment?
3. Is it something wrong, that we do?

If you do not need the full dump created by ADPlus when the process terminates, but just the mini dump of the AV is good for you, then I can upload just it - 5M. This time crash occurs when updating the database, rather than reading it.

@firebird-automations
Copy link
Collaborator Author

Commented by: @dyemanov

1. Tomorrow
2. It's what we're going to release as 2.5.1 before this year ends, so the answer is "hopefully yes, although untested"
3. I have no idea yet.

It seems that the exception you should have been getting is bugcheck 266: "next transaction is older than oldest active transaction". This isn't something that should happen in the real life, so something went seriously wrong. Have you validated the database after the crash? Is it healthy?

Mini dump is usually enough, so feel free to provide the new one. I will ask for a full dump, if necessary.

@firebird-automations
Copy link
Collaborator Author

Modified by: Mark Kharitonov (mark.kharitonov)

Attachment: 20101108_203654_Crash_Mode.7z [ 11816 ]

@firebird-automations
Copy link
Collaborator Author

Commented by: Mark Kharitonov (mark.kharitonov)

Hi,
I have attached the mini dump.
"hopefully yes, although untested" sounds not very promising... Would you advise taking a previous release of Firebird? Which one can you recommend? We need only the embedded server ability.

Thanks.

@firebird-automations
Copy link
Collaborator Author

Commented by: Mark Kharitonov (mark.kharitonov)

About the database corruption.

Can you explain how to avoid it? Because, I have assumed that if we access it only through FireBird embedded database engine, then it will never be corrupt, barring things we do not control, like failing disk sectors, electricity, etc...

@firebird-automations
Copy link
Collaborator Author

Commented by: @dyemanov

The second mini dump shows the same problem.

Many people use snapshot builds in production. It's the logical choice if you cannot wait for the upcoming maintenance release to appear officially. Alternatively, you may download the original 2.5.0 source code, apply this patch yourself and build the "stable" version for your needs. Or pay for that to be done by someone else.

It's worth checking whether the problem persists when you connect to the standalone FB server instead of the embedded one. Also, you could test this issue against the prior version (2.1.3). Regardless, I'm still interested in the database validation results.

@firebird-automations
Copy link
Collaborator Author

Commented by: @dyemanov

Regarding the corruption. There are many things you cannot control and they do affect the database validity. A faulty RAM chip is a common source of corruptions, for example. Also, in the case of embedded, it may be your application that is guilty, as the Firebird caches reside in the same address space and are read/write accessible (e.g. accidentally) to your process.

@firebird-automations
Copy link
Collaborator Author

Commented by: Mark Kharitonov (mark.kharitonov)

OK. I will try the next build.
Thanks.

@firebird-automations
Copy link
Collaborator Author

Commented by: Mark Kharitonov (mark.kharitonov)

On DB corruption. The client is a managed code, we access FB database only through the FB http://ADO.NET layer.
So it should not be able to reach the FB caches, shouldn't it?

It is a bit difficult to test against standalone FB server...

@firebird-automations
Copy link
Collaborator Author

Commented by: Mark Kharitonov (mark.kharitonov)

So, I took the latest build.

How do I know that there is a problem? The process is run from CDB (automatically through Image Execution Flags), so I can monitor the exceptions. What kind of an exception do I have to look for?

@firebird-automations
Copy link
Collaborator Author

Commented by: @dyemanov

Does it keep crashing or not? As for exceptions, it's up to you to decide what to look for. And once again, have you validated the database using gfix?

Regardless, it's going to turn into the support topic and the tracker is not a good place for that. Choose between firebird-support at YahooGroups (Firebird related) and firebird-net-provider at SourceForge (.NET related).

@firebird-automations
Copy link
Collaborator Author

Commented by: Mark Kharitonov (mark.kharitonov)

OK, I will do so.

As for the crash - no it did not crash since then.

Thanks. BTW, if does crash - I return here, OK?

@firebird-automations
Copy link
Collaborator Author

Commented by: @dyemanov

Sure. I'm closing this ticker as resolved now, but it can be re-opened as soon as you get back with a report that the issue still persists.

@firebird-automations
Copy link
Collaborator Author

Modified by: @dyemanov

status: Open [ 1 ] => Resolved [ 5 ]

resolution: Fixed [ 1 ]

Fix Version: 2.5.1 [ 10333 ]

Fix Version: 3.0 Alpha 1 [ 10331 ]

@firebird-automations
Copy link
Collaborator Author

Modified by: @pcisar

status: Resolved [ 5 ] => Closed [ 6 ]

@firebird-automations
Copy link
Collaborator Author

Modified by: @pavel-zotov

QA Status: No test

@firebird-automations
Copy link
Collaborator Author

Modified by: @pavel-zotov

status: Closed [ 6 ] => Closed [ 6 ]

QA Status: No test => Cannot be tested

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

2 participants