Question

I got the following exception in a MT application, multiple threads are querying and inserting objects. Any Ideas?

var entity = Query.SingleOrDefault(key);

Invalid OperationException The LinkedList node dsen't belong to the current LinkedList  
Data = {System.Collections.ListDictionaryInternal}
TargetSite = {Void Remove(System.Collections.Generic.LinkedListNode`1[T])}

StackTrace:
bei System.Collections.Generic.LinkedList`1.Remove(LinkedListNode`1 node)
bei Xtensive.Core.Collections.TopDeque`2.TryGetValue(K key, Boolean moveToTop, V& value)
bei Xtensive.Core.Caching.LruCache`2.TryGetItem(TKey key, Boolean markAsHit, TItem& item)
bei Xtensive.Storage.Internals.RecordSetReader.Read(IEnumerable`1 source, RecordSetHeader header)
bei Xtensive.Storage.Internals.Prefetch.EntityGroupTask.PutLoadedStatesInCache(IEnumerable`1 queryResult, RecordSetReader reader, HashSet`1 foundedKeys)
bei Xtensive.Storage.Internals.Prefetch.EntityGroupTask.UpdateCache(HashSet`1 foundKeys)
bei Xtensive.Storage.Internals.Prefetch.Fetcher.UpdateCacheFromAllEntityGroupTasks()
bei Xtensive.Storage.Internals.Prefetch.Fetcher.ExecuteTasks(IEnumerable`1 containers, Boolean skipPersist)
bei Xtensive.Storage.Internals.Prefetch.PrefetchManager.ExecuteTasks(Boolean skipPersist)
bei Xtensive.Storage.Providers.SessionHandler.FetchEntityState(Key key)
bei Xtensive.Storage.EntityState.Refresh()
bei Xtensive.Storage.TransactionalStateContainer`1.get_State()
bei Xtensive.Storage.EntityState.get_Tuple()
bei Xtensive.Storage.EntityState.get_IsNotAvailableOrMarkedAsRemoved()
bei Xtensive.Storage.Query.SingleOrDefault(Session session, Key key)
bei Xtensive.Storage.Query.SingleOrDefault(Key key)
bei Seo.Crawler.CrawlerService.ScheduablesCreateTasksThreadFkt() in D:\Work\SeoBaseLib\SeoCrawlerService\CrawlerService\Scheduler\ProcessScheduables.cs:Zeile 94.

Accepted Answer

0

It looks like there is a bug related to multithreaded usage of LruCache in RecordSetReader. We'll investigate this ASAP.

Issue 839 is created.

answered Oct 11 '10 at 23:21

Alex Yakunin
2971●4●4●12

edited Oct 21 '10 at 09:11

Added an issue for this.

(Oct 21 '10 at 09:11) Alex Yakunin

Ups, sorry - commented wrong issue.

(Oct 27 '10 at 09:15) Alex Yakunin

Good news:

1) The bug is found. The developer that wrote Prefetch code has made a mistake by improperly using non thread-safe cache:

if (!cache.TryGetItem(header, true, out cacheItem))
lock (_lock) {
  if (!cache.TryGetItem(header, false, out cacheItem)) {

(Oct 28 '10 at 18:40) Alex Yakunin

As you see, it looks very much like double-checked locking pattern, but actually, it's buggy: the author assumes cache supports single writer + multiple readers concurrent access semantics, but actually, it doesn't (the same is true for almost any other non-concurrent collection in .NET except Hashtable).

So this code inevitably leads to an error.

(Oct 28 '10 at 18:40) Alex Yakunin

We'll fix the bug today. Actually, it's completely strange there is such a mistake, since our developers are aware of this. So we'll consult with the author to ensure there are no any other similar cache usages patterns.

(Oct 28 '10 at 18:43) Alex Yakunin

2) Memory storage currently does not support concurrent access, so the same test isn't valid for it.

(Oct 28 '10 at 18:44) Alex Yakunin

3) There was a mistake in your catch block - to properly filter all the "acceptable" exceptions, you should use this code:

catch (Exception error) {
  var currentError = error;
  bool notImportantException = false;
  while (currentError!=null) {
    if (currentError is ThreadAbortException)
      return;

(Oct 28 '10 at 18:46) Alex Yakunin

continuing:

    notImportantException =
      currentError is ReprocessableException ||
      currentError is OperationTimeoutException ||
      currentError is UniqueConstraintViolationException ||
      (currentError is StorageException && currentError.Message.Contains("This SqlTransaction has completed"));
    if (notImportantException)
      break;
    currentError = currentError.InnerException;
  }
  if (notImportantException)
    continue;

  // Error is the one we're looking for, let's log it!

(Oct 28 '10 at 18:47) Alex Yakunin

The issue is fixed in stable branch.

(Oct 29 '10 at 03:23) Alex Yakunin

I'm a bit stupid now, but.. A few minutes ago I got a strange exception, it appears like the cache in DO 4 somehow got corrupted (I made several requests at the same time) and got some exception about that a LinkedList node didn't belong to the LinkedList or similar. Of course not I didn't save the stack trace.. :/ I think I remember that the top of the stack trace hade something to do with caching. A restart of the appool resolved the issue.

(Oct 30 '10 at 16:58) Onkelborg

Further, I've implemented a simple versioning system, and to avoid manually copying all properties of an object I've enumerated the fields, and then used SetProperty<object> and GetProproperty<object> for everything except EntitySets (where I get the enumerator and loops through the whole entityset.) What's most likely the cause of the problem? Bad copying, or bad cache? (Or something totally different? :/ )

(Oct 30 '10 at 17:02) Onkelborg

Most likely, you've got the same exception. It appears on prefetches, but currently DO uses prefetch API to do nearly anything except queries (i.e. regular fetches, EntitySet content fetches, lazy loading and so on). So it's acctually pretty easy to get this error.

The update fixing the issue will be available today. We cheched all similar scenarios - that was the only one. The fix was made with just 1 line change, but the bug is really important.

(Oct 31 '10 at 07:06) Alex Yakunin

Answer 2

Here are the execptions (Debug Diagnostic assemblies) as they occur. It starts with a NullReferenceException and continues with a InvalidOperationException.

Exception: UncheckedProxy, (12601078)-System.NullReferenceException: Der Objektverweis wurde nicht auf eine Objektinstanz festgelegt.
   bei System.Collections.Generic.LinkedList`1.AddFirst(LinkedListNode`1 node)
   bei Xtensive.Core.Collections.TopDeque`2.TryGetValue(K key, Boolean moveToTop, V& value) in c:\Temp\DataObjects.Net\Xtensive.Core\Xtensive.Core\Collections\TopDeque.cs:Zeile 78.
   bei Xtensive.Core.Caching.LruCache`2.TryGetItem(TKey key, Boolean markAsHit,TItem& item) in c:\Temp\DataObjects.Net\Xtensive.Core\Xtensive.Core\Caching\LruCache{TKey, TItem}.cs:Zeile 99.
   bei Xtensive.Storage.Internals.RecordSetReader.Read(IEnumerable`1 source, RecordSetHeader header) in c:\Temp\DataObjects.Net\Xtensive.Storage\Xtensive.Storage\Internals\RecordSetReader.cs:Zeile 59.
   bei Xtensive.Storage.Internals.Prefetch.EntityGroupTask.PutLoadedStatesInCache(IEnumerable`1 queryResult, RecordSetReader reader, HashSet`1 foundedKeys) in c:\Temp\DataObjects.Net\Xtensive.Storage\Xtensive.Storage\Internals\Prefetch\EntityGroupTask.cs:Zeile 141.
   bei Xtensive.Storage.Internals.Prefetch.EntityGroupTask.UpdateCache(HashSet`1 foundKeys) in c:\Temp\DataObjects.Net\Xtensive.Storage\Xtensive.Storage\Internals\Prefetch\EntityGroupTask.cs:Zeile 77.
   bei Xtensive.Storage.Internals.Prefetch.Fetcher.UpdateCacheFromAllEntityGroupTasks() in c:\Temp\DataObjects.Net\Xtensive.Storage\Xtensive.Storage\Internals\Prefetch\Fetcher.cs:Zeile 99.
   bei Xtensive.Storage.Internals.Prefetch.Fetcher.ExecuteTasks(IEnumerable`1 containers, Boolean skipPersist) in c:\Temp\DataObjects.Net\Xtensive.Storage\Xtensive.Storage\Internals\Prefetch\Fetcher.cs:Zeile 35.
   bei Xtensive.Storage.Internals.Prefetch.PrefetchManager.ExecuteTasks(BooleanskipPersist) in c:\Temp\DataObjects.Net\Xtensive.Storage\Xtensive.Storage\Internals\Prefetch\PrefetchManager.cs:Zeile 161.
   bei Xtensive.Storage.Providers.SessionHandler.FetchField(Key key, FieldInfo field) in c:\Temp\DataObjects.Net\Xtensive.Storage\Xtensive.Storage\Providers\SessionHandler.Fetching.cs:Zeile 85.
   bei Xtensive.Storage.Entity.EnsureIsFetched(FieldInfo field) in c:\Temp\DataObjects.Net\Xtensive.Storage\Xtensive.Storage\Entity.cs:Zeile 415.
   bei Xtensive.Storage.Entity.SystemBeforeGetValue(FieldInfo field) in c:\Temp\DataObjects.Net\Xtensive.Storage\Xtensive.Storage\Entity.cs:Zeile 577.
   bei Xtensive.Storage.Persistent.GetFieldValue[T](FieldInfo field) in c:\Temp\DataObjects.Net\Xtensive.Storage\Xtensive.Storage\Persistent.cs:Zeile 237.
   bei Xtensive.Storage.Persistent.GetFieldValue[T](String fieldName) in c:\Temp\DataObjects.Net\Xtensive.Storage\Xtensive.Storage\Persistent.cs:Zeile 198.
   bei Seo.Crawler.Model.Engine.UncheckedProxy.get_Host() in D:\Work\SeoBaseLib\SeoCrawlerModel\Model\Engine\UncheckedProxy.cs:Zeile 0.
   bei Seo.Crawler.Model.Engine.UncheckedProxy.get_Address() in D:\Work\SeoBaseLib\SeoCrawlerModel\Model\Engine\UncheckedProxy.cs:Zeile 74.
   bei Seo.Crawler.Model.Engine.UncheckedProxy.CreateTasks(CrawlerEngine engine, DateTime currentTime) in D:\Work\SeoBaseLib\SeoCrawlerModel\Model\Engine\UncheckedProxy.cs:Zeile 154.
   bei Seo.Crawler.CrawlerService.ScheduablesCreateTasksThreadFkt() in D:\Work\SeoBaseLib\SeoCrawlerService\CrawlerService\Scheduler\ProcessScheduables.cs:Zeile 107.

Exception: UncheckedProxy, (12668898)-System.InvalidOperationException: Der LinkedList-Knoten gehört nicht zur aktuellen LinkedList.
   bei System.Collections.Generic.LinkedList`1.Remove(LinkedListNode`1 node)
   bei Xtensive.Core.Collections.TopDeque`2.TryGetValue(K key, Boolean moveToTop, V& value) in c:\Temp\DataObjects.Net\Xtensive.Core\Xtensive.Core\Collections\TopDeque.cs:Zeile 78.
   bei Xtensive.Core.Caching.LruCache`2.TryGetItem(TKey key, Boolean markAsHit,TItem& item) in c:\Temp\DataObjects.Net\Xtensive.Core\Xtensive.Core\Caching\LruCache{TKey, TItem}.cs:Zeile 99.
   bei Xtensive.Storage.Internals.RecordSetReader.Read(IEnumerable`1 source, RecordSetHeader header) in c:\Temp\DataObjects.Net\Xtensive.Storage\Xtensive.Storage\Internals\RecordSetReader.cs:Zeile 59.
   bei Xtensive.Storage.Internals.Prefetch.EntityGroupTask.PutLoadedStatesInCache(IEnumerable`1 queryResult, RecordSetReader reader, HashSet`1 foundedKeys) in c:\Temp\DataObjects.Net\Xtensive.Storage\Xtensive.Storage\Internals\Prefetch\EntityGroupTask.cs:Zeile 141.
   bei Xtensive.Storage.Internals.Prefetch.EntityGroupTask.UpdateCache(HashSet`1 foundKeys) in c:\Temp\DataObjects.Net\Xtensive.Storage\Xtensive.Storage\Internals\Prefetch\EntityGroupTask.cs:Zeile 76.
   bei Xtensive.Storage.Internals.Prefetch.Fetcher.UpdateCacheFromAllEntityGroupTasks() in c:\Temp\DataObjects.Net\Xtensive.Storage\Xtensive.Storage\Internals\Prefetch\Fetcher.cs:Zeile 97.
   bei Xtensive.Storage.Internals.Prefetch.Fetcher.ExecuteTasks(IEnumerable`1 containers, Boolean skipPersist) in c:\Temp\DataObjects.Net\Xtensive.Storage\Xtensive.Storage\Internals\Prefetch\Fetcher.cs:Zeile 36.
   bei Xtensive.Storage.Internals.Prefetch.PrefetchManager.ExecuteTasks(BooleanskipPersist) in c:\Temp\DataObjects.Net\Xtensive.Storage\Xtensive.Storage\Internals\Prefetch\PrefetchManager.cs:Zeile 162.
   bei Xtensive.Storage.Providers.SessionHandler.FetchField(Key key, FieldInfo field) in c:\Temp\DataObjects.Net\Xtensive.Storage\Xtensive.Storage\Providers\SessionHandler.Fetching.cs:Zeile 85.
   bei Xtensive.Storage.Entity.EnsureIsFetched(FieldInfo field) in c:\Temp\DataObjects.Net\Xtensive.Storage\Xtensive.Storage\Entity.cs:Zeile 416.
   bei Xtensive.Storage.Entity.SystemBeforeGetValue(FieldInfo field) in c:\Temp\DataObjects.Net\Xtensive.Storage\Xtensive.Storage\Entity.cs:Zeile 579.
   bei Xtensive.Storage.Persistent.GetFieldValue[T](FieldInfo field) in c:\Temp\DataObjects.Net\Xtensive.Storage\Xtensive.Storage\Persistent.cs:Zeile 237.
   bei Xtensive.Storage.Persistent.GetFieldValue[T](String fieldName) in c:\Temp\DataObjects.Net\Xtensive.Storage\Xtensive.Storage\Persistent.cs:Zeile 198.
   bei Seo.Crawler.Model.Engine.UncheckedProxy.get_Host() in D:\Work\SeoBaseLib\SeoCrawlerModel\Model\Engine\UncheckedProxy.cs:Zeile 0.
   bei Seo.Crawler.Model.Engine.UncheckedProxy.get_Address() in D:\Work\SeoBaseLib\SeoCrawlerModel\Model\Engine\UncheckedProxy.cs:Zeile 74.
   bei Seo.Crawler.Model.Engine.UncheckedProxy.CreateTasks(CrawlerEngine engine, DateTime currentTime) in D:\Work\SeoBaseLib\SeoCrawlerModel\Model\Engine\UncheckedProxy.cs:Zeile 154.
   bei Seo.Crawler.CrawlerService.ScheduablesCreateTasksThreadFkt() in D:\Work\SeoBaseLib\SeoCrawlerService\CrawlerService\Scheduler\ProcessScheduables.cs:Zeile 107.

Subscription:

Related questions