Hi,

I've got the feeling the FutureQueries are not well documented. The only documentation I find is in the Manual and this is not extensive. In the Reference Guide we can't even find the Methods (Execute(), ExecuteFuture(), etc..). There are no Future Queries in the Samples and the Wiki. If there is any documentation then I've missed it. Any pointers here would be valued.

We discovered for instance that the ExecuteFuture() only works when a transaction is active. If this interpretation is true and there is no other way to execute them (i.e. without an open transaction) the future queries can not be used in our scenario.

We centrally open a Session, add a DisconnectedState and call it's Connect(). We would like to use the Future Queries to preload certain tables (reference tables) at the start of the application. Future Queries would be the perfect means to reach that goal, but since we totally rely on auto-transactions, no transaction is open from the start of the program and Future Queries will fail. We also tried opening our own transaction but as soon as that is Disposed, the Future Queries that were open inside transaction, cannot be accessed from outside the transaction.

Ok, here are some questions:

  • Can we use Future Queries without a transaction?
  • If not, is there another way to use Future Queries in our scenario?

Regards


Updated at 27.07.2010 12:33:03

Just found the following link (via Google) http://help.dataobjects.net. There the Methods are described but still not extensively.


Updated at 29.07.2010 10:06:04

Hi Alex,

I still think we don't fully understand the concept of DO. Everything in DO seem transaction based. This would mean that there are no 'disconnected' Entities. Any change made to an Entity is always persisted to the DB.

We interpreted the DisconnectedState as a means to create disconnected recordsets. In a way that's what they are but in another way they aren't. If we want to use the full fledged functionality of DO, we need to open a transaction.

What's really unclear is how the DisconnectedState and (nested) transaction cooperate. Questions like:

  • What happens when we open a transaction. Does it open 1 in the DB too?
  • Can we keep a transaction open accross ApplyChanges() borders? Isn't that going to produce locks in the DB?

are not really answered in the Documentation.

  • What we mis is a complete conceptual discription of DO.
  • How does it handle it's tasks? What are the deeper thoughts behind the concept of DO?
  • What is the real function of the DisconnectedState, what is it's purpose?

I could think of a lot more questions, but the basic question is 'What are the concepts behind DO?'.

Regards


Updated at 29.07.2010 10:15:38

Aha.. my colleaque Marco just found out that Future Queries should be executed inside a transaction but we can access the objects read from them outside the transaction. Pfeww that makes life easier. Now we can generate code that preloads all the reference tables in just 1 batch. That's great.

This thread was imported from our support forum. The original discussion may contain more detailed answer.

asked Jul 27 '10 at 12:18

Paul%20Sinnema's gravatar image

Paul Sinnema
261868896

edited Sep 03 '10 at 15:41

Alex%20Yakunin's gravatar image

Alex Yakunin
29714412


One Answer:

Alex (Xtensive) wrote:

Just checked this - true, .CHM reference from .NET 4.0 version has a set of strange lacks (mainly, absence of member documentation). Almost 100% sure this is a result of some Sandcastle bug, since...

The good news is that reference from .NET 3.5 version seems fully correct. Right now I'm updating it to downloads section.

.NET 3.5 version is ~99.9% identical to .NET 4.0 version by its members, so please replace its .CHM file by that one. Hopefully, I'll be able to implement its automatic inclusion to installer shortly (the problem is that actually these versions reside in different branches - so from the point of current build process they have nothing common)...


Alex (Xtensive) wrote:

Concerning future queries: the example form Manual project: http://goo.gl/0iMh

You can't use them without a transaction - we create internal query tasks, when such queries are built, and these tasks are executed inside the next batch sent to server, but batches can't cross transaction boundaries. Normally it's even wrong to execute a set of correlated queries without a transaction: you expect the consistent result there (i.e. if first query fetches Authors and second - Books, they must properly relate to each other; now imagine query sent in between simply deletes all the Books), but there is no way to ensure this without a transaction.

Concerning your case: I don't understand what prevents you from creating a transaction manually to run a set of queries in batch - can you explain this?


Alex (Xtensive) wrote:

Let's start from simple questions:

> Marco just found out that Future Queries should be executed inside a transaction but we can access the objects read from them outside the transaction.

That's true. The transaction is necessary here just to ensure the data is fetched as part of one logical operation.

> What happens when we open a transaction. Does it open 1 in the DB too?

I know you're talking about the case when DisconnectedState (DS further) is attached to Session, and moreover, it is already "connected". So:

  • When you open a transaction, nothing happens in DB, but logical transaction is opened in DS. This means you can change something in entities and then roll back it - the state will be rolled back.

  • DB-level transaction starts when the first operation hitting DB is executed by DS. E.g. query.

  • DB-level transaction gets closed when the transaction in DS gets closed.

So in short, it's ok if you open many transactions in DS. Actual transactions in DB will run only when you really access the DB inside these transactions.

> Can we keep a transaction open accross ApplyChanges() borders? Isn't that going to produce locks in the DB?

Yes, you can. Locks will be acquired as they should - DO isn't responsible for this, this is solely a problem of SQL Server. But if you want to release X-locks acquired during ApplyChanges, you must complete the transaction (completing the transaction is the only way to release the locks in DB).

Note that ApplyChanges creates either nested or outermost transaction while it runs, and commits/rolls back it dependently on result of the whole operation.


Alex (Xtensive) wrote:

I also want to answer the question about asynchronous loading\fetches - I remember there was a question touching this.

Imagine we have a Session (usSession) with attached DisconnectedState (uiDS); UI is bound to entities provided by this Session. And now we'd like to load a large set of entities into this Session - asynchronously. To achieve this, you must:

  • create a background thread

  • create its own Session (bgSession) and its own DisconnectedState (bgDS) attached to it

  • run the queries fetching the data to bgSession. Since it has attached DS, all the entity states fetched by these queries will be cached by bgDS. Note that you must enumerate the query result (e.g. using our IEnumerable<t>.Run() extension method from Xtensive.Core) to ensure it is fully materialized - DO delays nearly any operation until the data is really necessary, so if this isn't done, nothing will be cached.

  • detach bgDS, dispose bgSession

  • pass bgDS to the UI thread

  • call uiDS.Merge(bgDS). This operation must be synchronous; also note MergeMode can be passed to Merge method (if it is not specified, uiDS.MergeMode will be used), so merge conflicts are resolved at this stage. You can get an exception, if mergeMode==MergeMode.Strict, and there is a version conflict for some entity (i.e. its version cached in uiDS differs from version in bgDS).

That's it.


Alex (Xtensive) wrote:

Forgot to add: if set to load asynchronously is really large (e.g. 100K...1M entities - I know it's a bad idea to cache this set on the client, so this is just FYI), I'd even split the whole operation to a set of smaller "page load" operations, where each page includes ~ 1K...10K entities.

I.e. the described routine must be used sequentially in this case.


LeonardiMarco wrote:


Marco just found out that Future Queries should be executed inside a transaction but we can access the objects read from them outside the transaction.

That's true. The transaction is necessary here just to ensure the data is fetched as part of one logical operation.


But now we have the following problem:

using (Transaction trans = Transaction.Open(session)
{
   IEnumerable<Person> persons = Query.ExecuteFuture<Person>(() => .....
   IEnumerable<Organization> orgnaization = ...
   ...
   orgnaization.ToList
   trans.Complete();
}

Outside of the transaction I want to foreach all entities:

using (IEnumerator<T> en = persons.GetEnumerator())
                {
                    while (en.MoveNext())
                    {
                        Add(en.Current);
                    }
                }

But on the persons.GetEnumerator() I got the follow Excpetion:

"InvalidOperationExcpetion occured: The current transaction is different from the transaction bound to this instance."

I don't understand why he needs transaction, beacuse he already has the data from the DB.

You agree?

Regards Marco Leonardi


psulek wrote:

... I don't understand why he needs transaction, beacuse he already has the data from the DB.

No, it does not have data for persons from DB because you create just IEnumerable<person> persons, which does not load data from DB until it is enumerated or called ToList() like you does with orgnaization.ToList.

Correction:

List<Person> personList;
using (Transaction trans = Transaction.Open(session)
{
   IEnumerable<Person> persons = Query.ExecuteFuture<Person>(() => .....
   personList = persons.ToList();
   trans.Complete();
}

As you can see here i add line personList = persons.ToList(); which iterate through IEnumerable and loads data from DB to generic list List<person> and outside transaction you can iterate variable personList without active transaction.


Alex (Xtensive) wrote:

Btw, that's why you get IEnumerable<t> as result of Query.ExecuteFuture<t>(() => ...):

  • we queue the query for future execution when you run Query.ExecuteFuture - nothing like this happens with regular compiled query;

  • when you enumerate the enumerable, we either materialize already fetched result (if it's there at the moment of materialization), or force execution of prepared query batch to ensure the result is available.

Let's compare this to compiled query (Query.Execute<t>(() => ...)):

  • Invocation of this method simply looks up for query compilation result in cache (+ compiles the query and adds it, if nothing is cached), and binds it with enumerable it returns;

  • Enumeration of this enumerable leads to actual query parameterization and execution - as with regular LINQ query. The only difference is that translation stage isn't necessary here.

Caching key is MethodInfo of delegate you pass to Query.Execute<t>(() => ...) - if you write it as anonymous delegate, its MethodInfo is unique in each case, i.e. that's exactly what is necessary here. The delegate itself is invoked just once.

Finally, it worth to mention once more that we never materialize the whole result at once. Instead we materialize it in batches; batch size grows up as power of 2 from ~ 16 to 1024, and stops growing. This, in combination with weak reference-based entity state cache, allows DataObjects.Net to browse potentially infinite result sets.


LeonardiMarco wrote:


As you can see here i add line personList = persons.ToList(); which iterate through IEnumerable and loads data from DB to generic list List<person> and outside transaction you can iterate variable personList without active transaction.


Psulek, your above conclusion is not right: If you apply ToList also on persons and you want to iterate it outside of the transactionit throws the same Excpetion. This confirms the statement of Alex: " when you enumerate the enumerable, we either materialize already fetched result (if it's there at the moment of materialization), or force execution of prepared query batch to ensure the result is available "

The solution of my problem is to use the List wicht returns the IEnumerable<person>.ToList(). This list must never refresh/update/ to the DB beacuse during the whole programm execution this list does never change. (maybe persons was not the right example, rather a list which fills a ComboBox).

Regards Marco Leonardi


psulek wrote:

I dont understand you, because i write that you must use:
personList = persons.ToList() where variable persons is of type IEnumerable<person> and you tell me that this is not good but your solution is to use IEnumerable<person>.ToList(). I think this is same as my solution, right?


LeonardiMarco wrote:

I dont understand you, because i write that you must use:
personList = persons.ToList() where variable persons is of type IEnumerable<person> and you tell me that this is not good but your solution is to use IEnumerable<person>.ToList(). I think this is same as my solution, right?

Hi psulek, What I mean is follow:

Requires a transaction:

IEnumerable<Person> persons = Query.ExecuteFuture<Person>(() => .....
using(Transaction trans = TranscationScope.Open(..)
{
   persons.ToList(); // data is loaded from DB
   trans.Complete
}
 using (IEnumerator<T> en = persons.GetEnumerator()) // Would raise an exception
{
    while (en.MoveNext())
    {
       Add(en.Current);
     }
}

Doesn't require a transaction:

List<Person> personList;
IEnumerable<Person> persons = Query.ExecuteFuture<Person>(() => .....
using(Transaction trans = TranscationScope.Open(..)
{
   personList = persons.ToList(); // data is loaded from DB
   trans.Complete
}
foreach(Person pers in personList) // doesn't require a transaction
{
    Console.Writeline(pers.Firstname);
}

LeonardiMarco wrote:

I have the follow problem:

List<PostalCode> pCodes;
List<Country> pCountrys;

using (TransactionScope trans = Transaction.Open(session))
{
    pCodes = Query.All<PostalCode>().ToList();
    pCountrys = Query.All<Country>().ToList();

    trans.Complete();

    foreach (PostalCode pc in pCodes)
    {
                Console.WriteLine(pc.IdP.Value);
    }
}

foreach (Country co in pCountrys)
{
    Console.WriteLine(co.IdP.Value);
}

The first foreach which iterates the 'pCodes' takes the entities from the cache and needs no other connection to the DB. But the second foreach (situated outside of the transaction) which iterates the 'pCountrys' executes for each item in the list an SQl query, although he already got the data. Why he doesn't take the data from the cache?

Outside of this transaction I'm using 'DisconnectedState'. Hope you can help me

Regards Marco Leonardi


Alex (Xtensive) wrote:

> But the second foreach (situated outside of the transaction) which iterates the 'pCountrys' executes for each item in the list an SQl query, although he already got the data.

This may happen only if DisconnectedState isn't attached to the current Session.

And one more note: you run the code after calling TransactionScope.Complete() - that's wrong: in our case TransactionScope.Complete just sets the flag that commit must happen instead of rollback, i.e. everything is ok, and thus it's safe to throw an exception from IDisposable.Dispose (see the end of this post, there is an explanation).

So TransactionScope.Complete must normally be called right before leaving corresponding using block.


LeonardiMarco wrote:

This may happen only if DisconnectedState isn't attached to the current Session.

Thats right, the DisconnectedState was not yet connected to the current Session. I have to open the transacation also after the Connect of the DisconnectedState.

Thanks for your help

Regards Marco Leonardi

answered Jul 28 '10 at 16:25

Editor's gravatar image

Editor
46153156157

Yes, exactly.

(Jul 28 '10 at 16:25) Alex Yakunin Alex%20Yakunin's gravatar image

Hi Alex,

I could think of a lot more questions, but the basic question is 'What are the concepts behind DO?'.

This got snowed under a bit. Maybe on your blog or in the Wiki?

Regards

(Jul 28 '10 at 16:25) Paul Sinnema Paul%20Sinnema's gravatar image

Yes, I'll do this quite soon. After finishing the fixes I'm busy with right now...

(Jul 28 '10 at 16:25) Alex Yakunin Alex%20Yakunin's gravatar image

Yes, in case with future queries transaction should be anyway opened.

(Jul 28 '10 at 16:25) Alex Yakunin Alex%20Yakunin's gravatar image
Your answer
Please start posting your answer anonymously - your answer will be saved within the current session and published after you log in or create a new account. Please try to give a substantial answer, for discussions, please use comments and please do remember to vote (after you log in)!
toggle preview

powered by OSQA