Question

I am in the process of migrating a legacy binary file format. Part of this involves converting objects to a byte array. But after a while I run into an OutOfMemoryException. I created a small sample program to demonstrate the problem. You can download it here.

for (int i = 0; i < 20; i++)
{
    Training training = service.GetNewTraining();
    training.Name = Guid.NewGuid().ToString();

    for (int j = 0; j < 10; j++)
    {
        Run run = service.GetNewRun();

        run.Created = DateTime.Now;
        run.Distance = GetRandom(10000, 40000); // Random between 10k and 40k.

        TimeSpan time = TimeSpan.FromHours((run.Distance / 1000) / 25.0);
        for (int k = 0; k < time.TotalSeconds; k++)
        {
            RunDataSample sample = new RunDataSample
            {
                Cadence = GetRandom(60, 110),
                Distance = (float)(time.TotalHours * 25),
                Time = TimeSpan.FromSeconds(k),
                HeartBeat = GetRandom(80, 180),
                Speed = 25,
                Power = GetRandom(200, 400)
            };

            run.RunSamples.Add(sample);
        }

        training.Runs.Add(run);
    }

    service.Save();
}

I once tried adding a GC.Collect() call after the service.Save() call, but that didn't help either.

I have no need for the objects to be in the memory after I saved them. How can I clean them up properly?

In the included sample I am using an in-memory database, but in my real application I am using an SqlCe database with an x86 build.

Answer 1

0

Hello jensen,

To import a huge amount of data I strongly recommend you to use SessionOptions.ServerProfile & manually open and complete transactions. This will boost the entire import process drastically.
In case you definitely want to do the import with SessionOptions.ClientProfile then reopen sessions from time to time, as Session in SessionOptions.ClientProfile doesn't clean its caches at all because it is not intended to be used with that amount of data.
Session.Persist in your case is called so frequent because you use auto transactions (which are used intentionally in SessionOptions.ClientProfile), so every setting a persistent property or adding an item to EntitySet results into commit of a small auto transaction, which in turn lead to Persist method call. To avoid it, open and commit the transactions manually.

BTW, I've updated the sample and add a transaction for each step of the outer for loop (20 iterations), as a result, I got only 20 Persist calls.

answered May 17 '11 at 10:11

Dmitri Maximov
2211●1●2●11

I can do that for the importer, but what if I'm unable to frequently re-open a session during the run of my application?

I have a few services which are initialized at the start of the application and which should exist for the entire run of the application. These services contain some objects from the database like the current player, currently selected training and so forth. If I need to re-open the database I lose the current object and need to get all of those again, increasing complexity of my application, if it would be possible at all.

(May 17 '11 at 10:24) jensen

I also have a screen which provides an overview of all runs. You can select one, which loads the byte[] of samples into the memory. When you close the screen and go back to the overview I perform a new query, but as far as I understand all the loaded bytes will still be kept in memory, attached to the session. Or will it somehow be cleaned?

If I understand correctly, I still would run into the same exception when analyzing a lot of different runs.

(May 17 '11 at 10:25) jensen

The case with the long running services and overview screen are OK, they won't consume lot's of data. As for the screen that loads byte[] of samples, I'd open it in a new Session and dispose it after the screen is closed.

(May 17 '11 at 10:39) Dmitri Maximov

Subscription:

Related questions