1
1

How do I remove a type by migrating fields to parent class?

Example:

Version 1.0

[HiearchyRoot]
public class A : Entity
{
  [Field, Key]
  public long Id {get; private set;}
}

public class B : A
{
  [Field]
  public string Text {get;set;}
}

Version 2.0

[HiearchyRoot]
public class A : Entity
{
  [Field, Key]
  public long Id {get; private set;}

  [Field]
  public string Text {get;set;}
}

I want my instances of B to become instances of A, and Text field to be filled with information defined in B class.

Is this possible? How do I do this? I tried with CopyFieldHint, but if I remove B type all my instances are removed.

asked Oct 04 '10 at 10:23

olorin's gravatar image

olorin
358878792

edited Oct 05 '10 at 00:45

Alex%20Yakunin's gravatar image

Alex Yakunin
29714412


2 Answers:

Sorry, I discovered that initially we wrote a test for different case (propagation to derived type), so it took some time to test and described this.

First of all, there is no easy way to do this now - mainly, because there is no hint allowing to merge two types into one. So if you need to merge


Upgrade way 1: RDBMS-indepenent, based on schema hints

Old model:

[HierarchyRoot]
public class Base : Entity
{
  [Field, Key]
  public long Id { get; private set; }
}

public class Derived : Base
{
  [Field]
  public string Text { get; set; }
}

New model:

[HierarchyRoot]
public class Base : Entity
{
  [Field, Key]
  public long Id { get; private set; }

  [Field]
  public string Text { get; set; }
}

// It's necessary to keep the data till the final stage of upgrade;
// but note we converted this type to hierarchy root.
[HierarchyRoot]
[Recycled, Obsolete]
public class Derived : Entity
{
  // Necessary, because this is a hierarchy root now!
  [Field, Key]
  public long Id { get; private set; }

  [Field]
  public string Text { get; set; }
}

Overriding OnSchemaReady method in UpgradeHandler:

public override void OnSchemaReady()
{
  // Table names can actually be extracted from models,
  // but I'm intentionally simplifying the code here

  string baseTableNodePath = "Tables/Base";
  string baseTableColumnPath = "Tables/Base/Columns/";
  string derivedTableNodePath = "Tables/Derived";
  string derivedTableColumnPath = "Tables/Derived/Columns/";

  if (UpgradeContext.Stage==UpgradeStage.Upgrading) {
    // Replacing Dervided type's TypeId in Base table
    // to Base type's TypeId; if there are many ancestors,
    // this action must be performed for each of them.
    UpgradeContext.SchemaHints.Add(new UpdateDataHint(
      baseTableNodePath,
      new List<IdentityPair> {
        new IdentityPair(baseTableColumnPath + "TypeId",
          UpgradeContext.ExtractedTypeMap[typeof (Derived).FullName].ToString(), true)
      },
      new List<Pair<string, object>> {
        new Pair<string, object>(baseTableColumnPath + "TypeId",
          UpgradeContext.ExtractedTypeMap[typeof (Base).FullName])
      }));
  }
  if (UpgradeContext.Stage==UpgradeStage.Final) {
    // Copying the data from "Text" column in "Dervided" table
    // to "Base" table.
    UpgradeContext.SchemaHints.Add(new CopyDataHint(
      derivedTableNodePath,
      new List<IdentityPair> {
        new IdentityPair(
          derivedTableColumnPath + "Id", baseTableColumnPath + "Id", false)
      },
      new List<Pair<string>> {
        new Pair<string>(
          derivedTableColumnPath + "Text", baseTableColumnPath + "Text"),
      }));
  }
}

The description:

  • We convert Derived type to a hierarchy root that must be recycled on final stage of upgrade
  • On Upgrade stage: we remap TypeId of old type to new one using schema hint
  • On Final stage: we copy the data from Text column by the same way.

The following SQL is executed in this case:

UPDATE [dbo].[Base] SET [TypeId] = 101 
WHERE ([Base].[TypeId] = '102');
...
UPDATE [dbo].[Base] SET [Text] = [th].[Text] FROM (
  SELECT [a].[Id], [a].[Text] FROM [dbo].[Derived] [a]) [th] 
WHERE ([Base].[Id] = [th].[Id]);

So it's nearly the fastest way to perform such upgrade.


Upgrade way 2: RDBMS-dependent, based on direct SQL execution

Models are the same.

Overriding OnStage method in UpgradeHandler:

public override void OnStage()
{
  var session = Session.Demand();
  var directSql = session.Services.Demand<DirectSqlAccessor>();
  if (!directSql.IsAvailable)
    return; // IMDB, so there is nothing to uprade
  if (UpgradeContext.Stage==UpgradeStage.Initializing) {
    // Relying on Metadata.Type, because
    // only system types are registered in model @ this stage.
    int baseTypeId = Query.All<Metadata.Type>()
      .Where(t => t.Name==typeof(Base).FullName).Single().Id;
    int derivedTypeId = Query.All<Metadata.Type>()
      .Where(t => t.Name==typeof(Derived).FullName).Single().Id;
    var command = directSql.CreateCommand();
    command.CommandText = @"
      UPDATE [dbo].[Base] SET [TypeId] = {0} 
      WHERE ([Base].[TypeId] = {1});
      ".FormatWith(baseTypeId, derivedTypeId);
    command.ExecuteNonQuery();
  }
  if (UpgradeContext.Stage==UpgradeStage.Upgrading) {
    var command = directSql.CreateCommand();
    command.CommandText = @"
      UPDATE [dbo].[Base] SET [Text] = [th].[Text] FROM (
        SELECT [a].[Id], [a].[Text] FROM [dbo].[Derived] [a]) [th] 
      WHERE ([Base].[Id] = [th].[Id]);

      DELETE FROM [dbo].[Derived];
      ";
    command.ExecuteNonQuery();
  }
}

As you see, we do nearly the same, but using direct SQL commands. You might notice we use different upgrade stages here - that's because schema hints must be generated in the same stage when schema changes, but OnStage method is executed when domain is already built, i.e. after schema changes. That's why we use earlier stages here.


Upgrade way 3: RDBMS-independent, based on custom upgrade code

You may find that usage of direct SQL commands is actually an optimization in previous case. You can achieve the same with this OnStage code:

public override void  OnStage()
{
  if (UpgradeContext.Stage==UpgradeStage.Upgrading) {
    foreach (var derived in Query.All<Derived>()) {
      new Base {Text = derived.Text};
    }
  }
}

And actually, I'd advise you to use such code. It's slower than a single update, but must be pretty good as well (DO batches everything batching). But it's easier to understand and refactor.

So use this approach, if upgrade performance isn't essential.


Conclusion

As you see, the process is a bit tricky first two cases, but such upgrades are possible. The third option is pretty simple, so likely, it will be the best choice in majority of cases.

answered Nov 16 '10 at 19:36

Alex%20Yakunin's gravatar image

Alex Yakunin
29714412

edited Nov 16 '10 at 19:39

Sadly, I have 1 367 109 instances of this particular class so method 3 is not possible ;) Method 3 will work only if you have few instances : else we risk the timeout and rollback of update.

However I did move the fields to 'Base' class without any problem using hints : I just didn't remove the 'Derived' class.

(Nov 17 '10 at 05:41) olorin olorin's gravatar image

So I must use method 1 or 2, but I'd rather wait the development of an "TypeMerge" hint to do this : is it possible that you develop such an hint for next release? I've the feeling we'll get this migration case from time to time, and this UpgradeHint would very helpful.

(Nov 17 '10 at 05:44) olorin olorin's gravatar image

Sadly, I have 1 367 109 instances ...

ASP.NET MVC Sample creates ~ 100K instances on startup. It takes ~ 10 seconds on good PC, entity has several fields. I.e. entity creation performance is about 10K entities per second in case this is a hierarchy root.

So in this case it should take about 2.5-3 minutes to handle the migration using this method (it's better to do this in transactions processing ~ 10K entities at once). If you need a single transaction, timeouts can be adjusted to ensure it will be finished, but this will take more time in this case.

(Nov 17 '10 at 06:41) Alex Yakunin Alex%20Yakunin's gravatar image

I think even 1 hour is acceptable time for migrations in most of cases. At least, many Google services (e.g. Google Code) frequently use ~ such time periods for updates.

(Nov 17 '10 at 06:42) Alex Yakunin Alex%20Yakunin's gravatar image

is it possible that you develop such an hint for next release?

For now this isn't planned - frankly speaking, I'd prefer to postpone this until release of v4.5 beta (caching + likely, security). We're targeting to release it closer to new year.

That's a feature that really simplifies handling such cases, but since typical development pattern with DO is "develop, develop, develop, write a migration code, release", this is isn't what people strongly need in development, + there are workarounds like listed ones. I think it's better to bring few more features first...

(Nov 17 '10 at 06:45) Alex Yakunin Alex%20Yakunin's gravatar image

I'll test method 3 that when I have the time. But how I can split the update in several transaction? In UpgradeHandler there is already an opened transaction for update, so should I do it outside UpgradeHandler?

By the way, the duration is not the only problem : if I create 1 million objects in one transaction wouldn't the transaction log grow to an huge size?

Another question: with method 3, how do you remove the instances of 'Derived' class?

(Nov 18 '10 at 04:33) olorin olorin's gravatar image

The timing for the new hint is yours : while it's not that important, I think it would be useful.

(Nov 18 '10 at 04:35) olorin olorin's gravatar image

In UpgradeHandler there is already an opened transaction for update, so should I do it outside UpgradeHandler?

No, you shouldn't, but you can control it: UpgradeContext.TransactionScope is the scope controlling upgrade transaction. So it must be completed & disposed to commit it; futher you can create & assign a new one there.

(Nov 18 '10 at 05:01) Alex Yakunin Alex%20Yakunin's gravatar image

If I create 1 million objects in one transaction wouldn't the transaction log grow to an huge size?

Yes, it will. So a solution with intermediate commits might be helpful in such cases.

(Nov 18 '10 at 05:02) Alex Yakunin Alex%20Yakunin's gravatar image

Another question: with method 3, how do you remove the instances of Derived class?

They'll be automatically removed on Final upgrade stage - the type is marked as [Recycled].

(Nov 18 '10 at 05:03) Alex Yakunin Alex%20Yakunin's gravatar image
1

Most likely type merge will be done with issue 857 (i.e. as part of v4.4.X).

(Nov 28 '10 at 15:46) Alex Yakunin Alex%20Yakunin's gravatar image

Custom upgrade handler must help here:

  1. Put [Recycled, Obsolete] @ class B, but leave it.
  2. Make necessary changes in class A (add Text field)
  3. Add custom upgrade handler with the code described below.

Upgrade handler code, option 1 (custom upgrade code):

public class Upgrader : UpgradeHandler
{
  public override void OnUpgrade()
  {
    foreach (var b in Query.All<B>())
      new A { Text = b.Text };
  }

  public override bool CanUpgradeFrom(string oldVersion)
  {
    return oldVersion=="1.0";
    // return base.CanUpgradeFrom(oldVersion); // Either the same version, 
    //                                         // or a new DB.
  }
}

Upgrade handler code, option 2 (with upgrade hints):

public class Upgrader : UpgradeHandler
{
  protected override void AddUpgradeHints(ISet<UpgradeHint> hints)
  {
    hints.Add(new RenameTypeHint(typeof (B).FullName, typeof (A)));
    hints.Add(new CopyFieldHint(typeof (B).FullName, "Text", typeof (A)));
    // Or, if B is already deleted (i.e. no any [Recycled, Obsolete]):
    // hints.Add(new RenameTypeHint("BNamespace.B", typeof (A)));
    // hints.Add(new CopyFieldHint("BNamespace.B", "Text", typeof (A)));
  }

  public override bool CanUpgradeFrom(string oldVersion)
  {
    return oldVersion=="1.0";
    // return base.CanUpgradeFrom(oldVersion); // Either the same version, 
    //                                         // or a new DB.
  }
}

Difference between these two versions:

  • First version implies old B instanced are destroyed, but new A instances are created. I.e. if there were references to Bs, they'll be nullified. To keep them, you must manually write the code doing this in OnUpgrade method (i.e. finding all references pointing to old B instances and changing them to corresponding A instances). Btw, you can use ReferenceFinder to do this - it operates quite efficiently.
  • Second version does not recreate anything - it is purely hint-based solution. You don't need to keep [Recycled, Obsolete] public class B there.

Upgrade handler is a regular class inherited from Xtensive.Storage.Upgrade.UpgradeHandler, that must be registered in DomainConfiguration.Types - normally, as part of the whole assembly with persistent types.

answered Oct 05 '10 at 00:15

Alex%20Yakunin's gravatar image

Alex Yakunin
29714412

edited Oct 05 '10 at 00:44

Just edited my answer.

(Oct 05 '10 at 00:38) Alex Yakunin Alex%20Yakunin's gravatar image

Very interesting! I missed the RenameTypeHint. I will test this right away.

(Oct 05 '10 at 03:43) olorin olorin's gravatar image

I will add that option 2 is better for performance. I have ~500 000 entities to migrate and that will take forever with option 1.

(Oct 05 '10 at 03:46) olorin olorin's gravatar image

I tried option 2 but I get an exception on Build: "Sequence contains no matching element"

Stack:

System.Linq.Enumerable.Single[TSource]
Xtensive.Storage.Upgrade.HintGenerator.GenerateTypeIdFieldRemoveHintsForConcreteTable()
Xtensive.Storage.Upgrade.HintGenerator.GenerateHintsXtensive.Storage.Upgrade.UpgradingDomainBuilder.BuildSchemaHints
...

Do you need more information or a sample?

(Oct 05 '10 at 04:13) olorin olorin's gravatar image

Did you apply [Recycled] to B? If not, could you try this?

In worst case, sample is preferable. This definitely should work.

(Oct 05 '10 at 05:32) Alex Yakunin Alex%20Yakunin's gravatar image

I removed the type.

(Oct 05 '10 at 06:13) olorin olorin's gravatar image

If I keep the type (without fields) and add [Recycled], I get an ArgumentException ("An item with the same key has already been added.") at

...

System.Core.dll!System.Linq.Enumerable.ToDictionary<xtensive.storage.upgrade.renametypehint,string>

Xtensive.Storage.dll!Xtensive.Storage.Upgrade.HintGenerator.RewriteGenericTypeHints

...

(Oct 05 '10 at 06:20) olorin olorin's gravatar image

Clear. I'll create a bug test for this and notify you.

(Oct 05 '10 at 07:03) Alex Yakunin Alex%20Yakunin's gravatar image

Ok. Thank you!

(Oct 05 '10 at 10:41) olorin olorin's gravatar image

Any news on this issue? Do you need a sample?

(Oct 13 '10 at 08:50) olorin olorin's gravatar image

Sorry, it's still under investigation - we need ~ 2 more days, since there are lot of similar work in queue. The sample isn't necessary.

(Oct 18 '10 at 00:08) Alex Yakunin Alex%20Yakunin's gravatar image

Issue 841 is created.

(Oct 21 '10 at 09:16) Alex Yakunin Alex%20Yakunin's gravatar image

Done. Full explanation will follow up soon.

(Oct 23 '10 at 04:05) Alex Yakunin Alex%20Yakunin's gravatar image

Is this fixed in build 6600?

(Nov 04 '10 at 09:33) olorin olorin's gravatar image

Yes, it is - sorry, I forgot to mention this. Today I'll publish a sample showing what must be done to describe such changes.

(Nov 06 '10 at 05:03) Alex Yakunin Alex%20Yakunin's gravatar image

I still can't get it to work.

Upgrader code: hintSet.Add(new RenameTypeHint("B", typeof(A)));

If I remove the type I get this exception: System.ArgumentException occurred Message=An item with the same key has already been added in Xtensive.Storage.Building.SchemaComparer.Compare

If I add the attribute [Recycled] I get: System.ArgumentException occurred Message=An item with the same key has already been added in Xtensive.Storage.Upgrade.HintGenerator.RewriteGenericTypeHints

I you need the full stack, ask me : I can't post it here.

(Nov 10 '10 at 04:52) olorin olorin's gravatar image

Please see the answer I just posted.

(Nov 16 '10 at 19:36) Alex Yakunin Alex%20Yakunin's gravatar image
Your answer
Please start posting your answer anonymously - your answer will be saved within the current session and published after you log in or create a new account. Please try to give a substantial answer, for discussions, please use comments and please do remember to vote (after you log in)!
toggle preview

powered by OSQA