[Schevo-devel] Schema evolution

[Matthew Scott]

On 12/7/05, Tom Locke <tom at livelogix.com> wrote:
> OK - whats puts the evo in schevo?

Partially, because the way you define a schema in Schevo is evolved :)

But more importantly, because it gives support for the task of
migrating from one schema version to the next.

Keep in mind that the main focus has lately been on same-version
evolution, where you are starting from scratch and are making minor,
incremental changes during development.

Version-to-version evolution has not been tackled at all in Schevo 3,
although it will build on the intra-version evolution that is already
in place.  Schevo 2 had working version-to-version evolution, and Pat
and I have successfully used it.  Yet another thing that we'll need
soon, that that we've been slowly chipping away at for a bit already
:)


>
> Easy one first - how would I rename a field?

Current, intra-version:  Add the new field.  Update the database by
setting all old field values to the new field when iterating over all
entities.  Remove the old field.  Update the database.

Ideal, version-to-version:  Remove the old field 'old_field'.  Add the
new field, passing it the argument was='old_field':

  class Foo(E.Entity):
     new_field = f.unicode(was='old_field')

The idea is that during migration, the field spec for Foo will be a
union of the old and new, so that anything (extents, fields) removed
in the new schema is temporarily available for migration purposes.
Renaming things, like in this example, should be fast operations too,
since internally we keep a mapping between extent/field names and
random unique identifiers.

And of course, if any point in the migration fails, the entire
database is rolled back and continues to use the old schema :)

The 'was' name would also apply for extents:

  class Bar(E.Entity):
      new_field = f.unicode(was='old_field')
      _was = 'Foo'

That would not only rename the field, but also rename the extent. 
Foo.old_field is now Bar.new_field.


>
> Then a harder one - I have a db with People who have an address field
> and 0-or-more PhoneNumbers (which have a name field - work, home etc.).
>
> Now I want to separate Homes from People:
>
> A Person has a Home and 0-or-more PhoneNumbers (cell phones, work
> phones, i.e. not related to the home)
>
> A Home has an address and 0-or-more PhoneNumbers.
>
> So some of the PhoneNumbers got moved out to the Home along with the
> address (those named "Home"), and the others stayed with the Person -
> make sense?
>
> The reason for the change is that the database has a lot of families, so
> there's a ton of redundant info (same home phone number and address for
> all members of the family).
>
> Doable?

Doable :)

Here's a simplistic view of it.

-------------------------------

#First version schema:

class Person(E.Entity):

    name = f.unicode()
    address = f.unicode()

    _key(name)

    _plural = 'People'


class PhoneNumber(E.Entity):

    contact = f.entity('Person')
    number = f.unicode()

    _key(who, number)


-------------------------------

#Second version schema:

class Person(E.Entity):

    name = f.unicode()
    home = f.entity('Home')

    _key(name, home)

    _plural = 'People'


class Home(E.Entity):

    address = f.unicode()

    _key(address)


class PhoneNumber(E.Entity):

    contact = f.entity('Home', 'Person')
    number = f.unicode()

    _key(contact, number)


-------------------------------

#The evolution script might look like this (untested, and cannot
currently do this)

def evolve(db):
    for person in db.Person.find():
        # Get their old address field.
        address = person.address
        # Create a Home if necessary, or return the existing one if
it's already there.
        tx = db.Home.create_if_necessary()
        tx.address = address
        home = db.execute(tx)
        # Update the person to give them a home.
        tx = person.t.update()
        tx.home = home
        db.execute(tx)
        # After the database begins using the new schema,
        # the 'address' field is no longer accessible on Person entities.

-------------------------------

Essentially, because you can have heterogeneous entity references (the
'contact' field in the second 'PhoneNumber' definition), that part is
easy; you can keep the existing phone numbers in place associated with
Person entities, and add new ones that can point to Home entities
instead.  The evolve script would take care of the more complex,
database-wide stuff that isn't easy to declare directly in the entity
definition.



--
Matthew R. Scott