[Schevo-devel] scalability of schevo

[Matthew Scott]

David Binger wrote:
> 
> On Feb 28, 2006, at 2:08 AM, Matthew Scott wrote:
> 
>> Transactions would be able to cross database boundaries, but would only
>> result in a write-lock being obtained on databases that the transaction
>> is actively attempting to change.
> 
> Do you have some ideas about how to make simultaneous
> multi-database transactions atomic?


Off the top of my head, the simplest approach would use synchronization,
and some sort of configurable time limit to prevent deadlocks on the
remote database.  It might be slow, but it could be a starting point at
least.

Let's say db-a has this:

class Foo(E.Entity):

    # /db-b/Bar is a reference to an aliased database which is
    # assumed to have been connected with db-a already.  ('name', )
    # is the key spec used to store OID-independent remote
    # references within db-a.
    bar = f.ext_entity(('/db-b/Bar', ('name', ), required=False)
    level = f.integer()

...and db-b has this:

class Bar(E.Entity):

    name = f.unicode()
    skill = f.float()

    _key(name)


db-a.Foo could have this transaction, as long as db-b trusted db-a to
lock the database for the purposes of atomic commits:


class _CreateWithBar(T.Transaction):

    name = f.unicode()
    skill = f.float()

    def _execute(self, db):
        # First, acquire a lock on the external database.
        # If the lock has already been acquired by a parent tx,
        # a reference count is updated and that lock is returned.
        db_b = db.lock_ext('/db-b')
        try:
            # For convenience, db_b has the Schevo database API.
            # Create a create transaction on db-b.
            tx = db_b.Bar.t.create(name=self.name, skill=self.skill)
            # bar becomes an external hard reference (i.e. referenced
            # by OID) to the new Bar entity.
            bar = db_b.execute(tx)
            # Now create the local Foo.
            tx = db.Foo.t.create(bar=bar, level=int(bar.skill * 2) / 3)
            foo = db.execute(tx)
            return foo
        finally:
            db_b.release()


(I think Python 2.5 will have a syntax that will make that even cleaner
to write.  Don't have time to look it up right now, but looking forward
to it :)

Now, if you didn't care that a Bar was created remotely even if the
local transaction failed, you don't have to create a lock; you'd just do:

db_b = db.ext('/db-b')
tx = db_b.Bar.t.create(name=..., skill=...)
bar = db_b.execute(tx)

If that last line succeeded, you're guaranteed that there is a new Bar
in db-b, whether or not the calling transaction succeeds or fails.


There is of course trust and timeout issues required for one database to
allow a remote database to lock it.

There is also room for developing techniques for using distributed
databases like this without necessarily requiring atomicity for certain
transactions.


Inter-connected local databases would not need to worry so much about
trust and latency though.


-- 
Matthew Scott
Springfield Technology