[Schevo-devel] scalability of schevo

[Matthew Scott]

In describing Schevo to some newcomers in recent days, one of the things
I have been questioned about is how Schevo scales in a variety of ways.

Here are some quick notes to incite some conversation in this area.
What I'll be describing below are some "vectors" of scalability that I
foresee needed by the apps that I myself will have responsiblity for in
terms of developing, deploying, and managing.

All of these items have at least been partially discussed on this list,
on other lists, or in conversations amongst core developers, so there's
nothing brand-new presented here -- it's more of a compilation of
several existing notions.

I hope this provides some insight for newcomers and old-timers alike as
far as some of the types of long-term improvements we have envisioned
for Schevo.


Process:Database ratio scalability
==================================

Right now, the only process that has direct access to the Python API for
a Schevo database is the process that has the database open.

Ideally, a client written in any language should be able to connect to a
Schevo server using commonly-accepted lightweight protocol combos, such
as HTTP+REST+POX, and then expose an API to the database not unlike
using Schevo directly.  The server would be responsible for limiting
read/write access to the database based on context such as a user's
credentials, location, whatever.

Work on this has started, in the guise of building web UI support into
Schevo, and the client/server concept has been experimented on briefly
in the past with Schevo.

I hope that by refining the latest incarnation of that over time, it
becomes possible to extend it so that instead of outputting pretty HTML
suitable for human consumption, we can output pretty XML or JSON or
whatever that is suitable for client-side code consumption. :)


Database size scalability
=========================

To my knowledge (and the Durus folks can confirm this, or let me know
I'm on crack) one current limitation of Durus, the underlying storage
engine used by Schevo, is that an efficient index to all Python objects
tracked in a Durus database is kept in-memory.

While this index is a fraction of the size of the data it indexes, it
still grows linearly with database size.

To tackle this, the Schevo team should work with the Durus team to
identify a solution to this that can at least optionally be used to
limit the amount of memory Durus is allowed to use for in-memory indices
(e.g. if performance suffers slightly with a paged index vs. a
completely in-memory index).

Before this is done though, it would be good to perform some benchmarks
to see if it is satisfactory to just let Durus's in-memory index grow
unbounded, and let the operating system's swap facilities take care of
paging it to/from disk.


Read scalability
================

In Schevo 2, we implemented an experimental feature wherein you could
have a master Schevo database, and several trusted replica Schevo
databases.

The master and replicas would all stay in sync, and the master would
control transaction execution order and act as a transaction repeater.
The net effect is that the master and replicas always had the same data
available locally, which would be quickly read from, and reliably
written to, with a fairly simple algorithm running the show.

By re-implementing something like this for Schevo 3, perhaps eventually
using some sort of mesh or multi-master topology for hot failover, we
gain the ability to scale up the capacity of databases that are
primarily read-driven, and have much less frequent writes.


Transactional scalability
=========================

For a database that is read-intensive but light on writes, one solution
that has been on the "mental to-do list" table for some time, but has
not yet received attention, is inter-database references.

That is, a database could be designed so that a new "external entity"
field could reference an instance in another database that was opened
and associated with the referring database.

Transactions would be able to cross database boundaries, but would only
result in a write-lock being obtained on databases that the transaction
is actively attempting to change.

Thus, if designed correctly, a set of databases using this approach can
scale with respect to transaction execution, since it would become
possible to simultaneously write to two databases at the same time.

With this, we must take care to avoid dead-locks, such as where two
databases mutually refer to eachother, and two transactions are
initiated, each starting in one database but then performing
subtransactions in the other.

I think we could tackle this though, such as by having transactions
pre-specify what databases other than the initial one they must have
write access to, and denying them write access if that isn't done.
That's just one example, there could be other approaches to try.


Schema complexity scalability
=============================

Schevo currently stores the database's schema inside the database, so
that there is no chance of using the wrong schema when opening a database.

Recently, we've been experimenting with including "standard" pieces of
schemata within another, but this introduces a conundrum wherein a
database doesn't have all necessary bits of source code for the schema
explicitly included.

To solve this, Schevo needs a way of specifying two things within a
schema:  1) any other schemata that it wishes to include/depend on, 2)
any Python packages (as per pkg_resources/setuptools) that a schema
depends on.

By doing this, Schevo will be able to introspect schema modules and
their source, store source code for all schemata used by a database, and
perform sufficient magic to correctly instantiate that source code as
unique modules when a database is opened.

Another item that will help with Schema complexity is of course
inter-database references as described above in `transactional
scalability`_.