Skip to content

Operations

Migrations

myxo migrate LAYOUT creates the framework's tables and runs each service's migrate. Run it before any process starts: each export target does. It is safe to run again.

myxo_versions records the changes made to the framework's tables, so a later version of Myxo changes them in place. A database that a later version migrated is left as it is when an earlier one runs migrate, as at a rollback.

Myxo has no migration tool for applications. A service's migrate may call Alembic, or anything else that brings the service's storage up to date.

Upgrades

Messages change under one rule, so that a process of one version understands a process of the next:

  • a message gains a field only with a default;
  • a message keeps each field that a reader of the previous version needs, with the same type;
  • a class that moves or is renamed keeps the name it is stored under, with @myxo.topic.

The rule holds for calls, events and jobs, in both directions. A reader ignores a field it does not know, and gives a missing field the default. A message that breaks the rule fails to decode. A call then raises Invalid, or RemoteError for the answer. A job is retried, then kept failed. An event stops the consumer, as a handler's bug does (events).

The framework's tables change only through myxo migrate, in steps that the previous version still reads.

An upgrade whose migrations only add, such as a table, a column with a default or an index, can replace the processes one at a time while the installation runs:

  1. Install the new version beside the running one, so that the old processes keep their code.
  2. Run the new version's myxo migrate, while the old processes run.
  3. Replace each process in turn: stop it, start it from the new version, and wait until /readyz answers on it before the next.

During the rollout, old and new processes call each other and read each other's events and jobs, under the rule above. With one instance of a process, the processes that call it get Unavailable while it restarts, for about a second. A worker's restart costs nothing, since the events and jobs wait in the database. E15 ran this procedure three times on PostgreSQL, on one host, with jobs queued and events pending, and lost nothing. It has not been tried on Hop3 or Kubernetes.

An upgrade that drops or renames what the running version reads needs every process stopped: stop them, run the migrations, and start the new version. Or split it in two releases: the first stops reading the column, and the second drops it.

Configuration

The layout reads the environment, so secrets stay out of it. env.example, from each export, lists the variables it reads. A variable with no default, ${NAME}, stops the command with a message if it is unset.

Logs

Each process logs to stderr, as time pid logger level message. A service logs with Python's logging, through logging.getLogger(__name__), and those lines take the same form. A process's start shows the stages, which tell you what it runs:

INFO stage 0 ready: database
INFO stage 1 ready: notes, cache:purge
INFO stage 2 ready: events:notes, jobs:CountWords:0, jobs:CountWords:1, schedule:notes.tick
INFO stage 3 ready: events:prune
INFO stage 4 ready: rpc

A failure logs a traceback, then restarting NAME in 0.5s. Litestar sets up its own logging, so a process that runs it may print some lines in two formats.

Myxo has no tracing yet: OpenTelemetry is on the list of known limits.

Health

/healthz and /readyz answer on each process's listener. Query a Unix socket with curl:

$ curl -s --unix-socket run/web.sock localhost/readyz
[]

An empty list means ready. On 503, the list says what is wrong: book: database: ..., or book is not running.

Backups

An installation's state is in its database, by the rules for application code: your services' tables, the events, the jobs and the schedules. Back that up with your database's own tools. The shared cache can be left out.

On SQLite, the database runs in WAL mode, so take the backup with sqlite3 FILE ".backup ...", which sees the -wal file. The *.lock files beside it matter only while the processes run.

The framework's tables

myxo migrate creates only those the layout's services use:

Table Holds
myxo_events events, kept an hour once handled
myxo_groups each consumer group's position
myxo_jobs jobs waiting, running or failed
myxo_schedules the last period each schedule started
myxo_cache the shared cache
myxo_versions the changes made to the tables above, by the framework's part that owns them

Two conditions need a person:

  • Failed jobs stay until you act. See the jobs guide.
  • A consumer group that stopped for good holds events back until you remove its row. See the events guide.

Moving to PostgreSQL

Change DATABASE_URL, create the database and its user, run the migrations, and start. The installation starts empty: nothing moves the data from SQLite yet, so export and import your services' tables with your own tools if you need them.