Failures¶

Any call may fail, wherever the callee runs. Myxo makes failures look the same in every layout, so that code handling them doesn't depend on where the services run.

The errors of a call¶

All four derive from myxo.ServiceError, which gives each a status. A service raises the first three on purpose. The runtime raises RemoteError for any other exception, and logs the traceback where it happened. One handler in the guestbook's web service turns every error into an HTTP answer:

def error(_request: Request, exc: ServiceError) -> Response:
    body = {"error": type(exc).__name__, "detail": str(exc)}
    return Response(body, status_code=exc.status)

Retrying¶

The runtime can't know which operations are safe to repeat, so it never retries a call by itself. You choose where:

from myxo import retry

entry = await retry(self.book.get, EntryRef(1))

retry makes the call again with a backoff while it raises Unavailable. It tries 5 times by default, with jitter, and delays that double from 0.5 s up to 10 s. With attempts=None, it retries until cancelled.

Inside a process: services restart¶

Each service, consumer, worker and listener runs under a supervisor, which restarts it on any exception, with a fresh instance of the service. The delay starts at 0.5 s and doubles to 30 s. A run longer than 60 s resets it. After more than 5 failures within 60 s, the process gives up and exits with status 1, for whatever runs it to restart. The [supervisor] section of the layout sets these numbers.

A restart is one for one: the others keep running, and calls to the restarting service raise Unavailable meanwhile.

Around a process: the platform restarts it¶

The measurements come from E10: a killed worker, and the time until it finished a new job. Each is a single run.

All of them stop a process with SIGTERM. The process ends long polls and streams, stops the services, and exits with status 0.

Readiness¶

Every process with a url serves two paths on the listener:

• /healthz answers 200 while the process listens.
• /readyz answers 200 once the process's services run and pass their check(), or 503 with the list of what is wrong. An empty list means ready.

A service's check() covers what it owns, such as a database. The processes it calls are left out on purpose: behind services that route only to ready pods, two processes that call each other would wait for each other forever. A caller handles an unavailable callee as it handles any failed call.

Events and jobs¶

Work that outlives a call has separate guarantees:

• A consumer of events sees each event at least once. Unavailable brings the event back after a backoff. Any other error stops the consumer, which the supervisor restarts.
• A job runs at least once. A job that raises is retried with a backoff, up to the attempts limit, then kept and marked failed, with the error.
• A schedule runs at most once per period. A period no process led is skipped. A failed run is not retried.

So a handler or a job must be harmless to repeat, and work that must happen goes in a job.