Graceful shutdown of a node (REQ-CLU-07).
On SIGTERM the application calls drain/1 before it stops. The node
- stops accepting new work: the gateway answers
503 shutting_down(withRetry-After) andGET /readyreturns 503, so a load balancer or Kubernetes sends traffic elsewhere; RPC refusessession.openandsession.submit; - waits for the conversation machines on this node to finish their running turns, up to
config :super_pi, drain_timeout_ms(default 30 000); - stops. Idle machines under
Horde.DynamicSupervisorare restarted on the other nodes, and a turn that did not finish in time is resumed from the durable state by the machine that replaces it.
Summary
Functions
True while the node accepts new work.
Ids of the conversations on this node whose machine is in the middle of a turn.
Stops accepting work and waits until no local machine is busy. Returns :drained or
{:timeout, busy_conversation_ids}.
Accepts work again (used by tests, and after a cancelled shutdown).
Stops accepting new work.
Functions
True while the node accepts new work.
Ids of the conversations on this node whose machine is in the middle of a turn.
Stops accepting work and waits until no local machine is busy. Returns :drained or
{:timeout, busy_conversation_ids}.
Accepts work again (used by tests, and after a cancelled shutdown).
Stops accepting new work.