Durable task state machine scheduler (REQ-FSM-01..REQ-FSM-05, REQ-OBAN-01..REQ-OBAN-02).
Uses Fsmx inside Ecto.Multi transactions to enforce valid state transitions:
pending->running|aborted|cancelled|orphaned|failedrunning->waiting(all_settled|fail_fast) |sleeping(sleep_until) |completing|completed|failed|aborted- Open-time recovery (
reconcile_on_open!/1): resets survivingrunningtasks topending, incrementsattempt, cancels tasks withabandon_on_restart: true, and fails tasks that exceedrestart_limit(default5). - Bottom-up subtree abort (
abort_task!/1): marks the target task and its foreground-owned childrenabort_requested: true, settling children before completing the parent's abort. - A parent in
waitingthat a settled child wakes (pending) is announced to its conversation's machine, on any node, once the transaction has committed ({:task_woken, task_id}). A machine that is gone finds the woken task when it next opens.
Summary
Functions
Bottom-up subtree abort (spec.md §5.4):
Marks the target task and all non-background owned descendants abort_requested: true.
Settles leaf descendants as "aborted" first, then settles the parent task.
Cancels a "running" or "sleeping" task with result (a user cancel). Live children are
aborted and a waiting parent is reconciled, as for fail_task/2. A sleeping task's wake job
finds it no longer sleeping and does nothing.
Checkpoints a running task to a new phase ({phase: next_phase, ...}).
Completes a task with result.
Creates a new durable task in "pending" status.
Rejects creation if owner_task_id is already completing, terminal, or abort_requested.
Fails a task with error_map and triggers fail_fast or all_settled reconciliation on its parent.
The task as stored, or nil.
The tasks with these ids, in id order.
The foreground tasks that task_id owns and that have not settled.
The oldest pending pi.generation task of the conversation, or nil. A machine that starts
after a crash or a failover resumes it (see Machine, :reconcile).
The oldest pi.generation task of the conversation that waits on its children (waiting), or
nil. A machine that starts while a turn is parked resumes the park (see Machine, :reconcile).
Open-time reconciliation (spec.md §2.5, §5.4, REQ-FSM-04, REQ-FSM-05)
Transitions a running task to "sleeping" until wake_at (DateTime.t()) and schedules an
Oban wake job (REQ-OBAN-01).
Atomically transitions a task to "running". The task row is locked for the
read and the write, so two workers cannot both claim it: the loser gets
{:error, :already_running}.
Performs an atomic Fsmx transition under a row lock (REQ-FSM-01, REQ-FSM-02).
Returns {:ok, task}, {:error, :not_found} or {:error, changeset}.
Transitions a running task to "waiting" on child_task_ids with policy "all_settled" or "fail_fast".
Wakes a "sleeping" task back to "pending" when its deadline arrives.
Functions
Bottom-up subtree abort (spec.md §5.4):
Marks the target task and all non-background owned descendants abort_requested: true.
Settles leaf descendants as "aborted" first, then settles the parent task.
Cancels a "running" or "sleeping" task with result (a user cancel). Live children are
aborted and a waiting parent is reconciled, as for fail_task/2. A sleeping task's wake job
finds it no longer sleeping and does nothing.
Checkpoints a running task to a new phase ({phase: next_phase, ...}).
Completes a task with result.
- If the task still has live foreground owned child tasks, transitions to
"completing"and holdsresult. - Otherwise transitions to
"completed"and reconciles any parent task waiting on it.
Creates a new durable task in "pending" status.
Rejects creation if owner_task_id is already completing, terminal, or abort_requested.
Fails a task with error_map and triggers fail_fast or all_settled reconciliation on its parent.
The task as stored, or nil.
The tasks with these ids, in id order.
The foreground tasks that task_id owns and that have not settled.
The oldest pending pi.generation task of the conversation, or nil. A machine that starts
after a crash or a failover resumes it (see Machine, :reconcile).
The oldest pi.generation task of the conversation that waits on its children (waiting), or
nil. A machine that starts while a turn is parked resumes the park (see Machine, :reconcile).
Open-time reconciliation (spec.md §2.5, §5.4, REQ-FSM-04, REQ-FSM-05):
- Resets surviving
"running"tasks forconversation_idto"pending"and incrementsattempt. - If
abandon_on_restart: true, settles the task as"cancelled". - If
attempt > restart_limit, settles the task as"failed"with"restart_limit_exceeded".
Transitions a running task to "sleeping" until wake_at (DateTime.t()) and schedules an
Oban wake job (REQ-OBAN-01).
Atomically transitions a task to "running". The task row is locked for the
read and the write, so two workers cannot both claim it: the loser gets
{:error, :already_running}.
Performs an atomic Fsmx transition under a row lock (REQ-FSM-01, REQ-FSM-02).
Returns {:ok, task}, {:error, :not_found} or {:error, changeset}.
Transitions a running task to "waiting" on child_task_ids with policy "all_settled" or "fail_fast".
The parent row is locked while the children are read and the parent is
written. A child that settles concurrently takes the same lock in
reconcile_parent_after_child_settled!/2, so it either sees "waiting" and
wakes the parent, or commits first and this call sees it settled. No lost wakeup.
Options: :phase and :state are checkpointed together with the move to "waiting". The
task is no longer running afterwards, so a later checkpoint_task/3 would be refused.
Wakes a "sleeping" task back to "pending" when its deadline arrives.