Phase 2 saga engine: compensation-based runner + step contract
The architectural spine of every write workflow phase 2+ — provisioning,
suspension, offboarding, updates, rollback — all ride this engine. Each
becomes a step list, not new orchestration code.
Schemas:
- saga_runs — kind, status (pending/running/completed/failed/
compensating/rolled_back), step_modules, current
step idx, accumulating context (jsonb), cancel
flag, error.
- saga_step_results — per-step audit ledger: status (running/completed/
failed/compensated), output, attempts, timings,
unique (saga_id, step_idx).
- cloud_provisioned — desired-state for resources WE provisioned. spec
+ spec_version + saga_id; FK to cloud_resources.
Phase 2's drift-detection diff lands here.
Step contract (ArcadiaCloud.Provisioning.Step):
- execute(state) -> {:ok, state} | {:error, reason}
- compensate(state) -> :ok | {:error, _} (optional)
- name() -> String.t()
SagaState carries the live execution state — accumulating context,
immutable inputs, current step_idx. Helpers: get_output/put_output
(context r/w), get_input (inputs read-only).
Runner (Oban worker, queue: provisioning, max_attempts: 1):
- kick_off: pending -> running, run_step(0)
- run_step: idempotent re-entry on saga.current_step_idx; persists step
result + saga context after each step; recursive forward walk through
the whole step list within one perform/1 call.
- safe_execute: try/rescue/catch around module.execute so a raised
exception triggers compensation rather than blowing up the worker.
- start_compensation: status=compensating, walk from idx-1 down to 0,
calling compensate/1 where it's exported; logs but doesn't halt on
compensate failure (best-effort + audit log).
- cancellation: checked between steps; cancel_requested=true -> trigger
compensation from current idx.
- crash recovery: max_attempts: 1 + run_step keyed on
saga.current_step_idx means Oban requeue picks up at the right place,
but full crash-resume infra is deferred to phase 2.5 (manual re-enqueue
works for now).
Two proof-of-concept steps (Steps.Echo, Steps.Fail) demonstrate the
engine without any DO API exposure. First real DO write step lands in
the next chunk.
Provisioning context provides start_saga/1, list_sagas/1,
list_step_results/1, cancel_saga/1, upsert_step_result/3.
Live smoke verified end-to-end:
- [Echo, Echo, Echo] happy path: all 3 completed, context accumulated
echoed_at_step_0/1/2 = "hello".
- [Echo, Echo, Fail] failure path: step 2 failed, compensation walked
back through step 1 then step 0; final status rolled_back with error
{compensate_from_idx: 1, reason: "step_failed:fail"}; ledger shows
echo/echo/fail with statuses compensated/compensated/failed.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
This commit is contained in:
29
lib/arcadia_cloud/provisioning/step.ex
Normal file
29
lib/arcadia_cloud/provisioning/step.ex
Normal file
@@ -0,0 +1,29 @@
|
||||
defmodule ArcadiaCloud.Provisioning.Step do
|
||||
@moduledoc """
|
||||
Contract every saga step module implements.
|
||||
|
||||
Steps MUST be:
|
||||
1. Idempotent — re-running produces the same effect; check context
|
||||
for prior outputs (`SagaState.get_output/2`) before doing work.
|
||||
2. Compensable — has an undo or explicitly declares it doesn't need
|
||||
one (compensate is optional; default = noop).
|
||||
3. Self-describing — writes its result into the saga context via
|
||||
`SagaState.put_output/3` so later steps + compensation can find
|
||||
what to act on.
|
||||
|
||||
Failure modes:
|
||||
{:ok, %SagaState{}} — step succeeded, advance.
|
||||
{:error, reason} — step failed, runner triggers compensation
|
||||
of all completed steps in reverse order.
|
||||
"""
|
||||
|
||||
alias ArcadiaCloud.Provisioning.SagaState
|
||||
|
||||
@callback execute(SagaState.t()) :: {:ok, SagaState.t()} | {:error, term()}
|
||||
|
||||
@callback compensate(SagaState.t()) :: :ok | {:error, term()}
|
||||
|
||||
@callback name() :: String.t()
|
||||
|
||||
@optional_callbacks [compensate: 1]
|
||||
end
|
||||
Reference in New Issue
Block a user