Phase 2 saga engine: compensation-based runner + step contract

The architectural spine of every write workflow phase 2+ — provisioning,
suspension, offboarding, updates, rollback — all ride this engine. Each
becomes a step list, not new orchestration code.

Schemas:
- saga_runs           — kind, status (pending/running/completed/failed/
                        compensating/rolled_back), step_modules, current
                        step idx, accumulating context (jsonb), cancel
                        flag, error.
- saga_step_results   — per-step audit ledger: status (running/completed/
                        failed/compensated), output, attempts, timings,
                        unique (saga_id, step_idx).
- cloud_provisioned   — desired-state for resources WE provisioned. spec
                        + spec_version + saga_id; FK to cloud_resources.
                        Phase 2's drift-detection diff lands here.

Step contract (ArcadiaCloud.Provisioning.Step):
- execute(state) -> {:ok, state} | {:error, reason}
- compensate(state) -> :ok | {:error, _}   (optional)
- name() -> String.t()

SagaState carries the live execution state — accumulating context,
immutable inputs, current step_idx. Helpers: get_output/put_output
(context r/w), get_input (inputs read-only).

Runner (Oban worker, queue: provisioning, max_attempts: 1):
- kick_off: pending -> running, run_step(0)
- run_step: idempotent re-entry on saga.current_step_idx; persists step
  result + saga context after each step; recursive forward walk through
  the whole step list within one perform/1 call.
- safe_execute: try/rescue/catch around module.execute so a raised
  exception triggers compensation rather than blowing up the worker.
- start_compensation: status=compensating, walk from idx-1 down to 0,
  calling compensate/1 where it's exported; logs but doesn't halt on
  compensate failure (best-effort + audit log).
- cancellation: checked between steps; cancel_requested=true -> trigger
  compensation from current idx.
- crash recovery: max_attempts: 1 + run_step keyed on
  saga.current_step_idx means Oban requeue picks up at the right place,
  but full crash-resume infra is deferred to phase 2.5 (manual re-enqueue
  works for now).

Two proof-of-concept steps (Steps.Echo, Steps.Fail) demonstrate the
engine without any DO API exposure. First real DO write step lands in
the next chunk.

Provisioning context provides start_saga/1, list_sagas/1,
list_step_results/1, cancel_saga/1, upsert_step_result/3.

Live smoke verified end-to-end:
- [Echo, Echo, Echo] happy path: all 3 completed, context accumulated
  echoed_at_step_0/1/2 = "hello".
- [Echo, Echo, Fail] failure path: step 2 failed, compensation walked
  back through step 1 then step 0; final status rolled_back with error
  {compensate_from_idx: 1, reason: "step_failed:fail"}; ledger shows
  echo/echo/fail with statuses compensated/compensated/failed.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

lib/arcadia_cloud/provisioning.ex

@@ -0,0 +1,112 @@
+defmodule ArcadiaCloud.Provisioning do
+  @moduledoc """
+  Context for saga orchestration — provisioning, suspension, offboarding,
+  updates, anything that wants compensation-based rollback over Oban.
+
+  Pattern: caller assembles a step list (per template or hand-rolled),
+  calls `start_saga/1`, the Runner Oban worker walks the steps,
+  persisting results and rolling back on failure.
+  """
+
+  import Ecto.Query, warn: false
+
+  alias ArcadiaCloud.Repo
+  alias ArcadiaCloud.Provisioning.{SagaRun, SagaStepResult}
+
+  @doc """
+  Inserts a saga_runs row + enqueues the Runner job.
+
+  Required:
+    :kind            — provision | suspend | offboard | update | rollback | test
+    :step_modules    — ordered list of step module atoms or fully-qualified strings
+    :inputs          — map of saga inputs (stored in context.__inputs__)
+
+  Optional:
+    :deployment_id   — links the saga to a deployment (nil for skyai-internal)
+    :triggered_by    — user_id or "system:<reason>"
+  """
+  def start_saga(opts) when is_list(opts) do
+    start_saga(Map.new(opts))
+  end
+
+  def start_saga(%{} = attrs) do
+    step_modules = Enum.map(attrs[:step_modules] || [], &to_string/1)
+    inputs = attrs[:inputs] || %{}
+
+    saga_attrs = %{
+      kind: attrs[:kind],
+      step_modules: step_modules,
+      deployment_id: attrs[:deployment_id],
+      triggered_by: attrs[:triggered_by],
+      context: %{"__inputs__" => inputs}
+    }
+
+    with {:ok, saga} <- create_saga(saga_attrs),
+         {:ok, _job} <-
+           %{"saga_id" => saga.id}
+           |> ArcadiaCloud.Provisioning.Runner.new()
+           |> Oban.insert() do
+      {:ok, saga}
+    end
+  end
+
+  def create_saga(attrs) do
+    %SagaRun{}
+    |> SagaRun.changeset(attrs)
+    |> Repo.insert()
+  end
+
+  def get_saga(id), do: Repo.get(SagaRun, id)
+  def get_saga!(id), do: Repo.get!(SagaRun, id)
+
+  def update_saga(%SagaRun{} = saga, attrs) do
+    saga
+    |> SagaRun.changeset(attrs)
+    |> Repo.update()
+  end
+
+  def list_sagas(opts \\ []) do
+    base = from(s in SagaRun, order_by: [desc: s.inserted_at])
+
+    base
+    |> maybe_filter(:status, opts[:status])
+    |> maybe_filter(:kind, opts[:kind])
+    |> maybe_filter(:deployment_id, opts[:deployment_id])
+    |> maybe_limit(opts[:limit])
+    |> Repo.all()
+  end
+
+  def list_step_results(saga_id) do
+    from(r in SagaStepResult,
+      where: r.saga_id == ^saga_id,
+      order_by: [asc: r.step_idx]
+    )
+    |> Repo.all()
+  end
+
+  def cancel_saga(%SagaRun{} = saga) do
+    saga
+    |> SagaRun.changeset(%{cancel_requested: true})
+    |> Repo.update()
+  end
+
+  def upsert_step_result(saga_id, step_idx, attrs) do
+    case Repo.get_by(SagaStepResult, saga_id: saga_id, step_idx: step_idx) do
+      nil ->
+        %SagaStepResult{}
+        |> SagaStepResult.changeset(Map.merge(attrs, %{saga_id: saga_id, step_idx: step_idx}))
+        |> Repo.insert()
+
+      existing ->
+        existing
+        |> SagaStepResult.changeset(attrs)
+        |> Repo.update()
+    end
+  end
+
+  defp maybe_filter(q, _f, nil), do: q
+  defp maybe_filter(q, field, value), do: from(s in q, where: field(s, ^field) == ^value)
+
+  defp maybe_limit(q, nil), do: q
+  defp maybe_limit(q, n), do: from(s in q, limit: ^n)
+end