Wire a full tenant deployment as one orchestrated, compensating saga:
mark → create droplet → wait active → register in inventory → link to
deployment → point DNS → activate. A failure anywhere rolls the whole
thing back — droplet destroyed, DNS reverted, deployment moved to
cancelled.
- New lifecycle state `provisioning`; deployments created via the
provision path enter here and only reach `active` once the saga's
ActivateDeployment step runs.
- Four new steps: MarkDeploymentProvisioning (owns the deployment's
failure state), LinkDeploymentResource, PointDeploymentDns,
ActivateDeployment.
- Provisioning.provision_deployment/2 assembles + starts the saga.
- DeploymentController: POST /deployments with provision:true creates
in `provisioning` and kicks the saga (202); GET /deployments/:id now
returns the provisioning saga + per-step progress.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
The most load-bearing write workflow — droplet provisioning is the spine
of phase 4a deployment onboarding.
DigitalOcean.Client: create_droplet, get_droplet, list_droplets_by_tag,
destroy_droplet. list_paginated/3 now threads caller-supplied params
(opts[:params]) through pagination so tag-filtered listing works.
Four droplet saga steps:
- CreateDroplet — POST a droplet, tagged arcadia-saga-<saga8> +
managed-by-arcadia-cloud. Idempotency: re-run checks context for
droplet_id, then queries DO by the saga tag, so a crash between POST
and context-save adopts the existing droplet. compensate destroys it.
- WaitDropletActive — polls get_droplet until status "active" (96x5s);
records the public IP. No compensation (waiting has no side effect).
- RegisterDroplet — fetches the droplet, upserts it into cloud_resources
(inventory consistent immediately, not at next 15-min sync) and writes
cloud_provisioned desired-state {size_slug, region, image}. compensate
removes the DB rows (the droplet itself is destroyed by CreateDroplet's
compensate).
- DestroyDroplet — DELETE the droplet + mark its cloud_resources row
deleted. Terminal/irreversible: compensate is a logged noop, per the
saga design destroy-class steps don't roll back.
Provisioning helpers:
- provision_droplet/1 — [CreateDroplet, WaitDropletActive, RegisterDroplet]
- destroy_droplet/2 — [DestroyDroplet]
Live smoke verified end-to-end (full create + destroy on a real
s-1vcpu-512mb-10gb droplet in syd1):
- provision saga completed: droplet 572017320 created, reached active
with public IP, registered into cloud_resources (status=active) +
cloud_provisioned (spec recorded).
- destroy saga completed: cloud_resources row marked deleted; droplet
confirmed 404 on DO afterward. Account back to its original 5
droplets, zero leftover, ~1 cent total cost.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
DigitalOcean.Client DNS record CRUD:
- list_domain_records/2, create_domain_record/3, update_domain_record/4,
delete_domain_record/3 on /v2/domains/:domain/records.
Steps.UpsertDnsRecord — idempotent ensure-record:
- (type, name) absent -> create
- present, data matches -> no-op
- present, data differs -> update, stashing prior state
Match comparison normalizes trailing dots (DO appends them to CNAMEs)
so a re-run doesn't false-positive into an update. compensate: deletes
records it created, restores prior data/ttl for records it updated.
Steps.DeleteDnsRecord — deletes a record by (type, name); idempotent
(already-absent succeeds); compensate recreates from stashed prior state.
Both read dns_domain / dns_record_type / dns_record_name / dns_record_data
/ dns_record_ttl from saga inputs.
Live smoke verified against sky-ai.com with TXT records (non-disruptive):
- [UpsertDnsRecord, Fail]: record created then compensation deleted it
(saga rolled_back, record absent after).
- [UpsertDnsRecord] twice: first outcome "created", second "noop"
(idempotency holds).
- [DeleteDnsRecord]: record deleted, absent after.
Both test records cleaned up — zero leftover.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
DigitalOcean.Client write methods:
- create_droplet_snapshot/3 — POST a snapshot action (async)
- get_droplet_action/3 — poll action status
- list_droplet_snapshots/2 — snapshots for a droplet
- delete_snapshot/2 — DELETE (used by compensation)
All use the "provisioning" token purpose.
Steps.CreateDropletSnapshot — the first saga step that touches real
infra:
- execute: deterministic snapshot name (arcadia-snap-<droplet>-<saga8>);
checks context for a prior snapshot_id, then checks DO for a snapshot
already carrying that name (crash-between-post-and-save recovery),
then posts the action, polls to completion, finds the resulting
snapshot, records snapshot_id + snapshot_name in context.
- compensate: deletes the snapshot; treats HTTP 404 as success.
Provisioning.snapshot_droplet/2 — convenience saga starter.
Two DO eventual-consistency gotchas surfaced + handled:
- After a snapshot action reports "completed", the snapshot lags a few
seconds before appearing in /droplets/:id/snapshots. The step now
retries the lookup (find_snapshot_with_retry, 12x5s) instead of
failing with :snapshot_not_found_after_completion.
- Deletion has the same lag the other way — a deleted snapshot lingers
in the listing briefly. compensate just trusts the DELETE 2xx/404;
no post-delete verification needed.
Live smoke verified end-to-end against holyspiritbraypark.com:
[CreateDropletSnapshot, Fail] saga — the step created real snapshot
229305609, the Fail step triggered compensation, compensation deleted
the snapshot. Final: saga rolled_back, ledger
[create_droplet_snapshot: compensated, fail: failed], zero leftover on DO.
Test-harness note: smoke tests create sagas via Provisioning.create_saga
(no Oban enqueue) so a single manual Runner.perform/1 owns execution —
start_saga/1 enqueues an Oban job, and running both racing the same saga
corrupts the step ledger. Production only ever runs via Oban.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
The architectural spine of every write workflow phase 2+ — provisioning,
suspension, offboarding, updates, rollback — all ride this engine. Each
becomes a step list, not new orchestration code.
Schemas:
- saga_runs — kind, status (pending/running/completed/failed/
compensating/rolled_back), step_modules, current
step idx, accumulating context (jsonb), cancel
flag, error.
- saga_step_results — per-step audit ledger: status (running/completed/
failed/compensated), output, attempts, timings,
unique (saga_id, step_idx).
- cloud_provisioned — desired-state for resources WE provisioned. spec
+ spec_version + saga_id; FK to cloud_resources.
Phase 2's drift-detection diff lands here.
Step contract (ArcadiaCloud.Provisioning.Step):
- execute(state) -> {:ok, state} | {:error, reason}
- compensate(state) -> :ok | {:error, _} (optional)
- name() -> String.t()
SagaState carries the live execution state — accumulating context,
immutable inputs, current step_idx. Helpers: get_output/put_output
(context r/w), get_input (inputs read-only).
Runner (Oban worker, queue: provisioning, max_attempts: 1):
- kick_off: pending -> running, run_step(0)
- run_step: idempotent re-entry on saga.current_step_idx; persists step
result + saga context after each step; recursive forward walk through
the whole step list within one perform/1 call.
- safe_execute: try/rescue/catch around module.execute so a raised
exception triggers compensation rather than blowing up the worker.
- start_compensation: status=compensating, walk from idx-1 down to 0,
calling compensate/1 where it's exported; logs but doesn't halt on
compensate failure (best-effort + audit log).
- cancellation: checked between steps; cancel_requested=true -> trigger
compensation from current idx.
- crash recovery: max_attempts: 1 + run_step keyed on
saga.current_step_idx means Oban requeue picks up at the right place,
but full crash-resume infra is deferred to phase 2.5 (manual re-enqueue
works for now).
Two proof-of-concept steps (Steps.Echo, Steps.Fail) demonstrate the
engine without any DO API exposure. First real DO write step lands in
the next chunk.
Provisioning context provides start_saga/1, list_sagas/1,
list_step_results/1, cancel_saga/1, upsert_step_result/3.
Live smoke verified end-to-end:
- [Echo, Echo, Echo] happy path: all 3 completed, context accumulated
echoed_at_step_0/1/2 = "hello".
- [Echo, Echo, Fail] failure path: step 2 failed, compensation walked
back through step 1 then step 0; final status rolled_back with error
{compensate_from_idx: 1, reason: "step_failed:fail"}; ledger shows
echo/echo/fail with statuses compensated/compensated/failed.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
