Round out DO sync workers: volumes, snapshots, floating IPs, firewalls, LBs

Five new Oban workers, all on cloud_sync_full following the established
droplet/domains pattern (list → normalize → upsert → mark_stale):

- VolumesWorker         — block storage
- SnapshotsWorker       — both droplet and volume snapshots (kind="snapshot"
                          with attrs.resource_type to differentiate)
- FloatingIpsWorker     — provider_id is the IP; status assigned/unassigned
- FirewallsWorker       — inbound/outbound rules + droplet_ids in attrs
- LoadBalancersWorker   — name + region + algorithm + forwarding rules

DigitalOcean.Client gains list_snapshots / list_firewalls / list_load_balancers.

ProjectsWorker URN normalization extended: "floatingip" → floating_ip,
"loadbalancer" → load_balancer, "dbaas" → managed_db. URNs DO emits don't
have underscores for these.

Cron updated: new workers run every 15min on cloud_sync_full; snapshots
moved to hourly (at :33) since they change slowly and listing them is
the most-paginated call we make.

InvoiceIngestWorker.derive_kind/2 reordered to check specific phrases
before generic products — "Droplet Snapshots"/"Droplet Backups" no longer
get bucketed as kind=droplet ahead of the actual snapshot check. Also
adds kind="droplet_backup" for DO's automated backup billing (separate
from the snapshot kind because backups aren't exposed via /v2/snapshots).

Live verified: 12 snapshots discovered + 1 firewall (account has no
volumes / floating IPs / LBs at the moment, so those workers ran clean).
April 2026 invoice match rate jumped from 18.2% → 51.5%. Of the
unmatched: 10 historic droplets that no longer exist on DO, 2 backups
(separate API surface), 1 Spaces bucket (S3 API, deferred), 1 GST
(correctly no kind). Effectively ~95% of currently-extant resources match.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
This commit is contained in:
2026-05-19 22:41:12 +10:00
parent 4f2e52af01
commit 8bdf500214
9 changed files with 256 additions and 2 deletions

View File

@@ -0,0 +1,45 @@
defmodule ArcadiaCloud.Sync.FirewallsWorker do
@moduledoc "Full sync of DO cloud firewalls."
use Oban.Worker, queue: :cloud_sync_full, max_attempts: 3
alias ArcadiaCloud.Cloud
alias ArcadiaCloud.DigitalOcean.Client
@kind "firewall"
@provider "digitalocean"
@impl Oban.Worker
def perform(_job) do
now = DateTime.utc_now() |> DateTime.truncate(:second)
with {:ok, firewalls} <- Client.list_firewalls() do
Enum.each(firewalls, fn f ->
Cloud.upsert_resource(normalize(f, now))
end)
Cloud.mark_stale(@kind, now)
:ok
end
end
defp normalize(f, now) do
%{
provider: @provider,
provider_id: f["id"],
kind: @kind,
name: f["name"],
region: nil,
status: f["status"] || "succeeded",
tags: f["tags"] || [],
attrs: %{
inbound_rules: f["inbound_rules"],
outbound_rules: f["outbound_rules"],
droplet_ids: f["droplet_ids"],
pending_changes: f["pending_changes"],
do_created_at: f["created_at"]
},
first_seen_at: now,
last_seen_at: now
}
end
end