Giuliano Silvestro
8bdf500214
Five new Oban workers, all on cloud_sync_full following the established
droplet/domains pattern (list → normalize → upsert → mark_stale):
- VolumesWorker — block storage
- SnapshotsWorker — both droplet and volume snapshots (kind="snapshot"
with attrs.resource_type to differentiate)
- FloatingIpsWorker — provider_id is the IP; status assigned/unassigned
- FirewallsWorker — inbound/outbound rules + droplet_ids in attrs
- LoadBalancersWorker — name + region + algorithm + forwarding rules
DigitalOcean.Client gains list_snapshots / list_firewalls / list_load_balancers.
ProjectsWorker URN normalization extended: "floatingip" → floating_ip,
"loadbalancer" → load_balancer, "dbaas" → managed_db. URNs DO emits don't
have underscores for these.
Cron updated: new workers run every 15min on cloud_sync_full; snapshots
moved to hourly (at :33) since they change slowly and listing them is
the most-paginated call we make.
InvoiceIngestWorker.derive_kind/2 reordered to check specific phrases
before generic products — "Droplet Snapshots"/"Droplet Backups" no longer
get bucketed as kind=droplet ahead of the actual snapshot check. Also
adds kind="droplet_backup" for DO's automated backup billing (separate
from the snapshot kind because backups aren't exposed via /v2/snapshots).
Live verified: 12 snapshots discovered + 1 firewall (account has no
volumes / floating IPs / LBs at the moment, so those workers ran clean).
April 2026 invoice match rate jumped from 18.2% → 51.5%. Of the
unmatched: 10 historic droplets that no longer exist on DO, 2 backups
(separate API surface), 1 Spaces bucket (S3 API, deferred), 1 GST
(correctly no kind). Effectively ~95% of currently-extant resources match.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
57 lines
1.4 KiB
Elixir
57 lines
1.4 KiB
Elixir
defmodule ArcadiaCloud.Sync.SnapshotsWorker do
|
|
@moduledoc """
|
|
Full sync of DO snapshots — both droplet snapshots and volume snapshots.
|
|
DO returns them together with a `resource_type` field that we use to
|
|
differentiate; both land as kind="snapshot" with `attrs.resource_type`.
|
|
"""
|
|
use Oban.Worker, queue: :cloud_sync_full, max_attempts: 3
|
|
|
|
alias ArcadiaCloud.Cloud
|
|
alias ArcadiaCloud.DigitalOcean.Client
|
|
|
|
@kind "snapshot"
|
|
@provider "digitalocean"
|
|
|
|
@impl Oban.Worker
|
|
def perform(_job) do
|
|
now = DateTime.utc_now() |> DateTime.truncate(:second)
|
|
|
|
with {:ok, snapshots} <- Client.list_snapshots() do
|
|
Enum.each(snapshots, fn s ->
|
|
Cloud.upsert_resource(normalize(s, now))
|
|
end)
|
|
|
|
Cloud.mark_stale(@kind, now)
|
|
:ok
|
|
end
|
|
end
|
|
|
|
defp normalize(s, now) do
|
|
region =
|
|
case s["regions"] do
|
|
[first | _] when is_binary(first) -> first
|
|
_ -> nil
|
|
end
|
|
|
|
%{
|
|
provider: @provider,
|
|
provider_id: to_string(s["id"]),
|
|
kind: @kind,
|
|
name: s["name"],
|
|
region: region,
|
|
status: "active",
|
|
tags: s["tags"] || [],
|
|
attrs: %{
|
|
resource_type: s["resource_type"],
|
|
resource_id: s["resource_id"],
|
|
size_gigabytes: s["size_gigabytes"],
|
|
min_disk_size: s["min_disk_size"],
|
|
regions: s["regions"],
|
|
do_created_at: s["created_at"]
|
|
},
|
|
first_seen_at: now,
|
|
last_seen_at: now
|
|
}
|
|
end
|
|
end
|