Supply pipeline — three-algorithm derivation, per-asset refresh

Every supply value on `/v1/assets/{id}` flows through one path, parameterised by one of three algorithms keyed on the asset class:

operator config: [supply] sdf_reserve_accounts /
                          watched_classic_assets /
                          watched_sep41_contracts /
                          sac_wrappers
                                │
                                ▼
                     one supply.Refresher per asset
                                │
                                ▼
                  (Algorithm 1)  (Algorithm 2)  (Algorithm 3)
                  XLMComputer    ClassicComputer SEP41Computer
                  ▼              ▼               ▼
                  (reads)        (reads)         (reads)
   ┌──────────────────────┐  ┌────────────────┐ ┌─────────────────┐
   │ account_observations │  │ trustline_obs  │ │ sep41_supply_   │
   │  (XLM balances of    │  │ claimable_obs  │ │  events         │
   │   SDF reserves)      │  │ lp_reserve_obs │ │  (mint / burn / │
   │                      │  │ sac_balance_obs│ │   clawback)     │
   └──────────────────────┘  └────────────────┘ └─────────────────┘
                                │
                                ▼
                     supply.Supply struct
                                │
                                ▼
                     Store.InsertSupply
                                │
                                ▼
                  asset_supply_history (hypertable)
                                │
                                ▼
                     Store.LatestSupply
                                │
                                ▼
                     /v1/assets/{id} F2 fields:
                       total_supply
                       circulating_supply
                       max_supply
                       market_cap_usd  (× current price)
                       fdv_usd         (× current price)
                       supply_basis

The three algorithms (per ADR-0011)

Circulating (per ADR-0011) is total − issuer/admin balance − Σ operator-locked-set balances for all three. The locked-set is operator-curated via supply.Policy.PerAsset.

Max supply is total for hard-capped assets (XLM), nil otherwise unless the operator supplies an override or a SEP-1 declaration overlay populates it.

The six observers

Every component the algorithms read is sourced from one of six LCM-stream observers. Each plugs into the dispatcher's hooks without changing dispatcher source — they're pure additive sources per the ingest-pipeline contract:

The first five are LCM ledger-entry observers (ADR-0021 + ADR-0022). The sixth is an event-stream observer (ADR-0023) — it classifies topics and accumulates amounts rather than reading state.

All six observers are now wired into the indexer's dispatcher (L2.12a closed via PRs #411 / #412 / #413). Registration is opt-in per the corresponding [supply] watched-set — pipeline.RegisterSupplyEntryDecoders handles the five LedgerEntryChangeDecoders (accounts / trustlines / claimable_balances / liquidity_pools / sac_balances) keyed off sdf_reserve_accounts / watched_classic_assets / [supply.sac_wrappers], and pipeline.RegisterSupplyEventDecoders attaches sep41_supply when watched_sep41_contracts is non-empty. Empty watched-set → observer skipped → no behaviour change. With any watched-set populated, the corresponding hypertable starts filling on every matching ledger close.

The chained-fallback reader pattern

Per ADR-0021, the supply readers compose a "live LCM-derived reader" with an "operator-static fallback" so the system works during observer bootstrap:

supply.Refresher.Tick()
    │
    ▼
supply.<Algorithm>Computer.Compute(asset, ledger, observedAt)
    │
    ▼
supply.<Algorithm>SupplyReader.Read(asset, locked, ledger)
    │
    ▼
chain reader:
    1. try live: query account_observations / trustline_observations / etc.
    2. on ErrNoObservation: fall through to operator-static config
       (reserve_balances_stroops / per-asset locked-set / etc.)
    3. otherwise: bubble error

For Algorithm 1 (XLM) specifically: supplyAggregatorChainReader in cmd/stellarindex-aggregator/main.go wraps supply.LCMReserveBalanceReader (live) with supply.ConfigReserveBalanceReader (static). When the AccountEntry observer hasn't backfilled the SDF reserves yet, the static config produces the answer; once the observer covers the live set, the static config can be left stale (the live reader wins).

For Algorithms 2 + 3: similar pattern, but the static fallback is per-component (operators populate reserve_balances_stroops for XLM analogues; they DON'T typically maintain manual trustline-component snapshots, so the classic / SEP-41 paths require the observer to be backfilled).

Two refresh paths (operator choice)

Per ADR-0011 / ADR-0021 / Task #57, operators have two paths to write asset_supply_history rows:

A. systemd-timer driven

stellarindex-ops supply snapshot subcommand, fired by deploy/systemd/supply-snapshot.timer daily at 04:42 UTC. Per supply-snapshot runbook.

XLM only at v1; the CLI rejects classic + SEP-41 with a "use the goroutine path" message.

Metrics: stellarindex_supply_snapshot_* textfile-emitted via internal/supply/textfile.go. Alerts in deploy/monitoring/rules/supply-snapshot.yml.

B. Aggregator-resident goroutine

[supply] aggregator_refresh_enabled = true runs a supply.Refresher goroutine per watched asset inside stellarindex-aggregator. One goroutine per (XLM | classic asset | SEP-41 contract) on the aggregator_refresh_cadence (default 5m).

Covers all three algorithms. Per-tick outcome counter stellarindex_aggregator_supply_refresh_total{asset_key, outcome} labels by both asset and outcome so operators can chart per- asset bootstrap progress + isolate failure modes per asset. Alerts in deploy/monitoring/rules/supply-refresh.yml.

Choice rules

• Classic + SEP-41 supply requires path B (the CLI doesn't

support those assets).

• XLM supply works on either path. Path A is simpler (no

aggregator dependency); path B is preferred when the LCM observer has backfilled (per-cadence freshness vs. per-day).

The two paths are mutually exclusive at the operator level — write idempotency makes a double-fire correctness-safe (the hypertable's (asset_key, ledger_sequence) PK and ON CONFLICT DO NOTHING dedupe), but operators should disable one when flipping to the other to avoid redundant work.

Cross-check between Algorithm 2 and Algorithm 3

A SAC-wrapped classic asset's supply is observable two ways: as a classic credit (Algorithm 2 — sums trustline + claimable + LP-reserve + SAC-wrapped contract balances) and as a SEP-41 token (Algorithm 3 — sums mint − burn − clawback events on the SAC contract). Per ADR-0011 the two MUST agree within 1 stroop because both observe the same underlying ledger state through different lenses. Disagreement beyond the float-rounding tolerance signals indexer corruption upstream.

The aggregator's supply.CrossCheckRefresher (internal/supply/crosscheck_refresher.go, wired in cmd/stellarindex-aggregator/main.go::buildCrossCheckRefresher) ticks on the same aggregator_refresh_cadence as the per-asset supply refreshers above. Pairs are derived at boot from the ∩ of:

• [supply].sac_wrappers (operator-declared classic↔SAC mapping)
• [supply].watched_classic_assets (Algorithm 2 watched-set)
• [supply].watched_sep41_contracts (Algorithm 3 watched-set)

Per tick, for each pair the refresher reads the latest snapshot for both the classic and the SAC sides via Store.LatestSupply, runs supply.CrossCheck, and emits:

• stellarindex_supply_cross_check_divergence_stroops{classic_key} —

gauge holding the absolute stroop divergence on within/over outcomes.

• stellarindex_supply_cross_check_total{outcome} — counter labelled

by within | over | missing_snapshot | read_error.

The supply.yml alert (stellarindex_supply_cross_check_divergence) fires when the gauge stays > 1 for ≥ 5 min. Runbook: `supply-cross-check-divergence`.

Empty pair-set is a no-op — operators that haven't declared any SAC-wrapper pairs (e.g. an SEP-41-only deployment with no classic side) get no gauge updates and no alerting noise.

Per-class storage tables (live-data side)

All hypertables on observed_at, 7-day chunks, compression segment-by the most-common reader-query column. PK convention drags observed_at into the key per Timescale's partition- column-in-PK rule.

Reader contracts

Each algorithm has a <X>SupplyReader interface in internal/supply/; the production impl is Storage<X>SupplyReader composing the storage primitives:

Each reader returns a <X>SupplyComponents struct that the matching <X>Computer reduces to a Supply snapshot.

API surface

/v1/assets/{id} reads from asset_supply_history via Store.LatestSupply; the F2 fields (total_supply / circulating_supply / max_supply / market_cap_usd / fdv_usd / supply_basis) populate when a row exists, stay JSON null when no snapshot has been written (per ADR-0011 "we don't fabricate"). The handler does NOT consult observer state directly — the snapshot table is the API source of truth.

Failure modes (per outcome label)

The aggregator-refresh metric labels each tick with one of:

Sustained non-ok for ≥ 30 min triggers stellarindex_aggregator_supply_refresh_error_dominant; no ok in ≥ 30 min triggers _stalled.

The cross-check refresher emits its own per-outcome counter:

Bootstrap-state (missing_snapshot) is intentionally NOT escalated — it's the normal state during first-tick warmup and the first moments after a new operator-watched asset is added. Sustained read_error would surface via the same storage-layer alerts the per-asset refreshers ride.

ADR map

• ADR-0011 — three-algorithm

spec

• ADR-0021 —

AccountEntry observer for live home-domain + reserve balances

• ADR-0022 —

Trustline / Claimable / LP / SAC observers

• ADR-0023 — SEP-41

supply event observer

• ADR-0003 — i128 / NUMERIC

end-to-end (every amount in this pipeline)

• ADR-0006 —

TimescaleDB storage, the hypertable convention

• ADR-0015 —

why the API serves CLOSED snapshots only

Repo map

internal/sources/{accounts,trustlines,claimable_balances,liquidity_pools,sac_balances,sep41_supply}/
        ↓ (LedgerEntryChange or events.Event hooks)
internal/dispatcher/             (4 hooks: Decoder, OpDecoder, ContractCallDecoder, LedgerEntryChangeDecoder)
        ↓ (consumer.Event)
internal/pipeline/sink.go        (type-switch routing)
        ↓
internal/storage/timescale/      (Insert{Supply, AccountObservation, TrustlineObservation, …}, Sum*, Latest*)
        ↓
internal/supply/                 (XLMComputer, ClassicComputer, SEP41Computer, Refresher, CrossCheckRefresher, chained readers)
        ↓
cmd/stellarindex-aggregator/      (buildSupplyRefreshers + buildCrossCheckRefresher; runSupplyRefresh + runCrossCheckRefresh — one goroutine per asset, plus one for cross-check)
        ↓
internal/api/v1/assets_f2.go     (populateMarketCap, F2 field rendering)
        ↓
GET /v1/assets/{id}              (asset_supply_history via Store.LatestSupply)

When to update this doc

Add a row, update a table, or extend the diagram when:

• A new algorithm class lands (no current candidates; the

three above cover all on-chain Stellar supply types).

• A new observer plugs in (e.g. operator-watched-set expansion

to issuer accounts triggering SEP-1 metadata refresh).

• A new operator-config knob materially changes the data flow.
• An ADR in the ADR map above supersedes another.

The matrix in coverage-matrix.md is the row-by-row tracker; this doc is the architecture-level overview.