feat(admin): IP2Proxy/IP2Location dataset refresh + freshness readout (BUNYIP-474) #470

Merged

longjacksonle merged 1 commit from feat/BUNYIP-474-dataset-age-and-refresh into main

2026-08-04 21:18:55 +02:00

longjacksonle commented

2026-08-04 21:14:10 +02:00

Owner

Closes BUNYIP-474 (BUNYIP-437 follow-up): automate the offline IP dataset refresh and make its age observable. Approach confirmed with the team lead: script + scheduled cron (not a compose sidecar).

The datasets are IP2Location LITE .BIN files bunyip reads via IP2LOCATION_DB_PATH (login country) and IP2PROXY_DB_PATH (ASN/VPN enrichment). The ip2location library never fetches them, so the deployment owns keeping them fresh.

Refresh (script + cron)

scripts/refresh-ip2-datasets.sh downloads and installs both LITE .BIN files from IP2Location using a free download token:

Stages each file in the destination directory and renames it into place, so a partial download never replaces a good file and the api never reads a half-written .BIN.
Exits non-zero if any dataset fails, so a scheduler surfaces the failure.
No long-running component, no committed secret; the token comes from the deployment's secret store.
Uses no force flags (per repo safety rule): atomic rename, curl --fail, recursive-not-forced temp cleanup.

docs/ip2-dataset-refresh.md documents the token, the env wiring, and a monthly cron / systemd-timer / Kubernetes CronJob example.

Observability

GET /v1/admin/health now returns a datasets block. For each file it reads the on-disk mtime and returns (configured, present, age_days, stale), distinguishing three real states: "operator did not deploy it", "path set but file missing", and "deployed but overdue" (stale past 40 days, a week of grace over the monthly cadence). The admin dashboard renders a Datasets card with a per-file age and a Fresh / Stale / Missing / Not configured badge, so a missed refresh is visible without reading logs. The card degrades to absent if the health call fails and never blocks the dashboard. Screenshot below.

Acceptance criteria

Dataset refresh automated: scripts/refresh-ip2-datasets.sh + documented monthly schedule.
Age observable to an admin: the dashboard Datasets card (age + staleness badge), backed by the /admin/health datasets block.

Tests / verification

API: the staleness helper (None / fresh / at-threshold / over-threshold) and the unconfigured / configured-but-missing dataset_health states.
Web: the card's four freshness states and singular/plural age wording.
just check-container green (fmt, clippy -D warnings, 182 tests). scripts/refresh-ip2-datasets.sh passes sh -n. Screenshot of the card:

Notes

The .BIN is opened at api startup, so a refresh is picked up on the next restart/deploy; the enrichment/geoip lookups always read the file that was open at boot. This is documented in the runbook.

🤖 Generated with Claude Code

https://claude.ai/code/session_018TXaT3P192nDsZzbzHETb9

Closes [BUNYIP-474](https://niceguyit.myjetbrains.com/youtrack/issue/BUNYIP-474) (BUNYIP-437 follow-up): automate the offline IP dataset refresh and make its age observable. Approach confirmed with the team lead: script + scheduled cron (not a compose sidecar). The datasets are IP2Location LITE `.BIN` files bunyip reads via `IP2LOCATION_DB_PATH` (login country) and `IP2PROXY_DB_PATH` (ASN/VPN enrichment). The `ip2location` library never fetches them, so the deployment owns keeping them fresh. ## Refresh (script + cron) `scripts/refresh-ip2-datasets.sh` downloads and installs both LITE `.BIN` files from IP2Location using a free download token: - Stages each file in the destination directory and renames it into place, so a partial download never replaces a good file and the api never reads a half-written `.BIN`. - Exits non-zero if any dataset fails, so a scheduler surfaces the failure. - No long-running component, no committed secret; the token comes from the deployment's secret store. - Uses no force flags (per repo safety rule): atomic rename, `curl --fail`, recursive-not-forced temp cleanup. `docs/ip2-dataset-refresh.md` documents the token, the env wiring, and a monthly cron / systemd-timer / Kubernetes CronJob example. ## Observability `GET /v1/admin/health` now returns a `datasets` block. For each file it reads the on-disk mtime and returns `(configured, present, age_days, stale)`, distinguishing three real states: "operator did not deploy it", "path set but file missing", and "deployed but overdue" (stale past 40 days, a week of grace over the monthly cadence). The admin **dashboard** renders a **Datasets** card with a per-file age and a Fresh / Stale / Missing / Not configured badge, so a missed refresh is visible without reading logs. The card degrades to absent if the health call fails and never blocks the dashboard. Screenshot below. ## Acceptance criteria - Dataset refresh automated: `scripts/refresh-ip2-datasets.sh` + documented monthly schedule. - Age observable to an admin: the dashboard Datasets card (age + staleness badge), backed by the `/admin/health` datasets block. ## Tests / verification - API: the staleness helper (None / fresh / at-threshold / over-threshold) and the unconfigured / configured-but-missing `dataset_health` states. - Web: the card's four freshness states and singular/plural age wording. - `just check-container` green (fmt, clippy `-D warnings`, 182 tests). `scripts/refresh-ip2-datasets.sh` passes `sh -n`. Screenshot of the card: ## Notes The `.BIN` is opened at api startup, so a refresh is picked up on the next restart/deploy; the enrichment/geoip lookups always read the file that was open at boot. This is documented in the runbook. 🤖 Generated with [Claude Code](https://claude.com/claude-code) https://claude.ai/code/session_018TXaT3P192nDsZzbzHETb9

longjacksonle added 1 commit

2026-08-04 21:14:10 +02:00

feat(admin): IP2Proxy/IP2Location dataset refresh + freshness readout (BUNYIP-474)

E2E PR gate / Install + reachability (no deployment secrets) (pull_request) Successful in 29s

Details

Check / fmt + clippy + build + tests (pull_request) Successful in 4m42s

Details

Create release / Create release from merged PR (pull_request) Has been skipped

Details

401fcb6de0

Closes the one BUNYIP-437 acceptance criterion left deferred: automate the offline IP dataset refresh and make its age observable. The datasets are the IP2Location LITE .BIN files bunyip reads via IP2LOCATION_DB_PATH (login country) and IP2PROXY_DB_PATH (ASN/VPN enrichment); the ip2location library never fetches them, so the deployment owns keeping them fresh.

Refresh (script + scheduled cron): scripts/refresh-ip2-datasets.sh downloads and installs both LITE .BIN files from IP2Location using a free download token, staging each in the destination directory and renaming it into place so a partial download never replaces a good file and the api never reads a half-written .BIN. It exits non-zero if any dataset fails so a scheduler surfaces it. docs/ip2-dataset-refresh.md documents the token, the env wiring, and a monthly cron / systemd-timer / CronJob example. It ships no long-running component and no secret; the token comes from the deployment's secret store.

Observability: GET /v1/admin/health now reports a datasets block. For each file it reads the on-disk mtime and returns (configured, present, age_days, stale), distinguishing "operator did not deploy it", "path set but file missing", and "deployed but overdue" (stale past 40 days, a week of grace over the monthly cadence). The admin dashboard renders a "Datasets" card with a per-file age and a Fresh / Stale / Missing / Not configured badge, so a missed refresh is visible without reading logs. The card degrades to absent if the health call fails and never blocks the dashboard.

Tests: the staleness helper (None/fresh/at-threshold/over-threshold) and the unconfigured/missing dataset_health states on the api side; the card's four freshness states and singular/plural age wording on the web side. Verified with just check-container (fmt, clippy -D warnings, 182 tests) and a screenshot of the card; scripts/refresh-ip2-datasets.sh passes sh -n and uses no force flags (atomic rename, curl --fail).

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_018TXaT3P192nDsZzbzHETb9

longjacksonle scheduled this pull request to auto merge when all checks succeed

2026-08-04 21:16:39 +02:00