feat(data): tenant data import - envelope validation + id-remap core (PMS-648) #439

Merged

nrupard merged 2 commits from feat/PMS-648-data-import into main 2026-07-09 20:57:06 +02:00

Conversation 0 Commits 2 Files changed 2 +453 -1

nrupard commented 2026-07-09 20:40:18 +02:00

Owner

Copy link

Backend import for PMS-646, sibling of the PMS-647 export (#438). Built in two commits so each piece is verified at the level it can be:

Stacking

This branch is stacked on feat/PMS-647-data-export (#438); it reuses that module's exclude/redaction consts. Until #438 merges, this PR shows the export commits too; it auto-cleans to just the import diff once #438 lands. Merge #438 first.

Commit 1 (this) - validated, locally verified

POST /api/v1/data/import (admin-only): 25 MB body cap (413 over-size), validates schema_version, rejects any unexpected/excluded table, checks the tenant-name confirmation, and computes the global old-id -> new-id remap over every row (so a malformed file fails during validation). Returns a dry-run summary and performs NO mutation.

The security-relevant, order-independent core is pure and unit-tested (build_id_map, remap_row): fresh id per row, pk/FK/tenant rewrite, and unrelated UUIDs left untouched. just pre-commit green (271 tests).

Commit 2 (next, this branch) - DB load, CI-verified

The destructive wipe-and-replace load: derive the FK-dependency topological order from the catalog, wipe the tenant's in-scope tables, load the remapped rows in dependency order (with self-referential-FK and dangling user-reference handling), emit the import audit action, and return the per-table summary - all in one transaction. This can only be exercised against Postgres, so it is validated by a round-trip integration test (tests/*.rs, run by CI integration.yml): export a seeded tenant (PMS-647) -> import -> assert structure matches and no secret substrings appear in the exported bytes.

#PMS-648

Backend import for PMS-646, sibling of the PMS-647 export (#438). Built in two commits so each piece is verified at the level it can be: ## Stacking This branch is stacked on `feat/PMS-647-data-export` (#438); it reuses that module's exclude/redaction consts. Until #438 merges, this PR shows the export commits too; it auto-cleans to just the import diff once #438 lands. **Merge #438 first.** ## Commit 1 (this) - validated, locally verified `POST /api/v1/data/import` (admin-only): 25 MB body cap (413 over-size), validates `schema_version`, rejects any unexpected/excluded table, checks the tenant-name confirmation, and computes the global old-id -> new-id remap over every row (so a malformed file fails during validation). Returns a dry-run summary and performs **NO mutation**. The security-relevant, order-independent core is pure and unit-tested (`build_id_map`, `remap_row`): fresh id per row, pk/FK/tenant rewrite, and unrelated UUIDs left untouched. `just pre-commit` green (271 tests). ## Commit 2 (next, this branch) - DB load, CI-verified The destructive wipe-and-replace load: derive the FK-dependency topological order from the catalog, wipe the tenant's in-scope tables, load the remapped rows in dependency order (with self-referential-FK and dangling user-reference handling), emit the `import` audit action, and return the per-table summary - all in one transaction. This can only be exercised against Postgres, so it is validated by a round-trip integration test (`tests/*.rs`, run by CI `integration.yml`): export a seeded tenant (PMS-647) -> import -> assert structure matches and no secret substrings appear in the exported bytes. #PMS-648

nrupard added 2 commits 2026-07-09 20:40:18 +02:00

feat(data): admin-only tenant data export endpoint ... 25a583e4d2

First slice of PMS-646. GET /api/v1/data/export returns a single versioned JSON snapshot of the caller's tenant business data as a browser download, audited as an export action.

Schema-driven generic dump (no per-table code): reads every table carrying a tenant_id from information_schema (so new business tables are picked up automatically), minus an explicit exclude set (integration secrets - payment_gateway_configs/credential_vault/api_keys/rmm_connections; auth identity/session/one-shot-token tables; audit/log tables). Every remaining row has secret-looking columns dropped (encrypted/password_hash/_secret/mfa_secret/api_key/api_secret/private_key) as defense-in-depth for secrets that live inside otherwise-business tables (e.g. contacts.portal_password_hash). Each table is dumped with row_to_json ... WHERE tenant_id = $1 under begin_with_tenant, tenant-scoped on both RLS and the explicit filter, and catalog table names are identifier-guarded before interpolation. Admin-gated (RequireAuth + RequireAdmin); emits the previously-unused AuditAction::Export.

Import (wipe-and-replace + FK remap) and the mokosh-apps UI are sibling children under PMS-646. Blob payloads are metadata-only for now (noted in the envelope).

#PMS-647

feat(data): tenant data import - envelope validation + id-remap core ... d193e4653c

First commit of PMS-648 (sibling of the PMS-647 export). Adds the POST /api/v1/data/import endpoint: admin-only, 25 MB body cap (413 over-size), validates schema_version, rejects unexpected/excluded tables, checks the tenant-name confirmation, and computes the global old-id -> new-id remap over every row so a malformed file fails during validation. Returns a dry-run summary and performs NO mutation - the destructive wipe-and-replace load plus the round-trip integration test land in the next commit.

The id-map build and the FK-aware row remap are pure functions with unit tests (locally verified); the DB load orchestration will be validated by the integration round-trip since it can only be exercised against Postgres.

#PMS-648

nrupard added 1 commit 2026-07-09 20:50:24 +02:00

feat(data): tenant data import wipe-and-replace load + round-trip test ... 88df5051c2

Second commit of PMS-648. Replaces the commit-1 dry-run with the real load: in one begin_with_tenant transaction, verify the tenant-name confirmation, reject schema-drift tables, derive the cross-table FK order from the catalog (topo-sort; self-refs handled by one jsonb_populate_recordset statement per table, whose non-deferrable FK check runs at statement end), wipe the in-scope tables children-first, then load the remapped rows parents-first. Emits the import audit action and returns a per-table summary; any failure rolls the whole thing back.

topo_order is a pure, unit-tested function. The DB behaviour is covered by tests/data_transfer.rs: seed a company + FK-referencing contact, export, import back wipe-and-replace, and assert the ids are remapped (the contact FK follows the company's new id) with no secret column in the export. That test runs in CI integration.yml (Postgres); the local --lib pre-commit only compiles it.

#PMS-648

nrupard merged commit 8f01cb1bc8 into main 2026-07-09 20:57:06 +02:00

nrupard deleted branch feat/PMS-648-data-import 2026-07-09 20:57:06 +02:00

nrupard referenced this pull request from a commit 2026-07-09 20:57:06 +02:00

Merge pull request 'feat(data): tenant data import - envelope validation + id-remap core (PMS-648)' (#439) from feat/PMS-648-data-import into main

nrupard referenced this pull request 2026-07-09 21:35:53 +02:00

test(data): fix red integration round-trip (robust to pre-seeded data, PMS-648) #440

Sign in to join this conversation.

Reviewers

No reviewers

Labels

Clear labels

No items

No labels

Milestone

Clear milestone

No items

No milestone

Projects

Clear projects

No items

No project

Assignees

Clear assignees

No assignees

1 participant

Notifications

Subscribe

Due date

The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set

Reference

psa-systems/mokosh-server!439

No description provided.