fix(classify): replace peak-counting bimodality with Otsu eta (MK-14) #32

Merged

David merged 1 commit from fix/classify-otsu-bimodality-MK-14 into main

2026-05-24 05:20:57 +02:00

David commented

2026-05-24 05:16:41 +02:00

Owner

Summary

bimodality_score returns max_t (between-class variance at threshold t) / (total variance), computed in one pass over the 64-bin luma histogram with O(1) per-threshold class moments via prefix sums. No new deps.
Eta is in [0, 1] regardless of how spread the dark population is: hard-binary inputs (white paper + black ink, however anti-aliased) score 0.84-0.94, photos and gradients < 0.30. Old detector returned 0.000 for every real scan because the dark ink population spread across many bins and no single bin cleared the 1%-of-total gate.
Retunes four thresholds whose old values were calibrated against peak-counting:
- bimodality_binary 0.85 -> 0.82 (anchored below the real-scan eta cluster).
- soft_scale_bimodality 0.05 -> 0.06.
- edge_density_text_min 0.10 -> 0.06 (admits light-print thermal receipts).
- min_score 0.50 -> 0.35 (Otsu eta produces naturally lower multiplicative class scores).
Adds tests/fixtures/ to .gitignore. Real validation imagery stays local; the directory layout tests/fixtures/classify/<class>/ is established convention for ad-hoc validation runs.

Validation

corpus	before	after
in-tree synthetic unit tests	14/14	14/14
`tests/fixtures/classify/binary-scan/`	0/21	21/21
`tests/fixtures/classify/receipt/`	2/5	5/5

No test code changes; only thresholds in classify.toml and the detector body in classify.rs.

Test plan

cargo test (synthetic suite, including the existing binary-scan and receipt cases).
cargo run --release -- image classify <path> against each file in the local fixture set (results above).
cargo clippy --all-targets -- -D warnings.
cargo fmt --check.
CI Docker stage on Forgejo.

Implements MK-14. Replaces the peak-counting bimodality detector with Otsu eta, the textbook between-class variance ratio. Validated against the user's local fixture set (21 letter-size mono scans + 5 thermal receipts). ## Summary - `bimodality_score` returns `max_t (between-class variance at threshold t) / (total variance)`, computed in one pass over the 64-bin luma histogram with O(1) per-threshold class moments via prefix sums. No new deps. - Eta is in `[0, 1]` regardless of how spread the dark population is: hard-binary inputs (white paper + black ink, however anti-aliased) score 0.84-0.94, photos and gradients < 0.30. Old detector returned 0.000 for every real scan because the dark ink population spread across many bins and no single bin cleared the 1%-of-total gate. - Retunes four thresholds whose old values were calibrated against peak-counting: - `bimodality_binary` 0.85 -> 0.82 (anchored below the real-scan eta cluster). - `soft_scale_bimodality` 0.05 -> 0.06. - `edge_density_text_min` 0.10 -> 0.06 (admits light-print thermal receipts). - `min_score` 0.50 -> 0.35 (Otsu eta produces naturally lower multiplicative class scores). - Adds `tests/fixtures/` to `.gitignore`. Real validation imagery stays local; the directory layout `tests/fixtures/classify/<class>/` is established convention for ad-hoc validation runs. ## Validation | corpus | before | after | |-----------------------------------------|-----------|----------| | in-tree synthetic unit tests | 14/14 | 14/14 | | `tests/fixtures/classify/binary-scan/` | 0/21 | 21/21 | | `tests/fixtures/classify/receipt/` | 2/5 | 5/5 | No test code changes; only thresholds in `classify.toml` and the detector body in `classify.rs`. ## Test plan - [x] `cargo test` (synthetic suite, including the existing binary-scan and receipt cases). - [x] `cargo run --release -- image classify <path>` against each file in the local fixture set (results above). - [x] `cargo clippy --all-targets -- -D warnings`. - [x] `cargo fmt --check`. - [ ] CI Docker stage on Forgejo.

David added 1 commit

2026-05-24 05:16:41 +02:00

fix(classify): replace peak-counting bimodality with Otsu eta

Check / fmt + clippy + build + tests (pull_request) Successful in 43s

Details

Create release / Create release from merged PR (pull_request) Has been skipped

Details

b0ebe00810

The MK-6 bimodality detector required a sharp second histogram peak holding at least 1% of pixels in a single 64-wide bin. Real document scans have ~90% of pixels concentrated at white (paper) and the dark ink population spread across twenty bins (anti-aliasing, JPEG/PNG compression, ink bleed), so no single dark bin clears the 1% gate. Result on the user-provided fixture set: every one of 21 letter-size mono scans and every one of 5 receipt scans returned `bimodality = 0.000`, zeroing the binary-scan score.

Replaces the peak-counting routine with Otsu's eta: the maximum between-class variance over the 64-bin luma histogram divided by the total variance. The metric is in [0, 1] regardless of how spread the dark population is, so hard-binary inputs (white paper + black ink, however smeared) score 0.84-0.94, while photos and gradients land below 0.30. Cost is one pass over the histogram with O(1) per-threshold class moments via prefix sums; no new dependencies.

Retunes the three thresholds whose old values were calibrated against the peak-counting score: `bimodality_binary` (0.85 -> 0.82) and `soft_scale_bimodality` (0.05 -> 0.06) anchor the logistic to the real-scan eta cluster, `edge_density_text_min` (0.10 -> 0.06) lets light-print thermal receipts decisively beat the binary-vs-receipt tie, and `min_score` (0.50 -> 0.35) accommodates the naturally lower multiplicative scores that Otsu eta produces. Inline TOML comments cite the empirical observations.

Ignores `tests/fixtures/` from the same commit so the real validation imagery the user dropped into `tests/fixtures/classify/<class>/` stays local and never bleeds into the repo (per the user's choice to host real fixtures in a separate repo).

Validates against synthetic fixtures: 14 of 14 in-tree unit tests pass with no test code changes. Validates against the local fixture set: 21 of 21 letter-size mono scans (`binary-scan/`) classify as `binary-scan`, 5 of 5 thermal-roll scans (`receipt/`) classify as `receipt`.

#MK-14 State Done

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>