Reverse Cross Evaluator Design¶
Status: Evaluator v1 and structural-quality foundations implemented on branch
jm/string-repetition at a12ff3b; calibration policy and human feedback remain future work
Scope: Deterministic puzzle evaluation, persisted evaluation results, structural generation policy, benchmark corpora, and the future human-feedback boundary
Audience: Reverse Cross engineers, playtest analysts, and reviewers of quality-policy changes
This is the durable evaluator contract, not a claim that a rollout flag is enabled in production. Persisted puzzle and play shapes are documented in Data Layout and Persistence.
Purpose¶
Reverse Cross currently proves that a generated puzzle is structurally legal and has at least one valid construction. Phase 7 adds an independent, versioned report describing the completed candidate. The report records facts and explicit findings before the product assigns difficulty labels or a single quality score.
The intended generation path is:
generate legal mask
-> fill from the eligible lexical catalog
-> validate the completed construction
-> evaluate the player-visible reconstruction problem
-> reject explicit policy failures or retain warnings
-> persist the puzzle and exact evaluation report
Generation should continue enforcing cheap, established invariants as early as possible. The evaluator is not a substitute for constraint-driven generation and should not turn generation into blind random sampling. It measures properties that require a complete mask, bank, and construction, and it provides one auditable definition of those properties for benchmarks and later calibration.
Terminology¶
- Construction is the generator's process of creating and filling a puzzle.
- Reconstruction is the player's problem: recover any rule-valid arrangement from the public mask and shuffled bank. Generator search cost and player reconstruction difficulty are distinct.
- Witness is the private slot-to-entry assignment found by the generator. It proves that the puzzle is solvable, but it is not an exclusive answer key; any rule-valid reconstruction is accepted.
- Lexical entry ID identifies a normalized language/grid-form pair in the shared catalog. The
current ID is
SHA256(language + NUL + gridForm). It does not identify a dictionary sense. IDs are unique within a catalog snapshot and must also be unique within one puzzle bank. - Canonical order is an explicitly defined ordinal ordering used before hashing or comparing unordered data. Presentation order must not affect identity.
Evaluator Contract¶
ReverseCrossPuzzleEvaluator is a side-effect-free domain service that returns an immutable
ReverseCrossEvaluation. The initial contract contains:
- evaluator version;
- structural metrics;
- constraint-reconstruction metrics;
- lexical metrics;
- aesthetic proxy metrics;
- stable, ordinally sorted warning codes; and
- stable, ordinally sorted rejection codes.
The evaluator does not initially publish Easy, Medium, or Hard, and it does not claim to
measure fun. Raw metrics are observations. Warning/rejection thresholds are versioned product
policy. Human playtesting is needed to establish which observations predict perceived difficulty,
fairness, or enjoyment.
Initial Metric Families¶
| Family | Initial measurements |
|---|---|
| Structure | Dimensions, playable/blocked cells, Across/Down slot counts, slot-length histogram, checked/unchecked cells, connected components, and 180-degree mask-symmetry mismatches. |
| Reconstruction | Initial viable candidates, forced placements, propagation progress, solution count or lower bound, search completeness/truncation, nodes, and backtracks. |
| Lexical | Entry-length histogram, lexical-form counts, source coverage, distinct sources, and score summaries kept separate by source score scale. |
| Aesthetic proxies | Grid balance, crossing distribution, interlock distribution, symmetry measurements, and entry-length variety. |
Connectivity and symmetry are measurements before they are judgments. Disconnected open cells form
independent orthogonally connected regions and usually weaken interlock, but rejecting them is an
explicit Reverse Cross policy. Rotational symmetry compares each cell's blocked/open state with the
cell at (rows - 1 - row, columns - 1 - column); asymmetry is not intrinsically defective.
Structural Generation Policy¶
The evaluator remains reverse-cross-evaluator-v1: changing which masks generation prefers does
not change how a fixed completed puzzle is measured. Generation options contract v2 records an
explicit mask strategy as part of replay identity:
IndependentV1preserves the original deterministic independent-cell algorithm for replay and immediate rollback.SymmetryPreferredV2attempts deterministic 180-degree rotational orbits first, including the one-cell center orbit on odd grids, then uses deterministic individual-cell fallback when needed to approach the requested blocked-cell count. Symmetry is a preference, not a rejection rule.
The first server-enforced structural policy requires a checked-cell ratio of 1.0 and one connected
open component. These are generation constraints, not claims about difficulty or fun. The
ReverseCrossStructuralQuality feature flag selects that policy and SymmetryPreferredV2 for
requests already admitted by ReverseCrossGeneration. When the quality flag is absent, disabled,
or cannot be evaluated, generation uses IndependentV1 with no structural-quality constraints.
A deterministic engineering cohort covers 32 fixed seeds and 25 mask attempts for each of 5x5, 7x7, and 9x9. It retains raw per-attempt samples and verifies deterministic replay, structural legality, aggregate symmetry improvement, and that every accepted strict candidate is fully checked and connected. Every represented size has at least one accepted seed in this cohort. This is a regression gate, not production success-rate evidence and not a substitute for player calibration.
Complete evaluator reports remain separate Cosmos items. The gameplay puzzle document stores only its compact evaluator-report reference; generation telemetry exports a bounded subset of structural values and does not embed the report in the hot puzzle document. Report creation is a best-effort, idempotent write after the winning puzzle document is saved. A report-write failure preserves puzzle availability but can leave a compact reference without its report; current cache-hit reads do not repair that condition. Persistence failures are logged and traced for operator follow-up.
Deterministic Reconstruction Analysis¶
The first reconstruction analysis is a constraint solver, not a simulation of human intuition:
- Extract every maximal Across and Down slot.
- Compute length-compatible bank entries and crossing compatibility.
- Apply forced assignments and propagate crossing constraints.
- Select the unresolved slot with the fewest viable entries.
- Try entries in stable ordinal ID order and backtrack on contradictions.
- Stop at deterministic node and solution-count limits and report truncation explicitly.
Elapsed-time limits and randomized traversal must not affect persisted evaluator results. Later, separately versioned strategy profiles may model longest-first, distinctive-entry, or familiarity-first behavior, but only after play data can test whether those profiles predict human experience.
Transpose-equivalent reconstructions¶
The v1 SolutionCount is a raw count of concrete rule-valid boards. In particular, when a square
mask is unchanged by reflection across its main diagonal and Across and Down use the same bank and
rules, transposing a valid board produces another valid board. A fully open square therefore
normally reports at least two concrete boards even when the second is only the Across/Down transpose
of the first. The current multiple-reconstructions finding is informational: it neither rejects a
puzzle nor changes a quality or difficulty score.
Before reconstruction multiplicity contributes to candidate ranking, rejection, or a named preset,
a versioned evaluator must preserve the raw concrete-board count while also grouping boards into
equivalence classes under transformations that preserve the public puzzle. Transpose-equivalent
boards count as one reconstruction for that quality signal; only a genuinely different equivalence
class may incur an ambiguity penalty. The implementation must not apply transpose equivalence to a
rectangular board or to a square mask, bank, or directional rule set that transposition does not
preserve. Regression cases must distinguish 2 concrete / 1 modulo transpose from puzzles with
genuinely different reconstructions.
Persistence¶
Evaluation is computed once for every successful generated candidate and carried through generation. The complete v1 report is persisted as a separate Cosmos item rather than embedded in the gameplay puzzle document:
ReverseCrossGenerationResult
-> ReverseCrossPuzzleDefinition
-> ReverseCross puzzle item: small evaluation report ID/version reference
-> ReverseCross evaluation-report item: complete immutable report
The initial implementation stores both item types in the existing puzzles container under the
same puzzleKey partition. Point-reading the puzzle by its existing ID does not retrieve the report,
so gameplay does not pay its response size, JSON materialization, or read-RU cost. A report uses a
distinct document discriminator and deterministic ID derived from puzzle identity and evaluator
version. An IReverseCrossEvaluationReportStore owns this boundary so a later move to a dedicated
container does not change evaluator or generator code.
V1 Daily and Practice reports deliberately have no TTL, independently of the 30-day Practice puzzle TTL. Report items can later be shortened, sampled, exported, or deleted by discriminator without rewriting or endangering playable puzzle documents. The current shared-container policy excludes the complete structural, reconstruction, lexical, aesthetic, warning, and rejection bodies from indexing while retaining top-level operational selectors. If report analytics become a sustained workload, move the store to a dedicated container with a partition and indexing policy designed from those queries rather than scanning the shared gameplay container.
Report persistence should be idempotent and must not make a successfully persisted puzzle unplayable when a research write fails. Both items share a logical partition in the initial design, so a transactional batch remains available if later evidence requires atomicity. Existing puzzle documents without a report reference remain readable. Fixture-backed puzzles continue to have no generation envelope or report. Loading a historical puzzle must never silently recompute an absent report under a newer evaluator.
Benchmark Corpus¶
The v1 benchmark corpus is a small, static test asset, not a copy of the production dictionary. It
belongs in ordinary Git under MiniGamesBackEnd.Tests/TestData/ReverseCross/Evaluation/; Git LFS is
not warranted at this size. Its versioned synthetic catalog contains only invented letter sequences
and source metadata.
The first corpus directly exercises the evaluator with six deliberately small cases: a clean unique interlock, an accepted ambiguous reconstruction, an impossible bank, a disconnected asymmetric mask, node-limit truncation, and solution-count truncation. Cases record a stable ID, known-good/known-bad classification, complete mask and bank references, deterministic logical limits, required and forbidden warning/rejection codes, evaluator/catalog versions, a public puzzle fingerprint, and a fingerprint of the complete raw evaluation.
The public fingerprint covers dimensions, blocked/open mask, canonical bank content, and language. The evaluation fingerprint freezes every structural, reconstruction, lexical, and aesthetic value, including source-score summaries, while readable finding-code assertions explain intentional policy behavior. Each case is reevaluated with both bank and lexical inputs reversed to verify canonical ordering. Tests do not assert wall-clock duration.
Generator replay identity and construction artifacts remain covered by generator-focused tests. When presets introduce seed-specific acceptance bands, add generated benchmark cases with complete generation options, catalog/content-policy version, generator version, public fingerprint, and construction fingerprint rather than overloading the direct evaluator corpus.
Private Calibration Workflow¶
tools/ReverseCrossCalibration provides a local-only workflow for manually transcribed completed
grids that the operator is authorized to access. It accepts only a minimal grid schema, derives the
public mask and bank, and writes deterministic reports under .local/reverse-cross-calibration/,
which Git ignores. It does not scrape or authenticate to editorial sources, and its schema excludes
clues and source-identifying provenance. Private examples and their generated reports must not be
committed or redistributed.
Human Feedback Boundary¶
Human feedback is empirical data, not part of deterministic puzzle evaluation. A future
server-enforced ReverseCrossPlaytestFeedback feature flag can expose feedback after completion or
abandonment of generated Practice puzzles. The initial taxonomy should keep overall reaction and
perceived difficulty independent, with optional reason codes and a short note.
Feedback records are stored separately and join to the puzzle by stable identity, fingerprint, generator version, evaluator version, capture-policy version, play status, and cohort/device dimensions. Feedback eligibility is assigned independently from report persistence. The v1 policy retains complete reports for all generated puzzles, so a later feedback submission can reference the existing report without rewriting the gameplay puzzle. If traffic later requires sampling, capture assignment must happen when the Practice puzzle is created, before full lexical metadata leaves memory. This preserves eligible non-responders as the response-rate denominator and permits comparison among completed, abandoned, and never-started puzzles.
Completion does not make an evaluator report more correct: the report describes the puzzle. It makes the report more useful when joined to an observed outcome. Submitted feedback gives a direct subjective label; completion without feedback still supplies behavioral evidence such as elapsed time and interaction counts; an explicitly defined abandonment supplies weaker but important evidence; and a never-started puzzle supplies no player-experience evidence. Full reports for the last category are expected sampling overhead and should be bounded by cohort size and TTL.
Synthetic feedback belongs only in isolated tests or explicitly marked demonstration data; it must not be presented as human evidence. The first evaluator does not consume feedback.
Open calibration decisions include sample-size requirements, treatment of abandonment, cohort and device segmentation, expert-versus-general-player interpretation, and whether evidence eventually changes rejection policy, candidate ranking, or named difficulty presets.
Implementation Sequence¶
- Add the versioned evaluation records, shared structural calculations, deterministic reconstruction analysis, lexical/aesthetic measurements, and focused unit tests.
- Evaluate successful generated candidates while selected
LexicalEntrymetadata is still available; persist complete reports as separate idempotent items, leave gameplay puzzle reads independent from report reads, and add backward-compatible mapper/store tests. - Add the versioned synthetic catalog and known-good/known-bad evaluator cases.
- Add replay-versioned legacy and symmetry-preferred mask strategies, with fully checked and connected constraints behind a fail-safe runtime feature flag.
- Add deterministic fixed-seed structural cohorts and evaluator-derived production telemetry for checked ratio, unchecked cells, connected components, and rotational mismatch pairs.
- Research crossword-construction guidance into a sourced rubric of hypotheses, clearly separating clue-dependent traditional crossword conventions from Reverse Cross's bank-reconstruction model.
- Add the feedback feature as one complete UI/API/storage/export slice. Add versioned Practice capture-policy selection only if report volume later requires sampling, and collect observations before defining calibration policy.