Skip to content

Reverse Cross Evaluator Design

Status: Evaluator v1 and structural-quality foundations implemented on branch jm/string-repetition at a12ff3b; calibration policy and human feedback remain future work

Scope: Deterministic puzzle evaluation, persisted evaluation results, structural generation policy, benchmark corpora, and the future human-feedback boundary

Audience: Reverse Cross engineers, playtest analysts, and reviewers of quality-policy changes

This is the durable evaluator contract, not a claim that a rollout flag is enabled in production. Persisted puzzle and play shapes are documented in Data Layout and Persistence.

Purpose

Reverse Cross currently proves that a generated puzzle is structurally legal and has at least one valid construction. Phase 7 adds an independent, versioned report describing the completed candidate. The report records facts and explicit findings before the product assigns difficulty labels or a single quality score.

The intended generation path is:

generate legal mask
    -> fill from the eligible lexical catalog
    -> validate the completed construction
    -> evaluate the player-visible reconstruction problem
    -> reject explicit policy failures or retain warnings
    -> persist the puzzle and exact evaluation report

Generation should continue enforcing cheap, established invariants as early as possible. The evaluator is not a substitute for constraint-driven generation and should not turn generation into blind random sampling. It measures properties that require a complete mask, bank, and construction, and it provides one auditable definition of those properties for benchmarks and later calibration.

Terminology

  • Construction is the generator's process of creating and filling a puzzle.
  • Reconstruction is the player's problem: recover any rule-valid arrangement from the public mask and shuffled bank. Generator search cost and player reconstruction difficulty are distinct.
  • Witness is the private slot-to-entry assignment found by the generator. It proves that the puzzle is solvable, but it is not an exclusive answer key; any rule-valid reconstruction is accepted.
  • Lexical entry ID identifies a normalized language/grid-form pair in the shared catalog. The current ID is SHA256(language + NUL + gridForm). It does not identify a dictionary sense. IDs are unique within a catalog snapshot and must also be unique within one puzzle bank.
  • Canonical order is an explicitly defined ordinal ordering used before hashing or comparing unordered data. Presentation order must not affect identity.

Evaluator Contract

ReverseCrossPuzzleEvaluator is a side-effect-free domain service that returns an immutable ReverseCrossEvaluation. The initial contract contains:

  • evaluator version;
  • structural metrics;
  • constraint-reconstruction metrics;
  • lexical metrics;
  • aesthetic proxy metrics;
  • stable, ordinally sorted warning codes; and
  • stable, ordinally sorted rejection codes.

The evaluator does not initially publish Easy, Medium, or Hard, and it does not claim to measure fun. Raw metrics are observations. Warning/rejection thresholds are versioned product policy. Human playtesting is needed to establish which observations predict perceived difficulty, fairness, or enjoyment.

Initial Metric Families

Family Initial measurements
Structure Dimensions, playable/blocked cells, Across/Down slot counts, slot-length histogram, checked/unchecked cells, connected components, and 180-degree mask-symmetry mismatches.
Reconstruction Initial viable candidates, forced placements, propagation progress, solution count or lower bound, search completeness/truncation, nodes, and backtracks.
Lexical Entry-length histogram, lexical-form counts, source coverage, distinct sources, and score summaries kept separate by source score scale.
Aesthetic proxies Grid balance, crossing distribution, interlock distribution, symmetry measurements, and entry-length variety.

Connectivity and symmetry are measurements before they are judgments. Disconnected open cells form independent orthogonally connected regions and usually weaken interlock, but rejecting them is an explicit Reverse Cross policy. Rotational symmetry compares each cell's blocked/open state with the cell at (rows - 1 - row, columns - 1 - column); asymmetry is not intrinsically defective.

Structural Generation Policy

The evaluator remains reverse-cross-evaluator-v1: changing which masks generation prefers does not change how a fixed completed puzzle is measured. Generation options contract v2 records an explicit mask strategy as part of replay identity:

  • IndependentV1 preserves the original deterministic independent-cell algorithm for replay and immediate rollback.
  • SymmetryPreferredV2 attempts deterministic 180-degree rotational orbits first, including the one-cell center orbit on odd grids, then uses deterministic individual-cell fallback when needed to approach the requested blocked-cell count. Symmetry is a preference, not a rejection rule.

The first server-enforced structural policy requires a checked-cell ratio of 1.0 and one connected open component. These are generation constraints, not claims about difficulty or fun. The ReverseCrossStructuralQuality feature flag selects that policy and SymmetryPreferredV2 for requests already admitted by ReverseCrossGeneration. When the quality flag is absent, disabled, or cannot be evaluated, generation uses IndependentV1 with no structural-quality constraints.

A deterministic engineering cohort covers 32 fixed seeds and 25 mask attempts for each of 5x5, 7x7, and 9x9. It retains raw per-attempt samples and verifies deterministic replay, structural legality, aggregate symmetry improvement, and that every accepted strict candidate is fully checked and connected. Every represented size has at least one accepted seed in this cohort. This is a regression gate, not production success-rate evidence and not a substitute for player calibration.

Complete evaluator reports remain separate Cosmos items. The gameplay puzzle document stores only its compact evaluator-report reference; generation telemetry exports a bounded subset of structural values and does not embed the report in the hot puzzle document. Report creation is a best-effort, idempotent write after the winning puzzle document is saved. A report-write failure preserves puzzle availability but can leave a compact reference without its report; current cache-hit reads do not repair that condition. Persistence failures are logged and traced for operator follow-up.

Deterministic Reconstruction Analysis

The first reconstruction analysis is a constraint solver, not a simulation of human intuition:

  1. Extract every maximal Across and Down slot.
  2. Compute length-compatible bank entries and crossing compatibility.
  3. Apply forced assignments and propagate crossing constraints.
  4. Select the unresolved slot with the fewest viable entries.
  5. Try entries in stable ordinal ID order and backtrack on contradictions.
  6. Stop at deterministic node and solution-count limits and report truncation explicitly.

Elapsed-time limits and randomized traversal must not affect persisted evaluator results. Later, separately versioned strategy profiles may model longest-first, distinctive-entry, or familiarity-first behavior, but only after play data can test whether those profiles predict human experience.

Transpose-equivalent reconstructions

The v1 SolutionCount is a raw count of concrete rule-valid boards. In particular, when a square mask is unchanged by reflection across its main diagonal and Across and Down use the same bank and rules, transposing a valid board produces another valid board. A fully open square therefore normally reports at least two concrete boards even when the second is only the Across/Down transpose of the first. The current multiple-reconstructions finding is informational: it neither rejects a puzzle nor changes a quality or difficulty score.

Before reconstruction multiplicity contributes to candidate ranking, rejection, or a named preset, a versioned evaluator must preserve the raw concrete-board count while also grouping boards into equivalence classes under transformations that preserve the public puzzle. Transpose-equivalent boards count as one reconstruction for that quality signal; only a genuinely different equivalence class may incur an ambiguity penalty. The implementation must not apply transpose equivalence to a rectangular board or to a square mask, bank, or directional rule set that transposition does not preserve. Regression cases must distinguish 2 concrete / 1 modulo transpose from puzzles with genuinely different reconstructions.

Persistence

Evaluation is computed once for every successful generated candidate and carried through generation. The complete v1 report is persisted as a separate Cosmos item rather than embedded in the gameplay puzzle document:

ReverseCrossGenerationResult
    -> ReverseCrossPuzzleDefinition
    -> ReverseCross puzzle item: small evaluation report ID/version reference
    -> ReverseCross evaluation-report item: complete immutable report

The initial implementation stores both item types in the existing puzzles container under the same puzzleKey partition. Point-reading the puzzle by its existing ID does not retrieve the report, so gameplay does not pay its response size, JSON materialization, or read-RU cost. A report uses a distinct document discriminator and deterministic ID derived from puzzle identity and evaluator version. An IReverseCrossEvaluationReportStore owns this boundary so a later move to a dedicated container does not change evaluator or generator code.

V1 Daily and Practice reports deliberately have no TTL, independently of the 30-day Practice puzzle TTL. Report items can later be shortened, sampled, exported, or deleted by discriminator without rewriting or endangering playable puzzle documents. The current shared-container policy excludes the complete structural, reconstruction, lexical, aesthetic, warning, and rejection bodies from indexing while retaining top-level operational selectors. If report analytics become a sustained workload, move the store to a dedicated container with a partition and indexing policy designed from those queries rather than scanning the shared gameplay container.

Report persistence should be idempotent and must not make a successfully persisted puzzle unplayable when a research write fails. Both items share a logical partition in the initial design, so a transactional batch remains available if later evidence requires atomicity. Existing puzzle documents without a report reference remain readable. Fixture-backed puzzles continue to have no generation envelope or report. Loading a historical puzzle must never silently recompute an absent report under a newer evaluator.

Benchmark Corpus

The v1 benchmark corpus is a small, static test asset, not a copy of the production dictionary. It belongs in ordinary Git under MiniGamesBackEnd.Tests/TestData/ReverseCross/Evaluation/; Git LFS is not warranted at this size. Its versioned synthetic catalog contains only invented letter sequences and source metadata.

The first corpus directly exercises the evaluator with six deliberately small cases: a clean unique interlock, an accepted ambiguous reconstruction, an impossible bank, a disconnected asymmetric mask, node-limit truncation, and solution-count truncation. Cases record a stable ID, known-good/known-bad classification, complete mask and bank references, deterministic logical limits, required and forbidden warning/rejection codes, evaluator/catalog versions, a public puzzle fingerprint, and a fingerprint of the complete raw evaluation.

The public fingerprint covers dimensions, blocked/open mask, canonical bank content, and language. The evaluation fingerprint freezes every structural, reconstruction, lexical, and aesthetic value, including source-score summaries, while readable finding-code assertions explain intentional policy behavior. Each case is reevaluated with both bank and lexical inputs reversed to verify canonical ordering. Tests do not assert wall-clock duration.

Generator replay identity and construction artifacts remain covered by generator-focused tests. When presets introduce seed-specific acceptance bands, add generated benchmark cases with complete generation options, catalog/content-policy version, generator version, public fingerprint, and construction fingerprint rather than overloading the direct evaluator corpus.

Private Calibration Workflow

tools/ReverseCrossCalibration provides a local-only workflow for manually transcribed completed grids that the operator is authorized to access. It accepts only a minimal grid schema, derives the public mask and bank, and writes deterministic reports under .local/reverse-cross-calibration/, which Git ignores. It does not scrape or authenticate to editorial sources, and its schema excludes clues and source-identifying provenance. Private examples and their generated reports must not be committed or redistributed.

Human Feedback Boundary

Human feedback is empirical data, not part of deterministic puzzle evaluation. A future server-enforced ReverseCrossPlaytestFeedback feature flag can expose feedback after completion or abandonment of generated Practice puzzles. The initial taxonomy should keep overall reaction and perceived difficulty independent, with optional reason codes and a short note.

Feedback records are stored separately and join to the puzzle by stable identity, fingerprint, generator version, evaluator version, capture-policy version, play status, and cohort/device dimensions. Feedback eligibility is assigned independently from report persistence. The v1 policy retains complete reports for all generated puzzles, so a later feedback submission can reference the existing report without rewriting the gameplay puzzle. If traffic later requires sampling, capture assignment must happen when the Practice puzzle is created, before full lexical metadata leaves memory. This preserves eligible non-responders as the response-rate denominator and permits comparison among completed, abandoned, and never-started puzzles.

Completion does not make an evaluator report more correct: the report describes the puzzle. It makes the report more useful when joined to an observed outcome. Submitted feedback gives a direct subjective label; completion without feedback still supplies behavioral evidence such as elapsed time and interaction counts; an explicitly defined abandonment supplies weaker but important evidence; and a never-started puzzle supplies no player-experience evidence. Full reports for the last category are expected sampling overhead and should be bounded by cohort size and TTL.

Synthetic feedback belongs only in isolated tests or explicitly marked demonstration data; it must not be presented as human evidence. The first evaluator does not consume feedback.

Open calibration decisions include sample-size requirements, treatment of abandonment, cohort and device segmentation, expert-versus-general-player interpretation, and whether evidence eventually changes rejection policy, candidate ranking, or named difficulty presets.

Implementation Sequence

  1. Add the versioned evaluation records, shared structural calculations, deterministic reconstruction analysis, lexical/aesthetic measurements, and focused unit tests.
  2. Evaluate successful generated candidates while selected LexicalEntry metadata is still available; persist complete reports as separate idempotent items, leave gameplay puzzle reads independent from report reads, and add backward-compatible mapper/store tests.
  3. Add the versioned synthetic catalog and known-good/known-bad evaluator cases.
  4. Add replay-versioned legacy and symmetry-preferred mask strategies, with fully checked and connected constraints behind a fail-safe runtime feature flag.
  5. Add deterministic fixed-seed structural cohorts and evaluator-derived production telemetry for checked ratio, unchecked cells, connected components, and rotational mismatch pairs.
  6. Research crossword-construction guidance into a sourced rubric of hypotheses, clearly separating clue-dependent traditional crossword conventions from Reverse Cross's bank-reconstruction model.
  7. Add the feedback feature as one complete UI/API/storage/export slice. Add versioned Practice capture-policy selection only if report volume later requires sampling, and collect observations before defining calibration policy.