Lexical and Game Architecture

This document is the durable backend design for sharing lexical content across MiniGames without sharing game rules. The Reverse Cross roadmap records the phased migration history; this document describes the intended steady state. The Lexical Memory Analysis Runbook records the heap-capture, object-ownership, and Azure methods used to measure this architecture.

Design goals

• Load, normalize, moderate, version, and retain a lexical corpus once per language and catalog version.
• Let each game admit different forms and sources without maintaining duplicate dictionaries or accidentally broadening another game's rules.
• Preserve Unicode display text and game-specific graphemes rather than treating UTF-16 code units as letters.
• Build game-specific search structures from shared immutable entries and keep those algorithms in the game that owns them.
• Make cold loading, refresh, failure, cancellation, provenance, and telemetry explicit enough to operate safely in scaled-to-zero containers.
• Keep Letter Golf behavior stable while the production path moves incrementally away from the legacy word abstractions.

Responsibility flow

lexical sources
    |
    v
normalization + suite-wide moderation + provenance
    |
    v
immutable, versioned WordCatalogSnapshot
    |
    +--> Letter Golf eligibility view --> LetterGolfCatalogIndex --> LetterGolfGenerator
    |
    +--> Reverse Cross eligibility view --> ReverseCrossFillIndex --> production provider/generator
    |
    +--> future game eligibility view --> game-specific index/algorithm

The names in that flow describe ownership:

• A source is raw upstream material plus its identity, content version or fingerprint, provenance, license/attribution, language coverage, and source-native score.
• A catalog snapshot is the normalized and moderated shared in-memory representation. It owns the accepted entry and grapheme data plus the source and category registries used by compact entries.
• A profile is a side-effect-free game/mode policy over catalog metadata. A view is that profile applied to one snapshot.
• An index is a game-specific search structure referencing entries in a view.
• A provider selects or loads a generated puzzle or fixture. A generator performs a construction/search algorithm. Reverse Cross's production provider selects bounded generation behind a default-off feature flag and falls back to the fixture provider.

Shared lexical layer

Entries and normalization

LexicalEntry separates display form, normalized grid form, and immutable grapheme units. It also retains entry identity, lexical form, and compact references to language, source provenance, categories, and source-native scoring. Production catalog entries derive language and descriptor metadata from one snapshot-owned LexicalMetadataRegistry. They store a 64-bit source mask, sparse (source ordinal, numeric value) scores, and sorted 16-bit category IDs; entries without categories retain no category array. Public Language, Sources, and Tags projections remain readable, and standalone entry construction remains available for compatibility and tests. LexicalEntry.Id is a fixed 32-byte value type LexicalId containing the SHA-256 digest of UTF-8 language + NUL byte + gridForm. Canonical external identity is textual: lex: followed by 64 lowercase hexadecimal characters. Canonical formatting/parsing from/to that textual representation happens only at API, persistence, fixture, export, compatibility, and diagnostic boundaries; internal catalog operations use the typed LexicalId for sorting, comparison, hashing, and dictionary lookups. The catalog builder applies the language normalizer and suite-wide deny policy before an entry can be published.

Lexical metadata dimensions remain independent. Entry shape, grammar, morphology, usage labels, and editorial classification must not collapse into one overloaded enum. In particular, Inflection includes plurals, possessives, conjugations, participles, comparisons, declensions, agreement, and other language-specific changes; it is not synonymous with declension. See the detailed metadata notes in the Reverse Cross roadmap.

Current metadata scope and deferred models

The catalog intentionally stores only metadata that a current game uses. Unknown metadata is not false metadata: an entry from a source without grammatical classification remains eligible under that source's profile unless another reliable source contributes a classification that the profile acts on.

For the prepared lemma/POS/inflection source, the runtime artifact is a lossy, game-oriented projection rather than a copy of every upstream column. It uses UTF-8 TSV with the required header surface<TAB>forms, one normalized surface per row, and a |-separated combination of these tokens:

• word for source codes v, n, r, j, u, m, and fw, including their expanded inflections;
• proper-noun for source code K;
• abbreviation for source code abbr.

Rows sharing a normalized surface are combined before export. A surface classified both as an ordinary word and a proper noun retains both flags, allowing Letter Golf to accept an ordinary reading such as carol; a surface known only as a proper noun, such as Christmas, can be excluded. The artifact preserves meaningful display casing, is sorted by normalized surface, and contains no duplicate normalized surfaces. It deliberately omits frequency, lemma relationships, and detailed parts of speech. The original source and transformation provenance must remain documented so a richer artifact can be regenerated later.

The following models are explicitly deferred:

• Detailed part of speech: there is no LexicalPartOfSpeech type. Noun, verb, adjective, adverb, interjection, numeral, and function-word distinctions do not currently affect game eligibility, generation, scoring, or display. Adding the type would affect raw and normalized entry contracts, catalog merging and fingerprinting, source parsers, evaluation reports, eligibility tests, and retained memory. Add it only when a game or editor has a concrete policy that distinguishes at least two of those classes.
• Lemma and morphology relationships: the catalog does not store links such as am to be. Current games consume playable surface forms, so inflections from the prepared source are emitted as ordinary words. Add lemma IDs and structured morphology only for a feature that uses word families, such as clue generation, family-aware duplicate avoidance, or morphology-based play.
• Foreign-term classification: ForeignTerm remains a provisional LexicalForm flag. A future language metadata model means typed, validated language or usage data, such as BCP 47 language identifiers and whether a form is established English, borrowed, or foreign in a particular sense. It does not mean a machine-learning model. A spelling alone cannot reliably decide this: the same form may be an English word and a word in another language. Replace the flag only when a game uses language-aware eligibility or clues and a source supplies defensible language data.
• Idiom classification: Idiom remains a provisional LexicalForm flag. Idiomatic status belongs to a usage or sense classification because one phrase can have both literal and idiomatic readings. Add a typed usage model only when clues, editorial workflows, or a game mode consume that distinction and source data identifies the intended sense.
• Open-ended themes and editorial categories: these cannot be a closed enum. The snapshot owns a deterministic, ordinal-case-insensitive category registry, while entries retain sorted compact IDs and continue to expose existing string tags as a compatibility projection. Adding a category still changes canonical catalog content and therefore catalog identity; compact IDs are local to their owning snapshot and are not persistence or API identifiers.

Snapshot views and membership

WordCatalogSnapshot retains the complete sorted entry array, indexed by LexicalId, and owns the shared metadata registry plus authoritative lookups by entry ID (keyed by LexicalId) and grid form. WordCatalogView filters that snapshot to entries eligible under a game's policy (profile, theme, context). Eligibility checks query source masks, compact scores, and category IDs directly; they do not materialize the readable metadata projections. A view retains those filtered entry references and a compact BitArray keyed by snapshot ordinal; it does not retain a second ID array or lookup map. FindById and FindByGridForm resolve through the snapshot's authoritative dictionaries (typed by LexicalId internally), recover the matching ordinal, and test the bitmap. Textual lex:-prefixed IDs are resolved to LexicalId at boundaries only. Views remain immutable and retain their owning snapshot.

CatalogEntryHandle is a dense ordinal paired with a private scope owned by one exact snapshot instance. Equality and resolution require that scope, so even two snapshots with the same version cannot exchange handles accidentally. Stable textual lexical IDs (lex: + 64 hex chars) remain the boundary identity for external APIs, persistence, and cross-snapshot references.

LexicalForm.ProperNoun, Inflection, ForeignTerm, and Idiom therefore remain provisional for compatibility; this decision does not endorse adding more unrelated concepts to that enum. Moving them requires a current consumer, migration of fixture metadata and profile rules, catalog-version changes, and compatibility updates for persisted evaluation data.

Immutable snapshots

A catalog build remains private until source acquisition, normalization, moderation, merge, and minimum-size validation all succeed. Publication is one reference swap to a complete WordCatalogSnapshot. Callers cannot mutate its entries, source descriptors, category registry, tags, or grapheme collections, and consumers of the same language/version receive the same snapshot object rather than copies of the raw corpus.

An existing caller may safely retain an older snapshot while a newer one is published. The provider retains only the current snapshot; historical retrieval would require a future artifact store and must never silently substitute a different version.

Single-flight loading and refresh

The catalog provider maintains at most one in-flight build per language. Concurrent cold-start callers await that shared task. During refresh, ordinary reads keep receiving the last known good snapshot, and concurrent refresh requests share one candidate build. A completed candidate becomes visible atomically. A failed initial task is removed so a later call can retry; a failed refresh does not displace the last known good snapshot.

The same rule applies to expensive derived indexes: their cache key contains the catalog version, profile ID/version, and index-options fingerprint. Concurrent callers for one key share a build, while a new snapshot or different profile/options creates a distinct index. A caller already holding an old index may continue using it because that index retains its immutable view.

Versioning, cancellation, and failure

The catalog version is derived deterministically from the schema, canonical language, normalization and content-policy versions, sorted source descriptors, and a canonical fingerprint of every accepted entry and its forms, graphemes, provenance, scores, and tags. The fingerprint is streamed from compact masks, scores, and category IDs through their registry values, avoiding compatibility-projection allocations while preserving the exact canonical bytes. Each production HTTP source uses a commit-pinned artifact URI and an immutable descriptor containing its source ID, language, commit version, declared license, provenance URI, and optional source-wide score scale. Scores from unrelated scales remain distinct. Mutable branch-head downloads do not qualify as production catalog sources.

Caller cancellation is scoped to the caller's wait. It does not cancel or poison shared work that other callers need. The builder receives a separate operation-lifetime token, bound in production to application shutdown and a three-minute catalog operation limit, and forwards it through source refresh, enumeration, and moderation waits. HTTP artifact requests have their own 60-second timeout. A canceled, unavailable, malformed, or unexpectedly small candidate is never published.

Telemetry covers catalog and index build/refresh duration, status, language, safe versions, source and accepted/rejected counts, and failure category. The current vocabulary is WordCatalog.Build, WordCatalog.SourceRefresh, WordCatalog.SourceRead, WordCatalog.Load, WordCatalog.Refresh, WordCatalog.ViewBuild, and LetterGolf.CatalogIndexBuild, alongside the older compatibility spans such as WordList.ColdStartLoad. The catalog and index spans are now on the live Letter Golf path. Raw accepted or rejected lexical content, denied terms, theme text, and entry IDs are never metric tags or log fields.

The measured cold-build cost and proposed investigation of immutable compiled artifacts, container/blob/OCI delivery, and compact or memory-mapped layouts are documented in Lexical Catalog Cold-Start and Compiled-Artifact Plan.

Production source composition

Production currently composes the native, pinned Common Words, Webster, dwyl, Frequency List, Verbolus common words, Crossword Nexus, Peter Broda scored crossword, and first-party Reverse Cross fixture sources. The prepared Frequency List, Verbolus common-word, and Peter Broda artifacts are packaged in the backend publish output and container image, while the other third-party sources remain pinned HTTP acquisitions. Frequency List and Verbolus common words contribute to Letter Golf and Reverse Cross views; Nexus and Peter Broda contribute only to Reverse Cross views. Each parser preserves its current filtering, accepts cancellation, and rejects an invalid or undersized artifact rather than converting failure into an empty source. The complete content pins, declared licenses or rights references, provenance links, and unresolved rights notes are maintained in Third-Party Notices.

NexusLexicalSource and PeterBrodaLexicalSource are pinned, cancellable, scored, and fail-closed. Peter Broda scores retain a distinct scale identity and are not interpreted as Nexus scores or as lexical-form classifications. Complete-source workstation measurements are recorded in the Reverse Cross roadmap; production-container startup, memory, and tail behavior remain staged-release gates. TestWordsSource has no verified production provenance and is not a production catalog source.

Game-owned layers

Letter Golf

Letter Golf's Standard and Common Seed profiles admit the forms and sources appropriate to its existing rules. LetterGolfCatalogIndex references the shared entries using typed LexicalId internally, derives the common-seed subset from provenance, stores view-local ordinals in its count-signature trie, and computes frequency data from the catalog's authoritative graphemes. Production queries resolve trie results directly through the index's entry array; compatibility APIs project textual lex:-prefixed IDs for external consumption. LetterGolfGenerator owns solution-first selection, difficulty options, and puzzle construction. The live generator resolves the singleton catalog index provider directly; it no longer builds independent string dictionaries, signature indexes, or frequency tables. Historical puzzles whose stored letter keys are lowercase are normalized before solution lookup so the cutover preserves their behavior.

The class was historically named TestLetterGolfGenerator and its keyed DI registration was "TestLetterGolfGen". The ownership refactor renamed them to LetterGolfGenerator and Consts.LetterGolfGeneratorServiceKey; it did not change the generation algorithm, endpoint contract, or the keyed substitution seam used by tests.

Reverse Cross

Reverse Cross has Crossword and Generated profiles applied to the same snapshot. Crossword Nexus, the local Peter Broda source, and first-party fixtures are eligible in both profiles, subject to each profile's explicit source policy. Peter Broda entries are admitted independently of their source-native score because that score is retained as provenance rather than treated as a lexical classifier. First-party fixture metadata contributes explicit forms and tags for proper nouns, abbreviations, phrases, idioms, and other crossword content without admitting those entries to Letter Golf. The fixture provider resolves every authored entry through that eligible view and rejects an incomplete fixture set.

The asynchronous, cancellable serving boundary is IReverseCrossPuzzleProvider. ReverseCrossProductionPuzzleProvider selects ReverseCrossGenerator only when the separate ReverseCrossGeneration feature flag is enabled; missing, disabled, or failed flag evaluation and bounded generation failures retain fixture behavior. Caller cancellation remains terminal.

ReverseCrossFillIndex is a process-local, reusable positional index over one catalog/profile and configured grapheme-length range. It groups shared LexicalEntry references by length and stores flat integer postings by position and grapheme; lexical strings, grapheme arrays, and compact provenance/category metadata remain snapshot-owned. Each length bucket also owns an index-wide ordinal offset, allowing fill search to track used candidates as integers. Puzzle-bank solution and reconstruction analyzers similarly carry puzzle-local ordinals with sorted candidates rather than hash textual IDs during search. Public and persisted Reverse Cross IDs remain textual. The singleton provider coalesces builds and caches by language, catalog/profile version, length bounds, and index-options version. Fill order, mask construction, uniqueness, structural validation, difficulty, and quality scoring remain game algorithms, not catalog responsibilities.

Theme and richer-content enrichment remain later capabilities. The proposed LLM-assisted offline pipeline, review boundaries, and theme-manifest direction are tracked in LLM-Assisted Content Enrichment and Theming.

Additional games

Word Spin, Word Cube, and future games follow the same boundary when they need lexical content: define an explicit eligibility profile, retain a view over the shared snapshot, and build only the index required by that game's algorithm. Games share acquisition, normalization, moderation, provenance, language/grapheme handling, catalog versioning, identity, persistence primitives, telemetry plumbing, stats/streak infrastructure, and group infrastructure where those contracts are truly game-neutral. They do not inherit from one generic game engine or share scoring, generation, validation, difficulty, or UI rules merely because both use words.

Physical ownership

MiniGamesBackEnd/
  Words/
    Domain/          lexical entries and source metadata
    Normalization/   language normalization and grapheme segmentation
    Sources/         source adapters
      Acquisition/   raw loading/downloading boundaries
    Content/         moderation inputs and eligibility policy
    Catalog/         immutable snapshot construction and publication
    Views/           game-profile projections over a snapshot
    Compatibility/   temporary IWords/MultiDict migration layer
  Games/
    LetterGolf/
      Domain/
      Content/
      Generation/
      Application/
      Validation/
    ReverseCross/
      Domain/
      Content/
      Generation/

Tests mirror these durable ownership areas. Phase or step numbers do not appear in permanent test filenames or class names; milestone-created assertions remain as ordinary regression coverage. Endpoint and contract tests stay at their transport boundary rather than being moved under a game solely because a route mentions that game.

Namespaces currently remain compatible with the pre-refactor code so file moves and behavior are separately reviewable. Namespace changes, if useful, should be their own mechanical migration with no DI, serialization, endpoint, or persisted-contract changes.

Migration away from IWords and MultiDict

The compatibility layer is intentionally transitional:

1. Common Words, Webster, dwyl, Crossword Nexus, and Peter Broda implement the native ILexicalSource acquisition path with pinned versions, cancellation, validation, and immutable source descriptors. Peter Broda reads its packaged local artifact; the other third-party sources use pinned HTTP artifacts. Production composition publishes one singleton catalog provider, view provider, and Letter Golf index provider.
2. LetterGolfGenerator consumes LetterGolfCatalogIndex directly. The keyed Consts.CURRENT_WORD_LIST registration now resolves CatalogWordsAdapter only for endpoint and play-validation callers that still require the IWords shape.
3. LegacyWordsLexicalSource remains a test/migration utility for exposing an old IWords source to WordCatalogBuilder; it is not part of production source composition. MultiDict must not become a catalog source because that would preserve duplicate aggregation and erase provenance.
4. NexusLexicalSource is registered to preserve the existing Crossword compatibility contract. The deterministic Phase 1 integration test is proof of shared snapshot identity, not production Reverse Cross wiring or puzzle generation.
5. After all validation paths have migrated, remove MultiDict, AbstractWords, IWords, and both compatibility adapters, or isolate any genuinely necessary compatibility surface away from new game code.

The Letter Golf production composition cutover and Phase 1 verification are complete. Source-license declarations and the completed dwyl/Moby provenance review are documented in Third-Party Notices. That inventory records the project's review; it is not a general legal-clearance claim for new sources or uses. No architectural milestone authorizes a merge, deployment, or cloud mutation by itself.