Skip to content

Data Layout and Persistence

Status: Current architecture on branch jm/string-repetition at a12ff3b; not a statement of production deployment state.

Audience: Backend engineers, reviewers, operators investigating storage, and maintainers planning compatible migrations.

Scope: In-memory lexical ownership, identity forms, Reverse Cross puzzle and play schemas, and the Cosmos DB partition and retention model. Product rules remain in the game specifications.

Reading guide

This page describes how the current branch works. The Compact Identities and Storage Plan preserves the measurements, phase decisions, and implementation record that led here. The Lexical and Game Architecture owns the broader source-to-game responsibility model, while the Reverse Cross specification owns player behavior.

Three identity scopes are intentionally different:

Identity Representation Scope May cross a boundary?
CatalogEntryHandle Dense integer ordinal One exact catalog snapshot No
LexicalId Fixed 32-byte SHA-256 value Stable lexical content Yes, after canonical text conversion
Reverse Cross entry ordinal Integer position in one puzzle entry bank One persisted puzzle Yes, only inside that puzzle's versioned payload

An ordinal is compact because its owner supplies the meaning. It is not a shorter global ID.

In-memory ownership

One immutable catalog snapshot owns lexical entries and compact metadata. Views retain eligible entry references and snapshot membership; game indexes retain local ordinals or postings rather than another copy of the corpus.

flowchart LR
    Sources["Pinned lexical sources"] --> Normalize["Normalize and merge"]
    Normalize --> Snapshot["WordCatalogSnapshot<br/>entries + metadata registry"]
    Snapshot --> LGView["Letter Golf view<br/>membership + entry references"]
    Snapshot --> RCView["Reverse Cross view<br/>membership + entry references"]
    LGView --> LGIndex["Signature index<br/>view-local ordinals"]
    RCView --> RCIndex["Fill index<br/>postings + local ordinals"]
    Snapshot --> Boundary["API, persistence, export,<br/>fixtures, diagnostics"]
    Boundary --> TextId["lex: + 64 lowercase hex characters"]

The stable lexical identity is

\[ \operatorname{LexicalId}(l,g) = \operatorname{SHA256}(\operatorname{UTF8}(l \mathbin{\|} \mathtt{NUL} \mathbin{\|} g)) \]

where \(l\) is the canonical language and \(g\) is the normalized grid form. Internally, the digest is a 32-byte value type. At public and persisted boundaries it is formatted as lex: followed by 64 lowercase hexadecimal characters. This preserves existing IDs without retaining one 68-character string per catalog entry.

Snapshot handles obey a stricter invariant:

\[ \operatorname{resolve}(S_2, h_{S_1}) \text{ is invalid whenever } S_1 \ne S_2 \]

even when both snapshots describe equivalent content. This prevents a dense ordinal from silently acquiring meaning under a different owner.

Measured retained-memory change

The table uses the same pinned sources, profile set, .NET 10 Release configuration, and production-size measurement harness across phases. Values are forced-GC stage deltas, not a process memory limit or a production service-level objective.

Stage Baseline, 2026-09-15 Current compact layout, 2026-09-17 Change
Shared catalog 599.5 MiB 392.9 MiB -206.6 MiB (-34.5%)
Letter Golf index 198.4 MiB 167.4 MiB -31.0 MiB (-15.6%)
Reverse Cross crossword view 54.2 MiB 8.7 MiB -45.5 MiB (-83.9%)
Reverse Cross fill index 36.2 MiB 16.2 MiB -20.0 MiB (-55.2%)
Sum of stage deltas 888.3 MiB 585.2 MiB -303.1 MiB (-34.1%)

The aggregate comparison is

\[ \Delta M = 585.2\ \text{MiB} - 888.3\ \text{MiB} = -303.1\ \text{MiB} \]

and the relative reduction is

\[ \frac{303.1}{888.3} \times 100\% \approx 34.1\%. \]

The catalog improvement came in three measured steps: removing low-risk retained duplication, replacing retained textual lexical IDs, and normalizing provenance and category metadata.

xychart-beta
    title "Retained memory by construction stage (MiB)"
    x-axis ["Catalog", "Letter Golf", "RC view", "RC fill"]
    y-axis "MiB" 0 --> 650
    bar [599.5, 198.4, 54.2, 36.2]
    bar [392.9, 167.4, 8.7, 16.2]

The first series is the 2026-09-15 baseline and the second is the 2026-09-17 compact layout. Cold construction timings varied between workstation runs and are evidence, not latency targets; the phase record retains those timings and the generation-probe outcomes.

Reverse Cross structural identity

Reverse Cross derives slots from the public mask in one canonical order. Puzzle and play schema v2 use that order as the owner of positional vectors.

flowchart TD
    Mask["rows, columns,<br/>blockedCellIndexes"] --> Derive["Derive maximal slots"]
    Derive --> Across["Across slots<br/>row-major"]
    Derive --> Down["Down slots<br/>column-major"]
    Across --> Order["Canonical slot order"]
    Down --> Order
    Bank["Entry bank<br/>puzzle-local ordinals"] --> Vector["entryBySlot vector"]
    Order --> Vector
    Vector --> Expand["Runtime placements<br/>slot ID + textual entry ID"]

For \(S\) canonical slots, a valid solution vector has exactly \(S\) positions:

\[ \mathbf{e} = (e_0, e_1, \ldots, e_{S-1}), \qquad 0 \le e_i < E \]

where \(E\) is the entry-bank size. A completed solution uses every required entry exactly once. An in-progress play uses null for an unfilled slot. Readers reject wrong vector lengths, out-of-range ordinals, duplicate completed assignments, and payloads that mix schema shapes.

Puzzle document schemas

New Reverse Cross puzzle entities default to schema v2. The mapper reads both v1 and v2 and rejects unknown, malformed, or mixed representations.

Concern Schema v1 Schema v2
Public mask Object per cell in cells blockedCellIndexes plus dimensions
Entry spelling displayForm, gridForm, and graphemes displayForm plus exact graphemes; grid form is reconstructed
Private solved grid Stored cell objects Reconstructed from entry bank and assignment
Private placements Objects containing textual slot and entry IDs entryBySlot entry-ordinal vector
Structural meaning Repeated in strings and objects Derived from canonical ordering rules

For a row-major cell index \(i\) on a board with \(C\) columns,

\[ \operatorname{row}(i) = \left\lfloor \frac{i}{C} \right\rfloor, \qquad \operatorname{column}(i) = i \bmod C. \]

That relationship lets v2 store only blocked positions while reconstructing every playable position from dimensions and the mask.

Why the structure changed

The 2026-09-15 read-only production audit measured 144 Reverse Cross puzzle documents. The dominant nested sections were structural repetition rather than the duplicated top-level Cosmos ID.

Section Sampled bytes Share of Reverse Cross puzzle JSON
Public entry bank 340,177 30.19%
Solution placements 237,382 21.07%
Solved cells 214,815 19.07%
Public cells 122,671 10.89%
Generation metadata 73,663 6.54%
Evaluation-report reference 9,112 0.81%

Property names occupied 392,415 bytes in those puzzle documents, including 312,236 repeated bytes inside individual documents. Schema v2 therefore removes repeated object structures while retaining readable top-level contracts; it does not abbreviate every property name.

Play document schemas

The shared PlayEntity envelope remains unchanged. Only its optional Reverse Cross payload is versioned.

Concern Schema v1 Schema v2
Board state Placement objects with textual slot and entry IDs Nullable entryBySlot ordinal vector
Puzzle compatibility puzzleSchemaVersion and version identities Same
Timing Revision, active segment, pause, and accumulated duration fields Same
Runtime model Expanded textual placements Expanded textual placements

The runtime model deliberately remains expressive. Compaction is a persistence-boundary concern: the storage mapper expands either schema to the same placement model before gameplay, validation, stats, merge, or export logic runs.

Play v2 writes are controlled by ReverseCrossPlayStorageV2. A disabled flag or feature-evaluation failure writes v1. Puzzle v2 writes are not controlled by that flag, so rollback must target a build that already reads both puzzle schemas.

flowchart LR
    V1Puzzle["Puzzle v1"] --> PuzzleReader["Dual-schema puzzle reader"]
    V2Puzzle["Puzzle v2"] --> PuzzleReader
    V1Play["Play v1"] --> PlayReader["Dual-schema play reader"]
    V2Play["Play v2"] --> PlayReader
    PuzzleReader --> Domain["Common runtime model"]
    PlayReader --> Domain
    Domain --> PuzzleWriter["Puzzle writer: v2"]
    Domain --> Gate{"ReverseCrossPlayStorageV2"}
    Gate -->|false or evaluation failure| V1Writer["Play writer: v1"]
    Gate -->|true| V2Writer["Play writer: v2"]

Cosmos DB layout

Terraform remains authoritative for container partition paths, indexing, and default TTL. Composite keys remain readable strings because they support direct point reads and operational diagnosis; the measured duplicate top-level identity was too small to justify changing partition strategy.

Container Partition key ID convention Retention
puzzles /puzzleKey id == puzzleKey No default expiry; Practice writes per-item 30-day TTL
plays /playKey id == playKey; the key joins the puzzle and user IDs 180-day default TTL
players /userId id == userId No expiry
groups metadata /groupId id == groupId No expiry
groups standings /groupId Composite group, standings, and puzzle key Per-item 7-day TTL

Cosmos point operations address an item with both its ID and partition-key value. Keeping those values equal is intentional even though JSON contains both properties. The production audit measured 20,942 duplicated identity bytes across sampled Reverse Cross puzzles and plays, about 1.50% of their 1,394,823 combined JSON bytes. Changing partition-key paths would require new containers and a data migration, so the branch retains the current keys.

Puzzle bodies and Reverse Cross evaluation detail branches are excluded from indexing because they are read as documents rather than queried by nested content. Identity fields remain indexed.

Compatibility and rollout rules

  1. Keep readers compatible before changing writers.
  2. Reject mixed or unknown schemas instead of guessing.
  3. Preserve canonical lexical text, catalog fingerprints, and replay identities at boundaries.
  4. Keep snapshot-local and puzzle-local ordinals inside their owning payloads.
  5. Roll back to a dual-reader build; an older v1-only build cannot read newly persisted v2 puzzles.
  6. Treat production deployment and feature state as operational facts to verify, not facts implied by this branch snapshot.

Stats recalculation, account merge, and player export accept both puzzle and play schema versions. Exports are additive: consumers can distinguish and preserve either representation without losing the versioned identity needed for comparison.

Measurement provenance

The retained-memory results came from opt-in production-source tests on a workstation, using pinned source versions and forced-GC stage measurements. The persistence figures came from a read-only Azure CLI audit on 2026-09-15; it consumed 51.34 RU, emitted aggregate measurements, retained no document bodies, and performed no writes. They are dated observations rather than forecasts.

Use the Lexical Memory Analysis Runbook to reproduce heap captures. Use the Compact Identities and Storage Plan for per-phase timing, allocation, compatibility, and decision evidence.