Data Layout and Persistence¶
Status: Current architecture on branch jm/string-repetition at a12ff3b; not a statement of
production deployment state.
Audience: Backend engineers, reviewers, operators investigating storage, and maintainers planning compatible migrations.
Scope: In-memory lexical ownership, identity forms, Reverse Cross puzzle and play schemas, and the Cosmos DB partition and retention model. Product rules remain in the game specifications.
Reading guide¶
This page describes how the current branch works. The Compact Identities and Storage Plan preserves the measurements, phase decisions, and implementation record that led here. The Lexical and Game Architecture owns the broader source-to-game responsibility model, while the Reverse Cross specification owns player behavior.
Three identity scopes are intentionally different:
| Identity | Representation | Scope | May cross a boundary? |
|---|---|---|---|
CatalogEntryHandle |
Dense integer ordinal | One exact catalog snapshot | No |
LexicalId |
Fixed 32-byte SHA-256 value | Stable lexical content | Yes, after canonical text conversion |
| Reverse Cross entry ordinal | Integer position in one puzzle entry bank | One persisted puzzle | Yes, only inside that puzzle's versioned payload |
An ordinal is compact because its owner supplies the meaning. It is not a shorter global ID.
In-memory ownership¶
One immutable catalog snapshot owns lexical entries and compact metadata. Views retain eligible entry references and snapshot membership; game indexes retain local ordinals or postings rather than another copy of the corpus.
flowchart LR
Sources["Pinned lexical sources"] --> Normalize["Normalize and merge"]
Normalize --> Snapshot["WordCatalogSnapshot<br/>entries + metadata registry"]
Snapshot --> LGView["Letter Golf view<br/>membership + entry references"]
Snapshot --> RCView["Reverse Cross view<br/>membership + entry references"]
LGView --> LGIndex["Signature index<br/>view-local ordinals"]
RCView --> RCIndex["Fill index<br/>postings + local ordinals"]
Snapshot --> Boundary["API, persistence, export,<br/>fixtures, diagnostics"]
Boundary --> TextId["lex: + 64 lowercase hex characters"]
The stable lexical identity is
where \(l\) is the canonical language and \(g\) is the normalized grid form. Internally, the digest is a
32-byte value type. At public and persisted boundaries it is formatted as lex: followed by 64
lowercase hexadecimal characters. This preserves existing IDs without retaining one 68-character
string per catalog entry.
Snapshot handles obey a stricter invariant:
even when both snapshots describe equivalent content. This prevents a dense ordinal from silently acquiring meaning under a different owner.
Measured retained-memory change¶
The table uses the same pinned sources, profile set, .NET 10 Release configuration, and production-size measurement harness across phases. Values are forced-GC stage deltas, not a process memory limit or a production service-level objective.
| Stage | Baseline, 2026-09-15 | Current compact layout, 2026-09-17 | Change |
|---|---|---|---|
| Shared catalog | 599.5 MiB | 392.9 MiB | -206.6 MiB (-34.5%) |
| Letter Golf index | 198.4 MiB | 167.4 MiB | -31.0 MiB (-15.6%) |
| Reverse Cross crossword view | 54.2 MiB | 8.7 MiB | -45.5 MiB (-83.9%) |
| Reverse Cross fill index | 36.2 MiB | 16.2 MiB | -20.0 MiB (-55.2%) |
| Sum of stage deltas | 888.3 MiB | 585.2 MiB | -303.1 MiB (-34.1%) |
The aggregate comparison is
and the relative reduction is
The catalog improvement came in three measured steps: removing low-risk retained duplication, replacing retained textual lexical IDs, and normalizing provenance and category metadata.
xychart-beta
title "Retained memory by construction stage (MiB)"
x-axis ["Catalog", "Letter Golf", "RC view", "RC fill"]
y-axis "MiB" 0 --> 650
bar [599.5, 198.4, 54.2, 36.2]
bar [392.9, 167.4, 8.7, 16.2]
The first series is the 2026-09-15 baseline and the second is the 2026-09-17 compact layout. Cold construction timings varied between workstation runs and are evidence, not latency targets; the phase record retains those timings and the generation-probe outcomes.
Reverse Cross structural identity¶
Reverse Cross derives slots from the public mask in one canonical order. Puzzle and play schema v2 use that order as the owner of positional vectors.
flowchart TD
Mask["rows, columns,<br/>blockedCellIndexes"] --> Derive["Derive maximal slots"]
Derive --> Across["Across slots<br/>row-major"]
Derive --> Down["Down slots<br/>column-major"]
Across --> Order["Canonical slot order"]
Down --> Order
Bank["Entry bank<br/>puzzle-local ordinals"] --> Vector["entryBySlot vector"]
Order --> Vector
Vector --> Expand["Runtime placements<br/>slot ID + textual entry ID"]
For \(S\) canonical slots, a valid solution vector has exactly \(S\) positions:
where \(E\) is the entry-bank size. A completed solution uses every required entry exactly once. An
in-progress play uses null for an unfilled slot. Readers reject wrong vector lengths, out-of-range
ordinals, duplicate completed assignments, and payloads that mix schema shapes.
Puzzle document schemas¶
New Reverse Cross puzzle entities default to schema v2. The mapper reads both v1 and v2 and rejects unknown, malformed, or mixed representations.
| Concern | Schema v1 | Schema v2 |
|---|---|---|
| Public mask | Object per cell in cells |
blockedCellIndexes plus dimensions |
| Entry spelling | displayForm, gridForm, and graphemes |
displayForm plus exact graphemes; grid form is reconstructed |
| Private solved grid | Stored cell objects | Reconstructed from entry bank and assignment |
| Private placements | Objects containing textual slot and entry IDs | entryBySlot entry-ordinal vector |
| Structural meaning | Repeated in strings and objects | Derived from canonical ordering rules |
For a row-major cell index \(i\) on a board with \(C\) columns,
That relationship lets v2 store only blocked positions while reconstructing every playable position from dimensions and the mask.
Why the structure changed¶
The 2026-09-15 read-only production audit measured 144 Reverse Cross puzzle documents. The dominant nested sections were structural repetition rather than the duplicated top-level Cosmos ID.
| Section | Sampled bytes | Share of Reverse Cross puzzle JSON |
|---|---|---|
| Public entry bank | 340,177 | 30.19% |
| Solution placements | 237,382 | 21.07% |
| Solved cells | 214,815 | 19.07% |
| Public cells | 122,671 | 10.89% |
| Generation metadata | 73,663 | 6.54% |
| Evaluation-report reference | 9,112 | 0.81% |
Property names occupied 392,415 bytes in those puzzle documents, including 312,236 repeated bytes inside individual documents. Schema v2 therefore removes repeated object structures while retaining readable top-level contracts; it does not abbreviate every property name.
Play document schemas¶
The shared PlayEntity envelope remains unchanged. Only its optional Reverse Cross payload is
versioned.
| Concern | Schema v1 | Schema v2 |
|---|---|---|
| Board state | Placement objects with textual slot and entry IDs | Nullable entryBySlot ordinal vector |
| Puzzle compatibility | puzzleSchemaVersion and version identities |
Same |
| Timing | Revision, active segment, pause, and accumulated duration fields | Same |
| Runtime model | Expanded textual placements | Expanded textual placements |
The runtime model deliberately remains expressive. Compaction is a persistence-boundary concern: the storage mapper expands either schema to the same placement model before gameplay, validation, stats, merge, or export logic runs.
Play v2 writes are controlled by ReverseCrossPlayStorageV2. A disabled flag or feature-evaluation
failure writes v1. Puzzle v2 writes are not controlled by that flag, so rollback must target a build
that already reads both puzzle schemas.
flowchart LR
V1Puzzle["Puzzle v1"] --> PuzzleReader["Dual-schema puzzle reader"]
V2Puzzle["Puzzle v2"] --> PuzzleReader
V1Play["Play v1"] --> PlayReader["Dual-schema play reader"]
V2Play["Play v2"] --> PlayReader
PuzzleReader --> Domain["Common runtime model"]
PlayReader --> Domain
Domain --> PuzzleWriter["Puzzle writer: v2"]
Domain --> Gate{"ReverseCrossPlayStorageV2"}
Gate -->|false or evaluation failure| V1Writer["Play writer: v1"]
Gate -->|true| V2Writer["Play writer: v2"]
Cosmos DB layout¶
Terraform remains authoritative for container partition paths, indexing, and default TTL. Composite keys remain readable strings because they support direct point reads and operational diagnosis; the measured duplicate top-level identity was too small to justify changing partition strategy.
| Container | Partition key | ID convention | Retention |
|---|---|---|---|
puzzles |
/puzzleKey |
id == puzzleKey |
No default expiry; Practice writes per-item 30-day TTL |
plays |
/playKey |
id == playKey; the key joins the puzzle and user IDs |
180-day default TTL |
players |
/userId |
id == userId |
No expiry |
groups metadata |
/groupId |
id == groupId |
No expiry |
groups standings |
/groupId |
Composite group, standings, and puzzle key | Per-item 7-day TTL |
Cosmos point operations address an item with both its ID and partition-key value. Keeping those values equal is intentional even though JSON contains both properties. The production audit measured 20,942 duplicated identity bytes across sampled Reverse Cross puzzles and plays, about 1.50% of their 1,394,823 combined JSON bytes. Changing partition-key paths would require new containers and a data migration, so the branch retains the current keys.
Puzzle bodies and Reverse Cross evaluation detail branches are excluded from indexing because they are read as documents rather than queried by nested content. Identity fields remain indexed.
Compatibility and rollout rules¶
- Keep readers compatible before changing writers.
- Reject mixed or unknown schemas instead of guessing.
- Preserve canonical lexical text, catalog fingerprints, and replay identities at boundaries.
- Keep snapshot-local and puzzle-local ordinals inside their owning payloads.
- Roll back to a dual-reader build; an older v1-only build cannot read newly persisted v2 puzzles.
- Treat production deployment and feature state as operational facts to verify, not facts implied by this branch snapshot.
Stats recalculation, account merge, and player export accept both puzzle and play schema versions. Exports are additive: consumers can distinguish and preserve either representation without losing the versioned identity needed for comparison.
Measurement provenance¶
The retained-memory results came from opt-in production-source tests on a workstation, using pinned source versions and forced-GC stage measurements. The persistence figures came from a read-only Azure CLI audit on 2026-09-15; it consumed 51.34 RU, emitted aggregate measurements, retained no document bodies, and performed no writes. They are dated observations rather than forecasts.
Use the Lexical Memory Analysis Runbook to reproduce heap captures. Use the Compact Identities and Storage Plan for per-phase timing, allocation, compatibility, and decision evidence.