Lexical Memory Analysis Runbook¶
Status: Active measurement procedure; it does not authorize production load or Phase 5 work.
Audience: Backend and platform engineers reproducing retained-memory evidence.
Purpose and decision boundary¶
This runbook records the tools and methods used to evaluate compact lexical storage after Phase 4. It supports the evidence gate in the Compact Identities and Storage Plan, Phase 5. It does not authorize a Phase 5 implementation or a production load exercise. The Data Layout and Persistence reference summarizes the current layout; this runbook retains the capture and attribution method.
The decision requires two independent findings:
- The targeted lexical forms and grapheme structures can plausibly remove at least 15% and about 60 MiB of catalog retention at current scale.
- The actual 2 GiB Azure container has a named operational or cost problem that those savings address.
The local heap evidence satisfies the first condition. Azure measurements must establish the second condition before a compact form or grapheme representation is prototyped.
Recorded Phase 4 heap evidence¶
The two dumps used independent .NET 10 Release test processes, the same pinned production lexical
sources, and catalog version
word-catalog-v1:fbb7a50a9ef85969f591d902b1cef8c88c2a52fb33eca46468e22ed2eb465d82.
Each dump was captured after a forced full collection.
| Checkpoint | Retained root or root union | Retained bytes |
|---|---|---|
| After catalog publication | WordCatalogSnapshot graph |
379.6 MiB |
| After derived indexes | Union of catalog, Letter Golf, views, and fill indexes | 566.1 MiB |
The field-level ClrMD analysis produced the same catalog-owned counts in both dumps:
| Category | Objects or references | Bytes |
|---|---|---|
LexicalEntry objects |
802,131 | 89,838,672 |
| Display-form strings | 802,131 | 34,499,250 |
| Grid-form strings | 802,131 | 34,499,250 |
| Shared display/grid string objects | 22 | 638 |
| Distinct display/grid strings | 1,604,240 | 68,997,862 |
| Per-entry grapheme arrays | 802,131 | 86,660,616 |
| Grapheme reference slots | 8,426,184 | 67,409,472 |
| Distinct grapheme strings | 36 | 864 |
| Compact source-score arrays | 605,560 | 24,222,400 |
| Category-ID arrays | 473 | 12,794 |
Distinct display/grid strings plus per-entry grapheme arrays retain 155,658,478 bytes, or 148.45 MiB. That is 39.1% of the 379.6 MiB catalog root and clears the local Phase 5 threshold. The grapheme-array object sizes already include their reference slots; do not add the reported 67,409,472 slot bytes again.
This result establishes an addressable category, not achievable savings. A prototype would still need offsets, lengths, tables, validation metadata, and access machinery. It must prove its net reduction and preserve behavior.
Capture harness¶
The opt-in checkpoint test is
ProductionSources_CaptureHeapAtRequestedCheckpoint in
MiniGamesBackEnd.Tests/Words/Catalog/ProductionWordCatalogMeasurementTests.cs. It accepts these
environment variables:
MINIGAMES_RUN_PRODUCTION_CATALOG_MEASUREMENT=1enables production-source measurement tests.MINIGAMES_PRODUCTION_CATALOG_HEAP_CHECKPOINT=after-catalogcaptures the snapshot before game views and indexes are built.MINIGAMES_PRODUCTION_CATALOG_HEAP_CHECKPOINT=after-derived-indexesretains the Letter Golf index, Reverse Cross crossword and generated views, and both fill indexes.MINIGAMES_PRODUCTION_CATALOG_HEAP_DUMP_PATHselects the dump output path.
Run each checkpoint in a separate process. Reusing one process would leave previously constructed static and dependency-injection roots reachable and invalidate the catalog-only comparison.
$test = 'MiniGamesBackEnd.Tests.ProductionWordCatalogMeasurementTests.ProductionSources_CaptureHeapAtRequestedCheckpoint'
$env:MINIGAMES_RUN_PRODUCTION_CATALOG_MEASUREMENT = '1'
$env:MINIGAMES_PRODUCTION_CATALOG_HEAP_CHECKPOINT = 'after-catalog'
$env:MINIGAMES_PRODUCTION_CATALOG_HEAP_DUMP_PATH = '.local\measurements\phase5-evidence\phase4-after-catalog.dmp'
dotnet test .\MiniGamesBackEnd.Tests\MiniGamesBackEnd.Tests.csproj -c Release --filter "FullyQualifiedName=$test"
Remove-Item Env:MINIGAMES_PRODUCTION_CATALOG_HEAP_CHECKPOINT
Remove-Item Env:MINIGAMES_PRODUCTION_CATALOG_HEAP_DUMP_PATH
Remove-Item Env:MINIGAMES_RUN_PRODUCTION_CATALOG_MEASUREMENT
Repeat in a new shell process with after-derived-indexes and a distinct dump path. The test calls
dotnet-dump collect --type Heap against its own process after forcing a full collection. It keeps
the requested roots alive through capture.
SOS retained-graph analysis¶
Open a dump with the same major dotnet-dump tool version used to capture it:
Use these SOS commands inside the analyzer:
dumpheap -stat
dumpheap -type MiniGamesBackEnd.Words.Catalog.WordCatalogSnapshot
objsize -stat <snapshot-address>
For the derived checkpoint, locate and inspect the snapshot, LetterGolfCatalogIndex,
WordCatalogView, and ReverseCrossFillIndex roots. dumpheap -stat is a type census of the whole
live managed heap. objsize -stat walks objects reachable from one root and groups the graph by
type.
Do not sum objsize totals from roots that share objects. Views and game indexes retain snapshot
entries by reference, so independently walking each root counts shared entries, strings, and arrays
more than once. Use the snapshot size for the catalog-only checkpoint and an object-identity union
for the combined checkpoint. The difference between those two independent root unions is the
derived-structure increment; it is not the sum of each root report.
Dump-file size is also not managed retained size. A dump contains runtime, module, thread, and diagnostic state in addition to managed objects.
ClrMD ownership attribution¶
SOS identifies large types and retained graphs, but type totals cannot answer which field owns a
shared string or array. The ignored local analyzer at .local/heap-analyzer uses
Microsoft.Diagnostics.Runtime to enumerate each valid LexicalEntry, read its backing fields,
and de-duplicate object addresses in HashSet<ulong> collections.
dotnet run --project .local\heap-analyzer\HeapAnalyzer.csproj -c Release -- .local\measurements\phase5-evidence\phase4-after-catalog.dmp
dotnet run --project .local\heap-analyzer\HeapAnalyzer.csproj -c Release -- .local\measurements\phase5-evidence\phase4-after-derived-indexes.dmp
The analyzer separately tracks display strings, grid strings, grapheme arrays, distinct grapheme strings, compact score arrays, and category arrays. It also computes intersections by object address. This distinguishes equal values from shared instances and avoids assigning one object to multiple owners accidentally.
Keep analysis code local unless it becomes a maintained regression tool. Its current project
references the ClrMD assembly shipped with the installed dotnet-dump; update that path when the
tool version changes.
Interpreting the measurements¶
- A forced-GC stage delta compares process-wide managed memory before and after construction. It is useful for repeated before/after experiments but is not an ownership graph.
- SOS retained size describes reachability from a root. It is the correct basis for ownership questions when roots are unioned by object identity.
- ClrMD field attribution describes selected portions of that graph. Categories may overlap unless the analyzer explicitly computes unions and intersections.
- Process working set includes native allocations, JIT output, mapped files, stacks, and runtime overhead. It is the relevant container-pressure signal but cannot be derived from managed heap totals.
- Peak construction memory and settled memory answer different operational questions. Catalog refresh can retain the old snapshot while allocating and publishing the new graph.
- Compare only runs with the same source versions, Release configuration, runtime, checkpoint, view/profile set, and warm-up sequence.
Azure 2 GiB container measurement¶
The production Container App uses one CPU, 2 GiB of memory, zero to two replicas, and a 30-minute
scale-down cooldown. Read-only historical analysis should query these one-minute Azure Monitor
metrics in the Microsoft.App/containerApps namespace:
| Metric | Aggregations | Use |
|---|---|---|
WorkingSetBytes |
Average, maximum | Settled and peak resident memory |
UsageNanoCores |
Average, maximum | CPU saturation during construction |
Requests |
Total | Correlate memory and CPU with request load |
Replicas |
Maximum | Separate scale events and replica counts |
RestartCount |
Maximum | Detect instability or memory-pressure restarts |
Group WorkingSetBytes, UsageNanoCores, and RestartCount by revisionName and podName when
possible. Record the query range, interval, aggregation, revision, and replica identity with every
result. A 2 GiB limit leaves the Phase 4 plan's required 500 MiB headroom only when peak working set
stays at or below 1.5 GiB.
Historical metrics can establish ordinary peaks and settled values but cannot prove catalog-only or refresh-overlap causality. A deliberate scale-from-zero, warm generation, concurrent request, or catalog-refresh exercise changes production load and requires explicit approval. For an approved exercise, record:
- scale-from-zero request latency and success;
- peak and settled working set by replica;
- peak CPU and time at saturation;
- managed committed and retained memory plus GC counts and pause time from application telemetry;
- concurrent lexical-request latency and failures during publication;
- old/new snapshot overlap and derived-index reconstruction duration; and
- restarts, replica changes, Cosmos throttles, and request volume in the same interval.
Keep Phase 5 deferred when the container retains at least 500 MiB of peak headroom, refresh and scale-from-zero remain reliable, and no lower memory tier or other cost target is viable. Clearing the local 15%/60 MiB gate alone is insufficient.
Read-only production observation - 2026-09-17¶
Azure Monitor and Application Insights were queried for the seven-day interval from 2026-09-10 22:15 UTC through 2026-09-17 22:15 UTC. No resource configuration, traffic, or state was changed.
- Per-pod hourly maximum working set reached 1,675,915,264 bytes, or 1,598.3 MiB. The 2 GiB allocation therefore had 449.7 MiB of peak headroom, 50.3 MiB below the Phase 4 target.
- Per-pod CPU reached 1,017,116,218 nanocores, approximately 101.7% of the nominal one-core
allocation.
RestartCountremained zero throughout the queried interval. - Application Insights contained 27 sampled, successful
WordCatalog.Buildspans. Their p50, p95, and maximum durations were 25.85, 32.52, and 32.91 seconds. Every recorded span hadrefreshSources=Falseand a distinct replica name, identifying initial construction rather than in-process refresh. - Correlated request telemetry included eight warm-up requests and four daily-pregeneration requests ending with HTTP 499 while their catalog build spans completed successfully. Successful warm-up requests had a 28.97-second p95. Sampling means these are observed records, not a complete event count.
This history establishes two named operational concerns: the 500 MiB peak-headroom target is missed, and scale-from-zero catalog construction contributes tens of seconds to request latency and client cancellation. It does not show refresh overlap, GC pauses, or how much of the managed 148.45 MiB candidate category would leave the resident working set after compaction. It also does not establish that form/grapheme compaction would improve the CPU-bound cold build; the immutable artifact work in Phase 6 is a more direct candidate for that latency.
Phase 5 therefore remains deferred. A bounded prototype or approved representative deployment comparison must demonstrate at least roughly 60 MiB of net retained savings, preserve behavior, and restore meaningful working-set headroom before the representation is adopted.
Artifact security and cleanup¶
Heap dumps can contain source words, configuration values, tokens, cookies, request data, and other process memory. Treat every dump as sensitive:
- write dumps only under the ignored
.local/tree or another access-controlled location; - never commit, attach, or upload a dump to an unapproved service;
- do not place dumps in the Markdown export;
- publish aggregate counts and byte totals, not extracted object contents;
- remove dumps and temporary analyzer output when the investigation and review window ends; and
- rotate any credential found in a dump rather than relying on deletion of the dump.
Reproduction checklist¶
- Record the commit, runtime,
dotnet-dumpversion, OS, source fingerprints, and catalog version. - Build and test in Release mode.
- Capture catalog-only and derived checkpoints in independent processes.
- Record whole-heap type statistics and object-identity retained-root unions.
- Run field-level ClrMD attribution on both dumps and compare address-de-duplicated categories.
- Query Azure metrics without changing resources or generating load.
- Obtain explicit approval before any deliberate production exercise.
- Compare the evidence with both Phase 5 gates and record the decision.
- Delete sensitive local artifacts after the review window.