Skip to content

Lexical Memory Analysis Runbook

Status: Active measurement procedure; it does not authorize production load or Phase 5 work.

Audience: Backend and platform engineers reproducing retained-memory evidence.

Purpose and decision boundary

This runbook records the tools and methods used to evaluate compact lexical storage after Phase 4. It supports the evidence gate in the Compact Identities and Storage Plan, Phase 5. It does not authorize a Phase 5 implementation or a production load exercise. The Data Layout and Persistence reference summarizes the current layout; this runbook retains the capture and attribution method.

The decision requires two independent findings:

  1. The targeted lexical forms and grapheme structures can plausibly remove at least 15% and about 60 MiB of catalog retention at current scale.
  2. The actual 2 GiB Azure container has a named operational or cost problem that those savings address.

The local heap evidence satisfies the first condition. Azure measurements must establish the second condition before a compact form or grapheme representation is prototyped.

Recorded Phase 4 heap evidence

The two dumps used independent .NET 10 Release test processes, the same pinned production lexical sources, and catalog version word-catalog-v1:fbb7a50a9ef85969f591d902b1cef8c88c2a52fb33eca46468e22ed2eb465d82. Each dump was captured after a forced full collection.

Checkpoint Retained root or root union Retained bytes
After catalog publication WordCatalogSnapshot graph 379.6 MiB
After derived indexes Union of catalog, Letter Golf, views, and fill indexes 566.1 MiB

The field-level ClrMD analysis produced the same catalog-owned counts in both dumps:

Category Objects or references Bytes
LexicalEntry objects 802,131 89,838,672
Display-form strings 802,131 34,499,250
Grid-form strings 802,131 34,499,250
Shared display/grid string objects 22 638
Distinct display/grid strings 1,604,240 68,997,862
Per-entry grapheme arrays 802,131 86,660,616
Grapheme reference slots 8,426,184 67,409,472
Distinct grapheme strings 36 864
Compact source-score arrays 605,560 24,222,400
Category-ID arrays 473 12,794

Distinct display/grid strings plus per-entry grapheme arrays retain 155,658,478 bytes, or 148.45 MiB. That is 39.1% of the 379.6 MiB catalog root and clears the local Phase 5 threshold. The grapheme-array object sizes already include their reference slots; do not add the reported 67,409,472 slot bytes again.

This result establishes an addressable category, not achievable savings. A prototype would still need offsets, lengths, tables, validation metadata, and access machinery. It must prove its net reduction and preserve behavior.

Capture harness

The opt-in checkpoint test is ProductionSources_CaptureHeapAtRequestedCheckpoint in MiniGamesBackEnd.Tests/Words/Catalog/ProductionWordCatalogMeasurementTests.cs. It accepts these environment variables:

  • MINIGAMES_RUN_PRODUCTION_CATALOG_MEASUREMENT=1 enables production-source measurement tests.
  • MINIGAMES_PRODUCTION_CATALOG_HEAP_CHECKPOINT=after-catalog captures the snapshot before game views and indexes are built.
  • MINIGAMES_PRODUCTION_CATALOG_HEAP_CHECKPOINT=after-derived-indexes retains the Letter Golf index, Reverse Cross crossword and generated views, and both fill indexes.
  • MINIGAMES_PRODUCTION_CATALOG_HEAP_DUMP_PATH selects the dump output path.

Run each checkpoint in a separate process. Reusing one process would leave previously constructed static and dependency-injection roots reachable and invalidate the catalog-only comparison.

$test = 'MiniGamesBackEnd.Tests.ProductionWordCatalogMeasurementTests.ProductionSources_CaptureHeapAtRequestedCheckpoint'
$env:MINIGAMES_RUN_PRODUCTION_CATALOG_MEASUREMENT = '1'
$env:MINIGAMES_PRODUCTION_CATALOG_HEAP_CHECKPOINT = 'after-catalog'
$env:MINIGAMES_PRODUCTION_CATALOG_HEAP_DUMP_PATH = '.local\measurements\phase5-evidence\phase4-after-catalog.dmp'
dotnet test .\MiniGamesBackEnd.Tests\MiniGamesBackEnd.Tests.csproj -c Release --filter "FullyQualifiedName=$test"
Remove-Item Env:MINIGAMES_PRODUCTION_CATALOG_HEAP_CHECKPOINT
Remove-Item Env:MINIGAMES_PRODUCTION_CATALOG_HEAP_DUMP_PATH
Remove-Item Env:MINIGAMES_RUN_PRODUCTION_CATALOG_MEASUREMENT

Repeat in a new shell process with after-derived-indexes and a distinct dump path. The test calls dotnet-dump collect --type Heap against its own process after forcing a full collection. It keeps the requested roots alive through capture.

SOS retained-graph analysis

Open a dump with the same major dotnet-dump tool version used to capture it:

dotnet-dump analyze .local\measurements\phase5-evidence\phase4-after-catalog.dmp

Use these SOS commands inside the analyzer:

dumpheap -stat
dumpheap -type MiniGamesBackEnd.Words.Catalog.WordCatalogSnapshot
objsize -stat <snapshot-address>

For the derived checkpoint, locate and inspect the snapshot, LetterGolfCatalogIndex, WordCatalogView, and ReverseCrossFillIndex roots. dumpheap -stat is a type census of the whole live managed heap. objsize -stat walks objects reachable from one root and groups the graph by type.

Do not sum objsize totals from roots that share objects. Views and game indexes retain snapshot entries by reference, so independently walking each root counts shared entries, strings, and arrays more than once. Use the snapshot size for the catalog-only checkpoint and an object-identity union for the combined checkpoint. The difference between those two independent root unions is the derived-structure increment; it is not the sum of each root report.

Dump-file size is also not managed retained size. A dump contains runtime, module, thread, and diagnostic state in addition to managed objects.

ClrMD ownership attribution

SOS identifies large types and retained graphs, but type totals cannot answer which field owns a shared string or array. The ignored local analyzer at .local/heap-analyzer uses Microsoft.Diagnostics.Runtime to enumerate each valid LexicalEntry, read its backing fields, and de-duplicate object addresses in HashSet<ulong> collections.

dotnet run --project .local\heap-analyzer\HeapAnalyzer.csproj -c Release -- .local\measurements\phase5-evidence\phase4-after-catalog.dmp
dotnet run --project .local\heap-analyzer\HeapAnalyzer.csproj -c Release -- .local\measurements\phase5-evidence\phase4-after-derived-indexes.dmp

The analyzer separately tracks display strings, grid strings, grapheme arrays, distinct grapheme strings, compact score arrays, and category arrays. It also computes intersections by object address. This distinguishes equal values from shared instances and avoids assigning one object to multiple owners accidentally.

Keep analysis code local unless it becomes a maintained regression tool. Its current project references the ClrMD assembly shipped with the installed dotnet-dump; update that path when the tool version changes.

Interpreting the measurements

  • A forced-GC stage delta compares process-wide managed memory before and after construction. It is useful for repeated before/after experiments but is not an ownership graph.
  • SOS retained size describes reachability from a root. It is the correct basis for ownership questions when roots are unioned by object identity.
  • ClrMD field attribution describes selected portions of that graph. Categories may overlap unless the analyzer explicitly computes unions and intersections.
  • Process working set includes native allocations, JIT output, mapped files, stacks, and runtime overhead. It is the relevant container-pressure signal but cannot be derived from managed heap totals.
  • Peak construction memory and settled memory answer different operational questions. Catalog refresh can retain the old snapshot while allocating and publishing the new graph.
  • Compare only runs with the same source versions, Release configuration, runtime, checkpoint, view/profile set, and warm-up sequence.

Azure 2 GiB container measurement

The production Container App uses one CPU, 2 GiB of memory, zero to two replicas, and a 30-minute scale-down cooldown. Read-only historical analysis should query these one-minute Azure Monitor metrics in the Microsoft.App/containerApps namespace:

Metric Aggregations Use
WorkingSetBytes Average, maximum Settled and peak resident memory
UsageNanoCores Average, maximum CPU saturation during construction
Requests Total Correlate memory and CPU with request load
Replicas Maximum Separate scale events and replica counts
RestartCount Maximum Detect instability or memory-pressure restarts

Group WorkingSetBytes, UsageNanoCores, and RestartCount by revisionName and podName when possible. Record the query range, interval, aggregation, revision, and replica identity with every result. A 2 GiB limit leaves the Phase 4 plan's required 500 MiB headroom only when peak working set stays at or below 1.5 GiB.

Historical metrics can establish ordinary peaks and settled values but cannot prove catalog-only or refresh-overlap causality. A deliberate scale-from-zero, warm generation, concurrent request, or catalog-refresh exercise changes production load and requires explicit approval. For an approved exercise, record:

  1. scale-from-zero request latency and success;
  2. peak and settled working set by replica;
  3. peak CPU and time at saturation;
  4. managed committed and retained memory plus GC counts and pause time from application telemetry;
  5. concurrent lexical-request latency and failures during publication;
  6. old/new snapshot overlap and derived-index reconstruction duration; and
  7. restarts, replica changes, Cosmos throttles, and request volume in the same interval.

Keep Phase 5 deferred when the container retains at least 500 MiB of peak headroom, refresh and scale-from-zero remain reliable, and no lower memory tier or other cost target is viable. Clearing the local 15%/60 MiB gate alone is insufficient.

Read-only production observation - 2026-09-17

Azure Monitor and Application Insights were queried for the seven-day interval from 2026-09-10 22:15 UTC through 2026-09-17 22:15 UTC. No resource configuration, traffic, or state was changed.

  • Per-pod hourly maximum working set reached 1,675,915,264 bytes, or 1,598.3 MiB. The 2 GiB allocation therefore had 449.7 MiB of peak headroom, 50.3 MiB below the Phase 4 target.
  • Per-pod CPU reached 1,017,116,218 nanocores, approximately 101.7% of the nominal one-core allocation. RestartCount remained zero throughout the queried interval.
  • Application Insights contained 27 sampled, successful WordCatalog.Build spans. Their p50, p95, and maximum durations were 25.85, 32.52, and 32.91 seconds. Every recorded span had refreshSources=False and a distinct replica name, identifying initial construction rather than in-process refresh.
  • Correlated request telemetry included eight warm-up requests and four daily-pregeneration requests ending with HTTP 499 while their catalog build spans completed successfully. Successful warm-up requests had a 28.97-second p95. Sampling means these are observed records, not a complete event count.

This history establishes two named operational concerns: the 500 MiB peak-headroom target is missed, and scale-from-zero catalog construction contributes tens of seconds to request latency and client cancellation. It does not show refresh overlap, GC pauses, or how much of the managed 148.45 MiB candidate category would leave the resident working set after compaction. It also does not establish that form/grapheme compaction would improve the CPU-bound cold build; the immutable artifact work in Phase 6 is a more direct candidate for that latency.

Phase 5 therefore remains deferred. A bounded prototype or approved representative deployment comparison must demonstrate at least roughly 60 MiB of net retained savings, preserve behavior, and restore meaningful working-set headroom before the representation is adopted.

Artifact security and cleanup

Heap dumps can contain source words, configuration values, tokens, cookies, request data, and other process memory. Treat every dump as sensitive:

  • write dumps only under the ignored .local/ tree or another access-controlled location;
  • never commit, attach, or upload a dump to an unapproved service;
  • do not place dumps in the Markdown export;
  • publish aggregate counts and byte totals, not extracted object contents;
  • remove dumps and temporary analyzer output when the investigation and review window ends; and
  • rotate any credential found in a dump rather than relying on deletion of the dump.

Reproduction checklist

  1. Record the commit, runtime, dotnet-dump version, OS, source fingerprints, and catalog version.
  2. Build and test in Release mode.
  3. Capture catalog-only and derived checkpoints in independent processes.
  4. Record whole-heap type statistics and object-identity retained-root unions.
  5. Run field-level ClrMD attribution on both dumps and compare address-de-duplicated categories.
  6. Query Azure metrics without changing resources or generating load.
  7. Obtain explicit approval before any deliberate production exercise.
  8. Compare the evidence with both Phase 5 gates and record the decision.
  9. Delete sensitive local artifacts after the review window.