Skip to content

Lexical Catalog Cold-Start and Compiled-Artifact Plan

Status: Deferred on 2026-09-17; separate from the Reverse Cross evaluator implementation

Scope: Production lexical acquisition, catalog construction, derived indexes, container cold start, immutable artifact delivery, memory layout, and measurement

Audience: Backend and platform engineers deciding whether to reopen compiled-artifact work.

The measurements in this document predate the compact in-memory implementation and remain the baseline for this deferred decision. For the current branch layout and later measurements, see Data Layout and Persistence.

Current Decision

Defer compiled-artifact implementation while the intended production operating model is to sustain enough regular use, or configure sufficient minimum capacity, that at least one backend replica remains warm. Scaling the backend to zero is not a desired long-term goal under that model, so this work does not currently solve a required user-facing cold-start target. It remains isolated from the later in-memory identity and Cosmos persistence compaction work and does not block that work.

There is a separate development and CI benefit: a test container could load the real production catalog from a verified immutable artifact instead of reacquiring and compiling all lexical sources on every run. Compilation would occur once per source, policy, and compiler fingerprint and the artifact could then be reused across container tests, substantially shortening a realistic test loop. Revisit this plan if repeated container-test compilation becomes material, or if replica startup, deployment readiness, source availability, or catalog reproducibility becomes a measured operational problem.

Current State

The four production lexical source files are not committed to this repository and do not appear in its Git history. Source adapters commit immutable upstream URLs and versions, then download the files over HTTPS when a process first requests the language catalog.

Pinned source artifact Raw size
common.json 0.01 MiB
dictionary_compact.json 21.42 MiB
words_dictionary.json 6.51 MiB
xwordlist.dict 7.92 MiB
Total 35.86 MiB

The InMemoryWordCatalogProvider coalesces concurrent builds inside one process, but the published snapshot and all derived views/indexes are process-local. Each new replica or restarted process pays the acquisition and build cost again. Readiness currently checks serving dependencies such as Cosmos DB but does not require lexical warm-up, so the first lexical request or explicit warm-up can trigger the work.

Container Apps does not meter request units. Cold lexical loading consumes network transfer, CPU, memory, latency, and billed replica time. Cosmos DB operations consume RUs; the dictionary downloads do not touch Cosmos DB. A future Azure Blob source would incur Blob transaction/storage/bandwidth charges, not Cosmos RUs.

Existing Measurements

The 2026-08-28 workstation measurement produced 767,019 merged entries and 378,789 Letter Golf entries. It observed:

  • first validation warm-up request: 17.49 seconds;
  • catalog build activity: 16.01 seconds;
  • warm catalog access: 0.065 ms;
  • Letter Golf index construction: 3.98 seconds;
  • warm index access: 0.591 ms;
  • catalog retained managed-memory delta: approximately 707.4 MiB;
  • Letter Golf index retained managed-memory delta: approximately 198.1 MiB; and
  • cumulative managed allocations during the run: approximately 3,278.3 MiB.

The 3.28 GiB allocation figure is allocation churn over the complete run, not simultaneous or steady-state memory. The measured retained catalog plus Letter Golf index delta was approximately 905.5 MiB. A broader Phase 3 checkpoint retained approximately 817.9 MiB of managed catalog and game views, ended at approximately 979.0 MiB process working set, and later observed approximately 1,006.0 MiB after the Reverse Cross fill index. These measurements used different checkpoints, so they must not be combined as one exact memory accounting. OS committed memory was not recorded as a separate metric.

Production currently assigns the backend 2 GiB after observed 0.5 GiB and 1 GiB out-of-memory failures. It permits scale-to-zero but retains an idle replica for 30 minutes to reduce repeated cold builds. Azure-container startup, load, and headroom measurements remain required before final sizing decisions.

Structured values retained as strings

The backend often retains structured values as generated strings even when their components are already implicit in the owning type. Examples include lexical IDs, fixture and entry IDs, slot IDs, puzzle/play keys, versions, modes, directions, and coordinates. Exact literals may be interned and many collections reuse an existing string reference, but interpolated or concatenated values are new strings; distinct strings do not share common prefixes or suffixes. Repeated property values therefore do not prove duplication, while generated high-cardinality identifiers can retain the same type/version text and hexadecimal encoding once per object.

This is a broader memory hypothesis, not work to fold into fixture authoring. Capture a warm production-sized GC dump before redesigning these contracts. Rank strings by retained bytes and duplicate content, inspect representative roots, and distinguish duplicate character payloads from multiple references to one object. Prioritize high-cardinality structures such as the catalog's lex:<64 hex characters> ID per entry, normalized display/grid pairs, and composite keys. The measured 767,019-entry catalog makes lexical IDs a plausible material cost, but only a heap profile can establish their actual contribution to the 2 GiB Container App requirement.

If measurements justify a change, prefer compact typed values internally: enums and packed scalar coordinates, binary hashes, integer handles, shared string tables, or immutable value records. Format existing textual IDs only at API, persistence, telemetry, and diagnostic boundaries where compatibility requires them. Do not globally intern unbounded runtime data; the intern pool can extend transient strings to process lifetime and increase retention.

Separate Artifacts

The evaluator benchmark catalog and production compiled catalog solve different problems:

  • The benchmark catalog is a small, readable, versioned test fixture committed to ordinary Git.
  • The production artifact contains hundreds of thousands of licensed/provenance-bearing entries, is generated, and should not be committed to Git or Git LFS by default.

They should share deterministic schema versions, canonical ordering, hashes, provenance, and fail-closed validation without sharing physical storage or release lifecycle.

Delivery Vocabulary

Azure Files is a managed network file-share service exposed primarily through SMB and, for some configurations, NFS. Multiple machines or containers can mount one filesystem. It is useful for applications requiring shared filesystem semantics, but an immutable catalog is more naturally an object or image artifact and should not acquire a network-filesystem dependency without measured benefit.

OCI is the Open Container Initiative specification for container images, image manifests, and content-addressed layers. Azure Container Registry stores OCI images and can also store general OCI artifacts. A catalog can therefore be shipped as a stable layer inside the application image or as a separately versioned OCI artifact. OCI describes packaging and distribution; it does not dictate the catalog's internal binary format.

Compiled Catalog Options

Package the raw sources in the application image

The lowest-risk experiment is to acquire the four commit-pinned source files during the controlled build/release process, verify their expected identities, and copy them into a dedicated container image layer. At runtime the existing source parsers read those local files instead of downloading them from GitHub. This is what “package the raw files in the image” means; it does not compile or otherwise change the source formats.

This moves the download from every cold replica to the image build/pull lifecycle. Container registries and hosts can cache the content-addressed layer when it is unchanged between application releases. It removes GitHub availability and four application-level HTTP requests from cold catalog loading, but retains source parsing, normalization, moderation, merge, canonical sort, hashing, managed object construction, and derived-index construction. The image grows by the compressed size of the source layer, and redistribution obligations still apply.

Packaging raw files is useful as a measurement control and resilience improvement. It is unlikely to remove most of the measured 16-second catalog build by itself because the current 35.86 MiB transfer is only one part of the cold path.

Precompile a canonical catalog and reconstruct current objects

The next experiment performs source parsing, normalization, moderation, merging, canonical ordering, and catalog-version calculation in a controlled offline build. It serializes the resulting canonical catalog records into a versioned artifact with source/policy provenance and an integrity checksum. A runtime loader deserializes those records and reconstructs the current LexicalEntry, snapshot, view, and index types where required.

This does not need to serialize the private implementation details of the current .NET object graph. Doing so would tightly couple the artifact to runtime type layouts and collection implementations. Prefer a stable artifact DTO/schema containing the logical catalog data, then deliberately create the current domain objects from it. The first prototype may use a straightforward format before optimizing for binary density; “precompiled” means the expensive semantic transformation already happened, not that arbitrary CLR objects were frozen byte-for-byte.

Artifact representation and delivery are separate choices. The same compiled artifact could be:

  • copied into a container-image layer;
  • stored as an immutable private Azure Blob and downloaded to replica-local ephemeral storage; or
  • published as a separately versioned OCI artifact in Azure Container Registry.

Blob Storage is therefore one possible home, not part of the definition of precompilation. Embedding the artifact in the application image is simpler and removes an application-level startup download; Blob or a separate OCI artifact permits catalog rollout independently of application rollout.

Deserializing canonical records is expected to be faster and allocate less transient memory than repeating source-specific parsing, normalization, merge, sorting, and content hashing on every cold replica. It may still retain close to the current steady-state memory because it deliberately recreates the same managed object graph and lookup structures. The prototype must measure the actual gain rather than assume it.

Build, Storage, and Deployment Lifecycle

The catalog does not need to exist in Git or Git LFS before it can become part of an image. A Docker multi-stage build has private intermediate filesystems. One stage can acquire and verify the pinned sources, run a catalog compiler, and write /catalog/catalog.bin; the final stage can then use COPY --from=catalog-build /catalog/catalog.bin /app/catalog/catalog.bin. Before the push, those bytes exist in the CI runner's BuildKit content store and intermediate layers. After the push, Azure Container Registry is the durable store for the final content-addressed image layers.

The current backend workflow checks out source, builds from the MiniGamesBackEnd directory, pushes minigamebackend.azurecr.io/minigamebackend:{commit-tag}, and tells Azure Container Apps to create a revision from that image. Container Apps authenticates to ACR with its pull identity, downloads the image layers, and starts the final filesystem. No image tarball or generated catalog needs to be checked into the repository.

Initial single-image pipeline

The simplest prototype keeps application and catalog compilation in one reproducible image build:

Git source: compiler code + source URLs/versions + expected checksums
    -> Docker catalog-build stage
       -> download or COPY packaged raw sources
       -> verify source identities
       -> ETL/normalize/moderate/merge
       -> serialize and verify catalog.bin
    -> Docker application publish stage
    -> final stage COPY --from=catalog-build catalog.bin
    -> build one immutable backend image
    -> test/scan that image
    -> push image layers and manifest to ACR
    -> deploy that exact image digest to Container Apps

In this model, compilation is an image-build step that happens in the workflow currently named deploy-backend; it should not be a runtime Container Apps startup step. Longer term, CI should build, test, scan, and identify the image once before environment promotion, while the deployment step only selects the already-built digest.

The final image is self-contained and rollback is simple: an older image digest contains its matching catalog. The drawback is that every application build may repeat catalog compilation unless Docker cache keys and remote cache storage are configured correctly. The current buildx command does not configure an explicit persistent cache-from/cache-to, so cache behavior must be measured rather than assumed.

Independent catalog-release pipeline

If catalogs change much less often than application code, compilation can become a separate, controlled release process:

Catalog source/compiler change
    -> acquire and verify pinned sources
    -> ETL and compile catalog.bin
    -> run catalog contract, size, and content tests
    -> publish immutable catalog artifact by digest
    -> update or approve an application catalog-manifest reference

Application image build
    -> resolve the approved immutable catalog digest
    -> COPY/fetch that exact artifact during the build
    -> include it in the final backend image
    -> test/scan and push backend image to ACR

Possible durable stores for the independently published artifact are:

  • a private Azure Blob named by content hash;
  • a dedicated OCI artifact or minimal catalog image in ACR; or
  • an Azure Artifacts Universal Package if package-feed governance is already desired.

An ACR catalog image supports a convenient multi-stage form in which the application Dockerfile uses a digest-pinned FROM ... AS catalog and COPY --from=catalog. Blob and Azure Artifacts require the workflow or image build to authenticate, download, and verify the artifact before copying it into the final image. Build-time credentials must use BuildKit secret mounts or an external download step and must never be persisted in an image layer.

GitHub Actions artifacts are suitable for temporary handoff between jobs in one workflow, such as passing catalog.bin from a compiler job to an image-build job. They have retention policies and are not the preferred long-term production source or runtime dependency. GitHub Releases could be a durable distribution mechanism, but private Blob or ACR better matches the current Azure deployment and identity model.

  1. In CI, compile the catalog inside a multi-stage Docker build and include it in the backend image. This proves artifact correctness, cold-load improvement, final image size, and rollback without adding a second production artifact lifecycle.
  2. Ensure tests exercise the exact final image and deploy that same image digest; do not compile the catalog again during deployment or at replica startup.
  3. If repeated image-build cost becomes material, publish a dedicated digest-addressed catalog OCI artifact in ACR and consume it from the application build. Keep application images self-contained unless independent runtime catalog updates become an explicit requirement.
  4. Use Blob delivery at runtime only if independent catalog rollout is worth adding a startup network dependency, local cache management, managed-identity authorization, and last-known-good behavior.

Compact and Memory-Mapped Layouts: Evidence-Gated Option

A basic compiled loader can deserialize canonical records and reconstruct the current .NET object graph. This should reduce network and catalog-build CPU but may retain nearly the same steady-state memory.

Compact arrays, string tables, entry handles, and directly queryable memory-mapped indexes require substantial changes to catalog APIs and consumers that currently retain LexicalEntry references. They are not the assumed implementation and should not begin with the initial compiled-artifact work. Consider them only if representative measurements show that packaging raw sources and then loading a precompiled canonical artifact leave unacceptable startup latency, allocation churn, steady-state memory, or replica cost.

A memory-mapped design is different. It stores a deliberately queryable binary layout, such as:

  • a file header with schema/version/checksum;
  • canonical fixed-width entry records;
  • a deduplicated UTF-8 string table;
  • grapheme/source/tag tables addressed by integer offsets;
  • sorted ID and grid-form lookup tables; and
  • game-specific flat postings where prebuilding them is justified.

The operating system maps the file into the process's virtual address space and loads pages on demand. Code reads validated records by offsets or spans instead of deserializing hundreds of thousands of independent .NET strings, arrays, records, dictionaries, and immutable collections. Clean mapped pages can be evicted and reloaded from the file, and replicas on the same host may benefit from the host page cache. The design still consumes virtual address space and resident pages, requires bounds/checksum validation, and needs explicit lifetime/version management.

A “memory-map queryable index” means the sorted tables and postings are directly searchable in that binary format, for example by binary search and integer ranges. It does not mean serializing an arbitrary .NET object graph and reviving those exact runtime objects from disk. Reconstructing .NET types from a compiled stream is the preferred lower-risk experiment. A zero-copy design is only a later candidate if that experiment does not meet measured goals.

Escalation gate

Do not redesign the catalog around handles, offsets, or memory mapping unless the lower-risk path has been measured in a representative Azure container and fails an agreed target. Before escalating, record at least:

  • compiled artifact size and image or download cost;
  • scale-from-zero and first-lexical-request latency;
  • deserialization and derived-index construction duration;
  • cumulative allocations, GC counts, retained managed memory, and process working set;
  • retained heap types, duplicate-string content, representative roots, and bytes attributable to IDs, display/grid forms, source metadata, tags, versions, and composite keys;
  • behavior with representative concurrent requests; and
  • the estimated engineering and operational cost of the compact alternative.

If canonical deserialization removes enough cold latency and allocation churn while remaining comfortably inside the 2 GiB container envelope, stop there. The object model is easier to maintain and its remaining overhead is then justified by measured adequacy.

Proposed Investigation

  1. Extend measurements to separate source download, source parse, normalization/merge, immutable finalization, content hashing, snapshot lookup construction, view construction, and each index.
  2. Record representative Azure scale-from-zero latency, CPU, GC, resident/working-set memory, committed memory where available, and concurrent-request headroom.
  3. Package the raw pinned sources in an application-image layer and measure how much cold latency and failure exposure comes from transfer versus processing.
  4. Build a deterministic, schema-versioned canonical catalog artifact whose loader reconstructs the current object model; measure artifact size, load duration, allocations, retained memory, and compatibility.
  5. Compare application-image layer, private Blob, and separate OCI-artifact delivery for rollout, startup, rollback, availability, and cost.
  6. Stop at canonical deserialization if it meets the agreed latency and memory targets. Only then, if evidence demonstrates a remaining problem large enough to justify the rewrite, prototype a compact string-table/array representation and separately assess memory mapping.
  7. Reconfirm redistribution and attribution obligations for packaged or compiled source content before publishing it outside the private build/runtime boundary.
  8. Define atomic publication and rollback: immutable content identity, integrity check, current manifest pointer, last-known-good retention, and fail-closed handling of malformed artifacts.

Decision Criteria

The chosen approach must preserve catalog semantics, provenance, moderation, deterministic content versioning, and last-known-good behavior. Compare candidates on cold first-request latency, peak and retained memory, total allocation churn, image/artifact size, network dependency, replica startup reliability, operational complexity, and rollback. Do not adopt memory mapping merely for its own sake if a precompiled loader removes enough cold-start cost within the 2 GiB production envelope. The default decision is to preserve the current object-oriented catalog API unless measurements justify the complexity of a handle- and offset-based replacement.