Phase 4 Reverse Cross Puzzle Persistence: Measurement Plan and Decision Record¶
Status: Active Phase 10 evidence plan built from the completed Phase 4 persistence design. No general Practice pool is authorized without the measurements below.
Audience: Backend and platform engineers evaluating generation latency, freshness, capacity, and pool necessity.
Objective¶
Phase 4 implements Daily, Practice, and 14-day pregeneration persistence workflows for Reverse Cross puzzles. This document records the measurement strategy, operational thresholds, and provisional decision about whether a general Practice puzzle pool is justified. It serves as a decision record and a practical runbook for collecting deployment telemetry in Phase 10.
Telemetry Infrastructure¶
All puzzle lifecycle and storage operations emit activities via the existing
Telemetry.ActivitySource (MiniGamesBackEnd), with structured tags for cache hit/miss, generation path,
Cosmos request charges, and status. Activities include:
ReverseCross.Daily.GetOrCreate: Lifecycle around daily puzzle get-or-create.ReverseCross.Practice.CreateNew: Lifecycle around practice puzzle creation (distinguishes server-generated vs client-key retry lookup).ReverseCross.Daily.PreGenerate: Summary activity for the 14-day pregeneration job (attempted count, success count, failure count, elapsed time).ReverseCross.Storage.Read: Point read telemetry with Cosmos request charge and status.ReverseCross.Storage.CreateIfNotExists: Point create telemetry with Cosmos request charge and status and conflict winner detection.
All tags are low-cardinality and safe: no puzzle keys, dates, seeds, entries, solutions, placements, or user input are recorded.
Measurement Criteria and Decision Thresholds¶
On-Demand P95 Latency (Daily and Practice)¶
Criterion: Median/p50 and p95 wall-clock time from request to result, measured in a representative deployed environment (staging or production with representative load).
- No pool: p95 < 3 seconds consistently → on-demand generation is fast enough; pool overhead not justified.
- Optional pool: p95 in 3–10 second range → tolerable but consider pregeneration for improved UX; revisit if p95 worsens.
- Pool required: p95 > 10 seconds → unacceptable for client UX; pregeneration or architectural change required.
Daily Puzzle Freshness and Pre-Generation Success Rate¶
Criterion: Percentage of the 14-day horizon that is successfully pre-generated by start of Pacific day.
- Target: > 99% of the horizon is available by daily reset time.
- Warning threshold: < 95% success rate for a given day → investigate failed dates and Cosmos throttling, then backfill the missing horizon before changing concurrency or capacity.
- Action: Log failures per date, monitor job elapsed time, ensure container has sufficient headroom (see below).
Steady-State Pre-Generation Elapsed Time¶
Criterion: Wall-clock time for the daily pregeneration job to complete (only applies after the initial run has backfilled the first 14 days).
- Target: < 120 seconds for a complete 14-day run in steady state (i.e., only the new day is generated; the rest are cache hits).
- Rationale: Job should complete well before the next daily reset, leaving time for debugging if needed.
- Monitor: Collect the
elapsed.mstag fromReverseCross.Daily.PreGenerateactivities.
Request Rate and Queue Buildup¶
Criterion: Peak request rate during normal play and pregeneration, monitored via Azure Monitor or Application Insights.
- Target: Sustain more than 5 Practice requests per second in a representative test without queue buildup.
- Action if missed: Identify whether generation concurrency, CPU, or Cosmos throughput is the bottleneck before considering a pool or capacity change.
Cosmos Container Headroom and Throttling¶
Criterion: Application-container memory headroom and Cosmos throttle rate.
- Headroom: Keep at least 500 MiB of memory free in the 2 GiB application container after warm generation.
- Throttling rate: Cosmos 429 (throttle) errors and request timeouts should be < 5% of total requests. If higher, increase RU provisioning or optimize queries (e.g., via query plans and diagnostics).
- Monitor: Cosmos diagnostic logs and Application Insights metrics for
cosmos.status_codetag values (especially 429).
Measurement Scenarios¶
Cold Start (First Run)¶
Scenario: Deployed container with no pre-generated puzzles; pregeneration job runs for the first time.
- Expected behavior: Job generates all 14 days (or fewer if the horizon is shorter). Each
GetOrCreateForDateAsynccall to the Daily provider will miss the cache and trigger generation. - What to measure:
- Total and per-date p50/p95/p99 elapsed time.
- Read and create request charges reported by Cosmos.
- Generation outcomes, horizon success rate, CPU, and peak memory.
Warm Start (Steady State)¶
Scenario: Container already has 14 days pre-generated; the daily pregeneration job runs again.
- Expected behavior: Most dates will be cache hits (point read finds document, returns immediately). Only the new day (or a few overdue days) will trigger generation.
- What to measure:
- Total elapsed time (target < 120 seconds).
- Ratio of cache hits to generated paths; a daily run should normally introduce one new horizon date.
- Per-read and per-create request charges as reported by Cosmos.
- Success count should equal HorizonDays (14) on each run.
Practice Puzzle Generation¶
Scenario: Exercise 5x5, 7x7, and 9x9 Practice generation separately, including retries with client-supplied keys and representative concurrency.
- Expected behavior:
- Server-generated calls (no requested key): cold generation + store create.
- Client-key retry calls (requested key already exists): cache hit on read, no generation.
- Client-key calls (requested key missing): cache miss, generate, then create.
- What to measure:
- Latency from request to response (p50, p95, p99).
- Percentage of calls with cache hits (successful retries).
- Percentage of server-generated vs client-key requested calls (product mix).
- Cosmos request charges for read/create per scenario.
- Generation success and fallback rate by dimensions.
- Queue time, CPU, and memory during a sustained test above 5 RPS.
Locally Measured Evidence (Phase 4)¶
The following measurements were made on the Phase 4 branch using the pinned production word-catalog artifacts, local fixture dimensions, and deterministic seeds, without live Cosmos DB or representative production load:
-
Sample generation latency: Nine fixed-seed generation samples spanning 5x5, 7x7, and 9x9 all succeeded. The observed sample maxima were 15.4 ms for 5x5, 667.5 ms for 7x7, and 1,281.8 ms for 9x9. These are generation-only development-machine measurements; three samples per size are too few to establish representative percentiles and exclude persistence, network I/O, queueing, and representative concurrency.
-
Test evidence: Deterministic activity-capture tests cover Daily cache hits, Practice create and retry paths, partial and canceled pregeneration, Cosmos read/create request charges, conflict-winner reads, and tag privacy. Mock request charges prove propagation, not actual RU cost.
Staged Measurement: What Requires Production or Staging Evidence¶
The following measurements CANNOT be made locally and require a deployed environment:
- Real Cosmos DB request charges and latency: Network I/O, container/partition hot-spot effects, and indexing policy impact can only be observed against actual Cosmos.
- Concurrent generation under load: Multiple simultaneous puzzle requests (>5 RPS) and cache contention in practice.
- Container growth and throttling: Actual storage size, TTL cleanup behavior (30-day expiration for practice puzzles), and 429 throttle rates.
- Daily job execution: Actual job invocation timing relative to Pacific daily reset and resources available to the Container Apps Job.
- Player behavior and request distribution: Real cache hit rates, retry patterns, dimension preferences (5x5 vs 7x7 vs 9x9), and load spikes.
Provisional Decision: No General Practice Pool Now¶
Status: Phase 4 Step 6 — Telemetry and measurement infrastructure are in place.
Decision: Do NOT build a pre-generated general Practice pool at this time.
Rationale:
- Local evidence does not justify pool complexity: All nine fixed-seed samples spanning 5x5/7x7/9x9 completed successfully, with a worst observed sample of 1,281.8 ms for 9x9. This is encouraging, but it is not a deployed p95 measurement.
- Cache hit rates for retries will be high: Players who retry a client-supplied puzzle key will almost always hit the cache; only first-time requests trigger generation.
- Pregeneration of all possible dimensions/complexity is infeasible: Unlike Daily (one puzzle per day), a general Practice pool would need to pre-generate multiple variants (5x5, 7x7, 9x9; easy, medium, hard if added), multiplying storage and maintenance cost.
- Operational overhead of pool management: A pool requires strategies for pool depth, refresh policy, and eviction; without clear deployment evidence that on-demand is insufficient, that complexity is not justified.
Action: Continue measuring on-demand generation performance via telemetry once deployed (Phase 10).
Re-evaluation trigger: Revisit this decision if any of the following are observed in a representative deployment:
- On-demand p95 latency reaches 3 seconds, making a pool optional; treat sustained p95 above 10 seconds as requiring a pool or equivalent architectural change.
- Daily pregeneration success rate falls below 95%.
- Cosmos throttle rate (429 errors) exceeds 5% of total requests.
- Application-container memory headroom falls below 500 MiB after warm generation.
- Player feedback or analytics indicate unacceptable wait times.
Deployment Checklist for Phase 10¶
When operationalizing Reverse Cross (Phase 10), ensure the following:
- Telemetry exports to Application Insights: Verify that activities and tags are being collected and available in the Kusto query console.
- Dashboards for key metrics: Set up monitoring dashboards showing:
- p50/p95/p99 latency for
ReverseCross.Daily.GetOrCreateandReverseCross.Practice.CreateNew. - Success count and failure count for
ReverseCross.Daily.PreGenerate. - Request charges (RU consumption) for
ReverseCross.Storage.ReadandReverseCross.Storage.CreateIfNotExists. - 429 throttle rate and storage utilization for the Puzzles container.
- Alerting: Set up alerts for:
- Daily pregeneration job success rate < 95%.
- Cosmos throttle rate > 5%.
- On-demand latency p95 > 10 seconds (only set this alert if pool implementation becomes necessary later).
- Periodic review: Review collected telemetry weekly for the first month post-launch, then monthly. Use the measurement criteria above to assess whether the provisional decision (no general pool) remains valid.
References¶
- Reverse Cross Specification — product behavior and rules.
- Lexical and Game Architecture — shared catalog design.
- Reverse Cross Roadmap — phase timeline and decisions.
- Telemetry — implementation of
ActivitySourceand tag patterns. - Storage — Cosmos integration and request charge instrumentation.