Feature-flag operator runbook¶
Status: Active operator procedure, verified against repository scripts on branch
jm/string-repetition at a12ff3b.
Audience: Operators authorized to change Azure App Configuration feature state.
Safety boundary: This runbook explains supported commands. It does not authorize a production change or replace rollout approval, monitoring, and rollback review.
This is the supported way to change one or more Azure App Configuration feature flags for selected
users or for everyone. It works from Windows PowerShell, WSL, Linux, Git Bash, and macOS without
jq.
The scripts accept canonical user IDs only. A display name is mutable and non-unique; an email
address is neither a stable identifier nor presently a safe query surface. The format of a user ID
may evolve from the original anonymous GUID, but it remains the exact opaque identifier evaluated
by HttpUserIdTargetingContextAccessor and App Configuration's Microsoft.Targeting filter.
Prerequisites¶
- Your Azure identity has App Configuration Data Owner on
minigame-api-config. - You have a canonical user ID for every intended recipient. Obtain it from the player/device workflow already used by the team; do not guess from the current display name.
- Azure CLI, Node.js 18 or later, and either PowerShell 7 or a POSIX shell are available.
- Copy the local configuration template and fill in the values. It is ignored by Git because the admin key is a secret:
API_BASE=https://your-api-host
APP_CONFIG_ENDPOINT=https://minigame-api-config.azconfig.io
ADMIN_DEBUG_KEY=the-existing-admin-debug-key
- Log in before running a command:
The backend's managed identity stays read-only. The script deliberately performs the App
Configuration writes with the operator's Azure identity, then calls a separate private API endpoint
using ADMIN_DEBUG_KEY to announce the accepted batch.
Change selected users¶
Use -State On to add each canonical user ID to Audience.Users and remove the same ID from
Audience.Exclusion.Users:
Use -State Off to remove each ID from Audience.Users and add it to
Audience.Exclusion.Users:
Exclusion has priority over direct inclusion, group membership, and percentage rollout, so explicit exclusion is the only reliable per-user Off operation. Both states preserve groups, rollout percentage, descriptions, unrelated filters, and the flag's global enabled state. Duplicate feature names and user IDs are ignored. A newly created flag is globally enabled, but changing selected users does not re-enable an existing globally disabled flag.
PowerShell accepts comma-separated arguments or arrays. The POSIX command accepts comma-separated lists. Use preview mode before a sensitive change; it reads and validates every flag and builds all payloads, but performs no PUT or announcement:
Change all users¶
-All -State Off is the global kill switch. It sets enabled to false and preserves the complete
targeting configuration so it can be reviewed or restored later:
-All -State On sets enabled to true, sets the targeting default rollout to 100%, and clears
targeting exclusions. It does not enumerate Cosmos users. Existing explicit users, groups, and
unrelated filters remain present:
All On rejects a flag whose conditions.requirement_type is All. With that requirement type,
unrelated filters would still be ANDed with the 100% targeting filter, so the command could not
truthfully guarantee that the feature is on for everyone. Review and change that composition
separately before rerunning the command.
Concurrency and batching¶
The command reads and builds every requested payload before writing any of them. It then writes flags sequentially with each key's App Configuration ETag precondition. If another operator changes a flag after it was read, that PUT is rejected rather than silently replacing the concurrent change.
App Configuration has no multi-key transaction. If a later write fails, earlier writes remain
committed. The command reports the committed prefix and attempts one announcement for every
committed write that returned a usable Sync-Token; rerun the same idempotent command against the
latest values after resolving the reported failure.
Reverse Cross Daily generation rollout¶
Daily generation is controlled by two flags evaluated with one stable synthetic system targeting
identity, system:reverse-cross-daily:
ReverseCrossDailyGenerationis the Daily-specific source gate for missing Daily dates.ReverseCrossGenerationis the master generated-serving gate used by the production-default provider.
A missing Daily date uses generated production-default only when both flags evaluate true for
system:reverse-cross-daily. If the Daily gate is off or fails evaluation, the missing date is
served from fixtures without consulting the master provider. If the Daily gate is on but the master
generation flag is off, fails evaluation, or bounded generation falls back, the missing date is
still served from fixtures. ReverseCrossStructuralQuality is also evaluated with
system:reverse-cross-daily after a Daily request is admitted to generation.
Do not user-target Daily generation flags. One reversecross|global|yyyy-MM-dd puzzle is shared by
everyone, and the implementation deliberately prevents caller-specific Daily source or quality
selection. A user-only allowlist will not enable Daily generation. Use only global -All / --all
changes for Daily rollout. The scripts can create or update App Configuration feature-flag keys;
Terraform does not automatically create these flags or their rollout policy.
Existing persisted Daily puzzle documents are immutable cache hits. Enabling or disabling these flags affects only dates that are not yet persisted. With the normal 14-day pre-generation horizon already full of fixture-backed Dailies, generated Dailies naturally appear only as new dates enter the horizon, roughly two weeks later. Removing future fixture documents can accelerate the change, but do that only through a reviewed data-operation plan.
Before enabling Daily generation, confirm the master generation path is healthy enough for global
Practice production-default traffic too; ReverseCrossGeneration is not Daily-only. Preview and
then enable the Daily and master gates globally:
.\scripts\set-feature-flags.ps1 -Features ReverseCrossGeneration,ReverseCrossDailyGeneration -All -State On -WhatIf
.\scripts\set-feature-flags.ps1 -Features ReverseCrossGeneration,ReverseCrossDailyGeneration -All -State On
./scripts/set-feature-flags.sh --features ReverseCrossGeneration,ReverseCrossDailyGeneration --all --state On --what-if
./scripts/set-feature-flags.sh --features ReverseCrossGeneration,ReverseCrossDailyGeneration --all --state On
After propagation, verify GET /v1/features reports both flags as true from more than one device;
this confirms the global rollout visible to callers, while the Daily source decision itself uses
the synthetic system identity. Then run or wait for the Daily pre-generation job and monitor:
ReverseCross.Daily.GetOrCreatecache.hit,path, dimensions, profile, catalog, and generator tags;ReverseCross.Daily.PreGenerateattempted, success, failure, elapsed, and status tags;ReverseCross.PuzzleGenerationoutcome, fallback reason, latency, attempts, nodes, backtracks, and selected dimensions; and- Container App memory, restart, throttling, and request-failure metrics from platform monitoring.
If generation must stop for future missing Daily dates while leaving any broader Practice rollout untouched, disable only the Daily gate:
If generated serving itself is unhealthy, disable the master gate too:
.\scripts\set-feature-flags.ps1 -Features ReverseCrossDailyGeneration,ReverseCrossGeneration -All -State Off
./scripts/set-feature-flags.sh --features ReverseCrossDailyGeneration,ReverseCrossGeneration --all --state Off
Rollback does not rewrite already persisted generated Dailies. It only makes missing future dates use fixtures again.
Reverse Cross structural-quality rollout¶
ReverseCrossStructuralQuality is an independent quality-policy switch inside the audience already
enabled for ReverseCrossGeneration. It affects both Daily and Practice requests after they are
admitted to generation. Daily evaluates it with system:reverse-cross-daily; Practice evaluates it
with the caller identity. It has no serving effect when generation is off or when a Daily request is
held at the Daily-specific fixture gate. When the quality flag and generation path evaluate true,
generation uses SymmetryPreferredV2, requires a checked-cell ratio of 1.0, and requires one
connected open region. When the quality flag is off or its evaluation fails, generation preserves
IndependentV1 without those strict constraints.
Use caller targeting only for newly generated Practice puzzles with unique puzzle keys. Do not use this procedure to compare policies on Daily puzzles: Daily flag evaluation does not inherit the requesting caller's targeting identity, and a user-only allowlist will not enable Daily generation or structural quality. Daily generation and Daily structural-quality rollout must use global changes.
Before enabling the quality flag, capture a representative legacy baseline for the intended
dimensions and cohort from ReverseCross.PuzzleGeneration activities. Compare at least:
options.mask_strategy,options.quality,outcome,status, andfallback.reason;elapsed_ms,mask_attempts,search_nodes, andbacktracks; and- on successful evaluations,
structural.checked_cell_ratio,structural.unchecked_cell_count,structural.connected_open_component_count, andstructural.rotational_mismatch_pair_count.
Monitor container working set and memory-limit pressure from the Container App/platform metrics separately; the generation activity does not contain a memory measurement. Define that source and its acceptable bound alongside the activity-based guardrails.
Define acceptable changes in generation success, fallback rate, and latency/resource use before changing the flag. The fixed-seed engineering cohort proves deterministic behavior and basic structural availability; it does not establish production service objectives or player quality.
For a new or already enabled flag, preview and then target a small internal subset of the existing generation audience:
.\scripts\set-feature-flags.ps1 -Features ReverseCrossStructuralQuality -UserIds canonical-user-id-1 -State On -WhatIf
.\scripts\set-feature-flags.ps1 -Features ReverseCrossStructuralQuality -UserIds canonical-user-id-1 -State On
./scripts/set-feature-flags.sh --features ReverseCrossStructuralQuality --user-ids canonical-user-id-1 --state On --what-if
./scripts/set-feature-flags.sh --features ReverseCrossStructuralQuality --user-ids canonical-user-id-1 --state On
Selected-user changes intentionally preserve an existing flag's disabled state, so they cannot re-enable a flag after rollback. To begin a later rollout, use the Azure App Configuration Feature manager or review and run the explicit All On command, then rerun the selected-user command if a restricted audience is still required.
For global Daily and Practice structural rollout, preview and then enable the quality flag globally:
.\scripts\set-feature-flags.ps1 -Features ReverseCrossStructuralQuality -All -State On -WhatIf
.\scripts\set-feature-flags.ps1 -Features ReverseCrossStructuralQuality -All -State On
./scripts/set-feature-flags.sh --features ReverseCrossStructuralQuality --all --state On --what-if
./scripts/set-feature-flags.sh --features ReverseCrossStructuralQuality --all --state On
After propagation, verify the intended test caller reports ReverseCrossGeneration and
ReverseCrossStructuralQuality as true from GET /v1/features. For targeted Practice
comparisons, create a new generated Practice puzzle for each comparison. Verify successful
generation activities report SymmetryPreferredV2 and structural-v1, while an untargeted
generation caller creating a different Practice puzzle remains on IndependentV1 and none.
For Daily rollout, verify newly persisted Daily generation activities instead of relying on a
human caller's user ID.
For every successful strict generation, unchecked cells must be zero and connected open components
must be one. Rotational mismatch is a distribution to compare, not a per-puzzle stop condition.
Stop expansion and roll back when the strict structural invariants are violated, required tags are missing, or pre-agreed success, fallback, latency, search, or memory guardrails regress. Disable the whole quality flag; this affects all of its recipients and is intentionally different from removing one user:
Confirm new targeted activities return to IndependentV1 and none. If generated serving itself
remains unhealthy, disable ReverseCrossGeneration through the same reviewed operator path to
restore fixture serving. Disabling either flag does not rewrite stored puzzles or evaluator reports;
existing replay identities and separate report items remain valid.
What happens after a successful command¶
- The script reads every requested
.appconfig.featureflag/<FeatureName>key through the App Configuration data-plane REST API and builds every payload before the first write. - It forwards accumulated App Configuration
Sync-Tokenvalues on later reads and sequential, ETag-protected writes. This makes every request causally at least as new as the accepted requests before it. - After all writes succeed, the script sends one representative feature name and the accumulated
token set to
POST /v1/admin/feature-flags/announcewithX-Admin-Key. - The API saves a monotonically increasing revision document in the dedicated Cosmos
configurationcontainer. The flag definition itself remains in App Configuration. - Each active API replica point-reads that compact document at most every 30 seconds on ordinary
application traffic. A new revision is converted to the .NET provider's supported
ProcessPushNotificationcall. The provider uses the resource URI to identify the App Configuration endpoint and marks all wildcard-selected feature-flag watchers dirty. It applies the accumulated sync tokens, waits a randomized 0–15 seconds to avoid a refresh stampede, and revalidates on that or the next normal request.
For an actively used service, a targeted change normally appears within roughly 30–45 seconds plus the time until the user's next request. An idle replica does no work until it receives traffic, which is intentional. A one-day ordinary App Configuration poll remains as a missed-announcement safety net without consuming the Free-tier request quota on every active replica.
If App Configuration is unreachable while an API process starts, the process still starts and
feature flags safely evaluate off until a later provider refresh succeeds. This prevents a
transient App Configuration, Entra credential, or network failure from taking gameplay offline;
it does not make a missing production AppConfig:Endpoint valid configuration.
If all App Configuration PUTs succeed but the announcement fails, every feature flag has changed in the source of truth but active replicas have not been prompted. Fix the reported API error and rerun the exact same command. User operations and global state changes are idempotent, and the new announcement safely advances the revision.
Maintenance Mode Feature Flags (deferred)¶
Temporary maintainer-only access must not overwrite the normal rollout audience or require an operator to save and restore a manual snapshot. A future design should keep three independent flags for each protected capability:
<Feature>stores the normal rollout policy.<Feature>.MaintenanceModeselects temporary maintenance policy.<Feature>.MaintenanceAccesstargets the maintainers allowed through during maintenance.
Every feature decision point would use one composed helper instead of evaluating the original flag directly:
Turning maintenance mode off immediately restores the untouched original rollout. This composition intentionally lets approved maintainers bypass the original audience while maintenance mode is on. It requires application code, naming and lifecycle rules, tests, and operator commands that update the related flags coherently. That helper and those flags are not implemented by the current scripts; this section records the design for later work.
Infrastructure and verification¶
The configuration Cosmos container must be created through the reviewed Terraform plan before
the first live announcement. It holds only the global revision signal, has no TTL, and is accessed
by point read and ETag-protected replace/create operations.
After an operator command reports success, verify from a browser/device whose canonical user ID was targeted:
The response returns only that caller's evaluated { "FeatureName": true|false } map; it never
returns the targeting audience. Use the page or API route that consumes the flag to confirm the
intended behavior, and verify an untargeted device remains off.
Local checks¶
node --test scripts/set-feature-flag-value.test.mjs
node --test scripts/set-feature-flags.test.mjs
node --check scripts/set-feature-flags.mjs
sh -n scripts/set-feature-flags.sh
$tokens = $null; $errors = $null
[System.Management.Automation.Language.Parser]::ParseFile(
(Resolve-Path scripts/set-feature-flags.ps1), [ref]$tokens, [ref]$errors)
$errors
The automated Node tests cover document transformations, sync-token accumulation, ETag headers, single-announcement behavior, and partial-write reporting. The backend tests cover the protected announcement endpoint and the replica-side use of the provider's push-refresh API.