Account Deletion Operator and Developer Runbook¶
Status: Active application procedure; external retention and legal obligations require separate review.
Audience: Authorized operators, backend maintainers, product, privacy, and legal reviewers.
Date: 2026-09-13
Scope: Application behavior and operator procedure supporting GDPR/EEA erasure requests.
Implementation commits: 5535760, 4ae1bf2
1. Scope and Explicit Non-Claims¶
This implementation provides a durable account deletion workflow intended to support erasure requests. This is not legal advice and does not by itself guarantee GDPR, EEA, or other regulatory compliance. Compliance also depends on policies, procedures, data inventories, retention decisions, and legal review outside this software.
Explicit Non-Deletions¶
The system does not delete:
- Identity Provider (IdP) and authentication provider accounts. External OAuth/OIDC provider accounts (e.g., Google, Microsoft Entra) remain under the provider's control and must be deleted through the provider's own processes or user self-service.
- Data outside the application stores handled by this workflow. Provider-side accounts, external integrations, backups, and telemetry are governed separately.
- Future billing, legal, or audit records. If such records exist, their lawful retention and erasure rules require a separate documented review.
Responsible Teams¶
- Product/Legal: Define and approve erasure request workflows and any additional data retention policies.
- Operations: Execute approved deletion requests and verify outcomes.
- Compliance/Legal: Document retention policies for billing, audit, and regulatory purposes; coordinate with data protection officers.
2. What Is Erased¶
When an account deletion request is approved and executed for a user, the following data is permanently removed from Verbolus:
User Identity and Authentication¶
- Canonical user ID and all merged alias user IDs. The deletion operation resolves the requesting user to their canonical root identity, discovers all identities merged into that account through the merge history, and removes all of them.
- External identity lookup links. Issuer and subject links are deleted, so a later sign-in with the same provider identity can create or link a fresh Verbolus account instead of recovering the deleted account.
Game History and Player State¶
- All plays/games: Every game record, score, and puzzle attempt by any identity in the closure.
- Player profile documents: Personal settings, preferences, and configuration data.
- Player statistics: Scores, streaks, cumulative stats, and derived metrics.
- Session authorization: Durable revocation markers block the deleted identity. The successful browser flow replaces its own application identity cookie; other browsers keep stale cookies until they encounter recovery behavior or expire.
Social and Group Data¶
- Group memberships: The user's positions in groups they were members of.
- Group standings: Closure-keyed entries are removed from retained groups' standings documents.
- Empty groups: If removing a user causes a group to become empty (no members remain), the entire group document and all associated standings entries are deleted, including:
- Group metadata (name, join code, creation timestamp).
- Every standings document in the group partition.
- The snapshotted and current join-code lookup when present.
- Preservation: Unrelated users, memberships, standings entries, and nonempty groups remain intact. If the deleted user owned a retained group, ownership transfers deterministically to the earliest joined remaining member, then by ordinal user ID.
3. Durable Phase Model and Retry Behavior¶
The deletion orchestrator uses a durable, multi-phase state machine to ensure safe resumption from failure and to avoid partial/corrupted deletion states.
Phases (in order)¶
- Initialized: Operation created and snapshot of closure identities and group targets stored in Cosmos.
- RevocationsCreated: Revocation markers written for each identity in the closure, preventing future API requests by those identities.
- GroupsCleaned: Groups containing closure members are cleaned (members removed, empty groups deleted, standings scrubbed).
- PlaysDeleted: All game records for closure members are removed.
- IdentityLookupsDeleted: External identity lookup entries are removed, preventing re-authentication via the same IdP identity.
- PlayersDeleted: Player profile documents are removed.
- Completed: Operation marked complete; sensitive arrays cleared; TTL shortened to 24 hours.
Phase Execution and Checkpoints¶
- Phase ordering is strict and deterministic. The orchestrator checks the current phase at the start of
ExecutePhasesAsyncand only executes phases that have not yet completed. This ensures safe resumption: if a failure occurs during a phase, re-running the deletion operation picks up from the last successful checkpoint. - ETag-based optimistic concurrency. Each phase is followed by a checkpoint operation that updates the operation document with ETag validation. If another process modifies the operation concurrently, the update fails with a 412 Precondition Failed error.
- Bounded retry on checkpoint conflicts. The checkpoint logic makes at most five save attempts when the operation save returns 412. On each retry, it reloads the operation and does not regress a phase another worker already advanced. Group cleanup independently makes at most five attempts for group ETag conflicts. Other 412 responses are not reclassified as checkpoint conflicts.
- Non-412 errors propagate immediately. If any phase operation fails with a non-412 error, the exception is raised and execution stops. The operator must investigate and may retry by calling the deletion endpoint again (see Safe Operator Procedure).
- Idempotency: Calling the deletion operation again is safe. A retry resumes from its last durable checkpoint. Work performed immediately before a failed checkpoint can be replayed, so destructive store operations are designed to tolerate already-missing data.
Snapshot and Determinism¶
Before any destructive phase begins:
- The closure set (canonical user ID + all merged aliases) is snapshot into the operation document.
- External identities (issuer + subject pairs) are snapshot.
- Group targets (group IDs and join codes containing closure members) are snapshot.
These captured targets remain the basis for retries and are cleared when the operation reaches Completed, minimizing how long transient identifiers remain stored.
4. Session Behavior¶
Revocation Marker Duration¶
Revocation markers are created for each identity in the closure during the RevocationsCreated phase. These markers remain in Cosmos until they are expired by the Cosmos TTL mechanism.
TTL Duration: 32 days (approximately one month, not exactly 30 days).
During this period, any request from a browser session carrying the revoked userId is rejected by the AccountDeletionSessionMiddleware with HTTP 401 and a problem details response.
Stale Session Handling¶
When a revoked session makes a request (other than exact POST /v1/players/me/delete, which remains available for deletion retry):
- The middleware checks the revocation status and returns HTTP 401 with error code
session_deleted. - The application recovery route can handle that response by:
- Clearing OIDC transaction and pending-identity cookies.
- Generating a fresh anonymous userId (UUID v4).
- Issuing a new signed userId cookie for the fresh identity.
- This rotation allows the browser to continue using the app under a new anonymous session.
Multi-Device and Multi-Tab Recovery¶
- Browser that completes deletion: A successful account-delete BFF response immediately rotates that browser to a fresh anonymous identity.
- Other tabs/devices: They receive 401
session_deletedwhile using the stale identity. Application calls wired to session recovery make one recovery attempt and rotate locally; raw API clients must handle the response themselves.
Re-Authentication After Deletion¶
If the user later signs in using the same external IdP (e.g., the same Google account):
- The external identity lookup for that issuer + subject will not exist (it was deleted during IdentityLookupsDeleted phase).
- The sign-in flow treats it as a new identity and can create a new, independent Verbolus account linked to that IdP identity.
- The user's previous data (games, stats, groups) remains permanently deleted and is not recovered or re-associated.
5. Retention Caveats and Restoration Warning¶
Completed Operation Metadata¶
Once an operation reaches the Completed phase:
- Sensitive arrays are cleared: The
ClosureUserIds,ExternalIdentities, andGroupTargetsarrays in the operation document are emptied. - TTL is shortened: The per-item TTL is changed from 32 days to 24 hours.
- Remaining metadata: The operation retains its opaque ID, phase, and timestamps briefly. Treat operational metadata as controlled data even though closure IDs and identity/group targets are cleared.
- Automatic expiration: After 24 hours, the operation is automatically removed from Cosmos by the TTL mechanism.
In-Progress Operation and Revocation Marker TTL¶
- In-progress operations: TTL = 32 days (same as revocation markers).
- Revocation markers: TTL = 32 days.
If an in-progress operation is abandoned (e.g., the operator never retries, and no other process resumes it), it and its revocation markers will remain for up to 32 days before automatic expiration.
Stale Sessions After Operation Expiration¶
The completed operation expires after 24 hours, while its revocation markers can remain for 32 days from creation. During that interval, ordinary stale-session requests remain blocked. A self-delete retry can find a marker whose operation has expired; the service fails closed instead of reconstructing deletion targets from incomplete state. Rotate or recover that browser identity and investigate through approved procedures if deletion completion is in doubt.
After the marker itself expires, it no longer blocks the old cookie. This does not restore erased data. A later provider sign-in can create or link a fresh account.
Cosmos Backups and Diagnostic Logs¶
- Cosmos backups: Backups may contain deleted account data for the provider-configured retention period. Verify the actual policy; this runbook does not assert a backup mode or duration.
- Diagnostic and audit logs: Telemetry may remain for its configured retention period. Access and erasure decisions for those systems require their own policy and legal review.
Restoration caveat: If Cosmos is restored from a backup taken before the deletion was completed, the deleted data will be reintroduced. Any restoration must be followed by:
- Immediate re-erasure: Re-run the deletion operation on affected user IDs to ensure deletion is complete.
- Reconciliation: Verify that all affected user data was re-deleted (e.g., confirm absence via approved data queries).
- Documentation: Record the incident, reason for restoration, and reconciliation steps taken.
This risk is inherent to backup/restore workflows and requires documented incident procedures. Do not restore an old backup as a casual rollback without understanding the compliance and data-protection implications.
6. Repair Utility Usage¶
The repair-account-deletions.mjs utility allows operators to safely retry or resume account deletion requests through an admin API endpoint, with built-in safeguards against credential exposure and network failures.
Prerequisites¶
- Node.js: Use the repository's supported Node.js environment.
- Network access: Connectivity to the Verbolus API base URL (HTTPS).
- Admin credentials: Access to
ADMIN_DEBUG_KEY(see Apply Mode).
Basic Syntax¶
--user-ids(required): Comma-separated list of Verbolus user IDs to delete. Trimmed; empty IDs cause an error.--apply(optional):trueorfalse; defaults tofalse(dry-run).
Dry-Run Mode (default)¶
Output:
- No network calls made. Environment variables are not read.
- No credentials required.
- Deduplicates user IDs and reports only the unique count.
- Dry-run validates local input only; it does not verify an environment or inspect server data.
Apply Mode¶
To actually delete accounts, use --apply true:
Apply mode requires two environment variables:
API_BASE: Approved absolute HTTPS URL for the target API.- Must use HTTPS (no HTTP).
- Must be absolute (no relative paths).
- Must not contain credentials, query strings, or fragments.
-
Trailing slash is optional and will be normalized.
-
ADMIN_DEBUG_KEY: The admin key for authenticating the deletion request. - Sent as the
X-Admin-KeyHTTP header. -
Never hardcode or persist this value in scripts, shell history, or configuration files.
-
Inject it into the process environment through the approved secret manager without printing it or persisting it in a file or shell history.
After the approved process-scoped environment contains both variables:
Limits and Constraints¶
- Maximum 50 distinct user IDs per invocation. Batches larger than this will fail. For large repair operations, invoke the utility multiple times with separate batches.
- API base must be HTTPS. HTTP connections are rejected.
- Request timeout: 30 seconds per API call.
- No direct Cosmos mutation. The utility does not connect to Cosmos; it calls the API endpoint, which handles all Cosmos operations.
Output (Apply Mode)¶
On success:
Details:
Processed: Number of unique user IDs after deduplication.Deleted: Number of successfully deleted accounts.- If
Deleted < Processed, the utility reports an error and exits with code 1 (see Failure Modes).
On failure, the utility prints an error message and exits with code 1.
7. Safe Operator Procedure¶
Pre-Approval¶
- Identify the request source: Verify that the deletion request originates from an approved channel (e.g., formal data-protection request via legal, customer support with documented consent, or automated compliance workflow).
- Verify the user ID: Confirm that the target user ID is correct and matches the request. Do not proceed with incorrect or ambiguous IDs.
- Target environment: Clearly identify whether the deletion should be applied to development, staging, or production. Do not casually run deletion against production without explicit approval.
Dry-Run¶
- Run a dry-run first:
This confirms the user IDs are well-formed and counted correctly, with no network calls or environment variables required.
Capture and Approval¶
- Record the operation:
- Note the count of user IDs and deduplicated count (in case of duplicates).
- Link to the approval ticket or request (e.g., a Jira ticket, support case, or internal workflow).
- Do not record raw user IDs in tickets or logs visible to unauthorized personnel. Use aggregated counts and reference the original request/approval instead.
- Never capture the API base URL, admin key, or any secrets.
Non-Production Testing¶
- Exercise the approved procedure in a non-production environment first when appropriate. Use synthetic identities and the environment's own credentials. Verify the aggregate counts without copying identifiers into routine logs.
Production Execution¶
- Apply only to the approved target environment. Confirm the target and change approval again immediately before setting the process-scoped environment and using
--apply true.
Verification¶
-
Inspect the aggregate result:
-
The utility deliberately reports counts only. If any item fails or the API response is inconsistent, it exits with an error.
- To isolate a failed target without exposing response identifiers, retry the approved inputs individually or in smaller batches after investigating the service failure.
- Use approved queries only to verify that user data is absent from Cosmos (e.g., a data-export query run by authorized personnel). Do not run ad-hoc Cosmos queries as part of this verification.
Session Verification¶
- Confirm stale sessions are blocked:
- If applicable, have a test user account with a session from the deleted user ID and confirm that requests return 401
session_deleted. - Confirm that the session rotation logic rotates the user to a fresh anonymous session.
Documentation¶
- Record evidence of completion:
- Date, time, and user IDs affected (in aggregate, not individual IDs if sensitive).
- Deletion status (succeeded, partial failure, etc.).
- Any retry operations and their outcomes.
- Approval reference (ticket, request ID).
- Store this record in a secure, auditable location (e.g., a compliance log, ticket system, or audit database).
- Do not store raw user IDs, API keys, or other sensitive values in the record; reference the original request instead.
Do Not¶
- Do not run deletion against production without explicit, documented approval. "It's only one user" and "I've done this before" are not sufficient justifications.
- Do not hardcode credentials in scripts or shell history. Use your organization's secrets management system.
- Do not skip the dry-run step, even for small operations. A dry-run is free and catches configuration errors early.
- Do not assume successful exit code means correct behavior. Always verify counts and, if high-value, test outcomes in staging first.
8. Failure Modes and Troubleshooting¶
Revocation Marker References Expired or Missing Operation¶
Symptom: The middleware or a diagnostic query reports that a revocation marker references an operation that no longer exists in Cosmos.
Cause: A completed operation has a 24-hour TTL, while its marker can remain for 32 days from creation. A marker can therefore outlive its operation.
Resolution:
- Ordinary requests still receive 401
session_deletedwhile the marker remains; rotate to a fresh identity. - A self-delete retry fails closed because the durable snapshot is unavailable. Do not manufacture or edit an operation document.
- Investigate using approved evidence and queries. Escalate to the development and privacy owners if completion cannot be established.
412 Precondition Failed During Checkpoint¶
Symptom: The deletion operation fails with HTTP 412 during a checkpoint phase.
Cause: Another deletion worker modified the operation document between read and write, invalidating the ETag.
Resolution:
- The checkpoint logic automatically retries up to 5 times per phase.
- If all 5 retries fail, the operation fails and returns an error to the caller.
- Operator action: Wait a moment and retry the entire deletion operation. The operation will reload the latest state and may succeed on re-run.
- If the issue persists across multiple retries, investigate whether another process is intentionally or unintentionally modifying the operation.
Non-412 Phase Failure¶
Symptom: A phase operation (e.g., deleting plays, removing from groups) fails with an error other than 412.
Cause: Examples include Cosmos connectivity issues, authentication failures, or invalid data state.
Resolution:
- The error is logged and propagated to the caller.
- The operation does not automatically retry; it remains at the failed phase.
- Operator action: Investigate the error (check API and Cosmos logs), fix the underlying issue, and re-run the deletion operation. The operation will resume from the failed phase upon retry.
Admin Batch Continuation and Generic Failure¶
Symptom: The repair utility's --apply true mode reports a failure for one or more user IDs in the batch.
Cause: The /v1/admin/users/delete API processes each user ID independently. A failure for one user does not prevent processing of others.
Resolution:
- The API returns a generic batch failure if any user deletions failed, even if most succeeded.
- The utility intentionally suppresses per-item identifiers and errors. Use approved service telemetry and smaller retries to isolate the affected approved input without copying identifiers into routine logs.
- Successful prior items remain safe to retry because deletion is idempotent.
9. Rollback and Restoration Warning¶
Account deletion is intentionally destructive and has no application-level rollback.
- No "undo" operation exists. Once a deletion completes, the data is gone. There is no "rollback account deletion" endpoint or procedure.
- Restore from backup only under incident procedure. Restoring a Cosmos backup to reintroduce deleted account data is a major operation requiring:
- Incident authorization and documentation.
- Legal review (restoration may re-introduce data the user requested deleted, creating compliance risk).
- Immediate re-erasure and reconciliation of affected accounts.
- Audit trail of the incident and all recovery steps.
- Do not use backup restore as a casual rollback. It is not a supported rollback method and creates operational and compliance risk.
- Communicate clearly to stakeholders: When an account deletion request is approved, ensure all teams understand the permanent nature of the operation.
10. Testing Evidence¶
The following focused test suites provide evidence of account deletion behavior:
Backend Tests¶
MiniGamesBackEnd.Tests/AccountDeletionServiceTests.cs- Tests the
AccountDeletionServiceorchestrator, including phase ordering, idempotency, ETag retry logic, and snapshot behavior. - Covers happy path (successful deletion across all phases) and retry scenarios (ETag conflicts, stale session recovery).
-
Validates that completed operations clear sensitive arrays and shorten TTL.
-
MiniGamesBackEnd.Tests/AccountDeletionGroupCleanupTests.cs - Tests the
AccountDeletionGroupCleanupgroup-removal logic, including member removal, ownership transfer, empty group deletion, and standings scrubbing. -
Covers edge cases (empty closure, empty groups, duplicate targets) and retry behavior on 412 errors.
-
MiniGamesBackEnd.Tests/AccountDeletionSessionMiddlewareTests.cs - Tests the
AccountDeletionSessionMiddlewarerevocation check, including allowing retry-permitted paths and returning 401session_deletedfor blocked sessions. -
Validates that the middleware does not log sensitive data.
-
MiniGamesBackEnd.Tests/CosmosAccountDeletionStoreTests.cs - Tests the storage layer, including revocation marker creation/lookup, operation snapshot and retrieval, and ETag handling.
Frontend Tests¶
frontend/minigames/src/app/api/auth/account-delete/route.test.ts- Tests successful deletion rotation and backend failure behavior.
frontend/minigames/src/app/api/auth/session-recovery/route.test.tsandfrontend/minigames/src/app/api.test.ts- Test recovery routing and one-attempt handling of 401
session_deletedresponses.
Repair Utility Tests¶
scripts/repair-account-deletions.test.mjs- Tests argument parsing, dry-run behavior, apply mode validation, environment variable handling, and API request construction.
- Tests deduplication, direct-call and batch limits, Windows CLI execution, response validation, and secret-safe error behavior.
Coverage Notes¶
- What is tested: The core deletion phases, ETag retry logic, session revocation, and repair utility operation.
- What is not claimed here: These checks are not a production-Cosmos end-to-end exercise, a full repository validation, or a compliance certification.
- This list documents focused unit and targeted integration suites only. Full suite coverage claims and compliance certifications require broader validation and are not claimed by this runbook.
References¶
- Implementation commits:
5535760,4ae1bf2 - Source files:
- Backend:
MiniGamesBackEnd/AccountDeletionService.cs,MiniGamesBackEnd/AccountDeletionGroupCleanup.cs,MiniGamesBackEnd/Configuration/AccountDeletionSessionMiddleware.cs,MiniGamesBackEnd/Storage/AccountDeletionOperationEntity.cs - Frontend:
frontend/minigames/src/lib/auth-session-rotation.ts - Repair utility:
scripts/repair-account-deletions.mjs