Interruptions & Rescue

Infrastructure operations can be interrupted — a Ctrl-C, a lost connection, a crashed process, a CI job cancelled mid-apply. Keel is built to make that hard to do by accident and recoverable when it happens anyway.

Interrupting keel up and keel destroy

Both run under an interrupt guard. OpenTofu runs in its own process group so a stray Ctrl-C can't reach it directly; Keel counts your interrupts, warns on each, and takes four before it actually aborts — a genuinely stuck apply must be escapable, but a half-applied stack shouldn't be one accidental keystroke away. SIGTERM (closed terminal, cancelled CI job) is handled the same way.

If an operation is aborted, the error points you at the repair:

the apply was stopped on request and may have left resources untracked;
run 'keel rescue' to check

keel rescue

keel rescue (admin) diagnoses and repairs an environment left inconsistent by an interruption. By default it only reports. Two things are checked:

• State lock — a held lock blocks every other command. The report names who took it, what they were doing, and how long ago, so the decision to break it is an informed one.
• Untracked resources — resources tagged ManagedBy=keel for this app and environment that OpenTofu's state does not account for. These are what an interrupted apply leaves behind: they exist and bill, and no later Keel command will touch them.

Repairs are opt-in:

keel rescue                     # report only
keel rescue --unlock            # release the state lock (confirms; only when nothing holds it)
keel rescue --converge          # re-run the apply — the repair for an interrupted keel up
keel rescue --finish-destroy    # re-run the teardown, for an interrupted keel destroy

--finish-destroy also accepts --delete-retained. Untracked resources are listed with the exact command that deletes each one, and are never deleted automatically — removing a resource Keel has lost track of is not something to do on a best guess.

Two honest caveats the report states rather than hides: the Resource Groups Tagging API does not cover every resource type, so the untracked list is a floor, not a ceiling; and a locked state cannot be read, in which case tracked state is reported as unknown rather than everything being called orphaned.

Releasing a state lock directly

When you already have the lock ID from an "Error acquiring the state lock" message:

keel infra unlock <LOCK_ID>        # admin; confirms first
keel infra unlock <LOCK_ID> --yes  # for scripts

keel rescue --unlock and keel infra unlock are a pair: rescue finds the lock and reports who holds it; infra unlock takes the ID you already have. Only break a lock nobody holds — two writers into one state corrupt it.

What survives a destroy

Three DynamoDB tables span every app and environment in the account and are deliberately outside any app's stack: the app registry (keel-apps), deployment history (keel-deployments), and deploy locks (keel-deploy-locks). A successful keel destroy releases the environment's deploy lock and deregisters the app, but deployment history survives by default — "we deployed abc123 to production before tearing it down" is worth keeping, and the dashboard labels a history whose environment is gone rather than hiding it. Remove it when it genuinely isn't wanted:

keel history prune              # admin; confirms with the record count
keel destroy --prune-history    # or in the same run as the teardown

For the design reasoning — why adoption of orphans via tofu import is deliberately not implemented, the process-group split, the two-hour staleness heuristic — see the Interrupted Operations design note.