Interrupted applies and destroys

What goes wrong

OpenTofu writes state after each resource it acts on. Killed between the AWS call and the state write, a resource exists and nothing tracks it:

• the next keel up creates a second one, or fails on a name collision;
• keel destroy never deletes the first, so it bills indefinitely;
• no plan mentions it, so nothing surfaces it.

Separately, the state lock lives in DynamoDB and outlives the process that took it. Every later command fails with Error acquiring the state lock, naming a UUID the operator has no way to look up.

Both are consequences of stopping an operation partway. So the design has two halves: make stopping hard, and make recovery possible.

Making it hard to stop

A terminal delivers SIGINT to the entire foreground process group. So before this, Ctrl-C during an apply reached tofu directly, whatever Keel's own signal handling did — Keel could not have protected it even in principle.

infra.detachProcessGroup puts the child in its own process group (Setpgid on Unix, CREATE_NEW_PROCESS_GROUP on Windows). The terminal's signal no longer reaches it, and Keel becomes the only thing that can stop it.

ui.InterruptGuard then decides whether to. It installs a handler for SIGINT and SIGTERM around the critical section of keel up, keel destroy and keel rescue --converge, prints a banner up front saying the operation will not stop and does not need watching, and counts interrupts:

• interrupts 1 through InterruptThreshold-1 print a warning naming how many are left;
• the last one cancels the context, which kills the child, and points at keel rescue.

Four interrupts cannot be typed by accident. One can. The escape hatch exists because a genuinely stuck apply has to be escapable — an ACM certificate waiting on DNS validation that will never arrive, for instance — and because a tool that cannot be interrupted at all is one people will kill -9, which is strictly worse.

SIGTERM is included because a closed terminal or a cancelled CI job arrives as one, and stopping an apply on either has the same consequence as a Ctrl-C.

Making recovery possible

keel rescue answers the two questions an interruption leaves open. It changes nothing by default; each repair is a separate explicit flag.

The lock

infra.ReadStateLock reads the row the S3 backend keys on <bucket>/<key> and reports who took it, what they were doing, and how long ago. That is the information the decision to break it needs, and it was not available through any Keel command — keel infra unlock could release a lock but only if you already knew its ID and had decided nobody held it.

--unlock releases it, after a confirmation that quotes the holder. Unattended it declines rather than defaulting to yes: breaking a lock somebody is genuinely holding lets two writers into one state, and a scripted run should not do that to a lock nobody has looked at.

The staleness heuristic (StateLock.Stale, two hours) is presented as evidence, never acted on: a six-hour apply is a real thing.

The orphans

The tags are the second, independent record of what exists. Keel stamps ManagedBy=keel plus the app and environment on everything it creates, so:

whatever carries Keel's tags for this environment and is absent from OpenTofu state was created by Keel and is no longer tracked by it

That difference is exactly what an interruption produces, and it is computable: awsclient.TaggedResources lists the tagged set through the Resource Groups Tagging API, and trackedIdentifiers collects every ARN and id the state holds.

Two caveats the report states rather than hides:

• The tagging API does not cover every resource type. A security group rule is not taggable. So the untracked list is a floor on what exists, never a ceiling.
• A locked state cannot be read, which is the case the command exists for. It reports the lock and says the tracked set is unknown, rather than reporting every tagged resource as orphaned.

Untracked resources are listed with the AWS CLI command that deletes each one, and are never deleted automatically. Keel has lost track of them by definition; deleting on a best guess is not a thing to do to something that might be a production database. Where one command will not do the job — a VPC has to be emptied first — no command is offered rather than one that will fail.

Converging

--converge re-runs the apply, which is the actual repair for an interrupted keel up: OpenTofu refreshes, sees what exists, and creates what is missing. It uses Apply rather than ApplyPlan deliberately — any saved plan predates the interruption and was computed against state that has since moved, so OpenTofu would refuse it.

A converge does not adopt the untracked resources. It creates its own, which is why the report tells you to delete or import the orphans first: otherwise the name collision fails the apply.

--finish-destroy re-runs the teardown, with SkipFinalSnapshot forced on. The first attempt already took a snapshot, and a second attempt with the same identifier fails on the collision.

Known gap

Adoption is not implemented. tofu import needs a resource address for each orphan, and mapping an ARN back to the address the generators would have used is a guess — a wrong one writes a resource into state under the wrong address, which is a worse outcome than the orphan. Deleting and re-applying is the supported path.