App- and Environment-Level Separation

Status: Design / not yet implemented Scope of first pass: Phases 1–3 (config + selection, infra isolation, registry + runtime). Phase 4 (auth isolation) is designed here but deferred. Back-compat: None required — Keel has not launched. Every stack is cleanly <app>-<env>; there is no legacy unqualified special case.

---

1. Motivation

Today Keel has exactly one scoping axis: the app name (cfg.Name from keel.yml). It is threaded into every AWS resource name, the OpenTofu state key, SSM parameter paths, CloudWatch log groups, the DynamoDB registries, and the local credential cache. There is no notion of an environment (dev / staging / production).

We want an app to own N environments, and every Keel operation to target an (app, env) pair.

2. The core model

An app owns N environments. Every operation targets an (app, env) pair.

We support both isolation models, selectable per environment:

• Model A — namespace isolation (default): environments live side-by-side in one AWS account, separated only by a <app>-<env> name prefix. Cheap, fast.
• Model B — account isolation: an environment targets its own AWS account and/or region. This is the AWS Well-Architected recommendation for production. Keel's per-env context directory already makes this natural — each env's auth.json carries its own AccountID / Region / role.

An environment opts into Model B simply by declaring an account (and optionally region) override. Absent that, it is Model A in the app's default account. The identity/naming layer is built once and is identical for both models; only the target account/region differ.

3. Config shape

One keel.yml per repo. App-level fields stay at the top; per-environment overrides live under environments:.

name: acme-api
region: us-east-1            # app default region

environments:
  production:
    region: us-east-1
    account: "111111111111"  # optional → Model B (separate account)
    services:
      web: { count: 3, cpu: 1024 }
    database: { size: db.r6g.large }
  staging:
    # no account/region override → Model A, app default account
    services:
      web: { count: 1 }
    database: { size: db.t4g.micro }

defaultEnvironment: production

• An environment's effective config = app-level base config deep-merged with that environment's overrides.
• If environments: is omitted entirely, Keel synthesizes a single environment named by defaultEnvironment (or default). Because there is no launched footprint, this synthetic env is still fully qualified as <app>-default — no special unqualified path.

Environment selection precedence

1. --env flag (global persistent flag on root command)
2. KEEL_ENV environment variable
3. Active-context file set by keel env use <env> (stored under ~/.keel/contexts/<app>/active-env)
4. defaultEnvironment from keel.yml

4. Identity & naming

Introduce a single stack identity: name_prefix = "<app>-<env>". Thread it through variables.tf / locals in internal/infra/hcl/generator.go so every inline-HCL generator inherits it. This is the single seam; ~15 downstream name sites pick it up automatically.

Resources that get qualified

Concern	Today	Proposed
OpenTofu state key	<app>/terraform.tfstate	<app>/<env>/terraform.tfstate (cli/helpers.go:97)
Working dir	.keel/infra (flat)	.keel/infra/<env> (infra/engine.go:27)
ECS cluster / task family / service	<app>-cluster, <app>-<svc>	<app>-<env>-cluster, <app>-<env>-<svc> (hcl/ecs.go:19,64,154)
ECR repo	<app>	<app>-<env> (hcl/ecr.go:9)
ALB / target group	<app>-alb, <app>-<svc>	<app>-<env>-… (see 32-char note) (hcl/alb.go:22,135)
IAM roles/policies	<app>-ecs-execution, etc.	<app>-<env>-… (hcl/iam.go:24,48,55,74)
Security groups	<app>-alb-sg, etc.	<app>-<env>-… (hcl/security_groups.go:34,73,126,157)
RDS subnet group / identifier	<app>-db, <app>-db-1	<app>-<env>-db… (hcl/rds.go:21,35,52,80)
ElastiCache	<app>-cache	<app>-<env>-cache (hcl/cache.go:23,35)
WAF	<app>-waf	<app>-<env>-waf (hcl/waf.go:46,54)
CloudWatch log group	/ecs/<app>	/ecs/<app>-<env> (hcl/cloudwatch.go:8)
CodeBuild project	<app>-<svc>-build	<app>-<env>-<svc>-build (hcl/codebuild.go:33)
SSM secrets path	/keel/<app>/<key>	/keel/<app>/<env>/<key> (hcl/ecs.go:128 + cli/env.go:53,88,119,164)
Registry (keel-apps)	PK=Name	PK=Name, SK=Environment (project/registry.go)
Deploy locks / history	PK=AppName	add Environment to key (auth/bootstrap.go:122,144)
Creds / context dir	~/.keel/contexts/<app>/	~/.keel/contexts/<app>/<env>/ (auth/config.go:41-46)

Gotchas found during mapping

1. 32-character AWS limit on ALB names and target-group names (hcl/alb.go:22,135). <app>-<env>-<svc> will overflow for many combinations. Use a deterministic truncate-plus-hash scheme: keep a readable prefix of the full stack name and append a short (e.g. 6-char) hash of the full name to guarantee uniqueness and stay under 32 chars.
2. Pre-existing SSM path bug — fix in the same pass. hcl/ecs.go:128 writes secrets to /keel/<app>/… but the IAM read policy at hcl/iam.go:43 grants /<app>/* (missing the keel/ prefix). They don't match today. Align both to the new env-qualified path /keel/<app>/<env>/*.
3. Shared literals — RDS database_name = "keeldb" / master_username = "keeluser" (hcl/rds.go:37-38,88-89) are identical across every app/env. Fine when the DB instance itself is env-qualified (separate instance per env), but revisit if we ever co-locate.
4. Shared-and-fine — the state bucket keel-state-<accountID>-<region> and lock table keel-locks (infra/backend.go:36-37) stay shared; lock IDs are per-state-key so per-env locking already works once the key is env-qualified.
5. Dead default — backend.go:59 sets Key = "keel/terraform.tfstate" but it's overridden at cli/helpers.go:97. Update the real seam (helpers.go), not the dead default.
6. Unused modules — internal/infra/modules/* are embedded but not imported by any generator (the inline-HCL generators are what actually run). If we ever switch to modules, they take a single app_name var and would need either an environment var or a composed app_name = "<app>-<env>".

5. Model B specifics (account isolation)

• Each environment resolves its own target account + region. Precedence: env override → app default → SDK default chain.
• The per-env context dir (~/.keel/contexts/<app>/<env>/) already isolates auth.json (AccountID / Region / RoleName / ExternalID), base_credentials, and STS session credentials. This is the natural home for a per-env account.
• keel auth setup / join must run per environment when envs live in different accounts, because the keel-admin/developer/viewer role trio and the keel-* backing tables are created per account (auth/bootstrap.go).
• The OpenTofu backend (S3 bucket + keel-locks table) is per account+region already, so a Model B env naturally gets its own isolated state backend in its own account. No extra work beyond pointing creds at the right account.

5b. Pre-existing infra/runtime naming mismatch (discovered during Phase 2)

While threading the env qualifier I found the runtime and the generated HCL already disagree on resource names, independent of environments — the tool has never been run end-to-end so these are latent:

Resource	HCL creates	Runtime targets
ECS cluster	<app>-cluster (now <app>-<env>-cluster)	bare <app> (clusterName/deployer.go:166)
ECS service	<app>-<svc> (now <app>-<env>-<svc>)	bare <svc> map key
ECR repo	single repo <app> (now <app>-<env>)	per-service <app>-<svc> (deployer.go:92)
CodeBuild project	<app>-<svc>-build (now <app>-<env>-<svc>-build)	<app>-<svc>-build (deployer.go:60)

Phase 3 must reconcile runtime naming with the HCL. Open product question: the HCL provisions a single ECR repo per app (all services share it, all pull :latest), but the deployer assumes a per-service repo. Either the HCL should create one repo per service (for_each) or the deployer should push per-service tags into one repo. This needs a decision before deploy can work; it is not caused by env separation.

Also fixed in Phase 2: the quito branch had regressed variables.tf type hints to the quoted legacy form (type = "string"), which fails tofu validate on the bundled OpenTofu 1.9; restored to the unquoted SetAttributeRaw form that origin/main uses.

5c. Deploy model — B2 (immutable, SHA-pinned)

Decided: one shared ECR repo per app-environment (<app>-<env>), images namespaced by service via the tag (<service>-<sha> and a moving <service>-latest). Deploys are immutable: keel registers a new ECS task-definition revision pinned to <service>-<sha> and points the service at it (aws/ecs.go DeployImage), instead of repointing :latest + force-redeploy. This makes rollback a matter of re-deploying a prior SHA and gives the tracker's rolled_back status real meaning.

• Local path: deployer.deployLocal builds/pushes both tags, then DeployImage.
• CodeBuild path: the inline buildspec (hcl/codebuild.go) only builds+pushes the service-namespaced tags; keel calls DeployImage after the build completes. Both paths share Deployer.PinImage.
• The initial task def created by keel up references <repo>:<service>-latest as its bootstrap image.

6. Phasing

Status: Phases 1, 2, 3 ✅ complete. Build + go vet + tests green; generated HCL tofu validate-clean; CLI smoke-tested. Phase 4 (per-env IAM least-privilege) still deferred.

Phase 3 as built:

• Naming: single source of truth in config/naming.go (NamePrefix, ClusterName, ServiceName, ECRRepo, CodeBuildProject, ImageTag/ImageTagLatest); the HCL generators and every runtime command derive names from it, fixing the pre-existing infra↔runtime mismatch (§5b).
• Credentials: scoped by a composite <app>/<env> context key passed as the existing appName argument → ~/.keel/contexts/<app>/<env>/ with no auth-package signature changes. Active-env stays app-level at ~/.keel/contexts/<app>/active-env. Helpers authCtx/authCtxKey in cli/helpers.go.
• DynamoDB: registry / deploy-lock / deployment-history isolated by a composite <app>/<env> partition value (plus descriptive App/Environment attributes) — no table-schema or bootstrap changes. keel apps shows an ENVIRONMENT column; history and locks are per-env.
• Deploy: B2 as above.

Phase 1 — Config + selection

• Add environments: map + defaultEnvironment to config.Config (config/config.go:20); deep-merge base + override.
• Global --env flag on root (cli/root.go), KEEL_ENV, and keel env use/list/current commands with an active-env file.
• Synthetic single environment when environments: is absent.

Phase 2 — Infra isolation

• name_prefix = "<app>-<env>" in generator locals (hcl/generator.go:108-142); every generator inherits it.
• Per-env working dir (engine.go:27) and per-env state key (helpers.go:97).
• 32-char truncate-plus-hash for ALB/TG names.
• Fix the SSM path inconsistency.

Phase 3 — Registry + runtime

• Composite registry key Name + Environment (project/registry.go); update Register / List / Deregister and AppEntry.
• Add Environment to deploy-lock and deploy-history keys.
• Per-env context dir for creds (auth/config.go).
• keel apps / keel env list surface the env axis; runtime commands (deploy, exec, scale, logs, status, config) resolve targets from (app, env).

Command naming: environment management is keel env (list/use/current); configuration variables/secrets moved from keel env to keel config (the config set/config get convention) to avoid the env/envs typo hazard. SSM parameters are scoped per environment at /keel/<app>/<env>/<key>.

Phase 4 — Auth isolation (DEFERRED, designed only)

• Per-env IAM roles and/or resource/tag-scoped policies so, e.g., a developer can deploy staging but not production within a single account (Model A fine-grained).
• For Model B, per-env accounts already provide a hard boundary; this phase is about intra-account least-privilege.
• Candidate approaches: (a) per-env role suffix keel-<env>-developer with policies scoped to the <app>-<env>-* name prefix / tag Environment=<env>; (b) permission boundaries; (c) SCPs when envs are separate accounts.

7. Open questions for later

• Do we want keel promote <from-env> <to-env> to copy an image/config between environments?
• Per-env keel.yml override files (keel.<env>.yml) vs. the single-file environments: map — we chose the single-file map; revisit if configs diverge heavily.
• How environment maps to Terraform workspaces vs. separate working dirs — we chose separate working dirs (.keel/infra/<env>) for clearer isolation and simpler state keys.

8. Key file seams (quick reference)

• internal/config/config.go:20 — add environments to Config
• internal/cli/root.go — --env flag, register keel env command
• internal/cli/helpers.go:97 — env-qualified state key; also loadAppName/newAWSClient/setupInfraEngine resolve (app, env)
• internal/infra/engine.go:27 — per-env working dir
• internal/infra/hcl/generator.go:108-142 — name_prefix seam
• internal/project/registry.go — composite key
• internal/auth/config.go:41-46 — per-env context dir
• internal/auth/bootstrap.go:122,144 — env in deploy-lock / history keys