# Database credentials ## Where the password lives Keel provisions RDS with `manage_master_user_password = true`. That means AWS — not Keel — generates the master password, stores it in Secrets Manager, and rotates it on a schedule. The consequence worth internalising: **Keel never holds the password.** It is not in `keel.yml`, not in OpenTofu state, not in `~/.keel`, and not in any log Keel writes. Nothing you can grep will produce it, because nothing on your machine has ever had it. The running tasks do not hold it either. Each service's task definition names the secret by ARN and ECS injects the values at container start: | Variable | Source | | ---------------------- | --------------------------------------------- | | `DATABASE_USER` | the secret's `username` key | | `DATABASE_PASSWORD` | the secret's `password` key | | `DATABASE_CREDENTIALS` | the whole JSON payload, for a client that wants it | | `DATABASE_HOST` | the endpoint, from the RDS resource | | `DATABASE_PORT` | the port, from the RDS resource | | `DATABASE_NAME` | the database name | | `DATABASE_SCHEME` | `postgres` or `mysql2`, per the engine | So an application never needs the admin credentials, and should not use them: it already has what it needs, and it keeps working across a rotation. ### There is no `DATABASE_URL`, and composing one needs care The parts are injected separately. The cache gets `CACHE_URL` and `REDIS_URL`, so the asymmetry is real and deliberate: a cache URL can be composed by Keel because Keel knows every part of it at apply time, and a database URL cannot, because the password is a value Keel never sees. There is nowhere in the pipeline that could assemble one — ECS resolves the secret at container start, after every task-definition value is fixed, and a URL baked at apply time would be a copy of a password that rotates every seven days. So the composition happens in your application, and the hazard is percent-encoding: ```sh # Wrong. A password containing ? # % or / silently produces a URL that parses # into something else — usually an auth failure, occasionally a connection to a # different database. DATABASE_URL="$DATABASE_SCHEME://$DATABASE_USER:$DATABASE_PASSWORD@$DATABASE_HOST:$DATABASE_PORT/$DATABASE_NAME" ``` RDS forbids `/`, `"`, `@` and space in a master password, so the two characters that break a URL most obviously cannot occur — which is exactly what makes this look like it works. `?`, `#` and `%` are permitted and each breaks the parse. Read the parts directly where the framework allows it. In Rails that is `config/database.yml`: ```yaml production: adapter: postgresql host: <%= ENV["DATABASE_HOST"] %> port: <%= ENV["DATABASE_PORT"] %> database: <%= ENV["DATABASE_NAME"] %> username: <%= ENV["DATABASE_USER"] %> password: <%= ENV["DATABASE_PASSWORD"] %> ``` Where a URL is the only input the framework takes — dj-database-url, Prisma — build it in the entrypoint with the encoding done properly: ```sh #!/bin/sh export DATABASE_URL="$(python3 - <<'PY' import os, urllib.parse as u q = lambda v: u.quote(v, safe="") print("%s://%s:%s@%s:%s/%s" % ( os.environ["DATABASE_SCHEME"], q(os.environ["DATABASE_USER"]), q(os.environ["DATABASE_PASSWORD"]), os.environ["DATABASE_HOST"], os.environ["DATABASE_PORT"], os.environ["DATABASE_NAME"])) PY )" exec "$@" ``` ## More than one database role Keel provisions one role: the RDS master, which AWS generates and rotates. It does not create application or migration roles, because creating a Postgres role means running SQL inside the VPC and there is no moment in an apply where that is a safe, idempotent thing for a provisioning tool to do on your behalf. The supported pattern is to create them yourself and let Keel deliver the credential to the right task. This matters more than it sounds: a runtime role that can bypass row-level security makes every tenant-isolation policy in the schema advisory, so "the migrator is more privileged than the application" is the normal arrangement rather than an exotic one. 1. Create the roles once, as the master user, from inside the VPC: ``` $ keel exec web -c 'psql "$(...)" -c " CREATE ROLE app_runtime LOGIN PASSWORD :'"'"'pw'"'"' NOSUPERUSER NOBYPASSRLS; CREATE ROLE app_migrator LOGIN PASSWORD :'"'"'pw2'"'"'; GRANT CREATE ON SCHEMA public TO app_migrator;"' ``` Or run the same statements from a migration, which is easier to keep in version control. 2. Store each credential as a Keel secret: ``` $ keel config set RUNTIME_DATABASE_URL 'postgres://app_runtime:...@...' $ keel config set MIGRATOR_DATABASE_URL 'postgres://app_migrator:...@...' ``` 3. Scope them. The runtime credential goes on the services; the migration credential goes on the release and nowhere else: ```yaml services: web: secrets: [RUNTIME_DATABASE_URL] worker: secrets: [RUNTIME_DATABASE_URL] release: command: bundle exec rails db:migrate service: web secrets: - MIGRATOR_DATABASE_URL ``` `release.secrets` puts the migrator's credential on a task definition of its own — `{app}-{env}-release` — which exists only for the length of one migration. Without it the credential has to be declared on a long-running service, where it sits in every internet-facing task for the life of the deployment in order to be read once a deploy. Two things this does not do. Keel does not rotate these passwords: they are ordinary SSM parameters, and rotating one means writing a new value and redeploying. And the release still runs with the release service's task role and security group — the isolation is of the database credential, not of the AWS identity. ## Each person as themselves Everything above shares one credential, which is the honest limit of it: `keel db psql`, a tunnel and their own client, and the secret pasted into a GUI are all the master user, so `%u` in `log_line_prefix` attributes every statement to the same name. `database.iam_auth` closes that (Pro): ```yaml database: iam_auth: true ``` ``` $ keel db grant alice # creates the database role "keel-alice" $ keel db revoke alice # drops it; alice's IAM access is untouched ``` **Two halves, and neither is sufficient on its own.** IAM authorises minting a token for a database user of a given name; the database must separately hold a role of that name that accepts token authentication (`rds_iam`, or `AWSAuthenticationPlugin`). So turning the flag on changes nothing until somebody is granted — which the refusal says, because a flag that appears to do nothing is one people set twice and then turn off. The database user is the IAM user name with a `keel-` prefix and nothing else done to it. Three strings have to be identical — the session tag IAM substitutes, the resource pattern the grant matches, and the role name the DDL creates — and IAM substitutes the *raw* name, so a sanitiser in the middle of that chain looks like hygiene while breaking it. An awkward name is refused rather than rewritten. A grant creates **a login and no privileges**. What that person may read stays the schema owner's decision, made in SQL where it is reviewable, exactly like the roles in the section above. Three consequences worth planning around: - **The application keeps the master secret.** A token expires in fifteen minutes and IAM authentication has a connection-rate ceiling. So the claim is that *human* access is attributable; application access is still one identity. - **Every failure falls back to the master credential** — the flag off, not an IAM user, not granted yet, a token that could not be minted — and none of them is an error. The shell says which identity it will connect with before it connects. - **TLS is `require`, not `verify-full`.** The token is signed for the database's endpoint while your client dials `127.0.0.1` through the tunnel, so no certificate can match the name that was dialled. RDS refuses an unencrypted token connection outright, so the encryption is not optional — but "encrypted" and "verified" are different claims, and this is the first. A session opened before the grant keeps running as the master user, and the correlation ID Keel stamps as `PGAPPNAME` is not joined to the database user. See `docs/design/database-identity.md`. ## Seeing that it exists The config views list it alongside the variables someone set, because a service's environment carries both: ``` $ keel config list NAME TYPE VERSION SECRET_KEY_BASE SecureString 3 Managed by AWS — injected into every service, not editable here: @database credentials (active) DATABASE_USER ← username the master username DATABASE_PASSWORD ← password generated by AWS, rotated on a schedule DATABASE_CREDENTIALS the whole JSON secret, for a client that wants it AWS Secrets Manager, generated and rotated by RDS rotated by AWS every 7 days — do not copy the value secret arn:aws:secretsmanager:us-west-2:123456789012:secret:rds!db-1a2b3c-AbCdEf values keel db credentials ``` The dashboard's Config tab shows the same thing, in a block below the variables table and outside the cursor's range — the selection there drives unset, and none of this can be unset. No value is read to produce that listing: it comes from `DescribeSecret`, which returns metadata only. So a viewer or a developer sees the secret exists, what it injects and how often it rotates, without holding any permission that could read it. `keel config set` and `keel config unset` refuse these names rather than reporting a missing SSM parameter: ``` $ keel config unset DATABASE_PASSWORD Error: DATABASE_PASSWORD is not a configuration variable and cannot be unset: it comes from @database credentials, which AWS generates and rotates. Read the values with 'keel db credentials'. ``` ## Reading them as a human For the cases an application does not cover — connecting with `psql`, running a one-off query, creating a second database user — there are two commands. Both need **admin** access, because both call `secretsmanager:GetSecretValue`. ``` $ keel db credentials acme (production) — postgres host acme-production-db.abc123.us-west-2.rds.amazonaws.com port 5432 database acme username keeladmin password •••••••••••••••• (--show to reveal) secret arn:aws:secretsmanager:us-west-2:123456789012:secret:rds!db-1a2b3c-AbCdEf ``` The password is masked by default: checking an endpoint or the admin username is the common case, and it should not put a credential into a shared screen or a scrollback buffer. `--show` prints it. For a connection string: ``` $ psql "$(keel db url)" ``` `keel db url` is the only Keel command that emits a password, and it emits it only when asked. Do not store what it prints — the password rotates, and a saved URL stops working. ## Reaching the database at all The database sits in private subnets, and its security group only admits the service security groups on the engine's port. It is not reachable from a laptop, whatever credentials you hold. There are two routes in, and both go through a task that is already running in the private subnets. The first needs no admin access and no admin password: ``` $ keel exec web -c \ 'psql "postgres://$DATABASE_USER:$DATABASE_PASSWORD@$DATABASE_HOST:$DATABASE_PORT/$DATABASE_NAME"' ``` That uses the service's own injected values and is the right answer for almost every "I need to look at the database" moment. The second forwards a local port, so your own client reaches the database from your machine: ``` $ keel tunnel @database Tunnel to @database (production) localhost:5432 → acme-production-db.abc123.us-west-2.rds.amazonaws.com:5432 through task 9f8e7d $ keel db psql # the same tunnel, plus psql and the master credential ``` `keel tunnel` is the primitive and everything else is a wrapper — TablePlus, DataGrip and a migration tool all work against it without Keel knowing about them. It is SSM port forwarding through the same data channel `keel exec` uses, with a different document: nothing new is exposed, no listening port, no bastion, and it costs nothing when idle. It does need a running task to go through, so a scaled-to-zero environment has nothing to tunnel from, which is reported rather than hung on. `keel db psql` (also `keel db mysql`, `keel db shell`) is **admin**, and it asks before connecting, naming the environment: the credential it hands the client is the master user, which can drop the schema. The password reaches the client through the environment (`PGPASSWORD`, `MYSQL_PWD`) and never the command line, where an argument list is world-readable in `ps`. `keel db credentials` is for the cases where you genuinely need the master user in your own hands — creating roles, granting to a new user, an out-of-band migration. ## Rotation AWS rotates the master password on the schedule the secret carries. Two things follow: - Copies expire. Anything you paste into a password manager, a `.env`, or a CI variable will stop working, silently, at a time nobody chose. Read the secret each time instead. - Nothing needs redeploying when it rotates. The task definitions reference the secret by ARN, so ECS picks up the new value the next time a task starts. ## If `keel db credentials` says the output is missing A stack applied before Keel emitted the `database_secret_arn` output has no record of where the secret is. Re-applying adds the output: ``` $ keel up --env production ``` Nothing is recreated — an output is not a resource.