Database credentials

Where the password lives

Keel provisions RDS with manage_master_user_password = true. That means AWS — not Keel — generates the master password, stores it in Secrets Manager, and rotates it on a schedule.

The consequence worth internalising: Keel never holds the password. It is not in keel.yml, not in OpenTofu state, not in ~/.keel, and not in any log Keel writes. Nothing you can grep will produce it, because nothing on your machine has ever had it.

The running tasks do not hold it either. Each service's task definition names the secret by ARN and ECS injects the values at container start:

Variable	Source
DATABASE_USER	the secret's username key
DATABASE_PASSWORD	the secret's password key
DATABASE_CREDENTIALS	the whole JSON payload, for a client that wants it
DATABASE_HOST	the endpoint, from the RDS resource
DATABASE_PORT	the port, from the RDS resource
DATABASE_NAME	the database name
DATABASE_SCHEME	postgres or mysql2, per the engine

So an application never needs the admin credentials, and should not use them: it already has what it needs, and it keeps working across a rotation.

There is no DATABASE_URL, and composing one needs care

The parts are injected separately. The cache gets CACHE_URL and REDIS_URL, so the asymmetry is real and deliberate: a cache URL can be composed by Keel because Keel knows every part of it at apply time, and a database URL cannot, because the password is a value Keel never sees. There is nowhere in the pipeline that could assemble one — ECS resolves the secret at container start, after every task-definition value is fixed, and a URL baked at apply time would be a copy of a password that rotates every seven days.

So the composition happens in your application, and the hazard is percent-encoding:

# Wrong. A password containing ? # % or / silently produces a URL that parses
# into something else — usually an auth failure, occasionally a connection to a
# different database.
DATABASE_URL="$DATABASE_SCHEME://$DATABASE_USER:$DATABASE_PASSWORD@$DATABASE_HOST:$DATABASE_PORT/$DATABASE_NAME"

RDS forbids /, ", @ and space in a master password, so the two characters that break a URL most obviously cannot occur — which is exactly what makes this look like it works. ?, # and % are permitted and each breaks the parse.

Read the parts directly where the framework allows it. In Rails that is config/database.yml:

production:
  adapter: postgresql
  host:     <%= ENV["DATABASE_HOST"] %>
  port:     <%= ENV["DATABASE_PORT"] %>
  database: <%= ENV["DATABASE_NAME"] %>
  username: <%= ENV["DATABASE_USER"] %>
  password: <%= ENV["DATABASE_PASSWORD"] %>

Where a URL is the only input the framework takes — dj-database-url, Prisma — build it in the entrypoint with the encoding done properly:

#!/bin/sh
export DATABASE_URL="$(python3 - <<'PY'
import os, urllib.parse as u
q = lambda v: u.quote(v, safe="")
print("%s://%s:%s@%s:%s/%s" % (
    os.environ["DATABASE_SCHEME"], q(os.environ["DATABASE_USER"]),
    q(os.environ["DATABASE_PASSWORD"]), os.environ["DATABASE_HOST"],
    os.environ["DATABASE_PORT"], os.environ["DATABASE_NAME"]))
PY
)"
exec "$@"

More than one database role

Keel provisions one role: the RDS master, which AWS generates and rotates. It does not create application or migration roles, because creating a Postgres role means running SQL inside the VPC and there is no moment in an apply where that is a safe, idempotent thing for a provisioning tool to do on your behalf.

The supported pattern is to create them yourself and let Keel deliver the credential to the right task. This matters more than it sounds: a runtime role that can bypass row-level security makes every tenant-isolation policy in the schema advisory, so "the migrator is more privileged than the application" is the normal arrangement rather than an exotic one.

1. Create the roles once, as the master user, from inside the VPC:

   $ keel exec web -c 'psql "$(...)" -c "
       CREATE ROLE app_runtime LOGIN PASSWORD :'"'"'pw'"'"' NOSUPERUSER NOBYPASSRLS;
       CREATE ROLE app_migrator LOGIN PASSWORD :'"'"'pw2'"'"';
       GRANT CREATE ON SCHEMA public TO app_migrator;"'

   Or run the same statements from a migration, which is easier to keep in version control.

2. Store each credential as a Keel secret:

   $ keel config set RUNTIME_DATABASE_URL 'postgres://app_runtime:...@...'
   $ keel config set MIGRATOR_DATABASE_URL 'postgres://app_migrator:...@...'

3. Scope them. The runtime credential goes on the services; the migration credential goes on the release and nowhere else:

   services:
     web:
       secrets: [RUNTIME_DATABASE_URL]
     worker:
       secrets: [RUNTIME_DATABASE_URL]
   
   release:
     command: bundle exec rails db:migrate
     service: web
     secrets:
       - MIGRATOR_DATABASE_URL

release.secrets puts the migrator's credential on a task definition of its own — {app}-{env}-release — which exists only for the length of one migration. Without it the credential has to be declared on a long-running service, where it sits in every internet-facing task for the life of the deployment in order to be read once a deploy.

Two things this does not do. Keel does not rotate these passwords: they are ordinary SSM parameters, and rotating one means writing a new value and redeploying. And the release still runs with the release service's task role and security group — the isolation is of the database credential, not of the AWS identity.

Each person as themselves

Everything above shares one credential, which is the honest limit of it: keel db psql, a tunnel and their own client, and the secret pasted into a GUI are all the master user, so %u in log_line_prefix attributes every statement to the same name. database.iam_auth closes that (Pro):

database:
  iam_auth: true

$ keel db grant alice        # creates the database role "keel-alice"
$ keel db revoke alice       # drops it; alice's IAM access is untouched

Two halves, and neither is sufficient on its own. IAM authorises minting a token for a database user of a given name; the database must separately hold a role of that name that accepts token authentication (rds_iam, or AWSAuthenticationPlugin). So turning the flag on changes nothing until somebody is granted — which the refusal says, because a flag that appears to do nothing is one people set twice and then turn off.

The database user is the IAM user name with a keel- prefix and nothing else done to it. Three strings have to be identical — the session tag IAM substitutes, the resource pattern the grant matches, and the role name the DDL creates — and IAM substitutes the raw name, so a sanitiser in the middle of that chain looks like hygiene while breaking it. An awkward name is refused rather than rewritten.

A grant creates a login and no privileges. What that person may read stays the schema owner's decision, made in SQL where it is reviewable, exactly like the roles in the section above.

Three consequences worth planning around:

• The application keeps the master secret. A token expires in fifteen minutes and IAM authentication has a connection-rate ceiling. So the claim is that human access is attributable; application access is still one identity.
• Every failure falls back to the master credential — the flag off, not an IAM user, not granted yet, a token that could not be minted — and none of them is an error. The shell says which identity it will connect with before it connects.
• TLS is require, not verify-full. The token is signed for the database's endpoint while your client dials 127.0.0.1 through the tunnel, so no certificate can match the name that was dialled. RDS refuses an unencrypted token connection outright, so the encryption is not optional — but "encrypted" and "verified" are different claims, and this is the first.

A session opened before the grant keeps running as the master user, and the correlation ID Keel stamps as PGAPPNAME is not joined to the database user. See docs/design/database-identity.md.

Seeing that it exists

The config views list it alongside the variables someone set, because a service's environment carries both:

$ keel config list

NAME              TYPE           VERSION
SECRET_KEY_BASE   SecureString   3

Managed by AWS — injected into every service, not editable here:

  @database credentials  (active)
    DATABASE_USER ← username               the master username
    DATABASE_PASSWORD ← password           generated by AWS, rotated on a schedule
    DATABASE_CREDENTIALS                   the whole JSON secret, for a client that wants it
    AWS Secrets Manager, generated and rotated by RDS
    rotated by AWS every 7 days — do not copy the value
    secret   arn:aws:secretsmanager:us-west-2:123456789012:secret:rds!db-1a2b3c-AbCdEf
    values   keel db credentials

The dashboard's Config tab shows the same thing, in a block below the variables table and outside the cursor's range — the selection there drives unset, and none of this can be unset.

No value is read to produce that listing: it comes from DescribeSecret, which returns metadata only. So a viewer or a developer sees the secret exists, what it injects and how often it rotates, without holding any permission that could read it.

keel config set and keel config unset refuse these names rather than reporting a missing SSM parameter:

$ keel config unset DATABASE_PASSWORD
Error: DATABASE_PASSWORD is not a configuration variable and cannot be unset: it comes
from @database credentials, which AWS generates and rotates. Read the values with
'keel db credentials'.

Reading them as a human

For the cases an application does not cover — connecting with psql, running a one-off query, creating a second database user — there are two commands. Both need admin access, because both call secretsmanager:GetSecretValue.

$ keel db credentials

acme (production) — postgres

  host       acme-production-db.abc123.us-west-2.rds.amazonaws.com
  port       5432
  database   acme
  username   keeladmin
  password   ••••••••••••••••   (--show to reveal)

  secret     arn:aws:secretsmanager:us-west-2:123456789012:secret:rds!db-1a2b3c-AbCdEf

The password is masked by default: checking an endpoint or the admin username is the common case, and it should not put a credential into a shared screen or a scrollback buffer. --show prints it.

For a connection string:

$ psql "$(keel db url)"

keel db url is the only Keel command that emits a password, and it emits it only when asked. Do not store what it prints — the password rotates, and a saved URL stops working.

Reaching the database at all

The database sits in private subnets, and its security group only admits the service security groups on the engine's port. It is not reachable from a laptop, whatever credentials you hold.

There are two routes in, and both go through a task that is already running in the private subnets.

The first needs no admin access and no admin password:

$ keel exec web -c \
    'psql "postgres://$DATABASE_USER:$DATABASE_PASSWORD@$DATABASE_HOST:$DATABASE_PORT/$DATABASE_NAME"'

That uses the service's own injected values and is the right answer for almost every "I need to look at the database" moment.

The second forwards a local port, so your own client reaches the database from your machine:

$ keel tunnel @database
Tunnel to @database (production)
  localhost:5432 → acme-production-db.abc123.us-west-2.rds.amazonaws.com:5432
  through task 9f8e7d

$ keel db psql          # the same tunnel, plus psql and the master credential

keel tunnel is the primitive and everything else is a wrapper — TablePlus, DataGrip and a migration tool all work against it without Keel knowing about them. It is SSM port forwarding through the same data channel keel exec uses, with a different document: nothing new is exposed, no listening port, no bastion, and it costs nothing when idle. It does need a running task to go through, so a scaled-to-zero environment has nothing to tunnel from, which is reported rather than hung on.

keel db psql (also keel db mysql, keel db shell) is admin, and it asks before connecting, naming the environment: the credential it hands the client is the master user, which can drop the schema. The password reaches the client through the environment (PGPASSWORD, MYSQL_PWD) and never the command line, where an argument list is world-readable in ps.

keel db credentials is for the cases where you genuinely need the master user in your own hands — creating roles, granting to a new user, an out-of-band migration.

Rotation

AWS rotates the master password on the schedule the secret carries. Two things follow:

• Copies expire. Anything you paste into a password manager, a .env, or a CI variable will stop working, silently, at a time nobody chose. Read the secret each time instead.
• Nothing needs redeploying when it rotates. The task definitions reference the secret by ARN, so ECS picks up the new value the next time a task starts.

If keel db credentials says the output is missing

A stack applied before Keel emitted the database_secret_arn output has no record of where the secret is. Re-applying adds the output:

$ keel up --env production

Nothing is recreated — an output is not a resource.