Start shipping →

Answers · Keys & Services

How do I rotate an API key without taking the app down?

Create the new key at the provider while the old one still works, set it, deploy, verify the app is using it, and only then revoke the old one. Most providers allow several active keys for exactly this reason. Revoking before deploying is what turns a routine rotation into an outage.

Updated · 2 min read

The order

  1. Create the new key in the provider's dashboard. The old one keeps working.
  2. Set it, in .kamal/secrets or your credentials file.
  3. Deploy. Both keys are now valid; only one is in use.
  4. Verify the app is actually using the new one.
  5. Revoke the old key.
bin/kamal secrets print
bin/kamal deploy

Step four is the one people skip, and it is why step five occasionally causes an incident: if the deploy did not pick up the new value, revoking removes the key that was actually working.

Verifying without printing the secret

bin/kamal app exec 'bin/rails runner "puts ENV[%q(STRIPE_SECRET_KEY)].to_s.last(4)"'

Last four characters is enough to tell two keys apart and is not enough to be worth protecting. Never print a whole key into a terminal, a CI log or a chat window, all three of which keep history.

Keys that cannot be rotated this way

RAILS_MASTER_KEY. It is not a provider credential; it is the only thing that decrypts config/credentials.yml.enc. Rotating it means re-encrypting the file, which means generating a new key and a new file together and deploying both. There is no window where both are valid.

Webhook signing secrets. Rotating one means Stripe signs with the new secret immediately, so there is a brief window where in-flight deliveries fail. They retry for three days, so the practical answer is to rotate and let the retries backfill rather than trying to avoid the gap.

Rotate on a trigger, not a calendar

The useful triggers: someone with access leaves, a key appeared somewhere it should not have, or the provider says so. Rotating on a schedule for its own sake mostly generates opportunities to get the order wrong.

Prevent the leak that causes the emergency

bin/check

That runs a secret scan over the tree, so a key pasted into a file fails the build rather than reaching the repository history, where rotation becomes the only remedy.

Related questions

Should I use Rails credentials or environment variables for secrets?

Either. The app reads ENV first and falls back to Rails.application.credentials, so a value set in the environment wins and a value in the encrypted file is the default. Use credentials when you want secrets versioned with the code, and environment variables when a platform or a team already manages them elsewhere.

What does SECRET_KEY_BASE actually do in a Rails app?

It is the root secret Rails derives every other key from: the one that signs session cookies, encrypts encrypted cookies, and verifies signed global ids and Active Storage URLs. Change it and every existing session becomes invalid, so every signed-in user is logged out at the moment of deploy.

What happens if I lose the Rails master key?

The encrypted credentials are gone. RAILS_MASTER_KEY is the only thing that decrypts config/credentials.yml.enc, and there is no recovery path by design. Delete both files, regenerate them, and re-enter every secret from its source. Production will not boot until the key reaching the server matches the file in the repo.