Cyber security · Secrets

Secrets Management: Stop Leaking Keys in Git and Logs

A practical workflow for dev, CI, and production.

Reading time: ~8–12 min
Level: All levels
Updated:

Secrets management is less about “hiding strings” and more about building a workflow where keys don’t end up in Git, don’t show up in logs, and can be rotated without drama. This post gives you a practical, copy/paste-friendly setup for local dev, CI, and production—plus the common failure modes that cause real-world leaks (even in otherwise well-run teams).


Quickstart

If you only do a few things, do these. They’re high-impact and they prevent the two most common leak paths: commits and logs.

1) Stop secrets at the door (Git)

  • Add .env/.secrets patterns to .gitignore (and review existing repos for misses).
  • Install a pre-commit secrets scanner so leaks are blocked before they land in history.
  • Enable a repo scanner in CI so pull requests can’t bypass local checks.

2) Move secrets to a “system of record”

  • Pick one store: a cloud secrets manager, Vault, or your platform’s managed secret store.
  • Store only references in code (names/paths), not values.
  • Give each app/environment its own secret namespace and IAM policy (least privilege).

3) Make CI use short-lived credentials

  • Prefer OIDC-based federation over long-lived access keys in CI.
  • Fetch secrets at job runtime (not as committed files, not as build args).
  • Mask secrets and avoid printing env vars in CI logs.

4) Prevent leaks in logs and error reports

  • Redact known secret patterns (tokens, Authorization headers) in your logger.
  • Never log entire request bodies by default; sample carefully.
  • Rotate credentials immediately if a secret hits logs (treat as compromised).
One-line rule that saves teams

Secrets should be written once (to a secret store), read at runtime (by the app), and never copied into Git, images, tickets, or dashboards.

Overview

A leaked API key is rarely “someone did something dumb”. It’s usually a predictable workflow failure: a developer pastes a token into a config file, a build step bakes it into an image, a debug log prints headers, or CI stores a long-lived credential that gets copied across repos.

This guide is a practical secrets management playbook designed for real-world teams: small projects, growing startups, and mature orgs that still occasionally ship a secret into the wrong place.

What you’ll get from this post

  • A working baseline for dev → CI → production secrets flow
  • How to prevent secrets from entering Git history and logs
  • How to choose a secret store (and when env vars are fine)
  • A step-by-step setup, plus mistakes + fixes and an incident mini-playbook

A simple threat model for secrets

Leak path What it looks like What to do
Git Token in a commit, PR, or git tags Block with scanning + rotate immediately
Logs Authorization header or env dump in CI/runtime logs Redact + stop logging sensitive fields
Images/build artifacts Secret baked into a container layer Fetch at runtime; never via build args
Over-permissioned access One token can read all environments Least privilege + separate namespaces
What counts as a secret?

Anything that grants access: API tokens, private keys, database passwords, session signing keys, webhook secrets, and even “harmless” internal endpoints when combined with other data. If it enables an action, treat it as a secret.

Core concepts

1) The secret lifecycle: create → store → access → rotate → revoke

Good secrets management is a lifecycle. Most teams only do “create” and “paste somewhere” — then wonder why rotation is painful. Design for rotation from day one:

  • Create: generate secrets with high entropy (avoid “memorable” strings).
  • Store: put them in a dedicated secret store (source of truth).
  • Access: apps fetch at runtime with least privilege.
  • Rotate: replace on schedule or after exposure; avoid manual steps.
  • Revoke: immediately invalidate leaked/unused credentials.

2) Config vs secret: don’t treat them the same

A healthy setup separates configuration (safe to see) from secrets (must be protected). When they’re mixed together, secrets leak via normal config workflows (Git commits, tickets, screenshots, pastebins).

Quick sorting rule

Type Examples Where it belongs
Config feature flags, non-sensitive URLs, timeouts, log levels Git (committed), config maps, app settings
Secrets API keys, DB passwords, private keys, signing keys Secret manager / Vault / platform secret store

3) Four principles that keep secrets from spilling

Least privilege

A single leaked token shouldn’t unlock every environment. Scope access by app + environment, and avoid “shared admin keys”.

One system of record

Secrets should live in one place (the secret store). Copies multiply leak paths and make rotation chaotic.

Runtime injection

Apps should receive secrets at runtime (env vars, mounted files, sidecars)—not at build time and not committed.

Assume exposure

If a secret appears in Git or logs, treat it as compromised. Redaction is not a fix; rotation is the fix.

Base64 is not encryption

Encoding (base64) only changes representation. Anyone can decode it. If you need to store sensitive values outside a secret store, you need real encryption (and a safe way to manage encryption keys).

4) Secret storage options (what to pick)

You don’t need the “perfect” tool. You need a setup that (1) prevents Git leaks, (2) supports runtime access, and (3) enables rotation. Here’s a practical comparison:

Option Good for Watch out for
Cloud secrets manager Most teams; managed rotation/integration; audit logs IAM complexity; costs if abused at high QPS without caching
Vault / dedicated secret platform Advanced needs; dynamic creds; multi-cloud Operational burden; must secure and monitor Vault itself
Platform secrets (CI, K8s) Convenient injection; close to runtime Be careful with RBAC; avoid storing long-lived “source of truth” only here
Encrypted files (SOPS, etc.) Infra-as-code; GitOps workflows; small teams Key management; decryption in CI; still need rotation discipline

Step-by-step

This is a practical secrets management workflow that scales from solo projects to teams. The steps are ordered to reduce risk quickly: first block Git leaks, then fix CI, then harden production and logging.

Step 1 — Inventory and classify secrets (you can’t protect what you don’t see)

  • List every secret type: API keys, DB creds, OAuth client secrets, JWT signing keys, webhook secrets.
  • Map them to who uses them (service), where (dev/CI/prod), and impact if leaked.
  • Decide ownership: which team/person can rotate or revoke each one.

Step 2 — Block secrets from entering Git (local + CI)

Git is “forever”. Even if you remove a secret later, it may still exist in history, forks, caches, and mirrors. The goal is to prevent the first bad commit.

Local dev: minimum baseline

  • Use .env for local-only values and never commit it.
  • Check that your Docker builds and tests do not copy .env into images/artifacts.
  • Use a pre-commit scanner so mistakes are blocked automatically.

Repo-level enforcement

  • Run a secret scanner in CI on every PR and push.
  • Block merges on findings (with a documented exception flow for false positives).
  • Educate contributors: “never paste tokens into issues/PRs either.”

Here’s a pre-commit configuration you can drop into most repos. It runs quickly and catches common credential patterns before they become a permanent part of your git history.

# .pre-commit-config.yaml
# Install: pipx install pre-commit && pre-commit install
# Run manually: pre-commit run --all-files

repos:
  - repo: https://github.com/gitleaks/gitleaks
    rev: v8.22.1
    hooks:
      - id: gitleaks
        name: "gitleaks (block secrets)"
        args: ["--no-git", "--redact"]
What if you already leaked a secret in Git?

Rotation comes first. Then clean up: remove from working tree, rewrite history if needed, and invalidate old credentials. Treat “we deleted the file” as incomplete until the secret is rotated and access is audited.

Step 3 — Put secrets in a central store (source of truth)

Central storage solves three problems at once: access control (who can read), auditing (who did read), and rotation (change the value once, not in five repos).

How to structure secrets so they’re manageable

  • Use names that encode app + env + purpose (e.g., payments/prod/stripe_api_key).
  • Keep dev and prod completely separate (different namespaces, accounts, projects, or mounts).
  • Prefer per-service credentials over shared “platform keys”.
  • Set a rotation policy (even if it’s manual at first).

Step 4 — Fix CI: short-lived access + runtime fetching

CI is a favorite target because it often has powerful permissions and verbose logs. Your goal is: no long-lived cloud keys stored in CI, and no secrets printed.

CI best practices (portable)

  • Use OIDC federation where possible (CI gets temporary credentials).
  • Fetch secrets at runtime from the secret store (don’t commit secret files).
  • Disable shell debug flags (set -x) around secret handling.
  • Mask secrets and never print full environment dumps.

What to avoid

  • Storing cloud access keys as CI variables “forever”.
  • Passing secrets as Docker build args.
  • Writing secrets into artifacts (test reports, cached build outputs).
  • Using one credential for all repos/environments.

Example: GitHub Actions using OIDC to assume a cloud role and fetch a secret at job runtime. The exact action/provider varies by cloud, but the pattern is the same: federatefetchuse.

# .github/workflows/ci.yml
name: ci

on:
  push:
  pull_request:

permissions:
  id-token: write   # required for OIDC
  contents: read

jobs:
  test:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4

      # Example: assume a cloud role via OIDC (AWS shown)
      - name: Configure AWS credentials (OIDC)
        uses: aws-actions/configure-aws-credentials@v4
        with:
          role-to-assume: ${{ secrets.AWS_ROLE_ARN }}
          aws-region: eu-central-1

      - name: Fetch secret at runtime
        shell: bash
        run: |
          set -euo pipefail
          SECRET_JSON="$(aws secretsmanager get-secret-value --secret-id app/prod/db --query SecretString --output text)"
          DB_PASSWORD="$(python -c 'import json,sys; print(json.loads(sys.argv[1])["password"])' "$SECRET_JSON")"
          echo "::add-mask::$DB_PASSWORD"
          DB_PASSWORD="$DB_PASSWORD" ./scripts/run-tests.sh
CI log safety checklist
  • Never enable verbose debug output around secret handling.
  • Mask secrets early (before any command could echo them).
  • Avoid printing request headers/bodies in integration tests.
  • Ensure “failure output” (stack traces) doesn’t include secret values.

Step 5 — Production: inject secrets at runtime (and keep them out of images)

The most common production mistake is accidentally baking secrets into a container image or deploy artifact. If an image has a secret, anyone who can pull the image can extract it.

Runtime injection options (choose one)

  • Env vars: good for many apps, but be careful with logging and process dumps.
  • Mounted files: reduces accidental prints; good for certificates/keys.
  • Sidecar/agent: fetches secrets and renews them dynamically.

Example: Kubernetes External Secrets pattern (operator-based) where the cluster reads from a secret store and materializes a Kubernetes Secret. The exact CRD may differ by operator/provider, but the structure is consistent.

# external-secret.yaml (example pattern)
apiVersion: external-secrets.io/v1beta1
kind: ExternalSecret
metadata:
  name: app-db
spec:
  refreshInterval: 1h
  secretStoreRef:
    name: prod-secret-store
    kind: SecretStore
  target:
    name: app-db
    creationPolicy: Owner
  data:
    - secretKey: DB_PASSWORD
      remoteRef:
        key: app/prod/db
        property: password

Step 6 — Fix logging: redact by default and be intentional about debug output

Logs are a data pipeline: CI logs, runtime logs, APM traces, error trackers, analytics. If a secret lands there, it gets copied, indexed, and retained. The winning approach is default redaction plus a culture of “log intent”.

What to log (safe)

  • Request IDs, user IDs (non-sensitive), timing, status codes
  • High-level event names (e.g., payment_failed)
  • Counts and aggregates, not raw payloads
  • Structured fields with allow-lists

What not to log (common leaks)

  • Authorization headers, cookies, session tokens
  • Full request bodies (especially webhooks and auth callbacks)
  • Entire environment dumps (printenv)
  • Secrets inside exception messages (be careful with SDKs)
Practical redaction strategy

Redact by key name (e.g., password, token, secret) and by pattern (JWT-like strings, long hex/base64 tokens), and use an allow-list for “safe fields” in request logs.

Step 7 — Rotation and incident response (don’t improvise later)

Rotation is the “fire drill” of secrets management. If rotation requires five people and a weekend, your system will drift into unsafe habits. Build a small playbook and practice it.

Rotation baseline

  • Know where each secret is used (service inventory).
  • Prefer dual-secret rotation where possible (new secret works before old is revoked).
  • Rotate on schedule (e.g., quarterly) and on incident (immediately).
  • After rotation, verify with a canary request and monitor error rates.
If a secret appears in logs or Git
  • Assume compromise: rotate/revoke immediately.
  • Identify where the secret spread (forks, caches, artifacts, log indexes).
  • Reduce blast radius: tighten permissions and segment environments.
  • Write a small postmortem: which workflow allowed it and what guardrail stops it next time.

Common mistakes

These are the patterns behind “we didn’t mean to leak anything.” Each mistake includes the practical fix you can apply immediately.

Mistake 1 — “We’ll remove it later” (Git is forever)

Once committed, secrets can be copied, cached, and forked. Deleting a file doesn’t undo exposure.

  • Fix: pre-commit scanning + CI scanning, block merges on findings.
  • Fix: if exposed, rotate first; then clean history if needed.

Mistake 2 — Baking secrets into images/build artifacts

Docker build args and copied files often end up in image layers and caches.

  • Fix: fetch secrets at runtime (env vars or mounted files).
  • Fix: audit build steps; ensure no .env or token files are copied into images.

Mistake 3 — Using long-lived CI credentials

A leaked CI secret can be catastrophic because CI often has wide access.

  • Fix: use OIDC federation (short-lived credentials).
  • Fix: scope permissions per repo + environment.

Mistake 4 — Logging headers, bodies, or env vars

Debug logs are a top source of accidental token exposure.

  • Fix: redact by default; allow-list safe fields.
  • Fix: disable verbose request logging in production; keep a safe “debug mode” with redaction.

Mistake 5 — One secret to rule them all

Shared keys across services/environments create a huge blast radius.

  • Fix: separate dev/staging/prod; use per-service credentials.
  • Fix: tighten IAM and add periodic access reviews.

Mistake 6 — No rotation plan

If rotation is hard, people avoid it—until an incident forces a chaotic scramble.

  • Fix: document a rotation playbook; practice on a low-risk secret first.
  • Fix: prefer dual-secret patterns and graceful reloads in apps.
The “safe by default” workflow

If doing the right thing takes extra steps, people will sometimes skip it. Bake guardrails into the workflow: scanners, templates, runtime injection, and redaction.

FAQ

Are environment variables a secure way to store secrets?

They can be okay, but they’re not automatically “secure”. Env vars are a delivery mechanism, not a vault. They’re useful for runtime injection, but you must still control access (who can read process/env), prevent logging/printing, and avoid exposing them via crash dumps, debug endpoints, or CI output.

Should I keep a .env file in my repo?

No, not for real secrets. Commit a .env.example with placeholder names and documentation, but keep real values out of Git. Use a secret store or local dev tooling that injects values without committing them.

What’s the fastest way to respond if an API key leaked to Git or logs?

Rotate/revoke first. Then contain spread: identify where it went (forks, mirrors, CI logs, dashboards), tighten permissions, and add guardrails (scanners + redaction) so it doesn’t happen again. Cleaning history is useful, but it does not replace rotation.

Do I need HashiCorp Vault, or is a cloud secrets manager enough?

Most teams are fine with a cloud secrets manager. Vault becomes valuable when you need advanced patterns like dynamic credentials, complex multi-cloud setups, or very strict secret lifecycle controls. The best choice is the one your team can operate reliably and audit properly.

How often should secrets be rotated?

Rotate on exposure immediately, and on a schedule based on risk. High-risk credentials (prod DB admin, signing keys) should have tighter rotation; lower-risk secrets can be quarterly or aligned to your compliance requirements. The real win is making rotation low-friction so you actually do it.

What about “API keys” in frontend apps—are those secrets?

If it’s shipped to the browser, it’s public. Some services use “publishable keys” designed to be public; treat those as configuration. Anything that grants privileged access must stay on the server side and be protected like a true secret.

Cheatsheet

Scan-fast checklist for secrets management across dev, CI, and production. Use it as a baseline and iterate.

Dev checklist

  • .env is gitignored; .env.example is committed.
  • Pre-commit scanner installed and enforced.
  • No secrets in README, tickets, or PR descriptions.
  • Local secrets injected via a safe method (not pasted into code).

CI checklist

  • Prefer OIDC/short-lived credentials; avoid long-lived access keys.
  • Secrets fetched at runtime from a store.
  • Secrets masked; no verbose logging around secret handling.
  • PR/branch protections prevent bypassing scans.

Production checklist

  • Secrets injected at runtime (env vars/mounted files/agent).
  • Strict RBAC/IAM per service and per environment.
  • Audit logs enabled for secret access.
  • Rotation playbook exists; at least one rotation practiced.

Logging checklist

  • Redaction by default for common sensitive keys/patterns.
  • No raw Authorization headers or cookies in logs.
  • No full request body dumps in production.
  • Error tracking configured to scrub sensitive fields.

Decision matrix: what to do when you’re unsure

Question If “yes” If “no”
Would exposure enable access or actions? Treat as a secret and store it securely It’s likely config; document it
Is it shipped to the browser or client app? Assume public; redesign to keep privileged access server-side Protect it as a server-side secret
Could we rotate it quickly today? Good—document and automate next Fix workflow; rotation should not be a fire drill
Your next best improvement

If you don’t have time for everything, start with: pre-commit scanning + CI scanning + rotation on exposure. Those three eliminate most “oops” leaks.

Wrap-up

Secrets management is a workflow problem. Once you have guardrails that block Git leaks, a secret store as the source of truth, short-lived CI access, and redacted logging, you’ll stop playing whack-a-mole and start rotating safely and routinely.

Next actions (pick one)

  • Today: add a pre-commit secrets scanner and a CI scanner gate.
  • This week: migrate one production secret into a secret store and update the app to fetch at runtime.
  • This month: implement OIDC-based CI credentials and document a rotation playbook.
Keep learning

The related posts below go deeper on threat modeling, dependency risk, and CI/CD secrets workflows. Combine them with this post and you’ll have a practical security baseline most teams never reach.

Scroll to Related posts for the next reads, or jump back to the Cheatsheet and implement the top items.

Quiz

Quick self-check (demo). This quiz is auto-generated for cyber / security / secrets.

1) What’s the safest default rule for secrets management?
2) Which control best prevents accidental secrets in Git history?
3) A secret appears in CI logs. What’s the correct first response?
4) Which CI approach reduces risk compared to long-lived access keys?