Secrets Management: Why Credentials Do Not Belong in Your Code

Fractional CTO · Founder of Injuria.ai
A single line of code in a public GitHub repository once cost a company thousands of dollars in a few hours. The developer had accidentally committed an AWS access key. Automated bots constantly scanning for such credentials found it, spun up a fleet of powerful virtual machines for crypto-mining, and presented the company with a shocking bill. This story, in various forms, has played out countless times. It highlights a fundamental principle of modern software engineering: credentials, and other secrets, do not belong in your code.
Storing secrets improperly is a top-tier vulnerability. The OWASP Top 10 2021 list places 'Identification and Authentication Failures' (which includes hardcoded credentials) as a major category of risk. Secrets management is the formal practice and tooling used to manage, secure, and control access to these sensitive pieces of information. It is not about obscurity or simple configuration files. It is a deliberate architectural choice to treat secrets as first-class citizens that are dynamically and securely delivered to applications at runtime, based on a verifiable identity.
The Anatomy of a Secret
In this context, a secret is any piece of information that grants access or unlocks functionality that should be protected. It is distinct from general configuration, which might define an application's behavior but is not sensitive. A database hostname is configuration; its password is a secret.[16] A feature flag is configuration; a private TLS key is a secret.
Common types of secrets include:
- Passwords and Credentials: Usernames and passwords for databases, message brokers, or other stateful services.
- API Keys and Auth Tokens: Tokens for authenticating with third-party services like Stripe, Twilio, or cloud provider APIs.
- Private Keys and Certificates: TLS certificates for servers, client certificates for mTLS, or private keys used for signing data (like JWTs).
- System-Level Encryption Keys: Keys used for encrypting data at rest in storage systems or application-level encryption.
Treating these items as mere configuration is the root of the problem. They require a separate, more secure lifecycle of creation, distribution, rotation, and revocation.
The Fundamental Flaw of Hardcoding Secrets
Committing secrets directly into source code, even in a private repository, introduces severe and unnecessary risks. The practice violates core security principles and creates a brittle, dangerous system.
First, source code is designed for sharing, not for secrecy. Once a secret is in your Git history, it is effectively permanent. Even if you remove it in a later commit, the original commit remains in the history. An attacker who gains read access to the repository, either through a platform breach or a compromised developer account, can scan the entire history for credentials. Public repositories are an obvious disaster, but private repositories are frequently leaked, made public by mistake, or accessed by former employees.
Second, it creates an unacceptably large blast radius. A single leaked credential from one part of the codebase can grant an attacker access to a critical system, like a production database. From there, they can pivot to other parts of the infrastructure. The secret is no longer protected by application-level access controls; it is exposed to anyone who can read a text file.
Third, hardcoding makes secret rotation a nightmare. To change a compromised or expired password, you must change the code, run it through CI/CD, and redeploy every application that uses it. This friction discourages frequent rotation, which is a key practice for limiting the window of opportunity for an attacker who obtains a credential. Ideally, secrets should be rotated automatically and frequently, a task that is impossible when they are static values in source files.[17]
Finally, it violates the principle of least privilege[1]. Every developer, CI server, and tool with access to the codebase also gets access to every secret within it. An intern working on a UI component has no need to see the production database password, yet hardcoding grants them that access implicitly.
How Modern Secrets Management Works: A Step-by-Step Mechanism
A modern secrets management system works by externalizing secrets from the application.[18] The application code contains no secrets. Instead, at runtime, the application proves its identity to a trusted secrets manager and receives its required secrets dynamically.
The core components are the application (workload), a secrets manager, and an identity provider. The central challenge is the "Secret Zero" or bootstrap problem: how does the application prove its identity to the secrets manager for the very first time?
The solution is to rely on a trusted identity provided by the execution environment itself. This is the crucial step that replaces a static, long-lived secret in code with a secure, often short-lived, and automatically-managed identity token.
Here is the step-by-step workflow:
- Bootstrap Identity: The application starts in a secure, managed environment. This environment injects a verifiable identity token into the workload. This token is the "Secret Zero".
- On AWS, an EC2 instance or ECS task can be assigned an IAM Role[2]. The AWS SDK, running inside the application, can then request temporary, short-lived credentials from the local EC2 metadata service. These credentials act as the identity token.
- In Kubernetes, a Pod is assigned a ServiceAccount. The Kubernetes API server can issue a signed JSON Web Token (JWT) for this ServiceAccount, which the application can read from a well-known file path inside the container.
-
Authentication: The application presents its bootstrap identity token to the secrets manager. For example, it might make an API call to AWS Secrets Manager and sign the request with the temporary credentials from its IAM Role, or it might send its Kubernetes ServiceAccount JWT to HashiCorp Vault's Kubernetes auth method.
-
Authorization: The secrets manager receives the authentication request and verifies the token's validity and origin. It then consults its internal policies to authorize the request. These policies are critical. They define which identity is allowed to access which secret. For example, a policy might state: "The identity corresponding to the Kubernetes ServiceAccount
prod/my-api-serviceis permitted to read the secret at pathproduction/database/credentials." -
Secret Delivery: If authentication and authorization succeed, the secrets manager delivers the secret to the application over a secure (TLS-encrypted) connection. The application can then load this secret into memory and use it to connect to the database or API it needs. The secret is never written to disk by the application and exists only for the lifetime of the process.
This flow elegantly solves the hardcoding problem. The application source code is credential-free.[18] Access is governed by explicit policies. Secrets can be rotated in the central manager at any time without requiring an application code change or redeployment.[17] Auditing is also centralized; the secrets manager logs every request, providing a clear record of which workload accessed which secret and when.
Some advanced systems, like HashiCorp Vault's database secrets engine[3], take this a step further with dynamic secrets. When an application requests database credentials, Vault connects to the database itself, generates a brand new username and password with a short time-to-live (TTL), and returns that to the application. When the TTL expires, Vault automatically revokes the credentials. This means even if a credential were to leak, it would be useless within minutes or hours.
A Concrete Example: Accessing a Database from a Kubernetes Pod
Let's illustrate this with a common scenario: a Python application running in a Kubernetes cluster needs to connect to a PostgreSQL database. The credentials are stored in AWS Secrets Manager.[16]
The Wrong Way: Hardcoded in Code
# app.py - DO NOT DO THIS
import os
import psycopg2
# Secret is hardcoded directly in the source code.
DATABASE_URL = "postgres://dbuser:V3ryS3cr3tP@ssw0rd@prod-db.example.com:5432/mydatabase"
def get_data():
conn = psycopg2.connect(DATABASE_URL)
# ... use the connection
conn.close()
This code is a ticking time bomb. The credential V3ryS3cr3tP@ssw0rd is now part of the source code image and Git history.
The Right Way: Workload Identity and External Secrets
This approach uses the Secrets Store CSI Driver[4], a standard way to mount secrets from external managers into Kubernetes pods.
Step 1: Configure Kubernetes Resources (YAML)
First, we define a SecretProviderClass, which tells the CSI driver how to fetch the secret. We also modify our application's Deployment to use a specific serviceAccountName and mount the secret as a volume.
# secret-provider.yaml
apiVersion: secrets-store.csi.x-k8s.io/v1
kind: SecretProviderClass
metadata:
name: my-app-db-creds
spec:
provider: aws
parameters:
objects: |
- objectName: "production/MyDatabase/Password"
objectType: "secretsmanager"
jmesPath:
- path: "SecretString"
objectAlias: "db_password"
---
# deployment.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
name: my-python-app
spec:
replicas: 3
template:
metadata:
labels:
app: my-python-app
spec:
serviceAccountName: my-app-sa # Important: Assigns identity
containers:
- name: my-app
image: my-company/my-python-app:1.2.3
volumeMounts:
- name: secrets-volume
mountPath: "/mnt/secrets"
readOnly: true
volumes:
- name: secrets-volume
csi:
driver: secrets-store.csi.k8s.io
readOnly: true
volumeAttributes:
secretProviderClass: "my-app-db-creds"
In AWS, you would also create an IAM Role that trusts the Kubernetes Service Account (my-app-sa) and has a policy granting it permission to read production/MyDatabase/Password from Secrets Manager.
Step 2: Modify Application Code (Python)
The application code is now much simpler and more secure. It reads the password from a file at the well-known mount path.
# app.py - THE RIGHT WAY
import os
import psycopg2
# Read configuration from environment variables (non-sensitive)
DB_USER = os.environ.get("DB_USER")
DB_HOST = os.environ.get("DB_HOST")
DB_NAME = os.environ.get("DB_NAME")
# Read the secret from the file mounted by the CSI driver
with open("/mnt/secrets/db_password", "r") as f:
db_password = f.read().strip()
DATABASE_URL = f"postgres://{DB_USER}:{db_password}@{DB_HOST}:5432/{DB_NAME}"
def get_data():
conn = psycopg2.connect(DATABASE_URL)
# ... use the connection
conn.close()
The secret db_password is never in the code. The CSI driver, using the pod's identity, authenticates to AWS, fetches the secret, and writes it to a file inside the pod. The application simply reads it. The credential's lifecycle is now completely decoupled from the application's deployment lifecycle.
The Historical Arc of Secrets Management
Understanding how we arrived at today's solutions provides valuable context.
In the early days of web development, hardcoding was common. As systems grew, the first step towards improvement was using environment variables. This practice was codified in the influential Twelve-Factor App methodology[5], which states configuration (including credentials) should be stored in the environment. This was a significant improvement, as it separated config from code. However, environment variables are often visible to any process or user on the same machine (via ps or /proc), are difficult to audit, and still don't solve the rotation problem gracefully.
As distributed systems became more complex, tools emerged to manage configuration and secrets across fleets of servers. But the true shift began with the introduction of dedicated, API-driven secrets management tools. The release of HashiCorp Vault[6] in 2015[7] was a watershed moment. It was built from the ground up to be a universal, secure, and highly-available system for secrets. It introduced concepts like dynamic secrets, encryption-as-a-service, and a pluggable authentication system to a wide audience.
Following this, major cloud providers launched their own integrated, managed services. Azure Key Vault[8] was introduced in 2015, followed by AWS Secrets Manager[9] in 2018 and Google Cloud Secret Manager[10] in 2019. These services offer deep integration with their respective cloud's identity and access management (IAM) systems, dramatically lowering the barrier to entry for adopting secure workload identity patterns.
Correcting Common Misconceptions
Several persistent myths surround secrets management, often leading teams to adopt insecure half-measures.
Misconception: "My repository is private, so it's safe to commit secrets."
This provides a false sense of security. Private repositories are a frequent target. They can be exposed through misconfigured permissions, leaked by insiders, or accessed via a compromised developer machine or CI/CD system. Security should be layered, not dependent on a single, fragile perimeter.
Misconception: "Environment variables are a secure and sufficient solution."
They are a step up from hardcoding but are not a complete solution. They are often logged, can be inspected by other processes on the same host, and lack the core benefits of a true secrets manager: centralized auditing, fine-grained access control, and easy rotation. They are best used for non-sensitive configuration.
Misconception: "Encrypting secrets and committing them to Git is good enough."
This refers to tools like git-crypt[11] or SOPS[12], now maintained by the getsops community. These tools encrypt secret files before they are committed, and decrypt them during deployment. While this prevents plaintext secrets from living in Git history, it only shifts the problem. You now have a new secret to protect: the decryption key. This is another "Secret Zero" problem, and managing access to that key can become just as complex as managing the original secrets. These tools also lack the dynamic, runtime features of a secrets manager, such as auditing, leasing, and on-the-fly rotation.
The Decision Lens: Tradeoffs and Costs
Adopting a proper secrets management solution is not free. Technical leaders must weigh the costs against the significant security benefits.
Operational Overhead: A secrets manager is a critical piece of infrastructure. If it goes down, your applications may not be able to start or function. If self-hosting a tool like Vault, you are responsible for its high availability, backups, and upgrades. Using a managed cloud service offloads this responsibility but requires expertise in configuring IAM policies and network controls correctly.
Added Latency: Applications must now make a network call at startup to fetch their secrets. While typically fast, this adds a small delay and a new potential failure point. Caching strategies can mitigate this, but they introduce their own complexity around cache invalidation when a secret is rotated.
Vendor Lock-in: Using a cloud provider's native secrets manager (e.g., AWS Secrets Manager) is convenient but tightly couples your application's security model to that provider. Migrating to another cloud or a hybrid environment becomes more difficult. Tools like Vault or standards-based approaches aim to mitigate this.
Complexity: A secrets management system is another moving part. It requires developers to understand the new workflow and requires operators to manage the policies and infrastructure. The initial setup can be a hurdle for small teams.
Despite these costs, the decision to adopt secrets management is straightforward for most professional software. You should adopt it if your system handles any user data, connects to paid third-party APIs, or requires compliance with security standards like SOC 2 or PCI DSS. The cost of a single credential leak, including financial terms, reputational damage, and engineering time spent in remediation, almost always outweighs the upfront cost of implementation.
The State of Secrets Management Today
The field of secrets management is mature, with clear, established patterns and powerful tools.
For most organizations operating within a single major cloud, the provider's managed service is the default and often best choice. The tight integration with the platform's native IAM system (AWS IAM, Azure AD, Google IAM) makes the workload identity pattern easy to implement and manage.
HashiCorp Vault remains the industry standard for organizations with more complex needs, such as multi-cloud, hybrid-cloud, or on-premises data centers. Its ability to act as a single, consistent control plane for secrets across disparate environments is its key strength, though it comes with higher operational cost.
In the Kubernetes ecosystem, the Secrets Store CSI Driver has emerged as the de-facto standard for integrating with external secrets managers, providing a consistent, vendor-agnostic interface for applications.
The frontier of this space is moving beyond just managing secrets and towards a future based on universal, verifiable workload identity. Projects like SPIFFE[13] (Secure Production Identity Framework for Everyone) and its implementation SPIRE[14] are pioneering a path where workloads can obtain short-lived, cryptographically verifiable identity documents (SVIDs) regardless of where they are running. This allows for passwordless, mTLS-based authentication between services, further reducing the reliance on traditional, long-lived secrets. It builds on established service-to-service authentication patterns like the OAuth 2.0 Client Credentials Grant[15] flow, but standardizes the identity bootstrapping process across all platforms. This identity-centric approach is the next logical step in building truly secure distributed systems.
Sources (18)
- Principle of least privilege - Wikipedia
- IAM roles for Amazon EC2 - AWS Docs
- Database Secrets Engine - HashiCorp Vault
- Secrets Store CSI Driver
- The Twelve-Factor App: Config
- HashiCorp Vault
- Vault: A Tool for Managing Secrets (Announcement)
- Azure Key Vault
- AWS Secrets Manager
- Google Cloud Secret Manager
- git-crypt - GitHub
- sops - GitHub
- SPIFFE - Secure Production Identity Framework for Everyone
- SPIRE Concepts - SPIFFE
- RFC 6749: The OAuth 2.0 Authorization Framework - Client Credentials Grant
- Jorge T. Negron, Christoffer Jones, George Sawyer, AWS Certified SysOps Administrator Study Guide (John Wiley & Sons, 2024)
- Mauricio Mu¿oz, Dario Lucas Goldfarb, Alexandre M. S. P. Moraes, AWS Certified Security Study Guide (John Wiley & Sons, 2025)
- Tanya Janca, Alice and Bob Learn Application Security (John Wiley & Sons, 2020)