application_security/secrets

92 notes in this subject, read out of this brain and free to use. This is page 1 of 2.

Cloud API Limits for Secrets Management

Cloud services provide limited API calls over a given period. Running into these limits can cause (D)DoS. Limits typically apply per account, project, or subscription—spread workloads to limit blast radius. Some services support data key caching preventing load on key management API (AWS data key caching example). Some services have built-in data key caching like S3 with bucket keys.

Fine-grained access control for multi-cloud secrets

Use fine-grained access control mechanisms to restrict access to secrets based on the principle of least privilege. Ensure that access control policies are consistently enforced across all cloud providers.

Secrets Management - General Introduction

Secrets are widely used in modern DevOps environments, including API keys, database credentials, IAM permissions, SSH keys, and certificates. Organizations often hardcode secrets in source code and configuration files, creating significant security risks. A centralized secrets management system should control storage, provisioning, auditing, rotation, and management of secrets to prevent leaks and compromise. Shared secrets across services make it difficult to identify the source of compromise or leak.

High Availability for Secrets Management

A secrets management solution must be robust enough to reliably service traffic. User secrets like SSH keys and root passwords require rapid provisioning during incident response to minimize downtime impact. Application secrets like database credentials must be performant to avoid degrading availability or increasing startup times. The service must handle considerable request volume in large organizations.

Centralize and Standardize Secrets Management

Secrets management solutions should be centralized and standardized across the organization, though this may involve multiple solutions for different teams (cloud-native teams using cloud provider solutions, private cloud using third-party solutions, etc.). Teams must standardize their interaction with different solutions to maintain usability during incidents. Even when centralizing to one solution, the primary secret of that system should be stored in a secondary secrets management solution. Standardization must include secrets lifecycle management, authentication, authorization, accounting (AAA), and clear documentation of what each secret is used for and where to find it.

Least Privilege Access Control for Secrets

Engineers should not have access to all secrets in the secrets management system. The Least Privilege principle must be applied, with fine-grained access controls configured on each object and component. When users can read or update secrets, that secret can leak through that user and their systems, so access must be strictly limited.

Automate Secrets Management - Secrets Pipeline

Manual secrets maintenance increases both leakage risk and human error. Automation should include a secrets pipeline that handles creation, rotation, and other management tasks automatically. This reduces human interaction with actual secrets and minimizes the surface area of credential exposure.

Automate Secrets Management - Dynamic Secrets

When applications start, they should request dynamically generated credentials from a secrets management system. Dynamic secrets are provided with new credentials for each session and expire upon application reboot. This reduces the surface area of credential reuse. If stolen, dynamic credentials will be expired upon service restart, limiting damage from compromise.

Automate Secrets Management - Automated Key Rotation

Key rotation should be automated rather than manual because manual rotation is error-prone and challenging. Rotating certain keys like encryption keys may trigger full or partial data re-encryption. Different rotation strategies exist: gradual rotation, introducing new keys for write operations while keeping old keys for read operations, rapid rotation, and scheduled rotation.

Kubernetes Sidecar Container Pattern for Secrets Rotation

In Kubernetes, a common architectural pattern uses a sidecar container to retrieve secrets from a secrets manager and make them available to the main application container. The sidecar (e.g., HashiCorp Vault Agent, CyberArk Conjur Secrets Provider) authenticates with the secrets manager using a Kubernetes Service Account, retrieves the secret, and writes it to a shared in-memory volume. The application container reads from this shared volume. The sidecar can periodically refresh the secret to ensure the application always has a valid, short-lived credential. This decouples the application from the specifics of the secrets management solution.

Kubernetes Sidecar Secrets Injection - Example Manifest

Kubernetes Pod manifest with sidecar pattern: ```yaml apiVersion: v1 kind: Pod metadata: name: my-app spec: serviceAccountName: my-app-sa containers: - name: my-app-container image: my-app-image volumeMounts: - name: secrets-volume mountPath: "/mnt/secrets" readOnly: true - name: vault-agent-sidecar image: vault:latest args: ["agent", "-config=/etc/vault/vault-agent-config.hcl"] volumeMounts: - name: secrets-volume mountPath: "/mnt/secrets" volumes: - name: secrets-volume emptyDir: medium: "Memory" ```

Serverless Function Database Credential Rotation Pattern

Cloud-native secret managers support automated rotation using serverless functions (AWS Lambda, Azure Functions). Architecture: A secret in the cloud secrets manager (e.g., AWS Secrets Manager) triggers a rotation Lambda function on schedule. The Lambda function has permissions to update the database password and the secret value in the secrets manager. The rotation process involves multiple steps (create new secret, set new secret, test new secret, finish rotation) to ensure safe transition.

AWS Lambda Secrets Rotation Function - Conceptual Example

AWS Lambda rotation function example in Python: ```python import boto3 import os def lambda_handler(event, context): secret_name = event['SecretId'] token = event['ClientRequestToken'] step = event['Step'] secrets_manager = boto3.client('secretsmanager') metadata = secrets_manager.describe_secret(SecretId=secret_name) if step == "createSecret": new_password = generate_new_password() secrets_manager.put_secret_value( SecretId=secret_name, ClientRequestToken=token, SecretString=f'{{"password":"{new_password}"}}', VersionStages=['AWSPENDING'] ) elif step == "setSecret": update_database_password(new_password) elif step == "testSecret": test_database_connection(new_password) elif step == "finishSecret": secrets_manager.update_version_stage( SecretId=secret_name, VersionStage="AWSCURRENT", MoveToVersionId=token ) ```

Handling Secrets in Memory - Data Structures

In .NET and Java, do not use immutable structures like Strings to store secrets because they cannot be forced to be garbage collected. Instead, use primitive types such as byte arrays or char arrays, where memory can be directly overwritten. This prevents secrets from lingering in memory after use.

Handling Secrets in Memory - Zeroing Memory

After a secret has been used, the memory it occupied should be zeroed out to prevent it from lingering in memory where it could be accessed. This is particularly important in languages where direct memory manipulation is possible, such as C/C++.

Handling Secrets in Memory - Encryption

In some cases, hardware or operating system features can encrypt the entire memory space of the process handling the secret. This provides an additional layer of security. The goal is to minimize the time window where the secret is in plaintext in memory.

Secrets Management - Threat Modeling for Memory Security

Before implementing memory security measures for secrets, develop a threat model to surface implicit assumptions about the application's deployment environment and the capabilities of potential adversaries. Often protecting secrets in memory is considered overkill because threat actors either lack the capabilities to execute such attacks or the cost of defense exceeds the likely impact. If an attacker already has access to the memory of the process handling the secret, a security breach may have already occurred. Attacks like Rowhammer, Meltdown, and Spectre demonstrate that the operating system alone is insufficient to protect process memory. The only foolproof approach is full physical isolation of process memory from untrusted processes. Despite implementation difficulties, in highly sensitive environments protecting secrets in memory can be a valuable additional security layer.

Auditing Secrets Management - Minimum Requirements

Auditing must be implemented securely to be resilient against tampering or deletion. Minimum auditing requirements: Who requested a secret and for what system and role; Whether the secret request was approved or rejected; When the secret was used and by whom/what; When the secret expired; Whether there were attempts to reuse expired secrets; Any authentication or authorization errors; When the secret was updated and by whom/what; Any administrative actions and user activity on the underlying infrastructure stack.

Auditing Secrets Management - Timestamp Requirements

All auditing must have correct timestamps. The secrets management solution should have proper time synchronization protocols set up in its supporting infrastructure. The stack should be monitored for possible clock-skew and manual time adjustments.

Secret Lifecycle Stages

Secrets follow a lifecycle with four main stages: Creation, Rotation, Revocation, and Expiration.

Container Secrets Injection - Build Time Not Recommended

Secrets can be injected into containers at build time (not recommended) or during orchestration/deployment. Build-time injection leaks the secret with the container definition, making it accessible to anyone with container image access.

Secret Creation - Secure Generation and Transmission

New secrets must be securely generated and cryptographically robust enough for their purpose. Secrets must have minimum privileges assigned to enable their required use and role. Credentials should be transmitted securely, ideally not sending the password with the username in the same request. Instead, send the password via a secure channel (mutually authenticated connection) or a side-channel such as push notification, SMS, or email. Applications may require separate provisioning processes for credential delivery.

Secret Rotation - Frequency and Strategy

Secrets should be regularly rotated so stolen credentials only work for a short time. Regular rotation reduces the tendency for users to reuse credentials. Depending on the secret's function and what it protects, lifetime could range from minutes (end-to-end encrypted chats with perfect forward secrecy) to years (hardware secrets). User credentials are excluded from regular rotation and should only be rotated if there is suspicion or evidence of compromise, according to NIST recommendations.

Secret Revocation

When secrets are no longer required or potentially compromised, they must be securely revoked to restrict access. With TLS certificates, this involves certificate revocation.

Secret Expiration

Secrets should be created to expire after a defined time where possible. Expiration can be active expiration by the secret consuming system, or an expiration date set at the secrets management system forcing supporting processes to trigger a secret rotation. Policies should ensure credentials are only made available for a limited time appropriate for their type. Applications should verify that the secret is still active before trusting it.

Transport Layer Security for Secrets

Never transmit secrets via plaintext. TLS must be used for all secret transmission. Secrets management solutions can also be used to provision TLS certificates.

Secrets Management - Downtime and Maintenance Planning

Consider that a secrets management service may become unavailable due to scheduled maintenance. It could be impossible to retrieve credentials needed to restore the service if they were not previously acquired. Maintenance windows should be chosen carefully based on earlier metrics and audit logs.

Secrets Management - Backup and Restore Requirements

Backup and restore procedures must be regularly tested and audited for security. Requirements: An automated backup procedure must be in place and executed periodically, with frequency based on the number of secrets and their lifecycle. Restore procedures must be frequently tested to guarantee backups are intact. Backups must be encrypted and stored securely with reduced access rights. The backup location must be monitored for unauthorized access and administrative actions.

Secrets Management - Break-Glass Emergency Procedures

Emergency ('break-glass') processes should be implemented to restore service if the system becomes unavailable for reasons other than regular maintenance. Emergency break-glass credentials should be regularly backed up securely in a secondary secrets management system and tested routinely to verify they work.

Secrets Management Policies - Complexity and Algorithms

Consistently enforce organization-wide policies defining minimum complexity requirements for passwords and approved encryption algorithms. Using a centralized secrets management solution helps implement these policies. An organization-wide secrets management policy helps enforce the best practices defined in secrets management guidance.

Secrets Metadata - Required Information

A secret management solution should provide the capability to store at least the following metadata about a secret: When it was created/consumed/archived/rotated/deleted; Who created/consumed/archived/rotated/deleted it (both the actual producer and the engineer using the production method); What created/consumed/archived/rotated/deleted it; Who to contact when having trouble with the secret or questions about it; What the secret is used for (designated intended consumers and purpose); What type of secret it is (AES Key, HMAC key, RSA private key, etc.); When it needs to be rotated if done manually. Storing metadata and preparing to move secrets reduces the probability of vendor lock-in.

Passwordless Authentication and Token Security - OIDC Benefits

Passwordless authentication mechanisms like OpenID Connect (OIDC) can significantly reduce the attack surface by moving away from user-managed passwords. Applications rely on trusted identity providers (IdPs) to authenticate users and receive secure tokens. Benefits: Eliminates threats like phishing, credential stuffing, and weak password practices; Authentication is handled by a specialized IdP which can enforce strong authentication policies like MFA; OIDC tokens are typically short-lived, limiting the window if a token is compromised.

Token Security Controls - Transmission and Storage

Adopting passwordless authentication shifts security focus from protecting static passwords to protecting dynamic tokens (ID tokens, access tokens, refresh tokens). Tokens are bearer tokens—anyone possessing one can use them. Critical controls: Always transmit tokens over TLS; Do not store tokens in insecure locations like browser local storage; Use secure, HTTP-only cookies or appropriate secure storage mechanisms for mobile applications.

Token Validation and Lifetime Management

For passwordless authentication systems: Always validate the signature, issuer, and audience of a token to ensure it is legitimate; Use short-lived access tokens and implement a secure refresh token rotation strategy.

Standardized security policies for multi-cloud secrets

Define and enforce standardized security policies for managing secrets across all cloud providers. This includes policies for key rotation, access control, and auditing.

CI/CD Pipeline Secrets - Hardening Requirements

CI/CD tooling consumes high-privilege credentials regularly. Guidelines to prevent hacking or misuse: Treat CI/CD tooling as a production environment—harden it, patch it, and harden the underlying infrastructure and services; Have Security Event Monitoring in place; Implement least-privilege access: developers only need to execute required functions like setting up pipelines and running them, not administrative functions; Pipeline output must not leak secrets and production pipelines cannot be listened to with debugging tools; No ability to exec into runners and workers; Proper authentication, authorization and accounting in place; Only approved processes can create pipelines with MR/PR review to ensure security review.

CI/CD Secrets Storage Options - In CI/CD Tooling

Secrets can be stored as part of CI/CD tooling in GitHub, GitLab, Jenkins, and other platforms. This is different from committing secrets to code. When stored in CI/CD tooling, secrets are exposed to CI/CD jobs and often configurable/viewable by authorized people like GitHub maintainers, GitLab project owners, or Jenkins admins.

CI/CD Tooling Secrets - Best Practices

Best practices when storing secrets in CI/CD tooling: No 'big secret'—ensure secrets are not long-term, don't have wide blast radius, don't have high value, and are not shared (never one password for all admin users); Have clear overview of which users can view or alter secrets; Reduce the number of people who can perform administrative tasks to limit exposure; Log & Alert—assemble logs and have rules to detect secret extraction or misuse through web interface or exfiltration attempts; Rotation—regularly rotate secrets; Forking should not leak—validate that repository forks or job definition copies do not copy secrets; Document—which secrets are stored in CI/CD tooling and why, to enable easy migration.

CI/CD Secrets Storage - In Secrets Management System

Secrets can be stored in designated secrets management solutions: Cloud provider solutions (AWS Secrets Manager, Google Secret Manager, Azure Key Vault) or dedicated solutions (HashiCorp Vault, Keeper, Conjur). CI/CD pipeline tooling requires credentials to authenticate against the secret management system. Best practices: Rotation/Temporality—credentials used by CI/CD are rotated frequently and expire after job completion; Scope of authorization—credentials only authorize required secrets and services; Attribution of caller—calls can be attributed to requesting person or service; Follow all practices from section 3.2.1; Backup secrets to separate storage for critical operations.

CI/CD Secrets - Consumer-Retrieved Pattern

Secrets do not necessarily need to be brought to consumers by CI/CD pipeline. Better: the consumer retrieves the secret. CI/CD pipeline instructs the orchestrating system (e.g., Kubernetes) to schedule a service with a service account that can retrieve the required secret. CI/CD tooling has credentials for the orchestrating platform but no longer accesses secrets themselves. Best practices for these credentials are similar to those for secrets management system credentials.

CI/CD Tooling Authentication and Authorization

CI/CD tooling should have designated service accounts operating only in the scope of required secrets or orchestration. A CI/CD pipeline run should be easily attributable to the one who defined or triggered the job to detect secret exfiltration attempts. With certificate-based auth, the caller's pipeline identity should be part of the certificate. With token authentication, the principal requesting actions must be identifiable. Verify periodically that attribution remains in place for effective logging and security alerting.

CI/CD Logging and Accounting Requirements

Attackers use CI/CD tooling to extract secrets via administrative interfaces or job creation with encryption or double Base64 encoding. Every action in CI/CD tools must be logged. Define security alerting rules for non-standard pipeline manipulation and administrative interface access. Logs must be queryable for at least 90 days and stored for extended period in cold storage, as security teams may take time to understand extraction or manipulation methods.

CI/CD Rotation vs Dynamic Secret Creation

CI/CD tooling can rotate secrets or instruct other components to rotate them, requesting rotation from a secrets management system or application. Alternatively, CI/CD can set up dynamic secrets—secrets required for a consumer that are invalidated when the consumer no longer exists. Dynamic secrets reduce leakage risk and enable easy misuse detection. If attacker uses a secret from unexpected IP, it can be easily detected.

Pipeline-Generated Secrets

CI/CD pipeline tooling can generate secrets and offer them directly to deployed services or provide to secrets management solution. Alternatively, encrypt the secret in git so secret and metadata are close to developer's workspace. Git-stored secrets require: developers cannot decrypt secrets themselves; every consumer has encrypted variant; secrets differ per DTAP environment with different encryption key; only designated consumer in that environment can decrypt. Consumers can decrypt using a sidecar. When pipeline creates secrets, scripts must adhere to best practices for generation: secure randomness, proper length, based on well-defined metadata in git.

AWS Secrets Manager - Recommended Solution

For AWS, the recommended solution for secrets management is AWS Secrets Manager. Permissions are granted at the secret level. Refer to AWS Secrets Manager best practices for implementation details.

AWS Systems Manager Parameter Store - Alternative with Limitations

AWS Systems Manager Parameter Store is a cheaper alternative to AWS Secrets Manager but has limitations: Must manually ensure encryption (Secrets Manager does this by default); Offers fewer auto-rotation capabilities (requires custom function); Does not support cross-account access; Does not support cross-region replication; Fewer Security Hub Controls available.

AWS Nitro Enclaves - Isolated Compute for Secrets

AWS Nitro Enclaves create isolated compute environments to protect and securely process highly sensitive data such as secrets. Enclaves are hardened and restrict operator access, providing a trusted execution environment. Key feature is cryptographic attestation, allowing verification of the enclave's identity and ensuring only authorized code is running before provisioning secrets. This is suitable for scenarios requiring high assurance in secret handling.

AWS CloudHSM - Hardware Security Module Control

AWS CloudHSM is used when secrets require more control over encryption and storage in highly confidential applications. It supports bring-your-own-key (BYOK) for AWS services, giving more control over key creation, lifecycle, and durability. CloudHSM allows automatic scaling and backup. Amazon (the cloud provider) does not have access to the key material stored in AWS CloudHSM.

GCP Secret Manager - Recommended Solution

For GCP, the recommended service for secrets management is Secret Manager. Permissions are granted at the secret level. Refer to GCP Secret Manager best practices for implementation details.

GCP Confidential Computing - Data Encryption In-Use

GCP Confidential Computing encrypts data in-use while being processed. Achieved through services like Confidential VMs and Confidential GKE Nodes, leveraging AMD Secure Encrypted Virtualization (SEV). Even Google personnel cannot view the contents of memory of virtual machines, providing high protection for secrets that must be held in memory.

Azure Key Vault - Recommended Solution

For Azure, the recommended service is Key Vault. Permissions are granted at the Key Vault level (not the individual secret level, unlike AWS and GCP). This means secrets for separate workloads and sensitivity levels should be in separate Key Vaults. Refer to Azure Key Vault best practices for implementation details.

Azure Confidential Computing - Trusted Execution Environments

Azure Confidential Computing creates trusted execution environments isolating sensitive data within protected containers. Data is encrypted at rest, in transit, and in use. Services like Azure Confidential Virtual Machines and Confidential Containers on ACI utilize Intel SGX and AMD SEV-SNP to create secure enclaves. This prevents unauthorized access from cloud administrators, malware, or other tenants.

Azure Dedicated HSM - Hardware Security Module

Azure Dedicated HSM is used when secrets require enhanced administrative and cryptographic control in Azure environments. It provides more control over secrets stored on it. Microsoft (the cloud provider) does not have access to the key material stored in Azure Dedicated HSM.

Cloud-Agnostic Secrets Management Solutions

For multi-cloud or cloud-agnostic environments, dedicated secrets management solutions should be used to avoid vendor lock-in: CyberArk Conjur, HashiCorp Vault, Pulumi ESC. These allow using the same solution across all cloud providers and on-premises environments.

Server-Side vs Client-Side Encryption for Secrets

Server-side encryption of secrets means the cloud provider encrypts secrets at rest. The secret is safeguarded against compromise while at rest but is decrypted when shared with intended services or users. Client-side encryption means the secret remains encrypted until actively decrypted by the consumer. Only the consumer has decryption capability. Client-side encryption can provide end-to-end encryption from producer to consumer, requiring proper crypto system implementation.

Bring Your Own Key (BYOK) vs Cloud Provider Key

When encrypting secrets at rest, the encryption key can be managed by the cloud provider or by the customer (BYOK). Less trust in the cloud provider means more desire to manage keys yourself. Cloud providers support BYOK where either directly imported or generated at the key management solution using cloud HSM. The customer's key can encrypt the provider's data key, which encrypts the secret. By managing the Customer Master Key (CMK), you control the data key at the secrets management solution. Importing key material can be done with all providers but is complex and not recommended unless threat model and policy specifically require it.

Envelope Encryption Pattern

Envelope encryption uses a two-layer approach: a data key encrypts the secret, and a master key encrypts the data key. This allows key rotation and management without re-encrypting all data. Customers can import or generate their own master key (BYOK) at the cloud provider's key management service.

IAM for Secrets Management - Access Policies and Roles

To effectively manage secrets, suitable access policies and roles must be set up. This goes beyond secrets policies and includes hardening the full IAM setup to prevent privilege escalation attacks. Never allow open 'pass role' privileges or unrestricted IAM creation privileges, as these can use or create credentials with access to secrets. Tightly control what can impersonate a service account: are machines' roles accessible by attackers? Can service roles from data-pipeline tooling access secrets? Include IAM for every cloud component in threat modeling.

IAM for Secrets - Temporality and Monitoring

Leverage the temporality of IAM principals: ensure only specific roles and service accounts requiring access can access secrets. Monitor these accounts to determine who/what used them and when to access secrets.

IAM for Secrets - Scoped Access

One should not simply be allowed to access all secrets. In GCP and AWS, create fine-grained access policies ensuring a principal cannot access all secrets at once. In Azure, accessing the Key Vault grants access to all secrets in that vault, so separate Key Vaults are essential for segregating access.

Give your agent this brain