Encrypted Secrets Vault¶

Secure Storage for Deployment Credentials using vault-tool

Document Version: 1.0.0
Last Updated: 2026-04-19

1. Overview¶

MazeVault supports an encrypted vault file (secrets.vault) as a secure alternative to storing sensitive credentials in .env files. The vault uses AES-256-GCM encryption with Argon2id key derivation — the same cryptographic standards used for secret values at rest.

Backward Compatible

The vault is fully opt-in. Without vault configuration, MazeVault operates exactly as before using environment variables from .env. You can migrate secrets gradually — the system falls back to environment variables for any secret not found in the vault.

Architecture¶

graph LR
    VK["🔑 vault.key<br/>(passphrase file)"] --> D["🔓 Decrypt"]
    VF["🏦 secrets.vault<br/>(encrypted file)"] --> D
    D --> MG["🛡️ memguard<br/>(locked memory)"]
    MG --> BE["⚙️ Backend"]
    MG --> OCSP["📡 OCSP Responder"]

    classDef key fill:#FFF8E1,stroke:#FF9800,stroke-width:2px,color:#E65100
    classDef vault fill:#E8EAF6,stroke:#3F51B5,stroke-width:2px,color:#283593
    classDef decrypt fill:#E8F5E9,stroke:#4CAF50,stroke-width:2px,color:#2E7D32
    classDef service fill:#EBF5FB,stroke:#2196F3,stroke-width:2px,color:#1565C0

    class VK key
    class VF vault
    class D decrypt
    class MG decrypt
    class BE,OCSP service

What's Protected¶

The vault stores up to 27 sensitive credentials including:

Category	Variables
Critical	`MAZEVAULT_MASTER_KEY`, `MAZEVAULT_JWT_KEY`, `DATABASE_URL`, `REDIS_URL`, `MAZEVAULT_SYNC_SECRET`
SSO / OAuth	`ENTRA_CLIENT_SECRET`, `GITHUB_CLIENT_SECRET`, `GITLAB_CLIENT_SECRET`
Integration tokens	`AGENT_BINARY_GITHUB_TOKEN`, `GATEWAY_BOOTSTRAP_TOKEN`, `DYNATRACE_TOKEN`
Notifications	`SMTP_USERNAME/PASSWORD`, `SMSEAGLE_`, `JIRA_`, `TEAMS_WEBHOOK_URL`, `O365_*`
License	`BUILD_AUTH_SECRET`, `LICENSE_PUBLIC_KEY`
Infrastructure	`SESSION_SECRET`, `POSTGRES_PASSWORD`, `REDIS_PASSWORD`

Security Properties¶

Property	Implementation
Encryption	AES-256-GCM (NIST SP 800-38D)
Key derivation	Argon2id (RFC 9106) — 64 MB memory, 3 iterations, 4 threads
Memory protection	Secrets held in `memguard` locked buffers (mlock, guard pages, core dump exclusion)
File format	Binary — `MVT1` magic header + random salt + random nonce + ciphertext
Passphrase handling	Always from file — never CLI arguments or environment variables

2. Getting Started¶

Prerequisites¶

MazeVault backend is installed and running
Access to the deployment host (SSH or console)
The vault-tool binary (included with the backend container)

Step 1: Generate a Vault Key¶

vault-tool init --key-file /opt/mazevault/vault.key

This creates a cryptographically random 32-byte passphrase file. Secure it:

chmod 400 /opt/mazevault/vault.key
chown root:root /opt/mazevault/vault.key

Critical — Back Up the Key

Without vault.key, the vault cannot be decrypted. Back it up immediately to a secure offline location (see Backup & Restore).

Step 2: Migrate Existing Secrets¶

If you already have a .env file with sensitive values:

vault-tool migrate-env \
  --env-file /opt/mazevault/.env \
  --vault-file /opt/mazevault/secrets.vault \
  --key-file /opt/mazevault/vault.key

This extracts only the 27 known sensitive keys from your .env and encrypts them into the vault. Non-sensitive configuration (ports, feature flags, etc.) remains in .env.

Step 3: Configure MazeVault to Use the Vault¶

Add these variables to your .env or environment:

# Path to the encrypted vault file
MAZEVAULT_VAULT_FILE=/opt/mazevault/secrets.vault

# Path to the vault key (passphrase file)
MAZEVAULT_VAULT_KEY_FILE=/opt/mazevault/vault.key

Step 4: Verify¶

# List secrets stored in the vault
vault-tool list --vault-file /opt/mazevault/secrets.vault --key-file /opt/mazevault/vault.key

# Restart MazeVault and check health
docker compose restart backend ocsp-responder
curl -sk https://localhost/api/v1/health | jq .

Step 5: Remove Secrets from `.env` (Optional)¶

Once verified, you can remove the sensitive values from .env to eliminate plaintext exposure. The vault takes priority; .env serves as fallback.

3. `vault-tool` CLI Reference¶

Commands¶

Command	Description
`init`	Generate a new random vault key file
`encrypt`	Create an encrypted vault from JSON input (stdin)
`decrypt`	Decrypt and display vault contents
`add`	Add or update a single secret
`list`	List secret names (without values)
`rotate-key`	Re-encrypt vault with a new passphrase
`migrate-env`	Extract sensitive keys from `.env` into a vault

`init` — Generate Vault Key¶

vault-tool init --key-file vault.key

Creates a 32-byte random passphrase file. Set permissions to 400 (owner read only).

`migrate-env` — Migrate from `.env`¶

vault-tool migrate-env \
  --env-file .env \
  --vault-file secrets.vault \
  --key-file vault.key

Reads .env, extracts known sensitive keys, and writes secrets.vault. Non-sensitive keys are ignored.

`add` — Add or Update a Secret¶

echo "my-new-token" | vault-tool add \
  --vault-file secrets.vault \
  --key-file vault.key \
  --name DYNATRACE_TOKEN

Reads the value from stdin. If the secret already exists, it is overwritten.

`list` — List Secret Names¶

vault-tool list --vault-file secrets.vault --key-file vault.key

Displays secret names only — values are never shown with this command.

`decrypt` — View Vault Contents¶

# Names only (default)
vault-tool decrypt --vault-file secrets.vault --key-file vault.key

# With values (use with caution)
vault-tool decrypt --vault-file secrets.vault --key-file vault.key --show-values

Operational Security

Use --show-values only in secure terminal sessions. Secret values in terminal history may be a risk.

`rotate-key` — Rotate Vault Passphrase¶

# Generate new key
vault-tool init --key-file vault.key.new

# Re-encrypt vault
vault-tool rotate-key \
  --vault-file secrets.vault \
  --key-file vault.key \
  --new-key-file vault.key.new

# Replace old key
mv vault.key.new vault.key
chmod 400 vault.key

`encrypt` — Create Vault from JSON¶

echo '{"MAZEVAULT_MASTER_KEY": "base64value...", "DATABASE_URL": "postgres://..."}' | \
  vault-tool encrypt --vault-file secrets.vault --key-file vault.key

4. Deployment Modes¶

On-Premise (File-Based)¶

The simplest mode. Both secrets.vault and vault.key reside on the same host:

MAZEVAULT_VAULT_FILE=/opt/mazevault/secrets.vault
MAZEVAULT_VAULT_KEY_FILE=/opt/mazevault/vault.key

File permissions:

File	Permissions	Owner
`vault.key`	`0400`	`root:root`
`secrets.vault`	`0600`	`root:root`

Azure (Key Vault Passphrase)¶

For Azure deployments, the passphrase can be stored in Azure Key Vault instead of a local file. MazeVault uses Managed Identity to retrieve it:

MAZEVAULT_VAULT_FILE=/opt/mazevault/secrets.vault
MAZEVAULT_VAULT_AKV_URL=https://myvault.vault.azure.net
MAZEVAULT_VAULT_AKV_SECRET=mazevault-vault-key   # optional, default name

In this mode, no vault.key file is needed on disk — the passphrase is fetched from Azure Key Vault at startup using the system's Managed Identity.

Legacy (No Vault)¶

If MAZEVAULT_VAULT_FILE is not set, MazeVault operates in legacy mode — all configuration is read from environment variables / .env as before. No changes are required.

5. How It Works¶

Startup Sequence¶

sequenceDiagram
    participant E as .env file
    participant V as secrets.vault
    participant P as VaultProvider
    participant S as Backend Services

    E->>P: godotenv.Load()
    alt MAZEVAULT_VAULT_FILE set
        alt + AKV URL set
            P->>P: Fetch passphrase from Azure Key Vault
        else + KEY_FILE set
            P->>P: Read passphrase from vault.key
        end
        V->>P: Decrypt secrets.vault
        P->>P: Store in memguard locked buffers
    else No vault configured
        P->>P: Use env-only mode (legacy)
    end

    P->>S: provider.Get("SECRET_NAME")
    Note over P,S: Vault value → env fallback → empty string

Secret Resolution Order¶

For every secret lookup:

Vault — If the secret exists in the encrypted vault, use it
Environment — Fall back to os.Getenv() (from .env or system env)
Empty — If neither source has the value, return empty string

This enables gradual migration: move secrets to vault one by one while .env continues to work for anything not yet migrated.

6. Backup & Restore¶

Critical Files¶

File	Backup Priority	Loss Impact
`secrets.vault`	Critical	All encrypted credentials lost
`vault.key`	Critical	Cannot decrypt vault — permanent data loss
`.env`	Important	Non-sensitive config must be recreated

Backup Procedure¶

# Back up vault and key to secure location
cp /opt/mazevault/secrets.vault /secure-backup/secrets.vault.$(date +%Y%m%d)
cp /opt/mazevault/vault.key /secure-backup/vault.key.$(date +%Y%m%d)

# Verify backup integrity
vault-tool list \
  --vault-file /secure-backup/secrets.vault.$(date +%Y%m%d) \
  --key-file /secure-backup/vault.key.$(date +%Y%m%d)

Storage Recommendations¶

Requirement	Recommendation
Physical separation	Store `vault.key` backup in a different physical location than `secrets.vault`
Offline storage	USB drive, hardware token, or sealed envelope in a safe
Access control	Only authorized administrators should have access
Verification	Quarterly: verify backup can decrypt test data
Retention	Permanent — keep all versions (for historical data recovery)

Restore Procedure¶

# Copy files to deployment host
cp /secure-backup/secrets.vault.20260419 /opt/mazevault/secrets.vault
cp /secure-backup/vault.key.20260419 /opt/mazevault/vault.key

# Set correct permissions
chmod 600 /opt/mazevault/secrets.vault
chmod 400 /opt/mazevault/vault.key

# Verify vault is readable
vault-tool list --vault-file /opt/mazevault/secrets.vault --key-file /opt/mazevault/vault.key

# Restart services
docker compose restart backend ocsp-responder
curl -sk https://localhost/api/v1/health | jq .

Disaster Recovery¶

If vault.key is lost and no backup exists, the vault cannot be recovered. You must:

Generate a new vault key: vault-tool init --key-file vault.key
Recreate all secrets manually (from original sources: Azure portal, SMTP provider, etc.)
Create a new vault: vault-tool encrypt --vault-file secrets.vault --key-file vault.key
Back up the new key immediately

7. Key Rotation¶

Rotate the vault passphrase periodically or after suspected compromise:

# 1. Generate new passphrase
vault-tool init --key-file vault.key.new

# 2. Re-encrypt vault with new passphrase
vault-tool rotate-key \
  --vault-file /opt/mazevault/secrets.vault \
  --key-file /opt/mazevault/vault.key \
  --new-key-file vault.key.new

# 3. Replace old key
mv vault.key.new /opt/mazevault/vault.key
chmod 400 /opt/mazevault/vault.key

# 4. Restart services to load with new key
docker compose restart backend ocsp-responder

# 5. Back up new key
cp /opt/mazevault/vault.key /secure-backup/vault.key.$(date +%Y%m%d)

# 6. Verify
vault-tool list --vault-file /opt/mazevault/secrets.vault --key-file /opt/mazevault/vault.key
curl -sk https://localhost/api/v1/health | jq .

Rotation does not change secret values

Key rotation only changes the encryption passphrase. The secrets inside remain the same — no service configuration changes are needed.

8. OCSP Responder¶

The OCSP Responder shares the same secrets.vault file as the backend. It reads three secrets:

Secret	Purpose
`MAZEVAULT_MASTER_KEY`	Decryption of stored CA private keys
`DB_CONNECTION`	PostgreSQL connection string
`REDIS_URL`	Redis cache URL

Configuration is identical — set MAZEVAULT_VAULT_FILE and MAZEVAULT_VAULT_KEY_FILE in the OCSP Responder's environment. If the vault is not configured, the OCSP Responder falls back to environment variables.

9. Troubleshooting¶

Symptom	Cause	Solution
`vault: cannot stat key file`	Key file missing or wrong path	Verify `MAZEVAULT_VAULT_KEY_FILE` points to correct file
`vault: decryption failed`	Wrong passphrase or corrupted file	Use correct `vault.key`; restore from backup
`vault key file has overly permissive permissions`	File permissions too open	`chmod 400 vault.key`
`vault: file too small to be a valid vault`	File is empty or truncated	Restore `secrets.vault` from backup
Backend starts but secrets are empty	Vault configured but key not present in vault	Verify with `vault-tool list`; add missing key with `vault-tool add`
Services work without vault	Expected — legacy fallback	`.env` values are used when vault is not configured or secret is missing

Encryption & Key Management — Cryptographic standards
Key Management — Key lifecycle and HSM integration
Backup & Restore — Database and system backup procedures
Environment Variables — Complete variable reference

Encrypted Secrets Vault¶

1. Overview¶

Architecture¶

What's Protected¶

Security Properties¶

2. Getting Started¶

Prerequisites¶

Step 1: Generate a Vault Key¶

Step 2: Migrate Existing Secrets¶

Step 3: Configure MazeVault to Use the Vault¶

Step 4: Verify¶

Step 5: Remove Secrets from .env (Optional)¶

3. vault-tool CLI Reference¶

Commands¶

init — Generate Vault Key¶

migrate-env — Migrate from .env¶

add — Add or Update a Secret¶

list — List Secret Names¶

decrypt — View Vault Contents¶

rotate-key — Rotate Vault Passphrase¶

encrypt — Create Vault from JSON¶

4. Deployment Modes¶

On-Premise (File-Based)¶

Azure (Key Vault Passphrase)¶

Legacy (No Vault)¶

5. How It Works¶

Startup Sequence¶

Secret Resolution Order¶

6. Backup & Restore¶

Critical Files¶

Backup Procedure¶

Storage Recommendations¶

Restore Procedure¶

Disaster Recovery¶

7. Key Rotation¶

8. OCSP Responder¶

9. Troubleshooting¶

Related¶

Step 5: Remove Secrets from `.env` (Optional)¶

3. `vault-tool` CLI Reference¶

`init` — Generate Vault Key¶

`migrate-env` — Migrate from `.env`¶

`add` — Add or Update a Secret¶

`list` — List Secret Names¶

`decrypt` — View Vault Contents¶

`rotate-key` — Rotate Vault Passphrase¶

`encrypt` — Create Vault from JSON¶