Securing LLM API Keys in Production

Let me walk through how LLM API key compromises actually happen, because it's almost never a sophisticated attack. Developer hardcodes a key during local testing. Code gets pushed to a public repo. Automated scanners pick up the key within minutes. Attacker starts running inference at max throughput. By the time the developer notices the charge alert, the key has been used to generate hundreds of thousands of tokens.

The second most common vector: keys stored in .env files that get copied to staging or production servers without proper protection. A misconfigured server, an exposed Docker layer, or a log file that accidentally prints environment variables — any of these can leak your key. And unlike a database password, an LLM API key gives the attacker a direct line to a paid service that charges per request with no upper bound by default.

I treat LLM API keys with the same rigor as payment processing credentials. They go in a secrets manager, never in source code or config files. They're rotated every 60 days. Every key has spending limits configured at the provider level. Usage is monitored in real time, and alerts fire if consumption exceeds 200% of the rolling 7-day average. These practices have prevented at least 3 key compromises across my client deployments — caught by anomaly alerts before any significant damage occurred.

LLM API keys belong in a dedicated secrets manager — HashiCorp Vault, AWS Secrets Manager, Azure Key Vault, or Google Cloud Secret Manager. Your agents retrieve keys at runtime through the secrets manager API, never from environment variables, config files, or hardcoded strings.

The retrieval pattern matters. Don't cache keys in memory indefinitely; re-fetch them periodically (every 30-60 minutes) so that rotated keys propagate automatically. If your secrets manager goes down, the agent should fail gracefully with a clear error rather than falling back to a cached key that might be expired or revoked.

For development environments, use separate keys with low spending limits. Developers should never have access to production keys. If your LLM provider supports it, create project-scoped keys that restrict usage to specific models, endpoints, or IP ranges.

Rotate LLM API keys every 60 days, or immediately if compromise is suspected. The rotation process should be automated: generate new key at the provider, update the secrets manager, verify agents can authenticate with the new key, then revoke the old key. Zero-downtime rotation requires a brief overlap where both old and new keys are active.

Maintain a key inventory that records every active key, which agent uses it, when it was last rotated, and when it expires. Monthly audits should verify that no keys have been active longer than the rotation policy allows, no revoked keys are still being used, and no keys exist without an assigned owner.

When an employee leaves or an agent is decommissioned, revoke associated keys within 24 hours. Don't wait for the next scheduled rotation. Keys associated with departed team members are a common source of unauthorized access — not necessarily malicious, but uncontrolled.