For using Raft/Consul Autopilot, see: Consul Autopilot OR Vault Integrated Storage Autopilot
There are many best practices for a production hardened deployment of Vault.
Practice defense in depth and follow the Vault security model
NOTE: There are specific recommendations for running Vault in Kubernetes that are not included on this page.
See 09 Vault Security Model
- General Recommendations
- Operating System
- Vault-Specific Configurations
- Secure Vault with TLS
- Secure Consul
- Enable Auditing
- Say No to Cleartext Credentials
- Upgrade Vault Frequently
- Stop Using Root Tokens, Seriously
- Verify the Integrity of the Vault Binary
- Disable the UI – if Not in Use
- Encrypt the Gossip Protocol (Consul)
- Secure the Unseal/Recovery Keys
- Minimize the TTLs for Leases and Tokens
- Follow the Principle of Least Privilege
- Perform Regular Backups
- Integrate with Existing Identity Providers
- The fewer shared resources, the better
- Think "single tenancy" where possible
- Secure deployments: Hardware > VMs > Containers
- Ultimately comes down to protecting memory contents
- Many customers will still use virtualization (VMware/Cloud) or containerization (Docker/K8s) but will deploy to dedicated clusters
- Reduce or eliminate access to Vault nodes
- Includes SSH/RDP and through platform-based access (i.e., AWS SSM, kubectl, etc)
- Instead, access Vault via API or CLI from your workstation or jump box
- If you REALLY need access, use HashiCorp Boundary to limit/control access
- Vault nodes should be dedicated to Vault services
- You should not have other services contending for resources
- More services = more firewall requirements
- Don't forget: Encryption keys are stored in memory
- Exception to this rule may include:
- Telemetry agent
- Log file agent (Splunk, SumoLogic, DataDog)
- Vault and Consul use dedicated ports for communication
- Permit only the required ports to reduce attack surface
- Many enterprise Vault deployments don’t even allow SSH or UI ports
- Default ports include:
- Vault: 8200, 8201
- Consul: 8500, 8201, 8301
- More ports will be needed for our implementation
- Immutable upgrades guarantee a known state because you know the result of your automation configurations
- Easy to bring new nodes online, destroy the old nodes
- Consul and Raft can use AutoPilot to assist with upgrades
- Care must be taken when using Raft because you need to ensure replication has been completed to newly added nodes
- Never run Vault as Root
- Running as root can expose Vault’s sensitive data
- Limit access to configuration files and folders to the Vault user
- Normally, I create a user named “vault”
- Don’t forget that the Vault user will need access to write local files, such as the database, audit logs, and snapshots
chmod –R vault:vault /opt/vault/data
- Protect and audit critical Vault directories and files, including directory for snapshots
- Ensure unauthorized changes can’t be made
- Includes binaries, config files, plugins files and directory, service configurations, audit device files and directory, etc.
# Set permissions on Vault folder
chmod 740 –R /etc/vault.d- Vault writes all configuration and data to the storage backend
- No storage backend = No Vault!
- Consul Storage Backend:
- Use Consul ACLs when running on Consul
- Limit access to any Consul node
- Enable verify_server_hostname in config file
- Disabling history prevents retrieval of commands
- Possible to discover credentials/tokens in history
- Can also disable just 'vault' command in history
history
1365 vault login hvs.RTMd9YZ5Np9WGjvTfARaqffQ
1366 vault policy list
1367 vault list sys/policies/acl
1368 vault secrets list
1369 vault list pki/roles
# Disable command history system wide
echo 'set +o history' >> /etc/profile- Don’t disable to make install/management easier
- Provides additional layers of protection for the OS
- Adhere to CIS or DISA to improve posture of the host OS
Hardening HashiCorp Vault with SELinux
- Core dumps could reveal encryption keys
- Different process to disable depending on the OS
- Make sure you know if this file is modified or replaced
- An attacker could point to compromised binaries to leak data
- This assumes you are running systemd, but any "service" file should be monitored and secured
- Make sure to patch the OS frequently
- Follow your standards or be more stringent for Vault
- Options include Satellite, SpaceWalk, or other solution
- If you use an immutable architecture, then replace the nodes often with known good "patched" images. Use Packer to help simplify this workflow.
- Vault stores sensitive data in-memory, unencrypted
- That data should never be written to disk
- Disabling swap provides an extra layer of protection
- Different process for different operating systems but again, you don't have to do this on the exam
- Example is enabling
mlockto prevent memory swap
- Vault contains sensitive data
- Communications should never occur without TLS in place
- Load Balancers used with Vault can terminate TLS or instead use pass through to the Vault nodes.
- Verify tls_disable configuration does not equal true or 1
- Default is false (not disabled)
- Consul contains your sensitive data
- Communications should never occur without TLS in place
- Issue a certificate from a trusted CA
- Enable Consul ACLs
- Configure gossip encryption (use
consul keygen) - In short, follow the Consul Security Model
- Use multiple Audit Devices to log all interactions
- Send that data to a collection server
- Archive log data based upon security policies
- Create alerts based on certain actions
- Don’t put credentials in configuration files
- Use Environment Variables, where supported
- Use cloud-integrated services, such as AWS IAM Roles or Azure Managed Service Identities
- New updates frequently include security fixes
- New cipher suites can be enabled or supported
- New functionality enabled
- Root tokens have unrestricted access to Vault with no TTL
- Not bound by ACL policies, ERPs, or RGPs
- Get rid of the initial root token after initial setup
vault token revoke hvs.xxxxxxxxx
- Always get Vault binaries directly from HashiCorp
- Use the HashiCorp checksum to validate
- Modified version of Vault binary could leak data
- Vault UI is disabled by default
- Configured in the Vault configuration file
- Do the same for Consul UI, as well
- TLS only secures the interfaces, not Consul gossip traffic
- Use the –encrypt flag in the Consul configuration file
- Uses a 32-byte key – can use
consul keygento generate
- Initialize Vault using PGP keys, such as keybase.io
- Distribute to multiple team members, no single person should have all the keys
- Don’t store the keys in Vault itself
- Use the smallest TTL possible for tokens
- Define Max TTLs to prevent renewals beyond reasonable time frame
- Minimizing TTL also helps reduce burden on the storage backend
- Only give tokens access to paths required for business function
- Separate policies for applications and users`
- Limit use of * and + in policies, where possible
- Templated policies can help with policy creation and maintenance
- Backup configuration files and directories
- Automate Vault backup using snapshots or equivalent depending on the storage backend
- Regularly test backups to ensure functionality
vault operator raft snapshot save monday.snapvault write sys/storage/raft/snapshot-auto/config/daily(Enterprise)
- Use your existing IdP to provide access to users
- If/when users leave, they immediately lose access to Vault
- The fewer places a user has credentials, the better
- Using locally defined credentials is an administrative burden