Hashicorp Vault Setup
Hashicorp Vault is something of the gold standard for secrets storage. It's got a load of other features to do other security related things, but secrets are what it excels at. It's a great way to keep all those passwords, certificates and whatnot that your servers need to use.
Just to cover this off, Vault isn't a replacement of your password manager that you run on your laptop. In fact, you may want to keep your password manager around to remember the "root" secrets you'll need for Vault ;-) Instead, you can think of Vault as being a password manager for your servers to use.
Vault has about a million different options for installation and running. We're not going to look at them all, we're going to pick on Raft, which is a distributed filesystem that doesn't need any additional infrastructure or cloud specifics to work. It makes a good choice for a fair few use-cases, so hopefully a good starting point.
We're also going to skip over installation as it's well covered already.
Certificates
Before we get too far into this, we'll need some certificates. Certificate management is a big subject, so this is an example way to get started (you should look to formalise and secure this method if you intend to keep it).
Create a certificate authority (CA):
openssl genrsa -out rootCA.key 4096
openssl req -x509 -new -nodes -key rootCA.key -sha256 -days 365 -out rootCA.crt -subj "/C=UK/ST=London/O=Pre-Emptive/CN=vault.pre-emptive.net"
Next up, create certs for each server in your cluster. For example:
openssl genrsa -out uk-lon-vault1.pre-emptive.net.key 2048
openssl req -new -sha256 -key uk-lon- vault1.ad.dlpartners.com.key \
-subj "/C=UK/ST=London/O=Pre-Emptive/CN=vault.pre-emptive.net" \
-out vault1.pre-emptive.net.csr -config <(
printf "\n[req]\ndistinguished_name=req\n[san]\nsubjectAltName=DNS:vault1.pre-emptive.net\n" )
openssl x509 -req -days 365 -CA rootCA.crt -CAkey rootCA.key \
-CAcreateserial -extensions san -extfile <(
printf "\n[req]\ndistinguished_name=req\n[san]\nsubjectAltName=DNS:vault1.pre-emptive.net\n" ) \
-in vault1.pre-emptive.net.csr -out vault1.pre-emptive.net.crt
We should now have two server certs and a root CA certificate. The root key is rather important, but not needed any further, so can be hidden away and protected somewhere (not in Vault though!).
Configuration
One of the nifty things about Raft is that you can use the same configuration on all servers in the cluster. It'll happily work out what it doesn't need and ignores it. However, if you're managing your configs with something like Ansible then you can of course just push out the exact bits that are required.
For the first server, you'll need a vault.hcl
something like this:
storage "raft" {
path = "/opt/vault/raft"
node_id = "vault1.pre-emptive.net"
retry_join {
leader_api_addr = "https://vault2.pre-emptive.net:8200"
leader_ca_cert_file = "/opt/vault/tls/rootCA.crt"
leader_client_cert_file = "/opt/vault/tls/vault2.pre-emptive.net.crt"
leader_client_key_file = "/opt/vault/tls/vault2.pre-emptive.net.key"
}
}
listener "tcp" {
address = "0.0.0.0:8200"
tls_cert_file = "/opt/vault/tls/vault1.pre-emptive.net.crt"
tls_key_file = "/opt/vault/tls/vault1.pre-emptive.net.key"
}
performance_multiplier = 1
cluster_name = "Vault"
disable_mlock = true
cluster_addr = "https://vault1.pre-emptive.net:8201"
api_addr = "https://vault1.pre-emptive.net:8200"
ui = true
For the second server and third server you can more or less change vault1
into
vault2
(etc). Next you can go through the first startup and clustering procedure.
Clustering
To get the cluster up for the first time:
- Start up
vault
on the first server (systemctl start vault
) - In a browser, visit the API address and create a new Raft
- You can select how many pieces of key are required to unseal the vault (if you're not sure, perhaps start with 1 key and 1 piece required)
- Most importantly: make a note of the root token and the unseal key!!
- Unseal the Vault and optionally log in using the root token
Back on the command line, use vault status
to show the state of the Vault.
It will probably start with HA Mode
set to standby
. You may be able to
proceed like that, but for clarity, wait a while for that to turn to
active
. Then start up the second server, and unseal it. It should then
automatically join the cluster, sync the Raft data and be ready to log on to.
You'll also hopefully find that you can log on to the standby node and that it works as if it's the primary. Actually, under the covers there's some redirection going on so that the actions you're taking are actually happening on the primary and then getting replicated to the secondary.
Backups and Recovery
Some sort of backups are very much a requirement. Vault can create encrypted "blobs" which you can store in a cloud bucket, or on a tape or whatever. To recover them, you'll need the root key, and then the unseal key. So make sure you keep them safely somewhere separate to the backups themselves. For simplicity you may also want to keep a backup of the certificates and keys you used on the servers, and of course, should keep the CA certificate and key somewhere safe too.
Backups are an disaster recovery feature, you can't selectively recover secrets from a backup - it's all or nothing. For finer grained recovery, Vault has a built-in version history for all secrets.
Other considerations
Other things you probably should think about:
- Running backups automatically (and using a suitable Vault client and logon to do so - eg. an "approle")
- Using a load balancer or reverse proxy to provide a browser service with a browser-trusted certificate (eg. Letsencrypt)
- Monitoring
- Adding the CA certificate to the server's trust store (to make using the
vault
binary simpler)
You'll also want to store some secrets, provide and restrict access to humans and computers and have users who can log on etc.
If you need help with Vault, or indeed anything else, contact us - we can help you figure out what you need and make it work for you.