Hashicorp Vault Setup

Hashicorp Vault is something of the gold standard for secrets storage. It's got a load of other features to do other security related things, but secrets are what it excels at. It's a great way to keep all those passwords, certificates and whatnot that your servers need to use.

Just to cover this off, Vault isn't a replacement of your password manager that you run on your laptop. In fact, you may want to keep your password manager around to remember the "root" secrets you'll need for Vault ;-) Instead, you can think of Vault as being a password manager for your servers to use.

Vault has about a million different options for installation and running. We're not going to look at them all, we're going to pick on Raft, which is a distributed filesystem that doesn't need any additional infrastructure or cloud specifics to work. It makes a good choice for a fair few use-cases, so hopefully a good starting point.

We're also going to skip over installation as it's well covered already.

Certificates

Before we get too far into this, we'll need some certificates. Certificate management is a big subject, so this is an example way to get started (you should look to formalise and secure this method if you intend to keep it).

Create a certificate authority (CA):

openssl genrsa -out rootCA.key 4096
openssl req -x509 -new -nodes -key rootCA.key -sha256 -days 365 -out rootCA.crt -subj "/C=UK/ST=London/O=Pre-Emptive/CN=vault.pre-emptive.net"

Next up, create certs for each server in your cluster. For example:

openssl genrsa -out uk-lon-vault1.pre-emptive.net.key 2048
openssl req -new -sha256 -key uk-lon- vault1.ad.dlpartners.com.key \
  -subj "/C=UK/ST=London/O=Pre-Emptive/CN=vault.pre-emptive.net" \
  -out vault1.pre-emptive.net.csr -config <(
    printf "\n[req]\ndistinguished_name=req\n[san]\nsubjectAltName=DNS:vault1.pre-emptive.net\n" )
openssl x509 -req -days 365 -CA rootCA.crt -CAkey rootCA.key \
  -CAcreateserial -extensions san -extfile <(
    printf "\n[req]\ndistinguished_name=req\n[san]\nsubjectAltName=DNS:vault1.pre-emptive.net\n" ) \
  -in vault1.pre-emptive.net.csr -out vault1.pre-emptive.net.crt

We should now have two server certs and a root CA certificate. The root key is rather important, but not needed any further, so can be hidden away and protected somewhere (not in Vault though!).

Configuration

One of the nifty things about Raft is that you can use the same configuration on all servers in the cluster. It'll happily work out what it doesn't need and ignores it. However, if you're managing your configs with something like Ansible then you can of course just push out the exact bits that are required.

For the first server, you'll need a vault.hcl something like this:

storage "raft" {
  path = "/opt/vault/raft"
  node_id = "vault1.pre-emptive.net"
  retry_join {
    leader_api_addr = "https://vault2.pre-emptive.net:8200"
    leader_ca_cert_file = "/opt/vault/tls/rootCA.crt"
    leader_client_cert_file = "/opt/vault/tls/vault2.pre-emptive.net.crt"
    leader_client_key_file = "/opt/vault/tls/vault2.pre-emptive.net.key"
  }
}

listener "tcp" {
  address       = "0.0.0.0:8200"
  tls_cert_file = "/opt/vault/tls/vault1.pre-emptive.net.crt"
  tls_key_file  = "/opt/vault/tls/vault1.pre-emptive.net.key"
}
performance_multiplier = 1
cluster_name = "Vault"
disable_mlock = true
cluster_addr = "https://vault1.pre-emptive.net:8201"
api_addr = "https://vault1.pre-emptive.net:8200"
ui = true

For the second server and third server you can more or less change vault1 into vault2 (etc). Next you can go through the first startup and clustering procedure.

Clustering

To get the cluster up for the first time:

Back on the command line, use vault status to show the state of the Vault. It will probably start with HA Mode set to standby. You may be able to proceed like that, but for clarity, wait a while for that to turn to active. Then start up the second server, and unseal it. It should then automatically join the cluster, sync the Raft data and be ready to log on to.

You'll also hopefully find that you can log on to the standby node and that it works as if it's the primary. Actually, under the covers there's some redirection going on so that the actions you're taking are actually happening on the primary and then getting replicated to the secondary.

Backups and Recovery

Some sort of backups are very much a requirement. Vault can create encrypted "blobs" which you can store in a cloud bucket, or on a tape or whatever. To recover them, you'll need the root key, and then the unseal key. So make sure you keep them safely somewhere separate to the backups themselves. For simplicity you may also want to keep a backup of the certificates and keys you used on the servers, and of course, should keep the CA certificate and key somewhere safe too.

Backups are an disaster recovery feature, you can't selectively recover secrets from a backup - it's all or nothing. For finer grained recovery, Vault has a built-in version history for all secrets.

Other considerations

Other things you probably should think about:

You'll also want to store some secrets, provide and restrict access to humans and computers and have users who can log on etc.

If you need help with Vault, or indeed anything else, contact us - we can help you figure out what you need and make it work for you.