Monitoring Application Version Updates

Abstract image representing an update

Introduction

Most IT estates have a handful of important apps running on servers (or in containers) that are more or less critical to the activities of the company. Some examples are things like Gitlab, Sonatype Nexus, Sonarqube, Hashicorp Vault, or a myriad of others. They in turn depend on other applications such as Nginx/Apache, Postgres/MySQL, Redis and so on. Part of the sysadmin's job is to keep all those things updated to the latest versions, especially so if any of them have Common Vulnerability Exposure (CVE) warnings on them.

This sort of problem is comprehensively solved by a security platform that is constantly checking every version of every installed application in the estate for known CVEs. It can then produce reports and alerts to prompt some action to be taken. As great as these platforms are, they take a fair amount of setup and management and are beyond the reach of smaller organisations. For those people, application version update monitoring may help out.

Application version update monitoring is really about asking an application or the Internet if a newer version is available. If it is, then have an existing monitoring platform raise an alert (either a warning or a critical alert, depending on requirements). Ideally, if it's possible to determine if there's a known CVE on the current version, then include that information in the decision about what sort of alert to generate.

Monitoring is a great way for a sysadmin to be pro-actively told that something needs some attention. It's hard to ignore, so it's unlikely an update will go unnoticed. Indeed it'll stay in an alert state until it is sorted out, so it forms something of a "to-do list" for even the most disorganised sysadmin. Monitoring alerts can usually be "downtimed" or muted for a period of time too, so that solves for the time between now and the next change window, or deploy opportunity or whatever.

How to do it

Exactly how you'd add an extra check to your monitoring platform is vendor dependent. A lot of platforms need you to save out a script in a particular location, and perhaps to modify a configuration file. You'd need to check with your monitoring platform documentation to work it out for your own circumstances.

Either way, you'll need some sort of script or program to make a query and to present the outcome to the monitoring platform in a format it can understand. Again, formats vary, so we won't get into it here.

Actually getting version and update information for different applications is also application dependent. There are some examples below which may give some clues of what's required. If the application is installed using an operating system package manager (eg. Apt or Yum/DNF) then it's relatively easy to see if updates are available (although you'll struggle to work out if there are any CVEs available that way). Other applications have special APIs to help you out.

Before we get into the examples, I'll just remind you that you probably only need to actually check for updates once or twice a day at most - certainly not as frequently as most monitoring systems run, so make sure you cache your results where you can.

Gitlab

Gitlab's browser-based UI alerts administrators of updates and CVEs. This alert is based on some information you can get from gitlab.com. You have to tell them which version you're currently using, and in return they'll tell you what the latest version is, and they'll set a "critical" flag if your current version has a known CVE.

Here's some Python code which fetches the update information from Gitlab:

import base64
import requests
import json

gitlab_version = "16.5.8-ce.0"

b64_version = base64.urlsafe_b64encode(json.dumps({ "version": gitlab_version}).encode('utf-8')).decode('utf-8')

url = 'https://version.gitlab.com/check.json?gitlab_info={}'.format(b64_version)

data = requests.get(url, headers={'Referer': 'https://yourgitlabservername.example.com'}).json()

# data now contains:
# {'latest_stable_versions': ['16.8.2', '16.7.5', '16.6.7'], 'latest_version': '16.8.2', 'severity': 'warning', 'critical_vulnerability': False, 'details': ''}
# if there are no new versions, the 'latest_stable_versions' key is an empty list.

By looking at the current version and the one returned here, we can work out if there's a bugfix, "dot release" or major release available, and the 'critical_vulnerability' and 'severity' flags tell us if it's an urgent upgrade requirement.

We're not 100% sure the 'critical' and 'severity' fields actually work. Seemingly they're used by the Gitlab software itself, so maybe they do work, but we tried quite a bit and couldn't get it to say anything was 'critical'.

Sonatype Nexus

There isn't a direct way to do this with the Nexus API (it's been asked for here: https://community.sonatype.com/t/monitoring-for-availabilty-of-updates-to-nexus-get-latest-version-by-api/11975 and raised internally, so that may change). Instead, it's possible to watch the public Github repository for releases there and use that information. There appear to be more releases on Github than formally documented in the Release Notes, so this may be slightly noiser than it needs to be. There's also no way to check for CVEs (which the Nexus admin UI does tell you if you log on to look at it). However, here's some Python to at least get the version information:

import requests
import json

response = requests.get("https://api.github.com/repos/sonatype/nexus-public/releases/latest")
version = response.json()["name"]

version = version.replace('release-', '')

# version now contains something like "3.64.0-01"

With this information, it's now possible to compare to the known current version and make an monitoring alert as required.

This same method can be extended to work with anything that has Github releases, no matter how it was installed or used.

Sonarqube

Sonarqube has an API which reveals this information, and it's also available in the web UI of you log on to look at it. There's no CVE notification though. Here's some Python which can get the information (it needs a Sonaqube username/password to access this API endpoint though):

import sys
import requests
import json
import re

updates_url = 'http://localhost:9000/api/system/upgrades'
username = 'someusername'
password = 'somepassword'

data = requests.get(updates_url, auth=(username, password)).json()

if 'upgrades' not in data or len(data['upgrades']) == 0:
    # Do whatever your monitoring tool needs here!
    print("No upgrades available")
    sys.exit(0)

latest = data['upgrades'][-1]

# Latest now contains a single dict which includes information about the update.
# Sadly there's no immediate 'version' key in it (with enough detail), but instead
# you can use the `'downloadUrl'` key to give you something like
# "https://binaries.sonarsource.com/Distribution/sonarqube/sonarqube-10.4.0.87286.zip"
# which you can snip the version from.

Other Applications

Other applications can be monitored quite easily if they get installed via Apt or Yum/DNF etc. For example, you could ask Apt if something needs updating as easily as this:

$ apt list nginx
Listing... Done
nginx/focal-updates,focal-security 1.18.0-0ubuntu1.4 all

If there's an update waiting, it'll look more like this:

$ apt list nginx
Listing... Done
nginx/focal-updates,focal-security 1.18.0-0ubuntu1.4 all [upgradable from: 1.17.0-0ubuntu1.4]

Some simple 'grepping' and maybe regexes to pull out the details you need and you have all the information to make a monitoring check from this.

Conclusions

Keeping critical software applications updated in a timely manner is a part of every Sysadmin's job. However, keeping on top of such things can be tricky.

The comprehensive solution is a security platform that inventories the estate and produces reports and alerts about anything installed there that needs an update. Such platforms also keep an eye on known security issues (CVEs) and can additionally alert if current versions have any in them.

However, smaller estates may not have the budget or resources for a comprehensive solution and so can elect to watch their key applications for updates using their existing monitoring systems. Some common apps have specific APIs to do this sort of thing, but some applications don't and may need some more reative thinking to monitor them.

If you need help keeping up to date with you updates, monitoring or obervability or general Sysadmin operations, please contact us - we can help you figure out what you need and make it work for you.

Image credit: deepai (it's what we got when we asked for 'update')