<img height="1" width="1" style="display:none" src="https://www.facebook.com/tr?id=521127644762074&amp;ev=PageView&amp;noscript=1">

Tell us more

Blog

Nova Open Source Tool - Monitor Helm for New Releases

Image of galaxy

When running a production Kubernetes cluster, one of the most important (and burdensome) tasks is keeping everything up-to-date. Kubernetes itself cuts quarterly releases, which quickly fall out of support—if you fail to upgrade, you could be left open to security holes and buggy features. Fortunately, the quarterly cadence at least makes these releases predictable: you can even set a reminder to go through the process every three months.

Cyber warfare cartoon "First we inundate them with quarterly Kubernetes releases."

Source: https://gravitational.com/blog/kubernetes-release-cycle/

But on top of Kubernetes, you probably have half a dozen add-ons running in your cluster, most likely installed  via Helm. Supplementary components like nginx-ingress, cert-manager and linkerd are often critical to running a production-grade cluster. You might even be using k8s-specific deployments for common applications like PostgreSQL or Wordpress. Each of these tools has its own release cadence, and some updates may come with critical security patches.

Installing a new patch typically isn’t that hard - usually it just means rerunning a helm install, command, or updating a single line in your  Infrastructure-as-Code repository. But how do you know when it’s time to update? Unlike Kubernetes itself, Helm chart updates are incredibly hard to monitor and predict.

Observe and Report

Nova_bfw_color

Enter Nova, an open source tool by Fairwinds.

Nova is a command-line interface for cross-checking the Helm charts running in your cluster with the latest versions available. Click to Tweet

Nova is a command-line interface for cross-checking the Helm charts running in your cluster with the latest versions available. Nova will let you know if you’re running a chart that’s out-of-date or deprecated, so you can make sure you’re always aware of updates.

Release Name      Installed    Latest     Old     Deprecated
cert-manager      v0.11.0      v0.15.2    true    false
insights-agent    0.21.1       0.21.1     false   false
grafana           2.1.3        3.1.1      true    false
metrics-server    2.8.8        2.11.1     true    false
nginx-ingress     1.25.0       1.40.3     true    false

This makes it easy to see how far behind you are on any installed Helm charts. In the example above, we’ve got the latest version of insights-agent,but there are minor updates available for some core infrastructure, including cert-manager,nginx-ingress,and metrics-server.grafanaalso has a new major version available, so we might be missing out on some cool new features!

Challenges

This turned out to be a harder problem than expected.

The first issue was in supporting both Helm 2 and Helm 3. Helm 2 stored chart metadata using ConfigMaps, which were fairly easy to parse. But Helm 3 uses Secrets, along with some byzantine encoding practices. With some help from  the Pluto team, we built support for both versions, as well as the ability to auto-detect charts from both versions. You can use this feature by specifying --helm-version=auto.

Even more challenging was matching the installed chart to the upstream source. Unfortunately, Helm doesn’t store info about the upstream repository in the chart metadata, making an exact match impossible. All we have to go on is the name, which might be duplicated across different repositories. For example, many of  the Bitnami charts were recently migrated out of the core charts repo, but kept the same names. 

To find a match, we have to look in all known repositories for charts with a matching name, then use heuristics like the chart’s version, home, description, and maintainers to find a good match. So far this strategy has a 100% success rate, but it feels like a bit of a hack. If you know of a better solution for finding the upstream repository for an installed Helm chart, let us know!

Putting it into Practice

Nova saves a lot of time and effort for anyone looking to keep their charts up-to-date. Instead of having to monitor each individual chart for updates, operators can now run a single CLI to detect old versions running in their cluster. But you still have to remember to run the CLI!

When managing hundreds of clusters, the chances of forgetting to run Nova on a particular cluster, misreading the output, or failing to act on the results, are all but certain at scale. Furthermore, some organizations need to be on an older version of a given tool (in particular, cert-manager has regularly introduced breaking changes). So we needed a way to operationalize Nova at scale.

The easiest way to do this is using  Fairwinds Insights. Insights can run Nova (along with other Kubernetes auditing tools) automatically on a regular schedule—every hour by default—and use the results to create  Action Items. These Action Items serve as a reminder to update your Helm Chart, and can be piped to Slack, Datadog, or anywhere else your engineers live.

Internally, we also maintain a YAML file that includes a set of standards - particular versions of each add-on that we expect our clients to run. We use Nova to make sure that YAML file stays up-to-date with the latest versions, and to keep our Infrastructure-as-Code in line with those standards.

What’s Next

Automatically checking Helm charts for updates solves a big problem for us at Fairwinds, and for the Kubernetes community as a whole. But the ecosystem moves fast, and there are many more pieces of software and infrastructure that we need to stay on top of. Other things we’d like to build automated checks for are:

  • Supported k8s versions on managed Kubernetes providers (like GKE, EKS, and AKS)
  • Docker tags (at least where semver tagging practices are used)
  • Docker base images

Whether these checks fit into Nova, or will justify creating a separate project, remains to be seen. 

In any domain, staying on top of the latest releases is a big, scary problem. We hope Nova helps the Kubernetes community keep their clusters reliable and secure.

Learn more about Nova on GitHub