Our NSA Kubernetes Hardening Guide series has looked at pod security, network access, authentication and authorization, audit logging and threat detection. In the final installment of the series, we look at upgrading and application security practices.
The NSA Kubernetes Hardening Guidelines aims to help organizations strengthen their container and Kubernetes security to minimize the chances of a breach, and ensure that the blast radius will be as small as possible if an attacker does infiltrate your cluster. Here we look at security patches and updates, vulnerability scanning and deleting unused components in clusters.
[S]ecurity is an ongoing process, and it is vital to keep up with patches, updates and upgrades. The specific software components vary depending on the individual configuration, but each piece of the overall system must be kept as secure as possible. This includes updating Kubernetes, hypervisors, virtualization software, plugins, operating systems on which the environment is running, applications running on the servers, all elements of the organization’s continuous integration/continuous delivery (CI/CD) pipeline and any other software hosted in the environment.
Keeping Kubernetes and all the underlying components and container images up-to-date can be a full-time job (or even multiple full-time jobs). Having the right tooling in place can help reduce the toil and engineering effort involved.
Fairwinds Insights, a Kubernetes governance platform to help implement security guardrails, provides two services which help here:
Nova scans the environment for any Helm releases which are out-of-date or deprecated. It will surface an Action Item for each release that has updates available.
Trivy scans every container in the environment for vulnerabilities (either in the operating system or in the software installed on top) and will recommend upgrades when a patch is available.
Additionally, it's important to ensure Kubernetes itself stays up-to-date, as each version is only kept in support for about one year.
Administrators should periodically check to ensure their system's security is compliant with current cybersecurity best practices. Periodic vulnerability scans and penetration tests should be performed on the various system components to proactively look for insecure configurations and zero-day vulnerabilities.
A Kubernetes cluster is not a static piece of infrastructure. It is a constantly evolving ecosystem, with new applications and updates being applied daily, often automatically. It's imperative that organizations routinely scan their Kubernetes environment for vulnerabilities using a combination of automation and manual penetration testing.
By default, Fairwinds Insights runs each of its components on an hourly basis and can be configured to run as often as every minute. This capability ensures issues are raised as quickly as possible when a new vulnerability is discovered.
In addition to the Trivy, Polaris and kube-bench integrations listed above, Insights also provides a kube-hunter integration, which automatically probes clusters for network vulnerabilities and other security issues that an attacker could exploit.
As administrators deploy updates, they should also keep up with uninstalling any old, unused components from the environment and deployment pipeline. This practice will help reduce the attack surface and the risk of unused tools remaining on the system and falling out of date.
Making sure that stale and unused deployments don't linger inside your Kubernetes environment is important for both security and general hygiene. Unfortunately, Kubernetes doesn't have a generic way to check the staleness of resources in the cluster. There is an open issue, though some things can be checked on a case-by-case basis.
Fairwinds Insights has two customizable OPA policies for detecting Helm charts and Deployments that have not been updated for a certain number of days, which greatly helps the toil and manual effort involved in detecting stale resources.
Fairwinds Insights, a platform for Kubernetes governance and security, can help accomplish many of the NSA’s most important guidelines. Utilizing Fairwinds Insights, in conjunction with other best-of-breed commercial and open source software, can help organizations’ achieve compliance with the NSA’s recommendations.
