Moving to the cloud remains an important business initiative for organizations around the world. According to the World Economic Forum, digital transformation can enable sustainable growth and innovation. Part of the move forward with digital transformation plans includes moving more applications and services to the cloud. However, managing security presents new challenges for organizations adopting cloud and cloud-native technologies such as Kubernetes. According to Red Hat’s 2022 State of Kubernetes security report, 93% of respondents experienced at least one security incident in their Kubernetes environments. As organizations increasingly move production workloads to Kubernetes, it is important to understand both how to secure all aspects of Kubernetes and track and monitor workload security over time.
Fairwinds gathered data from over 150,000 workloads and hundreds of organizations to assemble the 2023 Kubernetes Benchmark Report, analyze trends in 2022, and compare it to data from the previous year. While a recent CNCF report indicated that 96% of respondents were using or evaluating Kubernetes, aligning to best practices can be difficult for organizations of all sizes. Unfortunately, the lack of alignment comes with real consequences: elevated security risks, unmanaged cloud costs, and decreased reliability of cloud apps and services.
Kubernetes is not secure by default, but many developers are not aware of this. For example, some Linux capabilities are enabled by default for Kubernetes workloads, even though most workloads do not actually require those capabilities. Unfortunately, the latest benchmark data shows that organizations are not limiting these capabilities as much in 2022 as they did the previous year. In 2021, 42% of organizations turned off these capabilities for most workloads (only 0-10% of workloads were impacted). In 2022, only 10% of organizations had these same insecure capabilities turned off.
“33% of organizations have more than 90% of workloads running with insecure capabilities, an increase compared to the previous year.”
The security setting readOnlyRootFilesystem prevents a container from writing to its filesystem. It is important to enable this setting in case an organization is hacked. This setting can ensure that if an attacker gets in, they will be unable to tamper with the application or write foreign executables to disk. This is another instance where it is not set to true by default on Kubernetes workloads, which means teams must explicitly change the setting to ensure the most secure configuration possible. In the past, only 23% of organizations appeared to be unaware that they needed to change this setting to override the insecure default setting for 71%-100% of their workloads. Unfortunately, that number increased in 2022; now 56% of organizations are failing to make that override. As Kubernetes usage and adoption grows, it is alarming to see this trend in this direction.
Under some configurations, containers may have the ability to escalate its privileges. If you set allowPrivilegeEscalation to false, that sets the no_new_privs flag on the container process, which prevents setuid binaries from changing the effective user ID. It is especially important to set this flag when you are using runAsNonRoot, which can otherwise be circumvented. This security is also not set by default, which means that security-conscious teams must explicitly set it. This year, the benchmark report shows a disturbing increase in workloads open to privilege escalation. In 2021, 42% of organizations locked down the majority of workloads. In 2022, that number dropped to 10%.
The privileged command determines whether any container in a pod can enable privileged mode. By default, a container may not access any devices on the host. However, a privileged container has access to all devices on the host. When this feature is enabled, it allows the container nearly the same level of access as processes running on the host. This is useful for containers that need to use Linux capabilities, such as manipulating the network stack and accessing devices. In this case, the privileged flag is off by default. Likely for this reason, 87% of organizations have the privileged flag off, which increases the security of workloads. In 2021, 88% of organizations had the privileged flag off, so it decreased slightly in 2022.
Another insecure capability is running containers as root. Many workloads in the Benchmark Report, unfortunately, are allowed this capability. The benchmark data shows an increase in the number of workloads in which running as root is allowed. In 2022, 44% of organizations were running 71% or more of their workloads allowing root access. That is an increase of twenty-two points compared to 2022. Given that there are known vulnerabilities targeting this capability, it is an alarming increase to see in the report.
Workloads impacted by image vulnerabilities increased significantly in 2022. In 2021, 40% of organizations had fewer than 10% of workloads impacted by image vulnerabilities; in 2022, that fell to only 12% of organizations. Malicious actors exploit known vulnerabilities, therefore they must be patched or remediated as quickly as possible. According to the report, 62% of organizations have more than 50% of workloads impacted by vulnerabilities.
Is your organization running images with vulnerabilities?
Keeping up to date with the latest releases for all your cluster add-ons is challenging, so it’s not surprising that outdated Helm charts are a common issue across most organizations. In 2022, 46% of organizations have 50% or greater workloads impacted by running workloads from outdated Helm charts. The add-ons running your cluster are probably installed by Helm. Each add-on has its own release cadence, and some updates include critical security patches. Helm charts must be kept up to date, but they can be difficult to monitor and predict. Nova is an open source project that cross checks Helm charts running in the cluster with the latest version available, which makes it easy to see when updates are available.
This year we started benchmarking how many organizations are running outdated container images. The new benchmark data shows that either less than 10% or greater than 90% of workloads are impacted. Nova, an open source tool, can run a flag called “containers” to analyze all container images in a Kubernetes cluster and notify users if an updated version is available. Nova provides three alternatives for updating images:
the latest version
the latest minor version
the most recent patch version
This allows users to choose the patch that they feel most comfortable using for the specific container.
According to the benchmark data, most organizations have just a few workloads with deprecated API versions. Unfortunately, in this category it is also trending in the wrong direction. In 2021, 82% of organizations were up to date with API versions for the vast majority of their workloads. In 2022, that number decreased to 74%. Monitoring for deprecated APIs and keeping them up to date remains a critical step in reducing risk during Kubernetes upgrades.
Using containers and Kubernetes for container orchestration enables a shift to cloud-native applications and services. These infrastructure changes bring significant value to organizations today. However, as many rapidly adopt Kubernetes and endeavor to deploy more and more applications to Kubernetes, they must also understand the many configurations available and how to set them appropriately. The Kubernetes Benchmark report can help you understand both where configurations are deficient or trending in the wrong direction and how to make changes going forward to ensure that your organization’s deployment is as secure, reliable, and cost-efficient as possible.
Read the complete Kubernetes Benchmark Report today.
