Earlier this month, the National Security Agency (NSA) and the Cybersecurity and Infrastructure Security Agency (CISA) released the 1.0 version of the Kubernetes hardening guide in August 2021, updated it based on industry feedback in March 2022 (version 1.1). The most recent version of the Kubernetes hardening guidance was released in August 2022 with corrections and clarifications. Version 1.2 outlines a number of recommendations for hardening Kubernetes clusters. The guide outlines a strong defense-in-depth approach to ensure that when an attacker compromises your cluster, the blast radius will be as small as possible. The NSA Kubernetes hardening guide includes the following recommendations:
Scan Linux container images and Pods for vulnerabilities or misconfigurations.
Run containers and Pods with the least privileges possible. Use containers that have been built to run applications as non-root users. Run containers with immutable file systems if possible.
Use technical controls to enforce minimum levels of security, such as preventing privileged containers, denying container features that are frequently exploited to breakout (hostPID, hostIPC, hostNetwork, allowedHostPath), rejecting containers that execute as the root user (or allow elevation to root), hardening applications against exploitation.
Use network separation and hardening to control the amount of damage a compromise can cause. This includes locking down access to control plane nodes and using separate networks for control plane components and nodes, limiting access to the Kubernetes etcd server (etcd should be managed through the API server), configuring control plane components to use Transport Layer Security (TLS) certificates to ensure authenticated, encrypted communications, encrypting etcd at rest, setting up network policies to isolate resources, creating an explicit deny access policy, and putting credentials and sensitive information encrypted in Kubernetes Secrets.
Use firewalls to limit unneeded network connectivity and encryption to protect confidentiality. Do not expose the Kubernetes API server to the internet or to an untrusted network.
Use strong authentication and authorization to limit user and administrator access as well as to limit the attack surface. Make sure that anonymous login is disabled and create role-based access control (RBAC) policies with unique roles for the infrastructure team, service accounts, developers, users, and administrators.
Use log auditing so that administrators can monitor activity and be alerted to potential malicious activity. Enable audit logging and ensure that logs are available if nodes, posts, or containers fail. Configure logging in the environment, including cluster metric logs, cluster application program interface (API) audit event logs, Pod seccomp logs, application
logs, and repository audit logs. Aggregate the logs in a location that is external to the cluster and implement a log monitoring and alerting system.
Your security policy should require you to periodically review all Kubernetes settings and use vulnerability scans to help ensure risks are appropriately accounted for and security patches are applied. Follow application security best practices, including upgrading as needed, applying security patches and updates in a timely manner, performing vulnerability scans and penetration tests periodically, and removing unused components from the environment.
Organizations are adopting Kubernetes and delivering cloud native applications, deploying apps and services on cloud providers such as Amazon Web Services (AWS), Azure, and Google Cloud Platform. Kubernetes is not secure out-of-the-box. By default, everything you deploy to your cluster is able to communicate with everything else. Pod security is important because pods can run as root, and even if you disable privileged mode, there are still plenty of pod settings that an attacker inside your cluster can enable to get root access to the underlying node.
Unfortunately, there is a pretty standard process for attacking that cluster you’ve spent so much time working on. Consider the many possible attack vectors: using a leaked kubeconfig or leaked cloud credentials, a supply chain attack, an exposed dashboard, or exploiting a known security vulnerability. The attacker can then execute some of their own code within your cluster, escalate privileges, and then cover their tracks. All of this behavior happens while also creating a persistent foothold, enabling attackers to steal secrets and data, consume compute resources for financial gain, or achieve any number of other results.
Securing your microservices against attacks is getting harder and harder as architecture continues to grow increasingly complex. Do you know if every pod in your Kubernetes cluster has hostPath disabled? How about the flow of network traffic through your cluster — are pods able to communicate with control plane components or neighboring namespaces? These little details are the difference between an attack’s blast radius being limited to a single container, to having your cluster compromised, or even having your entire cloud environment compromised. This could happen if an attacker escaped the pod, used the node’s cloud credentials to gain access to your account, and found more ways to escalate privileges.
From an insider threat perspective, RBAC is essential for ensuring that every person and application with access to your cluster has an appropriate level of permission. Managing RBAC profiles, and even gaining visibility into what's configured, can be an overwhelming task. A malicious insider acting inside the cluster could use existing permissions or escalate their permissions by exploiting known vulnerabilities, which is why it's critical to manage RBAC, monitor for and patch known vulnerabilities in containers, and review the security permissions of your services.
The Cloud Native Computing Foundation continues to provide an open source hub for projects including Kubernetes and Prometheus and Grafana, among others. Using open source software, such as Fairwinds’ Polaris, is a great first step towards checking your service configuration. With Polaris, you'll get a point-in-time snapshot of your Kubernetes security posture. Polaris also provides other best practice recommendations to help you discover Kubernetes misconfigurations and avoid problems.
While Polaris improves Kubernetes deployments, the NSA Hardening Guide advocates for a more continuous approach to Kubernetes security. Fortunately, several of these recommendations can be implemented using Fairwinds Insights, which combines Polaris with other open source tools to provide a platform that simplifies Kubernetes complexity and reduces risk.
With Fairwinds Insights, DevOps teams will be able to operationalize Polaris to run on a continuous basis and integrate multiple open source security tools from CI/CD all the way through to production. Fairwinds Insights adds:
RBAC monitoring: flag any highly permissive profiles
Configuration scanning: ensure deployed workloads and pods are not using excessive security features of Kubernetes
Policy enforcement and guardrails: establish your minimum security requirements in both CI/CD and Admission Controller contexts. This will automatically prevent security misconfigurations from downstream teams who use the cluster
Runtime container vulnerability scanning: identify which deployed containers container high severity CVEs, or have recently become vulnerable
If you're interested in learning more, please try out Fairwinds Insights to see how it can help you harden your Kubernetes environments.
Originally published August 10, 2021, and updated January 23, 2023
