<img height="1" width="1" style="display:none" src="https://www.facebook.com/tr?id=521127644762074&amp;ev=PageView&amp;noscript=1">

Meeting the Challenges of Kubernetes Multi-cluster Management

Whether you are looking to separate security boundaries, isolate the impact of a scary configuration change, scale applications closer to customers, achieve data sovereignty, or going for the multi-cloud dream, you likely have multiple Kubernetes clusters in your life. In this post I will describe some characteristics and tools that increase your success managing multiple clusters.

Reliance on Standards and Process

As Kubernetes clusters multiply, following standards and having repeatable processes becomes doubly important to produce consistent behavior and maintain order.

An example of such a standard is all clusters having the same version of Ingress or DNS controllers installed. An accompanying process documents or codifies multiple perspectives from well-rested engineers about how to safely upgrade those controllers.

Standards and processes will adjust as your infrastructure expands and you support differences between clouds, services that are not available in all regions, or new architectures. It can sometimes feel like the extra burden of maintaining standards and processes gets in the way of progress, but these habits help your infrastructure grow faster and farther, once you reach escape velocity. Part of maintaining standards does include adjusting to change.

  • The Fairwinds open source tool Pluto helps you detect when Kubernetes API versions are deprecated or no longer available. For example, an upgrade to Kubernetes 1.16 may have broken your ability to upgrade applications if the manifests for those applications had not been upgraded before-hand — Pluto helps you stay aware of how your Kubernetes manifests or Helm charts need to evolve with the Kubernetes API.
  • Similarly, Fairwinds Nova is an open source tool that compares Helm releases with known Helm repositories, and notifies you if you have deprecated charts or out-dated versions installed.

Policy

Policy helps keep non-standard changes from being made to your Kubernetes clusters and workloads. Enforcement of policy should provide guardrails where necessary, early in development or engineering workflows. For example, if a Kubernetes Deployment should be disallowed due to an insecure configuration, ideally it should fail during Continuous Integration instead of when the application gets deployed to Kubernetes.

  • Fairwinds Polaris is an open source tool that compares your Kubernetes workloads to best practices via a CI/CD integration and Kubernetes Admission Controller. The Admission Controller blocks resources from being deployed to Kubernetes clusters at the API level. Polaris allows defining your own standards using JSON Schema, which matches against the Kubernetes API specification. Examples of custom Polaris checks include disallowing pulling images from non-standard registries, or requiring that a Kubernetes Deployment is also accompanied by a Pod Disruption Budget.

More specific policies can be accomplished using Open Policy Agent or similar tools. For example, a labeling standard for Kubernetes resources helps you track how your platform is being consumed, and eases deployment and cleanup of feature branches. Labels can be used to instruct CI/CD about where an app should be deployed, RBACBindings necessary for developer access to development environments, network access, or when an ephemeral development environment should be cleaned up. Labels are only reliable if they exist everywhere. A policy can enforce that particular labels are set to a range of values by writing a more expressive policy in Rego — the language used by Open Policy Agent.

Visibility

Reactive or short term adjustments can get lost in a sea of clusters. Here are some tools that help spot unintended deviations in your fleet.

  • Fairwinds Polaris provides a dashboard view of in-cluster workloads and how they relate to best practices.
  • Fairwinds Goldilocks helps get your Pod resource requests and limits "just right" by using vertical-pod-autoscaler in recommendation mode, visualized in a dashboard.
  • Fairwinds RBAC Lookup is a command-line tool that eases finding the roles attached to users, groups, or ServiceAccounts that authenticate to Kubernetes.

To help wrangle multiple dashboards and provide a unified place to manage results from great tools such as these ... there's an app for that. :)

Fairwinds Insights

Insights aggregates the results of great open source tools, including the ones described above, into a single view, which allows you to assess and enforce your standards across multiple clusters. Insights uses Polaris and Open Policy Agent to provide central policy management for some or all of your clusters. Whether you would like Slack alerts or ticket creation for issues surfaced by Insights, your engineers can track their fixes without constantly returning to the Insights dashboard.

Fairwinds Insights is available to use for free. You can sign up here.

Managing multiple Kubernetes clusters across teams and clouds inevitably introduces inconsistencies that can cause errors and cost time and productivity. Many of these challenges are addressed by our open source tools, but for a greater scale and improved management, check out Fairwinds Insights.

Join the Fairwinds Open Source User Group today