Building and managing a Kubernetes-based platform for multiple development teams is a complex project. On one side, you want to empower developers with the flexibility to innovate, experiment, and deploy quickly. On the other, you need to ensure security, reliability, and operational efficiency—especially when running shared infrastructure. Striking the right balance between flexibility and standardization is not just a technical challenge; it’s a core responsibility of modern platform teams.
Recently, I did a presentation at a Meetup in DC exploring the real-world challenges and best practices for building a Kubernetes platform that serves diverse teams, drawing on my insights from years of experience managing Kubernetes infrastructure for Fairwinds Managed Kubernetes-as-a-Service clients.
A Kubernetes platform is more than just a cluster. It’s a set of addons, interactions, and abstractions that enable developers to deploy microservices and applications efficiently while minimizing the operational burden. The ideal platform supports developers with different needs and experience levels. Some want to configure every detail, while others prefer a streamlined, self-service experience.
At its heart, a platform should be an enablement tool for self-service, allowing developers to own both how their apps are built and how they’re run. As my former colleague, staff engineer Rachel Sweeney, puts it, “If you force the team to use your abstractions, you’re always going to be reactionary to their needs.” Instead, the goal is to create a “Golden Path”—an approachable, well-documented, and safe way to deploy, while still allowing for experimentation and customization.
When multiple teams share a Kubernetes platform, the range of deployment patterns, tools, and workflows can quickly become overwhelming. Some teams may already be running applications in various ways—on Kubernetes, on managed services like ECS or Cloud Run, or even on legacy platforms. Bringing everyone onto a shared Kubernetes cluster introduces new challenges around security, reliability, and efficiency.
Flexibility in these types of environments is essential. Developers need the freedom to try new tools, experiment with different patterns, and iterate quickly. However, standardization is equally important. Without guardrails, the platform can become unstable, insecure, or inefficient, especially when workloads share resources and dependencies.
The challenge is to provide enough flexibility to support innovation while still enforcing the standards needed to keep the platform safe and reliable.
Let’s explore how platform teams have addressed this tension in several critical areas:
Ingress, the way that you route external traffic (aka the internet) into your applications, is a core consideration for any Kubernetes platform. Over time, the Kubernetes ecosystem has evolved from simple Service objects to Service Proxies, API Gateways, Service Mesh, and other tools in the cloud native network, such as Cilium, Container Network Interface (CNI), Open vSwitch (OvS), and other tools. Each offers different levels of control and complexity.
Platform teams must decide how much to abstract away for developers. For example, some teams use mutating admission controllers to automatically update Ingress objects to the latest standards, so developers don’t need to change their configurations manually. This approach allows for broad standardization without disrupting developer velocity.
In one client site, a platform team chose Gloo Gateway (Gloo Edge API) a tool from Solo.io based on Istio to provide both fine-grained control for experienced users and simple abstractions for those who just want to get their service online. (The open-source version is now a Cloud Native Computing Foundation (CNCF) project called kgateway.) By modularizing the configuration, they enabled developers to focus on what matters most to them, while the platform team managed the underlying complexity.
Kubernetes offers many ways to deploy applications—Helm charts, Kustomize, raw manifest files, and more. Platform teams can provide templates and guidance, but must also enforce best practices around security, resource allocation, and reliability.
Policy engines, such as Polaris, Kyverno, and Open Policy Agent (OPA), can automatically check for common misconfigurations, such as insecure images, missing resource limits, or containers running as root. These tools can be integrated into CI/CD pipelines or run as admission controllers, ensuring that only compliant workloads are deployed.
For example, one organization uses a policy engine to enforce that all workloads have PodDisruptionBudgets and resource requests set while allowing for more flexibility in other areas. This approach reduced the risk of cluster-wide failures and improved overall reliability.
Role-Based Access Control (RBAC) is essential for securing a Kubernetes platform. Developers need to have the permissions necessary to do their jobs, but not so much that they can accidentally (or intentionally) disrupt the cluster or access sensitive data.
Platform teams can use tools to simplify RBAC management, such as mapping users and groups to roles or integrating with identity providers. They can also provide developers with tools for local testing, such as Telepresence. Telepresence allows developers to intercept traffic from the cluster and route it to their local environment, so they can test changes without going through a full build and deploy process.
A Kubernetes platform generates vast amounts of metrics, logs, and traces. Managing this data is a significant challenge, both in terms of cost and complexity.
Platform teams can standardize on tools like Prometheus, Grafana, and Jaeger for open-source monitoring, or use vendor solutions, such as Datadog or Splunk. OpenTelemetry has emerged as a vendor-neutral standard for instrumenting applications, making it easier for different teams to send data to the tools of their choice.
When I worked at FEMA, I set up a comprehensive observability suite that provided a single pane of glass for monitoring, alerting, and troubleshooting. This approach not only improved incident resolution times but also made it easier for developers to understand and debug their applications.
Several key lessons emerge from working with our clients to build real-world Kubernetes platforms:
Platform teams are not an infinite resource. Every tool, abstraction, or service you offer requires ongoing maintenance, support, and expertise. The more flexibility you provide, the more complexity you introduce—and the more work is required to keep things running smoothly.
The best platform teams act as enablers, not gatekeepers. They provide the tools, documentation, and support that developers need to be successful, while ensuring that the platform remains secure, reliable, and efficient.
Ready to build your own Kubernetes platform, one that balances flexibility with standardization? Here are five solid steps to get you started
By finding that elusive balance between flexibility and standardization, you can build a Kubernetes platform that empowers your developers, keeps your infrastructure secure and reliable, and drives innovation across your organization.
If you want to focus on building new applications and services instead of spending time figuring out how to build out production-grade Kubernetes infrastructure, FairwindsManaged Kubernetes-as-a-Service can help.
