Phase 5: Control

The next phase of Kubernetes maturity is when you introduce more measurement and control of the environment. You and your team are functioning well within Kubernetes, have overall understanding and there is organization-wide adoption. You are developing a deeper functional understanding of Kubernetes and opinions on how things should be done within clusters and the overall environment. Further, the team is ready to tackle technical debt from previous phases.

Previous stages have introduced some monitoring and observability. In this stage, you will gather and process more data, insights and tooling for you to start understanding what to measure and track and how to control Kubernetes.

Employing more sophisticated monitoring and alerting is now a focus to help understand common issues, Kubernetes errors and how to solve them. You’ll be more familiar with building out monitoring dashboards to catch problems and common misconfigurations before requiring assistance. You’ll also know the general location of problems when they occur to more easily troubleshoot.

During this phase, you’ll start improving how you measure your Kubernetes environment and track success. Measurement will be around five key areas:

1. Security - You will measure how many and what vulnerabilities exist in your containers or clusters and how often/when are you patching workloads, clusters or add-ons.
2. Auditing - You’ll create an audit trail to understand who has performed recent actions and what actions workloads are taking in your clusters. You’ll be able to identify if unauthorized access or actions have occurred.
3. Drift - You will be able to identify which workloads do not conform to your standards, what versions of dependencies/cluster add-ons are running and if workloads are compatible with future versions of Kubernetes.
4. Efficiency - You’ll measure to track the typical or standard resource usage of your workloads and the typical capacity/usage of the nodes within your clusters. You’ll also know how often your clusters are scaling.
5. Velocity - You’ll measure to improve your development velocity. This will include understanding how often deployments are being shipped, how many users access your clusters and the most common actions being taken within your clusters.

You will experience pain around workloads and other Kubernetes resources mainly around consistency. Workloads may be inconsistently or manually deployed and then modified. There are likely discrepancies in configurations across containers and clusters which can be challenging to identify, correct and keep consistent. Workloads can be disorganized, impacting other workloads. There may be too much access from workloads causing security issues. There may be reliability or scalability issues (not scaling enough or scaling too frequently). Cost may creep up too high as too many resources are being used or workloads are not being cleaned up. 

All of these paint points are natural as you and your team mature. It is in this phase that you want to put control policies in place around security, configuration and workflows. Here are some examples of what you need to consider.

Kubernetes workload security is essential. You need control around cluster permissions and should be able to answer:

• Who has access to clusters
• What actions users can take within clusters?
• What actions workloads can take within clusters?
• What level of permissions workloads have within clusters?
• What are the network policies between workloads within your clusters?

Solid Kubernetes environments will have configuration standards for consistency. You should have controls in place around:

• Where Kubernetes resources live and are defined?
• What changes happen and when?
• What is your code review process for resources?
• What type of resources can be deployed in your clusters?
• Which namespaces are usable by which users?
• Which namespaces workloads are deployed to?
• How do you set the amount of resources available to a workload or namespace?
• What are your common standards/defaults across your workloads/deployments?

Similarly, you will have established workflows for how workloads and services are deployed, promotion paths and responsibility:

• Who can deploy workloads and services to your clusters?
• How workloads and services can be deployed to your clusters?
• What is the promotion path between environments?
• Who is responsible for what aspects of your environment?

By answering these questions, you will now have a set of policies to start implementing configuration changes within your clusters. You also have the Kubernetes experience to loop back to these more advanced topics. You’ll revisit configurations where you may have simply chosen default options, inspect what is happening and make changes.

1. Inconsistency in configurations and processes across workloads, clusters, team
2. Users and workloads have too much access and may be running/acting in an insecure manner
3. Common reliability issues are burdensome and not well understood for lack of effective monitoring/alerting

Phase five will allow you to improve and refine your Kubernetes environment to ensure it is delivering against your business requirements. It may also help you to revisit previous phases so that as you migrate new apps or set up new environments you avoid introducing unnecessary problems.

You may at this stage of your maturity require help to identify where you can make improvements. An audit of your Kubernetes environment is a great tool here. You can also employ configuration validation tools.

As you exit phase five, you’ll have established protocols and procedures so that the team interacts consistently with systems and understands priorities. You’ll also have a sophisticated understanding of infrastructure as code and CI/CD helping to one-off changes.