One of the questions that I get most frequently from our customers at Fairwinds is “How do we know what to set our resource requests and limits to?” Today we are open-sourcing a tool that will hopefully give everyone a little help in that area. Goldilocks is a Kubernetes controller that provides a dashboard that gives recommendations on how to set your resource requests.
In order to provide recommendations, we use theVertical Pod Autoscaler(VPA). The VPA controller stack contains a recommendation engine that takes into account the current resource usage of your pods in order to provide a guideline. The primary goal of VPA is to actually set those for you, but we aren’t currently comfortable with how it does this; more specifically, we like to run Horizontal Pod Autoscaler, which doesn’t play nice with the VPA. Instead, we just use the recommendation engine that VPA provides to give you good recommendations on how to set your resource requests and limits.
The way we utilize the VPA recommendation engine is simple. We run a controller in the cluster that watches for namespaces that are labelled withgoldilocks.fairwinds.com/enabled=true. Within those enabled namespaces, we look at every deployment object and create a corresponding VPA object. That VPA is set withMode: Offand doesn’t actually modify your resource requests and limits; it just sits there and provides a recommendation. This alone is sort of cool, but in order to view these recommendations, you would have to usekubectlto query each and every VPA object. For medium to large deployments, this can get very tedious. That’s where the dashboard comes in.
The Goldilocks dashboard provides a visualization of the VPA recommendations. Now you can visit a service in your cluster and see two types of recommendations, depending on what QoS Class you desire for your deployments. The dashboard looks like this:In
What if I want to exclude containers from the dashboard?
This was one of the first upgrades we added to Goldilocks, mostly because of the sidecar proxy pattern. Frequently we see clusters where every pod gets a sidecar, such as the Istio or Linkerd2 sidecars. You may not want to see resource recommendations for those containers because we may or may not have control over those values. There are two ways to exclude these containers.
The first method is global and can be set by running the dashboard with the flag--exclude-containers "istio-proxy,linkerd-proxy". This is set by default in the Helm chart installation.
The second method is to exclude containers per deployment using a label on the deployment itself:goldilocks.fairwinds.com/exclude-containers=linkerd-proxy,istio-proxy.
What is this QoS Class?
If you’re not already familiar with it, QoS Class refers to different ways of setting your resource requests and limits. There are two that we care about, and one that should almost never be used. You can read about them in detailin the Kubernetes documentation, but I will summarize them here.
Guaranteed— This is my favorite QoS Class as it generally lends itself to the most stable Kubernetes clusters. In this class, you set your resource requests and limits to exactly the same values which guarantees that the resources requested by the container will be available to it when it gets scheduled.
Burstable— This means that your resource requests are lower than your limits. Essentially, the scheduler will use the request to place the pod on a node, but then the pod can use more resources up to the limit before it’s killed or throttled.
BestEffort— You don’t set any resource requests or limits. We don’t recommend you do this, ever.
The Goldilocks dashboard will give you recommendations for both the Guaranteed and Burstable QoS classes. These are generated by different values from the VPA recommendation field. In the Guaranteed class, we suggest you set your requests and limits to the VPA “target” field. In general, we use this value along with the Horizontal Pod Autoscaler to allow our applications to scale. The Burstable class recommendation is taken from the VPA LowerBound and UpperBound fields. You can read more about the recommender from VPAin their repo.
Thanks for Reading
If you have a question (or have an idea for how it could be better!) we’d love to hear from you. The best way to reach us is with through a Pull Request or Issue onGithub. Thanks for reading!