In the last kops post, kops 101, I covered what kops is and why it’s the right choice for professional-grade Kubernetes installations. This week, I’ll cover the Fairwinds Way of setting up Kubernetes with kops.
I want to share an example of an Fairwinds-style Kubernetes configuration on AWS with kops. It includes an undocumented feature that the DevOps community may find pretty useful. Before we get too far, I also want to introduce the idea of the Fairwinds way. An ongoing process of trial and error has enabled our team to set best practices for creating infrastructure in a thoughtful, repeatable way. That means that we have an opinion and stand behind it.
With that, let’s dive in.
There are two steps/components to creating Kubernetes clusters with kops the Fairwinds way:
We use Terraform, an open source infrastructure management and provisioning tool by Hashicorp, to create our base VPC and networking layout. We always use the same initial structure that we’ve tested and verified. The VPC we create is intended to host both Kubernetes and any other cloud-based resources that we may want Kubernetes to interact with, such as hosted databases, caches, and messaging queues.
Just a quick overview of what our VPC gets you:
The VPC is flexible enough for almost any situation within our scope, and we engineer it so that there are no single points of failure.
Of course, you could just let kops create the VPC for you. This is the default mode, and to be fair it’s definitely an attractive way to work, especially since we’ve designed kops to automatically make sensible choices about your infrastructure. However, at Fairwinds, we recognize that Kubernetes is but one piece in a highly complex ecosystem. Thus, we set the stage with Terraform and then let kops position Kubernetes within that stage.
Now that we have our VPC and our basic networking resources in place, we can get into the Kubernetes-specific configuration.
To go from our Terraform VPC setup to a working cluster, use these four commands:
kops create -f cluster_spec.yml
kops create secret --name $CLUSTER_NAME sshpublickey admin -i $SSH_KEY_PATH
kops update cluster $CLUSTER_NAME # Sanity check the output. Make sure that kops is
only making the changes you expect.
kops update cluster $CLUSTER_NAME --yesWas that really all that we needed to deploy a cluster? Yes and no. While it’s probably not how you would deploy your first cluster, once you’ve gone through it a few times and you understand how the Cluster Spec works, it’s a great way to define and create your clusters.
So let’s take a moment to discuss the Cluster Spec. The Cluster Spec is a fundamental concept for kops. Not all Kubernetes clusters have a cluster spec, but all clusters created by kops do. The Cluster Spec is a yaml document created by kops that defines in code everything that kops needs to deploy and manage your cluster. It lives in your State Store, which for AWS is in S3. If you have a Cluster Spec, you have the blueprints necessary to create a Kubernetes cluster, which is why we can create a Kubernetes cluster with only a few commands (as shown above).
If you have a working cluster built with kops and you want to view or save the Cluster Spec, this is how to do it:
kops get cluster ${CLUSTER_NAME} -o yaml --full > cluster.yml
kops get ig nodes -o yaml > nodes.yml
kops get ig masters -o yaml > master.yml
…
Merge the .yml files and put “---” between them into one large cluster_spec.yml See
cluster_spec-example.ymlThere are a number of ways to interact with kops, but here I’m highlighting a relatively unknown feature of kops – directly creating your cluster via kops create -f. This feature was designed to mirror kubectl create -f, which is used to create and configure all types of Kubernetes resources. Here, kops intakes the specification for the entire cluster in one go, infers any values that are not specified and then creates your cluster. It’s particularly useful if you are scripting or templating your cluster deployments (see cluster_spec-template.yml.j2 for some ideas about how you can start thinking about developing a template).
The better known route to creating clusters with kops is with the use of a script that wraps the command line flags included in deploy.sh. If you run deploy.sh, it will create a base Cluster Spec that you can then either deploy immediately or edit to customize your cluster configuration via kops edit cluster. Finally, you can deploy it with kops update cluster. But for those who are used to fully automatic playbooks, cookbooks, or other automated methods of Infrastructure management, what I described above is really interactive and takes a lot of operator time. If you’re doing any customization (and we do a lot), it also introduces the opportunity for error. That’s where templating and kops create -f come in.
My suggestion for developing consistent, repeatable infrastructure is to initially create clusters a few times interactively by hand and pay close attention to how the Cluster Spec responds to your changes. At that point, you’ll have a base spec that meets your needs and that you can use as a template from which to build more clusters.
Note that we configure our Cluster Spec to launch Kubernetes in a secure, private topology, which means that the masters and the nodes are in private subnets. kops creates “utility” subnets 1:1 with the private subnets. The utility subnets host the elastic load balancers (ELB) for the Kubernetes API server, the ELBs for any external services launched in Kubernetes and a bastion host, if used.
For an RO-style Kubernetes cluster created with kops, we prefer to let kops completely manage its own configuration inside of our existing VPC (except for NAT gateways). We find that letting kops have control over the networking, the security groups, and the instances allows it to optimize management of upgrades and other changes to your infrastructure. For the NAT gateways, we direct kops to use the gateways that we initially generated as part of the VPC creation because they are scalable and redundant – and pretty expensive. Even if we have more than one cluster in a VPC, we can continue to get the benefits of private topology without scaling up the associated networking costs.
If all went well with kops update cluster, within minutes you can run kops validate cluster $CLUSTER_NAME and know that your cluster is ready to go.
This configuration may seem pretty complicated. So why do we work this way?
In short, Terraform is great at laying the groundwork. It shapes the landscape and creates a place for Kubernetes to rest. kops, in turn, is purpose-built for Kubernetes. It’s predictable and fast and handles all of the cluster provisioning and better updates and management of the cluster than you could get with any other tool. When you use Terraform and kops together, then you have the best tool for the task at hand every step of the way.
Here’s a link to sample scripts, programs and specifications: https://github.com/geojaz/ro-intro-to-kops-blog
