As we outlined in our first post in this series, there are some things that you should simply never, ever do in Kubernetes. Corey Quinn, founder of Screaming in the Cloud and Chief Cloud Economist at The Duckbill Group, spoke with Kendall Miller, President of Fairwinds, and Stevie Caldwell, Senior Site Reliability Engineer at Fairwinds about a few things that development and operations teams should never ever do in Kubernetes if they want to get the most out of the leading container orchestrator. Here is some essential Kubernetes security information to keep in mind.
(Remember, this is never should you ever, so the headlines might seem a bit wrong, a bit obvious, or maybe even surprising!)
This one actually generated more conversation in our webinar than you may have expected. Many people aren’t aware that you can run a container not as root. It's a very common practice to run as root, because for practical reasons many containers run as root by default. This is because it allows easier debugging, but it’s still best to avoid running as root. Containers that run as root usually have a lot more permissions than their workload requires, which could help an attacker if your container were compromised. This may seem obvious, but giving over privileged access to anything can lead to over privileged behaviors in your cluster. We strongly recommend that you not run your container as root.
Should you share your certificate authority (CA) key with the CTO? There's a school of thought that it’s important to give the CTO “break glass access” to stuff like this in case they absolutely need to. That's valuable. You want to be able to not have individual sysadmins hold the company hostage. On the other side of it, though, if they can’t touch something, that means it's way harder for them to break it, so they are forced to do it indirectly. Giving access to things like that is often not worth doing, not because they would do anything nefarious, but because it expands who might be responsible for a problem if something goes wrong. Chances are, your CTO doesn’t need the CA key. If something goes wrong, it’s easier to have a smaller group control the CA key, so you know who can quickly review the problem and resolve it.
Let's talk about storing all your secrets in plain text in Github. Never should you ever do that. That’s because bad actors are looking for people storing credentials in Github. If you check unencrypted credentials into a repository, even in a constrained test environment, someone else is going to find those, potentially spin up a Kubernetes cluster, and then get it mining Bitcoin.
It's tempting to check your Kubernetes Secrets into your infrastructure-as-code repository so your builds are reproducible. But if you care about security, don't. Once checked in, your Secrets are permanently exposed to anyone with access to your Git repository.
Let's talk about hard coding IPs and ignoring internal DNS. You should never ever do that. And why not? We’ve seen this happen at Fairwinds — one client came to us with a list of IP addresses in an Excel spreadsheet. They had hard coded cluster IPs into their applications. As most people know, and some people don't, your pod IPs change and your cluster IPs change. You can’t rely on that, and Kubernetes has those things built into it to make your life easier. It has DNS built into it, so use the DNS name for the services. Everything in operations is a trade off, and as long as you know what you’re doing, all of these things are things you can do in Kubernetes. It’s just really easy to mess things up.
