Fairwinds | Blog

Kubernetes Mutations with Polaris: How it Works

Written by Robert Brennan | Jul 12, 2022 9:00:00 AM

For the last three years, Polaris has allowed Kubernetes users to audit their clusters and Infrastructure-as-Code for best practices. It comes with over 20 built-in checks, plus support for custom checks using JSON Schema . It can look inside a running cluster to find problematic resources, run as an Admission Controller that blocks resources with high-severity issues, or run as part of a CI/CD process to scan infrastructure-as-code.

But fixing the issues Polaris finds can be tedious. You have to look up the syntax for modifying the particular attribute that needs to change, and edit every infrastructure-as-code file that needs improving. This is especially hard if your code is spread across many repositories or teams. And new YAML files are getting created all the time, meaning there's a steady stream of work being created for whoever is responsible for enforcing best practices.

To help solve this problem, we've built a concept of Mutations into Polaris. Mutations specify precisely what needs to be done to a piece of YAML to get it to comply with best practices. Mutations can be run on Infrastructure-as-Code files, so the changes can be checked into your repository, or they can be run as a Mutating Webhook, modifying resources as they enter your Kubernetes Cluster. Learn about practical applications.

Run Polaris in multiple clusters, track results over time and integrate with Slack, Datadog, and Jira with Fairwinds Insights, software to standardize and enforce development best practices. It's free to use! Compare Polaris and Insights.

Mutation Syntax

Polaris utilizes JSON Schema to decide whether a particular resource meets best practices. For example, this is what the check to make sure hostIPC is not configured looks like:

successMessage: Host IPC is not configured
failureMessage: Host IPC should not be configured
category: Security
target: PodSpec
schema:
  '$schema': http://json-schema.org/draft-07/schema
  type: object
  properties:
    hostIPC:
      not:
        const: true

To fix an issue with hostIPC, we just need to remove that field from the YAML. So we can add a mutations block to the YAML above that looks like this:

mutations:
  - op: remove
    path: /hostIPC

The syntax here is  JSON Patch , an IETF standard for modifying JSON (or YAML) data.

It's worth noting that not every mutation will work out of the box. For instance, Liveness and Readiness probes need to be tailored to your application. But Polaris can at least make a guess, making it easier for you to fill them out without having to dig through Kubernetes documentation. To make this explicitly clear, Polaris can add a comment prompting the user to make further modifications. Here's the syntax that adds a comment to the Liveness Probe check:

mutations:
  - op: add
    path: /livenessProbe
    value: {"exec": { "command": [ "cat", "/tmp/healthy" ] }, "initialDelaySeconds": 5, "periodSeconds": 5 }
comments:
  - find: "livenessProbe:"
    comment: "TODO: Change livenessProbe setting to reflect your health endpoints"

All of this syntax is available in Polaris custom checks. So if you've already built a library of custom checks, or are planning to create some, you can add mutations while you're at it!

Running Mutations on Infrastructure-as-Code

The polaris fix command will allow you to modify Infrastructure-as-Code files so that they adhere to best practices.

For instance, if we start with this minimal configuration, copied directly from the Kubernetes docs:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: nginx-deployment
  labels:
    app: nginx
spec:
  replicas: 3
  selector:
    matchLabels:
      app: nginx
  template:
    metadata:
      labels:
        app: nginx
    spec:
      containers:
      - name: nginx
        image: nginx:1.14.2
        ports:
        - containerPort: 80

We can run polaris fix --files-path ./deploy.yaml --checks=runAsRootAllowed to get:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: nginx-deployment
  labels:
    app: nginx
spec:
  replicas: 3
  selector:
    matchLabels:
      app: nginx
  template:
    metadata:
      labels:
        app: nginx
    spec:
      containers:
      - name: nginx
        image: nginx:1.14.2
        ports:
        - containerPort: 80
        securityContext:
          runAsNonRoot: true

Note that the configuration now has a securityContext, and is set to run as non-root.

Or, if we want to see everything Polaris can do with this file, we can run polaris fix --files-path ./deploy.yaml --checks=all to get:

apiVersion: apps/v1
kind: Deployment
metadata:
  labels:
    app: nginx
  name: nginx-deployment
spec:
  replicas: 3
  selector:
    matchLabels:
      app: nginx
  template:
    metadata:
      labels:
        app: nginx
    spec:
      containers:
      - image: nginx:1.14.2
        imagePullPolicy: Always
        livenessProbe: #TODO: Change livenessProbe setting to reflect your health endpoints
          exec:
            command:
            - cat
            - /tmp/healthy
          initialDelaySeconds: 5
          periodSeconds: 5
        name: nginx
        ports:
        - containerPort: 80
        readinessProbe: #TODO: Change livenessProbe setting to reflect your health endpoints
          exec:
            command:
            - cat
            - /tmp/healthy
          initialDelaySeconds: 5
          periodSeconds: 5
        resources:
          limits:
            cpu: 100m #TODO: Set this to the amount of CPU you want to reserve for your workload
            memory: 512Mi #TODO: Set this to the amount of Memory you want to reserve for your workload
          requests:
            cpu: 100m #TODO: Set this to the amount of CPU you want to reserve for your workload
            memory: 512Mi #TODO: Set this to the amount of Memory you want to reserve for your workload
        securityContext:
          capabilities:
            drop:
            - ALL
          readOnlyRootFilesystem: true
          runAsNonRoot: true

Note that Polaris has tightened the security, as well as set some fields like health probes and resource settings. As noted above, you can see comments next to some settings that should be fine-tuned by the user, rather than just accepted as-is.

Enabling the Mutating Webhook

It can be tedious to change all your infrastructure-as-code files, even with the polaris fix command helping to fill in the gaps. This is especially true as new deployment files get created - it can be hard to keep up!

For some mutations, it's easier to simply modify objects as they enter the cluster. One good example of this is imagePullPolicy - Polaris recommends setting this to Always, and you can be confident that nothing will break if you do.

To enable the Mutating Webhook, you can install Polaris via the Helm Chart . Be sure to add the following configuration to enable the webhook:

webhook:
  enable: true
mutate: true

By default, the only check that is enabled for mutations is pullPolicyNotAlways. If you'd like to enable other mutations, you can edit the mutations section of your Polaris configuration:

webhook:
  enableMutation: true
  mutatingRules:
  - cpuLimitsMissing
  - cpuRequestsMissing
  - dangerousCapabilities
  - deploymentMissingReplicas
  - hostIPCSet
  - hostNetworkSet
  - hostPIDSet
  - insecureCapabilities
  - livenessProbeMissing
  - memoryLimitsMissing
  - memoryRequestsMissing
  - notReadOnlyRootFilesystem
  - priorityClassNotSet
  - pullPolicyNotAlways
  - readinessProbeMissing
  - runAsPrivileged
  - runAsRootAllowed

Future Improvements

We're really excited about the progress we've made in implementing mutations in Polaris, but there are some limitations. In the coming months, we'll be working to improve this functionality in a few ways:

  • Ability to modify Helm templates: Currently the polaris fix command only works on raw YAML files, not on things like Helm templates. We have some ideas on how to make polaris fix work in a wider variety of contexts.

  • Preserve comments and formatting: The polaris fix command deserializes, modifies, and reserializes YAML, and some cosmetic features get lost in translation. For example, comments are removed, and field order might change. We're working on making this experience better.

  • More mutations: Currently, not every check has a mutation. The current JSON Patch syntax has some limitations, e.g. when modifying the contents of an array. We also would like to support mutations for multi-resource checks, e.g. creating a PDB if none exists.

If you give the current implementation a try, we'd love to hear your feedback! You can find us in the Fairwinds Community Slack, our Open Source User Group, or on GitHub.

Resources