Relying on cached versions of a Docker image can become a security vulnerability. By default, an image will be pulled if it isn't already cached on the node attempting to run it. This can result in variations in images that are running per node, or potentially provide a way to gain access to an image without having direct access to the ImagePullSecret.
For example, you might deliberately override a particular tag, and want your workload to pull in those changes. More specifically, many projects will release version 1.2.3 with the image tags 1, 1.2, and 1.2.3. In the Kubernetes config, you can just specify tag 1.2, so that when 1.2.4 is released with a security patch, it automatically flows in the next time that pod is rescheduled or restarted. But if you don’t specify imagePullPolicy: Always, the cluster will continue to use the cached 1.2 tag (which still points to 1.2.3).
To overcome this, Kubernetes uses imagePullPolicy. When set to always (imagePullPolicy: Always), “every time the kubelet launches a container, the kubelet queries the container image registry to resolve the name to an image digest. If the kubelet has a container image with that exact digest cached locally, the kubelet uses its cached image; otherwise, the kubelet downloads (pulls) the image with the resolved digest, and uses that image to launch the container.” source: Kubernetes documentation.
Avoid problems with images by ensuring that imagePullPolicy is set to Always. Your team can of course manually check for this policy, but it takes time and effort away from your app. Instead, use configuration validation software like Fairwinds Insights to automatically check your imagePullPolicy. Fairwinds Insights triggers a warning when a tag is not specified or if imagePullPolicy is not set to Always. Your team can then easily fix the manifest helping to avoid problems with your application (downtime, bugs, etc).
Fairwinds Insights can also run throughout your entire deployment process so that from CI/CD through production, imagePullPolicy is always specified. And while this blog post discusses how to identify offending workloads, Fairwinds Insights also helps to prevent those workloads from getting into your cluster in the first place by scanning Infrastructure-as-Code in CI, and running as an Admission Controller in front of the Kuberenetes API.
You can try it for free by creating an account, creating a cluster and installing the agent via the Helm chart. Once the Fairwinds Insights agent is installed you’ll get results in 5-10 minutes. You can easily check for over permissioned containers as well as other reliability issues like missing readiness or liveness probes.
