Istio recently released version 1.1.1 of their popular service mesh with several changes and improvements. Without a doubt you should likely upgrade from earlier versions to take advantage of these improvements in your Kubernetes clusters. But upgrading isn’t trivial - Istio’s service mesh affects nearly every aspect of cluster communication! In addition to upgrading Istio’s control plane, every sidecar must get updated. In the following sections I’ll outline some of the major changes that should be planned for and the steps you can take to upgrade Istio in your cluster.
If you’re a helm user the biggest change to be aware of is that CRDs (Custom Resource Definitions) are now installed by a separate helm chart called istio-init. This new chart should be installed before the control plane is upgraded or the upgrade may fail. To simplify installation configuration profiles have been added, which are helm values files to use for differing needs.
In prior releases readiness and liveness probes have been a pain point when mTLS is enabled because the kubelet did not have istio issued certificates to communicate with the services. This made the use of http and tcp checks impossible. Version 1.1 introduces a probe rewrite that can redirect liveness probe requests to the pilot agent. The pilot agent then redirects the request to the pod, allowing a means for the kubelet to send requests without the istio issued certificates. In other words, you can use http/tcp liveness and readiness probes again!
If you’re using annotations to override mTLS policy, I’m sorry to say this won’t work going forward. This has been replaced with a new method of overriding with authentication policies and destination rules. The use of Policy and DestinationRule objects provide more flexibility but you’ll need to keep this in mind during your upgrade.
Sidecars
In the past it has been difficult to configure proxy sidecars beyond the out of box configuration. A new sidecar resource has been introduced that lets you configure the ports and protocols proxies will accept traffic from(both ingress and egress). This resource is dynamic in a way that you can specify subsets of workloads in a namespace to apply the configuration to.
Ingress
The most exciting change in my opinion is the deprecation of the istio ingress. This paves the way for more native Kubernetes support and with this change you can use optional support of the Kubernetes ingress. This allows you to use ingress objects to define ingress points and you can use a new controller that dynamically loads and rotates external certificates, including LetsEncrypt.
These changes and a long list of others can be reviewed in detail at the Istio 1.1 Release Notes page.
Upgrading is generally done by backing up the istio CRDs, installing the new istio-init helm chart, upgrading the control plane with helm, and then upgrading all of the sidecars.
Back Up Your CRDs!
Backing up the existing CRDs is essential in case something goes wrong in the install. Kubectl makes this very simple to do:
Install the Istio initializer Chart
Once you’ve backed up the CRDs, it’s time to start installing the istio charts. These are all available in the Istio repo, so we need to clone it to our workstation first with the command below. All of the applicable helm charts will be in this repo in the install/kubernetes/helm subdirectory.
If you’re using Reckoner, you’re in luck because it can install charts from a git repo. Reckoner is a command line helper for Helm developed by Fairwinds that uses a YAML syntax to install and manage multiple Helm charts in a single file.
The first chart we need to install is the Istio initializer chart called istio-init. This manages all of the CRDs that istio requires. It’s good practice to install this chart to the same namespace you have Istio installed in. In this example, we have Istio in the istio-system namespace.
helm install install/kubernetes/helm/istio-init --name istio-init --namespaceistio-system
If you’re using Reckoner this snippet can be added to your course.yml:
Then, the chart can be installed with the command reckoner plot course.yml --heading istio-init.
After the chart is installed several jobs are run to create the CRDs. You’ll want to make sure they all complete before proceeding. You can monitor the status with watchand kubectl:watch -d "kubectl get job -n istio-system | grep istio-init-crd"
Upgrade the Control PlaneThe next step in the process is to upgrade the Istio control plane. Before you proceed, there are a couple of new Helm chart values to be aware of:
global.envoyStatsd.enabled. The built in statsd collector was removed in favor of using your own. You can still integrate with your own by setting this value to true.global.istio-egressgateway.enabled. By default the egress gateway is now disabled. If you’re using this you need to enable it by setting the value to true.
Once your chart values are configured you’re ready to upgrade the release with helm:helm upgrade istio install/kubernetes/helm/istio --namespace istio-system -f <path-to-values>.yml
Here’s an example snippet you can use with Reckoner to upgrade Istio. The chart can be upgraded using the command reckoner plot course.yml --heading istio.
Upgrade the Sidecars
Once you upgrade the control plane, all applications using istio will still be using the old version of the sidecar proxy. As pods get reloaded they will get the updated sidecar. This is perfectly acceptable to receive the update through natural pod restarts, but you may want to consider triggering a rolling update of all your deployments to get them all on the same sidecar version. Istio recommends triggering the update by editing a trivial field in each deployment spec that has istio enabled. This snippet accomplishes that by adding an annotation to each deployment:
Enjoy your upgraded installation of Istio with improved support for liveness/readiness probes, updated certificate handling, and greater flexibility in the Kubernetes ecosystem!
