Yet Another Kubernetes Intro - Part 5 - Deployments

It is time to have a look at the last pod management related resource type in Kubernetes, the Deployment. In part 3 I talked about using ReplicaSets to run multiple pods in our cluster, which is all nice and dandy until you want to update the service you are running. With pure RSs, this can get a bit complicated…

When rolling out a new version, we generally want to do this without downtime. This means replacing a few pods at the time, so that there are always pods up and running, as well as making sure that the new version is behaving as it should, and not taking down the whole system. Doing this manually is a bit tedious, and probably quite error prone. That is why K8s has the Deployment resource to help out.

K8s deployments support 2 types of update strategies, Recreate and RollingUpdate. Recreate just tears down the existing pods and puts up new ones, leaving us with downtime while the new pods come online. Using a RollingUpdate strategy on the other hand, allows us to roll out an update of a pod in the cluster, replacing a few pods at the time. Because of this, a rolling update is preferred in most situations.

But before we go into how we set one of these up, let’s have a quick look at how they work, at a high level.

Deployments

When you add a deployment resource to your cluster, it will create a ReplicaSet to manage the pods that the deployment defines. And the RS in turn, schedules the pods. So it is pretty much just a nice layer on top of ReplicaSets. However, the “magic” comes in when you deploy a new version of your deployment, with the update strategy set to perform a rolling update.

Note: Not all changes to your deployment will cause an update. For example, changing the replica count does not require an update. It will just cause the RS to be reconfigured. But every time you change your pod template, it will cause an update.

Rolling updates

When making a change that causes an update to happen, the first thing that happens is that the deployment (or rather the controller managing the deployment, but that is semantics) will set up a new RS to handle the new, updated pods. It then scales up that RS to create one or more pods depending on the configuration and existing replica count.

Once the first group of new pods have come online, the previous RS is scaled down to compensate for the newly created pods. Then the new RS is scaled up, and the old one scaled down. This dance then goes on until the old RS is scaled to 0, and the new RS has been scaled up to the full number of requested replicas. Along the way, the deployment can be requested to monitor the new pods to make sure that the new pods stay online for at least X amount of time before being considered available. This makes sure that any configuration problems, or container issues that would cause a pod to fail, doesn’t replicate throughout your cluster. Otherwise it would be very easy to take down your application with a small config mistake for example.

Configuration

The rolling update can be configured in a few ways. First of all, as just mentioned, we can have the deployment monitor the new pods for a while before carrying on with the roll out. This is a nice little insurance to make sure that we aren’t rolling out a botched pod that will cause the application to fail. Or in the words of the infinitely wise @DevOps_Borat

DevOps Borat

On top of that, we can define how many pods that can be allowed to be unavailable at any point. Either in fixed numbers, or a percentage of the total desired replica count. So for example, if we tell it that we want a maximum of 3 pods unavailable, it means that even if we have expressed a desire to have 12 pods, during the update, the cluster is allowed to go down as low as 9.

We can also define a maximum “surge”. This tells the deployment how many extra pods that can be allowed. For example, setting it to 3 with a desired state of 12, means that during the update, the cluster can scale up to a maximum of 15 pods.

Using these two configuration values, we can configure how fast the update is allowed to roll through the cluster. The reason for these for having these settings is that since the deployment uses 2 ReplicaSets, scaling up one and down the other, there is no way to make sure that there is no overlap of pods between the sets. So this allows there to be a certain fluctuation in the pod count of the combined RS.

Defining a deployment

As with most K8s resources, creating the deployment definition in a YAML file is recommended. This gives us a way to version it using source control etc.

A spec looks something like this

apiVersion: apps/v1
kind: Deployment
metadata:
  name: hello-world
spec:
  replicas: 3
  selector:
    matchLabels:
      app: hello-world
  strategy:
    type: RollingUpdate
    rollingUpdate:
      maxUnavailable: 1
      maxSurge: 1
  minReadySeconds: 10
  template:
    metadata:
      labels:
        app: hello-world
    spec:
      containers:
      - name: hello-world
        image: zerokoll.azurecr.io/zerokoll/helloworld:v1

As you can see, the deployment spec is pretty similar to a RS definition, which kind of makes sense as it is pretty much just a controlling layer on top of an RS.

The above example defines a deployment called hello-world that should have 3 replicas. It should also use a rolling update strategy with maximum deviation of 1 pod both up and down from the desired 3, allowing the pods to be ready for 10 seconds before continuing the rollout. And as it spins up new pods, it uses the defined template.

That’s all there is to it! And after adding it to the cluster like this

kubectl apply -f .\hello-world-deployment.yml

we can see that it creates 1 RS and 3 pods

kubectl get all

NAME                              READY   STATUS    RESTARTS   AGE
pod/hello-world-5f4c77d94-hphp5   1/1     Running   0          68s
pod/hello-world-5f4c77d94-jxbht   1/1     Running   0          68s
pod/hello-world-5f4c77d94-mvltd   1/1     Running   0          68s


NAME                 TYPE        CLUSTER-IP   EXTERNAL-IP   PORT(S)   AGE
service/kubernetes   ClusterIP   10.96.0.1    <none>        443/TCP   6d8h


NAME                          READY   UP-TO-DATE   AVAILABLE   AGE
deployment.apps/hello-world   3/3     3            3           68s

NAME                                    DESIRED   CURRENT   READY   AGE
replicaset.apps/hello-world-5f4c77d94   3         3         3       68s

Performing a rolling update

One the deployment is up and running, we can do a rolling update by simply changing the pod template to use another version like this

…
spec:
  containers:
  - name: hello-world
    image: zerokoll.azurecr.io/zerokoll/helloworld:v2

and then re-deploy the deployment using kubectl apply. While the update is rolling out, we can monitor it in a couple of different ways. First of all, we can obviously have a look at the resources in the cluster

kubectl get all

NAME                               READY   STATUS        RESTARTS   AGE
pod/hello-world-5f4c77d94-hphp5    0/1     Terminating   0          5m28s
pod/hello-world-5f4c77d94-jxbht    1/1     Running       0          5m28s
pod/hello-world-5f4c77d94-mvltd    1/1     Running       0          5m28s
pod/hello-world-6669969979-pxf64   1/1     Running       0          2s
pod/hello-world-6669969979-rdmqs   1/1     Running       0          2s


NAME                 TYPE        CLUSTER-IP   EXTERNAL-IP   PORT(S)   AGE
service/kubernetes   ClusterIP   10.96.0.1    <none>        443/TCP   6d8h


NAME                          READY   UP-TO-DATE   AVAILABLE   AGE
deployment.apps/hello-world   4/3     2            2           5m28s

NAME                                     DESIRED   CURRENT   READY   AGE
replicaset.apps/hello-world-5f4c77d94    2         2         2       5m28s
replicaset.apps/hello-world-6669969979   2         2         2       2s

As you can see from the output, doing the update has caused a second RS to be created. You can also see that the new RS (hello-world-6669969979) has already scaled up to 2 pods, and the original one (hello-world-5f4c77d94) has started scaling down by one.

Once the update has rolled through, the output looks like this

kubectl get all

NAME                               READY   STATUS    RESTARTS   AGE
pod/hello-world-6669969979-4zf5n   1/1     Running   0          2m3s
pod/hello-world-6669969979-pxf64   1/1     Running   0          2m14s
pod/hello-world-6669969979-rdmqs   1/1     Running   0          2m14s


NAME                 TYPE        CLUSTER-IP   EXTERNAL-IP   PORT(S)   AGE
service/kubernetes   ClusterIP   10.96.0.1    <none>        443/TCP   6d8h


NAME                          READY   UP-TO-DATE   AVAILABLE   AGE
deployment.apps/hello-world   3/3     3            3           7m40s

NAME                                     DESIRED   CURRENT   READY   AGE
replicaset.apps/hello-world-5f4c77d94    0         0         0       7m40s
replicaset.apps/hello-world-6669969979   3         3         3       2m14s

The new RS is running at full speed and the old one has scaled down to 0. However, the old RS is still in place, and isn’t removed. Instead, the deployment can keep track of the ReplicaSets, and pods, from a specific deployment, by adding a label to them called pod-template-hash. The value for this label is a hash of the pod template. This allows the deployment to make sure that the created label selectors for the ReplicaSets aren’t overlapping.

Besides just looking at the resources in the cluster, you can also monitor the rolling update by running

kubectl rollout status deployment.v1.apps/hello-world

This returns locks up your terminal during the update, keeping track of the progress as it is rolling through the cluster. The output looks something like this

Waiting for deployment "hello-world" rollout to finish: 2 out of 3 new replicas have been updated...
Waiting for deployment "hello-world" rollout to finish: 2 out of 3 new replicas have been updated...
Waiting for deployment "hello-world" rollout to finish: 2 out of 3 new replicas have been updated...
Waiting for deployment "hello-world" rollout to finish: 2 out of 3 new replicas have been updated...
Waiting for deployment "hello-world" rollout to finish: 2 of 3 updated replicas are available...
Waiting for deployment "hello-world" rollout to finish: 2 of 3 updated replicas are available...
deployment "hello-world" successfully rolled out

Finally, you can always use kubectl get and kubectl describe to get more information about your deployment. For example, running kubectl describe for your deployment will get you the following information.

kubectl describe deployment/hello-world

Name:                   hello-world
Namespace:              default
CreationTimestamp:      Tue, 28 Jan 2020 23:36:20 +0100
Labels:                 <none>
Annotations:            deployment.kubernetes.io/revision: 2
                        kubectl.kubernetes.io/last-applied-configuration:
                          {"apiVersion":"apps/v1","kind":"Deployment","metadata":{"annotations":{},"name":"hello-world","namespace":"default"},"spec":{"minReadySeco..."
Selector:               app=hello-world
Replicas:               3 desired | 3 updated | 3 total | 3 available | 0 unavailable
StrategyType:           RollingUpdate
MinReadySeconds:        10
RollingUpdateStrategy:  1 max unavailable, 1 max surge
Pod Template:
  Labels:  app=hello-world
  Containers:
   hello-world:
    Image:        zerokoll.azurecr.io/zerokoll/helloworld:v2
    Port:         <none>
    Host Port:    <none>
    Environment:  <none>
    Mounts:       <none>
  Volumes:        <none>
Conditions:
  Type           Status  Reason
  ----           ------  ------
  Available      True    MinimumReplicasAvailable
  Progressing    True    NewReplicaSetAvailable
OldReplicaSets:  <none>
NewReplicaSet:   hello-world-6669969979 (3/3 replicas created)
Events:
  Type    Reason             Age    From                   Message
  ----    ------             ----   ----                   -------
  Normal  ScalingReplicaSet  3d12h  deployment-controller  Scaled up replica set hello-world-5f4c77d94 to 3
  Normal  ScalingReplicaSet  3d12h  deployment-controller  Scaled up replica set hello-world-6669969979 to 1
  Normal  ScalingReplicaSet  3d12h  deployment-controller  Scaled down replica set hello-world-5f4c77d94 to 2
  Normal  ScalingReplicaSet  3d12h  deployment-controller  Scaled up replica set hello-world-6669969979 to 2
  Normal  ScalingReplicaSet  3d12h  deployment-controller  Scaled down replica set hello-world-5f4c77d94 to 0
  Normal  ScalingReplicaSet  3d12h  deployment-controller  Scaled up replica set hello-world-6669969979 to 3

In this massive output you can see the pod template, the current status of the deployment, the replica count, the name of the old and new RS etc. And you can also see the historical events for the deployment to keep track of what has happened over time.

Revision history

As you make changes to the deployments you are using, and rolling out new version, a history of the old version is recorded. This allows us to roll back a failed update in a very simple way. Just remember that the revision history is limited. But on the other hand, in most cases you only rollback the latest version anyway.

To look at the history of my deployment, you can run the following command

kubectl rollout history deployment.v1.apps/hello-world

deployment.apps/hello-world
REVISION  CHANGE-CAUSE
1         <none>
2         <none>

As you can see, in this case there are 2 revisions. The first one was the initial deployment, and the second one is when we upgraded to version 2. However, the CHANGE-CAUSE column is pretty useless right now. This is because I forgot to add the parameter --record then applying the deployment spec. Including the --record parameter like this

kubectl apply -f=hello-world-deployment.yaml --record=true

would have caused the change to be recorded in a resource annotation called kubernetes.io/change-cause. And this in turn would have changed the empty output from the kubectl rollout history command above, to something like this

deployment.apps/hello-world
REVISION  CHANGE-CAUSE
1         kubectl.exe apply --filename=.\hello-world-deployment.yml --record=true
2         kubectl.exe apply --filename=.\hello-world-deployment.yml --record=true

Not a whole lot better as such. But at least it is some information about what caused the changed. And if you had made manual updates to your deployment using the CLI instead of a YAML file, the commands you ran would show up in the change cause. For example, running the following command to roll out the v1 image again

kubectl set image deployment/hello-world hello-world=zerokoll.azurecr.io/zerokoll/helloworld:v1 --record

causes the rollout history to look like this

kubectl rollout history deployment.v1.apps/hello-world

deployment.apps/hello-world
REVISION  CHANGE-CAUSE
2         kubectl.exe apply --filename=.\hello-world-deployment.yml --record=true
3         kubectl.exe set image deployment/hello-world hello-world=zerokoll.azurecr.io/zerokoll/helloworld:v1 --record=true

This is obviously a bit more useful, but I still recommend using YAML files!

Note: As you can see, the full revision history is not kept. However, the number of revisions to keep can be defined in the deployment by setting the .spec.revisionHistoryLimit property for the deployment.

If you want more information about a specific revision, you can run

kubectl rollout history deployment.v1.apps/hello-world --revision=2

deployment.apps/hello-world with revision #2
Pod Template:
  Labels:       app=hello-world
        pod-template-hash=6669969979
  Annotations:  kubernetes.io/change-cause: kubectl.exe apply --filename=.\hello-world-deployment.yml --record=true
  Containers:
   hello-world:
    Image:      zerokoll.azurecr.io/zerokoll/helloworld:v2
    Port:       <none>
    Host Port:  <none>
    Environment:        <none>
    Mounts:     <none>
  Volumes:      <none>

This shows the pod template that was being used for that specific revision.

And if you ever want to roll back to a previous revision, you can easily do this by running

kubectl rollout undo deployment.v1.apps/hello-world

This will roll back the last revision. And if you add --to-revision=<REVISION ID>, you can roll back to a specific revision.

Managing roll outs

During the roll out a new version, you can pause the rollout by running

kubectl rollout pause deployment.v1.apps/hello-world

This will pause the rollout and allow you to make manual changes to the deployment using kubectl. And when you are done changing the deployment that you are currently in the process of rolling out, you can run

kubectl rollout resume deployment.v1.apps/hello-world

to resume the rollout with the new configuration.

However, as mentioned several times before, it is better to make your changes in a YAML file so that it can be recorded in source control. So if you find that you want to make a change to a deployment in the middle of a rollout, you can just update the deployment spec and apply it again. This will cause a “rollover”. Basically that means that the deployment will stop the current rollout, and spin up yet another RS. It will then create new pods in the new RS, replacing both the original, and semi rolled out pods, with new pods in this new RS.

I think that is pretty much all had to say about deployments this time. To sum it up, they are the way you should define your replicated pods in your cluster. They allow you to create replicated pods using an RS, but adding a nice way to roll out updates to your cluster.

The next part is available here, and talks about configuration.

zerokoll

Chris

Developer-Badass-as-a-Service at your service