Graceful Shutdown

Overview

Kubernetes pods are designed to be transient. Therefore, any work carried out by a pod is subject to disruption. Implementation choices are made within Constellations to ensure that when a pod is terminated, it can shutdown without issues.

How it works

Each of the services that read messages from a queue implement a dotnet IHostedService, which is responsible for subscribing to a queue and processing the messages. When a shutdown signal is received, classes that implement this interface will be instructed to stop. In addition to this, cancellation tokens are cancelled.

Since RabbitMQ messages have a delivery limit imposed on them, it is important that they finish processing before a service shuts down.

In order to accomplish this, the service processing the message must keep track of the outstanding thread. When a signal to shutdown is received, the service will wait for this thread to complete. If the service takes too long to complete however, the graceful period will expire and the pod will be terminated regardless. The services use the default dotnet shutdown timeout for this, which is 5 seconds.

In the event that a shutdown signal is received at startup, any messages received will be immediately rejected.

This applies to the following services

• Scan Controller
• Page Scanner
• Scan Preprocessor
• CDR Enabler
• Event Collation

Event Projection has a Cosmos DB change feed processor which it hosts and is implemented in a similar fashion using IHostedService. When it is instructed to stop, the service will use the Azure Cosmos DB dotnet library to signal the change feed processor to stop listen for new changes.