Dead Letters

Overview

In the day to day operations of a high volume throughput system such as Constellations, it is inevitable that errors will occur. Errors may be retried, but there will come a point where it no longer makes sense to keep retrying. In the context of RabbitMQ and queuing in general, this is where Dead Letter Messages come in.

Dead Letter Process

A service that subscribes to a queue will reject messages in the event of an exception that causes the message not to be fully processed. This incrememnts delivery count on the message in RabbitMQ.

Once a message is processed to the configured delivery limit of the policy, the message is considered dead lettered and normally this would mean the loss of a message.

In Constellations however, a dead letter exchange has been declared. Messages sent to this exchange are then routed to dead letter queue, which by default is named 'constellations-dlq'.

A RabbitMQ policy contains various settings that configure a queue's behavior. The key setting here is the delivery-limit, which specifies how many times it is retried before dead lettered. There are also settings to bind the exchange to each of the services queue's.

In Kubernetes, this policy is represented as an Object defined by the rabbitmq Kubernetes operator. These objects, such as Policy, Queues, Exchanges are declared using helm charts.

The policy matches all queues in the associated RabbitMQ resource. The only exclusion is itself. This means that all messages will end up in this dead letter queue.

Scan Controller

The Scan Controller consumes this dead letter queue. The purpose of this is to 'close the loop' of the underlying scan. It does this by parsing the message and attempting to find the Scan ID.

If found, the scan will be marked as errored, otherwise the dead letter is dropped. If an exception is raised during this process, it will be retried indefinitely.

Furthermore, a critical log is raised at this point. This log contains a wealth of infomation, such as dead letter reason as well as associated scan and queue.

For example, if the scans source container is deleted this may mean parts of the process will be retried. Eventually part of the scan will reach the retry limit and the dead letter will be sent to the Scan Controller, marking this scan as failed.