Retries

Glasswall Halo is comprised of multiple services that communicate using asynchronous messages rather than HTTP. For these services, we have made considerations about how the platform handles failure in the event of an error when processing a message.

Synchronous API and Asynchronous API

The APIs subscribe to response queues from the Glasswall Engine, however in the event of failure an error response will be returned to the HTTP client.

Engine Service

The architecture is Request/Response using RabbitMQ reply-to queues. This means in the event of failure, when handling a message we still send a response message containing the reason for failure.

Report Aggregator

The report aggregator behaves as a standard message consumer and extra functionality was introduced to enable message retrying. Two configuration options are present on the service.

RetryMessages is a bool defaulting to true that enables the functionality to retry messages. If turned off all failed messages are immediately put in a dead letter queue.

MessageTTL is an integer that represents the number of seconds a message can be in the queue. This is to avoid messages backing up because of a failed message infinitely retrying. When the time to live has elapsed for a message, it is put in the dead letter queue. The code default is 30 seconds.

The Report Aggregator's functionality is not critical to clients receiving their files and is used to produce aggregated report data. Due to this the dead lettering of a report aggregator message is not deemed a Critical failure and dead letters are tolerated (only in extreme cases).