Working with Dead-letter Queues in AWS SQS and C#

What happens when messages sent via a messaging queue fail to be processed? Is it skipped? Is the data or intention encapsulated in that message lost? This is the problem that dead-letter queues solve. Dead-letter queues are like any other, with the key differentiator being that they act as a "store" for unprocessed messages to be subsequently processed. After a specified number of retries (SQS defaults to 2), unsuccessfully processed messages are sent to the queue.

This article will explore implementing a dead-letter queue with AWS and C#.

Please note this article is a sequel to "Asynchronous Communication In Microservices Via C# and AWS SQS." Kindly go through that and return to proceed.

The code for this article is available here.

In the previous blog post, we created a messaging queue, and published and consumed messages from it. Right now, we would add a dead-letter queue to our code.

Let's get started!

Create a new queue on your AWS SQS dashboard.

Give it the name orders-dlq As part of the configurations while creating the queue, enable "Redrive allow policy," and select your main (orders) queue from the dropdown as the source queue.
Edit your main orders queue. Enable "Dead-letter queue" and set the newly created orders-dlq as the dead-letter queue.

And that's it!

To test that this works:

Publish a new message to our main orders queue by making a request to our Orders.Api

Simulate an error by throwing an exception that will be handled in the catch block. And subsequently placed in the dead-letter queue.

 try
 {
     throw new ApplicationException("unable to process message");
     await _mediator.Send(deserializedMessage, stoppingToken);

     await _amazonSqs.DeleteMessageAsync(new DeleteMessageRequest
     {
         QueueUrl = queueUrl.QueueUrl,
         ReceiptHandle = message.ReceiptHandle
     }, stoppingToken);
 }

Run the consumer to pull from the orders queue.
Confirm the presence of this message in our orders-dlq on the AWS SQS dashboard. As we can see, the unprocessed message gets inserted into our dead-letter queue.
Trigger a reprocessing by clicking the "start DLQ redrive" button. We can remove the exception we intentionally threw from our consumer app this time. The redrive would transport these unprocessed messages from orders-dlq queue to our main orders queue to be reprocessed.
Rerun the Consumer app. This time, we have no messages in our orders or orders-dlq meaning our messages are now being processed. We can also verify this from the output on our console.