Cloud Computing

Saying throughput improve and useless letter queue redrive assist for Amazon SQS FIFO queues

Spread the love


Voiced by Polly

With Amazon Easy Queue Service (Amazon SQS), you may ship, retailer, and obtain messages between software program parts at any quantity. Right this moment, Amazon SQS has launched two new capabilities for first-in, first-out (FIFO) queues:

  • Most throughput has been elevated as much as 70,000 transactions per second (TPS) per API motion in chosen AWS Areas, supporting sending or receiving as much as 700,000 messages per second with batching.
  • Lifeless letter queue (DLQ) redrive assist to deal with messages that aren’t consumed after a selected variety of retries in a manner just like what was already accessible for normal queues.

Let’s take a extra in-depth take a look at how these work in observe.

FIFO queues throughput improve as much as 70K TPS
FIFO queues are designed for purposes that require messages to be processed precisely as soon as and within the order through which they’re despatched. Whereas commonplace queues have a vast throughput, FIFO queues have an higher quota within the variety of TPS per API motion.

Normal and FIFO queues assist batch actions that may ship and obtain as much as 10 messages with a single API name (as much as a most whole payload of 256 KB). Which means a FIFO queue can course of as much as 10 instances extra messages per second than its most throughput.

At launch in 2016, FIFO queues supported as much as 300 TPS per API motion (3,000 messages per second with batching). This was sufficient for a lot of use circumstances, however some clients requested for extra throughput.

With excessive throughput mode launched in 2021, FIFO queues launched a tenfold improve of the utmost throughput and will course of as much as 3,000 TPS per API motion, relying on the Area. One yr later, that quota was doubled to as much as 6,000 TPS per API motion.

This yr, Amazon SQS has already elevated FIFO queue throughput quota two instances, to as much as 9,000 TPS per API motion in August and as much as 18,000 TPS per API motion in October (relying on the Area).

Right this moment, the Amazon SQS group has been capable of improve the FIFO queue throughput quota once more, permitting you to course of as much as 70,000 TPS per API motion (as much as 700,000 messages per second with batching) within the US East (N. Virginia), US West (Oregon), and Europe (Eire) Areas. That is greater than 200 instances the utmost throughput at launch.

DLQ redrive assist for FIFO queues
With Amazon SQS, messages that aren’t consumed after a selected variety of retries can robotically be moved to a DLQ. There, messages could be analyzed to know the explanation why they haven’t been processed accurately. Typically there’s a bug or a misconfiguration within the client utility. Different instances the messages comprise invalid knowledge from the supply purposes that must be mounted to permit the messages to be processed once more.

Both manner, you may outline a plan to reprocess these messages. For instance, you may repair the buyer utility and redrive all messages to the supply queue. Or you may create a devoted queue the place a customized utility receives the messages, fixes their content material, after which sends them to the supply queue.

To simplify shifting the messages again to the supply queue or to a special queue, Amazon SQS lets you create a redrive process. Redrive duties are already accessible for normal queues. Beginning as we speak, you may also begin a redrive process for FIFO queues.

Utilizing the Amazon SQS console, I create a primary queue (my-dlq.fifo) for use as a DLQ. To redrive messages again to the supply FIFO queue, the queue kind should match, so that is additionally a FIFO queue.

Then, I create a supply FIFO queue (my-source-queue.fifo) to deal with messages as common. After I create the supply queue, I configure the primary queue (my-dlq.fifo) because the DLQ and specify 3 because the Most receives situation beneath which messages are moved from the supply queue to the DLQ.

Console screenshot.

When a message has been acquired by a client for greater than the variety of instances specified by this situation, Amazon SQS strikes the message to the DLQ. The unique message ID is retained and can be utilized to uniquely monitor the message.

To check this setup, I exploit the console to ship a message to the supply queue. Then, I exploit the AWS Command Line Interface (AWS CLI) to obtain the message a number of instances with out deleting it.

aws sqs receive-message --queue-url https://sqs.eu-west-1.amazonaws.com/123412341234/my-source-queue.fifo

{
    "Messages": [
        {
            "MessageId": "ef2f1c72-4bfe-4093-a451-03fe2dbd4d0f",
            "ReceiptHandle": "...",
            "MD5OfBody": "0f445a578fbcb0c06ca8aeb90a36fcfb",
            "Body": "My important message."
        }
    ]
}

To obtain the identical message greater than as soon as, I watch for the time specified within the queue visibility timeout to move (30 seconds by default).

After the third time, the message shouldn’t be within the supply queue as a result of it has been moved to the DLQ. When I attempt to obtain messages from the supply queue, the record is empty.

aws sqs receive-message --queue-url https://sqs.eu-west-1.amazonaws.com/123412341234/my-source-queue.fifo

{
    "Messages": []
}

To verify that the message has been moved, I ballot the DLQ to see if the message is there.

aws sqs receive-message --queue-url https://sqs.eu-west-1.amazonaws.com/123412341234/my-dlq.fifo  

{
    "Messages": [
        {
            "MessageId": "ef2f1c72-4bfe-4093-a451-03fe2dbd4d0f",
            "ReceiptHandle": "...",
            "MD5OfBody": "0f445a578fbcb0c06ca8aeb90a36fcfb",
            "Body": "My important message."
        }
    ]
}

Now that the message is within the DLQ, I can examine why the message has not been processed (nicely, I do know the explanation this time) and determine whether or not to redrive messages from the DLQ utilizing the Amazon SQS console or the brand new redrive API that was launched a couple of months in the past. For this instance, I exploit the console. Again on the Amazon SQS console, I choose the DLQ queue and select Begin DLQ redrive.

In Redrive configuration, I select to redrive the messages to the supply queue. Optionally, I can specify one other FIFO queue as a customized vacation spot. I exploit System optimized in Velocity management settings to redrive messages with the utmost variety of messages per second optimized by Amazon SQS. Optionally, if there may be numerous messages within the DLQ, I can configure a customized most fee of messages per second to keep away from overloading customers.

Console screenshot.

Earlier than beginning the redrive process, I can use the Examine messages part to ballot and verify messages. I already determined what to do, so I select DLQ redrive to start out the duty. I’ve just one message to course of, so the redrive process completes in a short time.

Console screenshot.

As anticipated, the message is again within the supply queue and is able to be processed once more.

Console screenshot.

Issues to know
Lifeless letter queue (DLQ) assist for FIFO queues is out there as we speak in all AWS Areas the place Amazon SQS is obtainable aside from GovCloud Areas and people primarily based in China.

Within the DLQ configuration, the utmost variety of receives ought to be between 1 and 1,000.

There isn’t any further value for utilizing excessive throughput mode or a DLQ. Each Amazon SQS motion counts as a request. A single request can ship or obtain from 1 to 10 messages, as much as a most whole payload of 256 KB. You pay primarily based on the variety of requests, and requests are priced in a different way between commonplace and FIFO queues.

As a part of the AWS Free Tier, there isn’t any value for the primary million requests per 30 days for normal queues and for the primary million requests per 30 days for FIFO queues. For extra info, see Amazon SQS pricing.

With these updates and the elevated throughput, you may cowl the overwhelming majority of use circumstances with FIFO queues.

Use Amazon SQS FIFO queues to have excessive throughput, exactly-once processing, and first-in-first-out supply.

Danilo



Leave a Reply

Your email address will not be published. Required fields are marked *