Use Cases Of AWS SQS(Simple Queue Service)
What is SQS ?
SQS is distributed queuing system. Messages are not pushed to receivers. Receivers have to poll SQS to receive messages. Messages can’t be received by multiple receivers at the same time. Any one receiver can receive a message, process and delete it. Other receivers do not receive the same message later.
Polling inherently introduces some latency in message delivery in SQS unlike SNS where messages are immediately pushed to subscribers.
SQS is mainly used to decouple applications or integrate applications. Messages can be stored in SQS for short duration of time (max 14 days). SNS distributes several copies of message to several subscribers. For example, lets say you want to replicate data generated by an application to several storage systems. You could use SNS and send this data to multiple subscribers, each replicating the messages it receives to different storage systems (s3, hard disk on your host, database, etc.).
SNS supports several end points such as email, sms, http end point and SQS. If you want unknown number and type of subscribers to receive messages, you need SNS.
How SQS Work ?
A Real World Example of SQS :
Ok, let’s imagine we’re running the latest and greatest video streaming service. Orange Tube.
Well Orange Tube has tons of users who are constantly uploading videos to it’s servers hosted on AWS.
Those videos are stored on S3 and a message is created each time that is added to the SQS queue. These messages are then picked up by worker nodes that are LONG polling the queue for jobs.
The messages have a 30 minute visibility timeout setting and this gives a worker node that amount of time to encode the uploaded video in the format the Orange Tube understands and then deletes the message from the queue.
To deal with sudden spikes in upload traffic, we can place the worker nodes behind a auto scaling group and then monitor the SQS queue for if the number of tasks grows beyond a certain value then we can simply increase (or decrease) the number of nodes available to process videos.
What are the benefits of Amazon SQS over any other types of similar softwares ?
Amazon SQS provides several advantages over building your own software for managing message queues or using commercial or open-source message queuing systems that require significant up-front time for development and configuration.
These alternatives require ongoing hardware maintenance and system administration resources. The complexity of configuring and managing these systems is compounded by the need for redundant storage of messages that ensures messages are not lost if hardware fails.
- Amazon SQS requires no administrative overhead and little configuration.
- Amazon SQS works on a massive scale, processing billions of messages per day.
- You can scale the amount of traffic you send to Amazon SQS up or down without any configuration.
- Amazon SQS also provides extremely high message durability, giving you and your stakeholders added confidence.
Things you should know about SQS!
PULL not PUSH!
SQS is PULL based! What does this mean?
Well it means that message sit on the queue until a worker node requests to read the next message in the queue. So basically worker nodes will poll the SQS queue looking for work to carry out.
SQS will NOT push messages to the worker nodes. For that you’ll need something like Simple Notification Service SNS (which we’ll cover in a later post).
Message Size
The messages that SQS stores can be upto 256 kb in size.
Those messages can be in data formats such as TEXT, JSON, XML etc…
Messages typically contain information about the task you want to perform. They wouldn’t necessarily contain the information to process. Instead they could point to a storage location such as S3 if for instance the queue was encoding video files.
Message Life Span
Messages by default are kept for 4 days.
However their life span can be reduced to just 60 seconds or alternatively they can be increased to 14 days.
There are 2 Types of SQS queues
I could see knowing the distinction between the 2 queue types coming up in the exam for sure. So here’s a quick summary of the 2.
Standard queues
This is the queue type you get by default. It offers a few advantages over FIFO queues.
Such as:
Unlimited throughput basically meaning you can send as many messages into this queue type as you want without worrying about hitting limits.
At-least-once-delivery guaranteeing that each message will be delivered at least once. But occasionally more than once.
Best-Effort Ordering means basically that the order that the messages were originally sent to the queue. However sometimes they can come out of order.
FIFO queues
FIFO or First In First Out queues are where the order of the messages in the queues is guaranteed. With this guarantee there comes a few trade offs:
Throughput FIFO is limited to 300 messages per second. Although you can increase this by batching the messages.
Exactly-Once Processing messages are delivered exactly once and remains available until a consumer node deletes it.
Polling Types
What are polling types?
Well, remember how I said SQS is a PULL based system?
Short polling returns immediately even if the queue is empty. You get charged per usage, so the costs can rack up.
This can pose a problem for developers because if you want to know if there is work to be processed in the queue then you will have to constantly talk to (or poll) the queue to find that out. This incurs charges and can make running SQS expensive.
There is however a solution to this:
Long polling doesn’t return a message until there is a message in the queue or the long polling times out. This helps save money because you are not constantly polling the queue to check if there are messages to process.
Use Cases of SQS :
- You need a simple queue with no particular additional requirements.
- Decoupling two applications and allowing parallel asynchronous processing.
- Only one subscriber is needed.
Thank You for Reading
