AWS SQS — Real Industry Use Case & Case Study
What is Amazon Simple Queue Service (SQS) ???
- Amazon Simple Queue Service (Amazon SQS) offers a secure, durable, and available hosted queue that lets you integrate and decouple distributed software systems and components.
- Amazon SQS offers common constructs such as dead-letter queues and cost allocation tags.
- It provides a generic web services API that you can access using any programming language that the AWS SDK supports.
- Amazon SQS supports both standard and FIFO queues.
Benefits of using Amazon SQS :
- Security — You control who can send messages to and receive messages from an Amazon SQS queue.
- Server-side encryption (SSE) — lets you transmit sensitive data by protecting the contents of messages in queues using keys managed in AWS Key Management Service (AWS KMS).
- Durability — For the safety of your messages, Amazon SQS stores them on multiple servers. Standard queues support at-least-once message delivery, and FIFO queues support exactly-once message processing.
- Availability — Amazon SQS uses redundant infrastructure to provide highly-concurrent access to messages and high availability for producing and consuming messages.
- Scalability — Amazon SQS can process each buffered request independently, scaling transparently to handle any load increases or spikes without any provisioning instructions.
- Reliability — Amazon SQS locks your messages during processing, so that multiple producers can send and multiple consumers can receive messages at the same time.
- Customization — Your queues don’t have to be exactly alike — for example, you can set a default delay on a queue. You can store the contents of messages larger than 256 KB using Amazon Simple Storage Service (Amazon S3) or Amazon DynamoDB, with Amazon SQS holding a pointer to the Amazon S3 object, or you can split a large message into smaller messages.
Basic Amazon SQS architecture :
- There are three main parts in a distributed messaging system: the components of your distributed system, your queue (distributed on Amazon SQS servers), and the messages in the queue.
- In the following scenario, your system has several producers (components that send messages to the queue) and consumers (components that receive messages from the queue). The queue (which holds messages A through E) redundantly stores the messages across multiple Amazon SQS servers.
Message lifecycle :
The following scenario describes the lifecycle of an Amazon SQS message in a queue, from creation to deletion.
Amazon SQS Standard queues :
- Amazon SQS offers standard as the default queue type. Standard queues support a nearly unlimited number of API calls per second, per API action (
- Standard queues support at-least-once message delivery. However, occasionally (because of the highly distributed architecture that allows nearly unlimited throughput), more than one copy of a message might be delivered out of order.
- Standard queues provide best-effort ordering which ensures that messages are generally delivered in the same order as they're sent.
You can use standard message queues in many scenarios, as long as your application can process messages that arrive more than once and out of order, for example:
- Decouple live user requests from intensive background work — Let users upload media while resizing or encoding it.
- Allocate tasks to multiple worker nodes — Process a high number of credit card validation requests.
- Batch messages for future processing — Schedule multiple entries to be added to a database.
Message ordering :
A standard queue makes a best effort to preserve the order of messages, but more than one copy of a message might be delivered out of order.
If your system requires that order be preserved, we recommend using a FIFO (First-In-First-Out) queue or adding sequencing information in each message so you can reorder the messages when they’re received.
Amazon SQS stores copies of your messages on multiple servers for redundancy and high availability. On rare occasions, one of the servers that stores a copy of a message might be unavailable when you receive or delete a message.
If this occurs, the copy of the message isn’t deleted on that unavailable server, and you might get that message copy again when you receive messages. Design your applications to be idempotent (they should not be affected adversely when processing the same message more than once).
Amazon SQS documentation link for reference :
Real Industry Use Case & Case Study :
NASA — The NASA image & video library
The NASA Image and Video Library is a cloud-native solution, with the front-end web app separated from the backend API. It runs as immutable infrastructure in a fully automated environment, with all infrastructure defined in code to support continuous integration and continuous deployment (CI/CD).
In building the solution, ManTech International took advantage of the following AWS services :
• Amazon Elastic Compute Cloud (Amazon EC2), which provides secure, resizable compute capacity in the cloud. This enables NASA to scale up under load and scale down during periods of inactivity to save money, and pay for only what it uses.
• Elastic Load Balancing (ELB), which is used to distribute incoming traffic across multiple Amazon EC2 instances, as required to achieve redundancy and fault-tolerance.
• Amazon Simple Storage Service (Amazon S3), which supports object storage for incoming (uploaded) media, metadata, and published assets.
• Amazon Simple Queue Service (Amazon SQS), which is used to decouple incoming jobs from pipeline processes.
• Amazon Relational Database Service (Amazon RDS), which is used for automatic synchronization and failover.
• Amazon DynamoDB, a fast and flexible NoSQL database service, which is used to track incoming jobs, published assets, and users.
• Amazon Elastic Transcoder, which is used to transcode audio and video to various resolutions.
• Amazon CloudSearch, which is used to support searching by free text or fields.
• Amazon Simple Notification Service (Amazon SNS), which is used to trigger the processing pipeline when new content is uploaded.
• AWS CloudFormation, which enables automated creation, updating, and destruction of AWS resources. ManTech International also used the Troposphere library, which enables the creation of objects via AWS CloudFormation using Python instead of hand-coded JSON — each object representing one AWS resource such as an instance, an Elastic IP (EIP) address, or a security group.
- Amazon CloudWatch, which provides a monitoring service for AWS cloud resources and the applications running on AWS.
Thank You for reading.