AWS SQS and SNS: A Complete Guide to Messaging Services

Introduction

AWS provides two powerful messaging services, Simple Queue Service (SQS) and Simple Notification Service (SNS), for building scalable, decoupled, and event-driven applications. While SQS is a fully managed queuing service that enables asynchronous communication, SNS is a pub/sub messaging service for sending notifications and alerts.

AWS SQS (Simple Queue Service)

What is AWS SQS?

AWS SQS is a message queuing service that allows components of a distributed application to communicate asynchronously. It ensures reliable message delivery between services, even if they are running at different speeds.

Key Features of AWS SQS

• Fully Managed: AWS handles scaling and availability.
• Decoupling: Separates application components for better scalability.
• FIFO & Standard Queues: Ensures message ordering when needed.
• At-least-once Delivery: Guarantees message delivery.
• Dead-letter Queues (DLQ): Stores failed messages for later debugging.
• Message Retention: Stores messages from 1 minute to 14 days.

AWS SQS Architecture

AWS SQS follows a producer-consumer model:

1. Producer: Sends a message to the queue.
2. Queue: Temporarily stores the message.
3. Consumer: Retrieves and processes the message.

Use Cases of AWS SQS

• Decoupling microservices.
• Processing background jobs.
• Handling asynchronous tasks.
• Managing order processing systems.

Steps to Set Up an AWS SQS Queue

1. Log in to the AWS Console and navigate to SQS.
2. Click Create Queue and choose Standard or FIFO.
3. Configure settings like Retention Period and DLQ.
4. Click Create Queue.
5. Use SDK or CLI to send and receive messages:

   aws sqs send-message --queue-url <queue-url> --message-body "Hello SQS!"
   aws sqs receive-message --queue-url <queue-url>
   

AWS SNS (Simple Notification Service)

What is AWS SNS?

AWS SNS is a fully managed publish-subscribe (pub/sub) messaging service that allows sending notifications or messages to multiple subscribers.

Key Features of AWS SNS

• Pub/Sub Model: Multiple subscribers receive a message from one publisher.
• Multiple Protocols: Supports SMS, Email, HTTP, Lambda, and SQS.
• Event-Driven Notifications: Works with AWS services like Lambda and CloudWatch.
• Message Filtering: Subscribers receive only relevant messages.
• Fan-out Mechanism: Distributes messages to multiple endpoints.

AWS SNS Architecture

1. Publisher: Sends a message to an SNS topic.
2. SNS Topic: Acts as a communication channel.
3. Subscribers: Receive notifications via email, SMS, or other endpoints.

Use Cases of AWS SNS

• Sending alerts and notifications.
• Fan-out pattern for real-time updates.
• Mobile push notifications.
• Disaster recovery alerts.

Steps to Set Up an AWS SNS Topic

1. Log in to the AWS Console and navigate to SNS.
2. Click Create Topic and choose Standard or FIFO.
3. Configure topic settings and click Create Topic.
4. Add Subscribers (Email, SMS, SQS, HTTP, Lambda).
5. Use SDK or CLI to publish messages:

   aws sns publish --topic-arn <topic-arn> --message "Hello SNS!"
   

Exploring SNS Topic Filtering for Optimized Notifications

AWS SNS allows message filtering, enabling subscribers to receive only relevant messages instead of every message published to a topic. This helps optimize notification delivery and reduces unnecessary processing.

How SNS Message Filtering Works

1. Message Attributes: Publishers can add key-value pairs to messages.
2. Subscription Filters: Subscribers define filtering policies based on message attributes.
3. Filtered Delivery: SNS only delivers messages that match the subscriber's filter policy.

Steps to Implement SNS Topic Filtering

1. Create an SNS Topic as described earlier.
2. Create Multiple Subscribers (e.g., SQS queues, email, Lambda).
3. Define Message Attributes when publishing messages:

   aws sns publish --topic-arn <topic-arn> --message "New Order Received" --message-attributes '{"orderType": {"DataType": "String", "StringValue": "Premium"}}'
   

4. Apply Filter Policy to a subscriber:

   {
       "orderType": ["Premium"]
   }
   

5. Test Filtering by publishing messages with different attributes and observing the selective delivery.

Exploring Additional Filtering Options in SNS

AWS SNS provides advanced filtering capabilities to enhance notification efficiency. Some key options include:

• Attribute-based Filtering: Define complex filters using multiple attributes.
• Numeric Filtering: Filter messages based on numerical conditions like greater than or less than.
• Prefix and Suffix Matching: Match message attributes based on text patterns.
• Logical Operators: Combine multiple filtering rules using AND, OR, and NOT conditions.

Use Case for SNS Topic Filtering

• Sending targeted marketing notifications.
• Filtering alerts by severity (e.g., critical vs. informational).
• Routing messages to different services based on type.

Implementing SQS and SNS for a Real-World Event-Driven Microservices Project

Scenario: E-Commerce Order Processing System

Imagine an e-commerce platform where orders need to be processed asynchronously. The system uses SNS for notifications and SQS for order processing.

Architecture Overview

1. Customer places an order → Event is published to an SNS Topic.
2. SNS Fan-out → Notifies different services:
   • Payment Service (SQS Queue 1) processes the payment.
   • Inventory Service (SQS Queue 2) updates stock levels.
   • Notification Service (Email/SMS Subscriber) alerts the user.
3. Consumers pick up messages from SQS and process them independently.

Steps to Implement

1. Create an SNS Topic (OrderTopic).
2. Create SQS Queues (PaymentQueue, InventoryQueue).
3. Subscribe SQS Queues to SNS Topic:

   aws sns subscribe --topic-arn <OrderTopic-arn> --protocol sqs --notification-endpoint <PaymentQueue-url>
   aws sns subscribe --topic-arn <OrderTopic-arn> --protocol sqs --notification-endpoint <InventoryQueue-url>
   

4. Publish an Order Event:

   aws sns publish --topic-arn <OrderTopic-arn> --message "New order placed: #1234"
   

5. Consumers process messages from their respective queues.

Conclusion

AWS SQS and SNS are essential for building event-driven architectures. SNS topic filtering optimizes message delivery, while SQS queues help process asynchronous tasks efficiently. Together, they enhance application scalability and reliability.

---

Next Steps:

• Learn about AWS Lambda event triggers using SQS and SNS.
• Implement an end-to-end microservices project using AWS messaging services.
• Explore additional filtering options in SNS.