Amazon SQS (Queues & Decoupling)

STANDARD QUEUE:
├── Delivery: At-least-once (duplicates possible)
├── Ordering: Best-effort (may be out of order)
├── Throughput: Nearly unlimited TPS
├── Deduplication: None built-in
└── Best For:
    ├── High-volume workloads
    ├── Duplicate-tolerant processing
    ├── Application logs
    └── Background job processing

FIFO QUEUE:
├── Delivery: Exactly-once processing
├── Ordering: Strict FIFO (within message group)
├── Throughput: 300 TPS (3,000 with batching)
├── Deduplication: 5-minute window
└── Best For:
    ├── Financial transactions
    ├── Order processing (e-commerce)
    ├── Command sequences
    └── Events requiring strict ordering

FIFO QUEUE WITH MESSAGE GROUPS:

Message Group: "user-123"          Message Group: "user-456"
┌──────────────────────────┐       ┌──────────────────────────┐
│ Order 1 → Order 2 → 3    │       │ Order A → Order B → C    │
│ (processed in sequence)  │       │ (processed in sequence)  │
└──────────────────────────┘       └──────────────────────────┘
        │                                    │
        └──────────┬─────────────────────────┘
                   │
                   ▼
          PARALLEL PROCESSING
          (Different groups processed concurrently)

USE CASES:
├── Per-user: MessageGroupId = userId (user actions in order)
├── Per-order: MessageGroupId = orderId (order steps in order)
├── Per-device: MessageGroupId = deviceId (device events in order)
└── Single group: MessageGroupId = "all" (all messages in order)

VISIBILITY TIMEOUT FLOW:

Time 0s: Consumer A receives Message
         └── Message becomes INVISIBLE

Time 0-30s: Consumer A processing...
            └── Other consumers cannot see message

Time 30s (default timeout):
├── If deleted: Message removed permanently ✓
└── If NOT deleted: Message becomes VISIBLE again
                    └── Another consumer can receive it

CONFIGURATION:
├── Default: 30 seconds
├── Minimum: 0 seconds
├── Maximum: 12 hours
└── Can be extended programmatically (ChangeMessageVisibility)

BEST PRACTICE:
├── Set timeout > expected processing time
├── Add buffer for retries/errors
├── Extend timeout if processing takes longer
└── Rule of thumb: 6x average processing time

MESSAGE RETENTION:

├── Default: 4 days
├── Minimum: 1 minute
├── Maximum: 14 days

IMPORTANT CONSIDERATIONS:
├── Messages deleted after retention period
├── Cannot be recovered once deleted
├── DLQ retention should be longer than source queue
└── Original timestamp preserved when moved to DLQ

SCENARIO (DLQ RETENTION):
Source Queue: 4 days retention
DLQ: 14 days retention

Day 0: Message sent to source queue
Day 3: Message moved to DLQ (after 3 failed attempts)
       └── Message has 11 days left (14 - 3 = 11)
           NOT 14 days from DLQ arrival!

DELAY QUEUE:

├── Default: 0 seconds (no delay)
├── Maximum: 15 minutes
├── Applies to: All new messages in queue

USE CASES:
├── Rate limiting API calls
├── Scheduling delayed processing
├── Building retry mechanisms
├── Coordinating timed workflows

MESSAGE-LEVEL DELAY (Standard Queue only):
├── Override queue-level delay per message
├── Use DelaySeconds parameter when sending
├── Not supported in FIFO queues
└── Range: 0 to 15 minutes

DEAD LETTER QUEUE SETUP:

Source Queue ──────► Consumer
     │                  │
     │              Process
     │                  │
     │         ┌───────┴───────┐
     │         │               │
     │      Success         Failure
     │         │               │
     │      Delete          Return
     │      Message        to Queue
     │                        │
     │                    Retry...
     │                        │
     │              (after maxReceiveCount)
     │                        │
     └──────────► Dead Letter Queue

REDRIVE POLICY (Source Queue):
{
  "deadLetterTargetArn": "arn:aws:sqs:region:account:my-dlq",
  "maxReceiveCount": 3
}

KEY SETTINGS:
├── maxReceiveCount: Times message received before DLQ (1-1000)
├── DLQ must be same type (Standard → Standard, FIFO → FIFO)
├── DLQ retention > Source retention (best practice)
└── Redrive allow policy: Controls which queues can use DLQ

DLQ REDRIVE WORKFLOW:

1. IDENTIFY: Messages appear in DLQ
   └── Set CloudWatch alarm on ApproximateNumberOfMessagesVisible

2. INVESTIGATE: Analyze failed messages
   ├── Check message content
   ├── Review consumer logs
   └── Identify root cause

3. FIX: Resolve the underlying issue
   ├── Fix consumer code
   ├── Update dependencies
   └── Correct configuration

4. REDRIVE: Move messages back
   ├── AWS Console: Start message redrive
   ├── CLI: aws sqs start-message-move-task
   └── Messages return to source queue

REDRIVE ALLOW POLICY (on DLQ):
{
  "redrivePermission": "byQueue",
  "sourceQueueArns": [
    "arn:aws:sqs:region:account:source-queue"
  ]
}

SHORT POLLING (Default):
├── ReceiveMessageWaitTimeSeconds = 0
├── Returns immediately (even if empty)
├── Queries subset of SQS servers
├── May return empty even when messages exist
└── Higher cost (more API calls)

LONG POLLING (Recommended):
├── ReceiveMessageWaitTimeSeconds = 1-20
├── Waits for messages to arrive
├── Queries ALL SQS servers
├── Reduces empty responses
└── Lower cost, better efficiency

CONFIGURATION OPTIONS:
1. Queue-level: Set ReceiveMessageWaitTimeSeconds
2. Request-level: Use WaitTimeSeconds parameter

BEST PRACTICE:
├── Use long polling (20 seconds recommended)
├── Set at queue level for consistency
├── Reduces costs significantly
└── Note: Single-threaded apps polling multiple queues
    may need short polling

LAMBDA + SQS INTEGRATION:

                    Event Source Mapping
SQS Queue ─────────────────────────────────► Lambda Function
              │                                    │
              │  Polls queue                       │
              │  Batches messages                  │
              │  Invokes function                  │
              │  Manages scaling                   │
              └────────────────────────────────────┘

SCALING BEHAVIOR (Standard Queue):
├── Starts with 5 concurrent batches
├── Scales up to 300 more per minute
├── Maximum: 1,250 concurrent invocations
├── Scales down when traffic decreases
└── Minimum: 2 concurrent invocations

CONFIGURATION OPTIONS:
├── Batch Size: 1-10,000 messages (default: 10)
├── Batch Window: 0-300 seconds (wait for full batch)
├── Maximum Concurrency: Limit concurrent invocations
├── Report Batch Item Failures: Return partial failures
└── Function Error Handling: DLQ on function level

FIFO QUEUE WITH LAMBDA:
├── Maintains message order within group
├── One concurrent invocation per message group
├── Multiple groups = parallel processing
└── Scales based on number of message groups

{
  "EventSourceArn": "arn:aws:sqs:region:account:my-queue",
  "FunctionName": "my-function",
  "BatchSize": 100,
  "MaximumBatchingWindowInSeconds": 30,
  "ScalingConfig": {
    "MaximumConcurrency": 50
  },
  "FunctionResponseTypes": ["ReportBatchItemFailures"]
}

BATCH ITEM FAILURE REPORTING:

Scenario: Batch of 10 messages, 2 fail

WITHOUT ReportBatchItemFailures:
├── Entire batch retried
├── 8 successful messages reprocessed
├── Duplicates in downstream systems
└── Inefficient, wastes compute

WITH ReportBatchItemFailures:
├── Return failed message IDs
├── Only failed messages retried
├── Successful messages deleted
└── Efficient, no duplicates

LAMBDA RESPONSE FORMAT:
{
  "batchItemFailures": [
    { "itemIdentifier": "message-id-1" },
    { "itemIdentifier": "message-id-2" }
  ]
}

AUTO SCALING ARCHITECTURE:

SQS Queue ──► CloudWatch Metric ──► Auto Scaling Policy ──► EC2 Instances
                    │
     ApproximateNumberOfMessagesVisible
                    │
                    ▼
         Target: Messages per instance

SCALING METRICS:
├── ApproximateNumberOfMessagesVisible: Messages available
├── ApproximateNumberOfMessagesNotVisible: In-flight messages
├── ApproximateAgeOfOldestMessage: Queue backlog age
└── NumberOfMessagesReceived: Incoming rate

CUSTOM METRIC FORMULA:
Backlog per Instance = ApproximateNumberOfMessagesVisible
                       ─────────────────────────────────────
                       Running Capacity (instances)

Target Value = Acceptable backlog per instance

SCALING POLICY EXAMPLE:
├── Metric: ApproximateNumberOfMessagesVisible
├── Target: 100 messages per instance
├── Scale Out: When > 100 messages/instance
├── Scale In: When < 50 messages/instance
└── Cooldown: 300 seconds

ENCRYPTION AT REST:
├── SSE-SQS: AWS managed key (default, no cost)
├── SSE-KMS: Customer managed KMS key
│   ├── More control over key rotation
│   ├── Audit via CloudTrail
│   └── Cross-account access possible
└── Enable at queue creation or modify later

ENCRYPTION IN TRANSIT:
├── HTTPS endpoints enforced
├── TLS 1.2+ required
└── VPC endpoints available (PrivateLink)

ACCESS POLICY EXAMPLE (KMS):
{
  "Effect": "Allow",
  "Principal": {"AWS": "arn:aws:iam::account:role/consumer"},
  "Action": [
    "kms:Decrypt",
    "kms:GenerateDataKey"
  ],
  "Resource": "*"
}

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Principal": {
        "Service": "sns.amazonaws.com"
      },
      "Action": "sqs:SendMessage",
      "Resource": "arn:aws:sqs:region:account:my-queue",
      "Condition": {
        "ArnEquals": {
          "aws:SourceArn": "arn:aws:sns:region:account:my-topic"
        }
      }
    }
  ]
}

{
  "MessageBody": "Your message content here",
  "MessageAttributes": {
    "OrderType": {
      "DataType": "String",
      "StringValue": "Premium"
    },
    "Priority": {
      "DataType": "Number",
      "StringValue": "1"
    }
  },
  "MessageDeduplicationId": "unique-id-123",
  "MessageGroupId": "order-group-456"
}