Amazon Simple Queue Service

Amazon SQS (Simple Queue Service) is yet another amazing product from Amazon, which provides a reliable, easy & highly scalable hosted queue service for storing messages, which can be accessed by any computer on any network. This provides a way to simply move around messages between computers or application layers and removes the risk of losing messages, delays and does not require all components to be always online, so you may design such that a data collection system post the data into Amazon SQS which may be be processed later by another processing system, this way you may allocate resources in a timely fashion and not requiring them to be always online. You may create as many number of queues you need.

Amazon SQS being a simple web service can being accessed over HTTP and does not require any kind of development or maintenance on the users side. The best part being there is no setup fee and Amazon charges you per SQS request, as on date of writing this the cost is US $1 for 1 million requests, it means all kinds of requests like add, fetching, deleting etc.

The catch is that Amazon SQS delivers the queued message on an "at least once" policy, so chances are there that you may receive the message multiple times, so it is upto you to make sure your system can identify and remove a previously processed message. Also, Amazon SQS does not return queue items in any specific order, so you should plan the system keep this in mind.

Key Fundamentals

Let me explain the key fundamentals involved with a queue.

Messages

Amazon SQS messages can contain up to 64KB of text data, which may be XML, JSON or just simple unformatted text, but you will need to escape binary data/unicode text, the easiest way is to bas64 encode the data.

Message Visibility Timeout

Every Amazon SQS queue has a configurable visibility timeout, i.e. for the designated amount of time after a message is read from a queue, it will not be visible to any other reader. In case the process failed, the message will again be available after the timeout period has passed, till the message is not deleted.

Queue Name

Queue names are limited to 80 characters. Alphanumeric characters plus hyphens (-) and underscores (_) are allowed. Queue names must be unique within an Amazon account. After you delete a queue, you can reuse the queue name.

Example

Below is an example of two scripts where one queues data and the other one processes the data.

The program that will queue data:

Code:

use Amazon::SQS::Simple;

my $sqs = new Amazon::SQS::Simple( $your_access_key, $your_secret_key );

## get the queue end point, this may differ
my $queue = $sqs->GetQueue(q[http://queue.amazonaws.com/MyOldQueue]);

## open some data stream

open( my $access_log, '<', 'tail -f access_log|' );

while ( chomp( my $line = <$access_log> ) ) {
    $queue->SendMessage($line);
}

close($access_log);

The program that will process the data:

Code:

use Amazon::SQS::Simple;

my $sqs = new Amazon::SQS::Simple( $your_access_key, $your_secret_key );

## get the queue end point, this may differ
my $queue = $sqs->GetQueue(q[http://queue.amazonaws.com/MyOldQueue]);

while (1) {

    # get the messages, into @msg
    my @msg;

    eval {
        @msg = $q->ReceiveMessage( MaxNumberOfMessages => 10 );

    };

    if ( !scalar(@msg) ) {
        warn "No messages in queue, sleep for sometime!";
        sleep(5);
    }

    foreach my $msg (@msg) {
        my $message_body = $msg->MessageBody();

        ## do something with the message body

    }
}

References

Amazon SQS
Amazon SQS Developer Guide