(mis-)Adventures with Amazon Simple Email Services (SES)

PHP Web development Zend Framework March 23rd, 2011 by Eran Galperin

Amazon has recently launched a new service as part of its web-services offerings - Amazon Simple Email Service (or SES for short), an HTTP API for sending Emails. The main selling points are Amazon's usual scalability power, as well as a relatively low price point for sending Emails (10 cents per 1000 Emails, not counting bandwidth). They also promise to improve deliverability using filters and feedback loops.

Having used Sendgrid for a while at our startup, Binpress, we were overall satisfied with their service. We did have some problems with seemingly innocent Emails ending up in spam boxes, and combined with the better pricing we thought Amazon SES deserved a shot.

This ain't no rocket science

Amazon's claims of simplicity would be justified indeed if you compare it with building a complete Email sending infrastructure with similar scaling power as their own offering. However, compared to integrating competitors such as Sendgrid or any SMTP based service, SES integration is anything but simple.

Using any SMTP abstraction class, integrating an SMTP service involves passing the credentials and service endpoint IP. That's it. We use the Zend_Mail component from the Zend Framework and our code looked like this:

  1. $config = array(
  2. 'auth' => 'login',
  3. 'username' => 'user@domain.com',
  4. 'password' => 'ourpassword'
  5. );
  6. $transport = new Zend_Mail_Transport_Smtp('smtp.sendgrid.net', $config);
  7.  
  8. $mail = new Zend_Mail();
  9. // Prepare mail
  10. $mail -> send($transport);

Pretty straightforward. Getting Amazon SES up and running was anything but. I ended up writing a nice SES abstraction and even a Zend_Mail transport to go with it (you can check out the code here if you are not interested in the details).

Requests and authentication

The first hurdle to overcome is using Amazon's authentication scheme. Unlike most HTTP APIs, Amazon doesn't pass a request signature as a request parameter. Instead, it uses a custom HTTP header called 'X-Amzn-Authorization' - the documentation of which is hidden in one of the inner pages of the docs as a seemingly meaningless footnote. You cannot do anything without authorizing first! this section should've been front and center and explained much more clearly.

Since we need custom headers, that immediately rules out something simple like file_get_contents() (which probably should be avoided anyway for it's simplistic error handling). My first thought was to use cURL, like I do with most HTTP APIs - however after struggling with it for around an hour, I came to realize that cURL (at least for PHP) doesn't allow you to specify custom headers that are not a part of the HTTP standard - like Amazon's authentication header.

I briefly considered using http_request(), however that requires installing a PECL extension that is not bundled with the default installation, so that was scraped. The only viable option left for me was to construct the HTTP request myself from scratch and negotiate with the SES API using socket functions. Simple, right?

Luckily, SES docs do provide example requests. Being this was the first time in a while for me to experiment with raw HTTP requests and socket connections, it took me some time to get my bearings.

Socked up and raring to go

So was I ready to rock my Email world with Amazon SES at this point? not quite. Amazon's example POST request looks like this -

POST / HTTP/1.1
Host: email.us-east-1.amazonaws.com
Content-Type: application/x-www-form-urlencoded
Date: Tue, 25 May 2010 21:20:27 +0000
X-Amzn-Authorization: AWS3-HTTPS AWSAccessKeyId=AKIADQKE4EXAMPLE,Algorithm=HMACSHA256,Signature=lBP67vCvGl ...

The sample request was failing with a cryptic "Unable to determine service/operation name to be authorized" error. Can you tell what's missing? it's the Content-Length header. Sure enough, adding it finally made my first request to go through to the API. More time wasted because of a bad usage example in the docs.

Obviously, there's more

We've gone this far without mentioning one of the basic annoyances inherent to Amazon SES. You must verify each and every Email address you want to send to and from before you can actually do that. I'm sorry, come again? yes, in order to help protect itself from spammers, Amazon implemented yet another barrier to adoption. Here, user experience means something a user experiences as he contemplates smashing his screen and not a craft used to improve costumer satisfaction.

I mean, I understand the underlying reasoning, but this is going a bit far. Fortunately, requesting "production access" can alleviate verifying recipient addresses leaving the restriction only on source (from) addresses. Seeing that the service is basically useless without it, we sent out a request and put this integration project aside until we got it (which took about a week). It's a good thing they're cheap and scalable (at least, by reputation), cause simple it hasn't been so far.

Finally, up and running

After we got our production access and verified some of our Emails, we could finally put Amazon SES to use. The code provided in their SDK for PHP is simply terrible, and it doesn't even work for SES (UPDATE: while this was true when I tested it a few weeks ago, one of the commentators has clued me in to that it has been updated and it does work as of now), so I've completed my own API wrapper and even wrote a transport for Zend_Mail that we could plug instead of our previous SMTP transport (shown at the beginning of the post).

I even added a simple control panel for listing and editing verified Email addresses as well as viewing usage statistics and quotas. You can download it for free for non-commercial use (there are also licenses for commercial applications if you need it).

A couple of days ago we sent out a newsletter to about 350 recipients and in the middle Amazon SES crapped on us by throwing a "sending rate quota exceeded" exception. Since we don't have an internal queue (we were counting on theirs), we basically had no way of knowing which Emails got sent and which didn't.

It's been almost a month since we were granted production access, and according to their quota growth table, our rate should have been 90 Emails per second. However, it was still at the start value of 1 per second, which we exceeded apparently. I would've expected it to go into a queue and be sent slowly instead of throwing an error, but that's just me I guess.

Another issue that has popped up lately, is one of their API actions returning a malformed XML after the information grew beyond a certain size. Since there is no official technical support for non premium users like me, even for reporting bugs relating to their beta service, I posted the error on their forums and hoped for the best. I have received no response on it so far.

So that's the state of Amazon SES for you. A very cheap and apparently scalable service with some kinks that need to be ironed out. The API is nothing to write home about as well, but with some abstractions that is acceptable.

I hope this article has given you some sense what it takes to integrate Amazon SES and how to do it. If you have any particular questions, don't hesitate to probe me in the comments.

Tags: , ,

Enter your email address to receive notification about new posts.

If you liked this article you should follow me on Twitter and/or share below:
  • Auke

    file_get_contents does allow you to send custom headers, using a custom stream context. See: http://www.php.net/manual/en/context.http.php and http://www.php.net/manual/en/function.stream-context-create.php

  • http://www.binpress.com Eran Galperin

    You are right, I was not aware of that. It seems however that the restriction on the headers is similar to cURL, since this approach is failing for me. It’s hard to tell because of the file_get_contents() simplistic error handling – it throws a warning with the same error regardless of what I do “failed to open stream: HTTP request failed! ”

  • Ted Malo

    Some of those are valid complains but come on, blaming Amazon for your inexperience with working with HTTP protocol?

  • http://www.binpress.com Eran Galperin

    I guess I’ve been spoiled by practically any other HTTP API out there that allows you to access it without modifying the HTTP protocol format and constructing custom headers. Calling something “simple” but then eliminating the use of the common tools for working with APIs (such as cURL) is a valid complaint in my book.

    Not only that – it would have saved me much trouble if they have just provided a full working request example – which they didn’t (I’m not sure they even tested themselves)

  • Drew

    I’m very confused that your summary says it’s a ‘scalable service’, yet the article seems to indicate that you had significant trouble the minute you tried to scale (rate limiting). That seems to be the opposite of scaling to me. Sendgrid and PostmarkApp are my tools of choice specifically because e-mail sending is too important for a service that you can’t rely on.

  • http://www.binpress.com Eran Galperin

    You are right that this is a contradiction of sorts, but it’s another part of Amazon’s attempts to automatically stem spammers. The service is supposedly scalable because of AWS hosting power, however as an API user you are gradually given more resources as long as you don’t violate any terms for a duration and your usage indicates you need it.

    Personally, I find this method to be cumbersome at best and as my post indicates a real issue when you hit your quotas at inconvenient times. I’ve sent Amazon an inquiry regarding my quota two days ago and so far only received a response they are looking into it.

  • Have a look

    There you go :)

    true,
    CURLOPT_URL => ‘http://localhost’,
    CURLOPT_HTTPHEADER => array
    (
    ‘X-Amzn-Authorization: Foo’
    )
    )
    );

    curl_exec($ch);
    curl_close();

    $ php curl.php
    * About to connect() to localhost port 80 (#0)
    * Trying ::1… * Connection refused
    * Trying 127.0.0.1… * connected
    * Connected to localhost (127.0.0.1) port 80 (#0)
    > GET / HTTP/1.1
    Host: localhost
    Accept: */*
    X-Amzn-Authorization: Foo

    < HTTP/1.1 200 OK
    < …

  • http://www.binpress.com Eran Galperin

    What version are you using? I tried it exactly like this previously and not only was X-Amzn-Auth not appearing, the ‘Date’ header I was sending also was being reformatted by cURL

  • Rory

    I’m not sure what issues you had with the Amazon PHP SDK but if you tried it a month ago they have made improvements since then. I grabbed it from PEAR and was sending email basically straight away with: `$ses->send_mail($source, $destination, $message);`

  • http://www.binpress.com Eran Galperin

    Yes, you are right, I’ve picked up a fresh copy and it seems to work without a hitch. I’ll put an update in the post.

  • Have a look

    $ php -v
    PHP 5.3.2-1ubuntu4.7 with Suhosin-Patch (cli) (built: Jan 12 2011 18:36:08)
    Copyright (c) 1997-2009 The PHP Group
    Zend Engine v2.3.0, Copyright (c) 1998-2010 Zend Technologies
    with Suhosin v0.9.29, Copyright (c) 2007, by SektionEins GmbH

    $ curl –version
    curl 7.19.7 (i486-pc-linux-gnu) libcurl/7.19.7 OpenSSL/0.9.8k zlib/1.2.3.3 libidn/1.15
    Protocols: tftp ftp telnet dict ldap ldaps http file https ftps
    Features: GSS-Negotiate IDN IPv6 Largefile NTLM SSL libz

    but i’m pretty sure, that these options also work long time ago .. with a php 5.x and a curl 6.x installed

  • Niall

    The error you posted on the forum looks like it has something to do with transfer-concoding: chunked see http://en.wikipedia.org/wiki/Chunked_transfer_encoding

    ive has similar problem with getting large xml from a server before, not sure if that explains anything

  • http://profiles.google.com/bryan0101 bryan tai

    I just finished verifying and sending my first email address using the cmd tool. Then I read the API…https posting and such. It might be easy for certain type of programmer(perl, LAMP), but coming from java, where we have abstract APIs for pretty much the most complex things (e.g. look up google’s java API for Json and other lib…), this is pretty much not acceptable unless you are big corp looking to migrate, and have dedicate staff and write an extra layer, maintain the code change (it sure will change), talk with Amazon when throw exception….All this for email?! Isn’t the point is for startup/smallbiz to use? Might as well setup another AMI instance with postfix for all these trouble.
    I really wish sendgrid can success and work out the kinks. I got the code compile in an hour with a springbean+javax.mail (but still un provision, can’t test it yet), no scripts.
    It’s email…not nuclear reactor monitoring service.

  • morteza

    In the name of God the Merciful

    God is Great
    Iran is a powerful
    Imam Khamenei is our leader
    You are terrorists
    Islam is complete
    You’re a killer of Palestinian children
    We are not afraid of America
    Imam Khamenei has ordered you to destroy you

    O Allah, bless Muhammad and the family of Muhammad

    http://masafportal.com/uploads/fotos/1314303769_marg-bar-israel.jpg

    we are iranian

  • http://twitter.com/pickywebdesign Picky Web Design

    how SES working with Zend framework?