Foreclosure: The PayPal API is amongst the worst I've ever had to deal with - inconsistencies, sometimes poor and conflicting documentation, unpredictable failures and account changes and major differences between the live and sandbox versions all conspire to make working with the PayPal API quite a pain in the ass.

Unfortunately, there doesn't seem to be any better alternatives currently, so hopefully this guide will help ease the pain for some of you out there taking your lumps working the API into your applications.

The different payment options

PayPal offers a variety of payment options, which might be confusing at first:

• Express Checkout - The premier PayPal service. Express checkout allows you to receive payments without having a merchant account or meeting special requirements other than having your account verified (either via a bank account or a credit card).Previously you could receive Express Checkout payments from PayPal users only, but PayPal has since added a credit-card option for non-PayPal users making this service accessible to practically anyone with a major credit-card.Note that the Express Checkout process occurs on the PayPal platform and thus can never be fully integrated into your site experience completely.
• Direct Payment - The Direct Payment method allows you to receive credit-card payments directly through an API call. This allows you to host the payment process on your site in full, which might make for a more complete shopping experience for shoppers.The Direct Payment method has several variations that allow you to authorize a payment and complete it at a later date - the appropriately named methods Authorization and Capture. The direct payments methods are a part of the Website Payments Pro API, which is only available U.S accounts.
• Recurring Payments - allow you to set up a recurring transaction - i.e, a subscription payment.
• Mass Payments - allow you to transfer money to multiple accounts at once.

This is a comprehensive list, but it covers the main payment options (see the docs for more).

Making API requests

PayPal supports two main formats over HTTP - NVP and SOAP. NVP is short for Node-Value-Pairs and SOAP stands for Simple Object Access Protocol. I will cover the NVP approach which I prefer to SOAP's comparatively verbose and general complex syntax.

Each of the API methods have different parameters but they all share some basic parameters which are used to identify the API account and sign the transaction. Those include:

• USER - Your PayPal API Username.
• PWD - Your PayPal API Password.
• VERSION - Version number of the NVP API service, such as 56.0.
• SIGNATURE - Your PayPal API signature string. This parameter is optional if you use a certificate to authenticate

The last required parameter is METHOD which declares which API method we are calling.

Requests are made over HTTPS. We'll use cURL to build up our basic request, and encapsulate the process in a function:

function paypalApiRequest($method,$params = array()) {
	if( empty($method) ) { //Check if API method is not empty
		return false;
	}
 
	//Our request parameters
	$requestParams = array(
		'METHOD' => $method,
		'VERSION' => '65.1',
		'PWD' => 'ourAPIpassword',
		'USER' => 'ourAPIusername',
		'SIGNATURE' => 'ourAPIsignature'
	);
	//Building our NVP string
	$request = http_build_query($requestParams + $params);
 
	//cURL settings
	$curlOptions = array (
		CURLOPT_URL => $this -> getOption('endpoint'),
		CURLOPT_VERBOSE => 1,
		CURLOPT_SSL_VERIFYPEER => true,
		CURLOPT_SSL_VERIFYHOST => 2,
		CURLOPT_CAINFO => dirname(__FILE__) . '/cacert.pem', //CA cert file
		CURLOPT_RETURNTRANSFER => 1,
		CURLOPT_POST => 1,
		CURLOPT_POSTFIELDS => $request
	);
 
	$ch = curl_init();
	curl_setopt_array($ch,$curlOptions);
 
	//Sending our request - $response will hold the API response
	$response = curl_exec($ch);
 
	//Checking for cURL errors
	if (curl_errno($ch)) {
		//Handle errors
	} else  {
		curl_close($ch);
		$responseArray = array();
		parse_str($response,$responseArray); // Break the NVP string to an array
		return $responseArray;
	}
}

Note that I use a CA certificate file for SSL certificate validation. You can obtain the file from the cURL website or any trusted source. Update the file to the certificate file according to where you placed it.

The response returned would be in NVP format as well. A parameter named ACK signifies the status of the request - Success or SuccessWithWarning on a successful request and Error or Warning values when the request failed. There are many reasons that a request could fail, different ones for each API method, and those are covered in detail in the manual. Keep in mind that the parameter values are case-sensitive so you should code against it accordingly.

Express Checkout

In this post I'll focus on the Express Checkout process and use it to illustrate how you can work with the API:

The Express Checkout process works as follows:

• We request a checkout token from PayPal using the order details
• If successful, we redirect the user to the PayPal endpoint using the received token
• User completes / cancels payment on the PayPal platform and is redirected back to our site
• We complete the payment either when the user is redirected back or via IPN

1. Getting the checkout token - SetExpressCheckout

We initiate the Express Checkout process by passing the order details to the PayPal API and receive a token string that identifies it. This token would be used in the next step to redirect to PayPal.

Required parameters:

• METHOD - 'SetExpressCheckout' (the API method we're using)
• RETURNURL - The URL the user will be redirected to after payment process is complete
• CANCELURL - The URL the user will be redirected to after cancelling the payment process
• PAYMENTREQUEST_0_AMT - The transaction total amount.
• PAYMENTREQUEST_0_ITEMAMT - The cost total of the items in the order, excluding shipping, taxes and other costs. If there are no extra costs, it should be the same value as PAYMENTREQUEST_0_AMT.

There are additional parameters we can pass to add more information about the order, some of which have default values:

• PAYMENTREQUEST_0_CURRENCYCODE - Payment currency. A three-letter code, default is USD.
• PAYMENTREQUEST_0_SHIPPINGAMT - Total shipping costs for this order
• PAYMENTREQUEST_0_TAXAMT - Total tax amount for this order (required if per item tax is specified, see below)
• PAYMENTREQUEST_0_DESC - Order description

We can also add details about individual items in the order:

• L_PAYMENTREQUEST_0_NAMEm - Item name.
• L_PAYMENTREQUEST_0_DESCm - Item description.
• L_PAYMENTREQUEST_0_AMTm - Item cost
• L_PAYMENTREQUEST_0_QTYm - Item quantity

'm' is a variable index identifying the item (use the same number for all details of the same item).

There are many other optional options (read more in the API docs).

We'll use the function we wrote above for building the SetExpressCheckout request:

//Our request parameters
$requestParams = array(
	'RETURNURL' => 'http://www.yourdomain.com/payment/success',
	'CANCELURL' => 'http://www.yourdomain.com/payment/cancelled'
);
 
$orderParams = array(
	'PAYMENTREQUEST_0_AMT' => '500',
	'PAYMENTREQUEST_0_SHIPPINGAMT' => '4'
	'PAYMENTREQUEST_0_CURRENCYCODE' => 'GBP'
);
 
$item = array(
	'L_PAYMENTREQUEST_0_NAME0' => 'iPhone',
	'L_PAYMENTREQUEST_0_DESC0' => 'White iPhone, 16GB',
	'L_PAYMENTREQUEST_0_AMT0' => '500',
	'L_PAYMENTREQUEST_0_QTY0' => '1'
);
$response = paypalApiRequest('SetExpressCheckout',$requestParams + $orderParams + $item);

2. Redirecting to PayPal using the Checkout Express token

If the request was successful, we'll receive a checkout token in the 'TOKEN' parameter of the response.

if(is_array($response) && $response['ACK'] == 'Success') { //Request successful
      $token = $response['TOKEN'];
      header('Location: https://www.paypal.com/webscr?cmd=_express-checkout&token=' . urlencode($token));
}

The user will now go through the purchase process on the PayPal site. If he confirms or cancels it he will return to one of the URLs we specified in the request.

3. Completing the transaction

Assuming the user confirmed the transaction, he will be redirected to our site by PayPal. At this point we have two relevant API methods we should use - 'DoExpressCheckoutPayment' will complete the transaction, but before that we might want to get additional information on the buyer using 'GetExpressCheckoutDetails'.

PayPal will redirect the user back from the purchase with the checkout token, which we will use to call those methods. The token will be available in the URL query parameters via the 'token' parameter. We will check for its existence in the confirmation URL, and send our API requests if we find it.

The 'GetExpressCheckoutDetails' method requires only the checkout token. 'DoExpressCheckoutPayment' requires a couple of additional parameters:

• PAYMENTREQUEST_0_PAYMENTACTION - Payment action. Should be set to 'Sale' unless we specified a different action in the 'SetExpressCheckout' method (possible values include 'Authorization' and 'Capture')
• PAYERID - Unique PayPal account identification. This is returned in the URL query parameters as well, in the 'PayerID' parameter, and can also be retrieved from the details returned by 'GetExpressCheckoutDetails'.

if( isset($_GET['token']) && !empty($_GET['token']) ) { // Token parameter exists
	// Get checkout details, including buyer information.
	// We can save it for future reference or cross check with the data we have
	$checkoutDetails = paypalApiRequest('GetExpressCheckoutDetails', array('TOKEN' => $_GET['token']));
 
	// Complete the checkout transaction
	$requestParams = array(
		'PAYMENTREQUEST_0_PAYMENTACTION' => 'Sale',
		'PAYERID' => $_GET['PayerID']
	);
 
	$response = paypalApiRequest('DoExpressCheckoutPayment',$requestParams);
	if( is_array($response) && $response['ACK'] == 'Success') { // Payment successful
		// We'll fetch the transaction ID for internal bookkeeping
		$transactionId = $response['PAYMENTINFO_0_TRANSACTIONID'];
	}
}

Just the tip of the iceberg

In this post I showed a very simple way of working with the PayPal API. I didn't discuss error handling, more complex methods such as DirectPayment (for direct credit-card payments) and more. There is enough to do there that I could fill 10 more articles like this and not cover everything.

If you don't want to deal with the API yourself, you can check out my PayPal class over at Binpress. It covers additional APIs including handling Direct Payments (credit-card payments), Mass payments (paying multiple recipients at the same time) and IPN (instant payment notifications).

I understand the need for simplicity because of the wide audience of Smashing Magazine, but I'd wish they'd give something more than the absolute basics you could find in almost any other site out there. I also didn't like some of the methods mentioned there for profiling (or the code itself), so I here is my starter guide to optimizing database performance.

When do we optimize the database?

As is noted in the article, page load speed is important. It affects the user-experience as well as Google pagerank when it gets too slow. There are so many variables to account for when trying to improve page load, including page download weight (including all the various assets such as images, javascript and CSS), network latency, browser cache and server headers, server load (requests per second and memory and CPU usage) among others. Yahoo has a very nice guide for client-side performance tips. We're going to suspect the database as the culprit for the purposes of this article, but you should first observe the complete picture before deciding on what to optimize.

Aside from page load speed, a busy database can affect the rest of the server as well, meaning parts that don't use the database or have very fast running queries could start to slow down.

Profile first, optimize last

The basic rule of optimization is to never assume - always verify, using actual data. The process of collecting performance metrics and determining performance issues is called profiling. We want to know whether database performance is responsible for a significant part of our page load time.

Referring again to the smashing magazine article, the author suggests a profiling method that is basically correct however the implementation leaves a lot to be desired. We won't go into why using globals and outputting inside functions is not good practice, and the author even mentions that this could seriously mess up the layout of the site or the sessions and yet makes no attempt to give out a better solution.

We want to time how much queries are taking to run. There are plenty of timing solutions out in the open - such as PEAR_Benchmark, that there is simply no need to build your own unless you want the exercise. The concept is simple - store microtime() values before and after the query for later observation, and the difference would be the timing of the query with good accuracy.

If you are using a database abstraction class (and you should), incorporating a timer to profile every query should be a piece of cake - so no need to hunt down every query and modify the code around it as suggested in the SM article. Wrap the query method of your abstraction class with the timer and use the queries as the keys in the timing array. We used the Zend Framework for all of our previous projects and for our current startup, and it comes with a built-in support for profiling which makes it a breeze to get started.

Example code using Zend_Db_Profiler

$db = Zend_Db::factory('PDO_MYSQL', $config); //Set up the database object
 $db -> getProfiler()->setEnabled(true); // turn on profiler
 
//Queries are performed on the page
//...
 
// Where we want to show the results
$profiles = $profiler -> getQueryProfiles(); //An array of all the query profiling

If you are using Firebug (which you should if you are running Firefox) you can install the FirePHP plugin for completely unobtrusive output. The Zend Framework comes with a FirePHP profiler that can send query results directly into your Firebug console.

$profiler = new Zend_Db_Profiler_Firebug('All DB Queries');
$profiler -> setEnabled(true);
$db -> setProfiler($profiler);
 
//Queries are performed on the page
//...
// No need to output, query profiles will appear in your Firebug console

Pretty convenient. We can go over page by page without disturbing content or messing up sessions and even run this in a production environment, provided we load the profiler only for specific users. The output looks something like this:

It's very important to profile using a relevant dataset. If you profile on a development machine that has very different data (and probably much smaller tables) than your production machine, you will get very different results. You should create a test machine that resembles your live dataset as much as possible to get relevant data (as I've shown in an old article about profiling).

Another important note is to avoid looking at cached results. MySQL will cache certain queries - so verify the results of your profiling by running the queries while avoiding caches using SQL_NO_CACHE and other means. Compare the first run with subsequent runs of the same query to be sure you are seeing non-cached results.

Aside from profiling the queries in real time, we can also profile queries that are used by daemons and cron jobs and log the results to a file. MySQL has a built in feature in MySQL that can log slow queries for us while the database daemon is running. As of MySQL 5.1.21 we can get microsecond timing on queries (previously only one-second jumps were supported) so we can get very good measurements with the slow-query log.

The slow query log should be used for monitoring and be checked periodically for possible problems. The rate at which the log fills out also gives an indication of how much your database is slowing down over time and how much time you have left before you need to optimize.

Optimizing performance

Suppose we found out some problematic queries on slow pages. There are 4 basic ways to optimize query performance:

• Rewrite the queries
• Change indexing strategy
• Change schema
• Use an external cache

Examining query execution plans (EXPLAIN)

Before trying to optimize a slow query, we need to understand what makes it slow. For this purpose MySQL has a query examination tool called EXPLAIN. Add the reserved word 'EXPLAIN' at the beginning of your query to get the execution plan for the query. The execution plan literally 'explains' to us what the database is doing to optimize the query. The MySQL manual has a full reference guide to the different values that appear in the plan, and you can see a full walkthrough of using EXPLAIN to optimize a query in this slideshow on slideshare (as well as in the profiling article I linked to earlier).

This is a very useful tool, but like all other tools it should be used while being aware of its limitations.

Common optimizations

1. Looping queries

The most basic performance issues often will not be the fault of the database itself. One of the most common mistakes is to query in a loop without need. Most likely looped SELECT queries can be rewritten as a JOIN -

 
$query = 'SELECT id,name FROM categories';
$rows = $db -> fetchAll($query);
foreach($rows as $row) {
     $query = 'SELECT id,name FROM sub_categories WHERE category_id=' . (int) $row['id'];
     $subCategories = $db -> fetchAll($query);
     //...
}

(I'm using Zend Framework syntax since we already assumed we are using a database abstraction class)

This could be rewritten as a join -

 
$query = 'SELECT categories.id,categories.name,
sub_categories.id AS subcat_id,sub_categories.name AS subcat_name
FROM categories
LEFT JOIN sub_categories ON categories.id=sub_categories.category_id';
$rows = $db -> fetchAll($query);
foreach($rows as $row) {
   // Require a bit of additional logic to format the results, but we have one query instead of many
}

Inserting and updating rows in a loop can have major overhead as well, and those queries are generally slower than simple SELECT queries (since indexes often need to be updated) and they affect the performance of other queries since they use table / row locks while the data is written (this differs depending on the table engine). I wrote an article almost two years ago on multiple row operations that covers how to rewrite looped INSERT / UPDATE queries and includes some benchmarks to show how it improves performance.

2. Picking only needed columns

It is common to see a wildcard used to pick all columns ('SELECT * FROM ... ') - this however, is not efficient. Depending on the number of participating columns and their type (especially large types such as the TEXT variants), we could be selecting much more data from the database than we actually need. The query will take longer to return since it needs to transfer more data (from the hard-disk if it doesn't hit the cache) and it will take up more memory doing so.

Picking only the needed columns is a good general practice to use, and avoids those problems.

3. Filtering rows correctly and using indexes

Our main goal is to select the smallest amount of rows we need and doing so in the fastest way possible. We want to filter rows using indexes, and in general we want to avoid full table scans unless it is absolutely needed (aside from edge cases where it actually improves performance). The MySQL manual has some great information on optimizing the WHERE clause, and I'll dive into a bit more detail -

Filtering conditions include the WHERE, ON (for joins) and HAVING clauses. As much as possible, we want those clauses to hit indexes - unless we are selecting a very large amount of rows, index lookup is much faster than a full table scan. Those clauses should be used along with the LIMIT clause if relevant to filter the amount of rows / data returned by the query. The LIMIT clause itself can lend some important optimizations for queries if used correctly.

Since our goal is to hit indexes with our WHERE clause, an important rule is to avoid using calculations there. When the filtering condition has to be calculated for each row, the WHERE clause cannot use an index.

Example - fetching users created in the last 4 weeks:

SELECT id,name FROM users WHERE created - NOW() < INTERVAL 4 WEEK

Since the value of the `created` column changes from row to row, we now have a calculation in the left-hand side of the condition. This could be rewritten so that the calculation doesn't change and thus will only be performed once:

SELECT id,name FROM users WHERE created > NOW() - INTERVAL 4 WEEK

This query can use an index on the `created` column and should perform much better.

Another less common calculation but a very problematic one is correlated subqueries in the WHERE clause.

Selecting the lowest priced fruit from several fruit types:

SELECT type, variety, price
FROM fruits
WHERE price = (
    SELECT MIN(price) FROM fruits AS f WHERE f.type = fruits.type
)

(Example taken from the excellent Xaprb article on the topic - you should read it). This query could be rewritten as JOIN, moving the subquery from the WHERE clause -

SELECT f.type, f.variety, f.price
FROM (
   SELECT type, MIN(price) AS minprice
   FROM fruits
   GROUP BY type
) AS minfruits
INNER JOIN fruits AS f ON f.type = minfruits.type AND f.price = minfruits.minprice

Ideally, MySQL would've optimized both of those the same, but since it is usually not the case, rewriting correlated subqueries as joins is preferred.

4. Indexing correctly

Whether optimizations are needed or not is dependent on the EXPLAIN results we mentioned previously. If the execution plan indicates an index is not being used or a non-selective index has been picked, we need to understand why and change our indexing strategy accordingly (or use index hints).

MySQL can use one index per table alias in a query, so we need to plan our indexes to maximize their effectiveness. Using more indexes than is necessary can have adverse affects - as it slows down the operation of INSERT and UPDATE queries, while taking up more memory. Some indexes can even slow down performance depending on their selectivity.

If we use ordering clauses such as ORDER and GROUP BY, our indexes should often be composite indexes (indexes covering more than one column) to allow for both the filtering and ordering to use an index.

5. Picking the right engine for your data

MySQL has a pluggable engine design, which allows you to use different engine types to store your data, each with its own advantages and drawbacks. The two main engines are MyISAM and InnoDB, and the differences between them affect much more than just performance - InnoDB is an ACID compliant transactional engine, while MyISAM sacrifices some integrity and consistancy features for simplicity and performance. Having said that, MyISAM is not necessarily faster, only in some cases.

I use InnoDB for all of my table unless I need a full-text index (a MyISAM only feature), since my tables usually have a lot of write activity. InnoDB uses row-level locks which really helps performance for such use, while MyISAM uses table locks for write operations. It also optimizes write operations by treating indexes differently than MyISAM - InnoDB uses clustered indexes, so picking the right primary key is critical.

Caching

In the case that our optimizations does not yield sufficient performance benefits (either due to technical reasons or our own skill level), caching is a viable strategy to reduce database load. You should always try and optimize the database itself first, since caching will add another of complexity to our application.

MySQL has an internal query cache that caches results from frequently running queries if it meets certain requirements. If our queries are cached by MySQL (this can be verified by running the queries several times), there is no need to cache it - we just need to be aware that a MySQL service restart could cause a noticeable slow down while the cache is being primed again. You can read more about the dangers of the internal cache on the excellent MySQL performance blog (a must read for any serious MySQL user).

There are many caching strategies and that is the topic for another post. Common options include caching to disk (files) or caching to memory (using solutions such as memcache or APC). Another form of caching is to the database - by de-normalizing the schema to store data that is the result of expensive to run queries.

Server tuning and beyond

Everything covered here is just the tip of the iceberg - it gets rapidly more advanced as you get dig deeper, including tuning MySQL server variables (and you should - at least the basics), the server itself (hardware / software) and using related tools such as sphinx and lucene to offload some of the work. I tried to give a good starting point and as many references as possible for getting a good start to getting your database in shape.

I linked to several excellent resources in this article, such as the MySQL manual, MySQL performance blog and Xaprb (the last two are of Percona fame - world-class experts on MySQL). I suggest you start visiting those regularly as they offer excellent advice.

This ain't no rocket science

Amazon's claims of simplicity would be justified indeed if you compare it with building a complete Email sending infrastructure with similar scaling power as their own offering. However, compared to integrating competitors such as Sendgrid or any SMTP based service, SES integration is anything but simple.

Using any SMTP abstraction class, integrating an SMTP service involves passing the credentials and service endpoint IP. That's it. We use the Zend_Mail component from the Zend Framework and our code looked like this:

$config = array(
	'auth' => 'login',
	'username' => 'user@domain.com',
	'password' => 'ourpassword'
);
$transport = new Zend_Mail_Transport_Smtp('smtp.sendgrid.net', $config);
 
$mail = new Zend_Mail();
// Prepare mail
$mail -> send($transport);

Pretty straightforward. Getting Amazon SES up and running was anything but. I ended up writing a nice SES abstraction and even a Zend_Mail transport to go with it (you can check out the code here if you are not interested in the details).

Requests and authentication

The first hurdle to overcome is using Amazon's authentication scheme. Unlike most HTTP APIs, Amazon doesn't pass a request signature as a request parameter. Instead, it uses a custom HTTP header called 'X-Amzn-Authorization' - the documentation of which is hidden in one of the inner pages of the docs as a seemingly meaningless footnote. You cannot do anything without authorizing first! this section should've been front and center and explained much more clearly.

Since we need custom headers, that immediately rules out something simple like file_get_contents() (which probably should be avoided anyway for it's simplistic error handling). My first thought was to use cURL, like I do with most HTTP APIs - however after struggling with it for around an hour, I came to realize that cURL (at least for PHP) doesn't allow you to specify custom headers that are not a part of the HTTP standard - like Amazon's authentication header.

I briefly considered using http_request(), however that requires installing a PECL extension that is not bundled with the default installation, so that was scraped. The only viable option left for me was to construct the HTTP request myself from scratch and negotiate with the SES API using socket functions. Simple, right?

Luckily, SES docs do provide example requests. Being this was the first time in a while for me to experiment with raw HTTP requests and socket connections, it took me some time to get my bearings.

Socked up and raring to go

So was I ready to rock my Email world with Amazon SES at this point? not quite. Amazon's example POST request looks like this -

POST / HTTP/1.1
Host: email.us-east-1.amazonaws.com
Content-Type: application/x-www-form-urlencoded
Date: Tue, 25 May 2010 21:20:27 +0000
X-Amzn-Authorization: AWS3-HTTPS AWSAccessKeyId=AKIADQKE4EXAMPLE,Algorithm=HMACSHA256,Signature=lBP67vCvGl ...

The sample request was failing with a cryptic "Unable to determine service/operation name to be authorized" error. Can you tell what's missing? it's the Content-Length header. Sure enough, adding it finally made my first request to go through to the API. More time wasted because of a bad usage example in the docs.

Obviously, there's more

We've gone this far without mentioning one of the basic annoyances inherent to Amazon SES. You must verify each and every Email address you want to send to and from before you can actually do that. I'm sorry, come again? yes, in order to help protect itself from spammers, Amazon implemented yet another barrier to adoption. Here, user experience means something a user experiences as he contemplates smashing his screen and not a craft used to improve costumer satisfaction.

I mean, I understand the underlying reasoning, but this is going a bit far. Fortunately, requesting "production access" can alleviate verifying recipient addresses leaving the restriction only on source (from) addresses. Seeing that the service is basically useless without it, we sent out a request and put this integration project aside until we got it (which took about a week). It's a good thing they're cheap and scalable (at least, by reputation), cause simple it hasn't been so far.

Finally, up and running

After we got our production access and verified some of our Emails, we could finally put Amazon SES to use. The code provided in their SDK for PHP is simply terrible, and it doesn't even work for SES (UPDATE: while this was true when I tested it a few weeks ago, one of the commentators has clued me in to that it has been updated and it does work as of now), so I've completed my own API wrapper and even wrote a transport for Zend_Mail that we could plug instead of our previous SMTP transport (shown at the beginning of the post).

I even added a simple control panel for listing and editing verified Email addresses as well as viewing usage statistics and quotas. You can download it for free for non-commercial use (there are also licenses for commercial applications if you need it).

A couple of days ago we sent out a newsletter to about 350 recipients and in the middle Amazon SES crapped on us by throwing a "sending rate quota exceeded" exception. Since we don't have an internal queue (we were counting on theirs), we basically had no way of knowing which Emails got sent and which didn't.

It's been almost a month since we were granted production access, and according to their quota growth table, our rate should have been 90 Emails per second. However, it was still at the start value of 1 per second, which we exceeded apparently. I would've expected it to go into a queue and be sent slowly instead of throwing an error, but that's just me I guess.

Another issue that has popped up lately, is one of their API actions returning a malformed XML after the information grew beyond a certain size. Since there is no official technical support for non premium users like me, even for reporting bugs relating to their beta service, I posted the error on their forums and hoped for the best. I have received no response on it so far.

So that's the state of Amazon SES for you. A very cheap and apparently scalable service with some kinks that need to be ironed out. The API is nothing to write home about as well, but with some abstractions that is acceptable.

I hope this article has given you some sense what it takes to integrate Amazon SES and how to do it. If you have any particular questions, don't hesitate to probe me in the comments.

A short recap - my startup, Binpress - a marketplace for source-code, is running a programming contest for best submission of a source-code package in one of the 6 top web languages - PHP, Python, Ruby, Java, ASP.NET (C#, VB.NET) and Javascript. One grand winner and two runner ups will be chosen, as well as the top pick for each language. We have over $40,000 value in cash and prizes that is sponsored by companies such as Google, Amazon, PayPal and Zend.

So far we've had over 70 submissions in the various languages. Some of the submissions are really top notch and there's no clear winner right now. Submissions will start go out to our judges next week for review and the judging will take up to two weeks after the deadline for the contest.

Hope to see what you have to offer there!

To kick start our launch, we've organized a programming contest with over $40k in cash and prizes, sponsored by companies such as Google, Amazon, PayPal and Zend. Check out the contest site, and join if you think you have what it takes - the contest will run until 24th of Feb., so get cracking!

To overcome those issues, I wrote a utility class to automatically resolve Zend Framework dependencies by fetching them from the online SVN repository when they are needed (and then storing them locally). The class is available for download from Binpress under the MIT license.

How does it work:

• An autoload function is prepended to the spl autoload stack
• The auotload function catches requests for Zend Framework classes
• If the class is not available locally, the dependency manager downloads it from the SVN repository
• If the class has additional dependencies handled by internal class managers in the ZF (such as plugins and helpers), those are fetched as well
• All fetched class files are stored locally in a predefined location (such as /library/Zend as is the norm)
• The class is then loaded

Usage:

• Include the Zdm class (Zdm.php) at the beginning of the application (in the bootstrap, if you are using one)
• Run the static start() function
• The start() function accepts several optional parameters
  • 'libraryPath' - The local path to store fetched ZF classes. The default is the same directory as the Zdm class file (expecting it to be under /library)
  • 'zfVersion' - The Zend Framework version to fetch. Default is 1.10.8
  • 'prependStack' - By default the autoload function is prepended to the stack. If this interferes with any existing autoload functions, you can set this value to false and it will be appended to the stack instead.
  • 'repository' - Change the location of the repository to fetch ZF classes from. You can set this value to a local installation of the framework for better performance.
• The first time the application is run with the dependency manager, it might take some  time to download all the dependencies. To avoid the script timing out, you should set the time limit to '0' using set_time_limit()

Notes - you need to remove 'require' and 'require_once' statements that load Zend Framework classes. Those statements produce a fatal error when the file is not found and cannot be trapped by the autoload function. I tried to recover from the warning sent before the fatal error, however it could not prevent the fatal error from happening.

This class can be used to minimize the needed framework build for current projects as well. Rename your Zend Framework library directory, and allow the dependency manager to build the library from classes that are actually used in the application. This usually results in a much smaller framework footprint. I've successfully used this class in several active projects to recreate the needed framework library that runs the entire application.

It is likely there are some dependencies that are not mapped yet in the class. If you encounter such dependencies, please let me so I could add it to the class.

You need a software license when you are selling software

Seems obvious, right? however, many freelance developers do not use any license. Common misconception seems to be that when you develop custom code for a client, he basically owns the code when the job is done. That premise is usually false. While the license given to custom software solutions is usually liberal, allowing changes to source-code (with conditions) - there are many restrictions that apply on the buyer. For example, it's very rare to allow a client to resell your work as his - though without a license, he can certainly do that.

The purpose of a license is to protect your intellectual property (IP) and copyright. While you may intend to allow a client to modify the source code you provide to fit his needs, it does not necessarily mean he can sell it or include it in his other works.

Lets go over some decisions you have to make when writing a software license and explain the legal terms that describe it:

Do you want to allow clients to redistribute your software?

The most basic and critical question. If you answer YES, the license is called a Sublicensable license and it allows license holders (your clients) to resell your code with their own license, as long as that license does not contradict your original license.

Sometimes, an additional condition is added that the software cannot be sold as is and must be included in a greater work to be sublicensed. This condition prevents your client from becoming your competitor and basically selling the same product you are selling. A sublicesnable license should be used when you want to allow clients to sell solutions based on your work - for example, the client wants to sell CMS product that uses a plug-in developed by you.

The opposite of a sublicesnable license is a Personal license. A personal license prohibits license holders from reselling, renting or even allowing use of your code by 3rd parties. This type of license is common when you are selling premade (non-custom) code and want to limit use to the client only. The vast majority of desktop applications are sold under this license.

Do you want to allow clients to modify the source code?

Despite what it may seem, a sublicensable license by itself does not allow clients to modify your code. In order to allow that, the license must include the right to create derivative works. A derivative work is a modified version of your source code.

Having the right to create derivative works in a sublicensable license by itself does not give the client the right to sell modifications (only the original code). In order to allow the sale of a modified version of your source code (either as standalone or as a part of a larger work), the license must include the right to distribute derivative works.

How many sites and servers do you want to allow clients to use  your software on?

This web-oriented clause restricts the client's use of the software to pre-determined number of sites / servers. While more relevant if you are selling a non-custom solution, it can still apply to custom solutions as well. You can also provide the right to use the software on unlimited sites / servers. A common approach is to offer different pricing (mostly for non-custom solutions) depending on the extent of use granted.

Do you want to allow commercial use of your software?

While this condition is true for most custom solutions, there are cases - such as when you are giving a discount to a non-profit - where you want to prohibit commercial use. It is common for off-the-shelf solutions to have a pricing strategy in which the basic product is given for free and commercial versions or extensions are sold for a fee. In order to allow that to happen, commercial use must be covered by the license.

There are basically three main options - royalty free commercial use, non-commercial use and commercial use with conditions. The last one states that commercial use is allowed if certain criteria are met (criteria that you need to decide upon and must be specified in the license).

Do you want to allow clients to sell their license?

Self explanatory. In legal terms - a license that allows license holders to sell their license and their right to use the software to a 3rd party, is called an assignable license. A license that restricts sale of the license is called a non-assignable & non-transferable license.

Do you want to grant your clients an exclusive license?

An exclusive license allows you to sell your software to only one client (granting them exclusivity). This could be a requirement by a client, or your way to insure them that you will not be reselling the custom solution they paid you to develop. In most cases this only applies to custom solutions.

This is just a partial list - there are many more minor restrictions and rights that can be included in a software license. We cover some of those in our licensing guide on binpress, and there even more less common clauses that can be applied. If you want to create a legally termed and binding software license using those conditions, we provide a commercial license generator for registered developers in our marketplace.

If you have any questions regarding any of the topics I covered, I would be happy to elaborate or even pass it on to our attorney. The next time you will be talking to clients about a potential project, be sure to have a ready software license that covers their rights and restrictions. This can save you much anguish and provide you with legal protection in the case that things go south.

I've recently completed a project where a very bizarre bug would occur on some machines, and I wasn't able to reproduce it myself on any test machine that I tried. The bug manifested in a calendar system we built for the project, and in certain situations form submissions would not pass the date validation through Zend_Validate_Date, despite using the exact same input that was passing on the test machines.

After bringing in one of the "affected" machines for testing, I used the process of elimination to determine the problem originates in the Zend_Date::isDate() method, which has different behavior on different machines.

Zend_Date tries to validate dates according to a given format (with a default fallback). The dangerous behavior is that it tries to convert the given format to a localized format using Zend_Locale. Zend_Locale attempts to detect automatically the locale of the requesting client, and it appears that on the machines that were exhibiting the bug, a different locale was determined than those I was testing it on.

Despite trying to validate a non-localized format (to be exact, the MySQL timestamp format - YYYY-MM-dd HH:mm:sss), it was being converted on the affected machine sdue to the auto-detection by Zend_Locale (on a side note - it was happening on Internet Explorer 8 on Windows 7). To avoid this issue, I set the locale for all requests manually to en_US, since I don't have any other use for Zend_Locale in the application. I put the following lines in the bootstrap -

$locale = new Zend_Locale('en_US');
Zend_Registry::set('Zend_Locale', $locale);

As recommended by the manual to achieve this effect.

In my opinion, the Locale should be not be auto-detected by default - it should be an opt-in feature if it affects other components behind the scenes.

Too often I've seen such queries ran in long loops one at a time, which is very bad for performance (as I will show here) and sometimes equally bad for integrity (if the process is interrupted). So what are the alternatives?

Inserting multiple rows

Insertion of multiple rows comes about often in batch jobs, database migrations and handling table relationships. A naive approach, via PHP, would be to loop over the data to be inserted, inserting one row at a time. Suppose the data is already properly filtered and quoted:

foreach( $data as $row ) {
      $query = "INSERT INTO `test_table` (user_id,content)"
          . " VALUES ("
          . $row['user_id'] . ","
          . $row['content'] . ")";
      mysql_query($query);
}

Depending on the amount of rows to be inserted, this can be a costly process. A better approach would be to concatenate the values into one insert query and then execute it:

$values = array();
foreach( $data as $row ) {
    $values[] =  "(" . $row['user_id'] . "," . $row['content'] . ")";
}
if( !empty($values) ) {
    $query = "INSERT INTO `test_table` (user_id,content) VALUES "
             . implode(',',$values);
    mysql_query($query);
}

What's the difference? lets see some benchmarks:

A single query completes much faster than looping through multiple queries. At 2560 rows inserted, it took the loop ~36 seconds to complete, yet the single query took just 0.14 seconds.

Memory consumption shows a reverse trend however:

Since the values are concatenated to create the single query, it consumes more and more memory with more rows as it needs to hold a larger query string. At 2560 rows inserted, the single query approach consumed ~800kb in memory, while the loop consumed just ~60kb.

Since memory is much cheaper than CPU cycles and database connections, I'd usually opt for the single query approach. It's important though to be aware of the implications.

Inserting / Updating (multiple) values

Another use-case of multiple row operations is when we want to insert several rows that might exist already, and in case they exist we want to update existing values instead. Of course, we could check first which rows exist, update the ones that do and insert the ones that don't - for a total of at least three separate queries.

Inserting and updating in the same operation is done using INSERT ... ON DUPLICATE KEY UPDATE.

The KEY in this statement should be a unique key in the table you are inserting to (otherwise duplicates would be allowed and this operation would be moot). The syntax of this statement allows to pick which columns are updated in the case of matching rows. For example, suppose I'm adding translations for content pages. I have the following table schema:

 
pages
 - id
 - content
 - ...
 
pages_translations
 - page_id
 - lang_id
 - content
 

The pages_translations table has a primary (unique) key on ( page_id , lang_id ). When adding / updating a translation, the query would look something like:

INSERT INTO `pages_translations` (page_id,lang_id,content)
VALUES (5,2,'the brown fox ...')
ON DUPLICATE KEY UPDATE content=VALUES(content)
 

This will create a translation if one does not exist and update the content field if it does exist. We can combine this statement with the previous approach for multiple insertion to insert / update multiple rows with one statement.

Updating multiple rows can be done using this technique, or through a CASE .. WHEN condition for each row. This method is a bit more involved and usually results in bigger queries but it's worth noting. Example:

UPDATE `pages` SET content=(CASE
    WHEN id=1 THEN 'first page content ...'
    WHEN id=2 THEN 'second page content ...'
END)

We need to repeat this format for each row which results is a somewhat verbose query.

Inserting / Updating with multiple tables

There are some cases where we would like to use data in one or several tables to update / insert into another table.

With insertion the syntax is relatively straightforward - replace the VALUES part of the statement with a SELECT statement.

INSERT INTO `my_table` (col1,col2,col3)
SELECT col4,col5,col6
FROM `another_table`

The ON DUPLICATE KEY condition can be used with this statement as well.

Updating rows cross-tables is done with a JOIN statement (of the less declarative type).

UPDATE `my_table`,`other_table`
   SET `my_table`.col1 = `other_table`.col2
WHERE `my_table`.parent_id = `other_table`.id

It's important to try the INSERT / UPDATE select statements separately to determine how many rows they will select and how fast will they complete. INSERT / UPDATE operations are relatively costly and can be a severe bottleneck for the database if not managed correctly.

As I found no concrete online information on the test (and the guide link from Zend doesn't work), I might as well elaborate a little for the benefit of future test takers reading this blog -

The test is 1.5 hours long and composed of 75 questions. Most questions are multiple choice with the rest being open-ended (usually requiring to enter what you believe will the be the result input from several manipulations). The scope of the test is pretty encompassing, touching some modules that I wouldn't normally use (or even imagine a possible use scenario for), but if you have enough experience of the core features (MVC, Db, Cache, Filter/Validation, Localization / Internationalization and Security) and coding standards - you should do just fine.

The one thing to watch out for is the relatively high percentage of trick questions - which actually made the test somewhat harder than I'd anticipated (after you hit several trick questions in a row, you start being suspicious of every question). Some questions didn't even have an absolutely right answer, but sort of the answer of least incorrectness.

Thankfully, 1.5 hours is plenty long for delibrating some of those more ambigious questions - I had finished my first run in about 35 minutes and rechecked everything in 10 more minutes - leaving me 45 minutes to spare.

I do have some more respect for people holding the certification now (regarding knowledge and experience), though I'm not sure what regard does it hold in the industry (this is the first time I've encountered someone asking for it - and the client is Zend itself, so it's not surprising). If anyone has had previous experience with ZF certification qualification requirements, I would love to hear it.