I understand the need for simplicity because of the wide audience of Smashing Magazine, but I'd wish they'd give something more than the absolute basics you could find in almost any other site out there. I also didn't like some of the methods mentioned there for profiling (or the code itself), so I here is my starter guide to optimizing database performance.

When do we optimize the database?

As is noted in the article, page load speed is important. It affects the user-experience as well as Google pagerank when it gets too slow. There are so many variables to account for when trying to improve page load, including page download weight (including all the various assets such as images, javascript and CSS), network latency, browser cache and server headers, server load (requests per second and memory and CPU usage) among others. Yahoo has a very nice guide for client-side performance tips. We're going to suspect the database as the culprit for the purposes of this article, but you should first observe the complete picture before deciding on what to optimize.

Aside from page load speed, a busy database can affect the rest of the server as well, meaning parts that don't use the database or have very fast running queries could start to slow down.

Profile first, optimize last

The basic rule of optimization is to never assume - always verify, using actual data. The process of collecting performance metrics and determining performance issues is called profiling. We want to know whether database performance is responsible for a significant part of our page load time.

Referring again to the smashing magazine article, the author suggests a profiling method that is basically correct however the implementation leaves a lot to be desired. We won't go into why using globals and outputting inside functions is not good practice, and the author even mentions that this could seriously mess up the layout of the site or the sessions and yet makes no attempt to give out a better solution.

We want to time how much queries are taking to run. There are plenty of timing solutions out in the open - such as PEAR_Benchmark, that there is simply no need to build your own unless you want the exercise. The concept is simple - store microtime() values before and after the query for later observation, and the difference would be the timing of the query with good accuracy.

If you are using a database abstraction class (and you should), incorporating a timer to profile every query should be a piece of cake - so no need to hunt down every query and modify the code around it as suggested in the SM article. Wrap the query method of your abstraction class with the timer and use the queries as the keys in the timing array. We used the Zend Framework for all of our previous projects and for our current startup, and it comes with a built-in support for profiling which makes it a breeze to get started.

Example code using Zend_Db_Profiler

$db = Zend_Db::factory('PDO_MYSQL', $config); //Set up the database object
 $db -> getProfiler()->setEnabled(true); // turn on profiler
 
//Queries are performed on the page
//...
 
// Where we want to show the results
$profiles = $profiler -> getQueryProfiles(); //An array of all the query profiling

If you are using Firebug (which you should if you are running Firefox) you can install the FirePHP plugin for completely unobtrusive output. The Zend Framework comes with a FirePHP profiler that can send query results directly into your Firebug console.

$profiler = new Zend_Db_Profiler_Firebug('All DB Queries');
$profiler -> setEnabled(true);
$db -> setProfiler($profiler);
 
//Queries are performed on the page
//...
// No need to output, query profiles will appear in your Firebug console

Pretty convenient. We can go over page by page without disturbing content or messing up sessions and even run this in a production environment, provided we load the profiler only for specific users. The output looks something like this:

It's very important to profile using a relevant dataset. If you profile on a development machine that has very different data (and probably much smaller tables) than your production machine, you will get very different results. You should create a test machine that resembles your live dataset as much as possible to get relevant data (as I've shown in an old article about profiling).

Another important note is to avoid looking at cached results. MySQL will cache certain queries - so verify the results of your profiling by running the queries while avoiding caches using SQL_NO_CACHE and other means. Compare the first run with subsequent runs of the same query to be sure you are seeing non-cached results.

Aside from profiling the queries in real time, we can also profile queries that are used by daemons and cron jobs and log the results to a file. MySQL has a built in feature in MySQL that can log slow queries for us while the database daemon is running. As of MySQL 5.1.21 we can get microsecond timing on queries (previously only one-second jumps were supported) so we can get very good measurements with the slow-query log.

The slow query log should be used for monitoring and be checked periodically for possible problems. The rate at which the log fills out also gives an indication of how much your database is slowing down over time and how much time you have left before you need to optimize.

Optimizing performance

Suppose we found out some problematic queries on slow pages. There are 4 basic ways to optimize query performance:

• Rewrite the queries
• Change indexing strategy
• Change schema
• Use an external cache

Examining query execution plans (EXPLAIN)

Before trying to optimize a slow query, we need to understand what makes it slow. For this purpose MySQL has a query examination tool called EXPLAIN. Add the reserved word 'EXPLAIN' at the beginning of your query to get the execution plan for the query. The execution plan literally 'explains' to us what the database is doing to optimize the query. The MySQL manual has a full reference guide to the different values that appear in the plan, and you can see a full walkthrough of using EXPLAIN to optimize a query in this slideshow on slideshare (as well as in the profiling article I linked to earlier).

This is a very useful tool, but like all other tools it should be used while being aware of its limitations.

Common optimizations

1. Looping queries

The most basic performance issues often will not be the fault of the database itself. One of the most common mistakes is to query in a loop without need. Most likely looped SELECT queries can be rewritten as a JOIN -

 
$query = 'SELECT id,name FROM categories';
$rows = $db -> fetchAll($query);
foreach($rows as $row) {
     $query = 'SELECT id,name FROM sub_categories WHERE category_id=' . (int) $row['id'];
     $subCategories = $db -> fetchAll($query);
     //...
}

(I'm using Zend Framework syntax since we already assumed we are using a database abstraction class)

This could be rewritten as a join -

 
$query = 'SELECT categories.id,categories.name,
sub_categories.id AS subcat_id,sub_categories.name AS subcat_name
FROM categories
LEFT JOIN sub_categories ON categories.id=sub_categories.category_id';
$rows = $db -> fetchAll($query);
foreach($rows as $row) {
   // Require a bit of additional logic to format the results, but we have one query instead of many
}

Inserting and updating rows in a loop can have major overhead as well, and those queries are generally slower than simple SELECT queries (since indexes often need to be updated) and they affect the performance of other queries since they use table / row locks while the data is written (this differs depending on the table engine). I wrote an article almost two years ago on multiple row operations that covers how to rewrite looped INSERT / UPDATE queries and includes some benchmarks to show how it improves performance.

2. Picking only needed columns

It is common to see a wildcard used to pick all columns ('SELECT * FROM ... ') - this however, is not efficient. Depending on the number of participating columns and their type (especially large types such as the TEXT variants), we could be selecting much more data from the database than we actually need. The query will take longer to return since it needs to transfer more data (from the hard-disk if it doesn't hit the cache) and it will take up more memory doing so.

Picking only the needed columns is a good general practice to use, and avoids those problems.

3. Filtering rows correctly and using indexes

Our main goal is to select the smallest amount of rows we need and doing so in the fastest way possible. We want to filter rows using indexes, and in general we want to avoid full table scans unless it is absolutely needed (aside from edge cases where it actually improves performance). The MySQL manual has some great information on optimizing the WHERE clause, and I'll dive into a bit more detail -

Filtering conditions include the WHERE, ON (for joins) and HAVING clauses. As much as possible, we want those clauses to hit indexes - unless we are selecting a very large amount of rows, index lookup is much faster than a full table scan. Those clauses should be used along with the LIMIT clause if relevant to filter the amount of rows / data returned by the query. The LIMIT clause itself can lend some important optimizations for queries if used correctly.

Since our goal is to hit indexes with our WHERE clause, an important rule is to avoid using calculations there. When the filtering condition has to be calculated for each row, the WHERE clause cannot use an index.

Example - fetching users created in the last 4 weeks:

SELECT id,name FROM users WHERE created - NOW() < INTERVAL 4 WEEK

Since the value of the `created` column changes from row to row, we now have a calculation in the left-hand side of the condition. This could be rewritten so that the calculation doesn't change and thus will only be performed once:

SELECT id,name FROM users WHERE created > NOW() - INTERVAL 4 WEEK

This query can use an index on the `created` column and should perform much better.

Another less common calculation but a very problematic one is correlated subqueries in the WHERE clause.

Selecting the lowest priced fruit from several fruit types:

SELECT type, variety, price
FROM fruits
WHERE price = (
    SELECT MIN(price) FROM fruits AS f WHERE f.type = fruits.type
)

(Example taken from the excellent Xaprb article on the topic - you should read it). This query could be rewritten as JOIN, moving the subquery from the WHERE clause -

SELECT f.type, f.variety, f.price
FROM (
   SELECT type, MIN(price) AS minprice
   FROM fruits
   GROUP BY type
) AS minfruits
INNER JOIN fruits AS f ON f.type = minfruits.type AND f.price = minfruits.minprice

Ideally, MySQL would've optimized both of those the same, but since it is usually not the case, rewriting correlated subqueries as joins is preferred.

4. Indexing correctly

Whether optimizations are needed or not is dependent on the EXPLAIN results we mentioned previously. If the execution plan indicates an index is not being used or a non-selective index has been picked, we need to understand why and change our indexing strategy accordingly (or use index hints).

MySQL can use one index per table alias in a query, so we need to plan our indexes to maximize their effectiveness. Using more indexes than is necessary can have adverse affects - as it slows down the operation of INSERT and UPDATE queries, while taking up more memory. Some indexes can even slow down performance depending on their selectivity.

If we use ordering clauses such as ORDER and GROUP BY, our indexes should often be composite indexes (indexes covering more than one column) to allow for both the filtering and ordering to use an index.

5. Picking the right engine for your data

MySQL has a pluggable engine design, which allows you to use different engine types to store your data, each with its own advantages and drawbacks. The two main engines are MyISAM and InnoDB, and the differences between them affect much more than just performance - InnoDB is an ACID compliant transactional engine, while MyISAM sacrifices some integrity and consistancy features for simplicity and performance. Having said that, MyISAM is not necessarily faster, only in some cases.

I use InnoDB for all of my table unless I need a full-text index (a MyISAM only feature), since my tables usually have a lot of write activity. InnoDB uses row-level locks which really helps performance for such use, while MyISAM uses table locks for write operations. It also optimizes write operations by treating indexes differently than MyISAM - InnoDB uses clustered indexes, so picking the right primary key is critical.

Caching

In the case that our optimizations does not yield sufficient performance benefits (either due to technical reasons or our own skill level), caching is a viable strategy to reduce database load. You should always try and optimize the database itself first, since caching will add another of complexity to our application.

MySQL has an internal query cache that caches results from frequently running queries if it meets certain requirements. If our queries are cached by MySQL (this can be verified by running the queries several times), there is no need to cache it - we just need to be aware that a MySQL service restart could cause a noticeable slow down while the cache is being primed again. You can read more about the dangers of the internal cache on the excellent MySQL performance blog (a must read for any serious MySQL user).

There are many caching strategies and that is the topic for another post. Common options include caching to disk (files) or caching to memory (using solutions such as memcache or APC). Another form of caching is to the database - by de-normalizing the schema to store data that is the result of expensive to run queries.

Server tuning and beyond

Everything covered here is just the tip of the iceberg - it gets rapidly more advanced as you get dig deeper, including tuning MySQL server variables (and you should - at least the basics), the server itself (hardware / software) and using related tools such as sphinx and lucene to offload some of the work. I tried to give a good starting point and as many references as possible for getting a good start to getting your database in shape.

I linked to several excellent resources in this article, such as the MySQL manual, MySQL performance blog and Xaprb (the last two are of Percona fame - world-class experts on MySQL). I suggest you start visiting those regularly as they offer excellent advice.

SELECT product_images.src,products.name
FROM product_images
INNER JOIN products ON product_images.parent_id=products.id

Literally: Get the name of the parent product for each row in the products_images table.

Going the other direction is not as trivial. If we just want all the referenced rows in the child table, the query seen before would do the trick. But if we wanted a specific row to be found - for example, the latest inserted image, it becomes somewhat more complicated.

Lets explore a common scenario - fetching the latest comment for each row in a list of posts in a forum.

Latest posts from the devnet forums

Dealing with the following simplified schema for a forum:

CREATE TABLE `threads` (
   `id` INT NOT NULL  AUTO_INCREMENT,
   `title` VARCHAR( 250 ) NOT NULL ,
   `created` TIMESTAMP NOT NULL DEFAULT CURRENT_TIMESTAMP ,
   PRIMARY KEY ( `id` )
) ENGINE = InnoDB;
 
CREATE TABLE `posts` (
   `id` INT NOT NULL  AUTO_INCREMENT,
   `thread_id` INT NOT NULL ,
   `content` TEXT NOT NULL ,
   `created` TIMESTAMP NOT NULL DEFAULT CURRENT_TIMESTAMP ,
   PRIMARY KEY ( `id` ) ,
   INDEX ( `thread_id` )
) ENGINE = InnoDB;

A naive approach to fetching 10 threads with the latest post for each thread would be something like:

SELECT threads.*, posts.*
FROM threads
LEFT JOIN posts ON posts.thread_id=threads.id
GROUP BY threads.id
ORDER BY posts.created DESC
LIMIT 10

This query might appear to work as long as there is only one post per thread. Once there are more, you will start getting unexpected results (go ahead, try it). Lets try and understand why:

In the SQL standard, a GROUP BY modifier can return only columns that are either aggregated or the same for the entire group. MySQL extends this and allows returning non unique or aggregated columns from the group, instead of throwing an error.

However, values from those columns could come from any row in the group and we have no control over which row it will be.

... all rows in each group should have the same values for the columns that are ommitted from the GROUP BY part. The server is free to return any value from the group, so the results are indeterminate unless all values are the same.
The MySQL manual on GROUP BY modifiers

In addition, grouping occurs before the order clause is even applied. After the grouping, unpredictable values for the order column (posts.created) would exist for each group, making the sorting afterward somewhat meaningless (since we wanted the latest post for each thread).

A more experienced SQL user might think - why not just use an aggregate function instead? leading to the following query:

SELECT threads.*, posts.*, MAX(posts.created) AS maxcreated
FROM threads
LEFT JOIN posts ON posts.thread_id=threads.id
GROUP BY threads.id
ORDER BY maxcreated DESC
LIMIT 10

This is somewhat better, as it does retrieve the timestamp belonging to the latest post in each thread - however, surprisingly (or not, if you've been following closely) the rest of the non grouped columns from the threads table are still indeterminate and do not necessarily belong to the row with the latest timestamp.

The solution to this issue is not completely trivial, but it makes sense once you understand the limitations. The key is to use the GROUP BY statement to return the filtering criteria (latest timestamps from the posts table) but to avoid it affecting the result set directly.

First, the filtering criteria -

SELECT thread_id,MAX(created) AS created
FROM posts
GROUP BY posts.thread_id

Here we return only aggregate (latest timestamp) and unique (thread_id) data for each group. Now we have the timestamp of the latest post from each thread, along with the matching thread identifier.

We will move the filtering criteria into a subquery, and JOIN that against the threads and posts table respectively.

SELECT threads.*, posts.*
FROM threads
LEFT JOIN (
   SELECT thread_id,MAX(created) AS created FROM posts
   GROUP BY thread_id
) AS latest ON latest.thread_id = threads.id
LEFT JOIN posts ON latest.created=posts.created AND
latest.thread_id=posts.thread_id
ORDER BY posts.created DESC
LIMIT 10

Literally: Fetch rows from the threads table, (left) join that with the timestamp from the latest post in each thread, and then (left) join that with the posts table with the condition that the timestamp and thread_id match. This returns all the rows in the threads table along with the latest post from each thread.

People have a disposition for avoiding subqueries due to performance concerns, but in this case it is a derived subquery (also referred to as a derived table) which will compute once for the entire query as opposed to a correlated subquery which computes once per row ( = bad performance with many rows).

* Note - in real-world situations you would like to specify the columns to be returned in the result set

For more advanced usage, I suggest you read Baron Schwartz's article on selecting the first/least/max row per group in SQL

Too often I've seen such queries ran in long loops one at a time, which is very bad for performance (as I will show here) and sometimes equally bad for integrity (if the process is interrupted). So what are the alternatives?

Inserting multiple rows

Insertion of multiple rows comes about often in batch jobs, database migrations and handling table relationships. A naive approach, via PHP, would be to loop over the data to be inserted, inserting one row at a time. Suppose the data is already properly filtered and quoted:

foreach( $data as $row ) {
      $query = "INSERT INTO `test_table` (user_id,content)"
          . " VALUES ("
          . $row['user_id'] . ","
          . $row['content'] . ")";
      mysql_query($query);
}

Depending on the amount of rows to be inserted, this can be a costly process. A better approach would be to concatenate the values into one insert query and then execute it:

$values = array();
foreach( $data as $row ) {
    $values[] =  "(" . $row['user_id'] . "," . $row['content'] . ")";
}
if( !empty($values) ) {
    $query = "INSERT INTO `test_table` (user_id,content) VALUES "
             . implode(',',$values);
    mysql_query($query);
}

What's the difference? lets see some benchmarks:

A single query completes much faster than looping through multiple queries. At 2560 rows inserted, it took the loop ~36 seconds to complete, yet the single query took just 0.14 seconds.

Memory consumption shows a reverse trend however:

Since the values are concatenated to create the single query, it consumes more and more memory with more rows as it needs to hold a larger query string. At 2560 rows inserted, the single query approach consumed ~800kb in memory, while the loop consumed just ~60kb.

Since memory is much cheaper than CPU cycles and database connections, I'd usually opt for the single query approach. It's important though to be aware of the implications.

Inserting / Updating (multiple) values

Another use-case of multiple row operations is when we want to insert several rows that might exist already, and in case they exist we want to update existing values instead. Of course, we could check first which rows exist, update the ones that do and insert the ones that don't - for a total of at least three separate queries.

Inserting and updating in the same operation is done using INSERT ... ON DUPLICATE KEY UPDATE.

The KEY in this statement should be a unique key in the table you are inserting to (otherwise duplicates would be allowed and this operation would be moot). The syntax of this statement allows to pick which columns are updated in the case of matching rows. For example, suppose I'm adding translations for content pages. I have the following table schema:

 
pages
 - id
 - content
 - ...
 
pages_translations
 - page_id
 - lang_id
 - content
 

The pages_translations table has a primary (unique) key on ( page_id , lang_id ). When adding / updating a translation, the query would look something like:

INSERT INTO `pages_translations` (page_id,lang_id,content)
VALUES (5,2,'the brown fox ...')
ON DUPLICATE KEY UPDATE content=VALUES(content)
 

This will create a translation if one does not exist and update the content field if it does exist. We can combine this statement with the previous approach for multiple insertion to insert / update multiple rows with one statement.

Updating multiple rows can be done using this technique, or through a CASE .. WHEN condition for each row. This method is a bit more involved and usually results in bigger queries but it's worth noting. Example:

UPDATE `pages` SET content=(CASE
    WHEN id=1 THEN 'first page content ...'
    WHEN id=2 THEN 'second page content ...'
END)

We need to repeat this format for each row which results is a somewhat verbose query.

Inserting / Updating with multiple tables

There are some cases where we would like to use data in one or several tables to update / insert into another table.

With insertion the syntax is relatively straightforward - replace the VALUES part of the statement with a SELECT statement.

INSERT INTO `my_table` (col1,col2,col3)
SELECT col4,col5,col6
FROM `another_table`

The ON DUPLICATE KEY condition can be used with this statement as well.

Updating rows cross-tables is done with a JOIN statement (of the less declarative type).

UPDATE `my_table`,`other_table`
   SET `my_table`.col1 = `other_table`.col2
WHERE `my_table`.parent_id = `other_table`.id

It's important to try the INSERT / UPDATE select statements separately to determine how many rows they will select and how fast will they complete. INSERT / UPDATE operations are relatively costly and can be a severe bottleneck for the database if not managed correctly.

Suppose we want to find the 6 closest numbers to the number 2500 (including) from a numbers table:

SELECT number, ABS( number - 2500 ) AS distance
FROM numbers
ORDER BY distance
LIMIT 6

This returns:

(6 total, Query took 2.2792 sec)

This works nicely, and is actually relatively performant if the number column is indexed. Despite having to run a full table scan in order to calculate the distance for every number in the table, running this on a ~2 million row table completes in just over 2 seconds.

This kind of performance would be quite enough for smaller tables, but often in real time applications we would like faster response times than 2 seconds for completing a query.

Since we know exactly what we need, we can help MySQL by limiting the range of numbers it has to calculate the distance for. Since we want the 6 closest numbers, we can be sure they'll be at most in a range of 6 numbers lower and 6 numbers higher than our selected value.

If we can do that, then the calculation would run for only 13 numbers, hopefully leading to much improved performance. Selecting those numbers can be done using a union on a couple of SELECT statements:

(
   SELECT number
   FROM `numbers`
   WHERE number >= 2500
   ORDER BY number
   LIMIT 7
) UNION ALL (
   SELECT number
   FROM `numbers`
   WHERE number < 2500
   ORDER BY number DESC
   LIMIT 6
)

Notice the first one includes the actual value (in case it exists) and hence has one more value to select in its limit clause.

Combining this with our previous successful attempt produces the following ungainly query:

SELECT number, ABS( number - 2500 ) AS distance FROM (
	(
		SELECT number
		FROM `numbers`
		WHERE number >=2500
		ORDER BY number
		LIMIT 6
	) UNION ALL (
		SELECT number
		FROM `numbers`
		WHERE number < 2500
		ORDER BY number DESC
		LIMIT 6
	)
) AS n
ORDER BY distance
LIMIT 6

It does return the same results:

Is much more performant:

(6 total, Query took 0.0011 sec)

And there you have it.

A couple of notes:
- All queries were ran with SQL_NO_CACHE at least 5 times to ensure the timings were indicative of the performance.
- The queries were ran against a ~2 million table filled with randomly generated values.
- An index was created on the number column.

MySQL 5.0 added the index_merge select type, which allows the query optimizer to possibly select several indexes from a single table and merge them to improve query performance. I say possibly, since leaving such decisions to the optimizer is risky at best. In fact, as I will show next, you are sometimes left with no indexes selected out of several possible options, resulting in a full table scan.

Continuing from my last post, I'll use a real-world example to show the different paths the queries optimizer can take when preparing an execution plan for our queries.

I'll actually be working with the same query I profiled last time, with a minor change. The relevant table structure is as follows:

--
-- Table structure for table `tasks`
--
 
CREATE TABLE IF NOT EXISTS `tasks` (
  `id` int(13) NOT NULL AUTO_INCREMENT,
  `list_id` int(13) NOT NULL,
  `user_id` int(13) NOT NULL,
  `task` varchar(255) collate utf8_unicode_ci NOT NULL,
  `due` timestamp NULL DEFAULT NULL,
  `created` timestamp NOT NULL DEFAULT CURRENT_TIMESTAMP,
  `checked` timestamp NULL DEFAULT NULL,
  `assigned` int(13) DEFAULT NULL,
  `done` enum('1') collate utf8_unicode_ci DEFAULT NULL,
  PRIMARY KEY  (`id`),
  KEY `user_id` (`user_id`,`due`),
  KEY `assigned` (`assigned`,`due`),
  KEY `due` (`due`)
) ENGINE=InnoDB

And the query itself:

SELECT `tasks`.`id`,
           `tasks`.`task`,
           UNIX_TIMESTAMP(due) AS `time`,
           `lists`.`name` AS `list_name`,
           `lists`.`id` AS `list_id`
FROM `tasks`
INNER JOIN `lists` ON lists.id=tasks.list_id
WHERE (tasks.user_id='1' OR assigned='1')
     AND (tasks.done IS NULL)
     AND (tasks.due IS NOT NULL)
ORDER BY `tasks`.`due` ASC

This query is almost the same as the one I worked with last time. The only difference being the first statement in the WHERE clause, involving an OR operator:

(tasks.user_id='1' OR assigned='1')

Just to understand what I'm doing here - The query selects from the tasks table with several filtering criteria. The last OR statement conveys the condition that the tasks selected either belong to specific user (i.e created by him) or assigned to him (those two possibly coincide).

For testing purposes I will be running the queries against a testing database I set up with plenty of mock data. The database is around 1Gb in total size, with the tasks table at about 1.8 million rows. It's not very large, but enough for significant data to be obtained while allowing relatively online tampering with schema (modifying keys takes only around 4 minutes to complete).

Running in original form computes as (average of 10 queries):

(304 total, Query took 6.84 sec)

6.84 sec is way too long for running a query in a typical web application. If you'd recall, I last got this query running at 0.0008 sec without the additional OR condition, meaning it is running at around 7,000 times slower (yikes). In such a case you would suspect the indexes are not selective enough, and sure enough an EXPLAIN reveals:

MySQL is performing a regular index select and not an index_merge as we'd like, and doing that it selects the worst possible index - the only one that doesn't filter the result set. Sure enough, all 1.8M rows are scanned and the query is underperforming badly. Trying to force the issue, I first add an IGNORE INDEX(due) to remove it from the equation:

(304 total, Query took 1.41 sec)

The query optimizer has decided not to use an index at all, despite several good candidates. However, query performance is improved since a full table scan with no index is performed straight up. It's still way too slow, and we'd like it to use a filtering index so it can avoid a full table scan. Trying to force the other indexes doesn't work so we're left with no choice but to try alternative query structures.

Our first candidate is replacing our OR condition with two UNION'ed select statements (the inspiration is from a couple of posts over at the MySQL performance blog). Breaking the original query into a UNION form results in:

(SELECT `tasks`.`id`,
            `tasks`.`task`,
            UNIX_TIMESTAMP(due) AS `time`,
            `lists`.`name` AS `list_name`,
            `lists`.`id` AS `list_id`
FROM `tasks`
INNER JOIN `lists` ON lists.id=tasks.list_id
WHERE (tasks.done IS NULL)
    AND (tasks.due IS NOT NULL)
    AND (tasks.user_id='1')
) UNION ALL (
 SELECT `tasks`.`id`,
            `tasks`.`task`,
            UNIX_TIMESTAMP(due) AS `time`,
            `lists`.`name` AS `list_name`,
            `lists`.`id` AS `list_id`
FROM `tasks`
INNER JOIN `lists` ON lists.id=tasks.list_id
WHERE (tasks.done IS NULL)
    AND (tasks.due IS NOT NULL)
    AND (assigned='1' AND tasks.user_id!='1')
) ORDER BY `time` ASC
 

This unsightly looking query looks much more complex, but actually gets the job done:

(304 total, Query took 0.0021 sec)

A big improvement to say the least (~3500 faster than the original query).
The explain shows why:

Only 304 rows are scanned (as opposed to the entire 1.8M table before). Despite the filesort, we are in the area of usability for this query.

Another approach would be to determine why the index_merging isn't take place and how can we enforce it. By reducing the WHERE clause to include only the columns covered by an index, index_merging kicks in resulting in a similar performance as the union, albeit with less selective filtering:

(368 total, Query took 0.0018 sec)

We could filter our result set in the application, but I'd like to keep it in the SQL where it belongs. In order to integrate the rest of the filtering statements, I use an IN condition on another index I have available - the primary key. This results in:

SELECT `tasks`.`id`,
           `tasks`.`task`,
           UNIX_TIMESTAMP(due) AS `time`,
           `lists`.`name` AS `list_name`,
           `lists`.`id` AS `list_id`
FROM `tasks`
INNER JOIN `lists` ON lists.id=tasks.list_id
WHERE (tasks.user_id='1' OR assigned='1')
   AND tasks.id IN
     (SELECT id
      FROM tasks
      WHERE (tasks.done IS NULL)
        AND (tasks.due IS NOT NULL)
     )
ORDER BY `tasks`.`due` ASC
 

Which brings us back to the performance levels of the UNION form:

(304 total, Query took 0.0021 sec)

And we can see in the EXPLAIN that it is indeed an index_merge operation:

So there you have it. From 6.83 seconds to 0.0021 seconds with some tweaking to the query structure. I am still not completely satisfied with the fact that it's using filesort, but I couldn't get it to use another index for the operation. Without the sorting the query is twice as fast:

(304 total, Query took 0.0010 sec)

So if it ever becomes an issue I could move sorting to the application code. Hopefully by then merge_index is better implemented in MySQL.

Regarding the query strucutre - personally I use the subquery format since it is more compact and maintainable. It is always good however to be familiar with all the alternatives.

The Performance Equation

Database performance is affected by many different variables - the running machine specs, OS, database engine and configuration, table schema and the queries running against it. Since I'm dealing mostly (only?) with MySQL, this article covers it mainly (though it is probably relevant to a large degree for other engines). As for OS and machine specs, I'll take them out of the equation as I'm interested in optimizing relative performance on the same machine.

Basically I'm interested in optimizing:

• The structure of my database tables (schema)
• The structure of my application-level queries (SELECT, UPDATE, INSERT, DELETE)

Profiling Database Performance

Blindly optimizing queries and database schema is counter-productive.  First we should know what we should be optimizing, and for that we need data. The act of gathering data for optimization is called profiling or performance analysis.

The first step I take when profiling database performance for a web app is to measure the running time of all the queries running in it. Absolute run time of a query is not necessarily a good measure of how optimized / performant it is, since some queries are naturally more complex or pull more data - It will give me a good idea however of where to start improving the response time of the application I'm optimizing. My main goal is not specific query performance, but overall system performance.

Measuring the run time of a query can be done with simple timers using microtime() calls (Check out this post for an abstraction), running it in a tool that automatically provides such statistics (phpMyAdmin for example) or using an integrated profiler.

I am using the Zend_Db_Profiler, which is convenient for me since I'm using the Zend Framework and all database access converges to a Zend_Db_Adapter connection. The profiler basically uses the microtime() approach but integrates it transparently into all of the queries without me having to wrap them one by one.

Usage is pretty simple. First you need to pass an extra parameter to your Zend_Db_Adapter instance to activate profiling:

$params = array(
    'host'     => 'localhost',
    'username' => 'dbusername',
    'password' => 'dbpassword',
    'dbname'   => 'dbname',
    'profiler' => true  // turn on profiler, disabled by default
);
 
$db = Zend_Db::factory('PDO_MYSQL', $params);

The Zend Framework manual shows some advanced usage (such as profiling directly into firebug, which is very convenient), however for our purposes we simply want to dump the queries and their run time.

$profiler = $db -> getProfiler();
 
$profile = '';
foreach($profiler -> getQueryProfiles() as $query) {
	$profile .= $query -> getQuery() . "\n"
	     . 'Time: ' . $query -> getElapsedSecs();
}
 
echo $profile;

This snippet goes of course after the view has been rendered and all queries executed.

Using a relevant dataset

It's important to know whether the data you are profiling against is relevant for when you believe you will start hitting performance issues. Granted, you can never really know when will that be, however it is always preferable to work with a dataset that resembles your (projected) production environment.

I will walk through a specific use-case I recently went through while preparing for the beta release of my startup, business platform Octabox (since defunct).

Running the profiler on one of the views in the application produced the following output on my development machine:

SELECT `history`.`id`, `history`.`type`, `history`.`action`, `tags`.`color`, `tags`.`id` AS `tag_id`
FROM `history`
LEFT JOIN `tags`
ON tags.id=history.tag_id AND tags.user_id=1
WHERE (history.user_id=1) AND (history.logged_on >= '2008-09-27 00:00:00')
ORDER BY `history`.`logged_on` DESC

Time: 0.00269

SELECT `tasks`.`id`, `tasks`.`task`,`lists`.`id` AS `list_id`
FROM `tasks`
INNER JOIN `lists` ON lists.id=tasks.list_id
WHERE (tasks.done IS NULL) AND (tasks.due IS NOT NULL) AND (tasks.user_id=1)
ORDER BY `tasks`.`due` ASC

Time: 0.000592

SELECT COUNT(*) AS `count`
FROM `dispatch`
WHERE (dispatch.user_id=1) AND (dispatch.STATUS=0)

Time: 0.00044

At first sight it would appear that the first query would be our immediate suspect for optimization, though the gains would not be great (completes in just over 0.002 seconds). Switching to a database I've prepared before hand with plenty of dummy data (~1.1Gb, some tables over 5M rows) the output looks very different:

SELECT `history`.`id`, `history`.`type`, `history`.`action`, `tags`.`color`, `tags`.`id` AS `tag_id`
FROM `history`
LEFT JOIN `tags`
ON tags.id=history.tag_id AND tags.user_id=1
WHERE (history.user_id=1) AND (history.logged_on >= '2008-09-27 00:00:00')
ORDER BY `history`.`logged_on` DESC

Time: 0.000426

SELECT `tasks`.`id`, `tasks`.`task`,`lists`.`id` AS `list_id`
FROM `tasks`
INNER JOIN `lists` ON lists.id=tasks.list_id
WHERE (tasks.done IS NULL) AND (tasks.due IS NOT NULL) AND (tasks.user_id=1)
ORDER BY `tasks`.`due` ASC

Time: 58.16 //Hmmm... problem?

SELECT COUNT(*) AS `count`
FROM `dispatch`
WHERE (dispatch.user_id=1) AND (dispatch.STATUS=0)

Time: 0.000322

A query that took just over 0.0005 seconds on a small database, took almost a minute to complete on a bigger one. Surprisingly enough the other queries remained blazingly fast (even faster than on my development machine with a small database, which is a credit to our server's power).

Optimizing The Errant Query

So I've found a very problematic query to say the least. Taking 58.1 seconds to complete is obviously not acceptable for any real time application. Running EXPLAIN against the query revealed the following execution plan:

The EXPLAIN results show the most immediate problem with the query - it was performing a join against every row in a 1.8M row table.

Examining the details of the query plan, it appears both selects are using an index, the first one limited by a WHERE (on the surface at least - since all rows were scanned). Since despite using an index the entire table was scanned very slowly, I tried to see the query will respond if I removed the index. Using IGNORE INDEX(user_id), I re-ran the query (this time in the Mysql command line):

304 rows in set (1.53 sec)

Wow, that's a big difference for not using an index. Running EXPLAIN on this reveals:

Despite using filesort to sort the query, it was running almost 30 times faster than before. By forcing it not to use an index which was not selective enough (i.e. not at all), the query ran a full table scan instead and completed much faster due to a better execution plan.

So we've improved greatly, but still not enough for my needs. Having one query taking over a second to complete when all the others are completing in sub 0.01 second times means it will be a bottleneck as the database grows (remember, the same query completed in just over 0.0005 seconds on a much smaller database).

Examining the structure of the table in question (tasks) revealed another interesting revelation - there is no index on the user_id column. So what happened? it appears there used to be an index on the user_id column, however it was removed just before all the dummy data was inserted into the database. Running ANALYZE TABLE tasks repaired the key information to the current state (no index on user_id).

I wanted to try the original query, this time with an actual index on the filtering column (user_id). After adding the index, I re-ran the query:

(304 total, Query took 0.0021 sec)

Much better! but I was not done. Running EXPLAIN yet again reveals the following execution plan:

So indeed, using an actual index was a big help, filtering the set to only 368 rows before processing the rest of the query. However, you'd notice it is running a filesort which we'd like to avoid. Preferably, I could set up an index combination that both filters and sorts using an index.

Creating an index composed of both the 'user_id' and 'due' columns got it right on the head, leading to:

(304 total, Query took 0.0008 sec)

Not as dramatic as previous improvements, but still three times better than the last iteration.

Final Words

The entire process took several hours of head scratching to arrive at all the results I've shown here. I managed to take a query that would cripple our server (58 seconds for one completion!) to an extremely fast one (0.0008 seconds against a 1.8M row table), through the use of some basic profiling and examining the execution plan.

Being an edge case (a non existent index used to filter against a 1.8M row table...), I gained valuable experience on potential pitfalls and on reading between the lines in the execution plan.

Some technical gibberish:

All queries were run with SQL_NO_CACHE to prevent caching from tampering with the results. I ran each query at least 5 times to make sure I was getting the right readings.

The specs of the server machine that ran the queries are as follows:

• Intel \ 2.4 GHz 1066FSB - Conroe \ Xeon 3060 (Dual Core)
• 2 x Generic \ 1024 MB \ DDR2 667 ECC
• 2 x Maxtor \ 146GB:SAS:10K RPM \ Atlas 10K - SAS
• CentOS Enterprise Linux - x86_64 - OS ES 5.0
• Running PHP 5.2.6 with MySQL 5.0.45 on Apache 2.2.4

First I'd like to expand on the (soft) release of Octabox - Octabox is a web platform for professional individuals and small businesses, providing a complete informartion management and collaboration environment. The concept was borne in the mind of my good friend and Octabox co-founder, Adam Benayoun, as the result of accumulated experience in building information management systems for many small businesses over the years.

Octabox is a cross between the vast functionality offered by application platform Salesforce and the simple and usable approach of basecamp (of 37signals). It is an application platform in the sense that it offers many on-demand applications for managing information (such as task management, contact relationship mangement, whiteboard/brainstorming and much more), but the focus is on keeping everything as simple and usable as possible to accomodate the needs of small business operations (all the way down to single individuals - freelancers, consultants and so forth). It is our belief that this sector is feeling the need for more powerful tools, yet is reluctant to use so called "enterprise" applications due to complexity and cost.

Octabox now enters a private beta phase to weed out issues and to figure what features are in demand so they'd be integrated in the public release. Hence, the "soft" release - the marketing effort has not begun yet (as can be seen by missing content on the site). We are handing out beta invitations if you register through our waiting list on the site. Blog readers get special treatment, so contact me through the contact form to secure your invitation .

Octabox by the way runs on PHP 5.2.6 with MySQL 5.1 with Zend Framework 1.6 as the abstraction layer. I have much to say about building a high-availability entreprise web-application using those tools, so stay tuned as I already have a couple article drafts on the pipeline on framework performance, mysql query optimizations, server setup and much more.

Still, a nice tool for web designers wishing to test their HTML and CSS layouts against several generations of IE, without having to resort to hacking multiple installations of different versions (such as multipleIE).

IETester [via LifeHacker]