MySQL 5.0 added the index_merge select type, which allows the query optimizer to possibly select several indexes from a single table and merge them to improve query performance. I say possibly, since leaving such decisions to the optimizer is risky at best. In fact, as I will show next, you are sometimes left with no indexes selected out of several possible options, resulting in a full table scan.

Continuing from my last post, I'll use a real-world example to show the different paths the queries optimizer can take when preparing an execution plan for our queries.

I'll actually be working with the same query I profiled last time, with a minor change. The relevant table structure is as follows:

--
-- Table structure for table `tasks`
--
 
CREATE TABLE IF NOT EXISTS `tasks` (
  `id` int(13) NOT NULL AUTO_INCREMENT,
  `list_id` int(13) NOT NULL,
  `user_id` int(13) NOT NULL,
  `task` varchar(255) collate utf8_unicode_ci NOT NULL,
  `due` timestamp NULL DEFAULT NULL,
  `created` timestamp NOT NULL DEFAULT CURRENT_TIMESTAMP,
  `checked` timestamp NULL DEFAULT NULL,
  `assigned` int(13) DEFAULT NULL,
  `done` enum('1') collate utf8_unicode_ci DEFAULT NULL,
  PRIMARY KEY  (`id`),
  KEY `user_id` (`user_id`,`due`),
  KEY `assigned` (`assigned`,`due`),
  KEY `due` (`due`)
) ENGINE=InnoDB

And the query itself:

SELECT `tasks`.`id`,
           `tasks`.`task`,
           UNIX_TIMESTAMP(due) AS `time`,
           `lists`.`name` AS `list_name`,
           `lists`.`id` AS `list_id`
FROM `tasks`
INNER JOIN `lists` ON lists.id=tasks.list_id
WHERE (tasks.user_id='1' OR assigned='1')
     AND (tasks.done IS NULL)
     AND (tasks.due IS NOT NULL)
ORDER BY `tasks`.`due` ASC

This query is almost the same as the one I worked with last time. The only difference being the first statement in the WHERE clause, involving an OR operator:

(tasks.user_id='1' OR assigned='1')

Just to understand what I'm doing here - The query selects from the tasks table with several filtering criteria. The last OR statement conveys the condition that the tasks selected either belong to specific user (i.e created by him) or assigned to him (those two possibly coincide).

For testing purposes I will be running the queries against a testing database I set up with plenty of mock data. The database is around 1Gb in total size, with the tasks table at about 1.8 million rows. It's not very large, but enough for significant data to be obtained while allowing relatively online tampering with schema (modifying keys takes only around 4 minutes to complete).

Running in original form computes as (average of 10 queries):

(304 total, Query took 6.84 sec)

6.84 sec is way too long for running a query in a typical web application. If you'd recall, I last got this query running at 0.0008 sec without the additional OR condition, meaning it is running at around 7,000 times slower (yikes). In such a case you would suspect the indexes are not selective enough, and sure enough an EXPLAIN reveals:

MySQL is performing a regular index select and not an index_merge as we'd like, and doing that it selects the worst possible index - the only one that doesn't filter the result set. Sure enough, all 1.8M rows are scanned and the query is underperforming badly. Trying to force the issue, I first add an IGNORE INDEX(due) to remove it from the equation:

(304 total, Query took 1.41 sec)

The query optimizer has decided not to use an index at all, despite several good candidates. However, query performance is improved since a full table scan with no index is performed straight up. It's still way too slow, and we'd like it to use a filtering index so it can avoid a full table scan. Trying to force the other indexes doesn't work so we're left with no choice but to try alternative query structures.

Our first candidate is replacing our OR condition with two UNION'ed select statements (the inspiration is from a couple of posts over at the MySQL performance blog). Breaking the original query into a UNION form results in:

(SELECT `tasks`.`id`,
            `tasks`.`task`,
            UNIX_TIMESTAMP(due) AS `time`,
            `lists`.`name` AS `list_name`,
            `lists`.`id` AS `list_id`
FROM `tasks`
INNER JOIN `lists` ON lists.id=tasks.list_id
WHERE (tasks.done IS NULL)
    AND (tasks.due IS NOT NULL)
    AND (tasks.user_id='1')
) UNION ALL (
 SELECT `tasks`.`id`,
            `tasks`.`task`,
            UNIX_TIMESTAMP(due) AS `time`,
            `lists`.`name` AS `list_name`,
            `lists`.`id` AS `list_id`
FROM `tasks`
INNER JOIN `lists` ON lists.id=tasks.list_id
WHERE (tasks.done IS NULL)
    AND (tasks.due IS NOT NULL)
    AND (assigned='1' AND tasks.user_id!='1')
) ORDER BY `time` ASC
 

This unsightly looking query looks much more complex, but actually gets the job done:

(304 total, Query took 0.0021 sec)

A big improvement to say the least (~3500 faster than the original query).
The explain shows why:

Only 304 rows are scanned (as opposed to the entire 1.8M table before). Despite the filesort, we are in the area of usability for this query.

Another approach would be to determine why the index_merging isn't take place and how can we enforce it. By reducing the WHERE clause to include only the columns covered by an index, index_merging kicks in resulting in a similar performance as the union, albeit with less selective filtering:

(368 total, Query took 0.0018 sec)

We could filter our result set in the application, but I'd like to keep it in the SQL where it belongs. In order to integrate the rest of the filtering statements, I use an IN condition on another index I have available - the primary key. This results in:

SELECT `tasks`.`id`,
           `tasks`.`task`,
           UNIX_TIMESTAMP(due) AS `time`,
           `lists`.`name` AS `list_name`,
           `lists`.`id` AS `list_id`
FROM `tasks`
INNER JOIN `lists` ON lists.id=tasks.list_id
WHERE (tasks.user_id='1' OR assigned='1')
   AND tasks.id IN
     (SELECT id
      FROM tasks
      WHERE (tasks.done IS NULL)
        AND (tasks.due IS NOT NULL)
     )
ORDER BY `tasks`.`due` ASC
 

Which brings us back to the performance levels of the UNION form:

(304 total, Query took 0.0021 sec)

And we can see in the EXPLAIN that it is indeed an index_merge operation:

So there you have it. From 6.83 seconds to 0.0021 seconds with some tweaking to the query structure. I am still not completely satisfied with the fact that it's using filesort, but I couldn't get it to use another index for the operation. Without the sorting the query is twice as fast:

(304 total, Query took 0.0010 sec)

So if it ever becomes an issue I could move sorting to the application code. Hopefully by then merge_index is better implemented in MySQL.

Regarding the query strucutre - personally I use the subquery format since it is more compact and maintainable. It is always good however to be familiar with all the alternatives.

The Performance Equation

Database performance is affected by many different variables - the running machine specs, OS, database engine and configuration, table schema and the queries running against it. Since I'm dealing mostly (only?) with MySQL, this article covers it mainly (though it is probably relevant to a large degree for other engines). As for OS and machine specs, I'll take them out of the equation as I'm interested in optimizing relative performance on the same machine.

Basically I'm interested in optimizing:

• The structure of my database tables (schema)
• The structure of my application-level queries (SELECT, UPDATE, INSERT, DELETE)

Profiling Database Performance

Blindly optimizing queries and database schema is counter-productive.  First we should know what we should be optimizing, and for that we need data. The act of gathering data for optimization is called profiling or performance analysis.

The first step I take when profiling database performance for a web app is to measure the running time of all the queries running in it. Absolute run time of a query is not necessarily a good measure of how optimized / performant it is, since some queries are naturally more complex or pull more data - It will give me a good idea however of where to start improving the response time of the application I'm optimizing. My main goal is not specific query performance, but overall system performance.

Measuring the run time of a query can be done with simple timers using microtime() calls (Check out this post for an abstraction), running it in a tool that automatically provides such statistics (phpMyAdmin for example) or using an integrated profiler.

I am using the Zend_Db_Profiler, which is convenient for me since I'm using the Zend Framework and all database access converges to a Zend_Db_Adapter connection. The profiler basically uses the microtime() approach but integrates it transparently into all of the queries without me having to wrap them one by one.

Usage is pretty simple. First you need to pass an extra parameter to your Zend_Db_Adapter instance to activate profiling:

$params = array(
    'host'     => 'localhost',
    'username' => 'dbusername',
    'password' => 'dbpassword',
    'dbname'   => 'dbname',
    'profiler' => true  // turn on profiler, disabled by default
);
 
$db = Zend_Db::factory('PDO_MYSQL', $params);

The Zend Framework manual shows some advanced usage (such as profiling directly into firebug, which is very convenient), however for our purposes we simply want to dump the queries and their run time.

$profiler = $db -> getProfiler();
 
$profile = '';
foreach($profiler -> getQueryProfiles() as $query) {
	$profile .= $query -> getQuery() . "\n"
	     . 'Time: ' . $query -> getElapsedSecs();
}
 
echo $profile;

This snippet goes of course after the view has been rendered and all queries executed.

Using a relevant dataset

It's important to know whether the data you are profiling against is relevant for when you believe you will start hitting performance issues. Granted, you can never really know when will that be, however it is always preferable to work with a dataset that resembles your (projected) production environment.

I will walk through a specific use-case I recently went through while preparing for the beta release of my startup, business platform Octabox.

Running the profiler on one of the views in the application produced the following output on my development machine:

SELECT `history`.`id`, `history`.`type`, `history`.`action`, `history`.`value`, UNIX_TIMESTAMP(history.logged_on) AS `time`, `history`.`user_id`, `history`.`user_name`, `history`.`parent_id`, `history`.`parent_type`, `history`.`parent_name`, `history`.`list_id`, `history`.`list_type`, `history`.`list_name`, `history`.`logging_user_id`, `tags`.`color`, `tags`.`id` AS `tag_id` FROM `history` LEFT JOIN `tags` ON tags.id=history.tag_id AND tags.user_id='1' WHERE (history.user_id=1) AND (history.logged_on >= '2008-09-27 00:00:00') ORDER BY `history`.`logged_on` DESC
Time: 0.0026991367340088

SELECT `tasks`.`id`, `tasks`.`task`, UNIX_TIMESTAMP(due) AS `time`, `lists`.`name` AS `list_name`, `lists`.`id` AS `list_id` FROM `tasks` INNER JOIN `lists` ON lists.id=tasks.list_id WHERE (tasks.done IS NULL) AND (tasks.due IS NOT NULL) AND (tasks.user_id='1') ORDER BY `tasks`.`due` ASC
Time: 0.00059294700622559

SELECT COUNT(*) AS `count` FROM `dispatch` WHERE (dispatch.user_id='1') AND (dispatch.status='0')
Time: 0.00044488906860352

SELECT COUNT(*) AS `count` FROM `conversations_to_users` WHERE (user_id='1' AND status='0')
Time: 0.00038599967956543

At first sight it would appear that the first query would be our immediate suspect for optimization, though the gains would not be great (completes in just over 0.002 seconds). Switching to a database I've prepared before hand with plenty of dummy data (~1.1Gb, some tables over 5M rows) the output looks very different:

That's right, there was no output. My development machine was hanging on this particular view when switched to the dummy database (I confirmed this was database related by running the queries independently in the MySQL command line). I switched over to our staging server to see if it can handle rendering this view against the dummy database. Luckily, it could and the suspect query immediately showed up in the output:

SELECT `history`.`id`, `history`.`type`, `history`.`action`, `history`.`value`, UNIX_TIMESTAMP(history.logged_on) AS `time`, `history`.`user_id`, `history`.`user_name`, `history`.`parent_id`, `history`.`parent_type`, `history`.`parent_name`, `history`.`list_id`, `history`.`list_type`, `history`.`list_name`, `history`.`logging_user_id`, `tags`.`color`, `tags`.`id` AS `tag_id` FROM `history`
LEFT JOIN `tags` ON tags.id=history.tag_id AND tags.user_id='1' WHERE (history.user_id=1) AND (history.logged_on >= '2008-09-27 00:00:00') ORDER BY `history`.`logged_on` DESC
Time: 0.00042605400085449

SELECT `tasks`.`id`, `tasks`.`task`, UNIX_TIMESTAMP(due) AS `time`, `lists`.`name` AS `list_name`, `lists`.`id` AS `list_id` FROM `tasks`
INNER JOIN `lists` ON lists.id=tasks.list_id WHERE (tasks.done IS NULL) AND (tasks.due IS NOT NULL) AND (tasks.user_id='1') ORDER BY `tasks`.`due` ASC
Time: 58.160161972046 //OMFG! what gives...

SELECT COUNT(*) AS `count` FROM `dispatch` WHERE (dispatch.user_id='1') AND (dispatch.status='0')
Time: 0.00032281875610352

SELECT COUNT(*) AS `count` FROM `conversations_to_users` WHERE (user_id='1' AND status='0')
Time: 0.00026297569274902

A query that took just over 0.0005 seconds on a small database, took almost a minute to complete on a bigger one. Surprisingly enough the other queries remained blazingly fast (even faster than on my development machine with a small database, which is a credit to our server's power).

Optimizing The Errant Query

So I've found a very problematic query to say the least. Taking 58.1 seconds to complete is obviously not acceptable for any real time application. Running EXPLAIN against the query revealed the following execution plan:

The EXPLAIN results show the most immediate problem with the query - it was performing a join against every row in a 1.8M row table.

Examining the details of the query plan, it appears both selects are using an index, the first one limited by a WHERE (on the surface at least - since all rows were scanned). Since despite using an index the entire table was scanned very slowly, I tried to see the query will respond if I removed the index. Using IGNORE INDEX(user_id), I re-ran the query (this time in the Mysql command line):

304 rows in set (1.53 sec)

Wow, that's a big difference for not using an index. Running EXPLAIN on this reveals:

Despite using filesort to sort the query, it was running almost 30 times faster than before. By forcing it not to use an index which was not selective enough (i.e. not at all), the query ran a full table scan instead and completed much faster due to a better execution plan.

So we've improved greatly, but still not enough for my needs. Having one query taking over a second to complete when all the others are completing in sub 0.01 second times means it will be a bottleneck as the database grows (remember, the same query completed in just over 0.0005 seconds on a much smaller database).

Examining the structure of the table in question (tasks) revealed another interesting revealation - there is no index on the user_id column. So what happened? it appears there used to be an index on the user_id column, however it was removed just before all the dummy data was inserted into the database. Running ANALYZE TABLE tasks repaired the key information to the current state (no index on user_id).

I wanted to try the original query, this time with an actual index on the filtering column (user_id). After adding the index, I re-ran the query:

(304 total, Query took 0.0021 sec)

Much better! but I was not done. Running EXPLAIN yet again reveals the following execution plan:

So indeed, using an actual index was a big help, filtering the set to only 368 rows before processing the rest of the query. However, you'd notice it is running a filesort which we'd like to avoid. Preferably, I could set up an index combination that both filters and sorts using an index.

Creating an index composed of both the 'user_id' and 'due' columns got it right on the head, leading to:

(304 total, Query took 0.0008 sec)

Not as dramatic as previous improvements, but still three times better than the last iteration.

Final Words

The entire process took several hours of head scratching to arrive at all the results I've shown here. I managed to take a query that would cripple our server (58 seconds for one completion!) to an extremely fast one (0.0008 seconds against a 1.8M row table), through the use of some basic profiling and examining the execution plan.

Being an edge case (a non existent index used to filter against a 1.8M row table...), I gained valuable experience on potential pitfalls and on reading between the lines in the execution plan.

Some technical gibberish:

All queries were run with SQL_NO_CACHE to prevent caching from tampering with the results. I ran each query at least 5 times to make sure I was getting the right readings.

The specs of the server machine that ran the queries are as follows:

• Intel \ 2.4 GHz 1066FSB - Conroe \ Xeon 3060 (Dual Core)
• 2 x Generic \ 1024 MB \ DDR2 667 ECC
• 2 x Maxtor \ 146GB:SAS:10K RPM \ Atlas 10K - SAS
• CentOS Enterprise Linux - x86_64 - OS ES 5.0
• Running PHP 5.2.6 with MySQL 5.0.45 on Apache 2.2.4

A web browser built on the webkit engine (same as Safari), Chrome offers a simple UI and extensive support for web technologies. Having used it for a couple of days now, it is striking to me how obvious it is that Google is a web company - in bold contrast to another software giant currently pushing for their next-gen browser.

There is a lot to like about Chrome: It is open-source. The tab oriented UI is a great innovation. The simple UI allows for an impressively large work area in the browser. It comes with a built in DOM inspector and Javascript debugger (almost rivaling Firebug for functionality). The tab/process manager is an awesome feature (though a possible source of contention). And most importantly - aside from minor inconsistencies from the standards (shared by Safari), all web sites render perfectly under Chrome.

I was somehow surprised though it only reaches a score of 79 on the acid3 test, since it is based on an engine that already scored a perfect 100 (webkit). For comparison, Firefox 3.0.1 achieves a score of 71 (though with less graphical glitches), IE7 achieves a measly score of 13 in about triple the time, and IE8 beta 2 just barely beats that with a very low 21.

It is my personal hope that Chrome steals enough market share from Microsoft to help push out older IE versions, thus catapulting the web forward. This has been a fantastic move by Google, and the online world is watching to see how the market will respond. Is a Google OS next on the agenda?

1. Adding and encoding email headers
2. Protection against header injection
3. Multiple mail transports (SMTP is useful for sending multiple mails)
4. Creating and handling mime-compliant multipart messages
5. Composing HTML emails
6. Reading mail boxes

Sending mails

Using Zend_Mail is extremely straightforward: (Taken from the ZF documentation)

 require_once 'Zend/Mail.php';
$mail = new Zend_Mail();
$mail->setBodyText('This is the text of the mail.'); //Alternatively using setBodyHtml for HTML messages
$mail->setFrom('somebody@example.com', 'Some Sender'); //Adds a 'from' header
$mail->addTo('somebody_else@example.com', 'Some Recipient');
$mail->setSubject('TestSubject');
$mail->send();

Simple.

For sending multiple mails it is better to use the SMTP protocol for performance reasons. Doing this manually with PHP is quite elaborate however, but with Zend_Mail it is almost as simple as before:

 require_once 'Zend/Mail.php';
 
// Create transport
require_once 'Zend/Mail/Transport/Smtp.php';
$transport = new Zend_Mail_Transport_Smtp('localhost');
 
// Sending out multiple mails at once
for ($i = 0; $i > 5; $i++) {
    $mail = new Zend_Mail();
    $mail->addTo('studio@peptolab.com', 'Test');
    $mail->setFrom('studio@peptolab.com', 'Test');
    $mail->setSubject('Demonstration - Sending Multiple Mails per SMTP Connection');
    $mail->setBodyText('...Your message here...');
    $mail->send($transport); //Using the SMTP transport for sending the mail
}

* SMTP connections can be secured with SSL or TLS for sending sensitive mails.

This procedure reuses the SMTP connection for the lifetime of the entire process which is much better than sending out mails using the mail() function.

Reading mail boxes

Reading mail boxes is similarly simple. Zend_Mail supports POP3 and IMAP for remote connections, and Mbox and Maildir for local connections (mail boxes residing on the same server). Sample usage:

$mail = new Zend_Mail_Storage_Pop3( //Configuring with host and credentials
	array('host' => 'example.com',
	      'user' => 'test',
	      'password' => 'test'
	)
);
 
echo $mail->countMessages() . " messages found\n"; //Outputting message count in mail box
foreach ($mail as $message) { //Outputting message subjects
    echo "Mail from " . $message->from . ": " . $message->subject . "\n";
}

Using the different storage options is very similar. Messages returned are instances of the Zend_Mail_Message class, which provides an API for manipulating mail messages.

Advanced Usage

Working with multipart MIME messages is very tedious. The MIME specifications are rather complex and MIME messages aren't very readable to humans, and therefor we would like to be able to use Zend_Mail_Message API to reduce the pains of dealing with MIME messages. There are several general cases when working with MIME messages:

1. Working with messages which are stored as strings in a database
2. Working with messages which are read from files
3. Working with incoming messages directly (from an input stream)

Zend_Mail_Messsage can receive either a raw message string in its constructor, a file name or a file handle. Examples of instancing a Zend_Mail_Message object with all three:

//From a raw string
$message = new Zend_Mail_Message(array('raw' => $rawMessage)); 
 
//From a file name
$message = new Zend_Mail_Message(array('file' => '/path/to/mail/message'));
 
//From a file handle
$handle = fopen('/path/to/mail/message');
$message = new Zend_Mail_Message(array('file' => $handle));

The last one might seem a bit redundant. However it opens up creative ways to intercept mail directly using the input wrapper:

$handle = fopen("php://stdin", "r");
$message = new Zend_Mail_Message(array('file' => $handle));

The tricky part is directing the mail message stream into the script. You can read more about that in this nice tutorial from evolt.

Some minor bugs and fixes

Nothing is perfect however. Current version of Zend_Mail (shipped with the Zend_Framework ver. 1.5.2) has a buggy implementation of the headers encoding. This will become noticeable when working with UTF encoded mails. Fortunately there is a quick fix for that posted in the ZF issue tracker (and hopefully will be tested and integrated in the next release).

Direct usage statistics are hard to ascertain with great precision, but they revolve around 4%-6% of the total Internet population, which is not much. More feeds are being consumed indirectly by aggregation sites, such as my yahoo and iGoogle (as this Yahoo! paper shows), which shows that there is market ready to consume more feeds.

So why feed usage isn't more widespread?

1. Lack of awareness
Most people are simply unaware of the benefits of using feeds. When asked about what are feeds, only 12% of the Internet public were aware of its meaning. I will admit myself to be one of the ignorant up until not too long ago (about 6 months). For me that is mind boggling - I consider myself an extremely technical user and I love the Internet. How was I not aware of how useful this technology is?

2. Confusing terminology
What is a feed? syndication? aggregators? to someone hearing those terms for the first time it can be very discouraging. There is alternative terminology such as subscriptions and readers which is more similar to natural language but it isn't used enough to explain the technology.

3. Competing standards
Atom and RSS are competing standards for web syndication. Both are offering basically the same solution, so why haven't they merged yet? (actually there are reasons, but not very good ones). Most sites now offer both, which only add to the confusion as the names of the different formats are sometimes use interchangeably with the term Feed.

Content syndication is just waiting to erupt. Someone needs to take initiative and promote this technology to the masses:

1. Create awareness
The same way open standards such as OpenID made a splash some time ago, content syndication needs visibility. And in contrast to open standards, web syndication is useful now.

2. Improve description
Hand in hand in creating visibility, a low-tech explanation of the benefits of syndication should be drafted and used liberally. This will go a long way to break the barriers of adoption.

3. Unify standards
One to rule them all as they say. Since this is an open standard and certainly not for profit, the only real consideration would be backwards compatibility. Declare RSS officially as the standby for compatibility and Atom as the active standard for future development.

4. Do not call the technology by its standards name!
Stop using RSS or Atom feeds as the name for the subscription feature. People are much more likely to hit 'Subscribe' than 'Join my RSS feeds'...

P.S.
Syndication always reminds me of the PC classic 'Syndicate' in which rival clans fight for world domination in a futuristic setting using cyborg agents. Good times...

It's interesting to note that Dojo is not even featured on this list, though if you check the comments you'll see that it is used partially at the apple site (in the apple store). This makes me wonder even more regarding Zend's latest decision to integrate Dojo into their framework. As I commented in that release statement (comment #6 is mine), I feel that Dojo isn't appropriate as the default for the framework as it is more complex and much less documented than the other top frameworks.

As a long time jQuery developer I have no intentions of integrating a new Javascript library into my development environment, so I'm obviously biased. I still feel as though Zend missed an opportunity here to better cater to the needs of a broader user base and instead chose to prioritize its partners best interests.

I had believed I figured out most of the conventions used in the jQuery source code, however a recent addition has been bugging me and I could not find a reasonable explanation for it - I'm talking about the mysterious semi-colons appearing at the beginning of some of the source files in the library. What is its purpose? Does it make the closure invisible to giant robots from outer space? I had no leads to go on.

This blog post by the jQuery.rule team however, reveals the truth about the semi-colon debacle - apparently they're used for safe file concatenation (string join). Well that's a load off my chest. You learn something new every day.

In related news, jQuery UI 1.5 has been officially released, says the jQuery enquirer. jQuery 1.5 is an extensive UI oriented extension to jQuery, and version 1.5 bring forth many improvements such as a tighter API, an effects library called enchant, a skinning mechanism and plenty of bug fixes. I'm just excited they finally updated their documentation, as I've been using it for a while going only by source code.

Paginated queries typically look like this:

SELECT * FROM articles LIMIT 20, 10 //Selecting page 3 FROM TABLE articles (10 rows per page)

In order to generate page navigation, we would need to know the number of pages. We count the total rows that our query generates without the limit clause:

SELECT COUNT(*) FROM articles //Count ALL rows IN TABLE articles

And then we can perform simple arithmetics to calculate the number of pages.

It's relatively straightforward when dealing with simple queries, however for complicated queries (with multiple JOIN clauses) it can be quite a chore.

Lets switch to the Zend Framework and compose ourselves a somewhat elaborate query using Zend_Db_Select:

$select = new Zend_Db_Select();
$select -> from(array('pil' => 'project_item_list'),array('*'));
	-> join(array('pt' => 'project_tasks'),'pt.id=pil.task_id',array('task','ptid' => 'id'));
	-> join('projects','projects.id=pt.project_id',array());
	-> join(array('ptu' => 'projects_to_users'),'ptu.project_id=projects.id',array());
	-> join('users','users.id=pil.assigned_to',array('assignee' => 'name'));
	-> where("pt.project_id='" . $projectId . "'");
	-> where("ptu.user_id='" . $userId . "'");
	-> where("task_id='" . $taskId . "'");
	-> where("pil.assigned_to='" . $assignedTo . "'")
	-> limitPage(3,10);

Plenty of JOIN goodness. You'll notice the limitPage() call at the end, which is a nice Zend_Db_Select helper to add a limit clause by a page number (in this case it means limit by page 3, 10 rows per page).

So how do we count the number of rows? We'd want to have a COUNT() function returned from the base table (the one that appears at the from clause), however if we mix that with other columns, we'd have to use GROUP BY which would mess up our result-set.

There are two ways to handle this - generating a similar query sans the columns and the limit clauses, or using the SQL_CALC_FOUND_ROWS() function. The first approach results in two seperate queries while the second is just one query, however it is not that obvious which one is better for performance. For our case we'll use the first method, duplicating the original query while stripping out whats needed.

We could recompose the entire query using copy-paste methods, removing the columns and the limit clauses, but we would like a more general solution. I extend Zend_Db_Select and add a simple utility method:

class Techfounder_Db_Select extends Zend_Db_Select {
	public function prepareForCount($countColName = 'count') {
		$this -> reset(Zend_Db_Select::COLUMNS);
		$this -> reset(Zend_Db_Select::LIMIT_COUNT);
		$this -> reset(Zend_Db_Select::LIMIT_OFFSET);
		$table = key($this -> _parts['from']);
		$this -> _parts['columns'] = array(
			0 => array(
				0 => $table,
				1 => new Zend_Db_Expr('COUNT(*)'),
				2 => $countColName
			)
		);
		return $this;
	}
}

The prepareForCount() method modifies a select object to strip all previously set columns and LIMIT clauses, and to set a COUNT() function on the base table in the query (the row count is returned as a key named 'count'). We can now use this Select class to do the following:

$select = new Techfounder_Db_Select();
... //Multiple JOIN query, as composed previously
 
$countSelect = clone $select;
$countSelect -> prepareForCount();
$result = $db -> fetchRow($countSelect); //$db is an instance of Zend_Db_Adapter
$pageCount = ((int)(($result['count'] - 1) / $rowCount)) + 1; //Using the number of rows to calculate the number of pages.
//$rowCount is the number of rows per page

I go even a step further, and encapsulate this short process in a countPages() method in a custom wrapper for Zend_Db_Select (called Zend_Db_Gateway, for which I submitted a proposal and would talk about in my conclusion to my Models in the Zend Framework series).

In addition, jQuery UI has recently reached release-candidate status, which means all non minor or trivial issues were solved and it's getting to the point that it will production ready shortly. I have been using its components for a while in anticipation of such release, and hopefully they will have their documentation up soon.

Good times to be jQuery developers

I never knew it had so many to be honest. You know why? because I never bothered to go to the function index... PHP.net has a strategically located search bar in the top-right corner. Every time I needed to find a function I got to it from there or google. And it was always easy finding the right function (I guess the names aren't that bad after all).

Why is function diversity bad? While I agree that the naming conventions could be better, how should this be a problem to anyone? Unless of course you want to browse the function index...

It seems to me non-PHP developers are much more prone to deride PHP than PHP developers are to do the same for other languages. I guess it's because most PHP developers have no experience in other languages (expect maybe C/C++, which are not very relevant for compaison), so they don't have credentials to offer such criticism. Further more, it's more common to hear non-PHP developers complain about their own languages.

I have always maintained that the server-side language is irrelevant in the end. You can implement any web-based service / site / application in any of the modern web languages, it all depends on your development team skills. The user doesn't care what runs on the server, only what he is getting in the browser. And for all you detractors, yes you can build maintainable, secure and elegant applications in PHP quickly if you know what you are doing.

Having said that, in praise of PHP - It's open source and free, easy deployment (low-dependencies, simple packaging), great community (that has produced many professional-grade open-source packages), great documentation, available on most servers, dynamically typed and is easy to use. Also, from personal experience a PHP-dev in general has better command of client-side markup (HTML and CSS), as PHP blends easily with the markup and such blend is shown often in basic tutorials.

I liked this account by a guy who went back from ROR to PHP, as it puts the emphasis on the developer's skills instead of the language choice. Let everybody pick their favorite, why can't we all just get along?

(And by the way... RoR is a framework for Ruby, would people stop comparing it to languages? Thank you.)