Search-and-replace Vs. Include

As the title suggests, there are two prevalent forms of PHP templating, each with its own cons and pros:

1. Search-and-replace systems (to which Smarty belongs) involve running string replacement filters on a template, replacing placeholders with dynamic content.Search-and-replace systems suffer from parsing overhead which can be significant depending on the amount of content being parsed. More advanced systems in this category attempt to offset the overhead by caching parse results (Smarty for example relies heavily on caching for its performance), but this introduces the concern of maintaining the cache (determining when it becomes stale).

   Search-and-replace systems are also better suited for serving mostly static content, as regenerating content dynamically incurs the parsing overhead. Hence it is more suited for content web-sites than highly dynamic web-applications, in which every user sees a different page than the others, limiting the effectiveness of caching.

2. Include systems involve including PHP scripts which serve as templates, and passing variables directly to those. Wordpress for example uses a basic implementation of such a system for its templating mechanism.Include systems suffer much less overhead for parsing templates than search-and-replace schemes, simply due to the fact that no such action takes place.

   On the other hand, include systems are prone to scoping problems. If templates are included in the normal flow of the application, then they share and pollute the scope they are in - meaning they could affect data in other parts of the application which could lead to weird behavior and decrease in reusability of templates.

Templating with Objects

Ideally, we would like to combine the pros and solve the cons of both systems. We need a way to pass an arbitrary number of variables into a template without polluting the general scope of the application, while avoiding the parsing overhead of search-and-replace routines.

Fortunately PHP presents us with such a construct - Classes. Class methods are different from regular functions in that they have a shared local scope always available - the object instance ($this). Recent PHP templating systems use this property of classes to their advantage, creating a separate scope for templates thereby allowing them to process without affecting the rest of the application.

Savant is a good example of such a system, and so is the View component of the Zend Framework. I’ll be borrowing concepts from both to show how a simple implementation of such a system can be built.

In Code

In the most basic form, a templating system built with the principles I’ve mentioned above will look like this:

class View {
     public function render($script) {
        ob_start();
        $this->_include($script);
        return ob_get_clean();
    }

    public function __get($key) {
    	return (isset($this -> $key) ? $this -> $key : null);
    }

    protected function _include() {
        include func_get_arg(0);
    }
}

Usage is pretty simple - load some vars into a View class instance and pass a template into the render method. For a template that looks like:

<h2><?php echo $this -> name; ?></h2>
<p>Current position: <?php echo $this -> job; ?></p>

Passing parameters into a view instance:

$view = new View();

$view -> name = 'Obama';
$view -> job = 'President of the USA';

echo $view -> render('/path/to/template.phtml');

Will result in:

<h2>Obama</h2>
<p>Current position: President of the USA</p>

We can also invoke the render() method inside a template to chain several templates together:

<h2><?php echo $this -> name; ?></h2>
<p>Current position: <?php echo $this -> position; ?></p>

<?php echo $this -> render('footer.phtml'); ?>

‘footer.phtml’ being another separate template file.

In Review

Lets review what the class does:

     public function render($script) {
        ob_start();
        $this->_include($script);
        return ob_get_clean();
    }

The render() method receives a $script parameter which is the path to a template script. It passes that parameter to separate method called _include() and captures the output using output buffering.

Capturing the output into a string instead of outputting to the screen at once is very useful if we want to perform this as a part of a complete process, such as preparing headers and starting sessions(), which has to happen before any output is sent to the browser. It is also useful in case we want to perform additional transformations on the output before sending it to the browser (such as parsing and filtering).

    protected function _include() {
        include func_get_arg(0);
    }

The _include() method is the core of this templating approach. By including the template inside a class method with no arguments (calling func_get_arg() instead), the included script has only the object instance ($this) as the available scope to get information from and pass information to. Any variables we declare inside this scope have no bearing on the outside and so collisions and scoping issues are avoided.

     public function __get($key) {
    	return (isset($this -> $key) ? $this -> $key : null);
    }

This is really a utility method to prevent PHP from throwing a ‘variable is not defined’ notice in case the template attempts to access a non existing parameter.

Complete solutions like Zend_View and Savant offer many more features, such as plug-ins, filters, base directory configuration and more, but the principle is the same. While the same structure could be mimicked to some degree using procedural functions only, the simple and convenient interface of passing and retrieving parameter to/from templates is unique to this approach - making it the clear winner in my opinion.

As it might be expected, there are several techniques to optimize the delivery of web pages. The Exceptional Performance guide by Yahoo is a great resource for a multitude of optimizations practices, including specifically two techniques which I will address in this article - script minifcation and concatenation.

Reducing script size and the number of request

Modern web sites and web application serve increasingly heavier javascript and CSS files. The price for the increase in script payload is felt both on the client side, where loading time increase making pages feel slugish, and on the server side, where bandwidth and requests are at a premium.

Javascript files especially block the browser while downloading (no more than 2 components can be downloaded simultaneously as per HTML specifications), and are the type of scripts seeing the steepest increase in size in the last several years.

Minification and concatenation

Minifying is the process of removing all unnecessary characters from the source code without changing its functionality (removing whitespaces, line-breaks and comments, replacing variables names and more). Concatenation is the process of unifying two strings into one. Applying both to our scripts we reduce total script size and the amount of requests.

There are several public algorithms for minifying Javascript, and Dean Edwards gives a nice overview of those on his blog.

For PHP we have the handy Minify library, which handles both the minifcation and concatenation of scripts. Minify is quite robust, and comes with a set of standalone classes that can be used in other contexts as well. It also sets Etags and far-future headers for better client script caching.

Deploying Minify

Downloading and extracting the package available on google code creates several text files and two sub-directories. The directory that is of interest to us is ‘/min’ which contains the actual source code, and we’ll place it somewhere on our server about our document root (so it can be accessed by a regular http request).

The basic workflow of using Minify is to replace our <script> and <link> tags which load our javascript and CSS respectively with calls to the minify script. We need to tell the Minify script which scripts to process, and there are two main ways to accomplish that:

1. Passing the scripts as parameters in the call to the minify script. We pass the path to the scripts separated by commas, translating the following:

   <script type="text/javascript" src="/path/to/script_one.js"></script>
   <script type="text/javascript" src="/path/to/script_two.js"></script>
   

   Into:

   <script type="text/javascript" src="/path/to/min/?f=/path/to/script_one.js,/path/to/script_two.js"></script>
   

   There is actually a shorter way writing this, by using a common base path for both scripts:

   <script type="text/javascript" src="/path/to/min/?b=path/to&f=script_one.js,script_two.js"></script>
   

   Provided that all the paths are correct, we have successful concatenated two scripts into one minified script. This can repeated for as many scripts as we like, though there is an artificial limit (due to performance and memory limitations I assume) of 10 scripts per Minify request. This limit can be adjusted in the configuration file which exists at the base path of the library (config.php).

2. Using pre-defined script arrays. This method is less flexible as it requires hardcoding the script names into an array, but is slightly more performant. Minify calls those array ‘groups’, which look something like:

   return array(
       'js' => array('//path/to/script_one.js', '//path/to/script_two.js')
   );
   

   You can find an example of those in the installation folder of Minify (groupsConfig.php). The request to the script is then made by specifying the group name:

   <script type="text/javascript" src="/path/to/min/?g=js"></script>
   

   There are several options when using groups, and you can read on those on the library’s wiki.

Its important to set up the cache folder for Minify. Minifying the scripts can actually take a couple of seconds, which is a relatively long delay - however once generated once they will be cached until the scripts change (provided the cache folder is set-up properly).

To declare the cache folder, simply define a variable named $min_cachePath in the configuration file or the main app file before the main application start.

$min_cachePath = '/path/to/cache';

Make sure the folder exists and has write permissions by PHP.

Real world use

I’ve implemented Minify in all of my recent projects to great effect. An extreme case would be script deployment in my current startup, Octabox web platform. Being a heavy-duty web-application, it consumes much more javascript and styles than your average marketing site.

Without Minify

Javascript requests:

CSS requests:

With Minify

Javascript requests:

CSS requests:

51 total requests weighing 540kb reduced to 5 requests weighing 116kb.

MySQL 5.0 added the index_merge select type, which allows the query optimizer to possibly select several indexes from a single table and merge them to improve query performance. I say possibly, since leaving such decisions to the optimizer is risky at best. In fact, as I will show next, you are sometimes left with no indexes selected out of several possible options, resulting in a full table scan.

Continuing from my last post, I’ll use a real-world example to show the different paths the queries optimizer can take when preparing an execution plan for our queries.

I’ll actually be working with the same query I profiled last time, with a minor change. The relevant table structure is as follows:

--
-- Table structure for table `tasks`
--

CREATE TABLE IF NOT EXISTS `tasks` (
  `id` int(13) NOT NULL auto_increment,
  `list_id` int(13) NOT NULL,
  `user_id` int(13) NOT NULL,
  `task` varchar(255) collate utf8_unicode_ci NOT NULL,
  `due` timestamp NULL default NULL,
  `created` timestamp NOT NULL default CURRENT_TIMESTAMP,
  `checked` timestamp NULL default NULL,
  `assigned` int(13) default NULL,
  `done` enum('1') collate utf8_unicode_ci default NULL,
  PRIMARY KEY  (`id`),
  KEY `user_id` (`user_id`,`due`),
  KEY `assigned` (`assigned`,`due`),
  KEY `due` (`due`)
) ENGINE=InnoDB

And the query itself:

SELECT `tasks`.`id`,
           `tasks`.`task`,
           UNIX_TIMESTAMP(due) AS `time`,
           `lists`.`name` AS `list_name`,
           `lists`.`id` AS `list_id`
FROM `tasks`
INNER JOIN `lists` ON lists.id=tasks.list_id
WHERE (tasks.user_id='1' OR assigned='1')
     AND (tasks.done IS NULL)
     AND (tasks.due IS NOT NULL)
ORDER BY `tasks`.`due` ASC

This query is almost the same as the one I worked with last time. The only difference being the first statement in the WHERE clause, involving an OR operator:

(tasks.user_id=’1′ OR assigned=’1′)

Just to understand what I’m doing here - The query selects from the tasks table with several filtering criteria. The last OR statement conveys the condition that the tasks selected either belong to specific user (i.e created by him) or assigned to him (those two possibly coincide).

For testing purposes I will be running the queries against a testing database I set up with plenty of mock data. The database is around 1Gb in total size, with the tasks table at about 1.8 million rows. It’s not very large, but enough for significant data to be obtained while allowing relatively online tampering with schema (modifying keys takes only around 4 minutes to complete).

Running in original form computes as (average of 10 queries):

(304 total, Query took 6.84 sec)

6.84 sec is way too long for running a query in a typical web application. If you’d recall, I last got this query running at 0.0008 sec without the additional OR condition, meaning it is running at around 7,000 times slower (yikes). In such a case you would suspect the indexes are not selective enough, and sure enough an EXPLAIN reveals:

MySQL is performing a regular index select and not an index_merge as we’d like, and doing that it selects the worst possible index - the only one that doesn’t filter the result set. Sure enough, all 1.8M rows are scanned and the query is underperforming badly. Trying to force the issue, I first add an IGNORE INDEX(due) to remove it from the equation:

(304 total, Query took 1.41 sec)

The query optimizer has decided not to use an index at all, despite several good candidates. However, query performance is improved since a full table scan with no index is performed straight up. It’s still way too slow, and we’d like it to use a filtering index so it can avoid a full table scan. Trying to force the other indexes doesn’t work so we’re left with no choice but to try alternative query structures.

Our first candidate is replacing our OR condition with two UNION’ed select statements (the inspiration is from a couple of posts over at the MySQL performance blog). Breaking the original query into a UNION form results in:

(SELECT `tasks`.`id`,
            `tasks`.`task`,
            UNIX_TIMESTAMP(due) AS `time`,
            `lists`.`name` AS `list_name`,
            `lists`.`id` AS `list_id`
FROM `tasks`
INNER JOIN `lists` ON lists.id=tasks.list_id
WHERE (tasks.done IS NULL)
    AND (tasks.due IS NOT NULL)
    AND (tasks.user_id='1')
) UNION ALL (
 SELECT `tasks`.`id`,
            `tasks`.`task`,
            UNIX_TIMESTAMP(due) AS `time`,
            `lists`.`name` AS `list_name`,
            `lists`.`id` AS `list_id`
FROM `tasks`
INNER JOIN `lists` ON lists.id=tasks.list_id
WHERE (tasks.done IS NULL)
    AND (tasks.due IS NOT NULL)
    AND (assigned='1' AND tasks.user_id!='1')
) ORDER BY `time` ASC

This unsightly looking query looks much more complex, but actually gets the job done:

(304 total, Query took 0.0021 sec)

A big improvement to say the least (~3500 faster than the original query).
The explain shows why:

Only 304 rows are scanned (as opposed to the entire 1.8M table before). Despite the filesort, we are in the area of usability for this query.

Another approach would be to determine why the index_merging isn’t take place and how can we enforce it. By reducing the WHERE clause to include only the columns covered by an index, index_merging kicks in resulting in a similar performance as the union, albeit with less selective filtering:

(368 total, Query took 0.0018 sec)

We could filter our result set in the application, but I’d like to keep it in the SQL where it belongs. In order to integrate the rest of the filtering statements, I use an IN condition on another index I have available - the primary key. This results in:

SELECT `tasks`.`id`,
           `tasks`.`task`,
           UNIX_TIMESTAMP(due) AS `time`,
           `lists`.`name` AS `list_name`,
           `lists`.`id` AS `list_id`
FROM `tasks`
INNER JOIN `lists` ON lists.id=tasks.list_id
WHERE (tasks.user_id='1' OR assigned='1')
   AND tasks.id IN
     (SELECT id
      FROM tasks
      WHERE (tasks.done IS NULL)
        AND (tasks.due IS NOT NULL)
     )
ORDER BY `tasks`.`due` ASC

Which brings us back to the performance levels of the UNION form:

(304 total, Query took 0.0021 sec)

And we can see in the EXPLAIN that it is indeed an index_merge operation:

So there you have it. From 6.83 seconds to 0.0021 seconds with some tweaking to the query structure. I am still not completely satisfied with the fact that it’s using filesort, but I couldn’t get it to use another index for the operation. Without the sorting the query is twice as fast:

(304 total, Query took 0.0010 sec)

So if it ever becomes an issue I could move sorting to the application code. Hopefully by then merge_index is better implemented in MySQL.

Regarding the query strucutre - personally I use the subquery format since it is more compact and maintainable. It is always good however to be familiar with all the alternatives.

The Performance Equation

Database performance is affected by many different variables - the running machine specs, OS, database engine and configuration, table schema and the queries running against it. Since I’m dealing mostly (only?) with MySQL, this article covers it mainly (though it is probably relevant to a large degree for other engines). As for OS and machine specs, I’ll take them out of the equation as I’m interested in optimizing relative performance on the same machine.

Basically I’m interested in optimizing:

• The structure of my database tables (schema)
• The structure of my application-level queries (SELECT, UPDATE, INSERT, DELETE)

Profiling Database Performance

Blindly optimizing queries and database schema is counter-productive.  First we should know what we should be optimizing, and for that we need data. The act of gathering data for optimization is called profiling or performance analysis.

The first step I take when profiling database performance for a web app is to measure the running time of all the queries running in it. Absolute run time of a query is not necessarily a good measure of how optimized / performant it is, since some queries are naturally more complex or pull more data - It will give me a good idea however of where to start improving the response time of the application I’m optimizing. My main goal is not specific query performance, but overall system performance.

Measuring the run time of a query can be done with simple timers using microtime() calls (Check out this post for an abstraction), running it in a tool that automatically provides such statistics (phpMyAdmin for example) or using an integrated profiler.

I am using the Zend_Db_Profiler, which is convenient for me since I’m using the Zend Framework and all database access converges to a Zend_Db_Adapter connection. The profiler basically uses the microtime() approach but integrates it transparently into all of the queries without me having to wrap them one by one.

Usage is pretty simple. First you need to pass an extra parameter to your Zend_Db_Adapter instance to activate profiling:

$params = array(
    'host'     => 'localhost',
    'username' => 'dbusername',
    'password' => 'dbpassword',
    'dbname'   => 'dbname',
    'profiler' => true  // turn on profiler, disabled by default
);

$db = Zend_Db::factory('PDO_MYSQL', $params);

The Zend Framework manual shows some advanced usage (such as profiling directly into firebug, which is very convenient), however for our purposes we simply want to dump the queries and their run time.

$profiler = $db -> getProfiler();

$profile = '';
foreach($profiler -> getQueryProfiles() as $query) {
	$profile .= $query -> getQuery() . "\n"
	     . 'Time: ' . $query -> getElapsedSecs();
}

echo $profile;

This snippet goes of course after the view has been rendered and all queries executed.

Using a relevant dataset

It’s important to know whether the data you are profiling against is relevant for when you believe you will start hitting performance issues. Granted, you can never really know when will that be, however it is always preferable to work with a dataset that resembles your (projected) production environment.

I will walk through a specific use-case I recently went through while preparing for the beta release of my startup, business platform Octabox.

Running the profiler on one of the views in the application produced the following output on my development machine:

SELECT `history`.`id`, `history`.`type`, `history`.`action`, `history`.`value`, UNIX_TIMESTAMP(history.logged_on) AS `time`, `history`.`user_id`, `history`.`user_name`, `history`.`parent_id`, `history`.`parent_type`, `history`.`parent_name`, `history`.`list_id`, `history`.`list_type`, `history`.`list_name`, `history`.`logging_user_id`, `tags`.`color`, `tags`.`id` AS `tag_id` FROM `history` LEFT JOIN `tags` ON tags.id=history.tag_id AND tags.user_id=’1′ WHERE (history.user_id=1) AND (history.logged_on >= ‘2008-09-27 00:00:00′) ORDER BY `history`.`logged_on` DESC
Time: 0.0026991367340088

SELECT `tasks`.`id`, `tasks`.`task`, UNIX_TIMESTAMP(due) AS `time`, `lists`.`name` AS `list_name`, `lists`.`id` AS `list_id` FROM `tasks` INNER JOIN `lists` ON lists.id=tasks.list_id WHERE (tasks.done IS NULL) AND (tasks.due IS NOT NULL) AND (tasks.user_id=’1′) ORDER BY `tasks`.`due` ASC
Time: 0.00059294700622559

SELECT COUNT(*) AS `count` FROM `dispatch` WHERE (dispatch.user_id=’1′) AND (dispatch.status=’0′)
Time: 0.00044488906860352

SELECT COUNT(*) AS `count` FROM `conversations_to_users` WHERE (user_id=’1′ AND status=’0′)
Time: 0.00038599967956543

At first sight it would appear that the first query would be our immediate suspect for optimization, though the gains would not be great (completes in just over 0.002 seconds). Switching to a database I’ve prepared before hand with plenty of dummy data (~1.1Gb, some tables over 5M rows) the output looks very different:

That’s right, there was no output. My development machine was hanging on this particular view when switched to the dummy database (I confirmed this was database related by running the queries independently in the MySQL command line). I switched over to our staging server to see if it can handle rendering this view against the dummy database. Luckily, it could and the suspect query immediately showed up in the output:

SELECT `history`.`id`, `history`.`type`, `history`.`action`, `history`.`value`, UNIX_TIMESTAMP(history.logged_on) AS `time`, `history`.`user_id`, `history`.`user_name`, `history`.`parent_id`, `history`.`parent_type`, `history`.`parent_name`, `history`.`list_id`, `history`.`list_type`, `history`.`list_name`, `history`.`logging_user_id`, `tags`.`color`, `tags`.`id` AS `tag_id` FROM `history`
LEFT JOIN `tags` ON tags.id=history.tag_id AND tags.user_id=’1′ WHERE (history.user_id=1) AND (history.logged_on >= ‘2008-09-27 00:00:00′) ORDER BY `history`.`logged_on` DESC
Time: 0.00042605400085449

SELECT `tasks`.`id`, `tasks`.`task`, UNIX_TIMESTAMP(due) AS `time`, `lists`.`name` AS `list_name`, `lists`.`id` AS `list_id` FROM `tasks`
INNER JOIN `lists` ON lists.id=tasks.list_id WHERE (tasks.done IS NULL) AND (tasks.due IS NOT NULL) AND (tasks.user_id=’1′) ORDER BY `tasks`.`due` ASC
Time: 58.160161972046 //OMFG! what gives…

SELECT COUNT(*) AS `count` FROM `dispatch` WHERE (dispatch.user_id=’1′) AND (dispatch.status=’0′)
Time: 0.00032281875610352

SELECT COUNT(*) AS `count` FROM `conversations_to_users` WHERE (user_id=’1′ AND status=’0′)
Time: 0.00026297569274902

A query that took just over 0.0005 seconds on a small database, took almost a minute to complete on a bigger one. Surprisingly enough the other queries remained blazingly fast (even faster than on my development machine with a small database, which is a credit to our server’s power).

Optimizing The Errant Query

So I’ve found a very problematic query to say the least. Taking 58.1 seconds to complete is obviously not acceptable for any real time application. Running EXPLAIN against the query revealed the following execution plan:

The EXPLAIN results show the most immediate problem with the query - it was performing a join against every row in a 1.8M row table.

Examining the details of the query plan, it appears both selects are using an index, the first one limited by a WHERE (on the surface at least - since all rows were scanned). Since despite using an index the entire table was scanned very slowly, I tried to see the query will respond if I removed the index. Using IGNORE INDEX(user_id), I re-ran the query (this time in the Mysql command line):

304 rows in set (1.53 sec)

Wow, that’s a big difference for not using an index. Running EXPLAIN on this reveals:

Despite using filesort to sort the query, it was running almost 30 times faster than before. By forcing it not to use an index which was not selective enough (i.e. not at all), the query ran a full table scan instead and completed much faster due to a better execution plan.

So we’ve improved greatly, but still not enough for my needs. Having one query taking over a second to complete when all the others are completing in sub 0.01 second times means it will be a bottleneck as the database grows (remember, the same query completed in just over 0.0005 seconds on a much smaller database).

Examining the structure of the table in question (tasks) revealed another interesting revealation - there is no index on the user_id column. So what happened? it appears there used to be an index on the user_id column, however it was removed just before all the dummy data was inserted into the database. Running ANALYZE TABLE tasks repaired the key information to the current state (no index on user_id).

I wanted to try the original query, this time with an actual index on the filtering column (user_id). After adding the index, I re-ran the query:

(304 total, Query took 0.0021 sec)

Much better! but I was not done. Running EXPLAIN yet again reveals the following execution plan:

So indeed, using an actual index was a big help, filtering the set to only 368 rows before processing the rest of the query. However, you’d notice it is running a filesort which we’d like to avoid. Preferably, I could set up an index combination that both filters and sorts using an index.

Creating an index composed of both the ‘user_id’ and ‘due’ columns got it right on the head, leading to:

(304 total, Query took 0.0008 sec)

Not as dramatic as previous improvements, but still three times better than the last iteration.

Final Words

The entire process took several hours of head scratching to arrive at all the results I’ve shown here. I managed to take a query that would cripple our server (58 seconds for one completion!) to an extremely fast one (0.0008 seconds against a 1.8M row table), through the use of some basic profiling and examining the execution plan.

Being an edge case (a non existent index used to filter against a 1.8M row table…), I gained valuable experience on potential pitfalls and on reading between the lines in the execution plan.

Some technical gibberish:

All queries were run with SQL_NO_CACHE to prevent caching from tampering with the results. I ran each query at least 5 times to make sure I was getting the right readings.

The specs of the server machine that ran the queries are as follows:

• Intel \ 2.4 GHz 1066FSB - Conroe \ Xeon 3060 (Dual Core)
• 2 x Generic \ 1024 MB \ DDR2 667 ECC
• 2 x Maxtor \ 146GB:SAS:10K RPM \ Atlas 10K - SAS
• CentOS Enterprise Linux - x86_64 - OS ES 5.0
• Running PHP 5.2.6 with MySQL 5.0.45 on Apache 2.2.4

First I’d like to expand on the (soft) release of Octabox - Octabox is a web platform for professional individuals and small businesses, providing a complete informartion management and collaboration environment. The concept was borne in the mind of my good friend and Octabox co-founder, Adam Benayoun, as the result of accumulated experience in building information management systems for many small businesses over the years.

Octabox is a cross between the vast functionality offered by application platform Salesforce and the simple and usable approach of basecamp (of 37signals). It is an application platform in the sense that it offers many on-demand applications for managing information (such as task management, contact relationship mangement, whiteboard/brainstorming and much more), but the focus is on keeping everything as simple and usable as possible to accomodate the needs of small business operations (all the way down to single individuals - freelancers, consultants and so forth). It is our belief that this sector is feeling the need for more powerful tools, yet is reluctant to use so called “enterprise” applications due to complexity and cost.

Octabox now enters a private beta phase to weed out issues and to figure what features are in demand so they’d be integrated in the public release. Hence, the “soft” release - the marketing effort has not begun yet (as can be seen by missing content on the site). We are handing out beta invitations if you register through our waiting list on the site. Blog readers get special treatment, so contact me through the contact form to secure your invitation ;-).

Octabox by the way runs on PHP 5.2.6 with MySQL 5.1 with Zend Framework 1.6 as the abstraction layer. I have much to say about building a high-availability entreprise web-application using those tools, so stay tuned as I already have a couple article drafts on the pipeline on framework performance, mysql query optimizations, server setup and much more.

http://stuffthathappens.com/blog/2008/03/05/simplicity/

A web browser built on the webkit engine (same as Safari), Chrome offers a simple UI and extensive support for web technologies. Having used it for a couple of days now, it is striking to me how obvious it is that Google is a web company - in bold contrast to another software giant currently pushing for their next-gen browser.

There is a lot to like about Chrome: It is open-source. The tab oriented UI is a great innovation. The simple UI allows for an impressively large work area in the browser. It comes with a built in DOM inspector and Javascript debugger (almost rivaling Firebug for functionality). The tab/process manager is an awesome feature (though a possible source of contention). And most importantly - aside from minor inconsistencies from the standards (shared by Safari), all web sites render perfectly under Chrome.

I was somehow surprised though it only reaches a score of 79 on the acid3 test, since it is based on an engine that already scored a perfect 100 (webkit). For comparison, Firefox 3.0.1 achieves a score of 71 (though with less graphical glitches), IE7 achieves a measly score of 13 in about triple the time, and IE8 beta 2 just barely beats that with a very low 21.

It is my personal hope that Chrome steals enough market share from Microsoft to help push out older IE versions, thus catapulting the web forward. This has been a fantastic move by Google, and the online world is watching to see how the market will respond. Is a Google OS next on the agenda?

Some very well known and successful online entities can be considered as such - Wikipedia in which users contribute knowledge, Digg in which users help others find interesting articles by voting and facebook which is the current golden standard for social networking (in which user generated content - UGC - is a given).

However, those sites are not thought of as having strong business models. Wikipedia, of course, is free. Digg and facebook rely on advertising, which is the fallback business model on the web - conversion is relatively low and you can only count on decent revenue when you reach the size of the aforementioned sites.

As can be guessed from the title of this article, I would like to discuss a couple of sites / services that use UGC as the basis for a viable business model:

Case 1: Threadless

Threadless is an online t-shirt retailer with a twist - the concepts for the t-shirts are submitted by users, voted on by the community and finally hand picked by the staff. A winning design will be printed and sold through the site, and its creator wins a nice sum of 2,500$. That’s the basic premise - there are several variations such as contests and reprints, but at its base - its a marketplace for ideas.

Threadless is already an established online brand, existing since 2000. It proves that you can crowd source creativity for fresh product ideas in a way that is both beneficial for the product distributor and the creative contributor. Threadless builds on this premise and succeeds because of their excellent blend of branding, community integration and good service. I have personally purchased at threadless multiple times and I’ve had nothing but satisfaction and enjoyment from the service.

Threadless currently has revenue in excess of 30$ million, and income of 10$ million annually. Not bad for a startup funded with a 1000$ seed (earned in an online t-shirt content, no less).

Case 2: crowdSPRING

I had just recently discovered crowdSPRING, an online service that mates graphical design talent with design related projects. The concept is to try to improve traditional design projects proceedings for both sides:

- Project requesting parties (called buyers) are guaranteed to get at least 25 different concepts for their needs (be it a logo, website design, print and others), with a money-back guarantee.

- Graphical designers (called creatives) are given a global stage to show their work and generate income. Also, since buyers pay in advance, the winning piece is guaranteed to receive payment - there is no scenario in which a buyer can say he doesn’t like anything and walk away.

- crowdSPRING itself takes a small commission on top of the project award money. This is the main business model for the site.

What drives the service is the interaction between buyers and creatives. Creating a project and watching the concepts improve as both sides learn more about the requirements through iteration and communication is a very interesting experience.

We’ve recently created a logo design project on crowdSPRING, and the reaction has been phenomenal. Though we have a couple of designers on board at Octabox, we felt we needed a fresh approach as we have been too deeply involved for a long time now. There are still 9 days (out of 14) till the project ends, and we already have an incredible amount of entries (over 300).

The process itself was worth the price of admission - through the interaction with the many contributors, we achieved some insights on what we want in a logo and a design direction for our new website.

crowdSPRING is not yet an established entity like Threadless, but it is well on its way to becoming one. Another example of how to crowd source creativity in a win-win situation for all involved.

Web 2.0 as a viable business model

Those two sites are just a small sample of many successful sites / services built on UGC as the driving force. I believe this market is still mainly untapped - there are plenty of possibilities to be explored. Despite that the basic premise is always the same:

- Create a community around a product / service concept
- Allow / encourage the talent within the community to offer their skills to the rest of the community
- Facilitate the interaction between the talent and the community, while trying to interfere as little as possible

What other successful web2.0 business models have you seen?

Beta 2 is right around the corner and we are expanding our reach! If you wish to be a part of making IE better by contributing great bug reports then please email us at IESO@microsoft.com and tell us a little about yourself including why you’d be a great beta tester.

Reading the blog responses and this post on the subtly named ‘Why IE sucks’ blog, it seems the reactions are mostly negative. Aside from some unfortunate phrasing, I have to wonder why.

Most commentators are comparing it to open-source development and wonder out loud why should they contribute their precious time to a commercial product. The reason is simple: as long as Windows remains the dominant OS, Internet Explorer will remain the dominant browser. All of us developers will have to support IE8 in the near future - it is in our best interest that it will be as bug-free and standards-compliant as it can be.

Comparing IE development to open-source projects is a mistake. A more appropriate comparison should be to the public testing periods some computer games go through. Those testing releases are by invitation only most of the time, and the end product is almost never available for free (this is the business model of computer games, at least for now). Beta testers are motivated to participate since they intend the purchase the final product, and they want it to be the best it can be for their own personal benefit.

IE8 development is similar - possible users and developers are expected to contribute since it will be for their benefit. The question at the end of the day is - can you come to terms with Microsoft and do it their way? if previous history is any indication, complaining about it is pretty much ineffective. To quote the Flight of the Conchords - Be more constructive with your feedback, please.

I have submitted my tester-status request. What about you?

It is a topic I have very strong and passionate opinions of, and motivated by this beautiful prose by Jono over at Not the User’s Fault, these are my guidelines for user interaction design:

Know your users

The first step in interaction design is to know who it will be interacting with. Users can be profiled on many criteria, such as age, technical orientation, vocation, cultural background and more. The user profiles created from segmentation of those criteria are called Personas.

While defining Personas is a common practice for designing user interactions, it might not be possible to engage in all the steps required to fully understand the needs and tendencies of the users they represent - such as interviews, surveys, focus groups etc.

This is especially true on the web, where projects have limited funds and are very quick from inception to implementation. In this case, experience and common sense rule the day - but it is still important to define the base Personas for which the interaction under design applies. Going through the process brings out some considerations that can influence design decisions.

Watching actual users go through an interaction is very important to learn about its effectiveness. Watching live users in action is the best learning experience in interaction design.

Know yourself

Knowing your users is the most basic step to interaction design. Yet, for each user type and interaction requirements there are as many possible implementations as there are interaction designers. At this point the interaction designer has to make choices for his users based on his experience, attitude and style.

It is sometimes hard to avoid designing an interaction for yourself rather than for your users. It is a natural tendency to try to solve interaction problems in a way that seems most natural to you, however that might not always be in the user’s best interest. Ideally, the interaction designer is a part of the target audience. When that is not the case, observing prospective users is very important to understanding their needs and deciding on the approach to solve their problems.

I believe that good UI design is more intuitive than science, and in that respect it is not so different from graphical design. However the two should never be confused - as I’ve argued in my post on common misconceptions in web development.

Keep it as simple as possible

This old mantra is very much an integral part of a successful user interface. By keeping interactions as simple as possible you will:

• Have less opportunities to fail your users
• Give your users less to think about, allowing them to make easier decisions
• Reward your users quicker (at the completion of the interaction)
• Increase the chance that the interaction will get completed at all

A large part of the success of web-based services can be attributed to the simpler interfaces they provide compared to desktop solutions. Some of this is a result of technological limitations on the delivery software (ie, web browsers), but it’s hard to argue with the results.

There are several common ways to simplify interactions:

• Use intelligent defaults
• Hide optional paths (or form fields) by default (progressive disclosure)
• Remove unnecessary steps from the interaction (and do so aggressively)
• Reduce mouse clicks. Make each click do more

This is obviously a very partial list, but it’s a good start.

For me as a developer, the KISS principle is deeply ingrained in my thought process. Translating it into user interface design took some getting used to, but once it happened it became second nature.

In fact, many parallels can be drawn between UI design and software architecture design almost to the point you wonder why most developers aren’t interested in designing interactions (actually I know why - most developers resent users for constantly breaking their code. I know this since a couple years back I had the same mindset).

Don’t break conventions - And if you do, make it obvious

The tools available to us developers have evolved much in recent years, allowing us to create richer interfaces and interactions. With power comes responsibility - we need to apply discretion when using advanced techniques and tools, as to not confuse users. Breaking interface conventions by using new technologies where they are not needed is a mistake.

Conventions should only be broken when they result in a bad user experience or when the alternative is significantly better. The latter is very uncommon when the former does not apply, so be advised.

If you do design a unique interaction (or at least, one that isn’t in common use) - make it as obvious as possible for the user. A user can only begin to understand your new interaction when he realizes that something is different. Disguising buttons as links, hiding drop down menus in small target zones, making background changes to the document without notifying the user - all result in user confusion and a bad user experience.

The more you need to educate the user on how to complete an interaction - the less likely he will bother to. Good interactions are self explanatory.

Interactions should be fun

Users interact with your application since they want to achieve a goal. That goal might be to complete an item purchase, to indulge a curiosity, to gather information and many others. There are several factors that affect the user’s motivation to complete an interaction:

• How important is the interaction to achieving the user’s goal
• How unique is your application (ie, how easy would it be for the user to find a better place to achieve his goal)
• How hard it is for the user to complete the interaction

Negative factors can be offset by a fourth one:

• How fun is it to progress through the interaction

The fun factor in interactions is often ignored as they are considered strictly functional. It’s no coincidence the word functional begins with fun . The fun factor of an interaction increases motivation for completing it - it makes your application more unique and it increases the user’s tolerance for the interaction.

There are several ways to increase the fun factor of an interaction:

Increasing aesthetics

Making your interface prettier will have a positive effect on your users’ perception of it. It’s no secret that most of the illusion of Apple’s superior OS interface is made on the grounds of aesthetics.

Make it gamely

Computer games have some of the best interfaces of any computer software, and it’s no coincidence. You can make your interactions more gamely by adding feedback, reward the user for completing steps and transmit the feeling that the interaction is a part of the user’s journey towards his goal, rather than a functional requirement that he must take care of.

Make the interaction do more with less

The user’s time and patience are limited. Advance the interaction as much as you can with each user input. Make the user feel the interaction is smart and that it is working with him towards completion.

Learn from others

Those are my thoughts on user interaction design, born out of my experience in web development and of my introspection as a long-time user. If you are interested in user interface design, the GNOME Human Interface guidelines are as good as reference as you will find. My favorite web authors on the subject include Jono at Not the User’s Fault, Bill Scott at Looks Good Works Well and Aza at Aza’s Thoughts.