Web hosting is a competitive industry. Hosting companies can often find success by differentiating themselves through specialization. To that end, companies specializing in hosting certain popular web applications such as WordPress or Magento have been able to compete against the bigger players. Yet even within these niches, the fundamentals of support and performance end up being the deciding factors for most customers.
In the realm of Magento hosting — an application notorious for its performance challenges — the hosts that can demonstrate their ability to run faster sites find themselves in a stronger market position than their competitors. As such, Magento hosting companies like to use comparison benchmarking sites like Magento Speed Test in their marketing as evidence of their Magento optimization prowess. Unfortunately this (premature?) emphasis on performance can stimulate a surreptitious inclination to rig the tests in one’s favor.
Magento speed testing and comparison sites run automated HTTP load testing tools like Joe Dog Software’s siege against a standard list of URLs found in an out-of-the-box installation of Magento, and then publish the results on their site for visitors to review. It is important to note that there is no scientific rigor being employed here. There is an implicit assumption that hosts will install identical demo stores, (there is no requirement to do so) but this conflicts with an expectation that hosts will do whatever they can to make the application perform better.
At Nexcess, one of our goals has been to provide the most secure, optimized hosting platforms for Magento. In addition to providing a solid hardware infrastructure, the general strategy here is to inject as much caching as possible into various layers of the stack without breaking the functionality of the application itself. With session-dependent shopping cart software, this can be difficult. Although tweaking the lower layers for performance by tossing in plenty of RAM and utilizing tools like memcached and APC is relatively straightforward, caching the dynamically-generated HTML output can cause strange issues like users losing their cart contents or being unable to checkout. This is unfortunate because serving cached, static HTML via reverse-proxy like Squid or Varnish is orders of magnitude faster than serving each page dynamically, even with all of the underlying bytecode and object caching in place.
A quick web search will find several blog posts, project repositories, and Magento extensions where developers have attempted to find a viable solution that allows Magento to integrate with Varnish without breaking the cart functionality. Not having surveyed them all, I can’t say which ones are more successful than others. We too have been working on an in-house Magento extension that plugs into the Mage_PageCache module introduced in Magento 1.5. Our SIP-400 demo site is running a beta version of this extension which we will be making available to our customers in the near future. This server is currently in first place among the sites listed on www.magespeedtest.com.
This brings me back to my original point. The results you see on Magento Speed Test sites carry the implication that they’re an apples-to-apples comparison when in reality they’re not. They are simply another tool for hosting companies to demonstrate their market value in terms that potential customers (and Google’s ranking algorithm) care about. While we have striven to ensure that our demo stores are as close as possible to what our clients would get if they signed up for our Hosting service, there is no guarantee that their site — with custom code, third-party extensions, and image heavy themes — will perform as well as a demo store with sample data. Moreover, there is nothing to preclude other hosting companies from using disingenuous tactics in order to get the numbers that get you to walk in the door.
