Saturday, July 17, 2010

Web Browser Grand Prix The Top 5 Tested And Ranked

Since the time our first Web Browser Grand Prix debuted, the already-raging browser wars have become heated indeed. In case you haven't been keeping tabs on the browser news, let's begin by getting up to speed on the latest:
   Once again, we find ourselves in a situation where multiple parties are claiming the speed crown. Obviously, all of these claims can't be true. The fact is, it's easy enough to produce favorable results supporting ANY browser. You can even do this for IE6 if you try hard enough. Simply pick a single benchmark or a group of potato-oriented tests and viola, there's the fastest browser ever! That is why we run all of them. If we find a valid benchmark that runs on every browser, we use it.
While this is a follow-up to our original Web Browser Grand Prix, we have also included some standards compliance testing along with enhanced memory benchmarks. We feel that even though these tests do not directly affect speed, they are crucial to overall performance. But don't worry, we'll separate those benchmarks from our speed tests in the conclusion, so you'll still get to find out which Web browser is the fastest when judged purely on raw speed. Without further adieu, let's get to the testing!

Test System Specs
OS
Windows 7 Ultimate 64-bit
CPU
AMD Athlon 64 X2 3800+ @ 2.0 GHz (dual-core)
Motherboard
Biostar NF61S-M2 TE
Memory
4 GB DDR2 @ 800MHz (2 x 2 GB)
Graphics
EVGA Nvidia GeForce GTX 260 (896 MB GDDR3), PCI-E 1.0
Storage
Western Digital WD2500KS 250GB SATA 3 GB/s, 7,200 RPM, 16MB Cache
Optical
Setup
    A clean installation of Windows 7 Ultimate 64-bit was performed using the default partitioning scheme. All the available updates as of 6/16/2010 were applied, along with Nvidia driver package 257.21. We then installed the browsers, Flash, Java, and Silverlight. Finally, we switched off all power management and screen saver options, along with notifications. Before actually testing, we disabled automatic updates on browsers with that feature.

Methodology

A reboot was performed between each benchmark, per browser. The OS was allowed to “warm-up” before running a new test. Each time we finished benchmarking a browser, the cache was cleared and the system reset before beginning another benchmark or moving onto another browser. Most of the benchmarks were run for five iterations per browser, unless specifically mentioned otherwise.

Regarding Nontroppo

Thanks to the urging of our readers, we have retired all of the Nontroppo benchmarks from the previous Web Browser Grand Prix due to an incompatibility with WebKit-based browsers, such as Google Chrome and Apple Safari. The final load time reports completion before some of the elements on the page have loaded. This allows Chrome and Safari to illegitimately dominate every single Nontroppo test.

Regarding Internet Explorer

It has already been made clear from our last Grand Prix that Internet Explorer 8 (IE8) cannot really compete with the other top four Web browsers. We included IE8 again not to beat on it further, but to provide a graphic illustration of where the market share leader's performance stands. If IE didn't still dominate the landscape, it would have been removed.

Why not test the Internet Explorer 9 developer preview? Because I don't like to test products still in development. Even when I do, comparing them to stable products is not very informative. I'm sure that IE9 hits some high notes on several benchmarks, but IE9 isn't a finished product. It's not a release candidate or a beta; it's barely at alpha level. At this point, it's a window that houses a file menu and frame to view Web pages. No navigation bar or tabbed browsing. Not even an address bar. Let's face it, if you strip away essentially every bit of functionality from an app, that app will probably run faster. When IE9 is done, we'll be the first to benchmark it. Not before.

Editorial Disclosure

The author currently uses Google Chrome as his primary Web browser and Mozilla Firefox as a secondary. Bestofmedia uses Mozilla Firefox to interact with its Content Management System.

SunSpider v0.9.1
   Opera is the clear winner in the latest version of WebKit's SunSpider JavaScript benchmark. Trailing by one hundred milliseconds is Chrome, with Safari close behind in third. Fourth place finisher Firefox is nearly three times slower than the top three, while IE8 is just completely off the charts, taking more than five times longer to finish than Mozilla.

Google V8 Benchmark v5

  As we might have predicted, Google Chrome scores the most points in the company's V8 benchmark. Opera comes in 450 points behind Chrome's score, while third-place finisher Safari trails the leader by over one thousand points. Firefox earns just shy of 300 points. Again in last place, IE8 receives a score of slightly over sixty.

JSBenchmark

  The placing in JSBenchmark closely resembles that of the Google V8 test. Chrome again places first by a significant margin, but this time Apple's Safari slightly edges out Opera for second place. Firefox fills fourth place with IE8 floundering behind in fifth.

Mozilla Dromaeo JavaScript

  Once again, Opera displays troubling results. It is simply too far ahead of the other contenders for this score to be valid. Chrome therefore becomes the winner here, with second-place Safari pulling in less than half the number of runs per second. Firefox takes fourth with just shy of sixty runs per second, while Internet Explorer scores fifty.

  On a side note, Chrome v5.0.375.86 also displayed seriously skewed results in this test--almost a thousand more runs than Opera! However, the slight version change to 5.0.375.99 brought the outcome back in line with reality.
 
GUIMark 2 Flash: Vector Charting

  Safari manages to top this round, with Internet Explorer and Opera closely in tow. Firefox and Chrome place fourth and fifth, trailing the others by two frames per second.

GUIMark 2 Flash: Bitmap Gaming

  Opera has the slightest frame rate advantage over Firefox and Internet Explorer in Flash bitmap gaming. Safari comes in fourth, while Chrome again picks up the rear.

GUIMark 2 Flash: Text Columns

  IE8 finally pulls off a victory, beating second-place finisher Opera by a fraction of a frame. Firefox and Safari are close behind, with Chrome yet again lagging behind the pack.

  We modified our memory tests in accordance with some of the comments from the first article. Instead of just looking at how much memory each browser used, we also focused on how well that memory was managed. In the previous article, we tested each browser with ten, five, and a single tab open in a single browser window. This time, we took readings for a single tab after displaying google.com for five minutes. We then opened 39 more tabs, using sites that round out the Quantcast Top 40, ensuring that each site had fully loaded. After five minutes, a screen shot of Windows Task Manager was taken to record memory usage. We then closed all the tabs except the original containing google.com. We again waited five minutes before recording the numbers. Finally, we let the browser rest for an additional five minutes to see if the memory usage had dropped any further.

  We performed this sequence of testing three times per browser. As stated on the previous page, we used live Web sites for this test. Too many of the top 40 sites are content-driven pages with scrolling headlines. Saving them to the hard drive would have lost content, and therefore provided a lower memory reading than real-world usage.

Single Tab

  IE8 uses the least amount of memory in a single tab. Safari follows closely, with Chrome and Firefox in third and fourth (respectively). Opera uses the most memory in a single tab, more than twice as much as Microsoft's browser.

40 Tabs

  Unfortunately, Apple Safari would not open 40 tabs at once. After nearly two hours, Safari only managed to fully open eight tabs of the forty-tab load. While the browser is capable of doing this, we found that the app maxes out our test system's CPU. A recreation of the benchmark was done on a top-of-the-line Core i7 rig, and though Safari took a noticeable time to completely load all of the tabs, it did finish. We ended up having to open each of the forty tabs one at a time for this benchmark to work.

Side Note: Proper Page Loads

  During the 40-tab test, we noticed how each of the browsers performed when completely loading and properly rendering Web pages. Chrome had trouble opening all 40 tabs correctly. A handful of the last few tabs did not load at all. And about half of those that loaded needed refreshing for missing or broken elements. Firefox, IE, and Opera did not have nearly as many broken pages, just a few here and there between the three iterations. Obviously, we had a heck of a time with Safari.

-39 Tabs

  After closing all 39 additional tabs, leaving only Google open in a single tab, Google's Chrome returns nearly all of the previously-used system memory. Chrome only holds onto ten megabytes of excess RAM from the 39 closed tabs. This memory management is nothing short of exceptional. No other Web browser returns this amount of system memory after having so many tabs open. Even Firefox, which uses the least memory with a full load of tabs by far, returns less memory than Chrome (both in amount and in percentage). Firefox uses the second least amount of memory after closing 39 tabs, but only really returns half, holding onto 80 MB more than it had before the additional tabs were opened. Internet Explorer manages to surprise yet again, giving back three quarters of the 40-tab total. Safari also gives back three quarters of its 40-tab memory total, but that still puts it at 270 MB. Opera performs the worst in this arena, keeping over 350 megabytes after closing 39 of the forty tabs. Not only is this the largest amount of memory that a browser holds onto, but it is half of Opera's 40-tab total.

-39 Tabs: Five More Minutes

  After an additional five minutes, Firefox gives back a noticeable amount of system memory (ten megabytes). Opera, on the other hand, gives back a more significant amount of RAM (90 MB). But the Norwegian Web browser still occupies last place. The other browsers showed no significant changes, and the placing remains the same.
  With the exception of the Acid 3 completion times, these scores will be held separate from our final judgment on speed, but they will factor into our overall conclusion of performance.

HTML5 Test

  Google Chrome takes the HTML5 compliance test by storm, earning a solid 32 point lead over second-place finisher Safari. Apple manages to pull off a slight upset, beating Opera by six points. Firefox places fourth, twenty points behind Opera's 159. As expected, IE8 flunks out, earning a meager 27 points.

CSS3 Selectors Test

  Firefox tops the list for CSS3, bringing in a full 578 points. Opera places second with 576, while Chrome and Firefox tie for third with 574. Internet Explorer once again occupies last place, scoring only 349 points out of a possible 578.

Acid 3: Pass/Fail

  Chrome, Opera, and Safari receive perfect scores of 100 in the Acid 3 test. Firefox also gets an A at 94 out of 100. Internet Explorer predictably fails, managing to only score twenty points out of the possible hundred.

Acid 3: Speed

  In order to call a definitive winner, and to determine placement, we compared the completion times of the three browsers that earned perfect scores in Acid 3.

  Opera slaughters the browsers from both Apple and Google. The Norwegian app finishes Acid 3 in almost half the time it takes Safari or Chrome. While I'm sure someone will point out that we are talking about partial seconds here, let me just say that we're also talking about half the time to achieve the same score. A decisive victory is won by Opera. Apple Safari beats Google Chrome by a mere 0.03 seconds to come in second.

  Just as we promised in the introduction, we separated the complete standings from the speed-only standings. So we could potentially have two winners today!

Total Placing

1st
2nd
3rd
4th
5th
Chrome
10
6
6
4
3

Internet Explorer
1
3
5
3
11
Firefox
5
6
6
10
2
Opera
11
4
5
2
6
Safari
1
10
6
8
4


The overall winner, when including the non-speed benchmarks, is Google Chrome. Opera is very close on Google's heels, actually bagging more first-place finishes, but it's Chrome's excellent memory management and standards compliance scores (both non-speed tests) that keep Google the slight overall point leader.

In an unexpected upset, Mozilla Firefox fills the third place spot instead of Apple's “fastest browser in the world", despite Safari earning the most second place finishes. Safari 5 falls solidly behind Firefox 3.6.6, which places last the least out of any browser. IE8 once again gobbled up most of the losses, relegating Microsoft's offering to last place.

Speed Placing

1st
2nd
3rd
4th
5th
Chrome
7
6
3
4
3
Firefox
3
4
6
8
2
Opera
11
3
3
2
3
Safari
1
7
6
6
3


When focusing purely on speed, Opera screeches past the competition to take the speed crown away from Google. Unlike last time, it isn't really a photo-finish. Chrome grabs second, but it's not enough to give Opera a run for the money. Firefox and Safari are in a dead heat for third, with a ever so slight edge in Mozilla's favor. As we all already knew, Internet Explorer owns last place.

Like last time, we break down the results by category to determine how the browsers perform overall. Now, instead of just leaving you with a winner and calling it a day, we're also going to break down some of the strengths and weaknesses of these browsers, observed during the course of benchmarking. We spotlight the weak points, again, not to beat on any of these apps, but in the hope that it will not be an issue be when the time comes for Web Browser Grand Prix 3.

Winner By Category / Test

During the course of benchmarking, we observed that the WebKit browsers from Apple and Google displayed some troubling non-memory related behavior during the 40 tab memory usage test. Though not specifically tested, the CPU is maxed out by Safari in this benchmark, tying up the system indefinitely. While Google's browser didn't become unresponsive or exhibit any CPU issues, a few Web pages didn't load at all - they didn't freeze or stall, but simply appeared as blank white pages with no activity. Many of the pages that Chrome did load required reloading due to broken or missing elements - significantly more than Firefox, IE8, or Opera. We'll be looking into concocting ways of measuring CPU usage as well as proper, complete, and accurate page rendering for a future edition of the Web Browser Grand Prix.
Category / Test
Winner
Also Strong
Weak
Speed Tests
Startup Times
Opera
None
None
Page Load Times
Chrome
None
Opera
JavaScript
Chrome
Opera, Safari
Firefox, Internet Explorer
DOM
Opera
None
Internet Explorer
Peacekeeper
Opera
Chrome
Internet Explorer
HTML5 Performance
Opera
None
None
Acid 3
Opera
Chrome, Safari
Internet Explorer
Flash
Opera
Internet Explorer
Chrome
Java
Opera
All
None
Silverlight
Opera
All
None
Other Tests
Memory Usage
Firefox None
Safari
Memory Management
Chrome
None
Firefox, Opera
CSS3 Compliance
Firefox
Chrome, Opera, Safari
Internet Explorer
HTML5 Compliance
Chrome
Firefox, Opera, Safari
Internet Explorer

   In the previous Web Browser Grand Prix, Opera had a small lead on our winner Google Chrome when the results were broken down by category. This time around, Opera wrangles a massive victory, dominating in eight out of the fourteen categories. This makes Opera the hands down winner of Web Browser Grand Prix    2. Though Google Chrome holds a scant lead in points when taking all benchmarks into account, Opera easily wins the speed race and owns over half of the wins per category. Right now, the Norgwegian browser-maker is the only outfit that can claim to have "the world's fastest browser."
   Will Opera continue to best the competition in almost every single speed test? Will Chrome again dominate JavaScript and Page Load Times? Will Apple ever get around to fixing Safari's CPU and memory issues (will Opera fix the latter)? Can anyone reach Firefox-level peak memory use, or match Chrome in management? With Chrome 6, Firefox 4, and Internet Explorer 9 just around the bend, it won't be too long before we find out. Until then, stay tuned to Tom's Hardware.

No comments:

Post a Comment