NBR 3dmark Standard Proposition

It has come to my (and many others’ attention that there is a large amount of confusion surrounding the validity and similarity of 3dmark scores. On the discussion page of almost every single notebook review on this site, there is at least one post concerning 3dmark scores, and how they compare to other notebooks which are similarly spec’d.

The problem is resolution. I’ll primarily use 3dmark06 as my example. In 06, the unregistered, free version, defaults to 1280x1024, which is fine for many notebooks in the 15.4”+ range. However, for many other notebooks, including Thin & Lights and Ultraportables, this resolution is impossible, as they only support lesser ones. The way 3dmark works is that if it cannot display the 1280x1024, it takes the highest resolution possible, within a ratio defined by the screen of the notebook. (4:3, 16:10, 16:9). This leaves many notebooks being compared against others running 3dmark at a higher resolution, making the smaller notebook seem much more powerful.

I propose that Notebook Review devise some sort of standard for comparing 3dmark scores. I believe that it should be mandatory that either any benchmarks posted be compared against only those running at that resolution, or that the resolution be explicitly stated, as to avoid confusion. While this may cut down the list of comparable notebooks, it would eliminate any misperceptions someone may have about these synthetic scores.

While it is true that 3dmark06 does not measure performance as accurately as frames per second in a game, it is still considered a standard by many, and it should therefore be as simple and legitimate as possible.

Please, cast your vote whether you think Notebook Review needs to set some sort of standard, and maybe we can clear up the confusion.

 

3dmark is such a stretch of an evaluative tool in the first place...

its value as a standard of measurement can really only be useful for those who understand the resolution differences in the first place.

 

I agree with this, it's a good idea to try and have some standard, or at least explicitly state the res the benchmarks were run at (this would be good for any gaming benchmarks, including real-world gaming tests, etc.).

 

I just don't like it because it isn't as accurate as just playing games and measuring the fps

 

The relative scores do give you an idea of how well a machine will perform.

 

Well I prefer gaming benchmarks

See the 7600vsX1600 gaming benchmarks thread stickied, the Go7400 picture thread and this one for example

 

Actually I have a 17" notebook and its pretty high end. The resolution tends to vary by driver. In some drivers it defaults to 1280x8xx(some wierd numbers) and with other drivers I may get 1024x768. I agree that we should make some type of standard or AT LEAST state what resolution you benched your laptop on. Many people tend to just post 3dmark06 scores and not mention their resolution at all.

 

I say we need a better benchmark in general

 

QFT. I agree that 3dmark is hardly the best representation of performance, but what other options, aside from games, do we have?

 

Hardly. The new 8600 and below series of cards do phenomenally well in 3DMark but fall apart in many games compared to cards which do not score as well in 3DMark. It's a purely synthetic benchmark. I put about as much faith into it as I do something like SiSoft Sandra's CPU benchmarks.

I agree with Mr. Kubelwagon - there should be a process put in place to only compare 3DMark runs at the same resolution. Since there are many notebooks with a max 800px vertical resolution, it becomes unfair to use 3DMark 06 as a comparison otherwise. And it really shouldn't be too hard to do. Just a quick note about the resolution and make sure only runs with the same resolution are compared.

 

In what games would that be? From what I've read of people with MBP's and Asus G1S's, they seem to do much better than the previous generation in a number of actual game tests. What games do the 8600M GT cards fall apart in?

 

I'm not comparing them to previous generation mid-range cards. I'm saying that based on 3DMark, the 8600M GT does not perform on par. For example, with its ~ 4000 score in 3DMark 06, you're hitting the 7900GS or so. However in games, that's just not the case. In some newer games that really stress shaders, the 8600M GT does start to shine, but in most other cases, it's not performing where 3DMark would indicate.

I'm basing most of my arguments on desktop benchmarks. Since there's not really any standardized testing of games in the notebook scene, the desktop's where the most pertinent information can be found. Check the reviews of the 8600 series and note how extraordinarily well they do in 3DMark, then check the actual game benchmarks. It just doesn't isn't there.

 

I see. Has this been reflected on the mobile side though?

That is, are people seeing the exact same patterns as with the desktop cards as with the mobile cards?

From the real-world gaming benchmarks I've seen people posting with Asus G1S's and MBP's, it seems like they are doing pretty well (haven't seen how they compare to previous gen machines with the GeForce Go 7900GS though).

 

Yeah, that 4000 3DMark 06 number was in the mobile space.

To be honest, I haven't really seen too much in terms of game comparisons between notebooks. With a desktop, you can easily switch out a component and re-bench - obviously not really possible with notebooks. People are quoting FPS in games, but between different people with different setups and game settings make the comparisons pretty much moot.

That's why I've been mostly looking over at the desktop side of things. I don't see why the same thing wouldn't come about in the mobile space - after all these mobile parts are based on the same architecture as the desktop stuff.

The biggest thing against the 8600 and below is the lack of memory bandwidth. Although fill rates are pretty close to the Go 7900GS, the 8600M GT can't match the former's bandwidth. You're looking at something like 22.4GB/s versus 32GB/s. So at high resolutions or AA enabled, the 8600s will fall behind, sometimes substantially.

In shader-heavy games, the 8xxx series architecture really shines over those with fixed shaders. While in absolute values, the pixel shader throughput is pretty similar between the Go 7900GS and the 8600M GT, if a game doesn't require much pixel shader power, but a lot of vertex shader power, the 8600M GT can dynamically assign its shaders to do more vertex processing (ah, the beauty of unified shader design). If you take a look at some of the detailed 3DMark scores, you'll see that in terms of pixel shaders, the 8600s aren't anything terribly special, but look at the vertex numbers and they absolutely blow away the previous gen. Those cards had quite a few pixel shaders (20-24 for the 7900GS/GTX) but only 7-8 vertex shaders. Obviously by dynamically allocating shaders, the 8600s can really dominate.

 

A lot of notebooks can't benchmark at 1280*1024 but the resolution of 1440*900 is most time possible. The difference between both resolutions is 1% in pixels. In the score you wont even get 1% difference when you try both resolutions. So i think everyone should benchmark 1280*1024 and if not possible then try to benchmark at 1440*900. In that case scores ARE comparable.

 

While it's true that many notebooks cannot bench at 1280x1024, the next default resolution for many notebooks, as in the case with the Macbook Pro, is 1280x800, not 1440x900, as there is not enough horizontal reolution. This creates a 30% gap in resolution.