Making a StarCraft II Benchmark...

I cant take credit for it, because it was ViciousXUSMC idea. He pointed out that the only fair way to benchmark SC2 is to make custom map, with custom camera movements, and a pre-hashed battle.

I have a lot of experience with this stuff, as I made WC3 movies and maps all the time. So I'm setting out to get it done. Add to this thread if you have any ideas or can contribute in any way.

UPDATE (Aug 4): I've started the map. I've made some good progress, but I cant seem to find any integrated triggers for recording framerates.

 

Make sure it's a 200/200 (population max) battle and put it on the Zerg creep. I heard the creep adds some extra strain on the CPU/GPU.

 

If you really want to put it to the test have a mothership cloaking too

Dont forget when we get the map done that we need to make sure people are running the same resolution & settings to make equal comparisons.

This is why I like what they did with 3Dmark09 Vantage where each test has a fixed resolution. Performance was 1280x1024, High was 1680x1050, and Extreme is 1920x1200 so if you were comparing your Performance test results against somebody elses you knew that both the detail settings and the resolution matched.

In the previous version of 3dmark06 you could match the settings but people ran different resolutions all the time (sometimes not knowing) and compared it to others of a higher/low resolution giving false comparisons.

So once you get the custom map figured out I think we should copy this resolution based naming system and have probably 3 resolutions to test at so that anybody with high or low res screens can run with a set res and get equal results.

Once the results start rolling in I can make some spreadsheet forms to post them much like the SC2 friends database and make some fancy excel graphs to visualize the data.

Edit: I should note I think we should make our testing resolutions 16:9 based since that is the aspect ratio that shows the most on screen action at once and the target aspect ratio for SC2.

Here is a 16:10 vs 16:9 comparison I made



The 16:10 res is 1920x1200 and the 16:9 res is 1920x1080 but you can cleary see more in the 16:9 aspect ratio despite the lower pixel count. SC2 uses a Horizontal +/- FOV and a fixed vertical FOV.

The other reason for 16:9 is that its a lot more common now than 16:10 screens and you can run 16:9 res on a 16:10 screen without issues but not the other way around (can run 1920x1080 on a 1920x1200 screen but not the other way around forcing you to use a lower res benchmark even if you had a high res 16:9 screen)

So our extreme test should be 1920x1080 all Ultra
our high test should be 1280x720 all High
and our low test I am not sure if we need it yet, going lower than above would probably result is a benchmark that is testing CPU power more so than GPU & CPU power combined but I guess that could be interesting, I'll figure out the best res here later.

 

If you really want to go the extra mile have the map play out different scenarios to strain different settings like one with a lot of creep and shadows, one with a lot of lighting effects, one with a lot of physics interactions, etc....

 

I'll try to encompass all of the above strains.

This is my other problem: a framerate counter. I don't have time right now, but later in the week I'll check to see if there is a framerate display/count trigger (hoping). This way the actual benchmarking could be done in the map itself, and if I could use those numbers in calculations, and actually report a framerate at the end, instead of everybody using fraps or something.

 

The game itself has an FPS meter, its triggered by ctl+alt+f.

 

Is there a way to trigger a script to enable the FPS meter + logging? Then at the end output total from logging to screen? Maybe the recap also shows the current SC2 settings in game?

Just some thoughts. I like the idea of having the most intense battle possible. Something to let the gamer understand here is the worst you can expect at the current system level.

BW, StevenX

 

Ya...thats why I'm optimistic for a way to record/log FPS. Having an FPS meter makes me hopeful for one. Also I need to be able to set conditions based on it...not sure about that.

 

Have the fight take place with Zerg Creep in the Background. That would totally reduce Frames.

 

It's already been noted throughout the thread .

Btw: The game itself recognizes when the FPS is below a "certain threshold" for a prolonged period of time (which is to say, a couple of seconds dip into the low teens during a battle won't make SC2 show that message), as it'll ask you to either reduce graphical settings or close 'background applications'... Which means the game is at least tracking the FPS.

On the other hand, it could just be a hard-coded function in the game and there's no way to get at it through the map maker (which wouldn't surprise me).

 

I've started the map. I've made some good progress, but I cant seem to find any integrated triggers for recording framerates

 

good stuff, glad someone is making a benchmark map.

 

An option is to have as a requirement FRAPS loaded in the default directory. Have the benchmark via script kick-off F11 (logging) and then again at the end. This will then record hi-low-quantity-avg frame rate.

Take Care, StevenX

 

ya...I was thinking of using a .bat file for this.