A few words on WinSAT

(based on Vista, the same is true with some slight alterations for Win7, mainly more test videos and the final report is split into several files)

I didn’t expect I would post here again anytime soon (some people might know why – just my own general feelings… but well, here I am…
It started with myself cleaning out any Creative Junk left on my laptop, as I have now moved to a ProJect Audio Systems Dac Box USB – a European product made in Europe.

Along the way I stumbled across this folder:
C:\Windows\Performance
Which contains a folder aptly named WinSAT, which in turn contains:

1) 2 short videos each, used to assess video performance (decoding)
2) Two files named “ShaderCache_something”
3) One winsat.prx file (no clue what that does)
4) One WinSAT Log file
This one is actually quite interesting as it details exactly what is done when WinSAT is executed – for example, the power scheme is changed.
Further, it contains a detailed list of all tests run/executed in the course of running the WinSAT test.
5) One folder called DataStore
This is the goldmine, as it itself contains the WinSAT logfiles which put detailed numbers onto the Windows scores “we see” normally.


I do not know of a comfortable way to read xml files – so I suspect you will need to suffer through them by any means you have available. Personally I open them in Dreamweaver – which makes them somewhat legible, but I am sure there is a better way (I’d guess someone could write a tool to do this )

Reading through the files, the output is always in the language of the host OS – i.e., having a German OS; it says this:

HTML:

 <Name>WinSAT</Name>
      <Version>V6.0 Build-6002</Version>
      <Title>Windows-Systembewertungstool</Title>
      <ModulePath>C:\Windows\system32\winsat.exe</ModulePath>
      <CmdLine> …

Windows-Systembewertungstool -> Windows System Assessment/Evaluation etc.

As a result, when I quote from my own files, you may need to check with your own what a certain word/label means.

What makes the WinSAT files so interesting is the list of detailed measurements:
i.e.

HTML:

<Metrics>
    <CPUMetrics>
      <CompressionMetric units="MB/s">124.88278</CompressionMetric>
      <EncryptionMetric units="MB/s">63.50213</EncryptionMetric>
      <Compression2Metric units="MB/s">370.28751</Compression2Metric>
      <Encryption2Metric units="MB/s">552.53326</Encryption2Metric>
      <DshowEncodeTime units="s">5.92568</DshowEncodeTime>
    </CPUMetrics>
    <MemoryMetrics>
      <Bandwidth units="MB/s">4129.67432</Bandwidth>
    </MemoryMetrics> 

Yes, that is MB/s – not some obscure score of 5,4 that we cannot directly relate to anything.


And it goes on considering all sorts of little details – for example that a device is a mobile device, i.e. a laptop, as well as other detailed information like “is the CPU 64Bit capable” etc.

HTML:

<Platform>
        <IsMobile>1</IsMobile>
        <PlatformRole desc="Mobile">2</PlatformRole>
      </Platform>
      <Processor>
        <Instance id='0'>
          <ProcessorName>Intel(R) Core(TM)2 Duo CPU     T9300  @ 2.50GHz</ProcessorName>
          <TSCFrequency>2493800000</TSCFrequency>
          <NumProcs>1</NumProcs>
          <NumCores>2</NumCores>
          <NumCPUs>2</NumCPUs>
          <NumCPUsPerCore>1</NumCPUsPerCore>
          <NumCoresPerProcessor>2</NumCoresPerProcessor>
          <CoresAreThreaded>0</CoresAreThreaded>
          <X64Capable>1</X64Capable>
          <X64Running>0</X64Running>

As well as some detailed CPU information that I didn’t copy.

And it goes on, and on, and on. Ideally it is best if you look at your own WinSAT file to evaluate the information contained within.

For Vista it also has an informative detail at the end – the HDD/SSD is subject to only a sequential read test – repeated 12 times in consecutive HDD zones – i.e. it starts in one, finishes and starts exactly behind it:

HTML:

<ZoneStart units="bytes">0x1f0d5dfaa4</ZoneStart>
<ZoneEnd units="bytes">0x22284dad4e</ZoneEnd>
…
 <ZoneStart units="bytes">0x22284dad4e</ZoneStart>
 <ZoneEnd units="bytes">0x25433d5ff8</ZoneEnd>

And again, detailed “real” numbers are listed, rather than the ominous WinSAT score:

HTML:

 <Throughput units="mbs">257.07</Throughput>

Judging from this, while WinSAT has been shouted down a lot on here, it possibly is more accurate than many people think it is. It is just that the data isn’t displayed in a very detailed format and condensed down to an overall result. Further, the overall WinSAT rating is dependent on the lowest rating obtained – this approach might have also been used for the individual subpoints, leading to the sometimes significant changes effected by driver updates.

I wouldn’t exclude the option that the test has a flaw – although my guess is writing benchmarks is rather easy. It would be nice to have a more detailed documentation on WinSAT itself – and maybe it exists on MSDN.
I shall let others search for it – and hope that this little post/article has been of use to at least some.

 

Well, IE9 or Firefox 5 aren't any better on my system...

They just offer a tree view - same as Dreamweaver. On this note though - Firefox states that the document contains no style information, that's why.

 

Very interesting post! If I were a Windows user I'd request it to be made a sticky.

Benchmarking always works the same way: Stress some component, retrieve any returning values and compare these to other systems (or some fictive scale). The flaw in WinSAT is not the measurement, it's the way the results are presented.
The statement "Your sequential HDD read speed ranges from 40 to 60MB/s." gives me valuable info. "Your score is 1.5." does not, especially if I get the same score for a different HDD that reaches 20-80MB/s just because MS thinks the average value is all I'm interested in.

btw: Every script kiddie should be able to produce a more or less pleasant illustration of the raw values in the XML files that would be a 100 times more useful than the random numbers that are shown by the Windows dialogue.

disclaimer: All the numbers in my post are random. And so is the assumption about deriving a score only from the average transfer rate. It's merely meant to show the general problem that comes with oversimplification when measurements are interpreted.

 

I am guessing it would be rather simple to create an XSLT style sheet and a batch file to be used to present the data in a more human readible form. I am too swamped with other tasks at the moment to try to tackle this right now.

Gary

 

And which again is sort of troublesome, considering the files are actually there for easy viewing.
You don't even get hit with UAC trying to read them.

On the note with respect to presenting information by debguy:
This is a difficult topic - take my grandfather: Will he know what to make of 80MB/s sequential read speed? No. Neither would my mother - tell them they need a rating of say 2, and they can easily compare it. In this respect for the average person, WinSAT has its uses - on the downside, all those people that know a bit more than "Internet Explorer contains the internet" have little use for the oversimplified presentation as implemented.
However, this does follow a clear trend of dumbing everything down - which is fine for new users, but extremely frustrating if you want to find something specific. If I want error reporting in Vista I start typing "Proble" -> for "Problemberichte" and voilá - now try the same in Windows 7... by accident I found them in some control centre or similar - I forgot already. Key point: I can't easily access them.

Now for someone who has never used a computer before, their location is logical - for someone who knew how to easily access it before it is troublesome.

WinSAT is the same - it is ideal for people with very little idea of what they are looking at. On this note though, it would be very beneficial to have some easily accessible specifications, or maybe even detailed results for those people that are interested in the nitty gritty numbers.

 

This is the point that causes a lot of people to slam WEI and WinSAT. MSFT 'declines' to publish the mechanics of how they test systems and generate either the raw behind the scenes numbers or the relative WEI.

An undocumented and variable benchmark is no benchmark at all. It's a point in time snapshot.

 

Well, that's not entirely true.

The benchmark does not change on the same OS on the same computer. WinSAT (basis of WEI) "now" works the same as it did 2 years ago on Vista, the same is true for Windows 7. Now if they are comparable between systems is another question.

I also think it is reasonably safe to assume that between the same OS it is comparable between mobile (laptop) platforms and stationary (desktop) platforms.

The variables aren't endless. Of course it isn't perfect - but neither is any other benchmark.
Take CrystalDiskMark - not everybody can run a secondary drive.
So for the best numbers, use SafeMode, for the best real life OS figures, use the standard OS boot...

 

Microsoft has said, openly, that they freely change how the back end numbers are calculated and how the WEI is calculated.

 

I see a few problems with your statement:

1) Microsoft actually had a use for WEI at the start - for example games specifications. Yes, it doesn't beat knowing your hardware, but most people don't.
Further, it would give an idea as to whether a computer is fast enough for the OS - I believe with Vista, one should really aim for a score of 3.

2) If your graphics memory speed is 2000MB/s, it will stay that - period. If you measure the same thing, you should get roughly the same result.

3) If Microsoft randomly changes how WEI is calculated that would be an unauthorized change of data on your computer and thus a crime (at least in Europe) - the exception would be Windows Update -> you consent to them changing stuff: But then no Windows Update referred to WEI as far as I am aware.

4) Lastly: changing it all the time has little benefit for Microsoft itself - at least on the same OS.

 

Changing how WEI works is nothing more than anything else they do for security and functionality releases on a monthly basis. How is this a crime, you've already checked on the EULA for fixes haven't you?

WEI is and always has been a point in time snapshot of a system. It's not a benchmark.

 

WEI isn't really a benchmark. It's just Windows saying how well your hardware matches their OS. That's how I think about it.

 

This is basically how I think about it.

 

No, WEI is based upon hardware benchmarking.

 

Wrong. Changing WEI isn't a security fix.

Theoretical case:
Someone has an unencrypted open network - with open access to their computer. Creating a .txt file that says "your network is unsecure" is a crime.
Even if no damage is done.

Same for the Windows OS - if they change something without it, they technically commit a crime, having changed data (might be empty space too) without your consent.

Has there been an update that affected WEI? No. -> So it cannot have legally changed.

 

"without it"... without what? I don't follow what you are saying there.

If there has not been an update that affected WEI, how could there be a change?

Gary

 

Change something without user consent.
-> Now that doesn't mean the user needs to know the details - but they can't "just change it".

The last part is the point:
Newsposter comments that Microsoft apparently said they changed WEI - but as there is no update I am aware off that affects WEI, they cannot have legally changed it, as any change would have been without the user's consent, hence illegal.

 

lol no

10 chara

 

Not no, but yes.

But it could just be a US <-> Europe difference.

Changing any data - even empty space to a text document without the user's consent is illegal in Europe (at least Germany & the UK)
-> So the very moment Microsoft changes something on my computer without me accepting an update (with the rough idea what it is about - or the ability to have access to information what it changes) they would be changing data without my consent.

But I think the US is a bit behind Europe when it comes to privacy