I've really missed the 'Steady 70/30 Read/Write Mix' tests.

I wondered what had happened to HardOCP's SSD section... (although the story is a few weeks old now, I had stopped checking for new SSD reviews too regularly).

See:
http://www.hardocp.com/article/2015/03/24/where_ssd_market_headed_in_2015/#.VSdY8XktF9A

Pretty good read (although I noticed a few errors) on SSD's and the future.

What I hope the most from this new development (Chris Lonardo is HardOCP's new Storage Engineer and Editor) is that some of the last two years of SSD's they've missed get reviewed just like the previous SSD's from 2013 and before were measured.

My favorite part of their reviews (because it matched closely what I could experience using the SSD in question) was this part (specifically; Steady 70/30 Read/Write Mix). This handily proved/showed to others why I felt the Samsung 840 Pro felt slow/laggy vs. the SanDisk Extreme II at the time (middle of 2013).

See:
http://www.hardocp.com/article/2013/06/05/sandisk_240_gb_extreme_ii_ssd_review/8#.VSdcoHktGcw

Hope they not only continue with this type of testing, but also further improve on it too.


Have others missed their reviews like I have?

 

+1

I hope they continue that metric too. It was well ahead of other sites. Some are now adding the mixed workload.

 

Good article. Exciting times ahead in the SSD market. Let's hope it doesn't move at a snail's pace and we get market saturation in a couple years rather than a decade. I do think benchmarks need to be measured more than just "as fast as you can get" read and write speeds. It's fine as a baseline, but irrelevant for daily use.

 

I see most benchmarks like this:

A mechanic shows me that my car engine hits the RPM redline while it is sitting in his garage and asks to be paid.

I say let me do a few quarter mile runs first on the road - and I'll even pay double if you actually decreased those timed runs significantly (and with me behind the wheel)... and those decreased times continue and are consistent for the foreseeable future (i.e. the work performed actually made my platform more productive).

Most bm's are not only easy on the components in question, but they are also easy for the manufacturer to fake too. When our storage subsystems have cpu's, RAM and firmware (an O/S, in essence) at their disposal, it is a very to program it to give great 'scores'. I know, not everyone does this - but, it is in the realm of possibility and for some manufacturers, is too tempting to pass up (Samsung, I'm looking at you).

What impressed me the most about HardOCP at the time is that they took a simple bm and with a judicious implementation, they could handily show the differences between SSD drives in real world settings using an (shock!) s ynthetic benchmark utility.

Not only did this surpass what Anandtech was doing at the time (and continues today) with trace based runs (not allowed to run in real time unfortunately) but also correlated well with some of Anandtech's results too.

Today, with SamsungGate, we get new tools like FileBench007a.zip by BrainSplatter and SSDReadSpeedTester2.04.zip by Techie007 to test our drives 'real' performance with. I'd like to see Samsung employees going mental trying to figure out how to game these kind of utilities.

I also like PCMark 8 Advanced Workload Performance tests too. But because it is a major/popular program that can be easily detected by a devious SSD's firmware, I'm not too confident that it too won't be 'hacked' by unscrupulous manufacturers at some point in time.

What makes the above two tools harder to hack than PCMark 8? The simple fact that you can see the read speeds of each file and simply use Windows to copy them and verify the speeds yourself - without the utility running.

 

I like the fact that they used SNIA guidelines for putting the drives in steady state. Consistent, fair testing. All (or most) of the SSD manufacturers belong to this group. The test focused on low queue depth performance because SSDs are to0 fast for the QD32 results we too often see.

The results were for the typical consumer workload. Hopefully, they will add heavy usage results as well.

 

Yeah, Anandtech also does mixed read/write testing. But they don't follow SNIA guidelines (which is sad; they're the ones that introduced me to them).

From Anandtech:

See:
http://www.snia.org/sites/default/files/HoEasen_SNIA_Solid_State_Storage_Per_Test_1_0.pdf

From the SNIA pdf above, Anandtech doesn't fill the drive 2x as recommended. And we don't know if the measurement window is as stated on page 30 of the above pdf.

 

TweakTown uses SNIA. Does AnandTech even have an OS on the drive. As I recall, they didn't. I leaned more to [H]ardOCP and now TweakTown because of this.

http://www.tweaktown.com/reviews/7062/intel-750-1-2tb-nvme-pcie-gen3-x4-aic-ssd-review/index5.html

 

See:
http://www.hardocp.com/article/2015...world_benchmarks_in_ssd_reviews/#.VTKNQ3ktHIU


Real world benchmarks for SSD's. My mind is blown.

This is the reason I joined NBR almost 6 years ago and the questions I posted then are still mostly unanswered. How do SSD's stack up against each other. (How they now stack up against HDD's is kind of moot, but I would still be interested in comparing current and future SSD's to current HDD's as a base for how far SSD's actually help me, the user over traditional storage subsystem devices).


(Following bolded text is my emphasis).

Note: I feel the same about almost all current BM 'scores' for SSD's.

I'm not holding my breath hoping HardOCP will test SSD's how I want them tested, but at least the mainstream will now realize that synthetic benchmarks on their own are not the way to buy and/or compare components with. Without me going hoarse from typing the same thing over and over.

It may have taken over 5 years for 'my message' to get across to the actual reviewers of these components... and without seeing what they will offer us in future reviews, I can't say for certain today that they will actually give us what I have been asking for for so long now.

But what I am positive of is that once this idea has been out in the wild; and no matter how good or bad it is implemented by HardOCP initially, it will be eventually further refined and even improved by other reviewers who don't want to keep giving page after page of boring graphs of benchmark 'scores' with no relevance to their otherwise dwindling readership.

I'm excited because Chris Lonardo has even hinted that NVMe may not be all it's cracked up to be (or the foreseeable future).

It takes guts to call things as you see them and I'm glad he's not backing down from anything, right from the get go.

I hope he turns the manufacturers marketing propaganda on their heads and on the way there he uncovers some gems in the SSD offerings we can choose from. Who knows? This may even force a real standardization for SSD's that MUST be printed on the box.


While the products we buy are what make this 'experience' exciting, what is even more exciting to me is being able to make a purchase decision without having to invest in a week of my time testing things myself (after I have tested the tester, of course).

Is anyone else as excited as me about these developments in SSD real world testing?

 

Well, I read that article too but without knowing what he's going to do yet, I'm not all warm and fuzzy. Your same quote my bold is different.

Again, waiting to see what this means. Saying Benchmarks suck and then using them, well you know. The 70/30 Mix was done with Iometer after all. But is that gone as he stated "we’re working from a blank slate"? If there is something better or it was flawed, by all means, bring it on.
I don't want to see any FOB results with or without "real-world applications." They understood steady state testing before, will they still? Are these real-world applications going to be opening Photoshop on SSD-X and compare it to the field? SSD-X boots in 15 seconds and SSD-Y in 16. Yawn!

 

Bullrun,

If we are to believe what was written at face value for now, the synthetic benchmarks may become useful if presented in context (and/or in a more consistent fashion than what manufacturers are currently doing). Of course, the relevance of those 'scores' still need to be judged, but we can only see how closely they tie everything together once they deliver a review for us.

Testing for me is about changing a component for another and then seeing if my productivity actually increases in my workloads. What they're gearing up for is a little broader usage base, and I think I could make compensations for that depending on how they present the data they collect.

Btw, what is FOB results?

 

http://searchsolidstatestorage.techtarget.com/definition/fresh-out-of-the-box-FOB

 

Nice! The empty drive, useless benchmark.

I am taking a wait and see approach. I'm hopeful that there will be useful improvements in testing methods. And that the "real-world applications" won't be like most sites use them, on an empty drive. Testing with an OS and as secondary storage, light, moderate and heavy usage, all in steady state are the results that I want to see. This may help the consumer that does a little research before making a purchasing decision.

As you know very well, tilleroftheearth, the steady state 70/30 Mix and later PCMark 8 Consistency Test showed the FOB king 840 Pro wasn't a high performance drive.

I'm not against seeing ATTO results. It can determine if the manufacturer can't even get that right. But one of this type of synthetic is enough. Even thessdreview.com uses PCMark 8 Extended now and they aren't the complete shill they were to the manufacturers. There's hope!

 

OK http://www.hardocp.com/article/2015/05/08/intel_ssd_750_review_nvme_for_desktop/#.VU6U0xa5f51

I see that PCIe are good for file transfers but I don't see they suck as an OS drive, (not NVMe, SM951). You would think the SE Pro was a joke from this but PCMark 8 Extended has it all over the other drives (not 750) in this review.
No FOB results, wow, pre-conditioned the drives.

70/30 is at TweakTown
80/20 TomsHardware

EDIT: From the discussion link in the review.

It's really that narrow?

These are the "real world" benchmarks.

So, If this is what you do all day, every day buy this drive.

 

Thanks Bullrun. Just read the hardocp article. The performance is not (only) what blows me away, that was expected, the low price for that performance is the real news. The consistency vs. differing workloads is also a bonus.

The real world benchmarks vs. a 480GB SanDisk Extreme Pro are a revelation, especially for the CS6 results I live and breathe in everyday.

CS6 improvements of 2x the SATA3 drive are more than just impressive - this is disruptive tech. Once again, desktops are worlds away from notebooks systems once again for my workloads. Copy times of ~3x faster are also welcome and as real world as we can ask for (and if anyone says that these would not be noticeable by mere mortals then they must not be breathing at that time ).

18 Channels and (too) many watts of power needed though for a notebook today. Can't wait for Intel's interpretation of what a Skylake based notebook with a 'real' NVMe drive should look (and perform) like.

What is most interesting to me is that even with this enormous storage subsystem potential, workloads such as the VS2013 Chromium compile barely distinguishes even the Intel 750 series from the Sata3 based SanDisk Extreme Pro. Yeah, it is faster. But 3 minutes longer when you've been waiting already for over three quarters of an hour is not something to praise (or upgrade for).

As hinted in the article, with NVMe drives programmers are going to have to be smarter of how their programs access data going forward. Otherwise, the wall we'll hit is the one that is programmed in.

The only disappointing note I have for this review is that it is based on Win7Ex64SP1. Huh? HardOCP, 2011 called and want their obsolete OS back...

I hope that Win10x64Pro is the standard that they'll move to soon, Win8.1x64Pro is already tasting a little stale to me...