This Storage Intelligence (StorInt™) dispatch covers the new SPECsfs2014 and SPECsfs2008 benchmarks[1]. SPECsfs2008 is in “retirement status” and as a result, SPEC is no longer accepting any new submissions for SPECsfs2008.
There are still no new submissions for SPECsfs2014 other than the original 4-references solutions (one for each workload), submitted by the SPEC SFS®. Since the transition occurred, we have discussed the new benchmarks, NFS vs. CIFS/SMB ORT, new report formats and as of last quarter, NFS throughput per node. So, this quarter we again review another of our seldom seen SPECsfs2008 charts.
SPECsfs2008 results
Figure 1 is a bubble chart of all NFS submissions, size of bubble is capacity, horizontal axis is reported ORT (operational [average] response time) and vertical axis is NFS throughput.
Figure 1 SPECsfs2008 bubble chart NFS throughput vs. ORT with size = exported capacity
We seldom show the plot in Figure 1, but we like it as it provides an interesting comparison across NAS systems. Note we have plotted two types of subsystems in the chart above disk-only and hybrid and all-flash storage systems.
If one looks at the 1M NFS throughput operations/second vertical line one can see at least three different bubbles centered on this line. Each bubble has different ORTs ranging from 1.5, 2.5 and 2.9 msec, which represent three different vendor solutions, that being NetApp FAS6240 C-mode with 16 nodes, Huawei OceanStore 9000 with 20 nodes and Isilon S200 with 140 nodes, respectively. There is no pricing information so it’s hard to tell a $/NFS throughput operation. More node counts should generally cost more with the proviso that NetApp nodes will cost a lot more than Isilon nodes with Huawei OceanStore nodes somewhere in between those two.
Similarly, if one looks at ~1.2M NFS ops/sec there are two submissions of interest here. In this case, one at 0.75msec. and the other at 1.5 msec. ORT, which are Hitachi VSP G1000 FileModule 4100 (8 nodes) and the NetApp FAS6240 with 20 nodes, respectively.
Not much else you can tell without pricing information but there are a number of systems that can provide roughly the same throughput but provide vastly different average response times.
Significance
It’s now been over five quarters since SPECsfs2014 has been released. By this time in the SPECsfs2008 introduction there were at least 16 NFS and 8 CIFS non-reference submissions. The lack of any vendor submissions for SPECsfs2014, continues to be an major problem.
Yes, the changeover from SPEC SFS97_R1 to SPECsfs2008 was minor compared to the changeover from SPECsfs2008 to SPECsfs2014. But the changes were made to make the benchmark be more realistic and current. No submissions indicate there are bigger problems with the benchmark. Perhaps, results aren’t comparable, making vendors reluctant to release new benchmarks that show worse performance than older systems. In any case, if we don’t start seeing vendor submissions soon, we may have to abandon our SPECSFS reporting.
Until then, as always, suggestions on how to improve any of our performance analyses are welcomed…
[Also we offer more file storage performance information plus our NFS and CIFS/SMB ChampionsCharts™ charts in our recently updated (September, 2016) NAS Storage Buying Guide available for purchase on our website.]
[This performance dispatch was originally sent out to our newsletter subscribers in March of 2016. If you would like to receive this information via email please consider signing up for our free monthly newsletter (see subscription request, above right) and we will send our current issue along with download instructions for this and other reports. Dispatches are posted to our website at least a quarter or more after they are sent to our subscribers, so if you are interested in current results please consider signing up for our newsletter.]
Silverton Consulting, Inc., is a U.S.-based Storage, Strategy & Systems consulting firm offering products and services to the data storage community
[1] All SPECsfs2014 information is available at https://www.spec.org/sfs2014/ & SPECsfs2008 is available at https://www.spec.org/sfs2008/ as of 28Mar2016