NVMe Plugfest Webinar
An Interop Event for NVMe SSDs
Thursday October 2, 2014 2PM Eastern
Presented by:
David Woolf - UNH-IOL
david@iol.unh.edu
© 2014 University of New Hampshire InterOperability Laboratory
UNH-IOL
• Industry leading 3rd-party test facility for data,
telecom and storage networking technology &
consumer electronics
• 100% funded by commercial industry
• 150+ companies
• 32,000 sq. ft lab facility – Boston, USA Metro Area
• 7,200 sq. ft pre-wired space dedicated to Plugfests
© 2014 University of New Hampshire
InterOperability
2 Laboratory
2
Member Involvement
(150+ Companies)
3
3
UNH-IOL NVMe Consortium
Benefits
• Access to UNH-IOL NVMe test services.
• Attendance at NVMe Plugfest
• Access to UNH-IOL INTERACT PC Edition and UNH-IOL
INTERACT Teledyne-LeCroy Edition software for performing NVMe
Conformance Tests.
• Ability to list qualifying products on the NVMe Integrators List:
• Place products in the UNH-IOL NVMe Interop test bed for
continuous interop testing.
• Support for UNH-IOL Administration of open source tNVMe tool
PCIe Electrical testing available through UNH-IOL PCIe Consortium
4
NVMe Plugfest #3
UNH-IOL
November 10-13, 2014
Durham, NH USA
5
Agenda
•
•
•
•
•
NVMe Test Program Goals and Overview
Plugfest Components
• Interop Tests
• Conformance Tests
Plugfest Logistics
• Registration
• Test Stations Walk-Thru
• Reporting
Listing Products
Testing After the Plugfest
6
Agenda
•
•
•
•
•
NVMe Test Program Goals and Overview
Plugfest Components
• Interop Tests
• Conformance Tests
Plugfest Logistics
• Registration
• Test Stations Walk-Thru
• Reporting
Listing Products
Testing After the Plugfest
7
Growth of NVMe
•
“Enterprise SSD market is expected to reach $4 billion in
revenues by 2016, nearly six times that of 2011, while unit
shipments are expected to increase by ten times during
that period, to approximately 4 million units.”
- Forbes.com July 2014
•
NVMe is the next gen interface for both client and
enterprise systems.
NVMe products announcements are happening
There is a strong need to demonstrate the interoperability
of NVMe products.
The NVMe plugfest at UNH-IOL is a key component in
accomplishing this.
•
•
•
8
NVMe Test Program Overview
 Since 2012 NVMe Promoters Group
collaborating with UNH-IOL to manage the
Integrators List based Test Program for
NVMe.
9
NVMe Integrators List
NVMe Integrators List
hosted by UNH-IOL as
of Sept 30, 2014:
https://www.iol.unh.edu/services/testing/NVMe/integratorslist.php
10
NVMe Integrators List
• NVMe Integrators List is the industry
resource for listing products proven to
interoperable and conformance according to
industry accepted test practices.
• Listed products demonstrate commitment to
interoperability and open standards
conformance.
• OEMs viewing Integrators List qualification
as necessary for components.
11
NVMe Test Program Goals
 Prove the robustness and
interoperability of NVMe through
continually improving interop
and conformance testing.
 Help implementers ensure the
conformance and interoperability
of their products
 Provide recognition for products
meeting the necessary
qualifications backed by test
reports.
12
How do we populate the Integrators List?
 New Interop and Conformance
Testing Requirements approved by
NVMe Promoters group
 New Testing requirements launched
at plugfests.
 Testing for Integrators List
performed throughout the year at
UNH-IOL.
13
Agenda
•
•
•
•
•
NVMe Test Program Goals and Overview
Plugfest Components
• Interop Tests
• Conformance Tests
Plugfest Logistics
• Registration
• Test Stations Walk-Thru
• Reporting
Listing Products
Testing After the Plugfest
14
NVMe Plugfest Components
 NVMe Plugfest #3 November 10-13, 2014
will test according to version 1.1b of the
NVMe Test Program.
 First opportunity to test devices to the
new requirements and have them listed
on IL.
 After the Plugfest, UNH-IOL members
can request 1.1b Integrators List testing
throughout the year.
15
Test Plans
2 Test plans will be
executed at the plugfest.
1. UNH-IOL NVMe
Interoperability Test
Suite Document 1.1b
2. UNH-IOL NVMe
Conformance Test
Suite Document v1.1b
16
Common Test Terms
Mandatory
• DUT must pass test in order to qualify for IL.
Optional
• If DUT supports an optional NVMe feature, it
must pass test inorder to qualify for IL. If
option is not supported, the test is not
performed and does not affect IL qualification.
FYI
• Test does not affect IL qualification, but may
become mandatory in the future.
17
Passing Requirements Overview
Conformance Tests
• DUT must pass all mandatory conformance
tests supported by test tools
• If feature is supported, DUT must also pass
optional tests supported by test tools
18
Passing Requirements Overview
Interop Tests
Plugfest #3 (Today)
DUT must pass mandatory tests with 4/5 systems.
FYI tests do not affect IL qualification, are only
performed with 1 system
Plugfest #4 (2015)
DUT must pass mandatory tests with 5/6 systems.
FYI tests will become mandatory, are only performed
with 1 system
19
Interop Test Plan
Mandatory Tests (1.1, 1.2, 1.3, 1.4)
 Install SSD, identify drive in OS
 Format drive, write/read/verify for 3 minutes
 Reboot 3x/shut down 3x
FYI Tests (1.5, 1.6)
 1.5 - Boot from NVMe device
 1.6 - FYI Hotplug test (SFF-8639 only)
Will run all SSD and Host combinations possible at the event.
Publically available at:
https://www.iol.unh.edu/services/testing/NVMe/testsuites
20
Interop Test Tools – VDBENCH 5.04
 Command Line utility for generating IO
Workloads to disk.
 Java program works on Windows,
Solaris, Linux stations.
 Available at oracle.com

http://www.oracle.com/technetwork/serverstorage/vdbench-downloads-1901681.html
 Download vdbench Parameter File for
NVMe Interop Tests created by UNHIOL.
 https://www.iol.unh.edu/services/testing/
NVMe/unh_interop_1_1b_linux.txt
 https://www.iol.unh.edu/services/testing/
NVMe/unh_interop_1_1b_windows.txt
21
SFF-8639 / M.2 Adapters
• UNH-IOL will have 2 SFF-8639 to CEM adapters
on hand.
• If you have a non CEM form factor, please bring
it, and also please bring at least 2 of your own
adapters. This will facilitate interop testing.
PCI-AD-x439-01 adapter from
Serial Cables (serialcables.com)
22
22
How many samples?
• Conformance tests will only require 1 sample.
• Interop tests require that 2 samples be used
simultaneously.
• Please bring at least 2 samples of your product,
this will facilitate interop testing.
23
23
What Interop Tests are Required for Plugfest #3?
Test Group
Mandatory
Test 1.1 – Storage Device
Identified
Pass with 4/5 Hosts
Test 1.2 – Format Storage
Device
Pass with 4/5 Hosts
Test 1.3 – Write Read
Compare
Pass with 4/5 Hosts
Test 1.4 – Multiple Devices
on Bus
Pass with 4/5 Hosts
Optional/FYI
Test 1.5 – Boot from NVMe
Device
Mandatory for Plugfest #4
Pass with 1 Host:
Dell PowerEdge R720
Test 1.6 – Hotplug NVMe
Device
Mandatory for Plugfest #4
Pass with 1 Host:
TBD
Test 1.7 – Dual Port Device
Not implemented
24
Interop Systems
Platform
OS
Intended Testing
Dell PowerEdge R720
Windows Server 2012
(OFA Reference Driver)
Write/Read/Verify
Intel Patsburg CRB
Windows 8.1
(Microsoft Driver)
Write/Read/Verify
Intel Patsburg CRB
Ubuntu 14.04
(Reference Driver)
Write/Read/Verify
SUPERMICRO MBD-X10SAT-O
ATX Server Motherboard
RHEL 6.5
(Reference Driver)
Write/Read/Verify
SUPERMICRO MBD-X10SAT-O
ATX Server Motherboard
Windows Server 2012 R2
(Microsoft Driver)
Write/Read/Verify
TBD
TBD
Hotplug (FYI)
Dell PowerEdge R720
UEFI / Windows 8.1
Boot (FYI)
25
Conformance Test Plan
 Use Host Emulator to exercise and stimulate SSD.
 Focus on completion of Admin Commands, NVM
Commands, Controller Registers
Publically available at:
https://www.iol.unh.edu/services/testing/NVMe/
testsuites
26
Conformance Test Tools
 IOL INTERACT PC EDITION
 IOL INTERACT TELEDYNELECROY EDITION
27
Test Tools – IOL INTERACT PC EDITION
 Linux based NVMe Conformance Test
Station uses:
 tNVMe v.3.0 available from github
(UNH-IOL admin)
 IOL INTERACT 1.1 (available from
UNH-IOL) RELEASED July 30,
2014
 Ubuntu 10.10, 12.04, 13.04, 14.04




Executes most required conformance tests
defined in UNH-IOL NVMe Conformance Test
Suite Document
Installation walkthru:
http://www.flashmemorysummit.com/English/Collat
erals/Proceedings/2013/20130812_PreConfD_Mehta
_Woolf.pdf
Obtained through UNH-IOL membership
Contact: kerry.munson@iol.unh.edu
28
Test Tools – IOL INTERACT
TELEDYNE-LECROY EDITION
 Requires Teledyne-LeCroy Summit Z3
Exerciser and Analyzer, Windows PC
 Currently on LeCroy Software Build
6.75, aiming for 7.2 by plugfest time.
 Executes required conformance tests
defined in UNH-IOL NVMe
Conformance Test Suite Document
 For final IL Qualification UNH-IOL
will examine bus traces from IOL
INTERACT TELEDYNE-LECROY
EDITION
 Obtained through UNH-IOL
membership.
 Contact: kerry.munson@iol.unh.edu
29
Why 2 Conformance Tools?
•
•
•
Community needs a tool with a relatively low
cost barrier, to be able to determine
conformance. The PC Edition meets that
need.
Community also needs a tool with deep
debug and analysis capability. TeledyneLeCroy edition meets that need.
Tradeoff: Cost vs. Debug Capability + Trace
Analysis
30
Test Support
Test Group
IOL INTERACT PC
Edition v1.1
IOL INTERACT
Teledyne-LeCroy
Edition v1.2
Group 1: Admin Command
Set
All tests except 1.5 - Abort
Command
All tests supported
Group 2: NVM Command
Set
All tests except 2.7 - Write
Zeros
All tests supported
Group 3: NVM Features
Tests 3.2 and 3.3 not
implemented.
All Tests Supported
Group 4: Controller
Registers
All Tests Supported
All Tests Supported
Group 5: System Memory
Structure
All Tests Supported
All Tests Supported
Group 6: Controller
Architecture
No Tests implemented
All tests except 6.1
Group 7: Reservations
Not Implemented
Not Implemented
31
What Conformance Tests are Required?
Test Group
Optional (if supported
must be supported
correctly)
Mandatory
Group 1: Admin Command
Set
All Tests Mandatory
None
Group 2: NVM Command
Set
2.3, 2.4, 2.6
2.1, 2.2, 2.5, 2.7
Group 3: NVM Features
3.3
3.1, 3.2
Group 4: Controller
Registers
All tests except 4.6 –
Arbitration Mechanism
4.6 – Arbitration Mechanism
Group 5: System Memory
Structure
All Tests Mandatory
None
Group 6: Controller
Architecture
6.1, 6.2, 6.3
6.4
32
Agenda
•
•
•
•
•
NVMe Test Program Goals and Overview
Plugfest Components
• Interop Tests
• Conformance Tests
Plugfest Logistics
• Registration
• Test Stations Walk-Thru
• Reporting
Listing Products
Testing After the Plugfest
33
Plugfest Logistics - Registration
Registration open to
UNH-IOL
members:
 https://www.iol.unh.edu/servic
es/testing/NVMe/plugfest.php
Membership
 https://www.iol.unh.edu/servic
es/membership/form/
34
Plugfest Logistics – Who should attend?
• Companies developing or deploying NVMe
products
• Companies supporting NVMe products in their
systems
• Products should be functional to the point that
they are identified as a storage device in a
Windows or Linux system with an NVMe driver.
• No value in attending plugfest without a
functional product.
35
Plugfest Test Stations
 Each SSD will be cycled through multiple
conformance and interop test stations and
interop hosts.
 Conformance stations are test tools.
 Interop Stations are NVMe Hosts with
varying NVMe drivers.
 Not all stations required for Integrators List
qualification
 Some stations are “FYI”, to experiment for
future tests.
36
FYI Conformance Test Stations
PCIe PHY
Conformance
Test Station
(FYI)
 Following PCI-SIG
recommended test
procedures
 SFF-8639 only
 Help in tuning phy
parameters in new form
factors
 Tests are “FYI” only, not
required for Integrators List
qualification
37
FYI Conformance Test Stations
Protocol
Conformance
Test Station
(FYI)
 Additional Test equipment
companies may attend to
help improve their test
offerings.
 Any non UNH-IOL stations
should be considered FYI.
38
FYI Interop Test Stations
Hotplug
Interop Test
Station (FYI)
UEFI/Boot
Interop Test
Station (FYI)
 Hotplug Test Station
 Experimental test to help create
appropriate hot plug test cases
for future.
 UEFI /Boot Test Station
 Experimental test to exercise UEFI
NVMe driver.
 Tests are “FYI” only, not required for
Integrators List qualification today.
 These tests will be required at
Plugfest #4
39
Reporting
 Conformance Test Data will be collected directly
by UNH-IOL Staff
 Bus traces and logs
 Interop Data may be collected by UNH-IOL staff,
or engineers supporting their companies
equipment
 Summary and Error logs produced by
VDBENCH
40
Reporting
 UNH-IOL will use collected
test data to produce Interop
and Conformance Test
Reports for all products at
the Plugfest
 Test Reports are necessary
to list a product on the
NVMe Integrators List
 Reports are can be shared
with partners and
customers
41
Agenda
•
•
•
•
•
NVMe Test Program Goals and Overview
Plugfest Components
• Interop Tests
• Conformance Tests
Plugfest Logistics
• Registration
• Test Stations Walk-Thru
• Reporting
Listing Products
Testing After the Plugfest
42
Listing Products
 NVMe Integrators List Policy:

https://www.iol.unh.edu/services/testing/NVMe/20131219_NVMe_Integrators_List_Policy_v1_1b.pdf
 Test Reports are necessary to list a product
on the NVMe Integrators List
 UNH-IOL Report will indicate whether a
product is eligible or not
 Products are not automatically listed
 In order to list a product, companies must
make specific email request to UNH-IOL
through nvmelab@iol.unh.edu
43
Listing Products
Listed Information Includes:
 Product, includes Manufacturer, Model Name and Family
Name (i.e. different capacities of one SSD are in a single
family)
 Product type (accepted products types are: NVMe SSD, SSD
Controller, or SSD Controller IP, server, server board,
motherboard, add-in card, RAID Controller, or IP device).
 Operating System and Driver (Host Only)
 Firmware revision (Device Only)
 Interop Program Revision (i.e. 1.0, 1.1, 1.1b etc...)
 Date of listing and Test ID
 Further Information (primary contact or website)
44
Listing Similar Products
 Products with differences that do not affect
NVMe operation (form factor, storage
capacity) can be listed under a single
product listing, “listed by similarity”.
 The products listed by similarity must have
the same Firmware revision and Model
Name, and the listing company confirms
that the products are materially similar with
no substantive changes to the NVMe
interface.
45
Agenda
•
•
•
•
•
NVMe Test Program Goals and Overview
Plugfest Components
• Test Plans
• Test Tools
Plugfest Logistics
• Registration
• Test Stations Walk-Thru
• Reporting
Listing Products
Testing After the Plugfest
46
Post-Plugfest Testing
 Products will continue to
be tested at UNH-IOL
throughout the year.
 Test requests can be
made through:
https://www.iol.unh.edu/
my/
 Passing products can be
listed on the NVMe
Integrators List
47
Future Requirements
 Mandatory interop tests will increase from 4/5
to 5/6
 UEFI Boot test will be mandatory at Plugfest #4
 Hotplug test will be mandatory at Plugfest #4
48
UNH-IOL NVMe Consortium
Benefits:
• Access to UNH-IOL NVMe test services.
• Attendance at NVMe Plugfest
• Access to UNH-IOL INTERACT PC Edition and UNH-IOL
INTERACT Teledyne-LeCroy Edition software for performing
NVMe Conformance Tests
• Ability to list qualifying products on the NVMe Integrators
List
• Place products in the UNH-IOL NVMe Interop test bed for
continuous interop testing
• Support for UNH-IOL Administration of open source tNVMe
tool
PCI testing available through UNH-IOL PCIe Consortium
49
Review
•
•
•
•
•
NVMe Test Program focused on Integrators List to
showcase interoperable and conformant products.
Test tools and plans available from UNH-IOL.
Plugfest will allow the first opportunity to list
products on the NVMe Integrators list v1.1b.
Plugfest will walk SSDs through multiple
conformance and interop stations.
Testing can continue after the Plugfest for those
unable to attend.
50
Resources
•
UNH-IOL Membership
•
•
Plugfest Registration
•
•
https://www.iol.unh.edu/services/testing/NVMe/plugfest.php
NVMe Conformance and Interop Test Suites
•
•
https://www.iol.unh.edu/services/membership/form/
https://www.iol.unh.edu/services/testing/NVMe/testsuites/
NVMe Integrators List and Policy Document
•
https://www.iol.unh.edu/services/testing/NVMe/integratorslist.php
51
Q&A
Contact: david@iol.unh.edu
52
Q&A
• When will the 1.1b IOL-INTERACT
Linux and Lecroy tools be available for
download?
• As far as test support, the currently released
tools support all the necessary tests. Future
releases prior to the plugfest will contain bug
fixes and update LeCroy software support.
53
Q&A
• FYI boot from NVMe will require an OS
to be installed. What if the NVMe
device is too small for an OS to be
installed?
• Then the test cannot be performed. This is why
we have this as an FYI test today, so that you
can properly prepare for the next plugfest and
have a device large enough for an OS then.
54
Q&A
• How does the CEM adapter support
hot-plug testing as the server
motherboard PCIe slots also should
support the hot plug
• We will not be performing hotplug on CEM
card systems or devices. ·
55
Q&A
• What is the difference between github
version and iol-interact SW other than a
gui interface?
• IOL INTERACT supports test groups 3, 4, and
5, which tNVMe does not directly support.
56
Q&A
• Slide mentions IOL-Interact 1.1, but not
1.1b release
• In this case the 1.1 refers to the software
version. The 1.1 software version supports most
of the tests required for the NVMe IL version
1.1b (see table on slide 31 ). It’s just an
unfortunate coincidence that the numbers are
close.
57
Q&A
• Is PCIe phy conformance available for
non-CEM form factors
• Yes
• Is UEFI testing available on any
motherboard that can be done locally?
other platforms that support UEFI
• Yes, the test can be performed on any platform
that supports UEFI, but UNH-IOL will only
supply 1 platform at the plugfest.
58
Q&A
• Is there a deadline for registration?
• October 17, 2014
• Do you use Solaris host for interop test?
• Currently we don’t have a Solaris host for this
interop event. However, using a Solaris host as
one of the hosts for interop testing would still
help qualify an SSD for the IL.
59
Q&A
• We are interested in bringing a server
instead of a drive. Would you require 2
models the same as you want for
drives?
• Bringing a single server is fine. We only need
two samples of SSDs.
60