Penetrate - NVMe SSD Test Software

Introduction

Overall

Penetrate is a testing-purpose software package for tailoring tests on NVMe SSDs.

Penetrate is composed of Peanut interpreter, Peanut extension modules, and Peanut scripts. The major extension module is the NVMe extension module (pxm_nvme.pxm) which is devised to access NVMe SSDs in user space based on PCIe/NVMe architecture.

Document

NVMe Extension Module Manual

Features

The NVMe extension module includes the following functions:

Retrieve device information; includes

Number of devices
Number of functions of a device
Number of namespaces
Number of IO submission queues
Size of an IO submission queue
IO submission queue full/empty status

Automatic creations of IO Submission/Compltion queues
Manual creations of IO Submission/Compltion queues
Allocate/free DMA data buffers
Read/write DMA data buffers
Dump DMA data buffers
Support PRP and SGL buffers
Support multiple devices (controllers)
Support submission queue accesses by multitasks
Namespace with/without metadata
Metadata transfered separately
Metadata transfered as part of the LBA
Support IOMMU/Non-IOMMU
Suppot SR-IOV virtal functions
Issue commands to devices

Admin commands
IO commands
Vendor commands

Query status of each issued command

Completion status
Command specific status
Timeout status
Execution time

Retrieve SQE CDWs
Retrieve CQE DWs
Powercycle and recovery
Launch resets to the device and recovery

PCIe hot reset
Controller reset
Function level reset
NVM subsystem reset

Dump trace of the latest commands and events
Statistics of issued commands per OP code basis
Statistics of completed commands per OP code basis
Command excution time records for QoS statistics per SQ basis

Outstanding Efficiency

The Peanut extension module pxm_nvme.pxm supports functions to issue NVMe commands to the SSD in a high efficiency. In a random 4KB-read test, where both the host platform and the NVMe DUT are of PCIe Gen4 interface, we see a 1101K-IOPS performance. In the test, the script only invokes 1 IO submission queue and controls the maxmum number of commands issued as 16 in the script. During the tests, all commands are issued one by one in the script level.

Below figures show random 4KB-read performances in conditions configuring the number of IO submission queues and the maxmum number of issued commands in each queue.

Reference System and System Requirement

ASUS STRIX Z590-A WIFI GAMMING mother board
Ubuntu 22.04/20.04
1024 2MB-size huge pages
libusb-dev and libconfig-dev packages installed

Quick Start

After receiving the package, execute the following commands to decompress the penetrate.tar.gz file (note: the real filename is dependent on the package version), install the required library packages, and then run the Peanut hello.pnt script.

            
user1@pc:~$ tar -xvzf penetrate.tar.gz
user1@pc:~$ sudo apt-get update
user1@pc:~$ sudo apt-get install libconfig-dev libusb-dev
user1@pc:~$ cd penetrate 
user1@pc:~/penetrate$ sudo pni hello.pnt World

Next check hugepage configuration by the following command.

            
user1@pc:~/penetrate$ grep Huge /proc/meminfo

With huge pages configured properly, messages like the following would be displayed. If no hugepage is configured, refer to Configure Huge Pages for the setting.

            
AnonHugePages:         0 kB
ShmemHugePages:        0 kB
FileHugePages:         0 kB
HugePages_Total:    1024
HugePages_Free:     1024

Before running the script, there are two notes: (1) scripts to access the NVMe SSD should be run with sudoer privileges, and (2) the NVMe SSDs to be tested should be set as 'devices under test' (DUTs) by the sdut utility. Like the following as an example, /dev/nvme0 is set as a DUT.

            
user1@pc:~/penetrate$ sudo sdut --noiommu nvme0     or
user1@pc:~/penetrate$ sudo sdut --iommu nvme0

Another, the user can execute 'sudo -s' to start a new shell with sudoer privileges before setting up the NVMe DUTs or running scripts. Then setting up /dev/nvme0 as a DUT is done by the following command.

            
user1@pc:~/penetrate$ sudo -s
root@pc:/home/user1/penetrate$ sdut --noiommu nvme0

Now change the working directory to main/ and invoke pni to run the script like the following.

            
root@pc:/home/user1/penetrate$  cd main
root@pc:/home/user1/penetrate/main$ pni u00_identify.pnt

Package Contents

common/	The directory for commonly used script routines.
common_nvme/	The directory for frequently used script routines accessing nvme drives.
doc/	The directory for the documentations.
pxm/	The directory where the used extension modules are.
main/	The directory where the sample scripts are.
pni	The Peanut interpreter.
sdut	The utility used to set up DUTs, revert DUTs, and list DUTs information.

Configure Huge Pages

If there is no huge-page configured, run the following command to configure huge pages.

            
user1@pc:~$ sudo sysctl -w vm.nr_hugepages=1024

To ensure that the huge pages are configured after the system starts, add the following entry to /etc/sysctl.conf file.

            
vm.nr_hugepages=1024

sdut

sdut is used to set up DUTs, revert DUTs, and list DUTs.

A device can only be accessed by the extension module if the device is set as a DUT by sdut utility. The --noiommu or --iommu option is used to set a device as a DUT. If the host system supports IOMMU and IOMMU is enabled, use the --iommu option; otherwise, use the --noiommu option. The device names are specified after the --noiommu (or --iommu) option. The device name can be: nvme? or a BDF address, where ? is a number.

While the user wants some device(s) not be tested any more, the user can use the --revert option to do it. The DUT names are specified after the --revert option, and the DUT names are dut1 dut2 ...

And before running a script, the user may want to check DUTs information; she or he can use the --list option.

Using the --help option will show usage of the sdut utility like the following:

              
root@pc:/home/user1/penetrate$ sdut --help
Usage: sdut (--help | --version | --sync | --list)
       sdut --noiommu dname1 [dname2 ...]
       sdut --iommu dname1 [dname2 ...]
       sdut --revert dut1 [dut2 ...]
Set up and list NVMe devices under test.
  --help                    Display this help and exit.
  --version                 Output version information and exit.
  --list                    List NVMe DUTs currently set up.
  --sync                    Synchronous DUT settings.
  --noiommu dname1 ...      Set up NVMe DUTs running in NOIOMMU mode.
  --iommu dname1 ...        Set up NVMe DUTs running in IOMMU mode.
  --revert dut1 ...         Unset NVMe DUTs.
Example:
  sdut --noiommu nvme0 nvme1
  sdut --noiommu nvme1 0000:02:00.0
  sdut --revert dut1 dut2
  sdut --list