NVMe: Difference between revisions

From DWIKI
Line 5: Line 5:
*[https://wiki.archlinux.org/title/Solid_state_drive/NVMe arch linux nvme doc]
*[https://wiki.archlinux.org/title/Solid_state_drive/NVMe arch linux nvme doc]
*[https://www.pogolinux.com/blog/high-performance-multi-tenant-object-storage-nvme/ How to Deploy High-Performance Multi-Tenant Object Storage with NVMe]
*[https://www.pogolinux.com/blog/high-performance-multi-tenant-object-storage-nvme/ How to Deploy High-Performance Multi-Tenant Object Storage with NVMe]
 
*[https://docs.netapp.com/us-en/ontap-sanhost/nvme_sles15_sp3.html NVMe-oF Host Configuration for SUSE Linux Enterprise Server 15 SP3 with ONTAP]
==NVMe/TCP==
==NVMe/TCP==
*https://www.networkworld.com/article/3609921/nvme-over-tcp-how-it-supercharges-ssd-storage-using-standard-ip-networks.html
*https://www.networkworld.com/article/3609921/nvme-over-tcp-how-it-supercharges-ssd-storage-using-standard-ip-networks.html
Line 23: Line 23:
The old way, on standard motherboards
The old way, on standard motherboards
====U.2====
====U.2====
*https://www.drewthorst.com/posts/nvme/namespaces/readme/
Hotplug
Hotplug
*https://en.wikipedia.org/wiki/U.2
*https://en.wikipedia.org/wiki/U.2

Revision as of 16:44, 7 March 2023

NVM Express

Links

NVMe/TCP

Qemu and NVMe


Hardware

Slots

M.2

The old way, on standard motherboards

U.2

Hotplug

Tools

nvme-cli

nvmet-cli

https://github.com/JunxiongGuan/nvmetcli

Documentation

NVMe device names

Example: /dev/nvme0n2p3

Means nvme device 0, namespace 2, partition 3 The first namespace, n1, will always exist

Namespaces

MNAM: Maximum Number of Allowed Namespaces

HOWTO

List devices

nvme list

get details

nvme id-ctrl /dev/nvme0

if you want to find for example IP of a device:

nvme list-subsys /dev/nvme2n1

Namespaces

List namespaces

nvme list-ns /dev/nvme1

Show info about namespace

nvme id-ns /dev/nvme1n1

Show number of available namespaces

nvme id-ctrl /dev/nvme1|grep nn


Show total capacity

nvme id-ctrl /dev/nvme1|grep tnvmcap

Show unallocated capacity

nvme id-ctrl /dev/nvme1|grep unvmcap


NVMe over fabrics

https://www.linuxjournal.com/content/data-flash-part-iii-nvme-over-fabrics-using-tcp

You might want to use nvmetcli on the target:

https://github.com/JunxiongGuan/nvmetcli

Dependencies: python3-configshell-fb

NVMe/TCP Host

See Configuring an NVMe/TCP host

Client

Modules:

nvme_rdma nvme_core nvme_fabrics and more


Connect on boot

Check /etc/nvme/discovery.conf

systemctl enable nvmf-autoconnect.service

Show remote connections

nvme list-subsys

Monitoring nvme


FAQ

Failed to open /dev/nvme-fabrics: No such file or directory

modprobe nvme-tcp

or maybe rdma

Failed to write to /dev/nvme-fabrics: Connection refused

dmesg will probably show

nvme0: failed to connect socket: -111

Maybe you're using something like infiniband, try

nvme discover -t rdma ...


Failed to write to /dev/nvme-fabrics: Invalid argument

nvme nvme0: Invalid MNAN value 1024

Try

modprobe nvme_core multipath=N

(remember to rmmod first :)


nvme_fabrics: no handler found for transport drma.

Check your /etc/nvme/discover.conf, it should be rdma :)