NVMe
From DWIKI
NVM Express
Links
- nvmexpress.org
- arch linux nvme doc
- How to Deploy High-Performance Multi-Tenant Object Storage with NVMe
NVMe/TCP
- https://www.networkworld.com/article/3609921/nvme-over-tcp-how-it-supercharges-ssd-storage-using-standard-ip-networks.html
- nvme and tcp
- NVMe over TCP
- https://tekdeeps.com/what-is-nvme-over-tcp-how-to-use/
- https://www.linuxjournal.com/content/data-flash-part-iii-nvme-over-fabrics-using-tcp
- nvme-tcp storage operations
Qemu and NVMe
Hardware
Slots
M.2
The old way, on standard motherboards
U.2
Hotplug
Tools
nvme-cli
nvmet-cli
https://github.com/JunxiongGuan/nvmetcli
Documentation
NVMe device names
Example: /dev/nvme0n2p3
Means nvme device 0, namespace 2, partition 3 The first namespace, n1, will always exist
Namespaces
MNAM: Maximum Number of Allowed Namespaces
- nvme namespaces
- Drew Thorstensen - NVME namespaces
- https://narasimhan-v.github.io/2020/06/12/Managing-NVMe-Namespaces.html
HOWTO
List devices
nvme list
get details
nvme id-ctrl /dev/nvme0
if you want to find for example IP of a device:
nvme list-subsys /dev/nvme2n1
Namespaces
List namespaces
nvme list-ns /dev/nvme1
Show info about namespace
nvme id-ns /dev/nvme1n1
Show number of available namespaces
nvme id-ctrl /dev/nvme1|grep nn
Show total capacity
nvme id-ctrl /dev/nvme1|grep tnvmcap
Show unallocated capacity
nvme id-ctrl /dev/nvme1|grep unvmcap
NVMe over fabrics
https://www.linuxjournal.com/content/data-flash-part-iii-nvme-over-fabrics-using-tcp
You might want to use nvmetcli on the target:
https://github.com/JunxiongGuan/nvmetcli
Dependencies: python3-configshell-fb
NVMe/TCP Host
See Configuring an NVMe/TCP host
Client
Modules:
nvme_tcp nvme_core nvme_fabrics and more
Show remote connections
nvme list-subsys
Monitoring nvme
FAQ
Failed to open /dev/nvme-fabrics: No such file or directory
modprobe nvme-tcp
or maybe rdma
Failed to write to /dev/nvme-fabrics: Connection refused
dmesg will probably show
nvme0: failed to connect socket: -111
Maybe you're using something like infiniband, try
nvme discover -t rdma ...
Failed to write to /dev/nvme-fabrics: Invalid argument
nvme nvme0: Invalid MNAN value 1024
Try
modprobe nvme_core multipath=N
(remember to rmmod first :)
