NVMe: Difference between revisions
m (→FAQ) |
m (→Links) |
||
| Line 5: | Line 5: | ||
*[https://wiki.archlinux.org/title/Solid_state_drive/NVMe arch linux nvme doc] | *[https://wiki.archlinux.org/title/Solid_state_drive/NVMe arch linux nvme doc] | ||
*[https://www.pogolinux.com/blog/high-performance-multi-tenant-object-storage-nvme/ How to Deploy High-Performance Multi-Tenant Object Storage with NVMe] | *[https://www.pogolinux.com/blog/high-performance-multi-tenant-object-storage-nvme/ How to Deploy High-Performance Multi-Tenant Object Storage with NVMe] | ||
*[https://docs.netapp.com/us-en/ontap-sanhost/nvme_sles15_sp3.html NVMe-oF Host Configuration for SUSE Linux Enterprise Server 15 SP3 with ONTAP] | |||
==NVMe/TCP== | ==NVMe/TCP== | ||
*https://www.networkworld.com/article/3609921/nvme-over-tcp-how-it-supercharges-ssd-storage-using-standard-ip-networks.html | *https://www.networkworld.com/article/3609921/nvme-over-tcp-how-it-supercharges-ssd-storage-using-standard-ip-networks.html | ||
| Line 23: | Line 23: | ||
The old way, on standard motherboards | The old way, on standard motherboards | ||
====U.2==== | ====U.2==== | ||
*https://www.drewthorst.com/posts/nvme/namespaces/readme/ | |||
Hotplug | Hotplug | ||
*https://en.wikipedia.org/wiki/U.2 | *https://en.wikipedia.org/wiki/U.2 | ||
Revision as of 17:44, 7 March 2023
NVM Express
Links
- nvmexpress.org
- arch linux nvme doc
- How to Deploy High-Performance Multi-Tenant Object Storage with NVMe
- NVMe-oF Host Configuration for SUSE Linux Enterprise Server 15 SP3 with ONTAP
NVMe/TCP
- https://www.networkworld.com/article/3609921/nvme-over-tcp-how-it-supercharges-ssd-storage-using-standard-ip-networks.html
- nvme and tcp
- NVMe over TCP
- https://tekdeeps.com/what-is-nvme-over-tcp-how-to-use/
- https://www.linuxjournal.com/content/data-flash-part-iii-nvme-over-fabrics-using-tcp
- nvme-tcp storage operations
Qemu and NVMe
Hardware
Slots
M.2
The old way, on standard motherboards
U.2
Hotplug
Tools
nvme-cli
nvmet-cli
https://github.com/JunxiongGuan/nvmetcli
Documentation
NVMe device names
Example: /dev/nvme0n2p3
Means nvme device 0, namespace 2, partition 3 The first namespace, n1, will always exist
Namespaces
MNAM: Maximum Number of Allowed Namespaces
- nvme namespaces
- Drew Thorstensen - NVME namespaces
- https://narasimhan-v.github.io/2020/06/12/Managing-NVMe-Namespaces.html
HOWTO
List devices
nvme list
get details
nvme id-ctrl /dev/nvme0
if you want to find for example IP of a device:
nvme list-subsys /dev/nvme2n1
Namespaces
List namespaces
nvme list-ns /dev/nvme1
Show info about namespace
nvme id-ns /dev/nvme1n1
Show number of available namespaces
nvme id-ctrl /dev/nvme1|grep nn
Show total capacity
nvme id-ctrl /dev/nvme1|grep tnvmcap
Show unallocated capacity
nvme id-ctrl /dev/nvme1|grep unvmcap
NVMe over fabrics
https://www.linuxjournal.com/content/data-flash-part-iii-nvme-over-fabrics-using-tcp
You might want to use nvmetcli on the target:
https://github.com/JunxiongGuan/nvmetcli
Dependencies: python3-configshell-fb
NVMe/TCP Host
See Configuring an NVMe/TCP host
Client
Modules:
nvme_rdma nvme_core nvme_fabrics and more
Connect on boot
Check /etc/nvme/discovery.conf
systemctl enable nvmf-autoconnect.service
Show remote connections
nvme list-subsys
Monitoring nvme
FAQ
Failed to open /dev/nvme-fabrics: No such file or directory
modprobe nvme-tcp
or maybe rdma
Failed to write to /dev/nvme-fabrics: Connection refused
dmesg will probably show
nvme0: failed to connect socket: -111
Maybe you're using something like infiniband, try
nvme discover -t rdma ...
Failed to write to /dev/nvme-fabrics: Invalid argument
nvme nvme0: Invalid MNAN value 1024
Try
modprobe nvme_core multipath=N
(remember to rmmod first :)
nvme_fabrics: no handler found for transport drma.
Check your /etc/nvme/discover.conf, it should be rdma :)
