NVMe: Difference between revisions
mNo edit summary |
m (→Namespaces) |
||
| Line 60: | Line 60: | ||
==Namespaces== | ==Namespaces== | ||
===Difference between size and capacity=== | |||
nsze vs ncap | |||
flbas = blocksize | |||
To check them in a namespace: | |||
nvme id-ns /dev/nvme1n3 | egrep "nsze|ncap|flbas" | |||
===Create namespace=== | ===Create namespace=== | ||
*[https://narasimhan-v.github.io/2020/06/12/Managing-NVMe-Namespaces.html Managing NVMe namespaces] | *[https://narasimhan-v.github.io/2020/06/12/Managing-NVMe-Namespaces.html Managing NVMe namespaces] | ||
| Line 81: | Line 90: | ||
===Show unallocated capacity=== | ===Show unallocated capacity=== | ||
nvme id-ctrl /dev/nvme1|grep '''unvmcap''' | nvme id-ctrl /dev/nvme1|grep '''unvmcap''' | ||
==NVMe over fabrics== | ==NVMe over fabrics== | ||
Revision as of 14:03, 8 March 2023
NVM Express
Links
- nvmexpress.org
- arch linux nvme doc
- How to Deploy High-Performance Multi-Tenant Object Storage with NVMe
- NVMe-oF Host Configuration for SUSE Linux Enterprise Server 15 SP3 with ONTAP
NVMe/TCP
- https://www.networkworld.com/article/3609921/nvme-over-tcp-how-it-supercharges-ssd-storage-using-standard-ip-networks.html
- nvme and tcp
- NVMe over TCP
- https://tekdeeps.com/what-is-nvme-over-tcp-how-to-use/
- Data in a Flash, Part III: NVMe over Fabrics Using TCP
- nvme-tcp storage operations
Qemu and NVMe
Hardware
Slots
M.2
The old way, on standard motherboards
U.2
Hotplug
Tools
nvme-cli
nvmet-cli
https://github.com/JunxiongGuan/nvmetcli
Documentation
NVMe device names
Example: /dev/nvme0n2p3
Means nvme device 0, namespace 2, partition 3 The first namespace, n1, will always exist
Namespaces
MNAM: Maximum Number of Allowed Namespaces
- nvme namespaces
- Drew Thorstensen - NVME namespaces
- https://narasimhan-v.github.io/2020/06/12/Managing-NVMe-Namespaces.html
HOWTO
List devices
nvme list
get details
nvme id-ctrl /dev/nvme0
if you want to find for example IP of a device:
nvme list-subsys /dev/nvme2n1
Namespaces
Difference between size and capacity
nsze vs ncap flbas = blocksize
To check them in a namespace:
nvme id-ns /dev/nvme1n3 | egrep "nsze|ncap|flbas"
Create namespace
List namespaces
nvme list-ns /dev/nvme1
Show info about namespace
nvme id-ns /dev/nvme1n1
Show number of available namespaces
nvme id-ctrl /dev/nvme1|grep nn
Show total capacity
nvme id-ctrl /dev/nvme1|grep tnvmcap
Show unallocated capacity
nvme id-ctrl /dev/nvme1|grep unvmcap
NVMe over fabrics
https://www.linuxjournal.com/content/data-flash-part-iii-nvme-over-fabrics-using-tcp
You might want to use nvmetcli on the target:
https://github.com/JunxiongGuan/nvmetcli
Dependencies: python3-configshell-fb
NVMe/TCP Host
In this context Host means client/initiator
See Configuring an NVMe/TCP host
Client
Modules:
nvme_rdma nvme_core nvme_fabrics and more
Connect on boot
Check /etc/nvme/discovery.conf
systemctl enable nvmf-autoconnect.service
Show remote connections
nvme list-subsys
Monitoring nvme
FAQ
Failed to open /dev/nvme-fabrics: No such file or directory
modprobe nvme-tcp
or maybe rdma
Failed to write to /dev/nvme-fabrics: Connection refused
dmesg will probably show
nvme0: failed to connect socket: -111
Maybe you're using something like infiniband, try
nvme discover -t rdma ...
Failed to write to /dev/nvme-fabrics: Invalid argument
nvme nvme0: Invalid MNAN value 1024
Try
modprobe nvme_core multipath=N
(remember to rmmod first :)
nvme_fabrics: no handler found for transport drma.
Check your /etc/nvme/discover.conf, it should be rdma :)
