ZFS

Links

• http://open-zfs.org
• http://www.edplese.com/samba-with-zfs.html
• ZFS calculator
• another zfs calculator
• ZFS clustering
• https://jrs-s.net/2015/02/03/will-zfs-and-non-ecc-ram-kill-your-data/ ZFS and ECC]
• ZFS troubleshooting/disk replacement
• Creating a ZFS HA Cluster using shared or shared-nothing storage
• ZFS 101
• Raidz expansion
• Basic guide to working with zfs
• Archlinux page on ZFS
• Raidz basic concepts

Documentation

• zfs manpage
• ZFS on Linux
• openzfs wiki
• https://wiki.gentoo.org/wiki/ZFS
• ZFS cheatsheet
• http://wiki.freebsd.org/ZFSQuickStartGuide
• Opensolaris ZFS intro
• raidz types reference
• ZFS fragmentation
• raidz

ARC/Caching

• Configuring ZFS Cache for High-Speed IO
• ZFS Arc various sizes
• Activity of the ZFS ARC
• Understanding ARC hits
• ZFS Caching
• OpenZFS: All about the cache vdev or L2ARC

• What is stored in parity

Tuning ZFS

• ZFS Performance and Tuning
• Configuring ZFS Cache for High-Speed IO
• ZFS Tuning and Optimisation

(Monitoring and Tuning ZFS Performance

ARC statistics

• Tuning module parameters

cat /proc/spl/kstat/zfs/arcstats

data_size

size of cached user data

hdr_size

size of L2ARC headers stored in main ARC

metadata_size

size of cached metadata

Tools

• ztop
• iozstat
• arc_summary
• zfs-linux-tools kstat-analyzer is rather helpful

Processes

arc_evict

Evict buffers from list until we've removed the specified number of bytes. Move the removed buffers to the appropriate evict state. If the recycle flag is set, then attempt to "recycle" a buffer: - look for a buffer to evict that is `bytes' long. - return the data block from this buffer rather than freeing it. This flag is used by callers that are trying to make space for a new buffer in a full arc cache.

This function makes a "best effort". It skips over any buffers it can't get a hash_lock on, and so may not catch all candidates. It may also return without evicting as much space as requested.

Commands

Getting arc statistics

arc_summary

Tip, for details use

arc_summary -d

There is also

cat /proc/spl/kstat/zfs/arcstats

Getting IO statistics

zpool iostat -v 300

Terms and acronyms

vdev

Virtual Device.

• ZFS Virtual disks

ARC

Adaptive Replacement Cache

Portion of RAM used to cache data to speed up read performance

L2ARC

Level 2 Adaptive Replacement Cache

"L2ARC is usually considered if hit rate for the ARC is below 90% while having 64+ GB of RAM"

SSD cache

DMU

Data Management Unit

MFU

Most Frequently Used

MRU

Most Recently Used

zvol

kind of block device whose space is allocated from the pool, useful for iscsi targets

Scrubbing

Checking disks/data integrity

zpool status <poolname | grep scrub

and

zpool scrub <poolname>

probably taken care of by cron.

ZIL

the space synchronous writes are logged before the confirmation is sent back to the client

prefetch

See /proc/spl/kstat/zfs/zfetchstats

• Understanding ZFS prefetch
• Tuning of the ZFS module
• Understanding ZFS prefetch
• Some basic ZFS ARC statistics and prefetching
• Some notes on ZFS prefetch related stats
• Activity of the ZFS ARC

HOWTO

Getting statistics

zpool iostat

Create zfs filesystem

zfs create poolname/fsname

this also creates mountpoint

Add vdev to pool

zpool add mypool raidz1 sdg sdh sdi

Replace disk in zfs

Some links

• https://itectec.com/ubuntu/ubuntu-replacing-a-dead-disk-in-a-zpool/

Get information first:

Name of disk

zpool status

Find uid of disk to replace

take it offline

zpool offline poolname ata-WDC_WD20EFRX-68EUZN0_WD-WCC4M5RLZC6V

Get the disk guid:

zdb

guid: 15233236897831806877

Get list of disk by id:

ls -al /dev/disk/by-id

Save the id, shutdown, replace disk, boot:

Find the new disk:

ls -al /dev/disk/by-id

Run replace command. The id is the guid of the old disk, name is of the new disk

zpool replace tank /dev/disk/by-id/13450850036953119346 /dev/disk/by-id/ata-ST4000VN000-1H4168_Z302FQVZ

Showing information about ZFS pools and datasets

Show pools with sizes

zpool list 

or

zpool list -H -o name,size

Show reservations on datasets

zfs list -o name,reservations

Swap on zfs

https://askubuntu.com/questions/228149/zfs-partition-as-swap

vdevs

multiple vdevs

Multiple vdevs in a zpool get striped. What about balance?

invalid vdev specification

Probably means you need -f

show balance between vdevs

zpool iostat -v 'pool' [interval in seconds]

orjust

zpool iostat -vc 'pool'

Tuning arc settings

See Tuning ZFS modules parameters

zfs_arc_max

Linux defaults to giving 50% of RAM to arc, this is when:

cat /sys/module/zfs/parameters/zfs_arc_max
0
grep c_max /proc/spl/kstat/zfs/arcstats

To change this:

echo 5368709120 > /sys/module/zfs/parameters/zfs_arc_max

and add to /etc/modprobe.d/zfs.conf

zfs zfs_arc_max=5368709120

maybe you need

echo 3 > /proc/sys/vm/drop_caches

Tune zfs_arc_dnode_limit_percent

Assuming zfs_arc_dnode_limit = 0:

echo 20 > /sys/module/zfs/parameters/zfs_arc_dnode_limit_percent

In /etc/modprobe.d/zfs.conf:

options zfs zfs_arc_dnode_limit_percent=20

FAQ

Arc metadata size exceeds maximum=

So arc_meta_used > arc_meta_limit

show status and disks

zpool status

show drives/pools

zfs list
      

check raid level

zfs list -a

Estimate raidz speeds

raidz1: N/(N-1) * IOPS
raidz2: N/(N-2) * IOPS
raidz3: N/(N-3) * IOPS

VDEV cache disabled, skipping section

Looks like you just don't have l2arc cache

cannot export 'tank': pool is busy

After checking stuff like nfs etc try:

zfs unshare -a
zfs umount -a -f
zpool export -f tank