ZFS Feature Flags

Illumos Meetup January 2012

Christopher Siden

The Problem
ZFS on-disk formats are versioned using a single integer (last OpenSolaris version was 28 for pools) This worked when Sun controlled ZFS versioning, but in Illumos multiple organizations may be working on new ZFS versions in parallel What happens when Delphix uses ZFS version 30 for a new async destroy feature and Nexenta uses ZFS version 30 for RAIDZ 4? We don't want to have to coordinate allocation of version numbers between every organization working on ZFS Some organizations may want to maintain their own on-disk format changes separately from those maintained in Illumos

Feature Flags - A History

Originally proposed by the ZFS working group Described on zfs-discuss by Matt Ahrens in May (http://mail.opensolaris.
org/pipermail/zfs-discuss/2011-May/048514.html)

Last summer Basil Crow, an intern at Delphix, implemented the core components Delphix has integrated the "SPA/zpool versioning" part of that work into the next version of our product We plan to push this work back into Illumos soon

Version Numbers
Legacy version numbers still exist for pool versions 1-28 Continue using "zpool upgrade" to control legacy versions Feature flags has legacy version 1000 (implies 1-28) Legacy version never changes again Future on-disk format changes are enabled/disabled independently via feature flags Feature flags are exposed as pool properties Property name is: "feature@<org-name>:<feature-name>" <org-name> is the reverse DNS name of the organization which developed the feature (for guaranteed unique names) Properties can be shortened to "feature@<feature-name>" if this is unambiguous For example "feature@com.delphix:async_destroy" -> "feature@async_destroy"

Enabling Features
Feature properties can have one of three values: disabled - this feature will not be used, no on-disk format changes will be made (backwards compatible) enabled - this feature will be used, no on-disk format changes have been made yet, but the software may make them at any time (still backwards compatible) active - this feature has made backwards incompatible on-disk format changes to the pool The only value administrators can set is "enabled" Features cannot be set to "disabled" once enabled Enabling a feature automatically upgrades the pool legacy version to version 1000 Enabling a feature automatically enables any other features it depends on By default new pools are created with all supported features enabled (use the -d option to disable by default)

Example
root@dlpx:~# zpool create -o version=28 example ~/example # feature property exists, but is disabled due to pool version root@dlpx:~# zpool get all example NAME PROPERTY VALUE example version 28 example feature@com.delphix:async_destroy disabled root@dlpx:~# zpool set feature@async_destroy=enabled example # legacy version number has been automatically upgraded to 1000 # (displayed as '-' to indicate it is no longer in use) # feature property value has changed root@dlpx:~# zpool get all example NAME PROPERTY VALUE example version example feature@com.delphix:async_destroy enabled

Handling Incompatibilities
On-disk format changes can have one of two forms "features for write" must be supported by software in order to write to the pool, the feature does not need to be supported to open the pool in read-only mode "features for read" must be supported in order to read from the pool, there is no way to open a pool without reading from it For example, Delphix's async_destroy feature adds a new on-disk data structure to keep track of freed datasets, software does not need to know about this data structure to read the pool Trying to import a pool with unsupported features in the "active" state prints detailed error information listing the incompatible features

Examples
# import pool with unsupported feature for read root@dlpx:~# sudo zpool import -d ~/ example This pool uses the following feature(s) not supported by this system: com.example:unsupported (descriptive string) cannot import 'example': unsupported version or feature # import pool with unsupported feature for write root@dlpx:~# sudo zpool import -d ~/ example This pool uses the following feature(s) not supported by this system: com.example:unsupported (descriptive string) cannot import 'example': unsupported version or feature root@dlpx:~# sudo zpool import -d ~/ -o readonly=on example root@dlpx:~#

For sysadmins
The task of the average administrator has not changed 1. Upgrade software 2. Use the latest pool version zpool upgrade -a Additional benefits for administrators with special needs Evaluate and enable features one at a time (particularly useful if you are running a ZFS aware appliance) Gives features the option to "undo" their on disk changes changing the feature's state from "active" back to "enabled", making the pool compatible with an older software version again (some features may not be able to do this)

Async Destroy
Asynchronous destruction of ZFS filesystems Before Time to run "zfs destroy pool/fs" depends on size of pool/fs Interrupted destroy (e.g. by system crash) is completed the next time the pool is opened (i.e. next boot hangs) After Destroy operation does minimal metadata manipulation in a single transaction Data blocks are moved to free list Free list is processed in the background until empty

Adding New Features

Block comment in zfeature.c Each enabled feature has a reference count on each pool 0 means "enabled" Anything else means "active" Features are free to manipulate their reference count as long as they follow the conventions for "enabled" vs. "active" Good idea to use something that will naturally go to 0 if the feature gets disabled (e.g. number of vdevs using RAIDZ 4) Enabling a feature should be a fast operation Minimize "enable time" work, defer as much as possible Use spa_feature_is_enabled() to check if feature's code path should be taken Within the feature's code path use spa_feature_is_active() to see if the ondisk format changes need to be made This keeps features "inactive" as long as possible

Adding New Features

The default operation should always be "enable all features" Avoid "incompatible" features Avoid features which some people will not want to enable If your feature should include "enable/disable" functionality by design DO NOT use the feature@ properties to manage this, have your feature use its own properties (e.g. what dedup does) Feature flags does not help merge code for two conflicting features What happens when Delphix uses compression algorithm #14 for one algorithm and Joyent uses it for another? What happens when Delphix uses the 59th bit in the block pointer as a flag and Joyent uses bits 56-63 to store an enum value? For now we need to be careful, in the future we hope to have standard ways of handling this (the same way feature flags gives us a way to handle version numbers)

The Future
ZPL/filesystem feature flags Enum indirection tables object types checksum/compression algos send stream record types More useful frameworks to help with merging features