読者です 読者をやめる 読者になる 読者になる

TECHLOGICS

HYLOGICS分室。作業メモやガジェットのレビューなど、主に技術系のエントリを取り扱います。

【CentOS】ZFS on Linuxの導入

Linux

経緯

今更ZFSを導入した経緯は以下に書いた。

blog.hylogics.com

具体的な作業時の手順(とログ)について記載しておくことにする。

環境

[kirine@office ~]$ cat /etc/centos-release
CentOS Linux release 7.2.1511 (Core) 

[kirine@office ~]$ uname -r
3.18.25

zfsリポジトリの追加

[kirine@office ~]$ sudo yum install http://archive.zfsonlinux.org/epel/zfs-release.el7.noarch.rpmLoaded plugins: fastestmirror, langpacks
zfs-release.el7.noarch.rpm                                                                                                                           | 4.3 kB  00:00:00
Examining /var/tmp/yum-root-T5ovI7/zfs-release.el7.noarch.rpm: zfs-release-1-2.el7.centos.noarch
Marking /var/tmp/yum-root-T5ovI7/zfs-release.el7.noarch.rpm to be installed
Resolving Dependencies
--> Running transaction check
---> Package zfs-release.noarch 0:1-2.el7.centos will be installed
--> Finished Dependency Resolution

Dependencies Resolved

============================================================================================================================================================================
 Package                               Arch                             Version                                     Repository                                         Size
============================================================================================================================================================================
Installing:
 zfs-release                           noarch                           1-2.el7.centos                              /zfs-release.el7.noarch                           2.4 k

Transaction Summary
============================================================================================================================================================================
Install  1 Package

Total size: 2.4 k
Installed size: 2.4 k
Is this ok [y/d/N]: y
Downloading packages:
Running transaction check
Running transaction test
Transaction test succeeded
Running transaction
  Installing : zfs-release-1-2.el7.centos.noarch                                                                                                                        1/1
  Verifying  : zfs-release-1-2.el7.centos.noarch                                                                                                                        1/1

Installed:
  zfs-release.noarch 0:1-2.el7.centos

Complete!

zfsのインストール

[kirine@office ~]$ sudo yum install zfs
Loaded plugins: fastestmirror, langpacks
zfs                                                                                                                                                  | 2.9 kB  00:00:00
zfs/x86_64/primary_db                                                                                                                                |  67 kB  00:00:00
Loading mirror speeds from cached hostfile
 * base: ftp.iij.ad.jp
 * epel: ftp.iij.ad.jp
 * extras: ftp.iij.ad.jp
 * updates: ftp.iij.ad.jp
google-chrome                                                                                                                                                           3/3
Resolving Dependencies
--> Running transaction check
---> Package zfs.x86_64 0:0.6.5.7-1.el7.centos will be installed
--> Processing Dependency: zfs-kmod = 0.6.5.7 for package: zfs-0.6.5.7-1.el7.centos.x86_64
--> Processing Dependency: spl = 0.6.5.7 for package: zfs-0.6.5.7-1.el7.centos.x86_64
--> Processing Dependency: libzpool2 = 0.6.5.7 for package: zfs-0.6.5.7-1.el7.centos.x86_64
--> Processing Dependency: libzfs2 = 0.6.5.7 for package: zfs-0.6.5.7-1.el7.centos.x86_64
--> Processing Dependency: libuutil1 = 0.6.5.7 for package: zfs-0.6.5.7-1.el7.centos.x86_64
--> Processing Dependency: libnvpair1 = 0.6.5.7 for package: zfs-0.6.5.7-1.el7.centos.x86_64
--> Processing Dependency: libzpool.so.2()(64bit) for package: zfs-0.6.5.7-1.el7.centos.x86_64
--> Processing Dependency: libzfs_core.so.1()(64bit) for package: zfs-0.6.5.7-1.el7.centos.x86_64
--> Processing Dependency: libzfs.so.2()(64bit) for package: zfs-0.6.5.7-1.el7.centos.x86_64
--> Processing Dependency: libuutil.so.1()(64bit) for package: zfs-0.6.5.7-1.el7.centos.x86_64
--> Processing Dependency: libnvpair.so.1()(64bit) for package: zfs-0.6.5.7-1.el7.centos.x86_64
--> Running transaction check
---> Package libnvpair1.x86_64 0:0.6.5.7-1.el7.centos will be installed
---> Package libuutil1.x86_64 0:0.6.5.7-1.el7.centos will be installed
---> Package libzfs2.x86_64 0:0.6.5.7-1.el7.centos will be installed
---> Package libzpool2.x86_64 0:0.6.5.7-1.el7.centos will be installed
---> Package spl.x86_64 0:0.6.5.7-1.el7.centos will be installed
--> Processing Dependency: spl-kmod = 0.6.5.7 for package: spl-0.6.5.7-1.el7.centos.x86_64
---> Package zfs-dkms.noarch 0:0.6.5.7-1.el7.centos will be installed
--> Processing Dependency: dkms >= 2.2.0.3-20 for package: zfs-dkms-0.6.5.7-1.el7.centos.noarch
--> Running transaction check
---> Package dkms.noarch 0:2.2.0.3-34.git.9e0394d.el7 will be installed
---> Package spl-dkms.noarch 0:0.6.5.7-1.el7.centos will be installed
--> Finished Dependency Resolution

Dependencies Resolved

============================================================================================================================================================================
 Package                                Arch                               Version                                                   Repository                        Size
============================================================================================================================================================================
Installing:
 zfs                                    x86_64                             0.6.5.7-1.el7.centos                                      zfs                              330 k
Installing for dependencies:
 dkms                                   noarch                             2.2.0.3-34.git.9e0394d.el7                                epel                              78 k
 libnvpair1                             x86_64                             0.6.5.7-1.el7.centos                                      zfs                               32 k
 libuutil1                              x86_64                             0.6.5.7-1.el7.centos                                      zfs                               38 k
 libzfs2                                x86_64                             0.6.5.7-1.el7.centos                                      zfs                              119 k
 libzpool2                              x86_64                             0.6.5.7-1.el7.centos                                      zfs                              419 k
 spl                                    x86_64                             0.6.5.7-1.el7.centos                                      zfs                               28 k
 spl-dkms                               noarch                             0.6.5.7-1.el7.centos                                      zfs                              426 k
 zfs-dkms                               noarch                             0.6.5.7-1.el7.centos                                      zfs                              1.9 M

Transaction Summary
============================================================================================================================================================================
Install  1 Package (+8 Dependent packages)

Total download size: 3.3 M
Installed size: 17 M
Is this ok [y/d/N]: y
Downloading packages:
(1/9): dkms-2.2.0.3-34.git.9e0394d.el7.noarch.rpm                                                                                                    |  78 kB  00:00:00
(2/9): libuutil1-0.6.5.7-1.el7.centos.x86_64.rpm                                                                                                     |  38 kB  00:00:00
(3/9): libnvpair1-0.6.5.7-1.el7.centos.x86_64.rpm                                                                                                    |  32 kB  00:00:00
(4/9): libzfs2-0.6.5.7-1.el7.centos.x86_64.rpm                                                                                                       | 119 kB  00:00:00
(5/9): libzpool2-0.6.5.7-1.el7.centos.x86_64.rpm                                                                                                     | 419 kB  00:00:00
(6/9): spl-0.6.5.7-1.el7.centos.x86_64.rpm                                                                                                           |  28 kB  00:00:00
(7/9): spl-dkms-0.6.5.7-1.el7.centos.noarch.rpm                                                                                                      | 426 kB  00:00:00
(8/9): zfs-dkms-0.6.5.7-1.el7.centos.noarch.rpm                                                                                                      | 1.9 MB  00:00:00
(9/9): zfs-0.6.5.7-1.el7.centos.x86_64.rpm                                                                                                           | 330 kB  00:00:02
----------------------------------------------------------------------------------------------------------------------------------------------------------------------------
Total                                                                                                                                       766 kB/s | 3.3 MB  00:00:04
Running transaction check
Running transaction test
Transaction test succeeded
Running transaction
  Installing : libuutil1-0.6.5.7-1.el7.centos.x86_64                                                                                                                    1/9
  Installing : libnvpair1-0.6.5.7-1.el7.centos.x86_64                                                                                                                   2/9
  Installing : libzpool2-0.6.5.7-1.el7.centos.x86_64                                                                                                                    3/9
  Installing : dkms-2.2.0.3-34.git.9e0394d.el7.noarch                                                                                                                   4/9
  Installing : spl-dkms-0.6.5.7-1.el7.centos.noarch                                                                                                                     5/9
Loading new spl-0.6.5.7 DKMS files...
Building for 3.18.25-1
Building initial module for 3.18.25-1
Done.

spl:
Running module version sanity check.
 - Original module
   - No original module exists within this kernel
 - Installation
   - Installing to /lib/modules/3.18.25-1/extra/

splat.ko:
Running module version sanity check.
 - Original module
   - No original module exists within this kernel
 - Installation
   - Installing to /lib/modules/3.18.25-1/extra/
Adding any weak-modules

depmod................

DKMS: install completed.
  Installing : spl-0.6.5.7-1.el7.centos.x86_64                                                                                                                          6/9
  Installing : zfs-dkms-0.6.5.7-1.el7.centos.noarch                                                                                                                     7/9
Loading new zfs-0.6.5.7 DKMS files...
Building for 3.18.25-1
Building initial module for 3.18.25-1
Done.

zavl:
Running module version sanity check.
 - Original module
   - No original module exists within this kernel
 - Installation
   - Installing to /lib/modules/3.18.25-1/extra/

znvpair.ko:
Running module version sanity check.
 - Original module
   - No original module exists within this kernel
 - Installation
   - Installing to /lib/modules/3.18.25-1/extra/

zunicode.ko:
Running module version sanity check.
 - Original module
   - No original module exists within this kernel
 - Installation
   - Installing to /lib/modules/3.18.25-1/extra/

zcommon.ko:
Running module version sanity check.
 - Original module
   - No original module exists within this kernel
 - Installation
   - Installing to /lib/modules/3.18.25-1/extra/

zfs.ko:
Running module version sanity check.
 - Original module
   - No original module exists within this kernel
 - Installation
   - Installing to /lib/modules/3.18.25-1/extra/

zpios.ko:
Running module version sanity check.
 - Original module
   - No original module exists within this kernel
 - Installation
   - Installing to /lib/modules/3.18.25-1/extra/
Adding any weak-modules

depmod...

DKMS: install completed.
  Installing : libzfs2-0.6.5.7-1.el7.centos.x86_64                                                                                                                      8/9
  Installing : zfs-0.6.5.7-1.el7.centos.x86_64                                                                                                                          9/9
  Verifying  : spl-0.6.5.7-1.el7.centos.x86_64                                                                                                                          1/9
  Verifying  : dkms-2.2.0.3-34.git.9e0394d.el7.noarch                                                                                                                   2/9
  Verifying  : zfs-0.6.5.7-1.el7.centos.x86_64                                                                                                                          3/9
  Verifying  : libzfs2-0.6.5.7-1.el7.centos.x86_64                                                                                                                      4/9
  Verifying  : libnvpair1-0.6.5.7-1.el7.centos.x86_64                                                                                                                   5/9
  Verifying  : spl-dkms-0.6.5.7-1.el7.centos.noarch                                                                                                                     6/9
  Verifying  : libuutil1-0.6.5.7-1.el7.centos.x86_64                                                                                                                    7/9
  Verifying  : zfs-dkms-0.6.5.7-1.el7.centos.noarch                                                                                                                     8/9
  Verifying  : libzpool2-0.6.5.7-1.el7.centos.x86_64                                                                                                                    9/9

Installed:
  zfs.x86_64 0:0.6.5.7-1.el7.centos

Dependency Installed:
  dkms.noarch 0:2.2.0.3-34.git.9e0394d.el7   libnvpair1.x86_64 0:0.6.5.7-1.el7.centos   libuutil1.x86_64 0:0.6.5.7-1.el7.centos   libzfs2.x86_64 0:0.6.5.7-1.el7.centos
  libzpool2.x86_64 0:0.6.5.7-1.el7.centos    spl.x86_64 0:0.6.5.7-1.el7.centos          spl-dkms.noarch 0:0.6.5.7-1.el7.centos    zfs-dkms.noarch 0:0.6.5.7-1.el7.centos

Complete!

ZFSの有効化

[kirine@office ~]$ cat /proc/filesystems
nodev     sysfs
nodev     rootfs
nodev     ramfs
nodev     bdev
nodev     proc
nodev     cgroup
nodev     cpuset
nodev     tmpfs
nodev     devtmpfs
nodev     debugfs
nodev     securityfs
nodev     sockfs
nodev     pipefs
nodev     configfs
nodev     devpts
nodev     hugetlbfs
nodev     autofs
nodev     pstore
nodev     efivarfs
nodev     mqueue
     ext3
     ext2
     ext4
nodev     rpc_pipefs
nodev     nfsd
     vfat
     fuseblk
nodev     fuse
nodev     fusectl

[kirine@office ~]$ sudo /sbin/modprobe zfs

[kirine@office ~]$ cat /proc/filesystems
nodev     sysfs
nodev     rootfs
nodev     ramfs
nodev     bdev
nodev     proc
nodev     cgroup
nodev     cpuset
nodev     tmpfs
nodev     devtmpfs
nodev     debugfs
nodev     securityfs
nodev     sockfs
nodev     pipefs
nodev     configfs
nodev     devpts
nodev     hugetlbfs
nodev     autofs
nodev     pstore
nodev     efivarfs
nodev     mqueue
     ext3
     ext2
     ext4
nodev     rpc_pipefs
nodev     nfsd
     vfat
     fuseblk
nodev     fuse
nodev     fusectl
nodev     zfs

pool作成

[kirine@office ~]$ sudo zpool list
no pools available

[kirine@office ~]$ cat /proc/partitions
major minor  #blocks  name

   8       16 2930266584 sdb
   8       17     513024 sdb1
   8       18   52461568 sdb2
   8       19 2877085696 sdb3
   8       32 2930266584 sdc
   8       64 2930266584 sde
   8       65     204800 sde1
   8       66   41943040 sde2
   8       67    4194304 sde3
   8       68  441937920 sde4
   8       69 1464844288 sde5
   8       70  488669184 sde6
   8        0 2930266584 sda
   8        1     204800 sda1
   8        2     513024 sda2
   8        3   52461568 sda3
   8        4 2877085696 sda4
   8       48 2930266584 sdd
   8       80 2930266584 sdf
   8       96 1953514584 sdg
   8       97 1464844288 sdg1
   8       98  488669184 sdg2
   9      127   52428800 md127
   9      126     512960 md126
   9        2 1464712192 md2
   9        3 2876954624 md3
   9        4  488538112 md4

順調に終わるかと思いきや、蹴つまづいた。

[kirine@office ~]$ sudo zpool create tank raidz /dev/sdc /dev/sdd /dev/sdf
the kernel failed to rescan the partition table: 16
cannot label 'sdd': try using parted(8) and then provide a specific slice: -1

何かしらの原因でタイムアウトするとこのエラーが出るらしい。今回は停止したはずのRAID1ボリュームが自動再開されていた(mdadmが古いDiskを掴んだままだった)のが原因だった。再起動後、mdadmでスーパーブロックを削除してリトライし、成功。

[kirine@office ~]$ sudo zpool create tank raidz /dev/sdc /dev/sdd /dev/sdf -f

なお本件と直接の関連はないがfstabに編集漏れがあると正常に起動しないので注意(CentOS6まではfstabに不正なエントリがあってもシステム関連のボリュームでさえなければエラーだけ吐いて普通に起動した気がするのだが、CentOS7ではsystemdがお節介を焼いているようだ)。

確認

(作業時のログを消してしまったようなので、データ再配置後に改めて取得した情報を記載)

[kirine@office wait]$ sudo zpool list
[sudo] password for kirine: 
NAME   SIZE  ALLOC   FREE  EXPANDSZ   FRAG    CAP  DEDUP  HEALTH  ALTROOT
tank  8.12T  4.75T  3.37T         -    28%    58%  1.00x  ONLINE  -
[kirine@office wait]$ sudo zpool status tank
  pool: tank
 state: ONLINE
  scan: none requested
config:

    NAME        STATE     READ WRITE CKSUM
    tank        ONLINE       0     0     0
      raidz1-0  ONLINE       0     0     0
        sdc     ONLINE       0     0     0
        sdd     ONLINE       0     0     0
        sdf     ONLINE       0     0     0

errors: No known data errors

感想

お手軽ではあるものの、/dev配下に配置されなかったり自動でマウントされたりとそれなりに違和感がある。システムディスクはmdadmでRAID1のままなので運用コストが増えてしまったのも少し心残り。

あと、普通に(大きなファイルコピーしたりするとそれなりに気になるレベルで)アクセス遅いです。HDDの数を増やすとそれなりに速くなるとの噂なので、これから導入する方はご検討ください。

その他

参考:ZFS on Linuxを仮想CentOS7上で検証した - Qiita

ZFSは何故遅いのかについて:ftp-adminの憂鬱: RAID-Zは遅いよ