This blogs lists the steps around creating RAID5 volume on Linux using local disks.
We will start by installing software RAID on Linux system
# yum install mdadm Loaded plugins: ulninfo Resolving Dependencies --> Running transaction check ---> Package mdadm.x86_64 0:4.1-1.0.1.el7 will be installed --> Processing Dependency: libreport-filesystem for package: mdadm-4.1-1.0.1.el7.x86_64 --> Running transaction check ---> Package libreport-filesystem.x86_64 0:2.1.11-43.0.1.el7 will be installed --> Finished Dependency Resolution Dependencies Resolved ================================================================================ Package Arch Version Repository Size ================================================================================ Installing: mdadm x86_64 4.1-1.0.1.el7 ol7_latest 435 k Installing for dependencies: libreport-filesystem x86_64 2.1.11-43.0.1.el7 ol7_latest 40 k Transaction Summary ================================================================================ Install 1 Package (+1 Dependent package) Total download size: 475 k Installed size: 1.0 M Is this ok [y/d/N]: y Downloading packages: (1/2): libreport-filesystem-2.1.11-43.0.1.el7.x86_64.rpm | 40 kB 00:00 (2/2): mdadm-4.1-1.0.1.el7.x86_64.rpm | 435 kB 00:00 -------------------------------------------------------------------------------- Total 1.8 MB/s | 475 kB 00:00 Running transaction check Running transaction test Transaction test succeeded Running transaction Installing : libreport-filesystem-2.1.11-43.0.1.el7.x86_64 1/2 Installing : mdadm-4.1-1.0.1.el7.x86_64 2/2 Verifying : mdadm-4.1-1.0.1.el7.x86_64 1/2 Verifying : libreport-filesystem-2.1.11-43.0.1.el7.x86_64 2/2 Installed: mdadm.x86_64 0:4.1-1.0.1.el7 Dependency Installed: libreport-filesystem.x86_64 0:2.1.11-43.0.1.el7 Complete!
Now let’s identify disks which we will use for creating RAID5 volume. These disks can be local or storage LUNs. You can list the disks using running below command. In this example we will use 5 local disks – /dev/xvd(b-f)
# fdisk -l | grep /dev/ Disk /dev/xvda: 53.7 GB, 53687091200 bytes, 104857600 sectors /dev/xvda1 * 2048 4098047 2048000 83 Linux /dev/xvda2 4098048 86018047 40960000 83 Linux /dev/xvda3 86018048 104857599 9419776 82 Linux swap / Solaris Disk /dev/xvdd: 53.7 GB, 53687091200 bytes, 104857600 sectors Disk /dev/xvde: 53.7 GB, 53687091200 bytes, 104857600 sectors Disk /dev/xvdb: 53.7 GB, 53687091200 bytes, 104857600 sectors Disk /dev/xvdc: 53.7 GB, 53687091200 bytes, 104857600 sectors Disk /dev/xvdf: 53.7 GB, 53687091200 bytes, 104857600 sectors
Before you begin make sure that selected drives do not have existing RAID configured.
# mdadm -E /dev/xvd[b-f] mdadm: No md superblock detected on /dev/xvdb. mdadm: No md superblock detected on /dev/xvdc. mdadm: No md superblock detected on /dev/xvdd. mdadm: No md superblock detected on /dev/xvde. mdadm: No md superblock detected on /dev/xvdf.
Next step is to create partition on selected disks using fdisk. Follow below steps for each selected disks – /dev/xvd(b-f) in this example
# fdisk /dev/xvdc
Welcome to fdisk (util-linux 2.23.2).
Changes will remain in memory only, until you decide to write them.
Be careful before using the write command.
Device does not contain a recognized partition table
Building a new DOS disklabel with disk identifier 0x6845b853.
Command (m for help): n
Partition type:
p primary (0 primary, 0 extended, 4 free)
e extended
Select (default p): p
Partition number (1-4, default 1): 1
First sector (2048-104857599, default 2048):
Using default value 2048
Last sector, +sectors or +size{K,M,G} (2048-104857599, default 104857599):
Using default value 104857599
Partition 1 of type Linux and of size 50 GiB is set
Command (m for help): t
Selected partition 1
Hex code (type L to list all codes): fd
Changed type of partition 'Linux' to 'Linux raid autodetect'
Command (m for help): p
Disk /dev/xvdc: 53.7 GB, 53687091200 bytes, 104857600 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk label type: dos
Disk identifier: 0x6845b853
Device Boot Start End Blocks Id System
/dev/xvdc1 2048 104857599 52427776 fd Linux raid autodetect
Command (m for help): w
The partition table has been altered!
Calling ioctl() to re-read partition table.
Syncing disks.
Once you’ve successfully created the partitions for all disks, you can verify the changes using below command
# mdadm -E /dev/xvd[b-f] /dev/xvdb: MBR Magic : aa55 Partition[0] : 104855552 sectors at 2048 (type fd) /dev/xvdc: MBR Magic : aa55 Partition[0] : 104855552 sectors at 2048 (type fd) /dev/xvdd: MBR Magic : aa55 Partition[0] : 104855552 sectors at 2048 (type fd) /dev/xvde: MBR Magic : aa55 Partition[0] : 104855552 sectors at 2048 (type fd) /dev/xvdf: MBR Magic : aa55 Partition[0] : 104855552 sectors at 2048 (type fd)
Now create RAID device (/dev/md1 in below example) using all the newly created disk partitions
# mdadm --create /dev/md1 --level=5 --raid-devices=5 /dev/xvdb1 /dev/xvdc1 /dev/xvdd1 /dev/xvde1 /dev/xvdf1 mdadm: Defaulting to version 1.2 metadata mdadm: array /dev/md1 started.
Note that it takes some time to complete the RAID creation. You can monitor the progress using below command. Below command will show you current % completed.
# cat /proc/mdstat
Personalities : [raid6] [raid5] [raid4]
md1 : active raid5 xvdf1[5] xvde1[3] xvdd1[2] xvdc1[1] xvdb1[0]
209575936 blocks super 1.2 level 5, 512k chunk, algorithm 2 [5/4] [UUUU_]
[=>...................] recovery = 7.2% (3808256/52393984) finish=13.1min speed=61363K/sec
unused devices: <none>
But you prefer to watch screen till 100% completion then run below command. Result of this command will be persistent on the screen and will show you real time progress.
# watch -n1 cat /proc/mdstat
Every 1.0s: cat /proc/mdstat Mon Dec 2 09:41:04 2019
Personalities : [raid6] [raid5] [raid4]
md1 : active raid5 xvdf1[5] xvde1[3] xvdd1[2] xvdc1[1] xvdb1[0]
209575936 blocks super 1.2 level 5, 512k chunk, algorithm 2 [5/4] [UUUU_]
[==>..................] recovery = 13.9% (7335304/52393984) finish=12.1mi
n speed=61622K/sec
unused devices: <none>
Once RAID creation is 100% you can run below command to see the summary of the RAID device configuration. RAID device is /dev/md1 in below example with 5 devices – /dev/xvd(b-f)1
# mdadm --detail /dev/md1
/dev/md1:
Version : 1.2
Creation Time : Mon Dec 2 09:39:05 2019
Raid Level : raid5
Array Size : 209575936 (199.87 GiB 214.61 GB)
Used Dev Size : 52393984 (49.97 GiB 53.65 GB)
Raid Devices : 5
Total Devices : 5
Persistence : Superblock is persistent
Update Time : Mon Dec 2 09:53:20 2019
State : clean
Active Devices : 5
Working Devices : 5
Failed Devices : 0
Spare Devices : 0
Layout : left-symmetric
Chunk Size : 512K
Consistency Policy : resync
Name : ip-10-10-10-207:1 (local to host ip-10-10-10-207)
UUID : 07f28a7b:3c70d798:1deecf9a:292ddd2c
Events : 18
Number Major Minor RaidDevice State
0 202 17 0 active sync /dev/xvdb1
1 202 33 1 active sync /dev/xvdc1
2 202 49 2 active sync /dev/xvdd1
3 202 65 3 active sync /dev/xvde1
5 202 81 4 active sync /dev/xvdf1
At this point we have RAID5 device created (/dev/md1). Now we need to create file system. In below example we are creating EXT4 file system
# mkfs.ext4 /dev/md1
mke2fs 1.42.9 (28-Dec-2013)
Filesystem label=
OS type: Linux
Block size=4096 (log=2)
Fragment size=4096 (log=2)
Stride=128 blocks, Stripe width=512 blocks
13099008 inodes, 52393984 blocks
2619699 blocks (5.00%) reserved for the super user
First data block=0
Maximum filesystem blocks=2199912448
1599 block groups
32768 blocks per group, 32768 fragments per group
8192 inodes per group
Superblock backups stored on blocks:
32768, 98304, 163840, 229376, 294912, 819200, 884736, 1605632, 2654208,
4096000, 7962624, 11239424, 20480000, 23887872
Allocating group tables: done
Writing inode tables: done
Creating journal (32768 blocks): done
Writing superblocks and filesystem accounting information: done
Once the file system is created on RAID5 device, create the mount point and mount the RAID5 device on the same.
# mkdir /mnt/ebs_r5 # mount /dev/md1 /mnt/ebs_r5 # df -h Filesystem Size Used Avail Use% Mounted on devtmpfs 32G 0 32G 0% /dev tmpfs 32G 0 32G 0% /dev/shm tmpfs 32G 17M 32G 1% /run tmpfs 32G 0 32G 0% /sys/fs/cgroup /dev/xvda2 39G 2.9G 34G 8% / /dev/xvda1 1.9G 234M 1.6G 13% /boot tmpfs 6.3G 0 6.3G 0% /run/user/1000 /dev/md1 197G 61M 187G 1% /mnt/ebs_r5
Also make sure you’ve added entry for new mount point in the /etc/fstab file. So that mount point will be persistent across system reboots.
# vi etc/fstab /dev/md1 /mnt/ebs_r5 ext4 defaults 0 0
Lastly most important step is to save the RAID configuration. Run below command to make sure RAID configuration is saved.
# mdadm --detail --scan --verbose >> /etc/mdadm.conf
At this point you’ve successfully created RAID5 device on Linux.
