Module 6: Storage Management

Learning Objectives

By the end of this module, you will be able to:

Use disk partitioning tools to prepare storage devices
Create and manage different types of filesystems
Configure mount points and automate mounting with fstab
Implement Logical Volume Management for flexible storage allocation
Set up RAID arrays for improved performance or redundancy
Monitor storage health and troubleshoot common storage issues

1. Disk Partitioning Tools and Techniques

Understanding Storage Devices in Linux

In Linux, physical storage devices appear as files in the /dev directory. Traditionally, hard disks are named as:

/dev/sda, /dev/sdb, etc. for SATA/SCSI/USB devices
/dev/nvme0n1, /dev/nvme1n1, etc. for NVMe SSDs

Each physical device can be divided into partitions, which appear as:

/dev/sda1, /dev/sda2, etc. for traditional disks
/dev/nvme0n1p1, /dev/nvme0n1p2, etc. for NVMe drives

Partition Tables: MBR vs GPT

MBR (Master Boot Record):

Traditional format limited to 2TB disk size
Supports up to 4 primary partitions, or 3 primary + 1 extended (which can contain multiple logical partitions)
Contains the boot code in its first sector

Under the hood, MBR uses the first 512 bytes of the disk with a small table that contains partition information.

GPT (GUID Partition Table):

Modern format supporting disks larger than 2TB
Supports up to 128 partitions by default
Includes redundant partition tables for improved reliability
Required for UEFI boot systems

GPT stores its data structures at both the beginning and end of the disk, using unique GUIDs for each partition.

Using `fdisk`

# List all disks and partitions
sudo fdisk -l

# Partition a specific disk
sudo fdisk /dev/sdb

Within fdisk, common commands include m (help), n (new partition), d (delete), p (print table), t (change type), w (write changes), and q (quit without saving).

Using `parted`

# List all disk partitions
sudo parted -l

# Work with a specific disk
sudo parted /dev/sdb

# Create a new GPT partition table
sudo parted /dev/sdb mklabel gpt

# Create a partition using the entire disk
sudo parted /dev/sdb mkpart primary 0% 100%

Parted supports both interactive and command-line modes and works with various units (sectors, bytes, percentages).

Practical Exercise 1: Partition a Disk

Identify your disk:
```
sudo fdisk -l
```
Create partitions on your disk (replace /dev/sdX with your disk, e.g., /dev/sdb):
```
sudo fdisk /dev/sdX
```
Within fdisk:
- Type n to create a new partition
- Select p for primary
- Accept default partition number (1)
- Accept default first sector
- Type +1G for a 1GB partition
- Repeat as needed
- Type w to write changes and exit
Verify partitions:
```
sudo fdisk -l /dev/sdX
```

2. Filesystem Creation and Management

Understanding Linux Filesystems

A filesystem organizes data on a storage device. Common Linux filesystems include:

ext4: Default for many distributions
XFS: High-performance, especially for large files
Btrfs: Advanced features like snapshots and RAID
NTFS/FAT32: Windows-compatible
ZFS: Advanced, with integrated volume management

Filesystems organize data using inodes, directory entries, and data blocks.

Creating Filesystems with `mkfs`

# Format a partition with ext4
sudo mkfs.ext4 /dev/sdb1

# Format with XFS
sudo mkfs.xfs /dev/sdb2

# Format with a label
sudo mkfs.ext4 -L "DATAPART" /dev/sdb1

The mkfs commands create superblocks, inode tables, and journals (for ext4) on the partition.

Managing Filesystem Labels and UUIDs

# View UUIDs and labels
sudo blkid

# Set label on ext4
sudo e2label /dev/sdb1 "DATAPART"

# Set label on XFS
sudo xfs_admin -L "DATAPART" /dev/sdb1

Checking and Repairing Filesystems

# Check ext4 filesystem (unmounted)
sudo fsck.ext4 /dev/sdb1

# Force check on next boot (root filesystem)
sudo touch /forcefsck

# Check XFS filesystem
sudo xfs_repair /dev/sdb2

3. Mount Points and `fstab` Configuration

Understanding the Mount Process

# Create a mount point
sudo mkdir /mnt/data

# Mount a filesystem temporarily
sudo mount /dev/sdb1 /mnt/data

# Unmount a filesystem
sudo umount /mnt/data

Mounting updates the kernel’s mount table, integrating the filesystem into the directory tree.

Persistent Mounts with `/etc/fstab`

# /etc/fstab example
#                      
/dev/sdb1         /mnt/data      ext4          defaults        0       2
UUID=1234-5678    /mnt/backup    xfs           defaults        0       2
LABEL=DATAPART    /mnt/label     ext4          defaults        0       2

Using UUIDs or labels ensures persistent identification.

Common Mount Options

Examples:

# Mount with specific options
sudo mount -o noexec,nosuid /dev/sdb1 /mnt/data

In /etc/fstab:

/dev/sdb1   /mnt/data   ext4   noexec,nosuid   0   2

Testing `fstab` Configuration

sudo umount /mnt/data
sudo mount -a

Practical Exercise 2: Create and Mount a Filesystem

Create a filesystem:
```
sudo mkfs.ext4 /dev/sdX1
```
Create a mount point:
```
sudo mkdir /mnt/mydata
```
Mount the filesystem:
```
sudo mount /dev/sdX1 /mnt/mydata
```
Get the UUID:
```
sudo blkid /dev/sdX1
```

Add an entry to /etc/fstab:

sudo bash -c 'echo "UUID=your-uuid-here  /mnt/mydata  ext4  defaults  0  2" >> /etc/fstab'

Test:

sudo umount /mnt/mydata
sudo mount -a
df -h /mnt/mydata

4. Logical Volume Management (LVM)

Understanding LVM Concepts

LVM provides an abstraction layer between physical storage and filesystems.

Physical Volumes (PVs): Storage devices or partitions
Volume Groups (VGs): Pools of PVs
Logical Volumes (LVs): Virtual partitions created from VGs

This abstraction enables resizing filesystems, aggregating disks, and creating snapshots.

Setting Up LVM

# Install LVM tools
sudo apt install lvm2

# Create Physical Volumes
sudo pvcreate /dev/sdb /dev/sdc1

# Create a Volume Group
sudo vgcreate myvg /dev/sdb /dev/sdc1

# Create a Logical Volume
sudo lvcreate -n mylv -L 10G myvg

# Format the Logical Volume
sudo mkfs.ext4 /dev/myvg/mylv

# Mount the Logical Volume
sudo mkdir /mnt/mylvm
sudo mount /dev/myvg/mylv /mnt/mylvm

Managing LVM Components

# Display LVM information
sudo pvs
sudo vgs
sudo lvs

# Extend a Volume Group
sudo vgextend myvg /dev/sdd1

# Extend a Logical Volume and its filesystem
sudo lvextend -L +5G /dev/myvg/mylv
sudo resize2fs /dev/myvg/mylv

# Create an LVM snapshot
sudo lvcreate -s -n mylv-snapshot -L 1G /dev/myvg/mylv

Practical Exercise 3: Set Up and Manage LVM

Create Physical Volumes:
```
sudo pvcreate /dev/sdX1 /dev/sdX2
```

Create a Volume Group:

sudo vgcreate vg_data /dev/sdX1 /dev/sdX2

Create a Logical Volume:
```
sudo lvcreate -n lv_data -L 1G vg_data
```

Format and mount:

sudo mkfs.ext4 /dev/vg_data/lv_data
sudo mkdir /mnt/lvdata
sudo mount /dev/vg_data/lv_data /mnt/lvdata

Extend the Logical Volume:

sudo lvextend -L +500M /dev/vg_data/lv_data
sudo resize2fs /dev/vg_data/lv_data

Verify:
```
df -h /mnt/lvdata
```

5. RAID Configuration and Management

Understanding RAID

RAID combines multiple disks into a single logical unit for performance, capacity, or redundancy. Common RAID levels include:

RAID 0: Striping for performance (no redundancy)
RAID 1: Mirroring for redundancy
RAID 5: Distributed parity for redundancy with storage efficiency
RAID 6: Dual parity for higher fault tolerance
RAID 10: Combination of RAID 1 and RAID 0 for performance and redundancy

Under the hood, RAID distributes data blocks across disks according to specific algorithms for the RAID level.

Software RAID with `mdadm`

# Install mdadm
sudo apt install mdadm

# Create a RAID 1 array with two disks
sudo mdadm --create /dev/md0 --level=1 --raid-devices=2 /dev/sdb1 /dev/sdc1

# View RAID details
sudo mdadm --detail /dev/md0

# Check RAID status
cat /proc/mdstat

# Save RAID configuration
sudo mdadm --detail --scan >> /etc/mdadm/mdadm.conf

Managing RAID Arrays

# Add a spare disk
sudo mdadm --add /dev/md0 /dev/sdd1

# Remove a disk
sudo mdadm /dev/md0 --fail /dev/sdb1 --remove /dev/sdb1

# Grow a RAID array
sudo mdadm --add /dev/md0 /dev/sdd1
sudo mdadm --grow /dev/md0 --raid-devices=3

# Stop and reassemble an array
sudo mdadm --stop /dev/md0
sudo mdadm --assemble /dev/md0 /dev/sdb1 /dev/sdc1

6. Storage Monitoring and Troubleshooting

Monitoring Storage Usage

Regular monitoring prevents disk space issues:

df -h
du -h --max-depth=1 /var
find /var -type f -size +100M -exec ls -lh {} \;

Other tools include:

iotop - Monitor I/O usage by processes
iostat - Report CPU and I/O statistics
smartctl - Check disk health (from smartmontools)

sudo iotop
iostat -x 1
sudo smartctl -a /dev/sda

Common Storage Issues and Solutions

Disk Space Exhaustion: Identify large files with du and find, clean up temporary files and logs.
Inode Exhaustion: Check with df -i and remove numerous small files.
Slow Disk Performance: Monitor I/O wait with top, iostat, and identify high I/O processes with iotop.
Filesystem Errors: Run filesystem checks (fsck or xfs_repair), inspect logs, and run S.M.A.R.T. tests with smartctl.
Failed RAID Array: Check status with cat /proc/mdstat, replace failed disks, and rebuild the array.

Filesystem Maintenance Best Practices

Regularly monitor disk space, I/O performance, and S.M.A.R.T. status.
Tune filesystem parameters (e.g., using tune2fs).
Backup critical data frequently.
Schedule regular disk health checks with S.M.A.R.T.
Understand journaling and recovery capabilities of your filesystem.

Practical Exercise 4: Storage Monitoring and Troubleshooting

Check disk space and inode usage:
```
df -h
df -i
```
Find the largest directories in /var:
```
sudo du -h --max-depth=1 /var | sort -h
```

Find files larger than 100MB:

sudo find /var -type f -size +100M -exec ls -lh {} \; | sort -k5 -h

Check I/O statistics:
```
iostat -x 1 5
```
Check disk health:
```
sudo smartctl -a /dev/sda
```

Quick Reference Summary

Disk Partitioning

Command	Description
`sudo fdisk -l`	List partitions on all disks
`sudo fdisk /dev/sdb`	Partition a specific disk
`sudo parted -l`	List partitions using parted
`sudo parted /dev/sdb mklabel gpt`	Create a GPT partition table

Filesystem Management

Command	Description
`sudo mkfs.ext4 /dev/sdb1`	Create an ext4 filesystem
`sudo e2label /dev/sdb1 "DATAPART"`	Set a label on an ext4 filesystem
`sudo blkid`	List filesystems with UUIDs and labels
`sudo fsck.ext4 -f /dev/sdb1`	Force-check an ext4 filesystem

Mounting and `fstab`

Command	Description
`sudo mount /dev/sdb1 /mnt/data`	Mount a filesystem
`sudo umount /mnt/data`	Unmount a filesystem
`sudo mount -a`	Mount all filesystems in `/etc/fstab`
`df -h`	Check disk space usage

LVM Commands

Command	Description
`sudo pvcreate /dev/sdb1`	Create a physical volume
`sudo vgcreate myvg /dev/sdb1`	Create a volume group
`sudo lvcreate -n mylv -L 10G myvg`	Create a logical volume
`sudo lvextend -L +5G /dev/myvg/mylv`	Extend a logical volume
`sudo resize2fs /dev/myvg/mylv`	Resize the filesystem on a logical volume

RAID Management

Command	Description
`sudo mdadm --create /dev/md0 --level=1 --raid-devices=2 /dev/sdb1 /dev/sdc1`	Create a RAID 1 array
`sudo mdadm --detail /dev/md0`	Display detailed RAID information
`cat /proc/mdstat`	Check RAID status
`sudo mdadm --add /dev/md0 /dev/sdd1`	Add a spare disk

Storage Monitoring

Command	Description
`df -h`	Check disk space usage
`du -h --max-depth=1 /var`	Check directory sizes
`iostat -x 1`	Monitor I/O statistics
`sudo smartctl -a /dev/sda`	Check disk health via S.M.A.R.T.

This module covered essential storage management topics in Linux, from disk partitioning and filesystem creation to advanced techniques like LVM, RAID, and storage monitoring. With these skills, you can efficiently manage storage devices, automate mounting, implement redundancy, and troubleshoot common storage issues. Always back up critical data before making significant changes and monitor system health regularly.