Ubuntu 10.04 LTS. This server has an 8 disk raid 6 array.
cat /etc/mdadm/mdadm.conf
DEVICE partitions
HOMEHOST
MAILADDR my@email.com
ARRAY /dev/md0 level=raid6 num-devices=8 metadata=00.90 UUID=d6971ced:4f85c6ac:fea3373a:786177bd
This array is in the middle of rebuilding, which when rebuilding a single disk takes roughly 6-7 hours.
cat /proc/mdstat
Personalities : [linear] [multipath] [raid0] [raid1] [raid6] [raid5] [raid4] [raid10]
md0 : active raid6 sde1[8] sda1[0] sdh1[7] sdg1[6] sdf1[5] sdd1[3] sdc1[2] sdb1[1]
5860559616 blocks level 6, 64k chunk, algorithm 2 [8/7] [UUUU_UUU]
[>....................] recovery = 0.8% (8085504/976759936) finish=384.7min speed=41962K/sec
unused devices: none
This is pretty much how all the drives look:
sudo fdisk -l /dev/sda
Disk /dev/sda: 1000.2 GB, 1000204886016 bytes
255 heads, 63 sectors/track, 121601 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk identifier: 0x4c2d00f0
Device Boot Start End Blocks Id System
/dev/sda1 1 121601 976760001 fd Linux raid autodetect
This shows one drive as a spare, but that is how it looks when rebuilding, once rebuilt the array should show 8 drives:
sudo mdadm --examine /dev/sda1
mdadm: metadata format 00.90 unknown, ignored.
/dev/sda1:
Magic : a92b4efc
Version : 00.90.00
UUID : d6971ced:4f85c6ac:fea3373a:786177bd
Creation Time : Tue Aug 24 18:15:04 2010
Raid Level : raid6
Used Dev Size : 976759936 (931.51 GiB 1000.20 GB)
Array Size : 5860559616 (5589.07 GiB 6001.21 GB)
Raid Devices : 8
Total Devices : 8
Preferred Minor : 0
Update Time : Sun Dec 5 12:27:30 2010
State : clean
Active Devices : 7
Working Devices : 8
Failed Devices : 1
Spare Devices : 1
Checksum : 19f677b8 - correct
Events : 292
Chunk Size : 64K
Number Major Minor RaidDevice State
this 0 8 1 0 active sync /dev/sda1
0 0 8 1 0 active sync /dev/sda1
1 1 8 17 1 active sync /dev/sdb1
2 2 8 33 2 active sync /dev/sdc1
3 3 8 49 3 active sync /dev/sdd1
4 4 0 0 4 faulty removed
5 5 8 81 5 active sync /dev/sdf1
6 6 8 97 6 active sync /dev/sdg1
7 7 8 113 7 active sync /dev/sdh1
8 8 8 65 8 spare /dev/sde1
This particular server has the 8 1TB drives located in an external 8 drive enclosure connected via two eSATA cables to an add in card that resides in the main computer case.
I've found these cables to not stay attached very well and are prone to disconnect if the server or the external enclosure are moved or if the cats get too friskie around the machine. When this happens half of the drives drop out of the array and the array comes offline (obviously). What i've done in the past is to shut down the machine, re-seat the esata cables and reboot. At this point the boot process hangs.
Here is the boot log:
cat /var/log/boot.log
fsck from util-linux-ng 2.17.2
fsck from util-linux-ng 2.17.2
/dev/sdi1: clean, 224/124496 files, 80936/248832 blocks
/dev/mapper/ZEUS-root: clean, 96154/14630912 files, 1259268/58504192 blocks
*****Boot process hangs here******
Skipping /mnt/raid at user request
* Starting AppArmor profiles [ OK ]
* Starting disk temperature monitoring daemon hddtemp: [ OK ]
* Exporting directories for NFS kernel daemon... [ OK ]
* Starting NFS kernel daemon [ OK ]
* Starting Postfix Mail Transport Agent postfix [ OK ]
* Starting MD monitoring service mdadm --monitor mdadm: metadata format 00.90 unknown, ignored.
[ OK ]
Since it doesn't give any indication that it is attempting any raid business it is easy to assume that there is something else wrong with other drives/filesystems etc.
The message displayed on the screen durring boot is not always as innocuous as this, occasionally it can sound more serious such as:
mountall: fsck boot terminated with status 1 or there abouts. Which sounds a lot more serious.
Luckily (HA!) it is just stuck trying to initialize the array and all you have to do is press "s" on the keyboard and it will skip the raid initialization business on boot. You still will have to figure that out, but at least you will be able to boot and troubleshoot.
As for fixing the array, here is what I've done the last time this happened to me.
You can try to assemble the array, but it is probable that it won't work, or else you would have not had that trouble at boot, but using the --verbose flag can sometimes give you hints:
sudo mdadm --assemble --scan --verbose
I noticed a lot of "device busy" issues and I also noticed that one of my drives didn't look correct. All of my raid devices were partitions (e.g. /dev/sda1) except one was a device (e.g. /dev/sde). I thought this very strange, and wondered why I would create the array with partitions except for one create with the device. In all likeliness the partition table got goofed when the cord disconnected.
So I inspected the partition table for each drive
sudo fdisk -l /dev/sda
Disk /dev/sda: 1000.2 GB, 1000204886016 bytes
255 heads, 63 sectors/track, 121601 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk identifier: 0x4c2d00f0
Device Boot Start End Blocks Id System
/dev/sda1 1 121601 976760001 fd Linux raid autodetect
All of them looked like this except sde looked like this:
sudo fdisk -l /dev/sde
Disk /dev/sde1: 1000.2 GB, 1000202241024 bytes
255 heads, 63 sectors/track, 121600 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk identifier: 0x00000000
Disk /dev/sde doesn't contain a valid partition table
Or very similar to that, the main point is that sde looked substantially different then the others.
Lucky for me I chose to build my array as a raid6 array which means two drives can fail or be absent from the array and I can still operate the array. So what I decided to do was to force the array to assemble with 7 of 8 drives.
sudo mdadm --assemble --force --verbose /dev/md0 /dev/sd{a,b,c,d,f,g,h}1
Notice the lack of sde in the bunch.
To my joy the array started, I was able to mount it and all my data was there. a quick "cat /proc/mdstat" showed that the array was degraded but functioning (not rebuilding). WHEW!
Next I needed to setup sde for raid again, since it was all goofed up now.
sudo fdisk -l /dev/sde
Chose to create a new primary partition (command n) use the default values to create a partition that takes the whole disk
change the partitions system ID (command t ) to Linux raid auto (code fd)
write the table to the disk and exit (command w)
after that you should see output similar to this:
sudo fdisk -l /dev/sde
Disk /dev/sde: 1000.2 GB, 1000204886016 bytes
255 heads, 63 sectors/track, 121601 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk identifier: 0xe781d3a8
Device Boot Start End Blocks Id System
/dev/sde1 1 121601 976760001 fd Linux raid autodetect
Once you have the drive ready for the raid array you just need to re-add it.
sudo mdadm --re-add /dev/md0 /dev/sde1
If all goes well your array will start to rebuild.
cat /proc/mdstat
Personalities : [linear] [multipath] [raid0] [raid1] [raid6] [raid5] [raid4] [raid10]
md0 : active raid6 sde1[8] sda1[0] sdh1[7] sdg1[6] sdf1[5] sdd1[3] sdc1[2] sdb1[1]
5860559616 blocks level 6, 64k chunk, algorithm 2 [8/7] [UUUU_UUU]
[==>..................] recovery = 10.4% (102340224/976759936) finish=346.9min speed=42002K/sec
unused devices: none
After a few hours you should be all good :)
All of your drives in your array should look similar to this