Storage Management
a. Scanning storage Interconnects (scanning for new storage) b. Storage Adding Examples c. WWID and UUID d. Removing Storage from a system e. Re-read The Partition Table Without Rebooting
To check the Link status of an HBA Card
# cd /sys/class/fc_host/host# # cat port_state Online
To find out the WWPN
# cat /sys/class/fc_host/host/port_name 0x10000000c9b0ed59
Scanning Storage Interconnects
The following commands can be used to scan storage interconnects.
echo "1" > /sys/class/fc_host/hosth/issue_lip
This operation performs a Loop Initialization Protocol (LIP) and then scans the interconnect and causes the SCSI layer to be updated to reflect the devices currently on the bus. A LIP is, essentially, a bus reset, and will cause device addition and removal. This procedure is necessary to configure a new SCSI target on a Fibre Channel interconnect.
Bear in mind that issue_lip is an asynchronous operation. The command may complete before the entire scan has completed. You must monitor /var/log/messages to determine when it is done.
The lpfc (emulex) and qla2xxx (Qlogic) drivers support issue_lip
/usr/bin/rescan-scsi-bus.sh
This script is included in Red Hat Enterprise Linux 5.4 and all future updates as part of sg3_utils RPM. By default, this script scans all the SCSI buses on the system, updating the SCSI layer to reflect new devices on the bus. The script provides additional options to allow device removal and the issuing of LIPs.
echo "- - -" > /sys/class/scsi_host/hosth/scan
This command is used to add a storage device or path. In the above command, the Channel Number, SCSI Target ID, and LUN values are replaced by wildcards. This procedure will add LUNs, but not remove them.
rmmod driver-name or modprobe driver-name
These commands completely re-initialize the state of all interconnects controlled by the driver. Although this is extreme, it may be appropriate in some situations. This may be used, for example, to re-start the driver with a different module parameter value.
More information about storage reconfiguration can be found from the following Redhat Guide.
Storage Reconfiguration Guide
If the above methods are not working, try installing the latest driver by downloading it from Emulex or Qlogic web sites.
If emulex cards are used, One Connect Manager can be installed on the system to manage the LUNs and cards.
Storage Adding examples.
To configure the newly added LUNS on RHEL:
# ls /sys/class/fc_host host0 host1 # fdisk -l 2>/dev/null | egrep '^Disk' | egrep -v 'dm-' | wc -l # echo "1" > /sys/class/fc_host/host0/issue_lip # echo "- - -" > /sys/class/scsi_host/host0/scan # echo "1" > /sys/class/fc_host/host1/issue_lip # echo "- - -" > /sys/class/scsi_host/host1/scan # cat /proc/scsi/scsi | egrep -i 'Host:' | wc -l # fdisk -l 2>/dev/null | egrep '^Disk' | egrep -v 'dm-' | wc -l
To scan new LUNs on Linux operating system which is using QLogic driver
You need to find out driver proc file /proc/scsi/qlaXXX. For example on my system it is /proc/scsi/qla2300/0
Once file is identified you need to type following command (login as the root user):
# echo "scsi-qlascan" > /proc/scsi/qla2300/0 # cat /proc/scsi/qla2300/0
Now use the script rescan-scsi-bus.sh to configure new LUN as a device. Run script as follows:
# ./rescan-scsi-bus.sh -l -w
or
# /bin/rescan-scsi-bus.sh -w --hosts=4-5 --luns=0-31
For older systems, the script can be here
How to differentiate local storage from SAN LUN
The output of ls -l /sys/block/*/device should give you an idea about how each device is connected to the system.
In the example below red is a virtual disk on an internal RAID controller, green is a CD-ROM connected via an IDE controller, and the rest are SAN-connected SCSI disks where "hostN" refers to the instance of the Host Bus Adapter they are connected to.
# ls -l /sys/block/*/device lrwxrwxrwx 1 root root 0 Sep 19 02:11 /sys/block/cciss!c0d0/device -> ../../devices/pci0000:00/0000:00:04.0/0000:0d:00.0/disk0 lrwxrwxrwx 1 root root 0 Sep 19 02:11 /sys/block/hda/device -> ../../devices/pci0000:00/0000:00:1f.1/ide0/0.0 lrwxrwxrwx 1 root root 0 Sep 18 14:58 /sys/block/sda/device -> ../../devices/pci0000:00/0000:00:02.0/0000:13:00.0/host0/target0:0:0/0:0:0:0 lrwxrwxrwx 1 root root 0 Sep 18 14:58 /sys/block/sdb/device -> ../../devices/pci0000:00/0000:00:02.0/0000:13:00.0/host0/target0:0:0/0:0:0:1 lrwxrwxrwx 1 root root 0 Sep 18 14:58 /sys/block/sdc/device -> ../../devices/pci0000:00/0000:00:02.0/0000:13:00.0/host0/target0:0:0/0:0:0:64 lrwxrwxrwx 1 root root 0 Sep 18 14:58 /sys/block/sdd/device -> ../../devices/pci0000:00/0000:00:02.0/0000:13:00.0/host0/target0:0:0/0:0:0:120 ...
WWID and UUID
The World Wide Identifier (WWID) can be used in reliably identifying devices. It is a persistent, system-independent ID that the SCSI Standard requires from all SCSI devices. The WWID identifier is guaranteed to be unique for every storage device, and independent of the path that is used to access the device.
This identifier can be obtained by issuing a SCSI Inquiry to retrieve the Device Identification Vital Product Data (page 0x83) or Unit Serial Number (page 0x80). The mappings from these WWIDs to the current /dev/sd names can be seen in the symlinks maintained in the /dev/disk/by-id/' directory.
For example, a device with a page 0x83 identifier would have:
scsi-3600508b400105e210000900000490000 -> ../../sda
Or, a device with a page 0x80 identifier would have:
scsi-SSEAGATE_ST373453LW_3HW1RHM6 -> ../../sda
Red Hat Enterprise Linux automatically maintains the proper mapping from the WWID-based device name to a current /dev/sd name on that system. Applications can use the /dev/disk/by-id/name to reference the data on the disk, even if the path to the device changes, and even when accessing the device from different systems.
If there are multiple paths from a system to a device, device-mapper-multipath uses the WWID to detect this. Device-mapper-multipath then presents a single "pseudo-device" in /dev/mapper/wwid, such as /dev/mapper/3600508b400105df70000e00000ac0000.
The command multipath -l shows the mapping to the non-persistent identifiers:
Host:Channel:Target:LUN, /dev/sd name, and the major:minor number.
3600508b400105df70000e00000ac0000 dm-2 vendor,product [size=20G][features=1 queue_if_no_path][hwhandler=0][rw] \_ round-robin 0 [prio=0][active] \_ 5:0:1:1 sdc 8:32 [active][undef] \_ 6:0:1:1 sdg 8:96 [active][undef] \_ round-robin 0 [prio=0][enabled] \_ 5:0:0:1 sdb 8:16 [active][undef] \_ 6:0:0:1 sdf 8:80 [active][undef]
Device-mapper-multipath automatically maintains the proper mapping of each WWID-based device name to its corresponding /dev/sd name on the system. These names are persistent across pathchanges, and they are consistent when accessing the device from different systems.
When the user_friendly_names feature (of device-mapper-multipath) is used, the WWID is mapped to a name of the form /dev/mapper/mpathn. By default, this mapping is maintained in the file /var/lib/multipath/bindings. These mpathn names are persistent as long as that file is maintained.
UUID and Other Persistent Identifiers
If a storage device contains a filesystem, then that filesystem may provide one or both of the following:
- Universally Unique Identifier (UUID)
- Filesystem label
These identifiers are persistent, and based on metadata written on the device by certain applications.
They may also be used to access the device using the symlinks maintained by the operating system in the /dev/disk/by-label/ (e.g. boot -> ../../sda1 ) and /dev/disk/by-uuid/ (e.g. f8bf09e3-4c16-4d91-bd5e-6f62da165c08 -> ../../sda1) directories
Removing a Storage Device from system
To ensure a Clean Device Removal:
- Use umount to unmount any file systems that mounted the device
- Remove the device from any md and LVM volume using it
- If the device uses multipathing, run multipath -l and note all the paths to the device. Afterwards, remove the multipathed device using multipath -f device.
# multipath -l
# multipath -f mpath1 - Run blockdev -–flushbufs device to flush any outstanding I/O to all paths to the device.
# blockdev -–flushbufs /dev/sdc
# blockdev -–flushbufs /dev/sdd - Finally, remove each path to the device from the SCSI subsystem
# echo 1 > /sys/block/sdc/device/delete
# echo 1 > /sys/block/sdd/device/delete
Re-read The Partition Table Without Rebooting
If a new partition is created in the boot disk using fdisk, the Linux system need to be rebooted to get partition recognized. However with partprobe command you should able to create a new file system without rebooting the box. It is a program that informs the operating system kernel of partition table changes, by requesting that the operating system re-read the partition table.
After the fdisk command session (which makes changes to partition table) just type the following command:
# partprobe /dev/sdX
Multipath in RHEL
For detailed information for setting up DM-Multipath in RHEL, refer the official guide attached bellow.
'''RHEL Device Mapper Multipath Configuration and Administration Guide
- dm-multipath kernel module: It eroutes I/O and supports failover for paths and path groups.
- multipath command: Lists and configures multipath devices. Normally started up with /etc/rc.sysinit, it can also be started up by a udev program whenever a block device is added or it can be run by the initramfs file system.
- multipathd daemon: Monitors paths; as paths fail and come back, it may initiate path group switches. Provides for interactive changes to multipath devices. This must be restarted for any changes to the /etc/multipath.conf file.
- kpartx command: Creates device mapper devices for the partitions on a device It is necessary to use this command for DOS-based partitions with DM-MP. The kpartx is provided in its own package, but the device-mapper-multipath package depends on it.
Multipath Device Identifiers
Each multipath device has a World Wide Identifier (WWID), which is guaranteed to be globally unique and unchanging. By default, the name of a multipath device is set to its WWID. Alternately, you can set the user_friendly_names option in the multipath configuration file, which sets the alias to a node-unique name of the form mpathn.
You can also set the name of a multipath device to a name of your choosing by using the alias option in the multipaths section of the multipath configuration file.
For example, a node with two HBAs attached to a storage controller with two ports via a single unzoned FC switch sees four devices: /dev/sda, /dev/sdb, dev/sdc, and /dev/sdd. DM-Multipath creates a single device with a unique WWID that reroutes I/O to those four underlying devices according to the multipath configuration. When the user_friendly_names configuration option is set to yes, the name of the multipath device is set to mpathn.
When new devices are brought under the control of DM-Multipath, the new devices may be seen in three different places under the /dev directory: /dev/mapper/mpathn, /dev/mpath/mpathn, and /dev/dm-n.
- The devices in /dev/mapper are created early in the boot process. Use these devices to access the multipathed devices, for example when creating logical volumes.
- The devices in /dev/mpath are provided as a convenience so that all multipathed devices can be seen in one directory. These devices are created by the udev device manager and may not be available on startup when the system needs to access them. Do not use these devices for creating logical volumes or filesystems.
- Any devices of the form /dev/dm-n are for internal use only and should never be used.
Setting up DM-Multipath (Device Mapper Multipath
01. Install the device-mapper-multipath package if already not installed
# yum install device-mapper-multipath
02. Edit the multipath.conf file
- comment out the default blacklist
blacklist {
devnode "*"
} - change any of the existing defaults as needed
- save the configuration file
03. Start the multipath daemons and create the multipath device with the multipath command.
# modprobe dm-multipath # service multipathd start # multipath -v2 # chkconfig multipathd on
Since the value of user_friendly_name is set to yes in the configuration file the multipath devices will be created as /dev/mapper/mpathn.
Systems with separate /var filesystem
Problem: When using the user_friendly_names feature the device-mapper-multipath package stores a persistent database of device name to WWID mappings in the /var file system. This is problematic for systems that have been configured to mount a separate file system on the /var directory since the database will not be available during booting until this file system has been mounted.
This may lead to inconsistent device naming, configuration, and in some circumstances data corruption. This is due to devices being mis-identified during the boot process.
To avoid these problems it is required to relocate the bindings file database to a path within the root file system.
Solution: To relocate the bindings file on a system using multipath and having a separate /var file system, perform the following steps:
01. On systems that have been configured with a separate /var file system the following configuration directive should be added to the defaults section of the mulitpath.conf configuration file:
## Use user friendly names, instead of using WWIDs as names.
defaults {
user_friendly_names yes
bindings_file /etc/multipath/bindings
}
Use of a location within the /etc directory ensures that the bindings database is always available since this directory is required to be part of the root file system.
Note: Use of the bindings_file directive to specify a database path within the root file system is mandatory for systems configured with a separate /var file system.
02. Copy the existing bindings database to the new location
If multipath has already been configured on the system and device aliases have already been stored in the database, the existing bindings file should be copied to the new location
# cp /var/lib/multipath/bindings /etc/multipath/bindings
03. Flush and reconfigure all multipath devices or reboot
# service multipathd reload # multipath -F # multipath