As a Unix/Linux system administrator you need to deal with SAN storage and here are some of the functionalities.

1. New SAN storage request from SAN team.

2. Adding more storage into existing storage. ( As a system administrator you may need deal with this request more often )

3. SAN storage migration. (Most of the companies analyze their storage usage and migrate data into new storage according to their budget). As a system administrator you need to involve in the migration. When it comes to host base migration, system administrator needs to most of the job.

4. When spaces are not used by servers, system administrator gives to storage back SAN team.

5. SAN Storage troubleshooting

You can change the open file limits for individual users using the following method.

You might want to also set in /etc/security/limits.conf

userID soft nofile 8192
userID hard nofile 8192

If you want to set it for all the all the users, then you need to put *

* soft nofile 8192
* hard nofile 8192

Available lvm commands:
Use ‘lvm help ‘ for more information

dumpconfig      Dump active configuration
formats               List available metadata formats
help Display help for commands
lvchange            Change the attributes of logical volume(s)
lvconvert          Change logical volume layout
lvcreate             Create a logical volume
lvdisplay            Display information about a logical volume
lvextend             Add space to a logical volume
lvmchange        With the device mapper, this is obsolete and does nothing.
lvmdiskscan     List devices that may be used as physical volumes
lvmsadc             Collect activity data
lvmsar                 Create activity report
lvreduce             Reduce the size of a logical volume
lvremove           Remove logical volume(s) from the system
lvrename           Rename a logical volume
lvresize              Resize a logical volume
lvs                       Display information about logical volumes
lvscan                List all logical volumes in all volume groups
pvchange          Change attributes of physical volume(s)
pvresize            Resize physical volume(s)
pvcreate           Initialize physical volume(s) for use by LVM
pvdata              Display the on-disk metadata for physical volume(s)
pvdisplay        Display various attributes of physical volume(s)
pvmove          Move extents from one physical volume to another
pvremove     Remove LVM label(s) from physical volume(s)
pvresize          Resize a physical volume in use by a volume group
pvs                   Display information about physical volumes
pvscan           List all physical volumes
segtypes        List available segment types
vgcfgbackup Backup volume group configuration(s)
vgcfgrestore Restore volume group configuration
vgchange       Change volume group attributes
vgck                Check the consistency of volume group(s)
vgconvert    Change volume group metadata format
vgcreate       Create a volume group
vgdisplay      Display volume group information
vgexport        Unregister volume group(s) from the system
vgextend       Add physical volumes to a volume group
vgimport       Register exported volume group with system
vgmerge        Merge volume groups
vgmknodes  Create the special files for volume group devices in /dev
vgreduce      Remove physical volume(s) from a volume group
vgremove    Remove volume group(s)
vgrename     Rename a volume group
vgs Display   information about volume groups
vgscan           Search for all volume groups
vgsplit           Move physical volumes into a new volume group
version         Display software and driver version information

Some times when you boot the server it goes to grub prompt, you can assume there is a problem with grub loader. In order to resolve the problem, you need to boot up with recovery CD, and then you need to do the following command.

chroot /mnt/sysimage
grub-install /dev/hd(x)
reboot

echo “- – -” >/sys/class/scsi_host/host

Same for FiberChannel:

echo “1″ >/sys/class/fc_host/host

Also, you can reboot the system to Rescan.

Most of the time when there is a problem connecting to any specific port we use telnet command to verify, before that you can use NMAP command to verify whether port is open or close.

NMAP abbreviation is network mapper

NMAP is used to scan ports on a machine, either local or remote machine (just you require ip/hostname to scan).

NMAP is can be installed on windows, Sun Solaris machines too.

NMAP can be used to scan large networks, remember I am saying large networks.

Please do not try to use NMAP on machines which you don’t have permission.

Example1 : Using NMAP in normal way, i.e. to scan a particular system for open ports
#nmap hostname

Example2 : Scanning for a single port on a machine

#nmap –p 22 hostname
This will scan for 22 port is open on a host or not. And here –p indicates port.

Example3 : For scanning only ports
#nmap –F hostname
-F is for fast scan and this will not do any other scanning like IP address, hostname, operating system, and uptime etc. It’s very much fast as it said in man pages.

Example4 : For scanning only TCP ports
#nmap –sT hostname
Here s is for scanning and T is for only scanning of TCP ports

Example5 : For scanning only UDP ports
#nmap –sU hostname
Here U indicates UDP port scanning

Exmaple6 : Scanning for ports and to get what is the version of different services running on that machine
#nmap –sV hostname
V indicates version of each network service running on that host

Example7 : To check which protocol is supported by the remote machine
#nmap –sO hostname

Example8 : To scan a system for operating system and uptime details
# nmap -O hostname
-O is for operating system scan along with default port scan

Example9 : Scanning a network
#nmap networkID/subnetmask
For the above command you can try in this way
#nmap 192.168.0.0/24

HBA card is configuration is very important and if you can’t configure HBA card, then system won’t see your storage and you will have big issues.

Make sure HBA card instered properly, I had this problem with my Solaris box and after I have insereted the card properly, system detected the HBA card. Once it’s detected you can see it in BIOS.

Another important this is HBA card driver. If you don’t have proper driver, then it work and it’s doing all the communication between the SAN switches via the FC-SW Fibre Channel protocol and SAN storage arrays.

You need to keep the firmware up to date and upgrade to the latest version of firmware from the vendor support matrix site

If your system is hang or freeze during startup, it may caused by the device conflict (new device that recently installed). Remove the HBA card and reboot the system, if the problem is resolved, check the HBA card that you newly installed as it could be a faulty HBA card. If the problem still persists, try the HBA card in another machine. If the HBA card is work there, this is usually a conflict between the system and the HBA card. Contact the vendor to do the appropriate check on the HBA card by using card utility tool before you re-install the HBA card.

A problem HBA card may cause the link up and down. The healthy HBA card should never disconnect its light from the SAN switch for any reason unless the server is in the process of rebooting. For the server with new HBA card, the most common problem often either SAN switch port related or Fibre Channel cable related. To resolve the cable problem is easy, just replace another cable. If it is happen on the SAN switch level, check the GBIC / port, replace the GBIC if is it detected as faulty. Normally if the GBIC is unhealthy, the lights will go up and down or there is no signal from the GBIC at all.

Thank you sanduel.com

# vmstat  1   5
procs ———–memory———- —swap– —–io—- –system– —-cpu—-
r  b   swpd   free   buff  cache                          si   so    bi    bo   in    cs us sy id wa
0  0      0 129277344 346144 2033648    0    0     0     4    0     0  0  0 100  0
0  0      0 129277344 346144 2033648    0    0     0     0 1044   102  0  0 100  0
0  0      0 129277344 346144 2033648    0    0     0     0 1020    97  0  0 100  0
0  0      0 129277280 346144 2033648    0    0     0     0 1048    99  0  0 100  0
0  0      0 129277280 346144 2033648    0    0     0     0 1008    66  0  0 100  0

Proc:
——-
r: How many processes are waiting for CPU time.
b: Wait Queue – Process which are waiting for I/O (disk, network, user
input,etc..)

Memory:
———–
swpd: shows how many blocks are swapped out to disk (paged). Total Virtual
memory usage.

Note: you can see the swap area configured in server using “cat proc/swaps”

free: Idle Memory
buff: Memory used as buffers, like before/after I/O operations
cache: Memory used as cache by the Operating System

Swap:
———
si: How many blocks per second the operating system is swapping in. i.e
Memory swapped in from the disk (Read from swap area to Memory)
so: How many blocks per second the operating system is swaped Out. i.e
Memory swapped to the disk (Written to swap area and cleared from
Memory)

In Ideal condition, We like to see si and so at 0 most of the time, and we definitely don’t like to see more than 10 blocks per second.

IO:
——
bi: Blocks received from block device – Read (like a hard disk)
bo: Blocks sent to a block device – Write

System:
————-
in: The number of interrupts per second, including the clock.
cs: The number of context switches per second.

CPU:
——–
us: percentage of cpu used for running non-kernel code. (user time, including
nice time)
sy: percentage of cpu used for running kernel code. (system time – network, IO
interrupts, etc)
id: cpu idle time in percentage.
wa: percentage of time spent by cpu for waiting to IO.

If you used to monitor this data, you can understand how is your server doing during peak usage times.

Note: the memory, swap, and I/O statistics are in blocks, not in bytes. In Linux, blocks are usually 1,024 bytes (1 KB).

Some times system admins get confused that disks are from SAN or local disk. Please check the following information and this server is connected to SAN. If you look line 3,4,5 and 6, those are connected to storage.

[root@saturn block]# ls -l /sys/block/*/device
lrwxrwxrwx 1 root root 0 Jul 10 20:27 /sys/block/cciss!c0d0/device -> ../../devices/pci0000:00/0000:00:14.0/0000:03:00.0/disk0
lrwxrwxrwx 1 root root 0 Jul 10 23:00 /sys/block/hda/device -> ../../devices/pci0000:00/0000:00:06.1/ide0/0.0

lrwxrwxrwx 1 root root 0 Jul 10 23:00 /sys/block/sda/device -> ../../devices/pci0000:00/0000:00:13.0/0000:04:00.1/host1/target1:0:0/1:0:0:0
lrwxrwxrwx 1 root root 0 Jul 10 23:00 /sys/block/sdb/device -> ../../devices/pci0000:00/0000:00:13.0/0000:04:00.1/host1/target1:0:1/1:0:1:0
lrwxrwxrwx 1 root root 0 Jul 10 23:00 /sys/block/sdc/device -> ../../devices/pci0000:80/0000:80:13.0/0000:81:00.1/host3/target3:0:0/3:0:0:0
lrwxrwxrwx 1 root root 0 Jul 10 23:00 /sys/block/sdd/device -> ../../devices/pci0000:80/0000:80:13.0/0000:81:00.1/host3/target3:0:1/3:0:1:0

# fdisk -l

Disk /dev/cciss/c0d0: 146.7 GB, 146778685440 bytes
255 heads, 63 sectors/track, 17844 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes

Device Boot Start End Blocks Id System
/dev/cciss/c0d0p1 * 1 33 265041 83 Linux
/dev/cciss/c0d0p2 34 17844 143066857+ 8e Linux LVM

Disk /dev/sda: 107.3 GB, 107374182400 bytes
255 heads, 63 sectors/track, 13054 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes

Disk /dev/sda doesn’t contain a valid partition table

Disk /dev/sdb: 107.3 GB, 107374182400 bytes
255 heads, 63 sectors/track, 13054 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes

Disk /dev/sdb doesn’t contain a valid partition table

Disk /dev/sdc: 107.3 GB, 107374182400 bytes
255 heads, 63 sectors/track, 13054 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes

Disk /dev/sdc doesn’t contain a valid partition table

Disk /dev/sdd: 107.3 GB, 107374182400 bytes
255 heads, 63 sectors/track, 13054 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes

Disk /dev/sdd doesn’t contain a valid partition table

Most of the people get confuse about Linux memory usage. When you looked at used cloum, it says 3535 M, it doesn’t mean that system is using this memory.

# free -m
total used free shared buffers cached
Mem: 12010 3525 8484 0 424 1023
-/+ buffers/cache: 2077 9932
Swap: 3967 0 3967

Here is the correct way to check memory usage.

UsedMemory – Buffers – Cached/ TotalMemory * 100%
3525 – 424 – 1023 / 12010 * 100 = 17.3% memory used