Category Archives: Linux

VMware ESXi Host Memory Management, Monitoring, Alert Notification – Part 2

Posted on by
Reply

I have described memory monitoring and alert notification gauge on the previous article (Part 1) – let’s do the configuration.

There are so many ways to monitor and get alert notification of VMWare ESXi host memory usage status – most of well-known monitoring solutions come with VMware monitoring plugins pre-installed. vCenter server can also send alerts based on given conditions as well.

Here I will discuss how to configure Nagios Core to monitor memory usage and alert notification; NagiosXI (the commercial edition has a built-in nice easy web UI to do the same). Before moving forward, make sure Nagios server up and running – we need install the following software/tools on the Nagios server –

i. VMware vSphere Perl-SDK; the version should match to the vCenter/ESXi host version – version 5.5 can be download at https://developercenter.vmware.com/web/sdk/55/vsphere-perl
ii. Download and install check_vmware_esx.pl (this is a fork of check_vmware_api.pl) from https://www.monitoringexchange.org/inventory/Check-Plugins/Virtualization/VMWare-%2528ESX%2529/check_vmware_esx.pl—a-fork-of-check_vmware_api.pl-%2528check_esx3-pl%2529 or from https://github.com/BaldMansMojo/check_vmware_esx/blob/master/check_vmware_esx.pl
iii. Install the required Perl modules.

(Step 1 – install VMware vSphere Perl-SDK)

#tar zxvf  VMware-vSphere-Perl-SDK-5.5.0-1384587.x86_64.tar.gz
#cd vmware-vsphere-cli-distrib
#./vmware-install.pl

Accept the license agreement and install with default settings.

If the installation detect missing or old Perl modules – install them; easiest way is install them via CPAN.

(Step 2 – install & configure check_vmware_esx.pl Nagios check script)

Download this from the above mentioned web sites. Copy the “chech_vmware_esx.pl” script to Nagios libexec directory “/usr/local/nagios/libexec/”; make sure it is owned by “nagios” user/group with executable permission.

If you download the “check_vmware_esx_0.9.19.tgz” file – the installation process is following –

#tar zxvf check_vmware_esx_0.9.19.tgz
#cd check_vmware_esx_0.9.19
#cp check_vmware_esx.pl /usr/local/nagios/libexec
#chown nagios.nagios check_vmware_esx.pl
#chmod 751 check_vmware_esx.pl

Copy the perl modules within “check_vmware_esx_0.9.19/modules” to a directory – this can be inside “/usr/local/nagios/libexec” directory –

#mkdir /usr/local/nagios/libexec/vmware_modules
#cp –R /tmp/check_vmware_esx_0.9.19/modules /usr/local/nagios/libexec/vmware_modules/ 
#chown –R nagios.nagios /usr/local/nagios/libexec/vmware_modules

Also change following parameter in the check_vmware_esx.pl file –

use lib “modules”;
to
use lib /usr/local/nagios/libexec/vmware_modules/modules;

Again if the script execution complain about missing Perl module – install them via CPAN.

You should use a “session lock file” to minimize auth log entries on vCenter or ESXi host; every time nagios execute service check with this script – this will create auth log entries in vCenter/ESXi host – it’s huge! The default script will ask you to create session lock file in “/var/nagios_plugin_cache/” directory – create this directory and make sure it is owned by Nagios.

#mkdir /var/nagios_plugin_cache
#chown –R nagios.nagios /var/nagios_plugin_cache

You need to create an user account for this nagios script on your vCenter or on ESXi hosts you want to monitor. You should use “authfile”; this file contains Nagios monitoring user account/password created on vCenter or ESXi host.

#vi /usr/local/nagios/libexec/vmware_plugin/authfile

Enter the following –

username=nagios_userName_on_esxi
password=password_nagios

#chown nagios.nagios /usr/local/nagios/libexec/vmware_plugin/authfile

At this stage the script should be ready to execute! If not – it must be missing Perl modules :(.

(Step 3 – configure Nagios commands and service check)

This script is capable of monitoring lots of other vCenter objects such as cpu, network, datastore, virtual machines etc. Follow standard Nagios guidelines to create your check commands and service checks.

Usage:

To see all memory parameters of an esxi host–
./check_vmware_esx.pl -H 192.168.1.1 -f /location/of/authfile -S mem

mem usage=42.73% - consumed memory=24501.48 MB - swap used=35.87 MB - overhead=650.41 MB - memctl=0.00 MB: |'mem_usage'=42.73%;;;; 'consumed_memory'=24501.48MB;;;; 'mem_swap'=35.87MB;;;; 'mem_overhead'=650.41MB;;;; 'mem_memctl'=0.00MB;;;;

Set alert notification based on % of memory usage of an esxi host-
./check_vmware_esx.pl -H 192.168.1.1 -f /location/of/authfile -S mem -s usage

mem usage=42.73%|'mem_usage'=42.73%;;;;

./check_vmware_esx.pl -H 192.168.1.1 -f /location/of/authfile -S mem -s usage -w 40% -c 60%

Warning! mem usage=42.69%|'mem_usage'=42.69%;40;60;;

Set alert notification based on MB of total memory usage of an esxi host–
./check_vmware_esx.pl -H 192.168.1.1 -f /location/of/authfile -S mem -s consumed

consumed memory=24501.29 MB|'consumed_memory'=24501.29MB;;;;

./check_vmware_esx.pl -H 192.168.1.1 -f /location/of/authfile -S mem -s consumed -w 24000 -c 26000

Warning! consumed memory=24475.05 MB|'consumed_memory'=24475.05MB;24000;28000;;

To see swap memory usage only of an esxi host–
./check_vmware_esx.pl -H 192.168.1.1 -f /location/of/authfile -S mem -s swapused

swap used=35.87 MB|'mem_swap'=35.87MB;;;;

Screenshot of mem usage on Nagios web UI –

nagios-esxi-memcheck

This script also generate Nagios perfdata which is useful for graphing; if you have pnp4nagios graph installed you should be able to get graph like the following –

nagios-mem-graph

Linux LVM Cookbook with Examples

Posted on by
Reply

I often manage LVM volumes on Linux servers. LVM is the best way to manage on growing disk storage demands on Linux servers running on virtual infrastructure platform; the beauty is – the whole disk administration process is ONLINE without server reboot or any outage to running services. Here is below my LVM cookbook with examples –

Before working with LVM – make sure you are experienced with disk administration utilities such as “fdisk” and Linux file systems.

Before moving forward – let’s get familiar with LVM terminologies –

LVM Terminology Descriptions
Physical Volume (PV) A PV is the actual physical disk or a physical disk partition to be used in LVM. The disk partition ID for LVM PV is “8e”; whereas standard Linux partition ID is “83”. Example :/dev/sdc – the whole disk cab be a LVM PV/dev/sda6 – the logical partition sda6 can be a LVM PV
Volume Group (VG) A VG is a pool of Physical Volumes (PV); a VG can have a single PV only or bunch of PVs bundle together. Member PV disk size can be same or different in a VG. VGs are identified by unique names within a Linux server. Example:VolGrp00 – actual location is /dev/VolGrp00VolGrp01 – actual location is /dev/VolGrp01
Logical Volume (LV) LVs are the logical/virtual partitions in Linux systems. Linux file system mount points are assigned against LVs. A single LV or multiple LVs can be created on a single Volume Group (VG). A LV name is unique within a VG. Example:/dev/VolGrp00/LogicalVol00 can be mount as /usr/dev/VolGrp00/LogicalVol01 can be mount as /var/dev/VolGrp01/LogicalVol00 can be mount as /mydb
Physical Extents (PE) PEs are physical disk blocks in LVM. The size of PE is often in megabytes. Larger PE size allow large Volume Group (VG); i.e. PE size 32M support a VG up to 2TB, 64MB PE size support a VG up to 4TB, 256MB PE size support a VG up to 16TB. Once a VG is created, the PE size cannot be changed; that’s why it is important to decide PE size before creating LVM.
Logical Extents (LE) LEs are logical disk blocks in LVM. LEs have the same size as PEs.

1. Let’s start with the “creation” of LVM-

The LVM creation process involves five tasks –

  • create PV on physical disk or on a free disk partition
  • then create VG on PV(s); specify PE size if required
  • create LV(s) on VG(s)
  • create Linux fie system (ext3 ot ext4 or other) on LV
  • mount the LV disk to a Linux directory/mount point.

Following table describe the creation tasks Linux commands –

LVM Terminology Linux Commands Description
Physical Volume #pvcreate /dev/sdbor#pvcreate /dev/sdc3 The first command will create PV on the whole disk /dev/sdb.The second command will create PV on the disk partition /dev/sdc3. Other partitions on the same disk /dev/sdc might not be LVM PV – and it’s ok.
Logical Volume #vgcreate MyVolGrp00 /dev/sdbor#vgcreate MyVolGrp00 /dev/sdb –s 32Mor#vgcreate VolGrp01 /dev/sdb /dev/sdc3 /dev/sda7 –s 64M The first command will create VG named MyVolGrp00 on PV /dev/sda. The PE size is based on Linux system default – if you don’t specify.The second command will create VG named MyVolGrp00 on PV /dev/sdb with PE size of 32MB. This VG can grow upto 2TB.The third command will create VG named VolGrp01 on multiple PVs with PE size of 64MB. This VG can grow upto 4TB.
Logical Volume #lvcreate -l +100%Free MyVolGrp01 –n LVol00or#lvcreate –L 10G VolGrp00 –n LVol01 The first command will create LV LVol00 on MyVolGrp01 VG – and will consume the whole size of the VG.The second command will create LV LVol01 on VolGrp00 – the LV size is 10GB; if any free space available you can create more LVs on the same VG VolGrp00.
Fie System #mkfs.ext3 /dev/VolGrp00/LVol01#mkfs.ext3 /dev/VolGrp02/LVol00 This will create Ext3 file system on LVol01 & LVol00.
Mount Point #mount /dev/VolGrp00/LVol00 /mymountpoint For auto mount – add mount commands onto > /etc/fstab.

2. LVM – Display Details

Following are Linux commands to display detailed information about LVM,

LVM Term Linux Commands Description
PV #pvdisplay#pvdisplay /dev/sdc The first command will display all available PVs.The second command will display PV on disk /dev/sdc.
VG #vgdisplay#vgdisplay VolGrp01 The first command will display details of all available VGs. You can find the PE size here.The second command will display details of VolGrp01 only.
LV #lvdisplay#lvdisplay /dev/VolGrp00/LVol00 The first command will display details of all available LVs.The second command will display details of /dev/VolGrp00/LVol00 only.

3. LVM – Disk Expansion

LVM disk expansion process works as following –

-you can expand a logical volume (LV) only if there available free space on underlying volume group (VG).

-if no free space available on volume group (VG) – then (i)add new physical disk, convert it to PV then add to VG or (ii)extend existing PV physical disk; this is very common on virtual environment.

-finally tell Linux file system (ext3/ext4) to recognize and work with the new size.

The whole operation can be done ONLINE. Following table describe Linux commands for LVM disk expansion operation –

LVM Terminology Linux Commands Description
Physical Volume (PV) #pvresize /dev/sdc The first command will grow PV to all available free space on /dev/sdc physical disk – this is useful when we expand an existing hard disk on virtual machine.For “pvresize” nothing need to done on VG – VG will automatically recognize new disk size after the “pvresize” command.
#pvcreate /dev/sda6or#pvcreate /dev/sdd These commands will create a new PV to be added to existing VG; these commands are same as create a new PV for a new VG.New PV need to be added to VG as a part of VG extend.
Volume Group (VG) #vgextend VolGrp02 /dev/sdeor#vgextend VolGrp02 /dev/sde  /dev/sdf1 The first command will extend the VG VolGrp02 to new PV /dev/sde.The second command will extend the VG VolGrp02 to new multiple PVs /dev/sde & /dev/sdf1.
Logical Volume (LV) #lvextend –l +100%Free /dev/VolGrp02/Vol00or#lvextend –size +20G /dev/VolGrp02/Vol00or#lvextend –size 800GB /dev/VolGrp02/Vol00 The first command will extend the LV to all available free space on the VG /dev/VolGrp02.The second command will add additional 20GB space to the LV /dev/VolGrp02/Vol00 – make sure you have 20GB free space on the VG.The third command will make the LV /dev/VolGrp02/Vol00 size to 800GB (new final size) – make sure you have available space on the VG to have this 800GB LV.
File System #resize2fs /dev/VolGrp02/Vol00 You need to tell the Linux file system to recognize the new size ONLINE – this command will do that for LV Vol00. No system reboot or file system re-mount required for this.

4. LVM – Delete Operations

Before delete a logical volume LV make sure it is not mounted. Also delete operation will destroy all the data – so be careful before delete.

Command examples are following –

#lvremove /dev/VolGrp03/Vol01 ; this will remove LV “/dev/VolGrp03/Vol01”

#vgremove VolGrp03 ; this will remove VG “VolGrp03”

#pvremove /dev/sdc ; this will remove PV “/dev/sdc”

5. LVM – Disk Shrink

Disk size shrink of a LV can be done without deleting & recreating it – you might experience data loss while shrinking if not planned well (I experienced data loss and restored data from backup). Make sure you have data backup and the LV is not mounted before you move ahead. The new shrinked size must not be below the volume used size. Command examples are following –

Unmount the LV first,

#umount /dev/VolGrp02/Vol00   ; this will unmount “/dev/VolGrp02/Vol00”

Run file system check and make sure the file system is OK before moving ahead,

#e2fsck –f /dev/VolGrp02/Vol00

Reduce the file system size first (without losing data),

#resize2fs /dev/VolGrp02/Vol00 30G; this will reduce the current file system size to 30GB (let’s say the LV size is 50GB – and it has less than 30GB of data) – make sure volume usage is less than <30GB otherwise you will lose data.

Reduce the Logical Volume now,

#lvreduce –L 35G /dev/VolGrp02/Vol00; this will shrink the LV to 35GB (around 30GB of data + 5GB free space)

Finally resize the file system again to 35GB,

#resize2fs /dev/VolGrp02/Vol00

In this above example – our target is reducing a LV from 50GB to 35GB. The first “resize2fs” we shrink the file system from 50GB to 30GB (where data usage is less than 30GB out of total 50GB disk size) and then resize the Logical Volume to 35GB. This estimation helps eliminate data loss.

6. LVM – “scan” Commands

There are three scan commands – #pvscan, #vgscan & #lvscan.

LVM scan commands scan all the available PV/VG/LV in the system and build the LVM cache file “/etc/lvm/.cache” to maintain a listing of current LVM devices.

A Linux system auto execute “scan” commands every time we restart/power-on server and also during LVM create/expand/resize/reduce operations.

Configure Perl CPAN to use HTTP mirror instead of FTP – also access mirrors via HTTP Proxy

Posted on by
Reply

By default perl CPAN is configured to use FTP mirrors instead of HTTP mirrors.

If your environment only allows HTTP out through http proxy service – this prevents installation of new perl CPAN modules over the internet. I have seen lot of companies they allow access to internet only through http/https proxy server.

In this configure I will show –

(i)how to configure CPAN to use HTTP mirrors instead of FTP.

(ii)how to configure CPAN to use a http proxy server to access HTTP mirrors over the Internet.

Part 1: Configure CPAN to use HTTP mirror instead FTP

i. invoke perl cpan command shell first

#perl –MCPAN –e shell

If you ran CPAN for the first time in a computer – the above command might ask you

“Are you ready for manual configuration? [yes]”

Enter no.

ii. Set your preferred CPAN HTTP mirror URL

#cpan> o conf urllist push http://www.perl.com/CPAN; you can enter your preferred CPAN mirror URL here.

iii. List and verify your newly entered http mirror

#cpan> o conf urllist       ; this should return the above http mirror URL

iv. Save the new configuration

#cpan> o conf commit

CPAN is ready to use HTTP instead of FTP.

Part 2: Configure CPAN to use a HTTP proxy server to access Internet

i. invoke perl CPAN command shell first

#perl –MCPAN –e shell

If you ran CPAN for the first time in a computer – the above command might ask you

“Are you ready for manual configuration? [yes]”

Enter no.

ii. Set your HTTP proxy server address, port and auth details

#cpan> o conf init /proxy/

Your ftp proxy?                                 ;nothing to enter here

Your http proxy?myproxy:8080                 ;enter your proxy server IP or hostname with port number

Your proxy user id?                         ; if your proxy use proxy_auth enter username

iii. Save your configuration

#cpan> o conf commit

Your CPAN should be ready to access HTTP mirrors via proxy!