Archive for the ‘Linux’ Category

Installing NRPE

Nagios Remote Plugin Executor (NRPE) is used to remotely execute Nagios plugins on Linux/Unix machines. This makes it easy to monitor remote machine metrics such as disk usage, CPU load, number of running processes, logged in users etc.

On machine on which disk needs to be monitored install nrpe

yum install -y https://dl.fedoraproject.org/pub/epel/epel-release-latest-7.noarch.rpm
yum install nrpe -y

Installing disk plugin

List all available plugins

yum list nagios-plugins*

Install disk plugin:

yum install nagios-plugins-disk.x86_64

Uncomment and edit next lines in /etc/nagios/nrpe.conf

server_address=local IP
allowed_hosts=127.0.0.1,::1, nagios_server_ip
# allow arguments
dont_blame_nrpe=1
command[check_disk]=/usr/lib64/nagios/plugins/check_disk -w 20% -c 10% -p /opt/vbox

In above example partition /opt/vbox is monitored, warning is raised when free space falls below 20% and critical alert is created when free space falls below 10%.

When done, start and enable nrpe service (or restart it if it’s already installed)

systemctl enable nrpe
systemctl start nrpe

Steps on nagios server

Test if plugin works from nagios server:

/usr/lib64/nagios/plugins/check_nrpe -H 1.1.1.2 -c check_disk
DISK CRITICAL - free space: /opt/vbox 2080 MiB (0.61% inode=100%);| /opt/vbox=333875MiB;282112;317376;0;352640

On nagios server add command for disk plugin

Edit /usr/local/nagios/etc/objects/commands.cfg file

Add next lines:

define command {
command_name check_partition
command_line /usr/lib64/nagios/plugins/check_nrpe -H $HOSTADDRESS$ -c check_disk
}

Add reference to this command in monitored host file located in /usr/local/nagios/etc/objects/conf.d/ folder

define service{
        use                             generic-service         ; Name of service template to use
        host_name                       vagrant.test.local
        service_description             check vagrant partition
        check_command                   check_partition
        }

Restart nagios service

systemctl restart nagios

Below script worked fine up to Ubuntu 18 because that distro introduced netplan.

Function Set-VMNetworkConfiguration {
    [CmdletBinding()]
    Param (
        [Parameter(Mandatory=$true,
                   Position=1,
                   ParameterSetName='DHCP',
                   ValueFromPipeline=$true)]
        [Parameter(Mandatory=$true,
                   Position=0,
                   ParameterSetName='Static',
                   ValueFromPipeline=$true)]
        [Microsoft.HyperV.PowerShell.VMNetworkAdapter]$NetworkAdapter,
 
        [Parameter(Mandatory=$true,
                   Position=1,
                   ParameterSetName='Static')]
        [String[]]$IPAddress=@(),
 
        [Parameter(Mandatory=$false,
                   Position=2,
                   ParameterSetName='Static')]
        [String[]]$Subnet=@(),
 
        [Parameter(Mandatory=$false,
                   Position=3,
                   ParameterSetName='Static')]
        [String[]]$DefaultGateway = @(),
 
        [Parameter(Mandatory=$false,
                   Position=4,
                   ParameterSetName='Static')]
        [String[]]$DNSServer = @(),
 
        [Parameter(Mandatory=$false,
                   Position=0,
                   ParameterSetName='DHCP')]
        [Switch]$Dhcp
    )
 
    $VM = Get-WmiObject -Namespace 'root\virtualization\v2' -Class 'Msvm_ComputerSystem' | Where-Object { $_.ElementName -eq $NetworkAdapter.VMName } 
    $VMSettings = $vm.GetRelated('Msvm_VirtualSystemSettingData') | Where-Object { $_.VirtualSystemType -eq 'Microsoft:Hyper-V:System:Realized' }    
    $VMNetAdapters = $VMSettings.GetRelated('Msvm_SyntheticEthernetPortSettingData') 
 
    $NetworkSettings = @()
    foreach ($NetAdapter in $VMNetAdapters) {
        if ($NetAdapter.Address -eq $NetworkAdapter.MacAddress) {
            $NetworkSettings = $NetworkSettings + $NetAdapter.GetRelated("Msvm_GuestNetworkAdapterConfiguration")
        }
    }
 
    $NetworkSettings[0].IPAddresses = $IPAddress
    $NetworkSettings[0].Subnets = $Subnet
    $NetworkSettings[0].DefaultGateways = $DefaultGateway
    $NetworkSettings[0].DNSServers = $DNSServer
    $NetworkSettings[0].ProtocolIFType = 4096
 
    if ($dhcp) {
        $NetworkSettings[0].DHCPEnabled = $true
    } else {
        $NetworkSettings[0].DHCPEnabled = $false
    }
 
    $Service = Get-WmiObject -Class "Msvm_VirtualSystemManagementService" -Namespace "root\virtualization\v2"
    $setIP = $Service.SetGuestNetworkAdapterConfiguration($VM, $NetworkSettings[0].GetText(1))
 
    if ($setip.ReturnValue -eq 4096) {
        $job=[WMI]$setip.job 
 
        while ($job.JobState -eq 3 -or $job.JobState -eq 4) {
            start-sleep 1
            $job=[WMI]$setip.job
        }
 
        if ($job.JobState -eq 7) {
            write-host "Success"
        }
        else {
            $job.GetError()
        }
    } elseif($setip.ReturnValue -eq 0) {
        Write-Host "Success"
    }
}

Usage:

Get-VMNetworkAdapter -VMName Ubuntu18| Set-VMNetworkConfiguration -IPAddress 192.168.1.238 -Subnet 255.255.255.0 -DNSServer 8.8.8.8 -DefaultGateway 192.168.1.1

 

Netplan is based on YAML based configuration system that makes configuration process very simple. Netplan has replaced the old configuration file /etc/network/interfaces that we previously used for configuring network interfaces in Ubuntu.

As result, i wasn’t able to set Ubuntu 18.04 VM IP address from Hyper-V host.

Install Hyper-V tools on Ubuntu VM:

sudo apt-get update 
sudo apt-get install linux-image-virtual linux-tools-virtual linux-cloud-tools-virtual
sudo reboot now

Check if Hyper-V daemon is running:

systemctl status hv-kvp-daemon
● hv-kvp-daemon.service - Hyper-V KVP Protocol Daemon
   Loaded: loaded (/lib/systemd/system/hv-kvp-daemon.service; enabled; vendor preset: enabled)
   Active: active (running) since Wed 2020-02-12 09:07:53 UTC; 37min ago
 Main PID: 1463 (hv_kvp_daemon)
    Tasks: 1 (limit: 1054)
   CGroup: /system.slice/hv-kvp-daemon.service
           └─1463 /usr/lib/linux-tools/4.15.0-76-generic/hv_kvp_daemon -n

Check if there is any error message:

tail /var/log/syslog

If you get following errors:

sh: 1: /usr/libexec/hypervkvpd/hv_get_dns_info: not found
sh: 1: /usr/libexec/hypervkvpd/hv_get_dhcp_info: not found

Copy hyper-v daemon binaries to location mentioned in errors:

find /usr/|grep hv_set_ifconfig
/usr/src/linux-headers-4.15.0-55/tools/hv/hv_set_ifconfig.sh
/usr/sbin/hv_set_ifconfig
mkdir -p /usr/libexec/hypervkvpd/
cp /usr/sbin/hv_* /usr/libexec/hypervkvpd
systemctl restart hv-kvp-daemon

Disabling netplan

/usr/sbin/hv_set_ifconfig is python script which accepts IP, address, default gateway, network mask and DNS server as input parameters from Hyper-V host and edit /etc/network/interfaces file. I tried changing python script to edit /etc/netplan/50-cloud-init.yaml but it doesn’t work from Hyper-V host, so i disabled netplan completely

Edit /etc/default/grub file and add following line:

GRUB_CMDLINE_LINUX="netcfg/do_not_use_netplan=true"

update grub configuration

update-grub

Remove netplan and config files:

apt purge netplan.io
rm -rf /usr/share/netplan/*
rm -rf /etc/netplan/*

Install ifupdown

apt install ifupdown

Configure interface:

vi /etc/network/interfaces

auto eth0
iface eth0 inet static
address 192.168.0.10
netmask 255.255.255.0
gateway 192.168.0.1
broadcst 192.168.0.255
dns-nameservers 192.168.0.2 192.168.0.3
dns-search lan

Although we specified DNS servers in /etc/network/interfaces, it will be overwritten on reboot, to set DNS server permanently, edit /etc/systemd/resolved.conf :

[Resolve]
DNS=8.8.8.8 8.8.4.4 2001:4860:4860::8888 2001:4860:4860::8844

Reboot Ubuntu VM

shutdown -r now

Now try to change IP address using Powershell code from begining of this post and you should be able to change Ubuntu 18 VM IP address

RANCID – Really Awesome New Cisco config Differ monitors a router’s (or more generally a device’s) configuration, including software and hardware (cards, serial numbers, etc) and uses CVS (Concurrent Version System), Subversion or Git to maintain history of changes.

Rancid:

  • login to each device in the router table (router.db)
  • runs various commands to get the information that will be saved,
  • cook the output; re-format, remove oscillating or incrementing data,
  • email any differences (sample) from the previous collection to a mail list
  • commit those changes to the revision control system

Installing required components

dnf update -y
dnf install -y epel-release 
dns install -y python2 vim wget git mlocate 
dnf install dnf-utils http://rpms.remirepo.net/enterprise/remi-release-8.rpm 
dnf module reset php 
dnf module enable php:remi-7.4 
dnf install php php-opcache php-gd php-curl php-mysqlnd 
systemctl enable --now php-fpm 
rpm -ivh https://dev.mysql.com/get/mysql80-community-release-el8-1.noarch.rpm 
yum install -y wget ftp telnet mariadb-server mariadb perl tcl expect gcc cvs httpd autoconf php-common php-gd php-pear php-pecl-memcache php-mysql php-xml mod_ssl tar sendmail postfix  
pip2 install mysql-connector-python MySQL-Python 

If, during installing MySQL-Python you face following error:

Building wheels for collected packages: MySQL-python
Running setup.py bdist_wheel for MySQL-python ... error
Complete output from command /usr/bin/python -u -c "import setuptools, tokenize;file='/tmp/pip-build-3oXEzH/MySQL-python/setup.py';exec(compile(getattr( tokenize, 'open', open)(file).read().replace('\r\n', '\n'), file, 'exec'))" bdist_wheel -d /tmp/tmpXv05tnpip-wheel- --python-tag cp27:
running bdist_wheel
running build
running build_py
creating build
creating build/lib.linux-x86_64-2.7
copying mysql_exceptions.py -> build/lib.linux-x86_64-2.7
creating build/lib.linux-x86_64-2.7/MySQLdb
copying MySQLdb/init.py -> build/lib.linux-x86_64-2.7/MySQLdb
copying MySQLdb/converters.py -> build/lib.linux-x86_64-2.7/MySQLdb
copying MySQLdb/connections.py -> build/lib.linux-x86_64-2.7/MySQLdb
copying MySQLdb/cursors.py -> build/lib.linux-x86_64-2.7/MySQLdb
copying MySQLdb/release.py -> build/lib.linux-x86_64-2.7/MySQLdb
copying MySQLdb/times.py -> build/lib.linux-x86_64-2.7/MySQLdb
creating build/lib.linux-x86_64-2.7/MySQLdb/constants
copying MySQLdb/constants/init.py -> build/lib.linux-x86_64-2.7/MySQLdb/constants
copying MySQLdb/constants/CR.py -> build/lib.linux-x86_64-2.7/MySQLdb/constants
copying MySQLdb/constants/FIELD_TYPE.py -> build/lib.linux-x86_64-2.7/MySQLdb/constants
copying MySQLdb/constants/ER.py -> build/lib.linux-x86_64-2.7/MySQLdb/constants
copying MySQLdb/constants/FLAG.py -> build/lib.linux-x86_64-2.7/MySQLdb/constants
copying MySQLdb/constants/REFRESH.py -> build/lib.linux-x86_64-2.7/MySQLdb/constants
copying MySQLdb/constants/CLIENT.py -> build/lib.linux-x86_64-2.7/MySQLdb/constants
running build_ext
building 'mysql' extension
creating build/temp.linux-x86_64-2.7
x86_64-linux-gnu-gcc -pthread -DNDEBUG -g -fwrapv -O2 -Wall -Wstrict-prototypes -fno-strict-aliasing -Wdate-time -D_FORTIFY_SOURCE=2 -g -fstack-protector-strong -Wformat -Werror=format-security -fPIC -Dversion_info=(1,2,5,'final',1) -D__version=1.2.5 -I/usr/include/mysql -I/usr/include/ python2.7 -c _mysql.c -o build/temp.linux-x86_64-2.7/_mysql.o
In file included from _mysql.c:44:0:
/usr/include/mysql/my_config.h:3:2: warning: #warning This file should not be included by clients, include only <mysql.h> [-Wcpp]
#warning This file should not be included by clients, include only <mysql.h>
^
In file included from _mysql.c:46:0:
/usr/include/mysql/mysql.h:440:3: warning: function declaration isn’t a prototype [-Wstrict-prototypes]
MYSQL_CLIENT_PLUGIN_HEADER
^
/usr/include/mysql/mysql.h:585:1: warning: function declaration isn’t a prototype [-Wstrict-prototypes]
my_bool STDCALL mysql_embedded();
^
_mysql.c: In function ‘_mysql_ConnectionObject_ping’:
_mysql.c:2005:41: error: ‘MYSQL {aka struct st_mysql}’ has no member named ‘reconnect’
if ( reconnect != -1 ) self->connection.reconnect = reconnect;
^
error: command 'x86_64-linux-gnu-gcc' failed with exit status 1

Failed building wheel for MySQL-python
Running setup.py clean for MySQL-python
Failed to build MySQL-python

Edit  /usr/include/mysql/mysql.h. Add a line with unsigned int reconnect; after the line unsigned int warning_count;

Installing Rancid

groupadd netadm 
useradd -g netadm -c "Networking Backups" -d /home/rancid rancid 
mkdir /home/rancid/tar 
cd /home/rancid/tar/ 
wget ftp://ftp.shrubbery.net/pub/rancid/rancid-3.9.tar.gz 
tar -xzvf rancid-3.9.tar.gz 
cd rancid-3.9 
./configure --prefix=/usr/local/rancid 
make && make install

Configuring Rancid

Copy over the ‘cloginrc.sample into path /home/rancid/.cloginrc’ file. Then set the ownership and and permissions on the rancid files and directories.

cp cloginrc.sample /home/rancid/.cloginrc 
chmod 0640 /home/rancid/.cloginrc 
chown -R rancid:netadm /home/rancid/.cloginrc 
chown -R rancid:netadm /usr/local/rancid/ 
chmod 775 /usr/local/rancid/ 

Create folders “Routers Switches Firewalls”,you will see when you first log into the ViewVC web front end.

vim /usr/local/rancid/etc/rancid.conf 
Uncomment and add your groups, i.e. 
LIST_OF_GROUPS="Routers Switches Firewalls" 

Apply changes:

su - rancid 
/usr/local/rancid/bin/rancid-cvs
exit

Anytime when /usr/local/rancid/etc/rancid.conf is edited, above command needs to be executed.

Installing ViewVC

ViewVC is a browser interface for CVS and Subversion version control repositories. It generates templatized HTML to present navigable directory, revision, and change log listings. It can display specific versions of files as well as diffs between those versions.

cd /home/rancid/tar/ 
wget http://www.viewvc.org/downloads/viewvc-1.1.27.tar.gz 
tar -xzvf viewvc-1.1.27.tar.gz 
cd viewvc-1.1.27 
./viewvc-install 

Configuring ViewVC

vim /usr/local/viewvc-1.1.27/viewvc.conf 
#Uncomment and change the values, (as shown bellow)# 
root_parents = /usr/local/rancid/var/CVS : cvs 
rcs_dir = /usr/local/bin 
use_rcsparse = 1 

Enable ViewVC to work with Apache httpd, we need to copy over some CGI, and set some permissions.

cp /usr/local/viewvc-1.1.27/bin/cgi/*.cgi /var/www/cgi-bin/ 
chmod +x /var/www/cgi-bin/*.cgi 
chown apache:apache /var/www/cgi-bin/*.cgi 

MariaDB configuration

systemctl enable mariadb 
systemctl start mariadb 
mysql_secure_installation

We need to create a user in MariaDB SQL that ViewVC will use, to do this we need to log into SQL using the root password set up in previous step (mysql_secure_installation)

mysql -u root -p 
Enter your SQL root password 
CREATE USER 'VIEWVC'@'localhost' IDENTIFIED BY 'MyP4ssW0rd'; 
GRANT ALL PRIVILEGES ON *.* TO 'VIEWVC'@'localhost' WITH GRANT OPTION; 
FLUSH PRIVILEGES; 
quit 

Now we can setup database and get ViewVC to create its database.

cd /usr/local/viewvc-1.1.27/bin/ 
./make-database 

Create another user in MariaDB, that will be a ‘read-only’ user.

mysql -u root -p 
CREATE USER 'VIEWVCRO'@'localhost' IDENTIFIED BY 'MyP4ssW0rd'; 
GRANT SELECT ON ViewVC.* TO 'VIEWVCRO'@'localhost' WITH GRANT OPTION; 
FLUSH PRIVILEGES; 
quit 

Edit the ViewVC configuration so that it uses all the parameters you have setup.

vim /usr/local/viewvc-1.1.27/viewvc.conf

enabled = 1 
host = localhost 
port = 3306 
database_name = ViewVC 
user = VIEWVC 
passwd = MyP4ssW0rd 
readonly_user = VIEWVCRO 
readonly_passwd = MyP4ssW0rd

Rebuild the database

/usr/local/viewvc-1.1.27/bin/cvsdbadmin rebuild /usr/local/rancid/var/CVS/CVSROOT/

Apache configuration

Following configuration will:

  • enable SSL/https connection by self-signed certificate
  • will redirect http to https
  • will configure Active directory authentication
  • users must be members of specific AD group
dnf install mod_ldap
# create self-signed certificate
cd /etc/ssl/certs
sudo openssl req -x509 -nodes -days 3650 -newkey rsa:2048 -keyout rancid.key -out rancid.crt

Configure SSL Virtual host

vi /etc/httpd/conf.d/ssl.conf

#LDAPTrustedMode TLS
#LDAPTrustedMode SSL
#LDAPTrustedGlobalCert CERT_BASE64 /etc/pki/tls/certs/ca.cer
    
SSLCertificateFile /etc/ssl/certs/rancid.crt 
SSLCertificateKeyFile /etc/ssl/certs/rancid.key 
<VirtualHost _default_:443> 
DocumentRoot "/var/www" 
ScriptAlias /cgi-bin/ "/var/www/cgi-bin" 
ScriptAlias /viewvc /var/www/cgi-bin/viewvc.cgi 
ScriptAlias /query /var/www/cgi-bin/query.cgi 
ServerName rancid.test.com:443 

<Directory "/var/www/cgi-bin"> 
    AllowOverride None 
    Options None 
    #Order allow,deny 
    #Allow from all 
    AuthType Basic 
    AuthName "login to continue" 
    AuthBasicProvider ldap 
    AuthLDAPBindAuthoritative off 
    AuthLDAPURL "ldap://test.com/dc=test,dc=com?sAMAccountName 
    AuthLDAPBindDN "test@test.com" 
    AuthLDAPBindPassword "somepassword" 
    #require valid-user
    AuthLDAPSubGroupAttribute member
    AuthLDAPSubGroupClass group
    Require ldap-group CN=Rancid,OU=Security,OU=Groups,OU=example,DC=test,DC=com  
</Directory> 

Create virtual host and configure http to https redirection

vi /etc/httpd/conf.d/viewvc.conf

<VirtualHost *:80> 
        ServerAlias * 
        DocumentRoot /var/www 
        ScriptAlias /cgi-bin/ "/var/www/cgi-bin" 
        ScriptAlias /viewvc /var/www/cgi-bin/viewvc.cgi 
        ScriptAlias /query /var/www/cgi-bin/query.cgi 
        RewriteEngine on 
        RewriteRule "^/?(.*)"  "https://%{HTTP_HOST}/$1" [R=301] 
</VirtualHost> 

systemctl enable httpd && systemctl start httpd

Type http://ip/viewvc, you should be automatically forwarded to https/ip/viewvc, credentials pop-up will ask for username/password and following page should be seen

Adding devices to Rancid

Rancid groups are represented by respective folders in /usr/local/rancid/var folder

In this example switch will be added

vi /usr/local/rancid/var/Switches/router.db

Fileformat: ip-or-hostname;vendor;up

In this case content would be

1.1.1.1;cisco,up

Adding credentials

vi /home/rancid/.cloginrc

add user {ip-or-hostname} {username}
add password {ip-or-hostname} {password}
add method {ip-or-hostname} {ssh or telnet}
add autoenable {ip-or-hostname} 1

Applying changes

su rancid
/usr/local/rancid/bin/rancid-run

Above command won’t give any output, in case of errors examine logs

cd /usrcd /usr/local/rancid/var/logs/
ls
cat {log-name}

# or interactively debugging

/usr/local/rancid/bin/clogin -c "sh run" IP/or host name

Configuring notifications

which sendmail
/sbin

Edit /usr/local/rancid/bin/control_rancid

SENDMAIL=${SENDMAIL:=/sbin/sendmail};

Rancid uses groups in following format:

rancid-"group_name" 
rancid-admin-"group_name"

The first group will receive a report after a configuration change, the second one when there is error messages.Vi need to create those aliases so postfix can use it for sending emails.

vi /etc/aliases

rancid-Switches:        email@test.com
rancid-admin-Switches:  email@test.com 

Apply aliases, start sendmail and postfix:

newaliases
systemctl enable postfix && systemctl start postfix
systemctl enable sendmail && systemctl start sendmail

Recently i got long list of linux machines and had to check which of them support password authentication.

I found a tool Hydra , if it finds machine which do not support password authentication, it will print it in output

Hydra v8.2-dev (c) 2016 by van Hauser/THC - Please do not use in military
or secret service organizations, or for illegal purposes. Hydra
(http://www.thc.org/thc-hydra) starting at 2019-11-25 14:49:59 [DATA]
max 4 tasks per 4 servers, overall 64 tasks, 5 login tries (l:1/p:5),
~0 tries

per task [DATA] attacking service ssh on port 22
[ERROR] target ssh://1.1.1.1:22/ does not support password authentication.
[ERROR] target ssh://2.2.2.2:22/ does not support password authentication.

ERROR] target ssh://3.3.3.3:22/ does not support password authentication.

[ERROR] target ssh://4.4.4.4:22/ does not support password authentication.

4 of 4 targets completed, 0 valid passwords found Hydra 

(http://www.thc.org/thc-hydra) finished at 2019-11-25 14:50:01

So i created simple batch script which captures Hydra output into $command variable, then get string between [ERROR] target ssh:// and :22/ does not support into $out variable.

Then get IP address of masines – $filtered variable. Then print every IP into new line and write it to output.txt file.

Installing hydra (CentOS 7)

rpm -Uvh http://www6.atomicorp.com/channels/atomic/centos/7/x86_64/RPMS/atomic-release-1.0-21.art.noarch.rpm
yum install hydra

Put all your passwords to file pws.txt and machines IP into targets.txt

file: put every password/IP into new line

command=$((hydra -l root -P pws.txt -M targets.txt ssh -t 4) 2>&1)
echo $command
out=$(echo $command | grep -oP '(?<=ERROR] target ssh://).*(?=:22/ does not support)')
filtered=$(echo "$out" | sed 's|does not support password authentication.||g ; s|/||g ; s|ERROR||g ; s|target ssh||g ; s|:22||g ; s/[][]//g ; s|/||g ; s|:||g')
echo $filtered | xargs -n1 > output.txt

output.txt will contain IPs of machine which don’t support password authentication.

Keepalived is used for HA. Keepalived is a service that can monitor servers or processes in order to implement high availability on your infrastructure.

In this example Active-Passive HA is implemented. Nagios secondary will monitor connectivity to primary, when disruption is detected, Nagios secondary will start nagios and postfix service and will serve requests until nagios master is available, when connection to nagios master is restored, nagios secondary will stop nagios and postfix service. Both servers are reachable via keepalived virtual IP:192.168.0.30

Install rsync on nagios secondary:

dnf install rsync
systemctl start rsyncd && systemctl enable rsyncd

On both servers install keepalived

dnf install keepalived
systemctl start keepalived && systemctl enable keepalived

retention.dat file holds information about downtime, acknowledgement and comments. This file is read by CGI and shown in dashboard. This file (along with cfg files) will be regularly copied from master to slave nagios

On slave edit /usr/local/nagios/etc/nagios.cfg and set retention_update_interval=1 .It determines how often (in minutes) Nagios will automatically save retention data during normal operation (default is 60 minutes).

In /etc/keepalived create file exclude-list.txt to specify folder/files which don’t need to be synchronized with nagios slave

/etc/keepalived/exclude-list.txt

bin/
etc/cgi.cfg
etc/htpasswd.users
include/
libexec/
sbin/
share/
var/nagios.log
var/objects.cache
var/status.dat
var/archives/
var/rw/
var/spool/
var/spool/checkresults/

Keepalived config on Nagios master

set role to BACKUP, priority to 9, set virtual IP to 192.168.0.30 /etc/keepalived/keeplaived.conf:

! Configuration File for keepalived

global_defs {

    enable_script_security 1
    script_user root
   }
vrrp_instance VI_1 {
    debug 4
    interface eth0
    state BACKUP
    virtual_router_id 51
    advert_int 1
    priority 9
    virtual_ipaddress {
            192.168.0.30 dev eth0    # the virtual IP
       }
    unicast_src_ip 192.168.0.26 # Local IP
    unicast_peer {
      192.168.0.27 # Peer IP
    }
    authentication {
        auth_type PASS
        auth_pass XXXX
    }
 
}

Keepalived config on nagios_secondary

Set role to MASTER, priority to 10, detect failure (fall) and OK (rise) state on 2 attempts, define check script – track_script, (it will be bash script which will copy files from Nagios_master and report state: 0 if all is good, 1 if there is failure), reduce priority by 2 on check script failure (weight), when nagios_secondary becomes MASTER start nagios and postfix services (notify_master /etc/keepalived/stop_nagios.sh), and when becomes BACKUP (notify_backup /etc/keepalived/start_nagios.sh), start nagios and postfix service.

/etc/keepalived/keeplaived.conf:

! Configuration File for keepalived

global_defs {

   enable_script_security 1
   script_user root
   }


vrrp_script chk_service_health {
    script /etc/keepalived/check.sh
    interval 15
    fall 2
    rise 2
    weight -2
}

vrrp_instance VI_1 {
    debug 4

    interface eth0

    state MASTER

    virtual_router_id 51
    advert_int 1
    priority 10

    virtual_ipaddress {
            192.168.0.30 dev eth0    # the virtual IP
        }

    unicast_src_ip 192.168.0.27 # Local IP

    unicast_peer {
        192.168.0.21 # Peer IP
    }

    authentication {
        auth_type PASS
        auth_pass XXXX
    }

    track_script {
        chk_service_health
    }
    notify_master /etc/keepalived/stop_nagios.sh
    notify_backup /etc/keepalived/start_nagios.sh

}

/etc/keepalived/check.sh:

#!/bin/bash

 rsync -armzv --timeout=5 --delete 192.168.0.26:/usr/local/nagios /usr/local/nagios --exclude-from /etc/keepalived/exclude-list.txt 

 if [ "$?" -eq "0" ]
then
   exit 0 # All good. Nagios master reachable
else
  exit 1 # Failover trigger
fi

/etc/keepalived/stop_nagios.sh:

#!/bin/bash

logfile=/var/log/stop_nagios.txt
exec >> $logfile
exec 2>&1

# Define an array of processes to be checked.
# If properly quoted, these may contain spaces

check_process=("nagios" "postfix" )

for p in "${check_process[@]}"; do

   if (systemctl -q is-active $p)
    then
      echo "$p is running, stopping it"
      date
      systemctl stop $p
   fi
done
exit 0

/etc/keepalived/start_nagios.sh:

#!/bin/bash

logfile=/var/log/start_nagios.txt
exec >> $logfile
exec 2>&1
# Define an array of processes to be checked.
# If properly quoted, these may contain spaces

check_process=("nagios" "postfix" )

for p in "${check_process[@]}"; do

   if (systemctl -q is-active $p)
    then
      echo "$p is running"

   else
      date
      echo "Staring $p..."
      systemctl start $p
   fi
done
exit 0

Install Nagios Core on CentOS 8

Posted: November 4, 2019 in Linux

I disabled SELinux

sed -i 's/^SELINUX=.*/SELINUX=disabled/g' /etc/selinux/config
setenforce 0

 

Install python and other prerequisities

 

dnf install -y compat-openssl10 python3 perl gcc glibc glibc-common wget unzip httpd php gd gd-devel perl postfix 
alternatives --set python /usr/bin/python3 

Add nagios user and group

useradd nagios
groupadd nagcmd 

Add both the nagios user and the apache user to the nagcmd group 

usermod -G nagcmd nagios
usermod -G nagcmd apache

Download nagios setup

mkdir setup 
cd setup 
wget https://assets.nagios.com/downloads/nagioscore/releases/nagios-4.4.5.tar.gz 
tar xvf nagios-4.4.5.tar.gz

Install nagios

cd nagios-4.4.5 
./configure --with-command-group=nagcmd 
make all 
make install 
make install-init 
make install-commandmode 
make install-config 
make install-webconf 
# set nagiosadmin password 
htpasswd -s -c /usr/local/nagios/etc/htpasswd.users nagiosadmin 

Setup EventHandlers

cp -R contrib/eventhandlers/ /usr/local/nagios/libexec/ 
chown -R nagios:nagios /usr/local/nagios/libexec/eventhandlers

 

 

Download and install nagios plugins

yum install -y gcc glibc glibc-common make gettext automake autoconf wget openssl-devel net-snmp net-snmp-utils
cd /tmp
wget --no-check-certificate -O nagios-plugins.tar.gz https://github.com/nagios-plugins/nagios-plugins/archive/release-2.2.1.tar.gz
tar zxf nagios-plugins.tar.gz
cd /tmp/nagios-plugins-release-2.2.1/
./tools/setup
./configure --with-openssl --with-nagios-user=nagios --with-nagios-group=nagios
make
make install

 Apache configuration:

Create file (if not exists): /etc/httpd/conf.d/nagios.conf

ScriptAlias /nagios/cgi-bin "/usr/local/nagios/sbin"

<Directory "/usr/local/nagios/sbin">
  
   Options ExecCGI
   AllowOverride None
   <IfVersion >= 2.3>
      <RequireAll>
         Require all granted
#        Require host 127.0.0.1

         AuthName "Nagios Access"
         AuthType Basic
         AuthUserFile /usr/local/nagios/etc/htpasswd.users
         Require valid-user
      </RequireAll>
   </IfVersion>
   <IfVersion < 2.3>
      Order allow,deny
      Allow from all
      Order deny,allow
#     Deny from all
#     Allow from 127.0.0.1

      AuthName "Nagios Access"
      AuthType Basic
      AuthUserFile /usr/local/nagios/etc/htpasswd.users
      Require valid-user
   </IfVersion>
</Directory>

Alias /nagios "/usr/local/nagios/share"

<Directory "/usr/local/nagios/share">
  SSLRequireSSL
   Options None
   AllowOverride None
   <IfVersion >= 2.3>
      <RequireAll>
         Require all granted
#        Require host 127.0.0.1

         AuthName "Nagios Access"
         AuthType Basic
         AuthUserFile /usr/local/nagios/etc/htpasswd.users
         Require valid-user
      </RequireAll>
   </IfVersion>
   <IfVersion < 2.3>
      Order allow,deny
      Allow from all
     Order deny,allow
#     Deny from all
#     Allow from 127.0.0.1

      AuthName "Nagios Access"
      AuthType Basic
      AuthUserFile /usr/local/nagios/etc/htpasswd.users
      Require valid-user
   </IfVersion>
</Directory>

Install NRPE

NRPE allows you to remotely execute Nagios plugins on other Linux/Unix machines. This allows you to monitor remote machine metrics (disk usage, CPU load, etc.). NRPE can also communicate with some of the Windows agent addons, so you can execute scripts and check metrics on remote Windows machines as well.

# install nrpe 
dnf install openssl-devel 
wget https://github.com/NagiosEnterprises/nrpe/releases/download/nrpe-3.2.1/nrpe-3.2.1.tar.gz 
tar -xvf nrpe-3.2.1.tar.gz 
cd nrpe-3.2.1 
./configure --disable-ssl --with-nrpe-user=nagios --with-nrpe-group=nagios --with-nagios-user=nagios --with-nagios-group=nagios --libexecdir=/usr/local/nagios/libexec/ --bindir=/usr/local/nagios/bin/ --prefix=/usr/local/nagios 
make 
make install 
cp src/check_nrpe /usr/local/nagios/libexec/

Install NCPA 

NCPA is a cross-platform monitoring agent that runs on Windows, Linux/Unix, and Mac OS/X machines. Its features include both active and passive checks, remote management, and a local monitoring interface.

 

wget https://assets.nagios.com/downloads/ncpa/check_ncpa.tar.gz 
tar -zxf check_ncpa.tar.gz 
mv check_ncpa.py /usr/local/nagios/libexec/ 
chown nagios:nagios /usr/local/nagios/libexec/check_ncpa.py 
chmod 775 /usr/local/nagios/libexec/check_ncpa.py 

Next add your command (ncpa into commands.cfg) 

vim /usr/local/nagios/etc/objects/commands.cfg 

define command { 

    command_name    check_ncpa 

    command_line    $USER1$/check_ncpa.py -H $HOSTADDRESS$ $ARG1$ 

} 

Adding contact

edit contacts.cfg and change email address

define contact{
        contact_name            nagiosadmin             ; Short name of user
        use                     generic-contact         ; Inherit default values from generic-contact template (defined above)
        alias                   Nagios Admin            ; Full name of user
        email                   dragan.vucanovic@hotmail.com       ; <<***** CHANGE THIS TO YOUR EMAIL ADDRESS ******
        }

Enable and start nagios and httpd

systemctl start nagios
systemctl enable nagios
systemctl enable httpd
systemctl start httpd

Active directory authentication and LDAP over SSL

Install Root CA certificates

dnf install mod_ldap

This enables SSL, AD authentication and redirect http to https

Edit /etc/httpd/conf.d/nagios.conf

RewriteEngine On
RewriteCond %{HTTPS} off
RewriteRule (.*) https://%{HTTP_HOST}%{REQUEST_URI}

LogLevel warn

ScriptAlias /nagios/cgi-bin "/usr/local/nagios/sbin"

<Directory "/usr/local/nagios/sbin">
  SSLRequireSSL
   Options ExecCGI
   AllowOverride None
    Order allow,deny
    Allow from all
    AuthBasicProvider ldap file
    AuthType Basic
    AuthLDAPBindAuthoritative on
    AuthLDAPGroupAttributeIsDN on
    AuthName "Active Directory Login"
    AuthLDAPURL "ldaps://test.com:636/dc=test,dc=com?sAMAccountName
    AuthLDAPBindDN "bindtest@test.com"
    AuthLDAPBindPassword pass
    AuthUserFile /usr/local/nagios/etc/htpasswd.users 
    Require valid-user

    
</Directory>

Alias /nagios "/usr/local/nagios/share"

<Directory "/usr/local/nagios/share">
  SSLRequireSSL
   Options ExecCGI
   AllowOverride None
   AuthBasicProvider ldap file
   AuthType Basic
   AuthLDAPBindAuthoritative on
   AuthName "Active Directory Login 1"
   AuthLDAPURL "ldaps://test.com:636/dc=test,dc=com?sAMAccountName
   AuthLDAPBindDN "bindtest@test.com"
   AuthLDAPBindPassword pass
   AuthUserFile /usr/local/nagios/etc/htpasswd.users
   #Require valid-user
   Require ldap-group CN=mygroup ,OU=Security,OU=Groups,OU=test,DC=test,DC=com
    
</Directory>

/etc/httpd/conf.d/ssl.conf

SSLCertificateFile /etc/ssl/certs/nagios.cer
SSLCertificateKeyFile /etc/ssl/certs/nagios.key

To avoid

It appears as though you do not have permission to view information for any of the hosts you requested…

If you believe this is an error, check the HTTP server authentication requirements for accessing this CGI
and check the authorization options in your CGI configuration file.
Replace all of the “nagiosadmin” entries with “*” in /usr/local/nagios/etc/cgi.cfg

sed -i 's/nagiosadmin/*/' /usr/local/nagios/etc/cgi.cfg
systemctl restart nagios
systemctl restart httpd 

Using Start TLS

STARTTLS is an alternative approach that is now the preferred method of encrypting an LDAP connection. STARTTLS “upgrades” a non-encrypted connection by wrapping it with SSL/TLS after/during the connection process. This allows unencrypted and encrypted connections to be handled by the same port. This guide will utilize STARTTLS to encrypt connections.

Now our /etc/httpd/conf.d/nagios.conf looks like this:

RewriteEngine On
RewriteCond %{HTTPS} off
RewriteRule (.*) https://%{HTTP_HOST}%{REQUEST_URI}

LogLevel debug 

LDAPTrustedMode TLS

ScriptAlias /nagios/cgi-bin "/usr/local/nagios/sbin"
<Directory "/usr/local/nagios/sbin">
  SSLRequireSSL
   Options ExecCGI
   AllowOverride None
    Order allow,deny
    Allow from all
    AuthBasicProvider ldap file
    AuthType Basic
    AuthName "login to continue"
    AuthBasicProvider ldap
    AuthLDAPBindAuthoritative on
    AuthLDAPURL "ldap://test.com/dc=devtech,dc=local?sAMAccountName
    AuthLDAPBindDN "test@dtest.com"
    AuthLDAPBindPassword "pass"
    #require valid-user
    AuthLDAPSubGroupAttribute member
    #AuthLDAPGroupAttributeIsDN on
    AuthLDAPSubGroupClass group
    AuthUserFile /usr/local/nagios/etc/htpasswd.users
   Require ldap-group CN=Ansible AWX,OU=Security,OU=Groups,OU=test,DC=test,DC=com   

</Directory>
Alias /nagios "/usr/local/nagios/share"
<Directory "/usr/local/nagios/share">
  SSLRequireSSL
   Options None
   AllowOverride None
   AuthBasicProvider ldap file
   AuthType Basic
   AuthName "login to continue"
   AuthBasicProvider ldap
   AuthLDAPBindAuthoritative on
   AuthLDAPURL "ldap://test.com/dc=test,dc=com?sAMAccountName
   AuthLDAPBindDN "test@test.com"
   AuthLDAPBindPassword "pass"
   #require valid-user
   AuthLDAPSubGroupAttribute member
   #AuthLDAPGroupAttributeIsDN on
   AuthLDAPSubGroupClass group
   AuthUserFile /usr/local/nagios/etc/htpasswd.users
   Require ldap-group CN=mygroup,OU=Security,OU=Groups,OU=test,DC=test,DC=com 
  </Directory>

Recently i create new CentOS 7 Hyper-V VM, i set disk type to Dynamic with 127 GB, during installation, set automatic partition, but soon i realized i only have 50 GB of root partition, copied bunch of files to /opt directory and i left out of disk space.

First, on Hyper-V console, turn off VM and expand disk space

Turn VM and partition unallocated disk space. Check the name(s) of your scsi devices

ls /sys/class/scsi_device/
0:0:0:0  2:0:0:0

Then rescan the scsi bus. Replace the ‘0\:0\:0\:0’ with the actual SCSI bus name found with the previous command.

echo 1 > /sys/class/scsi_device/0\:0\:0\:0/device/rescan
echo 1 > /sys/class/scsi_device/2\:0\:0\:0/device/rescan

Create new partition

fdisk /dev/sda

Enter n, to create a new partition, then choose p to create a new primary partition, Choose partition number. I have /dev/sda1 and /dev/sda2, so the number would be 3,confirm  first and last cylinder,  type t to change the partition type. When prompted, enter the number of the partition you’ve just created in the previous steps. When you’re asked to enter the “Hex code”, enter 8e,  type w to write your partitions to the disk.

The device presents a logical sector size that is smaller than
the physical sector size. Aligning to a physical sector (or optimal
I/O) size boundary is recommended, or performance may be impacted.
Welcome to fdisk (util-linux 2.23.2).

Changes will remain in memory only, until you decide to write them.
Be careful before using the write command.


Command (m for help): n
Partition type:
   p   primary (2 primary, 0 extended, 2 free)
   e   extended
Select (default p): p
Partition number (3,4, default 3): 3
First sector (266338304-838860799, default 266338304):
Using default value 266338304
Last sector, +sectors or +size{K,M,G} (266338304-838860799, default 838860799):
Using default value 838860799
Partition 3 of type Linux and of size 273 GiB is set

Command (m for help): t
Partition number (1-3, default 3): 3
Hex code (type L to list all codes): 8e
Changed type of partition 'Linux' to 'Linux LVM'

Command (m for help): w
The partition table has been altered!

Calling ioctl() to re-read partition table.

WARNING: Re-reading the partition table failed with error 16: Device or resource busy.
The kernel still uses the old table. The new table will be used at
the next reboot or after you run partprobe(8) or kpartx(8)
Syncing disks.

Scan new partition

partprobe -s

And confirm partition is created

fdisk -l

Create the physical volume,replace /dev/sda3 with the newly created partition.

 pvcreate /dev/sda3

Extend logical volume with new partition, first find out volume group:

vgdisplay

Extend volume group by adding new partition

vgextend centos /dev/sda3

If getting “Device /dev/sda3 not found.” reboot VM

See newly added physical volume:

pvscan

Extend logical volume

lvextend /dev/mapper/centos-root /dev/sda3
Size of logical volume centos/root changed from 50.00 GiB (12800 extents) to <323.00 GiB (82687 extents).
Logical volume centos/root successfully resized.

Resize the file system to the volume group

xfs_growfs /dev/mapper/centos-root

xfs_growfs /dev/mapper/centos-root
meta-data=/dev/mapper/centos-root isize=512    agcount=4, agsize=3276800 blks
         =                       sectsz=4096  attr=2, projid32bit=1
         =                       crc=1        finobt=0 spinodes=0
data     =                       bsize=4096   blocks=13107200, imaxpct=25
         =                       sunit=0      swidth=0 blks
naming   =version 2              bsize=4096   ascii-ci=0 ftype=1
log      =internal               bsize=4096   blocks=6400, version=2
         =                       sectsz=4096  sunit=1 blks, lazy-count=1
realtime =none                   extsz=4096   blocks=0, rtextents=0
data blocks changed from 13107200 to 84671488

Root partition is resized