Monitor Rundeck Jobs with Zabbix

Posted: April 4, 2018 in Linux, RunDeck

It’s presumed that Rundeck server is monitored by Zabbix

Disable the token expiration time for rundeck API (so we can authenticate on Rundeck server when running API calls)

Edit the /etc/rundeck/rundeck-config.properties file and add the following line and restart Rundeck service

rundeck.api.tokens.duration.max = 0
service rundeckd restart

Authenticate with an admin account and click on the “profile” button at the top right of the page:Administrator-Profile-add API token-save it

Capture.PNG

I modified this script  (to include job name in output)

Change URL,Rundeck token, project name and jobname (this query runs in project scope)

#!/bin/bash
curl -s -H "Accept: application/json" -X GET "http://172.30.61.88:4440/api/20/project/demo/executions?authtoken=ZP9znayUp3Ktp26uQjSQGnEfTzDvqPDA" | python -m json.tool > /tmp/1.json

jq -r '.|[.executions[] | select(.job.name != null) | select(.job.name|contains("test")) ] | sort_by(.id) | reverse | .[0] | [.status, .job.name, ."date-started".date, ."date-ended".date, .job.project] | @csv' /tmp/1.json > /tmp/1.csv
sed 's/,/ /g' /tmp/1.csv>/tmp/st.txt

while read -r status name startdate enddate project; do

startdate=${startdate//\"/}
stime=$(date -d "${startdate/T/ }" +%s)
enddate=${enddate//\"/}
etime=$(date -d "${enddate/T/ }" +%s)
let elapsed=etime-stime

if [ "$status" == "\"aborted\"" ] && [ "$elapsed" -gt 300 ]; then echo $project"-"$name-"Long-Run"
elif [ "$status" == "\"aborted\"" ] && [ "$elapsed" -lt 300 ]; then echo $project"-"$name " Aborted"
elif [ "$status" == "\"failed\"" ]; then echo $project"-"$name "failed-Demo"
#elif [ "$status" == "\"succeeded\"" ]; then echo $project"-"$name "success"
elif [ "$status" == "\"succeeded\"" ] && [ "$elapsed" -gt 300 ]; then echo $project"-"$name" Takes too long"
fi

done</tmp/st.txt

Second version (without storing output to JSON file):

#!/bin/bash

curl -s -H "Accept: application/json" -X GET "http://192.168.253.21:4440/api/20/project/demo/executions?authtoken=kH44NoX35bp1zxohgkMtsOIC9H9tw6UI" | jq -r '.|[.executions[] | select(.job.name != null) | select(.job.name|contains("test")) ] | sort_by(.id) | reverse | .[0] | [.status, .job.name, ."date-started".date, ."date-ended".date, .job.project] | @csv'  | sed 's/,/ /g' > /tmp/st.tx


while read -r status name startdate enddate project; do

startdate=${startdate//\"/}
stime=$(date -d "${startdate/T/ }" +%s)
enddate=${enddate//\"/}
etime=$(date -d "${enddate/T/ }" +%s)
let elapsed=etime-stime

if [ "$status" == "\"aborted\"" ] && [ "$elapsed" -gt 300 ]; then echo $project"-"$name-"Long-Run"
elif [ "$status" == "\"aborted\"" ] && [ "$elapsed" -lt 300 ]; then echo $project"-"$name " Aborted"
elif [ "$status" == "\"failed\"" ]; then echo $project"-"$name "failed-Demo"
#elif [ "$status" == "\"succeeded\"" ]; then echo $project"-"$name "success"
elif [ "$status" == "\"succeeded\"" ] && [ "$elapsed" -gt 300 ]; then echo $project"-"$name" Takes too long"
fi

done</tmp/st.txt

Zabix agent configuration (Creating key)

Rundeck server is monitored by Zabbix, we’ll now tell Zabbix to run this script:

In my Centos box config file is in /etc/zabbix/zabbix_agentd.conf Uncomment

EnableRemoteCommands=1

Uncomment

UserParameter=rundeck,/home/a.sh

In above example i created a key named rundeck and specified a path for script

give zabbix user ownership over script:

chown zabbix:zabbix /home/a.sh

check zabbix user bash version

zabbix:x:997:995:Zabbix Monitoring System:/var/lib/zabbix:/bin/sh
su - zabbix
./a.sh

Now test if key works:restart zabbix agent and run test:

systemctl stop zabbix-agent.service && systemctl start zabbix-agent.service && zabbix_agentd -t rundeck
[root@localhost home]# zabbix_agentd -t rundeck
rundeck                                       [t|Job  "aa" success]

Now perform test on Zabbix server

zabbix_get -s 172.30.61.88 -p 10050 -k rundeck
Job  "aa" success

 

If get “permission denied” try following (on Rundeck server)

systemctl stop zabbix-agent.service
chown -R zabbix:zabbix /var/log/zabbix
chown -R zabbix:zabbix /var/run/zabbix
chmod -R 775 /var/log/zabbix/
chmod -R 775 /var/run/zabbix/
/usr/sbin/zabbix_agentd -c /etc/zabbix/zabbix_agentd.conf
systemctl start zabbix-agent.service

If get “unsupported key” error, try to set in Zabbix agent file as follows)

Mandatory: no
# Default:
Include=/etc/zabbix/zabbix_agentd.d
#Include=/etc/zabbix/zabbix_agentd.d/*.conf/

then restart zabbix agent

Creating item

Configuration-Host-click on host-items

Capture.PNG

Specify name-select key and specify Text as type of informtion

Capture.PNG

Creating trigger

In this example i created trigger for “failed” script output, if we need alerts for other outputs (“aborted”) we need to create another trigger

{rundeck:rundeck.str(“failed”)}=1

{hostname:keyname.str(script output)}=1

Capture.PNG

If we have some failed job, it will be detected on dashboard

Capture.PNG

Advertisements
Comments
  1. Kirk McCann says:

    Thats great but do you have a method of running rundeck jobs when zabbix creates an alert?

    Like

  2. Caio Monteiro says:

    aonde eu configuro esse script
    #! / bin / bash
    curl -s -H “Aceitar: aplicativo / json” -X GET “http://172.30.61.88:4440/api/20/project/demo/executions?authtoken=ZP9znayUp3Ktp26uQjSQGnEfTzDvqPDA” | python -m json.tool > /tmp/1.j

    Like

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Google photo

You are commenting using your Google account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s