Monitor Rundeck Jobs with Zabbix

Posted: April 4, 2018 in Linux, RunDeck

It’s presumed that Rundeck server is monitored by Zabbix

Disable the token expiration time for rundeck API (so we can authenticate on Rundeck server when running API calls)

Edit the /etc/rundeck/rundeck-config.properties file and add the following line and restart Rundeck service

rundeck.api.tokens.duration.max = 0
service rundeckd restart

Authenticate with an admin account and click on the “profile” button at the top right of the page:Administrator-Profile-add API token-save it

Capture.PNG

I modified this script  (to include job name in output)

Change URL,Rundeck token, project name and jobname (this query runs in project scope)

#!/bin/bash
curl -s -H "Accept: application/json" -X GET "http://172.30.61.88:4440/api/20/project/demo/executions?authtoken=ZP9znayUp3Ktp26uQjSQGnEfTzDvqPDA" | python -m json.tool > /tmp/1.json

jq -r '.|[.executions[] | select(.job.name != null) | select(.job.name|contains("test")) ] | sort_by(.id) | reverse | .[0] | [.status, .job.name, ."date-started".date, ."date-ended".date, .job.project] | @csv' /tmp/1.json > /tmp/1.csv
sed 's/,/ /g' /tmp/1.csv>/tmp/st.txt

while read -r status name startdate enddate project; do

startdate=${startdate//\"/}
stime=$(date -d "${startdate/T/ }" +%s)
enddate=${enddate//\"/}
etime=$(date -d "${enddate/T/ }" +%s)
let elapsed=etime-stime

if [ "$status" == "\"aborted\"" ] && [ "$elapsed" -gt 300 ]; then echo $project"-"$name-"Long-Run"
elif [ "$status" == "\"aborted\"" ] && [ "$elapsed" -lt 300 ]; then echo $project"-"$name " Aborted"
elif [ "$status" == "\"failed\"" ]; then echo $project"-"$name "failed-Demo"
#elif [ "$status" == "\"succeeded\"" ]; then echo $project"-"$name "success"
elif [ "$status" == "\"succeeded\"" ] && [ "$elapsed" -gt 300 ]; then echo $project"-"$name" Takes too long"
fi

done</tmp/st.txt

Second version (without storing output to JSON file):

#!/bin/bash

curl -s -H "Accept: application/json" -X GET "http://192.168.253.21:4440/api/20/project/demo/executions?authtoken=kH44NoX35bp1zxohgkMtsOIC9H9tw6UI" | jq -r '.|[.executions[] | select(.job.name != null) | select(.job.name|contains("test")) ] | sort_by(.id) | reverse | .[0] | [.status, .job.name, ."date-started".date, ."date-ended".date, .job.project] | @csv'  | sed 's/,/ /g' > /tmp/st.tx


while read -r status name startdate enddate project; do

startdate=${startdate//\"/}
stime=$(date -d "${startdate/T/ }" +%s)
enddate=${enddate//\"/}
etime=$(date -d "${enddate/T/ }" +%s)
let elapsed=etime-stime

if [ "$status" == "\"aborted\"" ] && [ "$elapsed" -gt 300 ]; then echo $project"-"$name-"Long-Run"
elif [ "$status" == "\"aborted\"" ] && [ "$elapsed" -lt 300 ]; then echo $project"-"$name " Aborted"
elif [ "$status" == "\"failed\"" ]; then echo $project"-"$name "failed-Demo"
#elif [ "$status" == "\"succeeded\"" ]; then echo $project"-"$name "success"
elif [ "$status" == "\"succeeded\"" ] && [ "$elapsed" -gt 300 ]; then echo $project"-"$name" Takes too long"
fi

done</tmp/st.txt

Zabix agent configuration (Creating key)

Rundeck server is monitored by Zabbix, we’ll now tell Zabbix to run this script:

In my Centos box config file is in /etc/zabbix/zabbix_agentd.conf Uncomment

EnableRemoteCommands=1

Uncomment

UserParameter=rundeck,/home/a.sh

In above example i created a key named rundeck and specified a path for script

give zabbix user ownership over script:

chown zabbix:zabbix /home/a.sh

check zabbix user bash version

zabbix:x:997:995:Zabbix Monitoring System:/var/lib/zabbix:/bin/sh
su - zabbix
./a.sh

Now test if key works:restart zabbix agent and run test:

systemctl stop zabbix-agent.service && systemctl start zabbix-agent.service && zabbix_agentd -t rundeck
[root@localhost home]# zabbix_agentd -t rundeck
rundeck                                       [t|Job  "aa" success]

Now perform test on Zabbix server

zabbix_get -s 172.30.61.88 -p 10050 -k rundeck
Job  "aa" success

 

If get “permission denied” try following (on Rundeck server)

systemctl stop zabbix-agent.service
chown -R zabbix:zabbix /var/log/zabbix
chown -R zabbix:zabbix /var/run/zabbix
chmod -R 775 /var/log/zabbix/
chmod -R 775 /var/run/zabbix/
/usr/sbin/zabbix_agentd -c /etc/zabbix/zabbix_agentd.conf
systemctl start zabbix-agent.service

If get “unsupported key” error, try to set in Zabbix agent file as follows)

Mandatory: no
# Default:
Include=/etc/zabbix/zabbix_agentd.d
#Include=/etc/zabbix/zabbix_agentd.d/*.conf/

then restart zabbix agent

Creating item

Configuration-Host-click on host-items

Capture.PNG

Specify name-select key and specify Text as type of informtion

Capture.PNG

Creating trigger

In this example i created trigger for “failed” script output, if we need alerts for other outputs (“aborted”) we need to create another trigger

{rundeck:rundeck.str(“failed”)}=1

{hostname:keyname.str(script output)}=1

Capture.PNG

If we have some failed job, it will be detected on dashboard

Capture.PNG

Comments
  1. Kirk McCann says:

    Thats great but do you have a method of running rundeck jobs when zabbix creates an alert?

    Like

  2. Caio Monteiro says:

    aonde eu configuro esse script
    #! / bin / bash
    curl -s -H “Aceitar: aplicativo / json” -X GET “http://172.30.61.88:4440/api/20/project/demo/executions?authtoken=ZP9znayUp3Ktp26uQjSQGnEfTzDvqPDA” | python -m json.tool > /tmp/1.j

    Like

  3. Thiago Nascimento says:

    Hi! Your tutorial are helping me a lot. I must to monitor a RDK job with my zabbix and im seeking for a wai to do that. But… when im triyng to use your scripts a got the message “file truncated” at st.txt file and no data is showed. Can you help me with this?

    Like

    • dragan979 says:

      what’s the content of /tmp/st.txt file ?

      if it’s empty, run curl -s -H “Accept: application/json” -X GET “http://172.30.61.88:4440/api/20/project/demo/executions?authtoken=ZP9znayUp3Ktp26uQjSQGnEfTzDvqPDA” | python -m json.tool

      replace IP,token and project name, you should get JSON output

      Like

      • Thiago Nascimento says:

        Hi, dragan! Thanks for reply. My st.txt file always return just one job running in the project. But i have at last three running at the same time. It comes with the new one id (sorted) and trunked. If the others jobs faileds it is not showed by the script. 😦

        Like

    • Thiago Nascimento says:

      Just in time. Im not apllying the job name filtering. Im tring to catch any one job failed.

      Like

  4. Pranesh says:

    what is that a.sh file contain?

    Like

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Google photo

You are commenting using your Google account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s