Archive for the ‘RunDeck’ Category

Monitor Rundeck Jobs with Zabbix

Posted: April 4, 2018 in Linux, RunDeck

It’s presumed that Rundeck server is monitored by Zabbix

Disable the token expiration time for rundeck API (so we can authenticate on Rundeck server when running API calls)

Edit the /etc/rundeck/rundeck-config.properties file and add the following line and restart Rundeck service

rundeck.api.tokens.duration.max = 0
service rundeckd restart

Authenticate with an admin account and click on the “profile” button at the top right of the page:Administrator-Profile-add API token-save it

Capture.PNG

I modified this script  (to include job name in output)

Change URL,Rundeck token, project name and jobname (this query runs in project scope)

#!/bin/bash
curl -s -H "Accept: application/json" -X GET "http://172.30.61.88:4440/api/20/project/demo/executions?authtoken=ZP9znayUp3Ktp26uQjSQGnEfTzDvqPDA" | python -m json.tool > /tmp/1.json

jq -r '.|[.executions[] | select(.job.name != null) | select(.job.name|contains("test")) ] | sort_by(.id) | reverse | .[0] | [.status, .job.name, ."date-started".date, ."date-ended".date, .job.project] | @csv' /tmp/1.json > /tmp/1.csv
sed 's/,/ /g' /tmp/1.csv>/tmp/st.txt

while read -r status name startdate enddate project; do

startdate=${startdate//\"/}
stime=$(date -d "${startdate/T/ }" +%s)
enddate=${enddate//\"/}
etime=$(date -d "${enddate/T/ }" +%s)
let elapsed=etime-stime

if [ "$status" == "\"aborted\"" ] && [ "$elapsed" -gt 300 ]; then echo $project"-"$name-"Long-Run"
elif [ "$status" == "\"aborted\"" ] && [ "$elapsed" -lt 300 ]; then echo $project"-"$name " Aborted"
elif [ "$status" == "\"failed\"" ]; then echo $project"-"$name "failed-Demo"
#elif [ "$status" == "\"succeeded\"" ]; then echo $project"-"$name "success"
elif [ "$status" == "\"succeeded\"" ] && [ "$elapsed" -gt 300 ]; then echo $project"-"$name" Takes too long"
fi

done</tmp/st.txt

Second version (without storing output to JSON file):

#!/bin/bash

curl -s -H "Accept: application/json" -X GET "http://192.168.253.21:4440/api/20/project/demo/executions?authtoken=kH44NoX35bp1zxohgkMtsOIC9H9tw6UI" | jq -r '.|[.executions[] | select(.job.name != null) | select(.job.name|contains("test")) ] | sort_by(.id) | reverse | .[0] | [.status, .job.name, ."date-started".date, ."date-ended".date, .job.project] | @csv'  | sed 's/,/ /g' > /tmp/st.tx


while read -r status name startdate enddate project; do

startdate=${startdate//\"/}
stime=$(date -d "${startdate/T/ }" +%s)
enddate=${enddate//\"/}
etime=$(date -d "${enddate/T/ }" +%s)
let elapsed=etime-stime

if [ "$status" == "\"aborted\"" ] && [ "$elapsed" -gt 300 ]; then echo $project"-"$name-"Long-Run"
elif [ "$status" == "\"aborted\"" ] && [ "$elapsed" -lt 300 ]; then echo $project"-"$name " Aborted"
elif [ "$status" == "\"failed\"" ]; then echo $project"-"$name "failed-Demo"
#elif [ "$status" == "\"succeeded\"" ]; then echo $project"-"$name "success"
elif [ "$status" == "\"succeeded\"" ] && [ "$elapsed" -gt 300 ]; then echo $project"-"$name" Takes too long"
fi

done</tmp/st.txt

Zabix agent configuration (Creating key)

Rundeck server is monitored by Zabbix, we’ll now tell Zabbix to run this script:

In my Centos box config file is in /etc/zabbix/zabbix_agentd.conf Uncomment

EnableRemoteCommands=1

Uncomment

UserParameter=rundeck,/home/a.sh

In above example i created a key named rundeck and specified a path for script

give zabbix user ownership over script:

chown zabbix:zabbix /home/a.sh

check zabbix user bash version

zabbix:x:997:995:Zabbix Monitoring System:/var/lib/zabbix:/bin/sh
su - zabbix
./a.sh

Now test if key works:restart zabbix agent and run test:

systemctl stop zabbix-agent.service && systemctl start zabbix-agent.service && zabbix_agentd -t rundeck
[root@localhost home]# zabbix_agentd -t rundeck
rundeck                                       [t|Job  "aa" success]

Now perform test on Zabbix server

zabbix_get -s 172.30.61.88 -p 10050 -k rundeck
Job  "aa" success

 

If get “permission denied” try following (on Rundeck server)

systemctl stop zabbix-agent.service
chown -R zabbix:zabbix /var/log/zabbix
chown -R zabbix:zabbix /var/run/zabbix
chmod -R 775 /var/log/zabbix/
chmod -R 775 /var/run/zabbix/
/usr/sbin/zabbix_agentd -c /etc/zabbix/zabbix_agentd.conf
systemctl start zabbix-agent.service

If get “unsupported key” error, try to set in Zabbix agent file as follows)

Mandatory: no
# Default:
Include=/etc/zabbix/zabbix_agentd.d
#Include=/etc/zabbix/zabbix_agentd.d/*.conf/

then restart zabbix agent

Creating item

Configuration-Host-click on host-items

Capture.PNG

Specify name-select key and specify Text as type of informtion

Capture.PNG

Creating trigger

In this example i created trigger for “failed” script output, if we need alerts for other outputs (“aborted”) we need to create another trigger

{rundeck:rundeck.str(“failed”)}=1

{hostname:itemname.str(script output)}=1

Capture.PNG

If we have some failed job, it will be detected on dashboard

Capture.PNG

Advertisements

In previous post we added linux node to Rundeck server.Now, we’ll add a Windows Server

Creating AD user

I’ll be adding Domain Controller to Rundeck, so i created Domain user and put it in Built-in Administrator group,username:rundeck@test.com

Capture.PNG

Installing OpenSSH server on Windows Server

In order to run inline scripts against Windows server we need password-less connection to Windows server (private/public key authentication), because Rundeck first copies script to remote node before executing it

Download OpenSSH server,unzip it and copy it to desired destination (i put it in C:\Program Files)

Capture.PNG

With powershell browse to unzipped folder and run ./install-sshd.ps1

Two services should be installed:sshd and ssh-agent,make sure both are running-set Startup type to Automatic

Capture.PNG

Open sshd_config_default file

Capture.PNG

edit is as following:

# This is the sshd server system-wide configuration file.  See
# sshd_config(5) for more information.

# The strategy used for options in the default sshd_config shipped with
# OpenSSH is to specify options with their default value where
# possible, but leave them commented.  Uncommented options override the
# default value.

#Port 22
#AddressFamily any
#ListenAddress 0.0.0.0
#ListenAddress ::

#HostKey __PROGRAMDATA__/ssh/ssh_host_rsa_key
#HostKey __PROGRAMDATA__/ssh/ssh_host_dsa_key
#HostKey __PROGRAMDATA__/ssh/ssh_host_ecdsa_key
#HostKey __PROGRAMDATA__/ssh/ssh_host_ed25519_key

# Ciphers and keying
#RekeyLimit default none

# Logging
#SyslogFacility AUTH
#LogLevel INFO

# Authentication:
RSAAuthentication yes
#LoginGraceTime 2m
#PermitRootLogin prohibit-password
PermitRootLogin yes
StrictModes no
#MaxAuthTries 6
#MaxSessions 10
RhostsRSAAuthentication yes
PubkeyAuthentication yes

# The default is to check both .ssh/authorized_keys and .ssh/authorized_keys2
# but this is overridden so installations will only check .ssh/authorized_keys
AuthorizedKeysFile	.ssh/authorized_keys

#AuthorizedPrincipalsFile none

# For this to work you will also need host keys in %programData%/ssh/ssh_known_hosts
#HostbasedAuthentication no
# Change to yes if you don't trust ~/.ssh/known_hosts for
# HostbasedAuthentication
#IgnoreUserKnownHosts no
# Don't read the user's ~/.rhosts and ~/.shosts files
#IgnoreRhosts yes

# To disable tunneled clear text passwords, change to no here!
PasswordAuthentication yes
#PermitEmptyPasswords no

#AllowAgentForwarding yes
#AllowTcpForwarding yes
#GatewayPorts no
#PermitTTY yes
#PrintMotd yes
#PrintLastLog yes
#TCPKeepAlive yes
#UseLogin no
#PermitUserEnvironment no
#ClientAliveInterval 0
#ClientAliveCountMax 3
#UseDNS no
#PidFile /var/run/sshd.pid
#MaxStartups 10:30:100
#PermitTunnel no
#ChrootDirectory none
#VersionAddendum none

# no default banner path
#Banner none

# override default of no subsystems
Subsystem	sftp	sftp-server.exe

# Example of overriding settings on a per-user basis
#Match User anoncvs
#	AllowTcpForwarding no
#	PermitTTY no
#	ForceCommand cvs server

In Rundeck user profile folder create folder .ssh

cd C:\Users\rundeck
mkdir .ssh

Create keypair on Rundeck server (if not created)

ssh-keygen

Copy  Rundeck public key (cat /root/.ssh/id_rsa.pub) to Windows machine to .ssh folder of rundeck user-authorized_keys file, if folder is not visible enable showing hidden folders and files

Capture.PNG

On Windows,make sure port 22 is opened, restart sshd,Restart-Service sshd

Try ssh connection to Windows server from Rundeck

ssh rundeck@192.168.0.13

You shouldn’t be asked for password

Capture.PNG

Creating project

Capture.PNG

Capture.PNG

Add node (resources.xml)

Linuxtopic/server.1key was created in previous post.

 

<node name=”dc” description=”My windows” tags=”node2″ hostname=”192.168.0.13″ osArch=”x86_64″ osFamily=”Windows” osName=”Windows Server 2016″ username=”rundeck” ssh-key-storage-path=”keys/Linuxtopic/server.1key” />

 

Password authentiation

If, for some reason Public key authentication doesn’t work (it happened to me with AWS EC2 Windows instance-Write Failed: broken pipe ) , we can try password authentication

 

0-1

 

Key Type: Password

 

0

Specify Password storage created in step above and password as SSH authentication

 

Capture

 

 

Untitled

 

resources.xml:

 

<node name=”windows” description=”My windows” tags=”node2″ hostname=”1.1.1.2″ osArch=”x86_64″ osFamily=”Windows” osName=”Windows Server 2016″ username=”rundeck” ssh-authentication=”password” ssh-password-storage-path=”keys/Windows” />

 

 

Creating Job

I added Powershell script to get AD user and to create OU

Capture.PNG

Capture.PNG

Capture.PNG

Creating Rundec ACL policies

Posted: February 9, 2018 in Linux, RunDeck

Creating role

vi /var/lib/rundeck/exp/webapp/WEB-INF/web.xml

search for section security-role

1

Creating a user

The format is

username:password,rolename

vi /etc/rundeck/realm.properties
demo:demo,user,demo

We created user demo with password demo and put it to demo role

Creating policy

In this example, we’ll create policy for allowing demo role to see only aws project

(-c Context: either ‘project’ or ‘application’.

-c application   Access to projects, users, storage, system info, execution management.

-c project  Access to jobs, nodes, events, within a project.

-a allow

  • Reading read
  • Deleting delete
  • Configuring configure
  • Importing archives import
  • Exporting archives export
  • Deleting executions delete_execution
  • Export project to another Rundeck instance promote
  • Full access admin

-g group

-p project

-j job (read,update,delete,run,runAs,kill,killAs,create)

 

Access to projects (read-only)

rd-acl is tool for creating code which we can append to policy file (usually to /etc/rundeck/admin.aclpolicy)

rd-acl create -c application -g demo -p aws -a read,delete,import>>/etc/rundeck/admin.aclpolicy

Command output:

---
by:
  group: demo
context:
  application: rundeck
for:
  project:
  - allow:
    - read
    - import
    - delete
    equals:
      name: aws
description: generated

Members of demo role will be able to see only aws project

1.PNG

If we need that role have access to multiple projects we just need to add following line in /etc/rundeck/admin.aclpolicy file

---
by:

group: demo

context:

application: rundeck

for:

project:

- allow:

- read

- import

- delete

equals:

name: demo

description: generated

Access to jobs

If we want to allow some jobs we need to type following:

rd-acl create -c project -p aws -g demo -j job2 -a read,run,kill>> /etc/rundeck/admin.aclpolicy

Code added to policy file:

---
by:
  group: demo
context:
  project: aws
for:
  job:
  - allow:
    - read
    - run
    - kill
    equals:
      name: 'jobs'

Access to Activity tab

-G (node,event,job),generic
-G event (read,create)

rd-acl create -c project -p aws -g demo -G event -a read >> /etc/rundeck/admin.aclpolicy

Code in policy:

---
by:
  group: demo
context:
  project: aws
for:
  resource:
  - allow: read
    equals:
      kind: event
description: generated

1.PNG

Access to nodes

-G node (read,create,update,refresh)

rd-acl create -c project -p aws -g demo -G node -a read>> /etc/rundeck/admin.aclpolicy

Policy code:

---
by:
  group: demo
context:
  project: aws
for:
  resource:
  - allow: read
    equals:
      kind: node
description: generated

Node access can be allowed based on node tag -t (read,create,update,refresh)

rd-acl create -c project -p aws -g demo -G node -t prod -a read,refresh
---by:
  group: demo
context:
  project: aws
for:
  node:
  - allow:
    - read
    - refresh
    contains:
      tags:
      - prod
description: generated

Now, users who belong to demo role can see only node with tag named prod

Example of admin ACL

description: Admin, all access.
context:
  application: 'rundeck'
for:
  resource:
    - allow: '*' # allow create of projects
  project:
    - allow: '*' # allow view/admin of all projects
by:
  group: [Rundeck_Admin]

description: Full access.
context:
  project: '.*' # all projects
for:
  resource:
    - allow: '*' # allow read/create all kinds
  adhoc:
    - allow: '*' # allow read/running/killing adhoc jobs
  job:
    - allow: '*' # allow read/write/delete/run/kill of all jobs
  node:
    - allow: '*' # allow read/run for all nodes
by:
  group: [Rundeck_Admin]

---

description: Admin, all access.
context:
  application: 'rundeck'
for:
  resource:
    - allow: '*' # allow create of projects
  project:
    - allow: '*' # allow view/admin of all projects
  project_acl:
    - allow: '*' # allow admin of all project-level ACL policies
  storage:
    - allow: '*' # allow read/create/update/delete for all /keys/* storage content


by:
  group: [Rundeck_Admin]

Read-Only ACL:

description: "Ops Engineers can launch jobs but not edit them"
context:

project: '.*' # all projects

for:

resource:

- equals:

kind: job

allow: [read,run] # allow create jobs

- equals:

kind: node

allow: [read,update,refresh] # allow refresh node sources

- equals:

kind: event

allow: [read] # allow read/create events

adhoc:

- allow: [read,run] # allow running/killing adhoc jobs

job:

- allow: [read,run] # allow create/read/write/delete/run/kill of all jobs

node:

- allow: [read,run] # allow read/run for nodes

by:

group: [Rundeck_Jobs_RunOnly]

In last post we added node to Rundeck, now we’ll add EC2 instance as node

First,we need to add AWS EC2 plugin

cd /var/lib/rundeck/libext/
wget https://github.com/rundeck-plugins/rundeck-ec2-nodes-plugin/releases/download/v1.5.1/rundeck-ec2-nodes-plugin-1.5.1.jar
systemctl restart rundeckd

Now create New project-Add source-AWS EC2 Resources

1.PNG

Specify Access Key, Secret Key, Endpoint (for list of endpoint refer to https://docs.aws.amazon.com/general/latest/gr/rande.html#ec2_region)

In mapping parameters field specify:

name.selector=tags/Name;

hostname.selector=publicIcDnsName;

description.default=Ec2 node instance;

osArch.selector=architecture;

osFamily.selector=platform;

osFamily.default=unix;

osName.selector=platform;

osName.default=Linux;

username.selector=tags/Rundeck-User;

username.default=root;

ssh-keypath.default=/var/lib/rundeck/.ssh/id_rsa;

editUrl.default=https://console.aws.amazon.com/ec2/home#c=EC2&s=Instances;

attribute.publicIpAddress.selector=publicIpAddress;

attribute.publicDnsName.selector=publicDnsName;

tags.selector=tags/Rundeck-Tags

Click Save, EC2 node(s) should be visible in Rundeck

Line in projet properties

resources.source.2.config.mappingParams=name.selector\=tags/Name;hostname.selector\=publicDnsName;description.default\=Ec2 node instance;osArch.selector\=architecture;osFamily.selector\=platform;osFamily.default\=unix;osName.selector\=platform;osName.default\=Linux;username.selector\=tags/Rundeck-User;username.default\=root;ssh-keypath.default\=/var/lib/rundeck/.ssh/id_rsa;editUrl.default\=https\://console.aws.amazon.com/ec2/home\#c\=EC2&s\=Instances;attribute.publicIpAddress.selector\=publicIpAddress;attribute.publicDnsName.selector\=publicDnsName;tags.selector\=tags/Rundeck-Tags

1.PNG

On Rundeck server, if not already done create key pair

ssh-keygen –t rsa
cp /root/.ssh/id_rsa /var/lib/rundeck/.ssh/id_rsa
cp /root/.ssh/id_rsa.pub /var/lib/rundeck/.ssh/id_rsa.pub

Now, copy content id_rsa.pub to EC2 instance to /root/.ssh/authorized_keys

In Rundeck GUI, click on project-Nodes, EC2 instance should be visible

 

1.PNG

 

Also, command should be executed

 

1.PNG

 

Running AWS CLI from Rundeck server:

Install AWS CLI, on Rundeck server

On Rundeck go to commands tab in node specify local server, enter following command in interface:

 

aws configure set aws_access_key_id your_access_key
aws configure set aws_secret_access_key your_secret_key
aws configure set default.region us-west-2

 

Rundeck is open source software that helps  automate routine operational procedures in data center or cloud environments

Installation:

Rundeck can be configured to use RDB instead of default file-based data storage. RDB is recommended in large environment.In this post we’ll use file-based storage.

Rundeck requires java

# yum install java-1.8.0-openjdk java-1.8.0-openjdk-devel -y

Create java.sh file in /etc/profile.d and and content below:

#!/bin/bash

JAVA_HOME=/usr/bin/java

PATH=$JAVA_HOME/bin:$PATH

export PATH JAVA_HOME

export CLASSPATH=.

Then make file executable

chmod +x /etc/profile.d/java.sh
source /etc/profile.d/java.sh

Rundeck is available on port 4440-that port needs to be open:

Add below line into file: /etc/sysconfig/iptables

-A INPUT -p tcp -m state --state NEW -m tcp --dport 4440 -j ACCEPT
/etc/init.d/iptables restart

Installing Rundeck:

rpm -Uvh http://repo.rundeck.org/latest.rpm 
yum install rundeck
/etc/init.d/rundeckd start

To make sure the service is running:

/etc/init.d/rundeckd status
netstat -anp | grep '4440\|4443'

The default username and password is admin:admin, if password change for admin is required then edit the file: /etc/rundeck/realm.properties

Comment out the following line in file: /etc/rundeck/rundeck-config.properties

# Comment this out from:
grails.serverURL=http://localhost:4440

# To:
grails.serverURL=http://ip address:4440

Modify the below lines in file: /etc/rundeck/framework.properties

framework.server.name = localhost
framework.server.hostname = localhost
framework.server.port = 4440
framework.server.url = http://localhost:4440

to

framework.server.name = ip address
framework.server.hostname = ip address
framework.server.port = 4440
framework.server.url = http://ip address:4440

Now, restart the service and try to login: http://ipaddress:4440

Adding nodes

At this moment, there is no feature which would allow adding nodes using GUI
https://github.com/rundeck/rundeck/issues/1584

Create New project

1.png

Clear SSH key path

1.png

And click Create

1.png

Go to /var/rundeck/projects//etc
Edit resources.xml file

Add following line for every new node (server which needs to be managed)

1.png

New node appears in Web interface

1.png

To add another node just copy node line and change name and node IP address

Creating keypair on Rundeck server

ssh-keygen

Copy private key to clipboard:

cat /root/.ssh/id_rsa

copy content to clipboard

Now, on Rundeck interface click settings (cog icon)-Key Storage

1.png

Click Add or Upload a Key

1.png

Make sure Private Key is selected from drop-down list, paste content of ~/.ssh/id_rsa
And give key a name. Note:storage path and key name must reflect names in /var/rundeck/projects//etc resources.xml file
(ssh-key-storage-path=”keys/Linuxtopic/server.key”)

Instead of Private/Public keys, password can be used as authentication method

1.png

On client (node) create authorized_keys file (under /root/.ssh)
Copy content of id_rsa.pub file (public key) from Rundeck server to authorized_keys file on node machine
Repeat same step for every new node (copy public key from Rundeck server to /root/.ssh/authorized_keys file on every node

Running command

Now when we added node, we can run command on it, from Rundeck server go to commands-type command
From nodes, type node name-Click Run on node

1.png

Key storage

Private key uploaded to Rundeck server in previous steps are located locally on Rundeck server

/var/lib/rundeck/var/storage/content/keys// folder

1.png

Passing Rundeck password storage to script

Create password storage:

Capture

Create job-add option-specify secure-select password storage created in previous step

Capture.PNG

In script option specify arguments

Capture.PNG

In script body specify argument:

jira_password=$1
curl -XN -u user:$1

CRON job to run every first day of the month at 09:00 AM

0 00 09 1 * ? *

Changing “from” Rundeck email address

edit /etc/rundeck/rundeck-config.properties and add

grails.mail.default.from=some@mail.com

Script to test if Rundeck service is running:

#!/usr/bin/python
import sys
import os
import commands

sys.stdout = open('log.txt','wt')

output = commands.getoutput('ps -A')
if 'runuser' in output:
print("Rundeck is up an running!")

else:
os.system("systemctl start rundeckd")
print("Rundeck service started")

We can execute this script via cron:

*/5 * * * * /usr/bin/python /root/scripts/service.py

Logrotate:

cat /etc/logrotate.d/rundeck
/var/log/rundeck/rundeck*.log {
su rundeck rundeck
copytruncate
daily
missingok
rotate 1
compress
delaycompress
notifempty
create 0664 rundeck rundeck
}

/var/log/rundeck/service.log {
su rundeck rundeck
copytruncate
daily
missingok
rotate 1
compress
delaycompress
notifempty
create 0664 rundeck rundeck
}

logrotate troubleshooting:

/usr/sbin/logrotate -d /etc/logrotate.conf