In order to change subnet of EC2 instance stop it firs

 

1.png

 

Then create AMI image from that instance

 

2.png

 

 

3.PNG

Click on new AMI then launch it

 

4.png

 

Select desired subnet

 

5.png

 

 

6.PNG

 

New instance is in different subnet and data are preserved

 

7.PNG

Advertisements

Install sasld

yum install cyrus-sasl cyrus-sasl-plain cyrus-sasl-md5
systemctl start saslauthd

in /etc/postfix folder create file sasl_passwd and put username and password of mailbox which will be used as relay

[smtp.office365.com]:587 user@domain.com:Pass

To set default “from” to be this email open file /etc/postfix/generic

Add this at bottom (this is amazon AWS instance)

root@ip-1-18-23-1.company.com user@domain.com

All emails will be rewritten with in from field with one we just specified

postmap hash:/etc/postfix/sasl_passwd
postmap hash:/etc/postfix/generic

now add following lines in /etc/postfix/main.cf

relayhost = [smtp.office365.com]:587
smtp_sasl_password_maps = hash:/etc/postfix/sasl_passwd
smtp_sasl_auth_enable = yes
smtp_generic_maps = hash:/etc/postfix/generic
smtp_tls_security_level = may
smtp_sasl_security_options = noanonymous

Restart postfix service and you should be able to send emails through Office 365

If you get “Office 365 unreachable” error, in /etc/postfix/main.cf
change inet_protocols line from all to ipv4

Certificate authentication

If we want to secure our connection we can secure it with certificate:

openssl s_client -showcerts -starttls smtp -crlf -connect smtp.office365.com:587

In my case i got 2 certificates, i copied it in cacert.pem file

Capture.PNG

then add following lines to /etc/postfix/main.cf

smtp_send_dummy_mail_auth = yes
smtp_always_send_ehlo = yes
smtp_tls_security_level = encrypt
smtp_use_tls = yes
smtp_tls_CAfile = /etc/postfix/cacert.pem

Restart postfix service and your traffic is now encrypted

Monitor Rundeck Jobs with Zabbix

Posted: April 4, 2018 in Linux, RunDeck

It’s presumed that Rundeck server is monitored by Zabbix

Disable the token expiration time for rundeck API (so we can authenticate on Rundeck server when running API calls)

Edit the /etc/rundeck/rundeck-config.properties file and add the following line and restart Rundeck service

rundeck.api.tokens.duration.max = 0
service rundeckd restart

Authenticate with an admin account and click on the “profile” button at the top right of the page:Administrator-Profile-add API token-save it

Capture.PNG

I modified this script  (to include job name in output)

Change URL,Rundeck token, project name and jobname (this query runs in project scope)

#!/bin/bash
curl -s -H "Accept: application/json" -X GET "http://172.30.61.88:4440/api/20/project/demo/executions?authtoken=ZP9znayUp3Ktp26uQjSQGnEfTzDvqPDA" | python -m json.tool > /tmp/1.json

jq -r '.|[.executions[] | select(.job.name != null) | select(.job.name|contains("test")) ] | sort_by(.id) | reverse | .[0] | [.status, .job.name, ."date-started".date, ."date-ended".date, .job.project] | @csv' /tmp/1.json > /tmp/1.csv
sed 's/,/ /g' /tmp/1.csv>/tmp/st.txt

while read -r status name startdate enddate project; do

startdate=${startdate//\"/}
stime=$(date -d "${startdate/T/ }" +%s)
enddate=${enddate//\"/}
etime=$(date -d "${enddate/T/ }" +%s)
let elapsed=etime-stime

if [ "$status" == "\"aborted\"" ] && [ "$elapsed" -gt 300 ]; then echo $project"-"$name-"Long-Run"
elif [ "$status" == "\"aborted\"" ] && [ "$elapsed" -lt 300 ]; then echo $project"-"$name " Aborted"
elif [ "$status" == "\"failed\"" ]; then echo $project"-"$name "failed-Demo"
#elif [ "$status" == "\"succeeded\"" ]; then echo $project"-"$name "success"
elif [ "$status" == "\"succeeded\"" ] && [ "$elapsed" -gt 300 ]; then echo $project"-"$name" Takes too long"
fi

done</tmp/st.txt

Second version (without storing output to JSON file):

#!/bin/bash

curl -s -H "Accept: application/json" -X GET "http://192.168.253.21:4440/api/20/project/demo/executions?authtoken=kH44NoX35bp1zxohgkMtsOIC9H9tw6UI" | jq -r '.|[.executions[] | select(.job.name != null) | select(.job.name|contains("test")) ] | sort_by(.id) | reverse | .[0] | [.status, .job.name, ."date-started".date, ."date-ended".date, .job.project] | @csv'  | sed 's/,/ /g' > /tmp/st.tx


while read -r status name startdate enddate project; do

startdate=${startdate//\"/}
stime=$(date -d "${startdate/T/ }" +%s)
enddate=${enddate//\"/}
etime=$(date -d "${enddate/T/ }" +%s)
let elapsed=etime-stime

if [ "$status" == "\"aborted\"" ] && [ "$elapsed" -gt 300 ]; then echo $project"-"$name-"Long-Run"
elif [ "$status" == "\"aborted\"" ] && [ "$elapsed" -lt 300 ]; then echo $project"-"$name " Aborted"
elif [ "$status" == "\"failed\"" ]; then echo $project"-"$name "failed-Demo"
#elif [ "$status" == "\"succeeded\"" ]; then echo $project"-"$name "success"
elif [ "$status" == "\"succeeded\"" ] && [ "$elapsed" -gt 300 ]; then echo $project"-"$name" Takes too long"
fi

done</tmp/st.txt

Zabix agent configuration (Creating key)

Rundeck server is monitored by Zabbix, we’ll now tell Zabbix to run this script:

In my Centos box config file is in /etc/zabbix/zabbix_agentd.conf Uncomment

EnableRemoteCommands=1

Uncomment

UserParameter=rundeck,/home/a.sh

In above example i created a key named rundeck and specified a path for script

give zabbix user ownership over script:

chown zabbix:zabbix /home/a.sh

check zabbix user bash version

zabbix:x:997:995:Zabbix Monitoring System:/var/lib/zabbix:/bin/sh
su - zabbix
./a.sh

Now test if key works:restart zabbix agent and run test:

systemctl stop zabbix-agent.service && systemctl start zabbix-agent.service && zabbix_agentd -t rundeck
[root@localhost home]# zabbix_agentd -t rundeck
rundeck                                       [t|Job  "aa" success]

Now perform test on Zabbix server

zabbix_get -s 172.30.61.88 -p 10050 -k rundeck
Job  "aa" success

 

If get “permission denied” try following (on Rundeck server)

systemctl stop zabbix-agent.service
chown -R zabbix:zabbix /var/log/zabbix
chown -R zabbix:zabbix /var/run/zabbix
chmod -R 775 /var/log/zabbix/
chmod -R 775 /var/run/zabbix/
/usr/sbin/zabbix_agentd -c /etc/zabbix/zabbix_agentd.conf
systemctl start zabbix-agent.service

If get “unsupported key” error, try to set in Zabbix agent file as follows)

Mandatory: no
# Default:
Include=/etc/zabbix/zabbix_agentd.d
#Include=/etc/zabbix/zabbix_agentd.d/*.conf/

then restart zabbix agent

Creating item

Configuration-Host-click on host-items

Capture.PNG

Specify name-select key and specify Text as type of informtion

Capture.PNG

Creating trigger

In this example i created trigger for “failed” script output, if we need alerts for other outputs (“aborted”) we need to create another trigger

{rundeck:rundeck.str(“failed”)}=1

{hostname:itemname.str(script output)}=1

Capture.PNG

If we have some failed job, it will be detected on dashboard

Capture.PNG

In previous post we added linux node to Rundeck server.Now, we’ll add a Windows Server

Creating AD user

I’ll be adding Domain Controller to Rundeck, so i created Domain user and put it in Built-in Administrator group,username:rundeck@test.com

Capture.PNG

Installing OpenSSH server on Windows Server

In order to run inline scripts against Windows server we need password-less connection to Windows server (private/public key authentication), because Rundeck first copies script to remote node before executing it

Download OpenSSH server,unzip it and copy it to desired destination (i put it in C:\Program Files)

Capture.PNG

With powershell browse to unzipped folder and run ./install-sshd.ps1

Two services should be installed:sshd and ssh-agent,make sure both are running-set Startup type to Automatic

Capture.PNG

Open sshd_config_default file

Capture.PNG

edit is as following:

# This is the sshd server system-wide configuration file.  See
# sshd_config(5) for more information.

# The strategy used for options in the default sshd_config shipped with
# OpenSSH is to specify options with their default value where
# possible, but leave them commented.  Uncommented options override the
# default value.

#Port 22
#AddressFamily any
#ListenAddress 0.0.0.0
#ListenAddress ::

#HostKey __PROGRAMDATA__/ssh/ssh_host_rsa_key
#HostKey __PROGRAMDATA__/ssh/ssh_host_dsa_key
#HostKey __PROGRAMDATA__/ssh/ssh_host_ecdsa_key
#HostKey __PROGRAMDATA__/ssh/ssh_host_ed25519_key

# Ciphers and keying
#RekeyLimit default none

# Logging
#SyslogFacility AUTH
#LogLevel INFO

# Authentication:
RSAAuthentication yes
#LoginGraceTime 2m
#PermitRootLogin prohibit-password
PermitRootLogin yes
StrictModes no
#MaxAuthTries 6
#MaxSessions 10
RhostsRSAAuthentication yes
PubkeyAuthentication yes

# The default is to check both .ssh/authorized_keys and .ssh/authorized_keys2
# but this is overridden so installations will only check .ssh/authorized_keys
AuthorizedKeysFile	.ssh/authorized_keys

#AuthorizedPrincipalsFile none

# For this to work you will also need host keys in %programData%/ssh/ssh_known_hosts
#HostbasedAuthentication no
# Change to yes if you don't trust ~/.ssh/known_hosts for
# HostbasedAuthentication
#IgnoreUserKnownHosts no
# Don't read the user's ~/.rhosts and ~/.shosts files
#IgnoreRhosts yes

# To disable tunneled clear text passwords, change to no here!
PasswordAuthentication yes
#PermitEmptyPasswords no

#AllowAgentForwarding yes
#AllowTcpForwarding yes
#GatewayPorts no
#PermitTTY yes
#PrintMotd yes
#PrintLastLog yes
#TCPKeepAlive yes
#UseLogin no
#PermitUserEnvironment no
#ClientAliveInterval 0
#ClientAliveCountMax 3
#UseDNS no
#PidFile /var/run/sshd.pid
#MaxStartups 10:30:100
#PermitTunnel no
#ChrootDirectory none
#VersionAddendum none

# no default banner path
#Banner none

# override default of no subsystems
Subsystem	sftp	sftp-server.exe

# Example of overriding settings on a per-user basis
#Match User anoncvs
#	AllowTcpForwarding no
#	PermitTTY no
#	ForceCommand cvs server

In Rundeck user profile folder create folder .ssh

cd C:\Users\rundeck
mkdir .ssh

Create keypair on Rundeck server (if not created)

ssh-keygen

Copy  Rundeck public key (cat /root/.ssh/id_rsa.pub) to Windows machine to .ssh folder of rundeck user-authorized_keys file, if folder is not visible enable showing hidden folders and files

Capture.PNG

On Windows,make sure port 22 is opened, restart sshd,Restart-Service sshd

Try ssh connection to Windows server from Rundeck

ssh rundeck@192.168.0.13

You shouldn’t be asked for password

Capture.PNG

Creating project

Capture.PNG

Capture.PNG

Add node (resources.xml)

Linuxtopic/server.1key was created in previous post.

 

<node name=”dc” description=”My windows” tags=”node2″ hostname=”192.168.0.13″ osArch=”x86_64″ osFamily=”Windows” osName=”Windows Server 2016″ username=”rundeck” ssh-key-storage-path=”keys/Linuxtopic/server.1key” />

 

Password authentiation

If, for some reason Public key authentication doesn’t work (it happened to me with AWS EC2 Windows instance-Write Failed: broken pipe ) , we can try password authentication

 

0-1

 

Key Type: Password

 

0

Specify Password storage created in step above and password as SSH authentication

 

Capture

 

 

Untitled

 

resources.xml:

 

<node name=”windows” description=”My windows” tags=”node2″ hostname=”1.1.1.2″ osArch=”x86_64″ osFamily=”Windows” osName=”Windows Server 2016″ username=”rundeck” ssh-authentication=”password” ssh-password-storage-path=”keys/Windows” />

 

 

Creating Job

I added Powershell script to get AD user and to create OU

Capture.PNG

Capture.PNG

Capture.PNG

In previous article we created federation trust between Azure and AWS by creating Amazon user and used it’s credentials to create trust between Azure and AWS (automatic provisioning).This method has 2 main drawbacks: it takes a long time for Azure to retrive all IAM roles,and it’s not possible to provide more than 1 IAM credentials (situation when need to federate same Azure Enterprise application with 2 or more AWS accounts).Most of the steps are same as for manual provisioning but i’ll put it here again for the sake of transparency

Adding Amazon Application to Azure portal

On Azure portal Azure Active Directory-Enterprise Applications-All applications-New Application

1

In search box type Amazone-select Amazon Web Services (AWS)

2

On AWS app properties click on Single sign-on

3

Click Add attribute

4

Add attributes as in picture below

Attribute name Attribute value Namespace
RoleSessionName user.userprincipalname https://aws.amazon.com/SAML/Attributes
Role user.assignedroles https://aws.amazon.com/SAML/Attributes

 

 

5

In the SAML Signing Certificate section, select Metadata XML. Then save the metadata file on your computer.

5-1

5-2

Then click Save

5-3

AWS Console:Creating Provider and IAM role

In AWS console we need to add Provider, IAM role and policy

Select Identity and access management-IAM

6

Identity Providers-Create Provider

7

Choose SAML as Provider Type,set name and browse for metadata file downloaded from Azure portal

9

10

Still in IAM Click Roles-Create Role

11

Select Saml 2.0 Federation-SAML provider-provider we created earlier-Allow programatic and AWS Management Console Access (Attribute and Value fields populate automatically)

12

In Attach permission policies click Next:Review

13

In Create Role create as many roles as you need

14

Besides Azure_Role, i created another one and attached one IAM policy, we’ll map this role to another Azure AD Group

Untitled

Azure portal:Create User and Group-add user to group

In this section we’ll map Azure AD group to AWS role we just created (Azure_Role)

Creating new user:

Azure Active Directory-Users-All Users:

26

Create user

27

Creating AD group

Azure Active Directory-Groups

Untitled

Specify Group Type,name-Membership Type:Assigned-specify user(s) to add to group-Select-Create

Untitled

In the same way i created another Azure AD group (AWS_Second_Test_Group) to map it with another IAM role we created earlier (AWS_Second_Test_Role), i added Don.Hall user to this group too

Editing Azure Active directory manifest file

Manifest file is a JSON file that represents application’s identity configuration.We’ll edit this file to map Azure AD group with AWS IAM Role.Access scopes and roles are exposed through this file

In Azure portal, in search box type App Registrations-Select Amazon Web Services (AWS)

Untitled.png

Click on Manifest

Untitled

Now, we’ll map IAM AWS roles to Azure AD groups:

IAM Role name Azure AD Group Name
 Azure_Role Azure_AD_Group
AWS_Second_Test_Role AWS_Second_Test_Group

Ideally, names of IAM Roles and groups should be the same to avoid confusion

In order to edit manifest file we need to obtanin IAM Role ARN,AWS Identity provider ARN and Azure AD group ID (Azure AD Group ID must be unique-as a rule of thumb i just changed last 2 digits)

Capture

AWS IAM role ARN:

Untitled.png

Untitled.png

AWS Identity provider ARN:

Untitled

Azure Group’s ID:

Click on group-Properties:

Capture

Untitled

Remember, Azure AD group ID’s needs to be unique, so change last digit(s) values

These 2 sections are added to manifest file:

displayName:Name of Azure AD group

id:id of Azure AD group (changes last 2 digits-needs to be unique)

value:AWS IAM role ARN,AWS identity provider ARN

"appRoles": [
    {
      "allowedMemberTypes": [
        "User"
      ],
      "displayName": "AWS_Second_Test_Group",
      "id": "faa9acbc-49db-4a04-9a66-2050998f1c15",
      "isEnabled": true,
      "description": "Azure AD Second group",
      "value": "arn:aws:iam::233135199200:role/AWS_Second_Test_Role,arn:aws:iam::233135199200:saml-provider/WindowsAD"
    },
    {
      "allowedMemberTypes": [
        "User"
      ],
      "displayName": "Azure_AD_Group",
      "id": "b40569c7-ebf0-4c32-959c-b0b3b1cbfc12",
      "isEnabled": true,
      "description": "Azure AD First group",
      "value": "arn:aws:iam::233135199200:role/Azure_Role,arn:aws:iam::233135199200:saml-provider/WindowsAD"
    },

If we need to map more roles to groups we just need to add allowedMemberTypes sections (separate each one with comma)

Here is complete manifest file:

{
  "appId": "1def2fa6-5467-4565-b3f0-e598b3007b42",
  "appRoles": [
    {
      "allowedMemberTypes": [
        "User"
      ],
      "displayName": "AWS_Second_Test_Group",
      "id": "faa9acbc-49db-4a04-9a66-2050998f1c15",
      "isEnabled": true,
      "description": "Azure AD Second group",
      "value": "arn:aws:iam::233135199200:role/AWS_Second_Test_Role,arn:aws:iam::233135199200:saml-provider/WindowsAD"
    },
    {
      "allowedMemberTypes": [
        "User"
      ],
      "displayName": "Azure_AD_Group",
      "id": "b40569c7-ebf0-4c32-959c-b0b3b1cbfc12",
      "isEnabled": true,
      "description": "Azure AD First group",
      "value": "arn:aws:iam::233135199200:role/Azure_Role,arn:aws:iam::233135199200:saml-provider/WindowsAD"
    },
    {
      "allowedMemberTypes": [
        "User"
      ],
      "displayName": "msiam_access",
      "id": "7dfd756e-8c27-4472-b2b7-38c17fc5de5e",
      "isEnabled": true,
      "description": "msiam_access",
      "value": null
    }
  ],
  "availableToOtherTenants": false,
  "displayName": "Amazon Web Services (AWS)",
  "errorUrl": null,
  "groupMembershipClaims": null,
  "optionalClaims": null,
  "acceptMappedClaims": null,
  "homepage": "https://signin.aws.amazon.com/saml?metadata=aws|ISV9.1|primary|z",
  "informationalUrls": {
    "privacy": null,
    "termsOfService": null
  },
  "identifierUris": [
    "http://awsDC46DF5ECB354EEA858E81622348A0BE",
    "http://instanceid_8b1025e4-1dd2-430b-a150-2ef79cd700f5_EAAEA402D2364790A14A5099A13A3B7E",
    "http://aws/d38c1eb9-ca01-420f-a982-210c0583dc49"
  ],
  "keyCredentials": [],
  "knownClientApplications": [],
  "logoutUrl": null,
  "oauth2AllowImplicitFlow": false,
  "oauth2AllowUrlPathMatching": false,
  "oauth2Permissions": [
    {
      "adminConsentDescription": "Allow the application to access Amazon Web Services (AWS) on behalf of the signed-in user.",
      "adminConsentDisplayName": "Access Amazon Web Services (AWS)",
      "id": "e81ccfaa-9095-4cbc-87fe-10538a57f314",
      "isEnabled": true,
      "type": "User",
      "userConsentDescription": "Allow the application to access Amazon Web Services (AWS) on your behalf.",
      "userConsentDisplayName": "Access Amazon Web Services (AWS)",
      "value": "user_impersonation"
    }
  ],
  "oauth2RequirePostResponse": false,
  "objectId": "dd1dc07d-87dc-48bb-9fd3-1c0274c789a5",
  "parentalControlSettings": {
    "countriesBlockedForMinors": [],
    "legalAgeGroupRule": "Allow"
  },
  "passwordCredentials": [],
  "publicClient": false,
  "replyUrls": [
    "https://signin.aws.amazon.com/saml"
  ],
  "requiredResourceAccess": [],
  "samlMetadataUrl": null
}

Azure Active Directory-Enterprise Applications-Amazon Web Services (AWS)-Users and Groups-Add User

Capture.PNG

In Users section Assign user- in Roles section new roles should appear-select role

Capture

And assign it

Capture

Capture

Make sure manual provision method is selected (Amazon Web Services (AWS)-Provisioning

Capture

Testing access to AWS console

Don.Hall should be able now to acces to Amazon Web Console

Go to http://myapps.microsoft.com, log in as Don.Hall

click on Amazon Web Service, you should be able to sign in automatically to AWS console

31

 

Capture

In this article we’ll create Azure AD User and log him in into Amazon Web Console using single sign-on

Adding Amazon Application to Azure portal

Azure Active Directory-Enterprise Applications-All applications-New Application

1

In search box type Amazone-select Amazon Web Services (AWS)

2

On AWS app properties click on Single sign-on

3

Click Add attribute

4

Add attributes as in picture below

Attribute name Attribute value Namespace
RoleSessionName user.userprincipalname https://aws.amazon.com/SAML/Attributes
Role user.assignedroles https://aws.amazon.com/SAML/Attributes

 

5.png

In the SAML Signing Certificate section, select Metadata XML. Then save the metadata file on your computer.

5-1

5-2

Then click Save

5-3

Configuring AWS part

In AWS console we need to add Provider, IAM role and policy

Select Identity and access management-IAM

6

Identity Providers-Create Provider

7.png

Choose SAML as Provider Type,set name and browse for metadata file downloaded from Azure portal

9

10

Still in IAM Click Roles-Create Role

11

Select Saml 2.0 Federation-SAML provider-provider we created earlier-Allow programatic and AWS Management Console Access (Attribute and Value fields populate automatically)

12

In Attach permission policies click Next:Review

13

In Create Role create as many roles as you need

14

Creating Policy

Policies-Create policy (this policy will grab all IAM roles from AWS account)

15

Click JSON tab and paste following code:

{
"Version": "2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Action": [
"iam:ListRoles"
],
"Resource": "*"
}
]
}

 

 

16

17

Creating new AWS user

We need to create new user,attach policy we just created,get credentials so we can submit it to Azure AWS application so we can get all Amazon AWS roles

18

19

20

Download CSV file (Access and shared access keys are there)

21

In Azure portal,in AWS app properties click Provisioning-for client secret enter AWS user access key,for Sercret token enter AWS user secret and click Test Connection

22

Scroll down, set On for Provision status then click Save

23

Creating Azure AD  user

Azure Active Directory-Users-Al Users

26

Create user

27

Enabling Azure Single sign-on for user

In AWS application properties select Users and Groups

28

Select user and click Select button

29

Click assign

30

Testing access to AWS console

Don.Hall should be able now to acces to Amazon Web Console

Go to http://myapps.microsoft.com, log in as Don.Hall

click on Amazon Web Service, you should be able to sign in automatically to AWS console

 

31

 

32

 

 

 

 

In this post we’ll get IAM reports using AWS CLI.

 #get IAM Users——————————————————

yes | cp -rf /media/IAM_REPORT/IAM_REPORT_users.csv /media/IAM_REPORT/IAM_REPORT_users_old.csv

aws iam list-users>/media/IAM_REPORT/users.json

jq -r ‘.Users[] | [.UserName] | @csv’ /media/IAM_REPORT/users.json > /media/IAM_REPORT/users.csv

{ echo “Users”; cat /media/IAM_REPORT/users.csv; } > /media/IAM_REPORT/IAM_REPORT_users.csv

comm -13 <(sort /media/IAM_REPORT/IAM_REPORT_users.csv) /media/IAM_REPORT/IAM_REPORT_users_changes.txt

#get IAM Groups———————————————————————-

yes | cp -rf /media/IAM_REPORT/IAM_REPORT_groups.csv /media/IAM_REPORT/IAM_REPORT_groups_old.csv

aws iam list-groups>/media/IAM_REPORT/groups.json

jq -r ‘.Groups[] | [.GroupName] | @csv’ /media/IAM_REPORT/groups.json > /media/IAM_REPORT/groups.csv

{ echo “Groups”; cat /media/IAM_REPORT/groups.csv; } > /media/IAM_REPORT/IAM_REPORT_groups.csv

comm -23 <(sort /media/IAM_REPORT/IAM_REPORT_groups.csv) /media/IAM_REPORT/IAM_REPORT_groups_changes.txt

#Get users with associated policies——————————————-

echo yes | cp /media/IAM_REPORT/IAM_REPORT_users_policies.csv /media/IAM_REPORT/IAM_REPORT_users_policies_old.csv

aws iam get-account-authorization-details > /media/IAM_REPORT/output.json

jq -r ‘.UserDetailList[] | .UserName as $u | .AttachedManagedPolicies[] | [$u, .PolicyName] | @csv’ /media/IAM_REPORT/output.json > /media/IAM_REPORT/userpolicies.csv

{ echo “Users,Policy”; cat /media/IAM_REPORT/userpolicies.csv; } > /media/IAM_REPORT/IAM_REPORT_users_policies.csv

comm -23 <(sort /media/IAM_REPORT/IAM_REPORT_users_policies.csv) /media/IAM_REPORT/IAM_REPORT_users_policies_changes.txt

#Get Groups with associated policies —————

echo yes | cp /media/IAM_REPORT/IAM_REPORT_group_policies.csv /media/IAM_REPORT/IAM_REPORT_group_policies_old.csv

jq -r ‘.GroupDetailList[] | .GroupName as $u | .AttachedManagedPolicies[] | [$u, .PolicyName] | @csv’ /media/IAM_REPORT/output.json > /media/IAM_REPORT/grouppolicies.csv

{ echo “Groups,Policies”; cat /media/IAM_REPORT/grouppolicies.csv; } > /media/IAM_REPORT/IAM_REPORT_group_policies.csv

comm -23 <(sort /media/IAM_REPORT/IAM_REPORT_group_policies.csv) /media/IAM_REPORT/IAM_REPORT_group_policies_changes.txt

#Roles assigned to policies———————————————

echo yes | cp /media/IAM_REPORT/IAM_REPORT_role_policies.csv /media/IAM_REPORT/IAM_REPORT_role_policies_old.csv

jq -r ‘.RoleDetailList | map(select(.AttachedManagedPolicies | length > 0))[] | .RoleName as $r | .AttachedManagedPolicies[] | [$r, .PolicyName] | @csv’ /media/IAM_REPORT/output.json>/media/IAM_REPORT/rolepolicy.csv

{ echo “Role,Policy”; cat /media/IAM_REPORT/rolepolicy.csv; } > /media/IAM_REPORT/IAM_REPORT_role_policies.csv

comm -23 <(sort /media/IAM_REPORT/IAM_REPORT_role_policies.csv) /media/IAM_REPORT/IAM_REPORT_role_policies_changes.txt

#Get IAM roles for group/user——————————————————-

echo yes | cp /media/IAM_REPORT/IAM_REPORT_role_assignment.csv /media/IAM_REPORT/IAM_REPORT_role_assignment_old.csv

jq -rc ‘.RoleDetailList | map(select((.AssumeRolePolicyDocument.Statement | length > 0) and (.AssumeRolePolicyDocument.Statement[].Principal.Service) or (.AssumeRolePolicyDocument.Statement[].Principal.AWS) or (.AssumeRolePolicyDocument.Statement[].Principal.Federated) or (.AttachedManagedPolicies | length >0) or (.RolePolicyList | length > 0)) )[] | [.RoleName,([.RolePolicyList[].PolicyName,([.AttachedManagedPolicies[].PolicyName] | join(“–“))] | join(” “)),(.AssumeRolePolicyDocument.Statement[] | .Principal.Federated + “” + (.Principal.Service | if type == “array” then join(“–“) else . end) + “”+.Principal.AWS)] | @csv’ /media/IAM_REPORT/output.json>/media/IAM_REPORT/roleassign.csv

#awk ‘BEGIN{ FS=OFS=”,” }$2 ~ /^”arn:aws:iam:/{ $2 = “,”$2 }1’ /media/IAM_REPORT/roles.csv>/media/IAM_REPORT/roleassign.csv

{ echo “Role,Policy,User/Group/Service”; cat /media/IAM_REPORT/roleassign.csv; } > /media/IAM_REPORT/IAM_REPORT_role_assignment.csv

comm -23 <(sort /media/IAM_REPORT/IAM_REPORT_role_assignment.csv) /media/IAM_REPORT/IAM_REPORT_role_assignemnt_changes.txt

#zip files———————————————————-

find /media/IAM_REPORT/*.txt -maxdepth 1 -size +0 -print | zip /tmp/IAM_REPORT.zip -@ /media/IAM_REPORT/IAM_REPORT_role_policies.csv /media/IAM_REPORT/IAM_REPORT_group_policies.csv /media/IAM_REPORT/IAM_REPORT_groups.csv /media/IAM_REPORT/IAM_REPORT_users.csv /media/IAM_REPORT/IAM_REPORT_users_policies.csv /media/IAM_REPORT/IAM_REPORT_role_assignment.csv

#send email

echo “IAM Reports-IAM_REPORT” | mail -r rundeck@company.com -s “AWS IAM reports ” -a /tmp/IAM_REPORT.zip “user@company.com”

########This section creates HTML mail and sends report as message body

#prepare HTML

#PYTHON_ARG=”$1″ python – <<END
#!/usr/bin/python

#import pandas as pd
#import sys
#import json
#sys.stdout = open('/media/IAM_REPORT/group_policy.html','wt')

#df = pd.read_csv('/media//IAM_REPORT/IAM_REPORT_group_policies.csv')
#pd.set_option('display.max_colwidth', -1)
#pd.DataFrame({'a': [1, 2]}).to_html()
#print(df.to_html(index=False))
#END

#PYTHON_ARG="$1" python – <<END
#!/usr/bin/python

#import pandas as pd
#import sys
#import json
#sys.stdout = open('/media/IAM_REPORT/groups.html','wt')

#df = pd.read_csv('/media/IAM_REPORT/IAM_REPORT_groups.csv')
#pd.set_option('display.max_colwidth', -1)
#pd.DataFrame({'a': [1, 2]}).to_html()
#print(df.to_html(index=False))
#END

#PYTHON_ARG="$1" python – <<END
#!/usr/bin/python

#import pandas as pd
#import sys
#import json
#sys.stdout = open('/media/IAM_REPORT/roles_assign.html','wt')

#df = pd.read_csv('/media/IAM_REPORT/IAM_REPORT_role_assignment.csv')
#pd.set_option('display.max_colwidth', -1)
#pd.DataFrame({'a': [1, 2]}).to_html()

#print(df.to_html(index=False))

#END

#PYTHON_ARG="$1" python – <<END
#!/usr/bin/python

#import pandas as pd
#import sys
#import json
#sys.stdout = open('/media/IAM_REPORT/roles.html','wt')

#df = pd.read_csv('/media/IAM_REPORT/IAM_REPORT_role_policies.csv')
#pd.set_option('display.max_colwidth', -1)
#pd.DataFrame({'a': [1, 2]}).to_html()

#print(df.to_html(index=False))

#END
#PYTHON_ARG="$1" python – <<END
#!/usr/bin/python

#import pandas as pd
#import sys
#import json
#sys.stdout = open('/media/IAM_REPORT/user_policy.html','wt')

#df = pd.read_csv('/media/IAM_REPORT/IAM_REPORT_users_policies.csv')
#pd.set_option('display.max_colwidth', -1)
#pd.DataFrame({'a': [1, 2]}).to_html()

#print(df.to_html(index=False))

#END

#PYTHON_ARG="$1" python – <<END
#!/usr/bin/python

#import pandas as pd
#import sys
#import json
#sys.stdout = open('/media/IAM_REPORT/users.html','wt')

#df = pd.read_csv('/media/IAM_REPORT/IAM_REPORT_users.csv')
#pd.set_option('display.max_colwidth', -1)
#pd.DataFrame({'a': [1, 2]}).to_html()
#print(df.to_html(index=False))
#END

#Add labels to html files————————————————

#echo -e "
\n\t
IAM Groups

\n\t
\n$(cat #/media/IAM_REPORT/groups.html)” > /media/IAM_REPORT/groups.html

#echo -e ”
\n\t
IAM Group Policies

\n\t
\n$(cat /media/IAM_REPORT/group_policy.html)” > /media/IAM_REPORT/group_policy.html

#echo -e ”
\n\t
IAM Roles Assignment

\n\t
\n$(cat /media/IAM_REPORT/roles_assign.html)” > /media/IAM_REPORT/roles_assign.html

#echo -e ”
\n\t
IAM Roles

\n\t
\n$(cat /media/IAM_REPORT/roles.html)” > /media/IAM_REPORT/roles.html

#echo -e ”
\n\t
IAM User Policies

\n\t
\n$(cat /media/IAM_REPORT/user_policy.html)” > /media/IAM_REPORT/user_policy.html

#echo -e ”
\n\t
IAM Users

\n\t
\n$(cat /media/IAM_REPORT/users.html)” > /media/IAM_REPORT/users.html

#Find changes if files (if any)—————————————————-

#find /media/IAM_REPORT/*.txt -maxdepth 1 -size +0 -print -exec cat {} \; > /media/IAM_REPORT/changes.txt

#if [ -s /media/IAM_REPORT/changes.txt ]
#then
#sed -i ‘s%^\/media/IAM_REPORT/%%;s/\.txt$//’ /media/IAM_REPORT/changes.txt>/media/IAM_REPORT/changes.txt
#PYTHON_ARG=”$1″ python – <<END
#!/usr/bin/python

#contents = open("/media/IAM_REPORT/changes.txt","r")
#with open("/media/IAM_REPORT/changes.html", "w") as e:
#for lines in contents.readlines():
#e.write("

" + lines + "

\n”)
#END

#echo -e ”
\n\t
Changes

\n\t
\n$(cat /media/IAM_REPORT/changes.html)” > /media/IAM_REPORT/changes.html

#cat /media/IAM_REPORT/mailheader /media/IAM_REPORT/changes.html /media/IAM_REPORT/users.html /media/IAM_REPORT/user_policy.html /media/IAM_REPORT/groups.html /media/IAM_REPORT/group_policy.html /media/IAM_REPORT/roles.html /media/IAM_REPORT/roles_assign.html | sendmail -t

#else
#cat /media/IAM_REPORT/mailheader /media/IAM_REPORT/no_changes.html /media/IAM_REPORT/users.html /media/IAM_REPORT/user_policy.html /media/IAM_REPORT/groups.html /media/IAM_REPORT/group_policy.html /media/IAM_REPORT/roles.html /media/IAM_REPORT/roles_assign.html | sendmail -t
fi

mailheader file content:

To: user@company.com
From: someuser@example.com
Subject: AWS IAM Reports
Content-Type: text/html