Archive for December, 2018

This script will search for any email¬† where subject starts with “Darktrace” subject and if it find any will move it to Processed folder, all other emails will be moved to backup folder and all emails in inbox will be deleted (to mark emails as processed), also it will extract email body in human readable format.

#!/usr/bin/python3

from email.message import EmailMessage
import email
import imaplib
import re
import sys
import logging
import base64
import email.parser
import html2text
import requests
import json
import argparse

parser = argparse.ArgumentParser()
parser.add_argument('-mpass', '-mailbox_password', dest = 'mailbox_password', help = 'mailbox password.')
args = parser.parse_args()

user = 'someuser@company.com'
mailbox_password = args.mailbox_password

def get_email_body(body):

       if body.is_multipart():
         for payload in body.get_payload():
             print('To:\t\t', body['To'])
             print('From:\t', body['From'])
             print('Subject:', body['Subject'])
             print('Date:\t',body['Date'])
             for part in body.walk():
               if (part.get_content_type() == 'text/plain') and (part.get('Content-Disposition') is None):
                output = part.get_payload()
       else:
         print('To:\t\t', body['To'])
         print('From:\t', body['From'])
         print('Subject:', body['Subject'])
         print('Date:\t', body['Date'])
         print('Thread-Index:\t', body['Thread-Index'])
         text = f"{body.get_payload(decode=True)}"
         html = text.replace("b'", "")
         h = html2text.HTML2Text()
         h.ignore_links = True
         output = (h.handle(f'''{html}''').replace("\\r\\n", ""))
         output = output.replace("'", "")
         # output in one line
         #output = output.replace('\n'," ")
         output = output.replace('*', "")
         return output

def clear_inbox(conn, dest_folder):
    output=[]
    result = conn.uid('COPY', emailid, dest_folder)
    output.append(result)
    if result[0] == 'OK':
     result = mov, data = conn.uid('STORE',emailid, '+FLAGS', '(\Deleted Items)')
     conn.expunge()

conn = imaplib.IMAP4_SSL("outlook.office365.com")
conn.login(user,mailbox_password)
conn.select("Inbox")

try:

  resp, items = conn.uid("search",None, 'All')
  items = items[0].split()

  for emailid in items:
   resp, data = conn.uid("fetch",emailid, "(RFC822)")
   if resp == 'OK':
     email_body = data[0][1].decode('utf-8')
     email_message = email.message_from_string(email_body)
     subject = email_message["Subject"]
     if subject.lower().startswith('Darktrace'.lower()):
         output = get_email_body(email_message)

         #do some task
         # move emails to Processed folder and clear Inbox
         clear_inbox(conn, "Processed")
     else:
        clear_inbox(conn, "backup")

except IndexError:
     print("No new email")

conn.close()
conn.logout()

Second method is using BeatiufoulSoup HTML parser

from bs4 import BeautifulSoup

def print_payload(message):
    print('')
    if message.is_multipart():
        for payload in message.get_payload():
            print_payload(payload)
    else:
         #print (message.get_payload())
         for part in message.walk():
             if part.get_content_type():
                 body = str(part.get_payload())
                 soup = BeautifulSoup(body,features="html.parser")
                 paragraphs = soup.find_all('p')
                 for paragraph in paragraphs:
                     print(paragraph.text.encode('utf-8').decode('ascii', 'ignore'))

#............................

mail = email.message_from_string(email_body)
print_payload(mail)
Advertisements

Rundeck – Nginx as SSL reverse proxy

Posted: December 14, 2018 in RunDeck

In this post we installed Rundeck, in this one we’ll access to Rundeck by typing https://FQDN, without specifying port number

rundeck.test.com is specified as name in rundeck properties before start nginx is installed.

Install nginx:

yum install nginx

Creating Self signed SSL certificate:

cd /etc/nginx
sudo openssl req -x509 -nodes -days 365 -newkey rsa:2048 -keyout /etc/nginx/rundeck.key -out /etc/nginx/rundeck.crt

Generating a 2048 bit RSA private key
.....................................+++
.........................................................+++
writing new private key to '/etc/nginx/rundeck.key'
-----
You are about to be asked to enter information that will be incorporated
into your certificate request.
What you are about to enter is what is called a Distinguished Name or a DN.
There are quite a few fields but you can leave some blank
For some fields there will be a default value,
If you enter '.', the field will be left blank.
-----
Country Name (2 letter code) [XX]:SR
State or Province Name (full name) []:
Locality Name (eg, city) [Default City]:Zemoon
Organization Name (eg, company) [Default Company Ltd]:
Organizational Unit Name (eg, section) []:
Common Name (eg, your name or your server's hostname) []:rundeck.test.com
Email Address []:

Make sure Rundeck FQDN (submitted during SSL certificate creation) is resolvable

Capture.PNG

cat /etc/nginx/conf.d/rundeck.conf
server {
listen 443 ssl;
server_name rundeck.test.com; # Replace it with your Subdomain

access_log /var/log/nginx/rundeck.yallalabs.local.access.log;

ssl_certificate /etc/nginx/rundeck.crt;
ssl_certificate_key /etc/nginx/rundeck.key;

ssl on;
ssl_session_cache builtin:1000 shared:SSL:10m;
ssl_protocols TLSv1 TLSv1.1 TLSv1.2;
ssl_ciphers HIGH:!aNULL:!eNULL:!EXPORT:!CAMELLIA:!DES:!MD5:!PSK:!RC4;
ssl_prefer_server_ciphers on;

location / {
#add_header Front-End-Https on;
proxy_set_header Host $host;
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header X-Forwarded-Proto $scheme;

proxy_pass http://rundeck.test.com:4440;
proxy_read_timeout 90;

proxy_redirect http://rundeck.test.com:4440 https://rundeck.test.com; # Replace it with your Subdomain
}

}

server {
listen 80;
server_name rundeck.test.com; # Replace it with your Subdomain
return 301 https://$host$request_uri;
}
cat /etc/rundeck/framework.properties
# framework.properties -
#

# ----------------------------------------------------------------
# Rundeck server connection information
# ----------------------------------------------------------------

framework.server.name = rundeck.test.com
framework.server.hostname = rundeck.test.com
framework.server.port = 4440
framework.server.url = https://rundeck.test.com

# ----------------------------------------------------------------
# Installation locations
# ----------------------------------------------------------------

rdeck.base=/var/lib/rundeck

framework.projects.dir=/var/rundeck/projects
framework.etc.dir=/etc/rundeck
framework.var.dir=/var/lib/rundeck/var
framework.tmp.dir=/var/lib/rundeck/var/tmp
framework.logs.dir=/var/lib/rundeck/logs
framework.libext.dir=/var/lib/rundeck/libext

# ----------------------------------------------------------------
# SSH defaults for node executor and file copier
# ----------------------------------------------------------------

framework.ssh.keypath = /var/lib/rundeck/.ssh/id_rsa
framework.ssh.user = rundeck

# ssh connection timeout after a specified number of milliseconds.
# "0" value means wait forever.
framework.ssh.timeout = 0
# ----------------------------------------------------------------
# Auto generated server UUID: 391d3428-9a67-44d9-9d55-9427b52387c0
# ----------------------------------------------------------------
rundeck.server.uuid = 391d3428-9a67-44d9-9d55-9427b52387c0
cat /etc/rundeck/rundeck-config.properties
#loglevel.default is the default log level for jobs: ERROR,WARN,INFO,VERBOSE,DEBUG
loglevel.default=INFO
rdeck.base=/var/lib/rundeck

#rss.enabled if set to true enables RSS feeds that are public (non-authenticated)
rss.enabled=false
# change hostname here
grails.serverURL=https://rundeck.test.com
dataSource.dbCreate = update
dataSource.url = jdbc:h2:file:/var/lib/rundeck/data/rundeckdb;MVCC=true
rundeck.log4j.config.file = /etc/rundeck/log4j.properties

Restart nginx and rundeckd

systemctl restart nginx && systemctl restart rundeckd

If all is fine, rundecl should be accessible from https//rundeck.test.com

In this post we configured Windows instances to go to hibernation. In this one we’ll modify Node.JS script to shut down EC2 instances if no one is connected via SSH connections.

Setting SSH idle timeout settings

cat /etc/profile.d/ssh-timeout.sh
export TMOUT=900
readonly TMOUT

Reboot

 

Installing SSM Agent on CentOS 7

yum install -y https://s3.amazonaws.com/ec2-downloads-windows/SSMAgent/latest/linux_amd64/amazon-ssm-agent.rpm
systemctl status amazon-ssm-agent
systemctl enable amazon-ssm-agent
systemctl start amazon-ssm-agent

How to create SSM role and how to assign it to instance check out here. and here

Create following tags:

 

Capture

 

Change only code in auto_stop/modules/control/index.js

// Import Dependencies
let AWS = require('aws-sdk');
AWS.config.region = "eu-west-1";

module.exports.getInstanceIds = () => {
return new Promise(
(resolve, reject) => {
let ec2 = new AWS.EC2();
let params = {
Filters: [
{ Name: "instance-state-name",
Values: ["running"]
},
{
Name: "tag:Auto_Stop_Schedule",
Values: ["1"]
},
{
Name: "tag:Auto_Stop_Enabled",
Values: ["True","true", "Yes", "yes"]
},
{
Name: "tag:Auto_Stop_Type",
Values: ["Linux","linux"]
}
]
};

ec2.describeInstances(params, (err, data) => {
if (err) reject(err);

let instanceIds = [];
let reservations = "";
try {
reservations = data.Reservations;
}
catch(err) {
reject(err);
}
if(Array.isArray(reservations)) {
reservations.forEach((reservation) => {
reservation.Instances.forEach((instance) =>{
instanceIds.push(instance.InstanceId);
});
});
if(instanceIds.length >= 1) {
resolve(
{
"InstanceIds": instanceIds
}
);
}
else {
console.log("[Info] getInstanceIds: No instances found.");
resolve();
}
}
else {
reject(new Error("[Error] getInstanceIds: Reservations is not an array."));
}
});
}
);
};

module.exports.shutdownInstances = (controlObj) => {
return new Promise(
(resolve, reject) => {
let ssm = new AWS.SSM();

let instanceIds = controlObj.InstanceIds;
instanceIds.forEach((i) => {
let ssmParams = {
InstanceIds: [i],
DocumentName: "AWS-RunShellScript",
Parameters: {
"workingDirectory":[""],
"executionTimeout":["300"],
"commands":["#!/bin/bash","LOGFOLDER=\"/var/log/ssh_check\"","LOGFILE=\"auto_stop_activity.log\"","","# Check if the ssh_check Log dir exists. If not, create it.","[ -d $LOGFOLDER ] || mkdir -p $LOGFOLDER","","if [[ \"$(/usr/bin/w | wc -l)\" -gt 2 ]];"," then"," echo \"$(date) >>> Live SSH session detected\" "," echo \"$(date) >>> Live SSH session detected\" >> \"$LOGFOLDER/$LOGFILE\""," else"," # If no active SSH sessions are found, shutdown the instance."," echo \"$(date) >>> No running sessions detected\" "," echo \"$(date) >>> Shutting down...\"",""," echo \"$(date) >>> No running sessions detected\" >> \"$LOGFOLDER/$LOGFILE\" "," echo \"$(date) >>> Shutting down...\" >> \"$LOGFOLDER/$LOGFILE\" "," shutdown -P -t 30 > /dev/null 2>&1"," exit 0"," fi"]
},
MaxErrors: "0",
TimeoutSeconds: 120
}

// Hibernate instances
ssm.sendCommand(ssmParams, function(err, data) {
if (err) {
console.log(`Error: ${err}`);
}
console.log("Command Sent");
console.log(`Instance to shutdown: ${instanceIds}`);
});
});
}
);
};

 

In this example 2 AD groups will be used

Capture

  • BitBucket.Admin (System Admin,Admin,Project creator,Bitbucket user)
  • BitBucket.User (Project creator,Bitbucket user)

Configuring Active Directory integration:

User directory-add directory-Microsoft Active Directory

Server Settings

Name: Active Directory server

Directory Type: Microsoft Active Directory

Hostname: example.com

Port: 389

Username: ldapuser for searching AD

LDAP Schema

BaseDN: DC=example,DC=com

LDAP Permissions:

Read only

Advanced settings

Check Enable nested groups (leave other settings)

User Schema Settings

Show only members of BitBucket.Admin/Bitbucket.User AD groups

UserObject class: user

UserObjectFilter:(&(objectCategory=Person)(sAMAccountName=*)(|(memberOf:1.2.840.113556.1.4.1941:=cn=BitBucket.Admin,OU=Groups,DC=example,DC=com (memberOf:1.2.840.113556.1.4.1941:=cn=BitBucket.User,OU=Groups,DC=example,DC=com)))

UserNameAtribute: sAMAccountName

UserName RDN Atribute: cn

User First Attribute Name: givenName

User Last Name attribute: sn

User Display Name Attribute: displayName

User Email Attribute: mail

User Password Attribute: unicodePwd

User Unique Attribute: objectGUID

 

Group Schema settings

 

Group Object Class: group

Group Obect Filter: (&(objectCategory=Group)(cn=BitBucket*))

Group Name attribute: cn

Group Description Attribute: description

Configuring group access and roles

In BitBucket go to Global permissions-add groups and assign roles