DevOps | Software Automation | Continuous Integration

Category: AWS

Top 10 DevOps Tools Or Services Used

Below are the list of top 10 tools I use on a daily basis in my job

  1. Configuration management tool

Configuration management tool – Ansible takes up 80% to 90% of my daily life. All servers provisioning, software installation and management are automated using it. Automation with configuration management tool allows repetition on multiple servers and avoids human error.

2. Jenkins

All software compilation, build and deploys are automated on Jenkins. Includes, writing Jenkins pipeline, installing, upgrading Jenkins and its plugins.

3. AWS

This is where all the servers and resources are. EC2, DNS and other services like Elastic Search etc.

4. Terraform

This is used in to provision the services and resources in AWS. I view it as the configuration management tool of AWS that allows repetition and eliminates human error.

5. Elastic Search

This is where all the logs go to. Maintenance work such as automating snapshot, backup and curator clean up are part of the job.

6. Operating system

System administrating work on operating systems like Ubuntu. Diagnosing, troubleshooting issues, installing and upgrading packages.

7. Nginx

Load balancing for applications and services.

8. Docker

Containerization has become important these days due to cost savings, therefore most servers are shifted towards being provisioned in Docker.

9. Monitoring tools

Integrating monitoring software into the applications, services and databases using services such as New Relic, AppDynamics and Datadog.

10. Hashicorp Vault

Used to store all secrets and sensitive information of applications.

AWS Elastic Search Snapshot & Restore

Create s3 repo

  • Create a s3 repo in AWS
  • Create a policy and attach it to a role

{
“Version”: “2012-10-17”,
“Statement”: [
{
“Action”: [
“s3:ListBucket”
],
“Effect”: “Allow”,
“Resource”: [
“arn:aws:s3:::snapshot_s3_repositorty”
]
},
{
“Action”: [
“s3:GetObject”,
“s3:PutObject”,
“s3:DeleteObject”
],
“Effect”: “Allow”,
“Resource”: [
“arn:aws:s3:::snapshot_s3_repository/*”
]
}
]
}

  • Establish a trust relationship for the snapshot role to allow access from Elastic Search service

{
“Version”: “2012-10-17”,
“Statement”: [
{
“Sid”: “”,
“Effect”: “Allow”,
“Principal”: {
“Service”: “es.amazonaws.com”
},
“Action”: “sts:AssumeRole”
}
]
}

Create a AWS user

  • Allow the user to use the snapshot role to snapshot/restore onto s3 and bunch of ES actions

{
“Version”: “2012-10-17”,
“Statement”: [
{
“Effect”: “Allow”,
“Action”: “iam:PassRole”,
“Resource”: “arn:aws:iam::123456789:role/elasticsearch_snapshot_role”
},
{
“Effect”: “Allow”,
“Action”: [
“es:ESHttpPut”,
“es:ESHttpGet”,
“es:ESHttpPost”,
“es:ESHttpDelete”
],
“Resource”: “arn:aws:es:us-east-1:123456789:domain/mydomain/*”
}
]
}

Register the snapshot repo

  • This is a shell command inside a Jenkins job
  • SECRET and ACCESS are the credentials for the AWS user created above and injected as passwords into the environment variables

!/bin/sh
pip install requests-aws4auth
pip install boto3
cat << EOF >> /home/jenkins/.aws/credentials
[default]
region=us-east-1
aws_access_key_id = $SECRET
aws_secret_access_key = $ACCESS
EOF
cat /home/jenkins/.aws/credentials
cat << ‘EOF’ >> register_es_snapshot.py
import boto3
import requests
from requests_aws4auth import AWS4Auth
host = ‘https://host.es.amazonaws.com/’
region = ‘us-east-1’
service = ‘es’
credentials = boto3.Session().get_credentials()
awsauth = AWS4Auth(credentials.access_key, credentials.secret_key, region, service, session_token=credentials.token)
Register repository
path = ‘_snapshot/my_es_snapshot’ # the Elasticsearch API endpoint
url = host + path
payload = {
“type”: “s3”,
“settings”: {
“bucket”: “snapshot_s3_repositorty”,
“region”: “us-east-1”,
“role_arn”: “arn:aws:iam::123456789:role/elasticsearch_snapshot_role”
}
}
headers = {“Content-Type”: “application/json”}
r = requests.put(url, auth=awsauth, json=payload, headers=headers)
print(r.status_code)
print(r.text)
EOF
chmod 755 register_es_snapshot.py
python register_es_snapshot.py

Creating a snapshot or backup

  • An error will occur if we use the same snapshot name, so $now is used to make snapshot name unique

!/bin/sh
pip install requests-aws4auth
pip install boto3
cat << EOF >> /home/jenkins/.aws/credentials
[default]
region=us-east-1
aws_access_key_id = $SECRET
aws_secret_access_key = $ACCESS
EOF
cat /home/jenkins/.aws/credentials
cat << ‘EOF’ >> backup_es_snapshot.py
import boto3
import requests
from requests_aws4auth import AWS4Auth
host = ‘https://host.us-east-1.es.amazonaws.com/’
region = ‘us-east-1’
service = ‘es’
credentials = boto3.Session().get_credentials()
awsauth = AWS4Auth(credentials.access_key, credentials.secret_key, region, service, session_token=credentials.token)
headers = {“Content-Type”: “application/json”}
Take snapshot
path = ‘_snapshot/my_es_snapshot/my_snapshot_$now’
url = host + path
r = requests.get(url, auth=awsauth)
print(r.status_code)
print(r.text)
EOF
chmod 755 backup_es_snapshot.py
python backup_es_snapshot.py

Restore single index

  • INDEX is a string parameter passed into the Jenkins job. For e.g: “logstash-2020.07.11”

!/bin/sh
pip install requests-aws4auth
pip install boto3
cat << EOF >> /home/jenkins/.aws/credentials
[default]
region=us-east-1
aws_access_key_id = $SECRET
aws_secret_access_key = $ACCESS
EOF
cat /home/jenkins/.aws/credentials
cat << EOF >> restore_single_index_es_snapshot.py
import boto3
import requests
from requests_aws4auth import AWS4Auth
host = ‘https://host.es.amazonaws.com/’
region = ‘us-east-1’
service = ‘es’
credentials = boto3.Session().get_credentials()
awsauth = AWS4Auth(credentials.access_key, credentials.secret_key, region, service, session_token=credentials.token)
headers = {“Content-Type”: “application/json”}
Restore snapshot (one index)
path = ‘_snapshot/my_es_snapshot/my_snapshot/_restore’
url = host + path
payload = {“indices”: “$INDEX”}
headers = {“Content-Type”: “application/json”}
r = requests.post(url, auth=awsauth, json=payload, headers=headers)
print(r.status_code)
print(r.text)
EOF
chmod 755 restore_single_index_es_snapshot.py
python restore_single_index_es_snapshot.py

Restore all indices

!/bin/sh
pip install requests-aws4auth
pip install boto3
cat << EOF >> /home/jenkins/.aws/credentials
[default]
region=us-east-1
aws_access_key_id = $SECRET
aws_secret_access_key = $ACCESS
EOF
cat /home/jenkins/.aws/credentials
cat << EOF >> restore_all_indices_es_snapshot.py
import boto3
import requests
from requests_aws4auth import AWS4Auth
host = ‘https://host.es.amazonaws.com/’
region = ‘us-east-1’
service = ‘es’
credentials = boto3.Session().get_credentials()
awsauth = AWS4Auth(credentials.access_key, credentials.secret_key, region, service, session_token=credentials.token)
headers = {“Content-Type”: “application/json”}
# Restore snapshot (all indices except Kibana and fine-grained access control)
path = ‘_snapshot/snapshot/my_es_snapshot/my_snapshot/_restore’
url = host + path
payload = {
“indices”: “-.kibana*,-.opendistro_security”,
“include_global_state”: False
}
headers = {“Content-Type”: “application/json”}
r = requests.post(url, auth=awsauth, json=payload, headers=headers)
print(r.status_code)
print(r.text)
EOF
chmod 755 restore_all_indices_es_snapshot.py
python restore_all_indices_es_snapshot.py

Solving Peer Not Authenticated Issue for Maven repository on AWS

Below the problem I got while migrating the Jenkins slave jobs over to AWS:

 

 

error2

The way I solve it is to get a copy of compiled dependencies on the AWS instance and build a AMI image out of it. So, the AWS instance will already have a copy of all needed dependencies, and will only check out the test code and run the test without have to worry about downloading and compiling its dependencies.

 

© 2020 Chuan Chuan Law

Theme by Anders NorenUp ↑