Chuan Chuan Law

DevOps | Software Automation | Continuous Integration

How To Fix Passenger + Nginx Issue On Ubuntu 18

Error

Nginx will not start due to the following error on Ubuntu 18:

nginx: [emerg] unknown directive "passenger_enabled" in /etc/nginx/sites-enabled/default:25

nginx: configuration file /etc/nginx/nginx.conf test failed

Resolution

Install Passenger + Nginx module

apt-get install -y libnginx-mod-http-passenger

Add a line on top of /etc/nginx/nginx.conf

load_module /usr/lib/nginx/modules/ngx_http_passenger_module.so;

Restart Nginx

service nginx restart

Terraform – External Data source

This can be used to pull data from external.

In resource.tf

data “external” “download” {
program = [“${path.module}/download.sh”, “${var.filename_zip}”]
}

In download.sh

!/bin/bash
FILE=$1
filename=$(curl -O $FILE)
echo -n “{\”Downloaded zip file\”:\”${filename}\”}”

In vars.tf

variable “filename” {
description = “Filename for lambda zip file”
default = “externalfile.zip”
}

Transitioning From SDET To DevOps

I started my career as a junior QA to Software Engineer In Test in Australia before a slight change in my IT career into a DevOps engineer in NYC.

Introduction

I would like to share about the technical skills that could be transferred from SDET and what extra skills that need to be picked up in order to be a DevOps engineer.

Transferable skills

  • Programming/automation

The programming skills from writing automated tests will be helpful in DevOps as part of the job requires programming to automate processes.

SDET will use more programming language such as Java, Ruby, etc but DevOps will use more of shell and bash script.

  • CI/CD or Deployment

SDET’s involvement in deployment pipeline automation with tools like Jenkins is definitely a core part of a DevOps engineer. The only difference is that a SDET will usually use Jenkins to set up the automated test build or integrate it into the CI/CD pipeline.

DevOps’s involvement will be helping the development team with building the entire pipeline from compilation till deployment

  • Tools

SDET uses lots of Open Source tools such as Selenium, Cucumber to develop test automation frameworks.

DevOps’s job leverage a lot on tools as well, but different sets. We will discuss more about this in the “Skills to be picked up” section.

  • General computing/system knowledge

General computer knowledge of operation systems etc will be used but more in depth in a DevOps’s role.

Skills to be picked up

  • New automation tool – configuration management tools

Configuration management tools such as Ansible and Puppet are a key part for DevOps to automate deployment and server configuration.

  • New tools

A major portion of a DevOp’s tasks involves installation, configuration, setting up, upgrading and managing a bunch of tools used by the development tools. A list of it include:

  1. Jenkins
  2. Git
  3. Artifactory
  4. Docker
  5. Kubernetes
  6. Nginx
  7. Consul
  8. Hashicorp
  9. Elastic Search
  10. Sonarqube
  11. New Relic
  12. Datadog
  • More in-depth operating system

A lot of DevOps’s work involve in troubleshooting system issues. Therefore, knowledge of the operation system in use such as Linux, Ubuntu, RedHat, etc is very important.

  • Cloud

Working with data centers or more applicable nowadays are the cloud such as AWS and Google Cloud.

That will include usage of related tools such as Terraform.

Conclusion

Both SDET and DevOps are exciting jobs. I would not recommend one over the other. However for those who want to transition, it’s definitely not a difficult task. There are transferable skills which you could leverage but also new skills to learn. Learning new skills is unavoidable in the IT world anyway as technology keeps on evolving.

Ansible: Install Different Versions Locally

If we want to use different versions of Ansible:

pip install virtualenv

virtualenv ansible194

ansible194/bin/pip install ansible==1.9.4

source ansible194/bin/activate

General Guideline To Take While Doing System Upgrade

Upgrading system and applications are important and major part of a DevOps role. Below are the general guideline to take while performing an upgrade.

Take a snapshot

Take a snapshot, create a backup, or image before an upgrade is very important in case we need to rollback.

Practice the upgrade process

Work out the process on a development environment, staging or test box before doing the actual upgrade, especially if it is for production environment.

Automate as much as you can

Try to automate the upgrade as much as you can, so we can repeat the process on other boxes or environment.

Make notes

Take notes especially on things that we cannot automate .

Have a runbook

List all the automation and manual steps to run in a runbook, which we can follow during the upgrade process

Prepare a rollback strategy

Ensure we know the steps to rollback if needed

Have peer reviews

Get peers or seniors to review the automated steps and runbook.

Have a helpline or support ready

Have a helpline or support ready during upgrade in case things go south during upgrade

How To Fix 0% Coverage On SonarCube

  • We need to make sure the Jacoco – Overall Coverage Summary is showing up test coverage result
  • Ensure the maven-sureflire-plugin is configured

<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-surefire-plugin</artifactId>
<version>${dep.apache.maven.test.plugins}</version>
<configuration>
<argLine>${surefireArgLine}</argLine>
<includes>
<include>**/*Test.java</include>
</includes>
<excludes>
<exclude>**/*IntegrationTest.java</exclude>
</excludes>
</configuration>
</plugin>

  • Ensure the jacoco-maven-plugin is configured

<plugin>
<groupId>org.jacoco</groupId>
<artifactId>jacoco-maven-plugin</artifactId>
<version>${jacoco-maven-plugin.version}</version>
<executions>
<execution>
<id>pre-unit-test</id>
<phase>initialize</phase>
<goals>
<goal>prepare-agent</goal>
</goals>
<configuration>
<propertyName>surefireArgLine</propertyName>
</configuration>
</execution>
<execution>
<id>pre-int-test</id>
<phase>pre-integration-test</phase>
<goals>
<goal>prepare-agent-integration</goal>
</goals>
</execution>
<execution>
<id>report-generation</id>
<phase>verify</phase>
<goals>
<goal>report</goal>
<goal>report-integration</goal>
</goals>
</execution>
</executions>
</plugin>

Ref: https://docs.sonarqube.org/display/PLUG/Usage+of+JaCoCo+with+SonarJava

Ansible – How to fix Failed to import docker-py – No module named websocket. Try `pip install docker-py`

Set the path of Python site-packages in your playbook:

environment:
PYTHONPATH: “{{ lookup(‘env’,’PYTHONPATH’) }}:/Users/claw/Library/Python/2.7/lib/python/site-packages:/Users/claw/Library/Python/2.7/lib/python/site-packages”

How To Migrate Elastic Search Cluster To New Cluster

Introduction

There are times when we will choose to set up a new cluster and migrate the data instead of updating existing one especially when when the version that we want to upgrade is many versions ahead of the current one. Thus, the risk associated with the upgrade will be higher and downtime of existing cluster due to the upgrade will also be longer.

Setup The New Elk Cluster

Install the new versions in the following sequence:

  • Elasticsearch
  • Kibana
  • Logstash

The Elastic Search official guide contains comprehensive guide on the installation, so this blog will talk about personal experience and problems encountered which are not part of the installation guide.

Elastic Search

The main thing here is to do Snapshot and Restore of current Elastic Search cluster to the new one.

  • Register a backup with the current ES. In this blog, we use S3 repository

curl -X PUT “http://oldelasticsearch:9200/_snapshot/s3_repository?verify=false” -H ‘Content-Type: application/json’ -d’
{
“type”: “s3”,
“settings”: {
“bucket”: “es-snapshot”,
“region”: “us-east-1”
}
}

  • Snapshot to S3

curl -XPUT “http://oldelasticsearch:9200/_snapshot/s3_repository/snap1?pretty?wait_for_completion=true

  • Register the backup on the new cluster

curl -X PUT “http://newelasticsearch:9200/_snapshot/s3_repository?verify=false” -H ‘Content-Type: application/json’ -d’
{
“type”: “s3”,
“settings”: {
“bucket”: “es-snapshot”,
“region”: “us-east-1”
}
}

  • Restore from S3 bucket on the new ES cluster

curl -X POST “http://newelasticsearch:9200/_snapshot/s3_repository/snap1/_restore

Kibana

The hip cups that you will see from the newly installed Kibana will be due to some conflicting indices from new and old cluster.

To solve this, do the following:

  • Close the kibana index

curl -X POST “http://newelasticsearch:9200/.kibana/_close

  • Delete the kibana index

curl -X DELETE “http://newelasticsearch:9200/.kibana

  • Restore kibana index from S3 (old cluster)

curl -X POST “http://newelasticsearch:9200/_snapshot/s3_repository/snap1/_restore” -H ‘Content-Type: application/json’ -d’
{
“indices”: “.kibana”,
“ignore_unavailable”: true,
“include_global_state”: true
}

  • Open the kibana index again

curl -X POST “http://newelasticsearch:9200/.kibana/_open

  • Restart kibana

Another issue with Kibana setup will be the Logtrial plugin. The plugin version needs to match exactly the Kibana version, thus we will need to do some manual hacks.

Below are the hacks in Ansible script:

  • Download logtrial

– name: download logtrail
get_url:
url: https://github.com/sivasamyk/logtrail/releases/download/v0.1.23/logtrail-5.6.5-0.1.23.zip
dest: /tmp

  • Unzip logtrial

– name: unzip logtrail
unarchive:
src: /tmp/logtrail-5.6.5-0.1.23.zip
dest: /tmp

  • Modify the kibana version in package.json

– name: modify the kibana version in package.json
lineinfile:
path: /tmp/kibana/logtrail/package.json
regexp: ‘”version”: “5.6.5”‘
line: ‘”version”: “5.6.15”‘

  • Zip it back

– name: zip logtrial back
archive:
path: /tmp/kibana
dest: /usr/share/kibana/bin/logtrail-5.6.5-0.1.23.zip
format: zip
mode: 0664

  • Install the modified logrial

– name: install modified logtrial
shell: ./kibana-plugin install file:///usr/share/kibana/bin/logtrail- 5.6.5-0.1.23.zip
chdir=/usr/share/kibana/bin

How To Solve DNS Issues in Ubuntu 18.04

Introduction

Ubuntu 18.04 Bionic has DNS bug where it is not able to resolved DNS.

Solution

We can do the following hack to solve it:

sudo rm -f /etc/resolv.conf
sudo ln -s /run/systemd/resolve/resolv.conf /etc/resolv.conf
reboot

Jenkins – How To Manually Upgrade Jenkins Plugin To Specific Versions

Introduction

Jenkins Plugin Manager will enable plugin installation of the latest version. Sometimes we might want to install plugin of a specific version, instead of the latest. To do that, we can do the following:

Steps

  • Older versions of plugins can be found here: https://updates.jenkins.io/download/plugins/
  • The plugins are in .hpi file extensions
  • Download the version you want and put it on the Jenkins server
  • Plugins are stored in /var/lib/jenkins
  • Backup – you might want to backup the current version of the plugins before upgrading. Each plugin has a directory and a .jpi file.

For eg: datadog plugin

mkdir datadog_backup

mv datadog* /datadog_backup

  • Place the .hpi file in /var/lib/jenkins
  • Restart Jenkins

service jenkins restart

  • A directory with the plugin name and a .jpi file will be created upon restart
  • Check if the plugin with the correct version appears on Jenkins website Manage Jenkins->Plugin Manager
  • If not, check the log on the server in /var/log/jenkins 
  • Log is usually called jenkins.log
  • Log will show the plugin installation failure as SEVERE
  • Usually this is due to version dependencies of other plugins as named in the log.

For eg:

SEVERE: Failed Loading plugin sauce-ondemand
java.io.IOException: Dependency workflow-job (1.15), workflow-cps (1.15), workflow-basic-steps (1.15), workflow-step-api (1.15) doesn’t exist

  • Download and install the missing dependencies plugin via the same method as above
  • Upon successful installation, you will see the plugin with the right version appearing in Manage Jenkins->Plugin Manager

« Older posts

© 2019 Chuan Chuan Law

Theme by Anders NorenUp ↑