Introduction

There are times when we will choose to set up a new cluster and migrate the data instead of updating existing one especially when when the version that we want to upgrade is many versions ahead of the current one. Thus, the risk associated with the upgrade will be higher and downtime of existing cluster due to the upgrade will also be longer.

Setup The New Elk Cluster

Install the new versions in the following sequence:

  • Elasticsearch
  • Kibana
  • Logstash

The Elastic Search official guide contains comprehensive guide on the installation, so this blog will talk about personal experience and problems encountered which are not part of the installation guide.

Elastic Search

The main thing here is to do Snapshot and Restore of current Elastic Search cluster to the new one.

  • Register a backup with the current ES. In this blog, we use S3 repository

curl -X PUT “http://oldelasticsearch:9200/_snapshot/s3_repository?verify=false” -H ‘Content-Type: application/json’ -d’
{
“type”: “s3”,
“settings”: {
“bucket”: “es-snapshot”,
“region”: “us-east-1”
}
}

  • Snapshot to S3

curl -XPUT “http://oldelasticsearch:9200/_snapshot/s3_repository/snap1?pretty?wait_for_completion=true“

  • Register the backup on the new cluster

curl -X PUT “http://newelasticsearch:9200/_snapshot/s3_repository?verify=false” -H ‘Content-Type: application/json’ -d’
{
“type”: “s3”,
“settings”: {
“bucket”: “es-snapshot”,
“region”: “us-east-1”
}
}
‘

  • Restore from S3 bucket on the new ES cluster

curl -X POST “http://newelasticsearch:9200/_snapshot/s3_repository/snap1/_restore“

Kibana

The hip cups that you will see from the newly installed Kibana will be due to some conflicting indices from new and old cluster.

To solve this, do the following:

  • Close the kibana index

curl -X POST “http://newelasticsearch:9200/.kibana/_close“

  • Delete the kibana index

curl -X DELETE “http://newelasticsearch:9200/.kibana“

  • Restore kibana index from S3 (old cluster)

curl -X POST “http://newelasticsearch:9200/_snapshot/s3_repository/snap1/_restore” -H ‘Content-Type: application/json’ -d’
{
“indices”: “.kibana”,
“ignore_unavailable”: true,
“include_global_state”: true
}
‘

  • Open the kibana index again

curl -X POST “http://newelasticsearch:9200/.kibana/_open“

  • Restart kibana

Another issue with Kibana setup will be the Logtrial plugin. The plugin version needs to match exactly the Kibana version, thus we will need to do some manual hacks.

Below are the hacks in Ansible script:

  • Download logtrial

– name: download logtrail
get_url:
url: https://github.com/sivasamyk/logtrail/releases/download/v0.1.23/logtrail-5.6.5-0.1.23.zip
dest: /tmp

  • Unzip logtrial

– name: unzip logtrail
unarchive:
src: /tmp/logtrail-5.6.5-0.1.23.zip
dest: /tmp

  • Modify the kibana version in package.json

– name: modify the kibana version in package.json
lineinfile:
path: /tmp/kibana/logtrail/package.json
regexp: ‘”version”: “5.6.5”‘
line: ‘”version”: “5.6.15”‘

  • Zip it back

– name: zip logtrial back
archive:
path: /tmp/kibana
dest: /usr/share/kibana/bin/logtrail-5.6.5-0.1.23.zip
format: zip
mode: 0664

  • Install the modified logrial

– name: install modified logtrial
shell: ./kibana-plugin install file:///usr/share/kibana/bin/logtrail- 5.6.5-0.1.23.zip
chdir=/usr/share/kibana/bin