There are times when we will choose to set up a new cluster and migrate the data instead of updating existing one especially when when the version that we want to upgrade is many versions ahead of the current one. Thus, the risk associated with the upgrade will be higher and downtime of existing cluster due to the upgrade will also be longer.
Setup The New Elk Cluster
Install the new versions in the following sequence:
The Elastic Search official guide contains comprehensive guide on the installation, so this blog will talk about personal experience and problems encountered which are not part of the installation guide.
The main thing here is to do Snapshot and Restore of current Elastic Search cluster to the new one.
- Register a backup with the current ES. In this blog, we use S3 repository
curl -X PUT “http://oldelasticsearch:9200/_snapshot/s3_repository?verify=false” -H ‘Content-Type: application/json’ -d’
- Snapshot to S3
curl -XPUT “http://oldelasticsearch:9200/_snapshot/s3_repository/snap1?pretty?wait_for_completion=true“
- Register the backup on the new cluster
curl -X PUT “http://newelasticsearch:9200/_snapshot/s3_repository?verify=false” -H ‘Content-Type: application/json’ -d’
- Restore from S3 bucket on the new ES cluster
curl -X POST “http://newelasticsearch:9200/_snapshot/s3_repository/snap1/_restore“
The hip cups that you will see from the newly installed Kibana will be due to some conflicting indices from new and old cluster.
To solve this, do the following:
- Close the kibana index
curl -X POST “http://newelasticsearch:9200/.kibana/_close“
- Delete the kibana index
curl -X DELETE “http://newelasticsearch:9200/.kibana“
- Restore kibana index from S3 (old cluster)
curl -X POST “http://newelasticsearch:9200/_snapshot/s3_repository/snap1/_restore” -H ‘Content-Type: application/json’ -d’
- Open the kibana index again
curl -X POST “http://newelasticsearch:9200/.kibana/_open“
- Restart kibana
Another issue with Kibana setup will be the Logtrial plugin. The plugin version needs to match exactly the Kibana version, thus we will need to do some manual hacks.
Below are the hacks in Ansible script:
- Download logtrial
– name: download logtrail
- Unzip logtrial
– name: unzip logtrail
- Modify the kibana version in package.json
– name: modify the kibana version in package.json
regexp: ‘”version”: “5.6.5”‘
line: ‘”version”: “5.6.15”‘
- Zip it back
– name: zip logtrial back
- Install the modified logrial
– name: install modified logtrial
shell: ./kibana-plugin install file:///usr/share/kibana/bin/logtrail- 5.6.5-0.1.23.zip