AWS Elasticsearch creating index snapshots periodically
We needed to create AWS ES index snapshots periodically and copy them to S3. I found an AWS documentation links which describes how to do this manually. But I wanted to automate these parts. There were three main parts in this solution.
1. Creating roles with correct privileges
This is the easiest part to automate with cloud formation. Just to define roles with correct privileges.
2. Register snapshot registration using a lambda backed custom resource
This is the most trickiest part of the solution. We need trigger a ES endpoint to register the ES snapshot repository. But this is an onetime thing and we need just to run it when stack is getting created. So I used a lambda backed custom resource to register S3 backed ES Snapshot repository.
3. Automating snapshot creation using a lambda triggered using a scheduled cloud watch rule.
And third part is to create a cloud watch rule to trigger an ES Snapshot creation by triggering ES endpoint using a lambda. Here I have used the same pattern used in other ES solutions I created and explained in some of my previous blogs.
All lambda solutions can be found in GitHub location [1] and the cloud-formation template is available at GitHub location [2].
References
[1] https://github.com/danushkaf/aws_elasticsearch_centralized_logging_with_rollover