Module update and path to supporting 7.x

[fatmcgav]

Hey all,

So I wanted to open this issue to give anyone watching an update on what's next for this module.

Firstly, apologies from Elastic that this module has not been given the time and effort it deserves.

The plan for the next steps for this module are as follows:

• Refresh dependencies and supported versions.

  This will unfortunately require dropping support for any Puppet versions older than 5.x, and any Elasticsearch version older than 6.x.

• Remove support for installing and managing multiple instances on a single host.

  This was the "recommended" method a few years ago in order to maximise the memory utilisation for a given host containing more than 32GB of RAM. However other methods have become more prevalent since, such as running Elasticsearch in Docker.
  Therefore in order to massively simplify this module we will be removing support for installing and managing multiple instances on a single host.

• Update the module to support installing Elasticsearch 7.

  This is the biggest thing that most of you are asking for - Support for installing and managing Elasticsearch 7.x.
  The good news is: It's coming!

Feel free to ask any questions or raise any concerns you might have with any of the above, and we'll do our best to work with you.

And thanks again to everyone who's taken the time to open issues or PR's - It's really appreciated.

[faxm0dem]

We're using both ansible and puppet for managing our infrastructure.
It would make sense to have both puppet and ansible modules to generate "similar" configuration state on the hosts, so that minimal conflict might happen when using both configuration management tools, or when switching from one to the other.

[zofog]

Please ensure support for upgrading from 6.x to 7.x for instances that were installed using this module.

[andrewbierbaum]

Since Ubuntu 20.04 is near and there are starting to be documtation for 8.x, please test at least some against if possible against those so the transition going forward is as easy as possible.

Also, version 7.x can be run on this module if jvm setting are set directly. It's not clear to me why those jvm settings are not just pulled more directly from the base installation with the ability to tweak them from there. Ideally this module would pull as much as possible from what's known to be a good base installation and provide the ability to change the important configurations from there.

Thanks for your work on this!

[cdenneen]

@fatmcgav I honestly disagree with the multiple instances removal for this module. The docker options while nice should be handled by other methods like docker-compose or helm charts for k8s. This is still a “puppet” module for deploying and maintaining elastic on a full VM/bare metal install. It definitely has different audience than the “docker” folks. Unless the plan for this module is to use docker module as dependency and spin up elastic inside containers leveraging ECE or something I feel that instance support should remain for the foreseeable future. The users of this module most likely wouldn’t even be using it, or puppet for that matter, in the case of deploying to a container setup. With that said I don’t think that allows for a “one size fits all model” unless, as mentioned above, this module will now support the installation of elastic on containers via puppet in which multi instance would still be supported but via containers rather than systemd/jvm processes.

[andrewbierbaum]

Master installed on the same host as a data node should absolutely be removed. That is already covered by having both node.master and node.data as true. This is a better solution than having two instances of elasticsearch running on the same VM.

https://www.elastic.co/guide/en/elasticsearch/reference/current/modules-node.html

Having two separate instances of elastic running was sited by the previous maintainer as being on of the main challenges of moving to his forward. It should be removed.

I'm also using both puppet and docker images for our deploy. I think most shops working with an updated puppet and elastic will have some docker workflow.

[cdenneen]

Running data and master in same node is great for non-prod use. However in a production level having the master and data on separate JVMs to avoid ingestion OOM exceptions taking down the master instance is always best. In a VM world this is fine but in real steel hardware can have tons of memory so to utilize hardware with 64gb+ memory best is to run multiple instances to utilize the memory properly.

[cdenneen]

I'm also using both puppet and docker images for our deploy. I think most shops working with an updated puppet and elastic will have some docker workflow.

I think that is a pretty big assumption. If you are using containers there is too much bloat in this module to maintain that deployment. You have no reason for half of what this controls. Unless your running full blown systemd containers not sure how this module would manage elastic on docker.
Moving towards containers I don’t disagree with but since this module wouldn’t be able to maintain it, it’s for a different audience than real steel or vm deployments.

[mookie-]

At first, thank you very much for providing a puppet module for elasticsearch. Today I've noticed this issue and I've read this:

Remove support for installing and managing multiple instances on a single host.

This was the "recommended" method a few years ago in order to maximise the memory utilisation for a given host containing more than 32GB of RAM. However other methods have become more prevalent since, such as running Elasticsearch in Docker.
Therefore in order to massively simplify this module we will be removing support for installing and managing multiple instances on a single host.

I don't understand it. If you use elasticsearch in docker, then you probably don't use this puppet module. I don't want to have 5 dockers, I want 5 elasticsearch instances on my machine. Also to massively simplify this module, the first steps from my point of view would be to make it more "puppet" like. You have a lot of custom functions/types/... in it which are not needed you could have done that with puppet standard features.
Why did you build a template engine in your module if there already is a template engine in puppet? I remember debugging this module multiple times and getting really frustrated due to this. This also makes it harder to contribute to this module.

[fatmcgav]

Firstly, thank you to everyone who has taken the time to provide feedback, 👍's etc, and apologies for not updating sooner.

In the background, we've been working on resolving some latent issues with the failing tests, updating dependencies etc.

Please be assured that we don't make the decision to remove the multi-instance functionality lightly.
However running multiple instances natively on a single node has often led to unpredictable performance profiles, unexpected behaviour, etc and therefore is no longer an officially supported deployment method.
It also results in much more complexity within this module, which makes adding support for new versions etc increasingly complicated and time-consuming.

With that being said, we appreciate that in certain scenarios, such as dev or test clusters etc, you are often forced to work with the hardware that's available. So we will be exploring whether we can provide some more guidance, examples etc around running multiple instances on a single node using other tools, such as Docker.

I'll follow up and address more specific feedback in separate comments.

[fatmcgav]

Please ensure support for upgrading from 6.x to 7.x for instances that were installed using this module.

@zofog We will be working to make the upgrade and migration path between 6.x and 7.x as painless as possible. However it's possible that there may not be a simple migration path. Further details will be provided when possible.

Since Ubuntu 20.04 is near and there are starting to be documtation for 8.x, please test at least some against if possible against those so the transition going forward is as easy as possible.

Also, version 7.x can be run on this module if jvm setting are set directly. It's not clear to me why those jvm settings are not just pulled more directly from the base installation with the ability to tweak them from there. Ideally this module would pull as much as possible from what's known to be a good base installation and provide the ability to change the important configurations from there.

Thanks for your work on this!

@andrewbierbaum Matching as much as possible the "default" JVM options etc will be part of this work, as we are aware that the divergence is an issue.
We'll review adding Ubuntu 20.04 (Focal) to the support matrix at the earliest possibility.

@cdenneen I appreciate that when running on bare metal, you will want to maximise the resource utilisation of the underlying hardware. However that is where Docker provides better guarantees around resource separation etc.
Whilst this module won't directly support provisioning multiple instances via Docker, we will be looking at providing further guidance to support this usage model in the near future.

I don't understand it. If you use elasticsearch in docker, then you probably don't use this puppet module. I don't want to have 5 dockers, I want 5 elasticsearch instances on my machine. Also to massively simplify this module, the first steps from my point of view would be to make it more "puppet" like. You have a lot of custom functions/types/... in it which are not needed you could have done that with puppet standard features.
Why did you build a template engine in your module if there already is a template engine in puppet? I remember debugging this module multiple times and getting really frustrated due to this. This also makes it harder to contribute to this module.

@mookie- It's worth noting that Puppet itself has changed a lot over recent years, with the introduction of richer typing etc. And the language is continually evolving and adding new functionality. So reviewing the existing "custom" pieces is part of the plan.

[cdenneen]

@cdenneen I appreciate that when running on bare metal, you will want to maximise the resource utilisation of the underlying hardware. However that is where Docker provides better guarantees around resource separation etc.
Whilst this module won't directly support provisioning multiple instances via Docker, we will be looking at providing further guidance to support this usage model in the near future.

I don't understand it. If you use elasticsearch in docker, then you probably don't use this puppet module. I don't want to have 5 dockers, I want 5 elasticsearch instances on my machine. Also to massively simplify this module, the first steps from my point of view would be to make it more "puppet" like. You have a lot of custom functions/types/... in it which are not needed you could have done that with puppet standard features.
Why did you build a template engine in your module if there already is a template engine in puppet? I remember debugging this module multiple times and getting really frustrated due to this. This also makes it harder to contribute to this module.
@fatmcgav The issue here is using docker as you said is outside the scope of this module and in that case you wouldn't honestly be using a puppet module but more of something like a helm chart. So since this module is geared more towards VM's or bare metal having the option to maintain multi-instance I think still makes sense.
If this module wants to completely revamp and handle ECE or something that will leverage containers under the covers then that handles the multi-instance model as well.

I honestly think if you diverge from being able to support multi-instance I see people moving away from this module completely in favor of ECK or Elastic's official Helm Chart. Not to say there wouldn't be some way to cobble this together in a container scenario but honestly if you are setting things up in a "container" fashion... you aren't going to use something that's basically shoe horning a "full stack" solution into a "container" model.