Elsticsearch 6.3 Upgrade Broken (in Debian based Systems)

[roock]

• Module version: 6.3.1
• Puppet version: 4.10.12 and 5.5.4
• OS and version: Debian 9 and Ubuntu 18.04

Bug description

Install any old version of elasticsearch before 6.3.0 e.g.

    include ::java

    class { 'elasticsearch':
      version => '6.2.0'
    }

    elasticsearch::instance { 'es-01': }

Then try to upgrade Elasticsearch to version 6.3.x. The upgrade will fail because it will try to create an initial keystore (this feature was introduced in version 6.3.0 elastic/elasticsearch#28928 ). The creating of the keystore will fail because elasticsearch-keystore doesn't find ES_PATH_CONF and exists with an errorcode.

==> puppet-test01.local: ES_PATH_CONF must be set to the configuration path
==> puppet-test01.local: dpkg: error processing package elasticsearch (--configure):
==> puppet-test01.local:  installed elasticsearch package post-installation script subprocess returned error exit status 1

The problem does not happen if the initial version is already 6.3.x because then the keystore is created before ES_PATH_CONF is removed from /etc/default/elasticsearch.

[archeious]

Did you figure out a workaround for this? I have not been able to figure out how to proceed.

[olevole]

same issue.

I used to for work-around global env:

file_line { 'es-path-conf':
          path  => '/etc/environment',
          line  => "ES_PATH_CONF=/etc/elasticsearch",
          match => 'ES_PATH_CONF=',
}

[tylerjl]

Thank you for the report, I'm looking into potential solutions in this branch. This is another victim of the complications this module has to make in order to support the concept of instances, unfortunately (and another potential nail in the coffin of continuing to support multiple instances in the future.)

[benedikt-haug]

I also got the problem when upgrading from 6.3.0 => 6.4.0

Had to copy ES_PATH_CONF
from /etc/default/elasticseach-$instance_name => /etc/default/elasticsearch

[smortex]

@tylerjl setting ES_PATH_CONF in /etc/environment seems to not do the trick:

$ sudo apt upgrade
Reading package lists... Done
Building dependency tree       
Reading state information... Done
Calculating upgrade... Done
The following packages will be upgraded:
  elasticsearch-oss kibana-oss
2 upgraded, 0 newly installed, 0 to remove and 0 not upgraded.
Need to get 110 MB of archives.
After this operation, 73.7 kB of additional disk space will be used.
Do you want to continue? [Y/n] 
Get:1 https://artifacts.elastic.co/packages/oss-6.x/apt stable/main amd64 elasticsearch-oss all 6.4.1 [33.7 MB]
Get:2 https://artifacts.elastic.co/packages/oss-6.x/apt stable/main amd64 kibana-oss amd64 6.4.1 [76.2 MB]
Fetched 110 MB in 7s (14.6 MB/s)                                                                                                                                                                                                            
(Reading database ... 237721 files and directories currently installed.)
Preparing to unpack .../elasticsearch-oss_6.4.1_all.deb ...
Unpacking elasticsearch-oss (6.4.1) over (6.4.0) ...
Preparing to unpack .../kibana-oss_6.4.1_amd64.deb ...
Stopping kibana service... OK
Unpacking kibana-oss (6.4.1) over (6.4.0) ...
Setting up elasticsearch-oss (6.4.1) ...
Installing new version of config file /usr/lib/systemd/system/elasticsearch.service ...
ES_PATH_CONF must be set to the configuration path
dpkg: error processing package elasticsearch-oss (--configure):
 subprocess installed post-installation script returned error exit status 1
Processing triggers for systemd (232-25+deb9u4) ...
Setting up kibana-oss (6.4.1) ...
Errors were encountered while processing:
 elasticsearch-oss
E: Sub-process /usr/bin/dpkg returned an error code (1)

Setting ES_PATH_CONF=/etc/elasticsearch/logs in /etc/default/elasticsearch (the actual path of my instance) may work, but apt ask for a confirmation I was not confident to accept: An elasticsearch keystore already exists. Overwrite? [y/N] 😱 .

Setting ES_PATH_CONF=/etc/elasticsearch in /etc/default/elasticsearch (the default path) make the update pass. The configuration is then somewhat broken (service does not start), but applying the Puppet catalog fixes everything.

[tylerjl]

@smortex out of curiosity, did you re-login/re-source your shell after making the change to /etc/elasticsearch before running apt? From what I can tell it seems to resolve some things on my end as well.

If it really doesn't work for some reason, we'll need to dramatically change some stuff with the module - such as supporting dedicated Elasticsearch instances at all - since setting ES_PATH_CONF in /etc/defaults/elasticsearch clobbers instance-specific ES_PATH_CONF values when invoking any CLI tools in the elasticsearch-env executable.

[smortex]

@tylerjl good point… I sudo-edited /etc/environment, and sudo-apt-upgraded elasticsearch… My regular-user shell was not reloaded after the change (so had not $ES_PATH_CONF set), but if it was, the sudo configuration would have unset $ES_PATH_CONF, so maybe this was a no-op in my workflow.

I assumed the unit would gather all information it needed (including sourcing /etc/environment), but I have not checked if it was actually doing so. Assuming is generally a good way to not find what is actually happening 😕

[roock]

I just checked the postinstall script again and in 6.4.x, it looks like this:
6.4

# the equivalent code for rpm is in posttrans
if [ "$PACKAGE" = "deb" -a ! -f /etc/elasticsearch/elasticsearch.keystore ]; then
    /usr/share/elasticsearch/bin/elasticsearch-keystore create
    chown root:elasticsearch /etc/elasticsearch/elasticsearch.keystore
    chmod 660 /etc/elasticsearch/elasticsearch.keystore
    md5sum /etc/elasticsearch/elasticsearch.keystore > /etc/elasticsearch/.elasticsearch.keystore.initial_md5sum
fi

It ssems that it would also be sufficient to have the puppet module create an empty file at /etc/elasticsearch/elasticsearch.keystore to fix the issue.

[minorOffense]

Just mentioning that when I run puppet the first time I do see this line here:

Notice: Augeas[/etc/sysconfig/elasticsearch](provider=augeas):
--- /etc/sysconfig/elasticsearch	2018-06-11 18:44:24.000000000 -0500
+++ /etc/sysconfig/elasticsearch.augnew	2018-10-03 13:12:32.590030428 -0500
@@ -9,8 +9,6 @@
 #JAVA_HOME=

 # Elasticsearch configuration directory
-ES_PATH_CONF=/etc/elasticsearch
-
 # Elasticsearch PID directory
 #PID_DIR=/var/run/elasticsearch

Looks like it's being removed explicitly.

Not sure if that's helpful.

[smortex]

Okay, new ElasticSearch update today, explicitly setting ES_PATH_CONF in the environment, and all went well.

# env ES_PATH_CONF=/etc/elasticsearch apt upgrade
Reading package lists... Done
Building dependency tree       
Reading state information... Done
Calculating upgrade... Done
The following packages will be upgraded:
  elasticsearch-oss kibana-oss
2 upgraded, 0 newly installed, 0 to remove and 0 not upgraded.
Need to get 110 MB of archives.
After this operation, 8192 B of additional disk space will be used.
Do you want to continue? [Y/n] 
Get:1 https://artifacts.elastic.co/packages/oss-6.x/apt stable/main amd64 elasticsearch-oss all 6.4.2 [33.7 MB]
Get:2 https://artifacts.elastic.co/packages/oss-6.x/apt stable/main amd64 kibana-oss amd64 6.4.2 [76.2 MB]
Fetched 110 MB in 6s (17.4 MB/s)                                                                                                                                                                                                          
(Reading database ... 237851 files and directories currently installed.)
Preparing to unpack .../elasticsearch-oss_6.4.2_all.deb ...
Unpacking elasticsearch-oss (6.4.2) over (6.4.1) ...
Preparing to unpack .../kibana-oss_6.4.2_amd64.deb ...
Stopping kibana service... OK
Unpacking kibana-oss (6.4.2) over (6.4.1) ...
Setting up elasticsearch-oss (6.4.2) ...
Installing new version of config file /usr/lib/systemd/system/elasticsearch.service ...
Created elasticsearch keystore in /etc/elasticsearch
Processing triggers for systemd (232-25+deb9u4) ...
Setting up kibana-oss (6.4.2) ...
#

This is probably the simplest workaround 🎉.

Puppet still fixes a lot of things after the update.

[zeha]

Are there any news on this? It's really annoying to run into this problem on each and every elasticsearch upgrade, and recently there have been more releases in a shorter time frame, so we're hitting this more often. The workaround ... works, but I don't see how I can persist it.

[tkuther]

Just mentioning that when I run puppet the first time I do see this line here:

Notice: Augeas[/etc/sysconfig/elasticsearch](provider=augeas):
--- /etc/sysconfig/elasticsearch	2018-06-11 18:44:24.000000000 -0500
+++ /etc/sysconfig/elasticsearch.augnew	2018-10-03 13:12:32.590030428 -0500
@@ -9,8 +9,6 @@
 #JAVA_HOME=

 # Elasticsearch configuration directory
-ES_PATH_CONF=/etc/elasticsearch
-
 # Elasticsearch PID directory
 #PID_DIR=/var/run/elasticsearch

Looks like it's being removed explicitly.

Not sure if that's helpful.

It seems to be the main cause of this trouble.

[tylerjl]

So here are the options to fix this that I can think of:

1. If anyone can confirm that making this change to /etc/environment resolves the issue, I can merge that fix it and release that to try and resolve it.
2. If option 1 doesn't work, we'll have to rewrite the module to drop support for running dedicated elasticsearch::instance resources entirely in a backwards-incompatible release (i.e., upgrading to the new Puppet module will likely not be possible.)

If anybody can provide feedback regarding whether setting /etc/environment via that Puppet change works - it seems to make things happy in my testing, but I'd like more confirmation - then we can potentially move forward with option 1) which will take significantly less time to fix.

[tkuther]

I can confirm that setting ES_PATH_CONF=/etc/elasticsearch in /etc/environment fixes installation/upgrade of the elasticsearch package.

But given other issues like #892 I'd vote for 2. in the long term.