Index migration to 7.17.1 doesnt create _meta field for .watches and .triggered_watches indices

[TheRiffRafi]

Elasticsearch Version

7.17.1

Installed Plugins

No response

Java Version

bundled

OS Version

Any OS

Problem Description

Upgrade to 7.17.1 fails to create mappings for the entire _meta object in the .watches and .triggered_watches indices.
So far we have only confirmed the behavior upgrading from version 7.13.1

Steps to Reproduce

So far it seems reproduction is not consistent, unsure on exact cluster settings or state needed to generate the problem.
We have confirmed from 4 users however that this happened just after upgrading to 7.17.1

Logs (if relevant)

Missing _meta field in mapping [_doc] of index [.triggered_watches], assuming mappings update required
Missing _meta field in mapping [_doc] of index [.watches], assuming mappings update required

[TheRiffRafi]

Current work-around is to add the field and a value below the current running version.
For example if the cluster is running version 7.17.1 then set 7.17.0 (or the version before upgrade if known) for the version value:

PUT .watches/_mapping
{
    "_meta": {
      "version" : "7.17.0"
  }
}

PUT .triggered_watches/_mapping
{
    "_meta": {
      "version" : "7.17.0"
  }
} 

[elasticmachine]

Pinging @elastic/es-core-infra (Team:Core/Infra)

[CyberneticNomad-v808]

I had the same issue with 8.1, I tried the workaround from @TheRiffRafi, the updates went through just fine but then I'm left with update_mapping [_doc] with no other context and the watcher service still not running. Any idea what's going on?

[williamrandolph]

I looked into this and managed to replicate it with the following steps:

1. Run Elasticsearch 6.8.20 with a trial license
2. Create a watch (the one from the docs is fine
3. Upgrade to 7.13.1. There will be a warning about failing to notify a cluster state updater, with a stack trace containing java.lang.IllegalStateException: Cannot read version string in index .watches
4. Upgrade to 7.17.1. There will be many warning messages saying `Missing _meta field in mapping [doc] of index [.watches], assuming mappings update required

Now that I've reproduced this, I don't think this actually has to do with the Feature Migration code, since I didn't have to run a feature migration to reproduce. I believe the bug has to do with the SystemIndexManager, which is supposed to handle mapping updates behind the scenes for internal system indices.

[williamrandolph]

It looks like we added some handling for a missing _meta field in #83896, which is why we see different messages in 7.17.1 for the same issue.

[williamrandolph]

After digging into this code a bit, I found a bug in the system index definitions for Watcher.

First, the thing that has been confusing me. The code in 7.17.x is supposed to quietly restore a missing _meta field in mappings, so these warnings are supposed to disappear soon after the index goes green.

However, before repairing mappings, our system index manager checks to see that the system index has the correct format. There are two ways that we could find a "wrong format":

1. The index's index.format setting does not match the index format number hardcoded in the system index settings.
2. The index's mapping type is incorrect, which, as far as I can tell, means that it is something other than _doc.

The bug (which I will fix) has to do with the first problem. The hard-coded index format number is incorrect in the watcher system index definition.

The way I've been reproducing the issue brings up both of these problems. The second problem occurs because the watcher mapping type in 6.8.x is doc, rather than _doc. This can only be fixed by reindexing.

EDIT: I no longer recommend doing what's below. I'd suggest trying to update the missing _meta field directly. See https://github.com/elastic/sdh-elasticsearch/issues/5736#issuecomment-1072532797 for details.

If you're on 7.17, the system feature migration can clean all of this up for you.

$ curl -s -XPOST 'localhost:9200/_migration/system_features'
{"accepted":true,"features":[{"feature_name":"watcher"}]}

$ curl -s -XGET 'localhost:9200/_migration/system_features
# lots of output, just make sure that the overall migration status is `NO_MIGRATION_NEEDED` before proceeding

$ curl -s -XGET 'localhost:9200/.watches/_mappings' | jq '.'
{
  ".watches-reindexed-for-8": {
    "mappings": {
      "dynamic": "strict",
      "_meta": {
        "version": "7.17.2"
      },
      "properties": {
        [...]
      }
    }
  }
}

In short, running a system feature migration should get rid of the warning.

It might be worth putting in a counter to make sure this message only gets logged once. It looks pretty serious when it shows up so much.