[ML] Inconsistent categorization_filters content for ML jobs in Kibana

[elasticmachine]

Original comment by @lcawl:

On 5.4 and 5.5 builds from s3, there is disparity with respect to the categorization_filters content.

For example, when I run the following API in the Dev Tools tab in Kibana:

PUT _xpack/ml/anomaly_detectors/it_ops_logs
{
  "description": "IT Ops application logs",
  "analysis_config": {
    "categorization_field_name": "message",
    "bucket_span":"30m",
    "detectors": [
      {
      "function": "count",
      "by_field_name": "mlcategory",
      "detector_description": "Unusual message counts"
    }],
    "categorization_filters":"\\[statement:.*\\]"
  },
  "analysis_limits": {
      "categorization_examples_limit": 5
  },
  "data_description": {
    "time_field": "time",
    "time_format": "epoch_ms"
  }
}

It returns the following:

...
"analysis_config": {
    "bucket_span": "30m",
    "categorization_field_name": "message",
    "categorization_filters": [
      """\[statement:.*\]"""
    ]
...

Ditto when I run GET _xpack/ml/anomaly_detectors/it_ops_logs from the Dev Tools tab.

If I run it from the command line, however, I get the following:

curl -u elastic:changeme -XGET 'localhost:9200/_xpack/ml/anomaly_detectors/it_ops_logs'
{"count":1,"jobs":[{"job_id":"it_ops_logs","job_type":"anomaly_detector","job_version":"5.5.0","description":"IT Ops application logs","create_time":1497295377972,"analysis_config":{"bucket_span":"30m","categorization_field_name":"message","categorization_filters":["\\[statement:.*\\]"],"detectors":[{"detector_description":"Unusual message counts","function":"count","by_field_name":"mlcategory","detector_rules":[],"detector_index":0}],"influencers":[]},"analysis_limits":{"categorization_examples_limit":5},"data_description":{"time_field":"time","time_format":"epoch_ms"},"model_snapshot_retention_days":1,"results_index_name":"shared"}]}Lisas-MBP:~ lcawley$ 

Note that the categorization_filters content matches what I specified in the PUT command in this case.

The "categorization_filters":"""[statement:.*]""" does not actually seem to be invalid (I can successfully create a job in the Dev Tools with that syntax, it's just a bit confusing that it's returning a different syntax in Dev Tools vs command line.

For interest's sake, to get this to work in the advanced job wizard I must use the following syntax:

.... the categorization_filters property then has the desired format in the Edit JSON tab:

[elasticmachine]

Original comment by @lcawl:

FYI, this problem was originally encountered in LINK REDACTED

[elasticmachine]

Original comment by @sophiec20:

Reviewing this ticket with 6.0.0 GA. (it's not actually changed)

UI text box
The text box in the UI does not (and should not) require escape characters.

This is the same format as could be used with regex101 etc.

UI Job List row expansion
Row expansion does not display the escape characters. This matches the UI text box when creating the job, which is intuitive.

UI JSON tab
This displays escape characters

Dev tools - syntax to PUT a job
This can be done using either

    "categorization_filters":"""\[statement:.*\]"""

or

    "categorization_filters":"\\[statement:.*\\]"

Dev tools - Output from PUT job and GET job
Both returns as follows:

    "categorization_filters":"""\[statement:.*\]"""

cUrl - PUT job

       "categorization_filters" : [
          "\\[statement:.*\\]"
        ],

cUrl - Output from PUT job and GET job

       "categorization_filters" : [
          "\\[statement:.*\\]"
        ],

TODO what does Postman accept?

The documentation shows both cUrl and the UI, with and without the double backslash. However, it does not specifically call out the difference.

Dev Tools seems to have it's own method for escaping, and I doubt there is scope for this to be changed. As long as any examples to PUT a job uses the double backslash, then we can be consistent in terms of syntax that works when creating a job.

Also, the example in the docs is greedy, meaning is selects from first [statement: to the very final ], and does not check if the square brackets are related. This example should probably be replaced with \[statement:.[^\]]*\] and its escaped version "\\[statement:.[^\\]]*\\]" to remove anything surrounded by their own square brackets.

In short, I think we can only fix this in the documentation. Also, maybe add to the tooltip for the categorization filter UI input.

[KOTungseth]

@lcawl is this issue still valid?