Add a property to disable HTTP Observations for actuator endpoints

[jonatan-ivanov]

It seems there is a good amount of users who are registering an ObservationPredicate in order to disable Observations for /actuator/** endpoints, something like this:

@Bean
ObservationPredicate actuatorServerContextPredicate() {
    return (name, context) -> {
        if (name.equals("http.server.requests") && context instanceof ServerRequestObservationContext serverContext) {
            return !serverContext.getCarrier().getRequestURI().startsWith("<actuator-base-path>");
        }
        return true;
    };
}

I think this common use-case could be simplified by creating a similar bean (for mvc and webflux) and let the users to configure this using a single property.

[jonatan-ivanov]

Connected issue in Framework: spring-projects/spring-framework#29210

[Saljack]

I implemented this solution in our services and it makes things worse than better because there is also Spring Security Observation. The security observation is not affected (because) this predicate disables only http.server.requests. This also creates multiple separated traces because they are created out of any parent trace id (http.server.requests trace). So then there are for example these traces:

643d472f5d60ba294b18360a79427310 - api-gateway: security filterchain after
643d472b0d5bde11b0fc3b53edc9ed36 - api-gateway: authorize -security-context-server-web-exchange
643d472646d2ef6b55c842b12e41d502 - api-gateway: security filterchain before

Of course I can disable Spring Security Observation but it does not make sense because then there can be for example Spring Data Observation.

[jonatan-ivanov]

There are a few connected issues to this, let me try to collect them here:

• Provide a way to make decisions based on the parent in ObservationPredicate micrometer-metrics/micrometer#3678
• Add request to Observation Context at creation to enable filtering by the request spring-security#12854
• Add ability to ignore request path prefixes in Observability server filters spring-framework#29210
• Provide configuration for ignoring routes from being sampled #34400

What you described as an issue can be a valid use-case for other users (ignoring parent observation but keeping children) but there is a solution/workaround to disable every Observation that was triggered through a specific HTTP endpoint (e.g.: /actuator) in spring-projects/spring-security#12854, please see spring-projects/spring-security#12854 (comment) and spring-projects/spring-security#12854 (comment)

[Saljack]

Ohh thanks. It is a nice summary. I was totally blind 🤦. Now it makes more sense to me. It is pretty hard to follow everything regarding observation because you need to follow/search multiple projects. I believe that most of these issues will be resolved in the Spring Boot 3.1. I really appreciate your work 👍.

[denniseffing]

@jonatan-ivanov Do you have working workarounds to disable every Observation that was triggered through a specific HTTP endpoint for WebFlux as well? ☺️

[jonatan-ivanov]

I might have something that could work for you but I have never tried with webflux and I did not test it a lot for webmvc, I threw it out because of the RequestContextHolder solution above seemed much better:

I think you might be able to write a getRoot method that recursively can go upwards on the parent chain (context.parent) and find the one that has no (null) parent. At that point, you get the root context/observation, something like this:

private Observation.Context getRoot(Observation.Context current) {
    ObservationView parent = current.getParentObservation();
    if (parent == null) {
        return current;
    }
    else {
        return getRoot((Context) parent.getContextView());
    }
}

And if you got the root, you can check if it is an "HTTP context" and the path so you can ignore actuator, something like this:

@Bean
ObservationPredicate noActuatorServerObservations() {
    return (name, context) ->
        Context root = getRoot(context);
        if (root instanceof ServerRequestObservationContext serverContext) {
            return !serverContext.getCarrier().getRequestURI().startsWith("/actuator");
        }
        else {
            return true;
        }
    };
}

I guess another workaround could be getting the current request from Reactor somehow, maybe @chemicL or @bclozel could help in that.

[chemicL]

From Reactor's perspective, I think not much can be done, aside from clearing the ObservationThreadLocalAccessor.KEY from the Reactor Context, but that doesn't disable the Observation that is downstream in the operator chain: if it's started, it is in effect, only the propagation would not reach some parts, but the framework codebase is where the creation and start process needs to be eliminated.