Skip to content

Pyroscope garbles application logs #4042

Open
@samthebest

Description

@samthebest

Describe the bug

We have some log entries like this:

2025-03-22 20:23:30.174{"level":"INFO","ts":"1742675010174","deviceID":"1234","message":"Bla bla"}
 [ERROR] Error uploading snapshot: 422 {"code":"unknown","message":"pushing IngestInput-pprof failed unknown: at least 2 live replicas required, could only find 1"}

This should be two log entries like this:

{"level":"INFO","ts":"1742675010174","deviceID":"1234","message":"Bla bla"}
2025-03-22 20:23:30.174 [ERROR] Error uploading snapshot: 422 {"code":"unknown","message":"pushing IngestInput-pprof failed unknown: at least 2 live replicas required, could only find 1"}

As you can see they are being interleaved. The second log line comes from pyroscope, the former from our application.

Unfortunately I'm not fully familiar with our infrastructure setup, especially pyroscope & grafana, and it would take me a few weeks of archeology to figure out answers to basic questions such as "how do we deploy pyroscope", etc.

All I really know is that the application is a JVM process, and we aggregate logs into Loki.

So I was just hoping that someone might be able to point me quickly to some documentation or a known issue where pyroscope is interleaving it's log lines with the JVM application. Does this problem have a solution? Can pyroscope be told to write it's logs to a different output? If so how?

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions