python: can't send data to pyroscope using internal ca certificates

[gira0]

Describe the bug you encountered:

Hello everyone, I'm currently evaluating a local pyroscope deployment, but I'm unable to send data to my test instance as soon as I get SSL involved. HTTP is working fine and without any issues, but with our internal ssl certs I get the errors as seen in the log below. I'm sure that the internal CA is the issue, because if I use a ceretificate from LetsEncrypt it works. The internal ca is inside the default OS ca trust store (/etc/ssl/certs/ca-bundle.crt) and environment variables like "SSL_CERT_FILE" had no effect. Unfortunately, I've never worked with rust and can't really debug this myself. The pyroscope config for python doesn't have many options so if theres a way to get more verbose output please let me know and I will provide the logs for that.

What did you expect to happen instead?

The python pyroscope module should provide an option to allow setting a diffent ca bundle or should use the OS cert store like the golang library that worked without any issues.

How did you install pyroscope-rs?
pip3.11 install -U pyroscope-io
pyroscope-io==0.8.7

---

pyroscope-rs version and environment

log output from the example app
2024-08-13T12:14:52.727Z INFO py_spy::python_spy > Found libpython binary @ /usr/lib64/libpython3.11.so.1.0
2024-08-13T12:14:52.728Z INFO py_spy::python_spy > Getting version from python binary BSS
2024-08-13T12:14:52.729Z INFO py_spy::python_spy > Failed to get version from BSS section: failed to find version string
2024-08-13T12:14:52.729Z INFO py_spy::python_spy > Getting version from libpython BSS
2024-08-13T12:14:52.729Z INFO py_spy::version > Found matching version string '3.11.9 (main, Jun 19 2024, 10:02:06) [GCC 8.5.0 20210514 (Red Hat 8.5.0'
2024-08-13T12:14:52.729Z INFO py_spy::python_spy > python version 3.11.9 detected
2024-08-13T12:14:52.729Z INFO py_spy::python_spy > got symbol _PyRuntime (0x00007f9e4fdc96e0) from libpython binary
2024-08-13T12:14:52.729Z INFO py_spy::python_spy > Found interpreter at 0x00007f9e4fdd7d18
2024-08-13T12:14:52.729Z INFO py_spy::python_spy > got symbol _PyRuntime (0x00007f9e4fdc96e0) from libpython binary
2024-08-13T12:14:52.729Z INFO py_spy::python_spy > Found _PyRuntime @ 0x00007f9e4fdc96e0, getting gilstate.tstate_current from offset 0x240
2024-08-13T12:15:00.000Z INFO Pyroscope::Session > Creating Session
2024-08-13T12:15:00.000Z DEBUG Pyroscope::Session > Encoding 3 reports to PPROF
2024-08-13T12:15:00.000Z DEBUG Pyroscope::Session > Compressing 1 reports to Some(GZIP)
2024-08-13T12:15:00.000Z INFO Pyroscope::Session > Sending Session: 1723551290 - 1723551300
2024-08-13T12:15:00.001Z DEBUG reqwest::connect > starting new connection: https://pyroscope.internal-adress.com/
2024-08-13T12:15:00.001Z DEBUG hyper_util::client::legacy::connect::dns > resolving host="pyroscope.internal-adress.com"
2024-08-13T12:15:00.002Z DEBUG hyper_util::client::legacy::connect::http > connecting to 192.168.32.10:443
2024-08-13T12:15:00.004Z DEBUG hyper_util::client::legacy::connect::http > connected to 192.168.32.10:443
2024-08-13T12:15:00.004Z DEBUG rustls::client::hs > No cached session for DnsName("pyroscope.internal-adress.com")
2024-08-13T12:15:00.004Z DEBUG rustls::client::hs > Not resuming any session
2024-08-13T12:15:00.013Z DEBUG rustls::client::hs > Using ciphersuite TLS13_AES_128_GCM_SHA256
2024-08-13T12:15:00.013Z DEBUG rustls::client::tls13 > Not resuming
2024-08-13T12:15:00.013Z DEBUG rustls::client::tls13 > TLS1.3 encrypted extensions: [ServerNameAck]
2024-08-13T12:15:00.013Z DEBUG rustls::client::hs > ALPN protocol is None
2024-08-13T12:15:00.014Z ERROR pyroscope::session > SessionManager - Failed to send session: error sending request for url (https://pyroscope.internal-adress.com/ingest?name=simple.python.app&from=1723551290&until=1723551300&format=pprof&sampleRate=100&spyName=pyspy)

[srstrickland]

I also ran into this problem today. Looks like at python version 0.8.2, there was a change from openssl -> rustls. I tried installing 0.8.1, and to my surprise and delight, it worked!

I am betting openssl also obeys environment variables as expected, too. I spent a while trying to figure out the same for rustls, but couldn't find anything that worked. I'm also not a rust expert, so there may be something that I missed. Just thought I'd share that 0.8.1 seems to work for this use case, in case it unblocks anyone who comes across this thread.

Incidentally, the logging in 0.8.2 at least seems to highlight the problem:

 2025-02-05T23:58:00.001Z DEBUG reqwest::connect   > starting new connection: https://pyroscope.XXXXXXXXXXXXXXXXXXXX
 2025-02-05T23:58:00.025Z DEBUG rustls::client::hs > No cached session for DnsName(DnsName(DnsName("pyroscope.XXXXXXXXXXXXXXXXXXXX")))
 2025-02-05T23:58:00.025Z DEBUG rustls::client::hs > Not resuming any session
 2025-02-05T23:58:00.048Z DEBUG rustls::client::hs > Using ciphersuite TLS13_AES_256_GCM_SHA384
 2025-02-05T23:58:00.048Z DEBUG rustls::client::tls13 > Not resuming
 2025-02-05T23:58:00.048Z DEBUG rustls::client::tls13 > TLS1.3 encrypted extensions: [Protocols([6832])]
 2025-02-05T23:58:00.048Z DEBUG rustls::client::hs    > ALPN protocol is Some(b"h2")
 2025-02-05T23:58:00.049Z WARN  rustls::conn          > Sending fatal alert BadCertificate
 2025-02-05T23:58:00.049Z ERROR pyroscope::session    > SessionManager - Failed to send session: error sending request for url (https://pyroscope.XXXXXXXXXXXXXXXXXXXX/ingest?name=example-web.cpu&...&from=1738799870&until=1738799880&format=folded&sampleRate=100&spyName=pyspy): error trying to connect: invalid peer certificate contents: invalid peer certificate: UnknownIssuer

... whereas 0.8.8 looks more like what's above; just a failure... no information. I was eventually able to deduce that the problem was TLS, but it would've been much more apparent with these logs above.

This is a pretty major regression IMHO, and I'm curious what rustls offers over openssl to justify this loss in functionality.