Bug: transient network errors attempting to use gRPC with proxy

[bcoe]

Problem description

A user in googleapis/nodejs-recaptcha-enterprise#231 is having trouble connecting to GCP APIs with with a proxy.

They've shared a full version of their error log here:

https://github.com/googleapis/nodejs-recaptcha-enterprise/files/6709916/recaptcha.error.txt

I was hoping something might jump out at you, or there might be additional information we could ask for.

[murgatroid99]

I think the problem is a TLS connection error, which currently isn't getting logged. I'll add some logging there to verify.

[murgatroid99]

OK, that took longer than I expected, but I published a new version of grpc-js with logging that should give more details in the scenario I think they're encountering. So, if they run again with the new version and the trace options, we should see a new log line.

[bcoe]

@furkee would you be able to provide a stack trace again, with the new trace logic @murgatroid99 has added?

[furkee]

same logs. the logs @murgatroid99 added with the release did not show up. (used 1.3.5 release of grpc and made the request through nodejs-recaptcha-enterprise package with grpx_proxy pointing to local mitmproxy)

error-log.txt

[murgatroid99]

That's very surprising. I am going to add more logging to try to understand what is happening. In the meantime, can you share the output of npm ls @grpc/grpc-js to verify that nodejs-recaptcha-enterprise is using the updated version?

[murgatroid99]

I have now published version 1.3.6 with even more logging. If you update again we should see at least one new log line.

[furkee]

Hey, thanks for quick updates. Here are the new logs:

error-logs-1.3.6.txt

edit: i did npm ls thing and realized there was another grpc package within google-gax and recaptcha was using that. sorry for misdirecting you there.

[murgatroid99]

OK, that log has the new log line, which does show why it's failing:

Failed to establish a TLS connection to recaptchaenterprise.googleapis.com:443 through proxy ::1:8080 with error self signed certificate in certificate chain

This could be a server problem, but if nobody else is experiencing it, it seems more likely that the problem is that the proxy is intercepting TLS connections and providing its own certificates.

[bcoe]

@furkee can you perhaps experiment with the certificate settings of mitmproxy?

This is outside my area of expertise, but it seems like there are some settings related to the certificate provided, perhaps as @murgatroid99 says a certificate is being used which is interfering with the connection.

[furkee]

i checked the logs produced in the staging environment where the proxy also has a valid certificate. i am back to 2021-07-30T08:21:38.787Z | subchannel | 127.0.0.1:3128 CONNECTING -> TRANSIENT_FAILURE. i am also trying to work this out with my company's internal proxy team. do these log tell anything useful?

logs.txt

[murgatroid99]

That is clearly not the whole log. I can't tell what it's supposed to illustrate, but it doesn't have the conclusion of the connection attempt through the proxy, which is really the important part for this issue.

[furkee]

i doubt this will be helpful for you maintainers but if anyone stumbles upon this, i resolved this by using google-auth-library to get a client and make the enterprise API request through that client, which does reach to the enterprise API. dunno what's wrong with (if anything) enterprise client library.