Description
Is your feature request related to a problem? Please describe.
Net::HTTP doesn't get very good parallel throughput because it's totally in Ruby and doesn't benefit from being able to manage its own I/O or threading in the same way that a C extension would. The AWS SDK supports pluggable handlers, and this was brought up in #301, but in SDKv3 I don't know what the current pattern for implementing such a handler would look like.
In particular, I'm issuing a ton of S3 API calls in threads, and throughput maxes out at a relatively low number of threads because everything is having to go through the Ruby interpreter, so there are only so many tasks that can run concurrently.
Describe the solution you'd like
I would love official support for Faraday (for multiple pluggable backends) or Typhoeus (which is what I would use, as it's a wrapper for cURL), possibly in a plugin. Typhoeus, in particular, would also work for H2 and H3/QUIC since it is using cURL internally. Alternatively, having documentation for how a custom handler might be implemented in SDKv3 would enable me to write my own handler.
Describe alternatives you've considered
I've thought about spawning multiple processes instead of multiple threads, but it would most likely perform worse without spawning processes and threads, and this would be a ton of work to get working properly - most likely more than writing a HTTP handler on top of Typhoeus.
I've also considered JRuby (whose Net::HTTP does hit a code path that runs in compiled code), but we don't use JRuby for anything else (and our other apps won't run under JRuby), so a broader solution that benefits more than this one-off thing would be preferred.
- 👋 I may be able to implement this feature request
Additional context
I am an Amazon employee (not in AWS), and so may be able to get the time to implement this and help you ship it.