Bazel: How can I compile gRPC java without having to compile protoc?

[ratnikov]

Hi! I'm on a journey to try to eradicate having bazel build protoc each time in favor of using prebuilt binaries. I'm really after kotlin, but it has a hard dependency on grpc-java and here I am.

https://github.com/aspect-build/toolchains_protoc has made some progress in that for regular proto generation, but it seems like grpc java generation still requires building protoc from source, plausibly to build the plugin. Looks like there're also precompiled plugin source distributed via source, but I don't see an easy way to switch to use them.

• Was there any work done in this area before? A quick issue search didn't come up with anything for me.
• Are there maybe any grpc-java specifics would make it hard (or impossible) to generate grpc java bindings without having to compile protoc?
• Seems like there's some toolchain-like functionality built into the grpc rules, but not quite a full-fledged toolchain that would allow override using precompiled binaries. Is there any documentation/guidance about intent for java_rpc_toolchain rule?
• Any other advice?

Hopefully patches are welcome. :)

[ejona86]

I know grpc-kotlin uses grpc-java runtime, and protoc kotlin for messages which uses protoc java for messages. But does the generated code actually depend on protoc-gen-grpc-java generated code? Looking at the example, it seems so.

I have spoken to Alex about aspect-build. But we (grpc) is not actively spending time on it. There are already pre-built binaries available on Maven Central, so most of what is needed is probably plumbing and glue.

Yes, grpc-java has a limited toolchain, but it was more to consume the other toolchain stuff that has been going on, in particular to trigger some specialized Android goo. No work has been made to allow the toolchain to be customized (and I still don't understand how the toolchains work in Bazel).

I have been waiting for proto_lang_toolchain to become available. Twice a year or so I check on it and it there hadn't been much to show for it over the years. I've not looked into it yet, but I understand that things have improved in recent Bazel with the movement of rules_proto finally out of Bazel and not being bound to com_google_protobuf for the runtime. Most of my time over the years on Bazel rules has been dealing with the com_google_protobuf dependency.

I'm quite open to changes, but we will need to think about the upgrade story and I assume support both old and new ways simultaneously.

[ratnikov]

Thanks @ejona86! This all sounds very reasonable!

I'd say the next steps (for me, or anyone else interested in this work) is:

1. Create a PoC for grpc-java with prebuilts for a java grpc example in toolchain_protoc: https://github.com/aspect-build/toolchains_protoc/blob/main/examples/BUILD.bazel
2. Refactoring some of the grpc_java pseudo-toolchain work to be a proper toolchain. Sounds like this may be a good stand-alone community contribution.

And yes, all the new Bazel 8.X can make this all moot, so I guess there's an option to wait until protocolbuffers/protobuf#19558 is resolved.

What's the best way to track this work? This issue?

[ejona86]

As discovered at #11767 (comment), there are problems depending on Bazel 8, rules_proto 7, and (already known) newer protobufs that probably have good things in them. I think rules_proto 6 may be enough to let things move forward, but that is a complication.

I see a notice that toolchains_protoc is currently broken with Bazel 8 and will only be fixed in a future protobuf release. We may find this issue to be premature, but there may still be ways to make progress. I think that notice is because protobuf made the toolchains private.

I'll note that any refactoring should probably first move to proto_lang_toolchain(). We wouldn't want to make our toolchain public without first being on proto_lang_toolchain(). I see protobuf has their toolchains as private. As we are refactoring, we should probably do the same, until we are certain things are ready.

As long as protobuf has a private toolchain, you must use the Bazel build from protobuf and not depend on Java binaries on Maven Central. I'd be very happy to split out the grpc-java rule to its own repository and not require building grpc-java's Java dependencies from source. But right now you need maven.override()s to avoid seeing duplicate classes on the classpath. That would probably need to be moved to protobuf's repo.