Use FastThreadLocalThread's in shared channel executor.

[buchgr]

While working on netty/netty@b88a980 I noticed that when using the blocking stub and the netty transport, we allocate in Netty's allocator from within the shared executor threads. Netty's allocator employs thread local caches that perform the majority of allocations. These caches are accessed via Netty's own threadlocal implementation FastThreadLocal, which is only "fast" when used in the context of a FastThreadLocalThread.

We should provide our own thread factory to the cached thread pool in order to create FastThreadLocalThreads. Additionally, we should override the Thread's run() method to do

  public void run() {
    try {
      super.run();
   } finally {
     FastThreadLocal.removeAll();
   }
  }

This will release all memory from a thread's cache when it is shutdown. That should happen quite frequently, with the pool sizing threads dynamically.

However, I don't know how to best implement this. The shared channel executor should be shared by all transport, client, server combinations running in a JVM. Also, I assume we can't rely on the Netty dependency to be there on Android?

[ejona86]

In what cases would the blocking call be running within the cached thread pool? I know there's the case where a server makes an outgoing call.

For all performance cases, I'd expect the user to override the executor being used, so I'd expect that our thread factory wouldn't be used when we'd want it. We could recommend our users to use Netty-aware threads though.

[buchgr]

yes, outgoing calls and responses on the server side, no? Good point with providing a different Executor.

That being said, I don't know what the benefit of using FastThreadLocalThreads for gRPC is. On my MBP, Netty's FastThreadLocalBenchmark runs ~4.6 times faster when using FastThreadLocalThreads. Now, that might just be cause the slowpath of FastThreadLocal is rather slow [1].

Anyways, is it feasable to have the GrpcUtil.SHARED_CHANNEL_EXECUTOR always use FastThreadLocalThreads? Would that work on Android (i.e. do we have the Netty dependency there?).

[1] netty/netty#4418

[ejona86]

Netty's FastThreadLocalBenchmark runs ~4.6 times faster

I question how much that matters. It doesn't matter the speedup if the operation doesn't consume much of the total CPU time. For example, if it is consuming 0.01% of the CPU today, then a 100000x speedup wouldn't be noticeable.

Anyways, is it feasable to have the GrpcUtil.SHARED_CHANNEL_EXECUTOR always use FastThreadLocalThreads?

No. In no way could we do that. Netty must not be a dependency of core.

[buchgr]

@ejona86

I question how much that matters. It doesn't matter the speedup if the operation doesn't consume much of the total CPU time. For example, if it is consuming 0.01% of the CPU today, then a 100000x speedup wouldn't be noticeable.

yup, that's also what I was thinking. I could look into this though.

No. In no way could we do that. Netty must not be a dependency of core.

yes, so my question: can we even implement this nicely? We could obviously use a different executor in the Netty specific builders, but then if someone was to use e.g. okhttp + netty transport in the same JVM, there would be two different executors...

I could maybe just look into it and see if we can get better performance from it, and we can then decide?

[ejona86]

but then if someone was to use e.g. okhttp + netty transport in the same JVM, there would be two different executors...

Not a huge concern to me. There is very little reason to do this and if it was done 1) two sets of executors isn't the end of the world and 2) the application could just override the executor.

I could maybe just look into it and see if we can get better performance from it, and we can then decide?

That could be fine. I'm mostly interested what a user would have to do in order to make use of the optimization, since the users that care about the optimization are should also be overriding the executor.

[buchgr]

That could be fine. I'm mostly interested what a user would have to do in order to make use of the optimization, since the users that care about the optimization are should also be overriding the executor.

They simply should use Netty's DefaultThreadFactory [1] in their Executor. It does all the things I mentioned. We can mention this in the docs for executor(Executor exec).

[1] https://github.com/netty/netty/blob/4.1/common/src/main/java/io/netty/util/concurrent/DefaultThreadFactory.java