Ability to specify worker_threads/tinyworker

[lateiskate]

Hello, I hope you are having a nice day (Stay safe!)

Would it be possible to make a pre-launch option where you could specify the method of spawning (but only for certain workers/pools ?)
Apparently, some modules like the canvas module are not context aware and therefore its impossible for me to use worker_threads... But then again I don't want to use child_processes for all pools/workers. Can spawning a certain pool using tinyworker without touching the other pools be possible without major changes?

PS: I'm on node12 but I had to make tweaks to force threads.js to use tiny-worker module but then again, all my stuff are now using child processes and consuming A LOT of resources.

If you have another advice I should use to combat the "non context aware" issue, I would love to hear it. Thanks a lot once again :D

[andywer]

Hey Kate! Hmm, can you elaborate a bit more on what you mean by "not context aware"?

In theory we could just add another option to the new Worker() constructor to explicitly state that you particularly want to use either tiny-worker or worker_threads by all means.

I am just not sure I understand why your canvas package would work in tiny-worker, but not in worker_threads. Maybe you can shine a light on that 😉

Stay safe as well!

[lateiskate]

@andywer Sorry about the late reply

Some modules are not context aware as in, they can't be loaded more than once by several threads.
This should shed more information about it: https://nodejs.org/api/addons.html#addons_context_aware_addons

The reason why the canvas package worked with tiny-worker is because it actually spawns child processes and not worker_threads, so each process has its own independent cache and its own canvas instance. And that's why it works just fine. Regular worker_threads will just lead to one thread loading canvas and all the others erroring "Module did not self-register".

Yeah an option would be absolutely awesome, especially when you can still use worker_threads with others things and just use tiny-worker for a certain thing.

Thank you!

[andywer]

How about this API? Would open a PR later.

new Worker("./path/to/worker", { _implementation: "tiny-worker" })

The leading underscore of the _implementation option is not overly pretty, but it's consistent with other custom threads.js options we allow you to pass (_baseURL).

[lateiskate]

Oh that would be absolutely perfect.
Thank you a lot Andy!

[andywer]

I was getting started as it just occured to me that it won't work the way I suggested… At least not easily / eleganty.

The issue is that we cannot pass the implementation selector to the new Worker() instantiation, as the Worker constructor is already part of the implementation you wish to select…

I see two alternative options here:

1. import { TinyWorker, WorkerThreadsWorker } from "threads" - Would be unusable for users who need to rely on import "threads/register", though
2. Wrap the Worker in yet another constructor which only abstracts the implementation selection - that new abstracted Worker could then not be instanceof NativeWorker, though

Need to think about it a little more…

[lateiskate]

Oh wow, that does seem like a lot of work and sacrifices made.
If it's too much, you should honestly not do it :D

[andywer]

Yeah, I could not come up with an easier solution so far… Turning down the feature request for now. Sorry.

If anyone else needs this, too, feel free to leave a comment here and we might put more effort into it 😉

[Ugzuzg]

This causes problems with some other packages which use native addons (same issue), so an option to specify what to use would be great for me too.

[andywer]

@Ugzuzg Switching to node 12+ is not an option at the moment?

[Ugzuzg]

I am using node 12+, but with threads@0.12.x.

[andywer]

Sorry, @Ugzuzg, but threads 0.x has reached its end of life. Version 0.12 is so old by now… May I ask what's stopping you from upgrading to a recent version of threads.js?

[Ugzuzg]

Yes, I understand, that's why I'm saying that having the option to choose the implementation would be useful for me.
I was hoping to upgrade to the latest version and use worker threads, but because of the issue described in the very first comment, I am forced to use the old version.

[andywer]

Do you guys need to run the code in node.js only or does it also need to work in browsers?