Enabling Multi-GPU training

[julkaztwittera]

How to enable multi-GPU training? No matter how many GPUs I use, only one process starts.

[maximepeabody]

I'm also curious about this

[Splendon]

Is there any progress?

[zyshin]

FYI, since the training code is wrapped in Accelerator, I just launch training with multiple GPUs and it seems work. Also, make sure that some functions are called only once in the main process by checking accelerator.is_main_process (e.g., create_output_folders, save_pipe, and the evaluation part)