Weird non-determinstic behavior on PyTorch imagenet

[jma127]

Repro

• Apply Add support for deterministic behavior in ImageNet benchmark #381 on this repo
• cd to the ImageNet folder
• Run python main.py --arch resnet18 --seed 0 --gpu 0 /path/to/imagenet/ on a multi-GPU machine, once with CUDA_VISIBLE_DEVICES=0 and once with CUDA_VISIBLE_DEVICES=1.

Environment

• PyTorch master
• CUDA 9.0
• Driver 384.81
• Ubuntu 16.04

Expected behavior

The two runs have the same output.

Actual behavior

The two runs have the same output when you run them one after the other (e.g. GPU 0 first, then Ctrl-C, then GPU 1). But when you run them at the same time, you get different output.

Suspicion

This is a driver bug. I dunno how PyTorch would be able to bypass CUDA_VISIBLE_DEVICES-based GPU segregation. But posting here for visibility anyways.

cc @shubho @ssnl @soumith @ailzhang

[ssnl]

cc @ngimel

…

On Sat, Jul 7, 2018 at 06:56 Jerry Ma ***@***.***> wrote: Repro - Apply #381 <#381> on this repo - cd to the ImageNet folder - Run python main.py --arch resnet18 --seed 0 --gpu 0 /path/to/imagenet/ on a multi-GPU machine, once with CUDA_VISIBLE_DEVICES=0 and once with CUDA_VISIBLE_DEVICES=1. Environment - PyTorch master - CUDA 9.0 - Driver 384.81 - Ubuntu 16.04 Expected behavior The two runs have the same output. Actual behavior The two runs have the same output when you run them one after the other (e.g. GPU 0 first, then Ctrl-C, then GPU 1). But when you run them at the same time, you get different output. Suspicion This is a driver bug. I dunno how PyTorch would be able to bypass CUDA_VISIBLE_DEVICES-based GPU segregation. cc @shubho <https://github.com/shubho> @ssnl <https://github.com/SsnL> @soumith <https://github.com/soumith> @ailzhang <https://github.com/ailzhang> — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#382>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AFaWZUyHLSDhEOhVdlhHmZn4JuiVkOrpks5uEJPJgaJpZM4VGUVT> .

[ngimel]

I remember at some point cudnn.deterministic=True and cudnn.benchmark=True did not guarantee deterministic behavior between runs, there was a discussion in one of the pytorch issues. Has this been fixed? (i'm on the phone now, so hard for me to search the issues and look at the code)