how can i use the imagenet-22k-wds and imagenet-w21-wds in huggingface on timm/train.py ?

[TheDarkKnight-21th]

i know that wds is the "web dataset" and it is different with image files(data) which is stored in local.

so how can i use the timm/imagenet-22k-wds and timm/imagenet-w21-wds in huggingface on timm/train.py ?

what should i do typing on that script (argument) ?

also is it possible to use the .tar(wds) streaming mode with url (without storing the data in local) ? and then
how can i get the url? like imagenet-12k-wds on webdataset example ? https://huggingface.co/docs/hub/datasets-webdataset

if Data loading with url is possible, which method will be faster between data loading with url or data loading with stored dataset in local?

• i alreay finished downloading the timm/imagenet-w21-wds on my local server with "dataset" library.

< imagenet-w21-wds path>

• plz check the timm/train.py https://github.com/huggingface/pytorch-image-models/blob/main/train.py

[rwightman]

@TheDarkKnight-21th you need to find the local path with the .tar files in it, if it also has the _info.json file you can just use:
python train.py --data-dir /path/to/imagenet-w21-wds/ --dataset wds/ --val-split '' --num-classes 21842

It's actually faster to download the large wds tar datasets using the cli tool and enabling HF transfer.

And it will use the shard info in the info file, there is no val split for that w21-wds which is why I use an empty '' to disable val. The in12k and in22k have a val subset that I created so wouldn't be needed. You can also manually specify the splits and use a subset of shards for val (they are shuffled). See below for manual split format, shard names followed by | and then # samples

python train.py --data-dir /imagenet-w21-wds/ --dataset wds/ --train-split 'imagenet_w21-train-{0000..1983}.tar|12741248' --val-split 'imagenet_w21-train-{1984..2047}.tar|411008' --num-classes 21842

[MaKaNu]

I also don't understand how to use the dataset if I downloaded it via datasets package.

Something like this:

python train.py --dataset timm/imagenet-1k-wds --data-dir ~/.cache/huggingface/datasets/timm___imagenet-1k-wds  --model resnet50 --sched cosine --epochs 1 --lr 0.05 --amp

and Returns:

RuntimeError: Found 0 images in subfolders of /home/test/.cache/huggingface/datasets/timm___imagenet-1k-wds. Supported image extensions are .png, .jpg, .jpeg

[rwightman]

@TheDarkKnight-21th hmm, that error wasn't really helpful as curl hides it behind a 22 exit code in both cases, I really think there is something wrong with the HFT being passed through in whatever env this is being run in, nothing else makes sense.

In any case, streaming isn't really very useful for long term training. With the downloaded form you need the same --dataset wds/ to tell it what dataset type to use. This is assuming that the cache dir you point to contains the .tar files and the _info.json file. I'm actually not sure what form it ends up in when datasets does the download, does it remain in .tar form or does datasets convert it?

When downloading wds datasets I usually recommend using the huggingface_cli tool to download as is ... also install hf-transfer

HF_TRANSFER=1 huggingface-cli download --repo-type dataset --local-dir /dest/dir timm/imagenet-1k-wds

[MaKaNu]

@rwightman yes for training I see the point for testing it could be very nice, if you just want to validate on a different system. Still wonder what the problem is since, since just curling the _info.json works:

curl -s -H "Authorization: Bearer $HFT" -f -L https://huggingface.co/datasets/timm/imagenet-12k-wds/resolve/main/_info.json

Only combination with train.py does create issues here.

Thanks for the hint with the download, now I was able to start a training.

[rwightman]

@MaKaNu you might get a bit more info from the output if you remove the '-s' in the CURL line along with the extra print... should at least give you a 401 vs 404. If you get a 401 the token is not being passed properly... could possibly be a shell issue, I always run from my python env in a bash shell.

[MaKaNu]

Perfect hint. It was a 401. And I misled myself that the token is correct, since the curl command worked in the shell, what I forgot was the forking aspect of the script.

Turning HFT="$(cat ~/.cache/huggingface/token)"; into export HFT="$(cat ~/.cache/huggingface/token)"; solved it.