Lmdb access slows down with number of iterations for LSUN

I am running the below DCGAN code for LSUN dataset with lmdb. I noticed that as the number of iterations increases the data loading becomes very slow (like after 4000 iterations with 4 or 16 workers the new batch takes a lot of time), is this a known issue or I am missing something ?

(https://github.com/pytorch/examples/tree/master/dcgan)