`rmtree` sometimes fails due to "directory not empty" on Linux

[christophfriedrich]

Bug report

Bug description:

I have a temp directory in which I have one zip file per day of a month (along the pattern ./agrisens_YYYYMMDD.zip) and the extracted folders (many files along the pattern ./agrisens_YYYYMMDD/foo.bar). The zip files are listed in a downloaded_files variable. My code to clear the temp directory (without deleting it entirely) is like this:

print('Cleaning up temporary files...')
for filename in downloaded_files:
    foldername = filename[0:-4]  # remove last 4 characters (i.e. the file extension (.zip))
    os.remove('temp/'+filename)  # delete zip file
    shutil.rmtree('temp/'+foldername)  # delete folder where zip file was extracted to (os.rmdir only deletes empty directories, but here we still have content in it (and don't want to go through it ourselves))
print('DONE')

Repeatedly, this loop failed half-way through:

Cleaning up temporary files...
Traceback (most recent call last):
  File "/datacube/Scripts/agrisens/dwd_data-v6.py", line 317, in <module>
    shutil.rmtree('temp/'+foldername)  # delete folder where zip file was extracted to (os.rmdir only deletes empty directories, but here we still have content in it (and don't want to go through it ourselves))
  File "/opt/conda/lib/python3.10/shutil.py", line 731, in rmtree
    onerror(os.rmdir, path, sys.exc_info())
  File "/opt/conda/lib/python3.10/shutil.py", line 729, in rmtree
    os.rmdir(path)
OSError: [Errno 39] Directory not empty: 'temp/agrisens_20231217'

Interestingly, when looking at the file system, the folder is actually empty!

(base) datacube@d6f03dd71d64:/datacube/Scripts/agrisens/temp$ ll
total 1222793
drwxrws--- 17 datacube 4010225     4096 Dec 18 17:16 ./
drwxrws---  5 datacube 4010225     4096 Dec 18 17:09 ../
drwxrws---  2 datacube 4010225     4096 Dec 18 17:16 agrisens_20231116/
drwxrws---  2 datacube 4010225     4096 Dec 18 16:39 agrisens_20231117/
-rw-rw----  1 datacube 4010225 86510075 Dec 18 16:38 agrisens_20231117.zip
drwxrws---  2 datacube 4010225     4096 Dec 18 16:39 agrisens_20231118/
-rw-rw----  1 datacube 4010225 89981192 Dec 18 16:38 agrisens_20231118.zip
...

(base) datacube@d6f03dd71d64:/datacube/Scripts/agrisens/temp/agrisens_20231217$ ll
total 1
drwxrws---  2 datacube 4010225 4096 Dec 18 18:06 ./
drwxrws--- 17 datacube 4010225 4096 Dec 18 18:06 ../
(base) datacube@d6f03dd71d64:/datacube/Scripts/agrisens/temp/agrisens_20231217$

So it's quite likely that some race condition happened so that the folder was not empty on the attempt to delete it, but shortly after.

Note also that the folder of the day before the criticised folder didn't get removed either. And neither the zip file of the day in question, despite the call to os.remove being before the call to shutil.rmtree in the loop.

---

The same issue also happened with the previous month:

Cleaning up temporary files...
Traceback (most recent call last):
  File "/datacube/Scripts/agrisens/dwd_data-v6.py", line 317, in <module>
    shutil.rmtree('temp/'+foldername)  # delete folder where zip file was extracted to (os.rmdir only deletes empty directories, but here we still have content in it (and don't want to go through it ourselves))
  File "/opt/conda/lib/python3.10/shutil.py", line 731, in rmtree
    onerror(os.rmdir, path, sys.exc_info())
  File "/opt/conda/lib/python3.10/shutil.py", line 729, in rmtree
    os.rmdir(path)
OSError: [Errno 39] Directory not empty: 'temp/agrisens_20231116'

In both cases, the fail happened quite exactly in the middle of the month -- might be coincidental, might be noteworthy.

---

Has this been discussed elsewhere?

I looked through open issues with "rmtree" and saw two that might be related:

• The bug sounds similar to shutil.rmtree will frequently fail on Windows under heavy load due to racy deletion #84324, but for me it's happening on Linux, not on Windows
• The actual bug might be workaroundable through shutil.rmtree and os.listdir cannot recover on error conditions #52769

Other notes
The file system being written to is a network storage that is mounted into a virtual machine, from where it is mounted into the Docker container where my actual code runs, so not the closest setup -> a good environment for race conditions to happen. To my knowledge, no Windows is involved anywhere in the chain.

Version

> python3 --version
Python 3.10.12

CPython versions tested on:

3.10

Operating systems tested on:

Linux

[nesergen]

Hello! I have the same problem. Raises an error about an non-empty folder, but really the folder is empty.
This behavior began after upgrade the system (Linux Mint 20.3 > 21 ) .The Python changed also from 3.8 to 3.10. With Python 3.8 all worked normally.

[hoodmane]

I am also having the same problem in Python 3.12.

[picnixz]

I think there are some flaky issues due to the OS itself (@barneygale maybe you have some idea?) I can suggest using ignore_errors=True to bypass errors or use an explicitly onexc handler to have a better control on the exceptions being ignored.

[hoodmane]

Thanks for the suggestion. What I currently have is:

            try:
                shutil.rmtree(self.build_dir)
            except OSError as e:
                if e.strerror == "Directory not empty":
                    # Not sure why this happens, but trying again seems to fix it usually?
                    shutil.rmtree(self.build_dir)
                else:
                    raise

which seems to reduce the frequency of the problem but not eliminate it entirely. Maybe I'll add a sleep for a few milliseconds before trying again, and do three tries. I bet that will be good enough, but I'll have to see.

[hoodmane]

So maybe:

def retrying_rmtree(d):
    for _ in range(3):
        try:
            return shutil.rmtree(d)
        except OSError as e:
            if e.strerror == "Directory not empty":
                # wait a bit and try again up to 3 tries
                time.sleep(0.01)
            else:
                raise
    raise RuntimeError(f"shutil.rmtree('{d}') failed with ENOTEMPTY three times")

[picnixz]

It's indeed a possibility to have a sleep/retry but I eventually think there's no real issue with shutil.rmtree. It's just that OSErrors are raised by default unless ignore_errors=True or an exception handler is specified. I'm going to close this one as wont fix unless someone can find a faithful reproducer which would highlight an issue in shutil.rmtree. However, I don't think the stdlib should do the "retry/sleep" (but we could perhaps document this)?

@barneygale WDYT? closing this one as wont fix or document the flakiness of shutil.rmtree in some cases? (or maybe there is an issue?)