HTTPError: 400 Only one sdist may be uploaded per release (when using --skip-existing)

[nanonano]

Describe the bug
Since 15th April my uploads to test.pypi.org have started failing with the error:

HTTPError: 400 Only one sdist may be uploaded per release.

Not sure if something has changed, I thought if the --skip-existing flag was set this was okay?

Below is the GitHub actions log output.

Run twine upload --repository-url https://test.pypi.org/legacy/ --skip-existing dist/*
[2](https://github.com/BittyTax/BittyTax/actions/runs/8754325941/job/24025816451#step:6:2)
  twine upload --repository-url https://test.pypi.org/legacy/ --skip-existing dist/*
[3](https://github.com/BittyTax/BittyTax/actions/runs/8754325941/job/24025816451#step:6:3)
  shell: /usr/bin/bash -e {0}
[4](https://github.com/BittyTax/BittyTax/actions/runs/8754325941/job/24025816451#step:6:4)
  env:
[5](https://github.com/BittyTax/BittyTax/actions/runs/8754325941/job/24025816451#step:6:5)
    pythonLocation: /opt/hostedtoolcache/Python/3.12.3/x[6](https://github.com/BittyTax/BittyTax/actions/runs/8754325941/job/24025816451#step:6:6)4
6
    PKG_CONFIG_PATH: /opt/hostedtoolcache/Python/3.12.3/x64/lib/pkgconfig
[7](https://github.com/BittyTax/BittyTax/actions/runs/8754325941/job/24025816451#step:6:7)
    Python_ROOT_DIR: /opt/hostedtoolcache/Python/3.12.3/x64
[8](https://github.com/BittyTax/BittyTax/actions/runs/8754325941/job/24025816451#step:6:8)
    Python2_ROOT_DIR: /opt/hostedtoolcache/Python/3.12.3/x64
[9](https://github.com/BittyTax/BittyTax/actions/runs/8754325941/job/24025816451#step:6:9)
    Python3_ROOT_DIR: /opt/hostedtoolcache/Python/3.12.3/x64
[10](https://github.com/BittyTax/BittyTax/actions/runs/8754325941/job/24025816451#step:6:10)
    LD_LIBRARY_PATH: /opt/hostedtoolcache/Python/3.12.3/x64/lib
[11](https://github.com/BittyTax/BittyTax/actions/runs/8754325941/job/24025816451#step:6:11)
    TWINE_USERNAME: __token__
[12](https://github.com/BittyTax/BittyTax/actions/runs/8754325941/job/24025816451#step:6:12)
    TWINE_PASSWORD: ***
[13](https://github.com/BittyTax/BittyTax/actions/runs/8754325941/job/24025816451#step:6:14)
Uploading distributions to https://test.pypi.org/legacy/
[14](https://github.com/BittyTax/BittyTax/actions/runs/8754325941/job/24025816451#step:6:15)
Uploading BittyTax-0.5.3.dev3-py3-none-any.whl
[15](https://github.com/BittyTax/BittyTax/actions/runs/8754325941/job/24025816451#step:6:16)
25l
[16](https://github.com/BittyTax/BittyTax/actions/runs/8754325941/job/24025816451#step:6:17)
  0% ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 0.0/935.3 kB • --:-- • ?
[17](https://github.com/BittyTax/BittyTax/actions/runs/8754325941/job/24025816451#step:6:18)
  0% ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 0.0/935.3 kB • --:-- • ?
[18](https://github.com/BittyTax/BittyTax/actions/runs/8754325941/job/24025816451#step:6:19)
 28% ━━━━━━━━━━━╺━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 262.1/935.3 kB • 00:01 • 3.1 MB/s
[19](https://github.com/BittyTax/BittyTax/actions/runs/8754325941/job/24025816451#step:6:20)
100% ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 935.3/935.3 kB • 00:00 • 5.7 MB/s
[20](https://github.com/BittyTax/BittyTax/actions/runs/8754325941/job/24025816451#step:6:21)
100% ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 935.3/935.3 kB • 00:00 • 5.7 MB/s
[21](https://github.com/BittyTax/BittyTax/actions/runs/8754325941/job/24025816451#step:6:22)
100% ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 935.3/935.3 kB • 00:00 • 5.7 MB/s
[22](https://github.com/BittyTax/BittyTax/actions/runs/8754325941/job/24025816451#step:6:23)
100% ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 935.3/935.3 kB • 00:00 • 5.7 MB/s
[23](https://github.com/BittyTax/BittyTax/actions/runs/8754325941/job/24025816451#step:6:24)
100% ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 935.3/935.3 kB • 00:00 • 5.7 MB/s
[24](https://github.com/BittyTax/BittyTax/actions/runs/8754325941/job/24025816451#step:6:25)
25hWARNING  Skipping BittyTax-0.5.3.dev3-py3-none-any.whl because it appears to    
[25](https://github.com/BittyTax/BittyTax/actions/runs/8754325941/job/24025816451#step:6:26)
         already exist                                                          
[26](https://github.com/BittyTax/BittyTax/actions/runs/8754325941/job/24025816451#step:6:27)
Uploading bittytax-0.5.3.dev3.tar.gz
[27](https://github.com/BittyTax/BittyTax/actions/runs/8754325941/job/24025816451#step:6:28)
25l
[28](https://github.com/BittyTax/BittyTax/actions/runs/8754325941/job/24025816451#step:6:29)
  0% ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 0.0/910.7 kB • --:-- • ?
[29](https://github.com/BittyTax/BittyTax/actions/runs/8754325941/job/24025816451#step:6:30)
100% ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 910.7/910.7 kB • 00:00 • 224.2 MB/s
[30](https://github.com/BittyTax/BittyTax/actions/runs/8754325941/job/24025816451#step:6:31)
100% ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 910.7/910.7 kB • 00:00 • 224.2 MB/s
[31](https://github.com/BittyTax/BittyTax/actions/runs/8754325941/job/24025816451#step:6:32)
100% ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 910.7/910.7 kB • 00:00 • 224.2 MB/s
[32](https://github.com/BittyTax/BittyTax/actions/runs/8754325941/job/24025816451#step:6:33)
25hWARNING  Error during upload. Retry with the --verbose option for more details. 
[33](https://github.com/BittyTax/BittyTax/actions/runs/8754325941/job/24025816451#step:6:34)
ERROR    HTTPError: 400 Bad Request from https://test.pypi.org/legacy/          
[34](https://github.com/BittyTax/BittyTax/actions/runs/8754325941/job/24025816451#step:6:35)
         Only one sdist may be uploaded per release.                            
[35](https://github.com/BittyTax/BittyTax/actions/runs/8754325941/job/24025816451#step:6:36)
Error: Process completed with exit code 1.

Expected behavior

I would expect just the warning as before:

WARNING Skipping BittyTax-0.5.3.dev3.tar.gz because it appears to already exist

Run twine upload --repository-url https://test.pypi.org/legacy/ --skip-existing dist/*
[2](https://github.com/BittyTax/BittyTax/actions/runs/8629194815/job/23652859082#step:6:2)
  twine upload --repository-url https://test.pypi.org/legacy/ --skip-existing dist/*
[3](https://github.com/BittyTax/BittyTax/actions/runs/8629194815/job/23652859082#step:6:3)
  shell: /usr/bin/bash -e {0}
[4](https://github.com/BittyTax/BittyTax/actions/runs/8629194815/job/23652859082#step:6:4)
  env:
[5](https://github.com/BittyTax/BittyTax/actions/runs/8629194815/job/23652859082#step:6:5)
    pythonLocation: /opt/hostedtoolcache/Python/3.12.2/x[6](https://github.com/BittyTax/BittyTax/actions/runs/8629194815/job/23652859082#step:6:6)4
6
    PKG_CONFIG_PATH: /opt/hostedtoolcache/Python/3.12.2/x64/lib/pkgconfig
[7](https://github.com/BittyTax/BittyTax/actions/runs/8629194815/job/23652859082#step:6:7)
    Python_ROOT_DIR: /opt/hostedtoolcache/Python/3.12.2/x64
[8](https://github.com/BittyTax/BittyTax/actions/runs/8629194815/job/23652859082#step:6:8)
    Python2_ROOT_DIR: /opt/hostedtoolcache/Python/3.12.2/x64
[9](https://github.com/BittyTax/BittyTax/actions/runs/8629194815/job/23652859082#step:6:9)
    Python3_ROOT_DIR: /opt/hostedtoolcache/Python/3.12.2/x64
[10](https://github.com/BittyTax/BittyTax/actions/runs/8629194815/job/23652859082#step:6:10)
    LD_LIBRARY_PATH: /opt/hostedtoolcache/Python/3.12.2/x64/lib
[11](https://github.com/BittyTax/BittyTax/actions/runs/8629194815/job/23652859082#step:6:11)
    TWINE_USERNAME: __token__
[12](https://github.com/BittyTax/BittyTax/actions/runs/8629194815/job/23652859082#step:6:12)
    TWINE_PASSWORD: ***
[13](https://github.com/BittyTax/BittyTax/actions/runs/8629194815/job/23652859082#step:6:14)
Uploading distributions to https://test.pypi.org/legacy/
[14](https://github.com/BittyTax/BittyTax/actions/runs/8629194815/job/23652859082#step:6:15)
Uploading BittyTax-0.5.3.dev3-py3-none-any.whl
[15](https://github.com/BittyTax/BittyTax/actions/runs/8629194815/job/23652859082#step:6:16)
25l
[16](https://github.com/BittyTax/BittyTax/actions/runs/8629194815/job/23652859082#step:6:17)
  0% ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 0.0/928.6 kB • --:-- • ?
[17](https://github.com/BittyTax/BittyTax/actions/runs/8629194815/job/23652859082#step:6:18)
 65% ━━━━━━━━━━━━━━━━━━━━━━━━━━╺━━━━━━━━━━━━━ 606.2/928.6 kB • 00:01 • 30.0 MB/s
[18](https://github.com/BittyTax/BittyTax/actions/runs/8629194815/job/23652859082#step:6:19)
100% ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 928.6/928.6 kB • 00:00 • 36.8 MB/s
[19](https://github.com/BittyTax/BittyTax/actions/runs/8629194815/job/23652859082#step:6:20)
100% ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 928.6/928.6 kB • 00:00 • 36.8 MB/s
[20](https://github.com/BittyTax/BittyTax/actions/runs/8629194815/job/23652859082#step:6:21)
100% ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 928.6/928.6 kB • 00:00 • 36.8 MB/s
[21](https://github.com/BittyTax/BittyTax/actions/runs/8629194815/job/23652859082#step:6:22)
100% ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 928.6/928.6 kB • 00:00 • 36.8 MB/s
[22](https://github.com/BittyTax/BittyTax/actions/runs/8629194815/job/23652859082#step:6:23)
100% ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 928.6/928.6 kB • 00:00 • 36.8 MB/s
[23](https://github.com/BittyTax/BittyTax/actions/runs/8629194815/job/23652859082#step:6:24)
25hWARNING  Skipping BittyTax-0.5.3.dev3-py3-none-any.whl because it appears to    
[24](https://github.com/BittyTax/BittyTax/actions/runs/8629194815/job/23652859082#step:6:25)
         already exist                                                          
[25](https://github.com/BittyTax/BittyTax/actions/runs/8629194815/job/23652859082#step:6:26)
Uploading BittyTax-0.5.3.dev3.tar.gz
[26](https://github.com/BittyTax/BittyTax/actions/runs/8629194815/job/23652859082#step:6:27)
25l
[27](https://github.com/BittyTax/BittyTax/actions/runs/8629194815/job/23652859082#step:6:28)
  0% ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 0.0/905.3 kB • --:-- • ?
[28](https://github.com/BittyTax/BittyTax/actions/runs/8629194815/job/23652859082#step:6:29)
100% ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 905.3/905.3 kB • 00:00 • 155.5 MB/s
[29](https://github.com/BittyTax/BittyTax/actions/runs/8629194815/job/23652859082#step:6:30)
100% ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 905.3/905.3 kB • 00:00 • 155.5 MB/s
[30](https://github.com/BittyTax/BittyTax/actions/runs/8629194815/job/23652859082#step:6:31)
100% ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 905.3/905.3 kB • 00:00 • 155.5 MB/s
[31](https://github.com/BittyTax/BittyTax/actions/runs/8629194815/job/23652859082#step:6:32)
25hWARNING  Skipping BittyTax-0.5.3.dev3.tar.gz because it appears to already exist

To Reproduce

My Platform

Current runner version: '2.315.0'
[2](https://github.com/BittyTax/BittyTax/actions/runs/8629194815/job/23652859082#step:1:2)
Operating System
[3](https://github.com/BittyTax/BittyTax/actions/runs/8629194815/job/23652859082#step:1:3)
  Ubuntu
[4](https://github.com/BittyTax/BittyTax/actions/runs/8629194815/job/23652859082#step:1:4)
  22.04.4
[5](https://github.com/BittyTax/BittyTax/actions/runs/8629194815/job/23652859082#step:1:5)
  LTS
[6](https://github.com/BittyTax/BittyTax/actions/runs/8629194815/job/23652859082#step:1:7)
Runner Image
[7](https://github.com/BittyTax/BittyTax/actions/runs/8629194815/job/23652859082#step:1:8)
  Image: ubuntu-22.04
[8](https://github.com/BittyTax/BittyTax/actions/runs/8629194815/job/23652859082#step:1:9)
  Version: 20240407.1.0
[9](https://github.com/BittyTax/BittyTax/actions/runs/8629194815/job/23652859082#step:1:10)
  Included Software: https://github.com/actions/runner-images/blob/ubuntu22/20240407.1/images/ubuntu/Ubuntu2204-Readme.md
[10](https://github.com/BittyTax/BittyTax/actions/runs/8629194815/job/23652859082#step:1:11)
  Image Release: https://github.com/actions/runner-images/releases/tag/ubuntu22%2F20240407.1

Additional context
None

[di]

The issue here seems to be the difference between bittytax-0.5.3.dev3.tar.gz (which is what you're trying to upload) and BittyTax-0.5.3.dev3.tar.gz (what you've already uploaded).

Packaging considers these the same file, because:

>>> from packaging.utils import parse_sdist_filename
>>> parse_sdist_filename('bittytax-0.5.3.dev3.tar.gz') == parse_sdist_filename('BittyTax-0.5.3.dev3.tar.gz')
True

but it looks like the logic in twine doesn't take this normalization into account: https://github.com/pypa/twine/blob/67e87ef4221123454403b11ee8f802e87fcc13fd/twine/repository.py#L229

...so it doesn't correctly skip the upload and attempts to do it anyways.

Our error message here is a little confusing though, because we also don't take normalization into account when checking for duplicate filenames:

warehouse/warehouse/forklift/legacy.py

Lines 910 to 939 in 5a2d6f8

	# Check to see if the file that was uploaded exists already or not.
	is_duplicate = _is_duplicate_file(request.db, filename, file_hashes)
	if is_duplicate:
	request.tm.doom()
	return HTTPOk()
	elif is_duplicate is not None:
	raise _exc_with_message(
	HTTPBadRequest,
	# Note: Changing this error message to something that doesn't
	# start with "File already exists" will break the
	# --skip-existing functionality in twine
	# ref: https://github.com/pypi/warehouse/issues/3482
	# ref: https://github.com/pypa/twine/issues/332
	"File already exists. See "
	+ request.help_url(_anchor="file-name-reuse")
	+ " for more information.",
	)
	# Check to see if the file that was uploaded exists in our filename log
	if request.db.query(
	request.db.query(Filename).filter(Filename.filename == filename).exists()
	).scalar():
	raise _exc_with_message(
	HTTPBadRequest,
	"This filename has already been used, use a "
	"different version. "
	"See "
	+ request.help_url(_anchor="file-name-reuse")
	+ " for more information.",
	)

And so this fails on our check for multiple source distributions instead:

warehouse/warehouse/forklift/legacy.py

Lines 941 to 953 in 5a2d6f8

	# Check to see if uploading this file would create a duplicate sdist
	# for the current release.
	if (
	form.filetype.data == "sdist"
	and request.db.query(
	request.db.query(File)
	.filter((File.release == release) & (File.packagetype == "sdist"))
	.exists()
	).scalar()
	):
	raise _exc_with_message(
	HTTPBadRequest, "Only one sdist may be uploaded per release."
	)

Fixing this is probably blocked on #12245, since there is not such a thing as a normalized source distribution filename in every case until that is resolved.

[nanonano]

Thanks for this, I didn't release the filename of the tar file had been changed to lowercase.

Now I need to figure out what has caused this change, the versions of build (1.2.1) and twine (5.0.0) are the same.

[di]

The change is caused by tools starting to support PEP 625. In your case, it is probably setuptools. In your case, bittytax-0.5.3.dev3.tar.gz is now the "right" filename for this, and nothing should change there.

Once #12245 is fixed, this should go away, and there's probably nothing you need to do here in the meantime.

[nanonano]

OK great that explains it, thanks for your help.

I'll probably just up-version to get around the issue.

[jpmckinney]

I don't know if this is also related to PEP 625, but I also tried to upload a package to test.pypi.org (using pypa/gh-action-pypi-publish@release/v1 with skip-existing: true, which is the same as twine --skip-existing), and I got:

ERROR    HTTPError: 400 Bad Request from https://test.pypi.org/legacy/          
         Only one sdist may be uploaded per release.  

I would have expected a warning.

At https://test.pypi.org/project/sphinxcontrib-opencontracting/#files, I can see that the sdist has two hyphens: sphinxcontrib-opencontracting-0.0.9.tar.gz

The file being uploaded (built with python -m build --sdist --wheel) has a hyphen only before the version: sphinxcontrib_opencontracting-0.0.9.tar.gz

Verbose info from pypa/gh-action-pypi-publish@release/v1 step:

Checking dist/sphinxcontrib_opencontracting-0.0.9-py3-none-any.whl: PASSED
Checking dist/sphinxcontrib_opencontracting-0.0.9.tar.gz: PASSED
Showing hash values of files to be uploaded:
/github/workspace/dist/sphinxcontrib_opencontracting-0.0.9.tar.gz

SHA256: 996837ed8abf4deb4938d4047dfe34de9a344e9ccfc005e538aee63be3e40c59
MD5: 2e4562014c683e026030b279b37189cb
BLAKE2-256: 32d0ce3c9412efbcb69bc1022d66b78b36012c4c196b6a5812cb3d1c1e3bf432

/github/workspace/dist/sphinxcontrib_opencontracting-0.0.9-py3-none-any.whl

SHA256: 2c1e3b6ec6f0ad29444cd149b650ac294df6e74fb5a745b6600429cb207104f6
MD5: 2ecad59c1d2f9e94fee80bbc0ae96b4f
BLAKE2-256: fa081a293eb89ce2a5720bb0275c37b390b49d47bc4c063a7bd73a7b38faae60

Uploading distributions to https://test.pypi.org/legacy/
INFO     dist/sphinxcontrib_opencontracting-0.0.9-py3-none-any.whl (10.1 KB)    
INFO     dist/sphinxcontrib_opencontracting-0.0.9.tar.gz (13.8 KB)              
INFO     password set by command options                                        
INFO     username: __token__                                                    
INFO     password: <hidden>                                                     
Uploading sphinxcontrib_opencontracting-0.0.9-py3-none-any.whl
INFO     Response from https://test.pypi.org/legacy/:                           
         400 File already exists                                                
         ('sphinxcontrib_opencontracting-0.0.9-py3-none-any.whl', with          
         blake2_256 hash                                                        
         'fa081a293eb89ce2a5720bb0275c37b390b49d47bc4c063a7bd73a7b38faae60').   
         See https://test.pypi.org/help/#file-name-reuse for more information.  
INFO     <html>                                                                 
          <head>                                                                
           <title>400 File already exists                                       
         ('sphinxcontrib_opencontracting-0.0.9-py3-none-any.whl', with          
         blake2_256 hash                                                        
         'fa081a293eb89ce2a5720bb0275c37b390b49d47bc4c063a7bd73a7b38faae60').   
         See https://test.pypi.org/help/#file-name-reuse for more               
         information.</title>                                                   
          </head>                                                               
          <body>                                                                
           <h1>400 File already exists                                          
         ('sphinxcontrib_opencontracting-0.0.9-py3-none-any.whl', with          
         blake2_256 hash                                                        
         'fa081a293eb89ce2a5720bb0275c37b390b49d47bc4c063a7bd73a7b38faae60').   
         See https://test.pypi.org/help/#file-name-reuse for more               
         information.</h1>                                                      
           The server could not comply with the request since it is either      
         malformed or otherwise incorrect.<br/><br/>                            
         File already exists                                                    
         (&#x27;sphinxcontrib_opencontracting-0.0.9-py3-none-any.whl&#x27;, with
         blake2_256 hash                                                        
         &#x27;fa081a293eb89ce2a5720bb0275c37b390b49d47bc4c063a7bd73a7b38faae60&
         #x27;). See https://test.pypi.org/help/#file-name-reuse for more       
         information.                                                           
                                                                                
                                                                                
          </body>                                                               
         </html>                                                                
WARNING  Skipping sphinxcontrib_opencontracting-0.0.9-py3-none-any.whl because  
         it appears to already exist                                            
Uploading sphinxcontrib_opencontracting-0.0.9.tar.gz
INFO     Response from https://test.pypi.org/legacy/:                           
         400 Only one sdist may be uploaded per release.                        
INFO     <html>                                                                 
          <head>                                                                
           <title>400 Only one sdist may be uploaded per release.</title>       
          </head>                                                               
          <body>                                                                
           <h1>400 Only one sdist may be uploaded per release.</h1>             
           The server could not comply with the request since it is either      
         malformed or otherwise incorrect.<br/><br/>                            
         Only one sdist may be uploaded per release.                            
                                                                                
                                                                                
          </body>                                                               
         </html>                                                                
ERROR    HTTPError: 400 Bad Request from https://test.pypi.org/legacy/          
         Only one sdist may be uploaded per release.                            

[di]

Yes, that is due to what I mentioned above:

Our error message here is a little confusing though, because we also don't take normalization into account when checking for duplicate filenames:

And so this fails on our check for multiple source distributions instead