Skip to content

PyTorchModel deployment infinite loop due to bad requirements.txt #3671

Open
@david-waterworth

Description

@david-waterworth

Describe the bug
I'm trying to deploy a pre-trained PyTorchModel that is dependent on a package that is not available on pypi

I couldn't find any documentation indicating if this is supported or not, but I added the filename to requirements.txt and placed the package .tar.gz file in the the code folder. When Sagemaker tries to create the endpoint it fails to install the package, but instead of failing to create the endpoint it continuously tries again - the log repeats the following every 2.5s

WARNING: Requirement 'warpspeed-multiclass-0.2.1.tar.gz' looks like a filename, but the file does not exist
...
subprocess.CalledProcessError: Command '['/opt/conda/bin/python', '-m', 'pip', 'install', '-r', '/opt/ml/model/code/requirements.txt']' returned non-zero exit s

To reproduce

Attempt to deploy a PyTorchModel (i.e. https://sagemaker-examples.readthedocs.io/en/latest/sagemaker-script-mode/pytorch_bert/deploy_bert_outputs.html) using requirements.txt that contains a file

Expected behaviour
If pip install fails, the endpoint deployment should fail

Screenshots or logs
If applicable, add screenshots or logs to help explain your problem.

System information
A description of your system. Please provide:

  • SageMaker Python SDK version: 2.133.0
  • Framework name (eg. PyTorch) or algorithm (eg. KMeans): PyTorchModel
  • Framework version: 1.13.1
  • Python version: 3.9
  • CPU or GPU: CPU
  • Custom Docker image (Y/N): N

Additional context
I fixed the symptom by identifying that the contents of the code folder get mounted as /opt/ml/model/code/ in the model container and using this path in requirements.txt, i.e.

/opt/ml/model/code/warpspeed-multiclass-0.2.1.tar.gz

rather than

warpspeed-multiclass-0.2.1.tar.gz

Metadata

Metadata

Type

No type

Projects

No projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions