Description
Describe the bug
When executing a transform job, the role is not passed from the Model or Estimator to the Transformer. Hence, down the line an error about permissions to read logs is encountered.
To reproduce
sagemaker.estimator.Model(
role="SagemakerTransformJobExecution",
image_uri = "custom-docker-img",
model_data="file://training_output/model.tar.gz"
).transformer(
instance_type = "local_gpu",
instance_count = 1,
output_path="file://transform_output"
).transform(
data='file://Manifest.json',
data_type="ManifestFile",
split_type='Line',
content_type='application/json',
)
The above is in local mode but looking at the sdk code this should make no difference (not any of the arguments used for the Model, Transformer or transform)
Expected behavior
The permissions to perform logs:DescribeLogStreams
over arn:aws:logs:eu-central-1:123456789101:log-group:/aws/sagemaker/TransformJobs:log-stream
were granted to the SagemakerTransformJobExecution
role. Hence I expected that role to be assumed.
Screenshots or logs
The error message I am getting:
arn:aws:sts::123456789101:assumed-role/SagemakerDevMachine/i-0f3b3b18092a62173 is not authorized to perform: logs:DescribeLogStreams on resource: arn:aws:logs:eu-central-1:1234567891011:log-group:/aws/sagemaker/TransformJobs:log-stream: because no identity-based policy allows the logs:DescribeLogStreams action
Note the offending role is SagemakerDevMachine
which is the instance role assigned to the EC2 instance I was running on.
System information
A description of your system. Please provide:
- SageMaker Python SDK version: 2.171.0
- Framework name (eg. PyTorch) or algorithm (eg. KMeans): PyTorch
- Framework version: N/A
- Python version: 3.7.16
- CPU or GPU: GPU
- Custom Docker image (Y/N): Y
Additional context
I looked at the sdk code. For comparison, when a training job is executed, the role is copied over from the estimator to the job config here
sagemaker-python-sdk/src/sagemaker/estimator.py
Line 2275 in 53fe9b7
When a Transformer is created here
sagemaker-python-sdk/src/sagemaker/model.py
Line 1508 in 53fe9b7