Skip to content

Transform job from Model does not pass role #3997

Open
@shakedel

Description

@shakedel

Describe the bug
When executing a transform job, the role is not passed from the Model or Estimator to the Transformer. Hence, down the line an error about permissions to read logs is encountered.

To reproduce

sagemaker.estimator.Model(
    role="SagemakerTransformJobExecution",
    image_uri = "custom-docker-img",
    model_data="file://training_output/model.tar.gz"
).transformer(
    instance_type = "local_gpu",
    instance_count = 1,
    output_path="file://transform_output"
).transform(
    data='file://Manifest.json',
    data_type="ManifestFile",
    split_type='Line',
    content_type='application/json',
)

The above is in local mode but looking at the sdk code this should make no difference (not any of the arguments used for the Model, Transformer or transform)

Expected behavior
The permissions to perform logs:DescribeLogStreams over arn:aws:logs:eu-central-1:123456789101:log-group:/aws/sagemaker/TransformJobs:log-stream were granted to the SagemakerTransformJobExecution role. Hence I expected that role to be assumed.

Screenshots or logs
The error message I am getting:

arn:aws:sts::123456789101:assumed-role/SagemakerDevMachine/i-0f3b3b18092a62173 is not authorized to perform: logs:DescribeLogStreams on resource: arn:aws:logs:eu-central-1:1234567891011:log-group:/aws/sagemaker/TransformJobs:log-stream: because no identity-based policy allows the logs:DescribeLogStreams action

Note the offending role is SagemakerDevMachine which is the instance role assigned to the EC2 instance I was running on.

System information
A description of your system. Please provide:

  • SageMaker Python SDK version: 2.171.0
  • Framework name (eg. PyTorch) or algorithm (eg. KMeans): PyTorch
  • Framework version: N/A
  • Python version: 3.7.16
  • CPU or GPU: GPU
  • Custom Docker image (Y/N): Y

Additional context
I looked at the sdk code. For comparison, when a training job is executed, the role is copied over from the estimator to the job config here

config = _Job._load_config(inputs, estimator)

When a Transformer is created here
return Transformer(
, there is no handing over of the role from the Model (if creating a Transformer from an Estimator - the role is passed to the Model, but again is not passed to Transformer - resulting in the same error).

Metadata

Metadata

Assignees

Type

No type

Projects

No projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions