AzureML SDKV2 - ml_client job stream error - Unauthorized

Question

I have a aml pipeline that uses azureml sdkv2. I submit the pipeline as:

def get_chained_credentials(client_id: str):
    try:
        managed_identity_creds = ManagedIdentityCredential(client_id=client_id)
        default_creds = DefaultAzureCredential()
        chained_creds = ChainedTokenCredential(managed_identity_creds, default_creds)
        return chained_creds
    except Exception as e:
        raise e

ml_client = get_client_workspace(
            get_chained_credentials(e.cluster_identity_id),
            e.subscription_id,
            e.resource_group,
            e.workspace_name,
        )
pipeline_run_job = ml_client.jobs.create_or_update(
                pipeline_job, experiment_name=e.experiment_name_forecast
            )
ml_client.jobs.stream(name=pipeline_run_job.name)

It submits the job on the compute cluster and the job completes as well, but it gives error on the stream:

The compute cluster has the identity attached to it. I am not sure why it gives authorization error when we start streaming. Also, this happens only when the job is submitted from devops. If I submit it manually from my compute it works fine with streaming.

trace:

[2024-05-28 11:30:01Z] Submitting 1 runs, first five are: 156e0e1b:dbc22428-a9da-4cdb-87c2-39ebd8961372
[2024-05-28 11:33:16Z] Completing processing run id dbc22428-a9da-4cdb-87c2-39ebd8961372.
[2024-05-28 11:33:17Z] Submitting 1 runs, first five are: 9f12649a:12a3ce89-f11e-452d-bb9f-0d3c4292778d
WARNING:azure.identity._credentials.chained:ChainedTokenCredential failed to retrieve a token from the included credentials.
Attempted credentials:
    ManagedIdentityCredential: ManagedIdentityCredential authentication unavailable. The requested identity has not been assigned to this resource.
    DefaultAzureCredential: Please run 'az login' to set up an account
To mitigate this issue, please refer to the troubleshooting guidelines here at https://aka.ms/azsdk/python/identity/defaultazurecredential/troubleshoot.
ERROR:__main__:Unable to complete the pipeline
Traceback (most recent call last):
  File "/opt/hostedtoolcache/Python/3.9.19/x64/lib/python3.9/runpy.py", line 197, in _run_module_as_main
    return _run_code(code, main_globals, None,
  File "/opt/hostedtoolcache/Python/3.9.19/x64/lib/python3.9/runpy.py", line 87, in _run_code
    exec(code, run_globals)
  File "/home/vsts/work/1/s/ml_service/pipelines/train_pipeline.py", line 308, in <module>
    main(args)
  File "/home/vsts/work/1/s/ml_service/pipelines/train_pipeline.py", line 208, in main
    raise excp
  File "/home/vsts/work/1/s/ml_service/pipelines/train_pipeline.py", line 205, in main
    ml_client.jobs.stream(name=pipeline_run_job.name)
  File "/opt/hostedtoolcache/Python/3.9.19/x64/lib/python3.9/site-packages/azure/core/tracing/decorator.py", line 78, in wrapper_use_tracer
    return func(*args, **kwargs)
  File "/opt/hostedtoolcache/Python/3.9.19/x64/lib/python3.9/site-packages/azure/ai/ml/_telemetry/activity.py", line 275, in wrapper
    return f(*args, **kwargs)
  File "/opt/hostedtoolcache/Python/3.9.19/x64/lib/python3.9/site-packages/azure/ai/ml/operations/_job_operations.py", line 788, in stream
    self._stream_logs_until_completion(
  File "/opt/hostedtoolcache/Python/3.9.19/x64/lib/python3.9/site-packages/azure/ai/ml/operations/_job_ops_helper.py", line 271, in stream_logs_until_completion
    _current_details: RunDetails = run_operations.get_run_details(job_name)