Seasoned ⭐️⭐️

Solved

Unable to run the CDF Transformation job created

19 replies
325 views

Hi,

I’m going through the Python SDK- CDF Transformation module in the DATA ENGINEER BASICS - TRANSFORM AND CONTEXTUALIZE course.

While I’m able to create the Transformation object, I’m unable to run due to some Authentication issues.

When I’m trying to run using the below code:

client.transformations.run(asset_transformation.id, wait=False)

I’m getting the following error :

Transformation job could not be created.
Error code: 403

API error: Invalid source/destination credentials: Could not authenticate with the OIDC credentials. Please check your credentials.

Request ID: 1a470c41-9001-989b-8f97-a2aaefdfd098

Is it due to some recent change in the Oidc credentials defined in the notebook currently?

Gaetan HelnessAnswer

MVP ⭐️⭐️⭐️⭐️⭐️

@Philips
Could you complete the last step 4 Clean up resources and start from scratch, or at least delete the transformation

client.transformations.delete(external_id=[f"{PREFIX}-assets", f"{PREFIX}-ts", f"{PREFIX}-dp"])

and recreate it.
The previous secret that was not working has been stored in the transformation and therefore the transformation needs to be recreated

Show previous replies

P

PhilipsAuthor

Seasoned ⭐️⭐️

You can then test the newly generated secret via

from cognite.client.credentials import OAuthClientCredentials

client_secret_auth = CogniteClient(ClientConfig(
    client_name="test-client",
    base_url=f"https://{CDF_CLUSTER}.cognitedata.com",
    project=COGNITE_PROJECT,
    credentials=OAuthClientCredentials(
    token_url=f"{AUTHORITY_URI}/oauth2/v2.0/token",
    client_id=CLIENT_ID,
    client_secret=CLIENT_SECRET,
    scopes = " ".join(SCOPES)
),
))

print(client_secret_auth.iam.token.inspect())

@Gaetan Helness , The above code did run without any error.

But when I’m trying to run the transformation (given below), the error remains the same as before:

client.transformations.run(asset_transformation.id, wait=False)

Error:CogniteAPIError: Invalid source/destination credentials: Could not authenticate with the OIDC credentials. Please check your credentials. | code: 403 | X-Request-ID: 42d5a64d-d981-9154-8e9c-e302a2082e38

Gaetan Helness

MVP ⭐️⭐️⭐️⭐️⭐️

@Philips thanks, did you regenerate a secret, updated it in the code and reran through the notebook?
What was the output of

print(client_secret_auth.iam.token.inspect())

The problem is coming from the secret according to the error message.

Gaetan Helness

MVP ⭐️⭐️⭐️⭐️⭐️

And if you can also send the output of

asset_transformation = client.transformations.retrieve_multiple(external_ids=[f'{PREFIX}-assets'])[0]
asset_transformation

P

PhilipsAuthor

Seasoned ⭐️⭐️

@Philips thanks, did you regenerate a secret, updated it in the code and reran through the notebook?
What was the output of

print(client_secret_auth.iam.token.inspect())

The problem is coming from the secret according to the error message.

Yes, I updated the secret in the code. The output is attached

output_cognite.txt

P

PhilipsAuthor

Seasoned ⭐️⭐️

And if you can also send the output of

asset_transformation = client.transformations.retrieve_multiple(external_ids=[f'{PREFIX}-assets'])[0]
asset_transformation

Output is attached

asset_tranform.txt

Gaetan Helness

MVP ⭐️⭐️⭐️⭐️⭐️

Thanks, everything looks quite good, it is hard to troubleshoot what is wrong.
We will have another look through your attachments and get back to you

Gaetan HelnessAnswer

MVP ⭐️⭐️⭐️⭐️⭐️

@Philips
Could you complete the last step 4 Clean up resources and start from scratch, or at least delete the transformation

client.transformations.delete(external_id=[f"{PREFIX}-assets", f"{PREFIX}-ts", f"{PREFIX}-dp"])

and recreate it.
The previous secret that was not working has been stored in the transformation and therefore the transformation needs to be recreated

P

PhilipsAuthor

Seasoned ⭐️⭐️

Yes, the cleaning up the transformation did the trick.

It is running fine now,

Thanks a lot everyone for the prompt support!