Skip to main content
Solved

Unable to run the CDF Transformation job created


Forum|alt.badge.img+3

Hi,

I’m going through the Python SDK- CDF Transformation module in the  DATA ENGINEER BASICS - TRANSFORM AND CONTEXTUALIZE course.

While I’m able to create the Transformation  object, I’m unable to run due to some Authentication issues. 

When I’m trying to run using the below code:

client.transformations.run(asset_transformation.id, wait=False)

I’m getting the following error :

Transformation job could not be created.
Error code: 403

API error: Invalid source/destination credentials: Could not authenticate with the OIDC credentials. Please check your credentials.

Request ID: 1a470c41-9001-989b-8f97-a2aaefdfd098

Is it due to some recent change in the Oidc credentials defined in the notebook currently?

 

Best answer by Gaetan Helness

@Philips 
Could you complete the last step 4 Clean up resources and start from scratch, or at least delete the transformation

client.transformations.delete(external_id=[f"{PREFIX}-assets", f"{PREFIX}-ts", f"{PREFIX}-dp"])

and recreate it. 
The previous secret that was not working has been stored in the transformation and therefore the transformation needs to be recreated

View original
Did this topic help you find an answer to your question?

19 replies

Sofie Haug
Practitioner
Forum|alt.badge.img+7
  • Cognite Academy Instructor
  • 148 replies
  • April 24, 2023

Hi @Philips 

Could are able to authenticate in the first cell in the notebook?


Forum|alt.badge.img
  • Seasoned Practitioner
  • 78 replies
  • April 24, 2023

Thanks for your feedback. 
It seems like an access right issue. 

Did you generate a secret and added it to the code? 
Did you encounter any errors earlier in the notebook, like in the first steps on the 
 

print(client.iam.token.inspect().projects)

And what information do you get via

print(client.iam.token.inspect())

 


Forum|alt.badge.img+3
  • Author
  • Committed
  • 12 replies
  • April 25, 2023

Hi @Sofie Haug , @Gaetan Helness 

Thanks for the response.

Yes, I was able to authenticate the first cell in the notebook (i.e. I added the secret that i generated in the course).

When this statement is run,  print(client.iam.token.inspect().projects) I encountered no error.

Following the output I got:

[{ "url_name": "cdf-fundamentals", "groups": [ 7233047623736400 ] }, { "url_name": "infield-training", "groups": [ 1376085367190226 ] }, { "url_name": "publicdata", "groups": [ 1701516184810448, 4860375157547584 ] }, { "url_name": "de-transformations", "groups": [ 3974891759857484 ] }]

 

The information I get via below is attached.

print(client.iam.token.inspect())

best regards,

philips

 

 

 

 


roman.chesnokov
Seasoned Practitioner

It seems like an error with generating secrets from the course. @Rahul Kumar , could you look into it? 


Forum|alt.badge.img
  • Seasoned Practitioner
  • 78 replies
  • April 25, 2023

Yes I agree with Roman

@Philips under the section Create credentials for use in Transformations, could you try to run this code?
This will try to authenticate with the client secret provided above in the notebook

from cognite.client.data_classes import OidcCredentials

creds = OidcCredentials(
        client_id = CLIENT_ID,
        client_secret = CLIENT_SECRET,
        scopes = " ".join(SCOPES),
        token_uri = f"{AUTHORITY_URI}/oauth2/v2.0/token",
        cdf_project_name = COGNITE_PROJECT
)

cnf_secret = ClientConfig(
    client_name="test-client",
    base_url=f"https://{CDF_CLUSTER}.cognitedata.com",
    project=COGNITE_PROJECT,
    credentials=creds,
)
client_secret_auth = CogniteClient(cnf_secret)

print(client_secret_auth.iam.token.inspect())

 


Forum|alt.badge.img
  • Seasoned Practitioner
  • 78 replies
  • April 25, 2023

The token inspect you printed is actually authenticated with your user, but great that works


Forum|alt.badge.img+3
  • Author
  • Committed
  • 12 replies
  • April 25, 2023

Hi Gaetan, 

On running the code you gave, I’m getting below error.

AttributeError: 'OidcCredentials' object has no attribute 'authorization_header'

 


Forum|alt.badge.img+6
  • Cognite Academy Instructor
  • 61 replies
  • April 25, 2023

Hello Philips, 

Please generate the new secret in the course  and use newly generated secret for auth in the code  

Regards
Kumar 


Forum|alt.badge.img
  • Seasoned Practitioner
  • 78 replies
  • April 25, 2023

You can then test the newly generated secret via

 

from cognite.client.credentials import OAuthClientCredentials

client_secret_auth = CogniteClient(ClientConfig(
    client_name="test-client",
    base_url=f"https://{CDF_CLUSTER}.cognitedata.com",
    project=COGNITE_PROJECT,
    credentials=OAuthClientCredentials(
    token_url=f"{AUTHORITY_URI}/oauth2/v2.0/token",
    client_id=CLIENT_ID,
    client_secret=CLIENT_SECRET,
    scopes = " ".join(SCOPES)
),
))

print(client_secret_auth.iam.token.inspect())

 


Forum|alt.badge.img
  • Seasoned Practitioner
  • 78 replies
  • April 25, 2023

@Philips let us know when you had a chance to test it out, but generating a new secret should fix the issue


Forum|alt.badge.img+3
  • Author
  • Committed
  • 12 replies
  • April 26, 2023
Gaetan Helness wrote:

You can then test the newly generated secret via

 

from cognite.client.credentials import OAuthClientCredentials

client_secret_auth = CogniteClient(ClientConfig(
    client_name="test-client",
    base_url=f"https://{CDF_CLUSTER}.cognitedata.com",
    project=COGNITE_PROJECT,
    credentials=OAuthClientCredentials(
    token_url=f"{AUTHORITY_URI}/oauth2/v2.0/token",
    client_id=CLIENT_ID,
    client_secret=CLIENT_SECRET,
    scopes = " ".join(SCOPES)
),
))

print(client_secret_auth.iam.token.inspect())

 

@Gaetan Helness , The above code did run without any error.

But when I’m trying to run the transformation (given below), the error remains the same as before:

client.transformations.run(asset_transformation.id, wait=False)

 

Error:CogniteAPIError: Invalid source/destination credentials: Could not authenticate with the OIDC credentials. Please check your credentials. | code: 403 | X-Request-ID: 42d5a64d-d981-9154-8e9c-e302a2082e38


Forum|alt.badge.img
  • Seasoned Practitioner
  • 78 replies
  • April 26, 2023

@Philips thanks, did you regenerate a secret, updated it in the code and reran through the notebook? 
What was the output of 

print(client_secret_auth.iam.token.inspect())

The problem is coming from the secret according to the error message. 

 


Forum|alt.badge.img
  • Seasoned Practitioner
  • 78 replies
  • April 26, 2023

And if you can also send the output of
 

asset_transformation = client.transformations.retrieve_multiple(external_ids=[f'{PREFIX}-assets'])[0]
asset_transformation

 


Forum|alt.badge.img+3
  • Author
  • Committed
  • 12 replies
  • April 26, 2023
Gaetan Helness wrote:

@Philips thanks, did you regenerate a secret, updated it in the code and reran through the notebook? 
What was the output of 

print(client_secret_auth.iam.token.inspect())

The problem is coming from the secret according to the error message. 

 

Yes, I updated the secret in the code. The output is attached


Forum|alt.badge.img+3
  • Author
  • Committed
  • 12 replies
  • April 26, 2023
Gaetan Helness wrote:

And if you can also send the output of
 

asset_transformation = client.transformations.retrieve_multiple(external_ids=[f'{PREFIX}-assets'])[0]
asset_transformation

 

Output is attached


Forum|alt.badge.img
  • Seasoned Practitioner
  • 78 replies
  • April 26, 2023

Thanks, everything looks quite good, it is hard to troubleshoot what is wrong. 
We will have another look through your attachments and get back to you


Forum|alt.badge.img
  • Seasoned Practitioner
  • 78 replies
  • Answer
  • April 26, 2023

@Philips 
Could you complete the last step 4 Clean up resources and start from scratch, or at least delete the transformation

client.transformations.delete(external_id=[f"{PREFIX}-assets", f"{PREFIX}-ts", f"{PREFIX}-dp"])

and recreate it. 
The previous secret that was not working has been stored in the transformation and therefore the transformation needs to be recreated


Forum|alt.badge.img+3
  • Author
  • Committed
  • 12 replies
  • April 26, 2023

Yes, the cleaning up the transformation did the trick.

It is running fine now, 

Thanks a lot everyone for the prompt support!


Forum|alt.badge.img
  • Seasoned Practitioner
  • 78 replies
  • April 26, 2023

Thanks for your patience and help for troubleshooting. 
Kudos to @roman.chesnokov , @Sofie Haug and @Rahul Kumar for the help as well


Reply


Cookie Policy

We use cookies to enhance and personalize your experience. If you accept you agree to our full cookie policy. Learn more about our cookies.

 
Cookie Settings