Solved

Unable to run the CDF Transformation job created

  • 24 April 2023
  • 19 replies
  • 181 views
P
Rank
Userlevel 1
Badge +2

Hi,

I’m going through the Python SDK- CDF Transformation module in the  DATA ENGINEER BASICS - TRANSFORM AND CONTEXTUALIZE course.

While I’m able to create the Transformation  object, I’m unable to run due to some Authentication issues. 

When I’m trying to run using the below code:

client.transformations.run(asset_transformation.id, wait=False)

I’m getting the following error :

Transformation job could not be created.
Error code: 403

API error: Invalid source/destination credentials: Could not authenticate with the OIDC credentials. Please check your credentials.

Request ID: 1a470c41-9001-989b-8f97-a2aaefdfd098

Is it due to some recent change in the Oidc credentials defined in the notebook currently?

 

icon

Best answer by Gaetan Helness 26 April 2023, 11:51

View original

19 replies

Sofie Haug
Rank
Userlevel 3
Badge +6

Hi @Philips 

Could are able to authenticate in the first cell in the notebook?

Sofie | Cognite Academy
G
Rank
Userlevel 3
Badge

Thanks for your feedback. 
It seems like an access right issue. 

Did you generate a secret and added it to the code? 
Did you encounter any errors earlier in the notebook, like in the first steps on the 
 

print(client.iam.token.inspect().projects)

And what information do you get via

print(client.iam.token.inspect())

 

P
Rank
Userlevel 1
Badge +2

Hi @Sofie Haug , @Gaetan Helness 

Thanks for the response.

Yes, I was able to authenticate the first cell in the notebook (i.e. I added the secret that i generated in the course).

When this statement is run,  print(client.iam.token.inspect().projects) I encountered no error.

Following the output I got:

[{ "url_name": "cdf-fundamentals", "groups": [ 7233047623736400 ] }, { "url_name": "infield-training", "groups": [ 1376085367190226 ] }, { "url_name": "publicdata", "groups": [ 1701516184810448, 4860375157547584 ] }, { "url_name": "de-transformations", "groups": [ 3974891759857484 ] }]

 

The information I get via below is attached.

print(client.iam.token.inspect())

best regards,

philips

 

 

 

 

1 Attachment
roman.chesnokov
Rank
Userlevel 3

It seems like an error with generating secrets from the course. @Rahul Kumar , could you look into it? 

G
Rank
Userlevel 3
Badge

Yes I agree with Roman

@Philips under the section Create credentials for use in Transformations, could you try to run this code?
This will try to authenticate with the client secret provided above in the notebook

from cognite.client.data_classes import OidcCredentials

creds = OidcCredentials(
        client_id = CLIENT_ID,
        client_secret = CLIENT_SECRET,
        scopes = " ".join(SCOPES),
        token_uri = f"{AUTHORITY_URI}/oauth2/v2.0/token",
        cdf_project_name = COGNITE_PROJECT
)

cnf_secret = ClientConfig(
    client_name="test-client",
    base_url=f"https://{CDF_CLUSTER}.cognitedata.com",
    project=COGNITE_PROJECT,
    credentials=creds,
)
client_secret_auth = CogniteClient(cnf_secret)

print(client_secret_auth.iam.token.inspect())

 

G
Rank
Userlevel 3
Badge

The token inspect you printed is actually authenticated with your user, but great that works

P
Rank
Userlevel 1
Badge +2

Hi Gaetan, 

On running the code you gave, I’m getting below error.

AttributeError: 'OidcCredentials' object has no attribute 'authorization_header'

 

R
Rank
Userlevel 2
Badge +6

Hello Philips, 

Please generate the new secret in the course  and use newly generated secret for auth in the code  

Regards
Kumar 

Rahul Kumar
G
Rank
Userlevel 3
Badge

You can then test the newly generated secret via

 

from cognite.client.credentials import OAuthClientCredentials

client_secret_auth = CogniteClient(ClientConfig(
    client_name="test-client",
    base_url=f"https://{CDF_CLUSTER}.cognitedata.com",
    project=COGNITE_PROJECT,
    credentials=OAuthClientCredentials(
    token_url=f"{AUTHORITY_URI}/oauth2/v2.0/token",
    client_id=CLIENT_ID,
    client_secret=CLIENT_SECRET,
    scopes = " ".join(SCOPES)
),
))

print(client_secret_auth.iam.token.inspect())

 

G
Rank
Userlevel 3
Badge

@Philips let us know when you had a chance to test it out, but generating a new secret should fix the issue

P
Rank
Userlevel 1
Badge +2

You can then test the newly generated secret via

 

from cognite.client.credentials import OAuthClientCredentials

client_secret_auth = CogniteClient(ClientConfig(
    client_name="test-client",
    base_url=f"https://{CDF_CLUSTER}.cognitedata.com",
    project=COGNITE_PROJECT,
    credentials=OAuthClientCredentials(
    token_url=f"{AUTHORITY_URI}/oauth2/v2.0/token",
    client_id=CLIENT_ID,
    client_secret=CLIENT_SECRET,
    scopes = " ".join(SCOPES)
),
))

print(client_secret_auth.iam.token.inspect())

 

@Gaetan Helness , The above code did run without any error.

But when I’m trying to run the transformation (given below), the error remains the same as before:

client.transformations.run(asset_transformation.id, wait=False)

 

Error:CogniteAPIError: Invalid source/destination credentials: Could not authenticate with the OIDC credentials. Please check your credentials. | code: 403 | X-Request-ID: 42d5a64d-d981-9154-8e9c-e302a2082e38

G
Rank
Userlevel 3
Badge

@Philips thanks, did you regenerate a secret, updated it in the code and reran through the notebook? 
What was the output of 

print(client_secret_auth.iam.token.inspect())

The problem is coming from the secret according to the error message. 

 

G
Rank
Userlevel 3
Badge

And if you can also send the output of
 

asset_transformation = client.transformations.retrieve_multiple(external_ids=[f'{PREFIX}-assets'])[0]
asset_transformation

 

P
Rank
Userlevel 1
Badge +2

@Philips thanks, did you regenerate a secret, updated it in the code and reran through the notebook? 
What was the output of 

print(client_secret_auth.iam.token.inspect())

The problem is coming from the secret according to the error message. 

 

Yes, I updated the secret in the code. The output is attached

1 Attachment
P
Rank
Userlevel 1
Badge +2

And if you can also send the output of
 

asset_transformation = client.transformations.retrieve_multiple(external_ids=[f'{PREFIX}-assets'])[0]
asset_transformation

 

Output is attached

1 Attachment
G
Rank
Userlevel 3
Badge

Thanks, everything looks quite good, it is hard to troubleshoot what is wrong. 
We will have another look through your attachments and get back to you

G
Rank
Userlevel 3
Badge

@Philips 
Could you complete the last step 4 Clean up resources and start from scratch, or at least delete the transformation

client.transformations.delete(external_id=[f"{PREFIX}-assets", f"{PREFIX}-ts", f"{PREFIX}-dp"])

and recreate it. 
The previous secret that was not working has been stored in the transformation and therefore the transformation needs to be recreated

P
Rank
Userlevel 1
Badge +2

Yes, the cleaning up the transformation did the trick.

It is running fine now, 

Thanks a lot everyone for the prompt support!

G
Rank
Userlevel 3
Badge

Thanks for your patience and help for troubleshooting. 
Kudos to @roman.chesnokov , @Sofie Haug and @Rahul Kumar for the help as well

Reply


Terms & ConditionsCookie Settings