Hi,
I’m going through the Python SDK- CDF Transformation module in the DATA ENGINEER BASICS - TRANSFORM AND CONTEXTUALIZE course.
While I’m able to create the Transformation object, I’m unable to run due to some Authentication issues.
When I’m trying to run using the below code:
client.transformations.run(asset_transformation.id, wait=False)
I’m getting the following error :
Transformation job could not be created.
Error code: 403
API error: Invalid source/destination credentials: Could not authenticate with the OIDC credentials. Please check your credentials.
Request ID: 1a470c41-9001-989b-8f97-a2aaefdfd098
Is it due to some recent change in the Oidc credentials defined in the notebook currently?
Thanks for your patience and help for troubleshooting.
Kudos to @roman.chesnokov , @Sofie Haug and @Rahul Kumar for the help as well
Yes, the cleaning up the transformation did the trick.
It is running fine now,
Thanks a lot everyone for the prompt support!
@Philips
Could you complete the last step 4 Clean up resources and start from scratch, or at least delete the transformation
client.transformations.delete(external_id=df"{PREFIX}-assets", f"{PREFIX}-ts", f"{PREFIX}-dp"])
and recreate it.
The previous secret that was not working has been stored in the transformation and therefore the transformation needs to be recreated
Thanks, everything looks quite good, it is hard to troubleshoot what is wrong.
We will have another look through your attachments and get back to you
And if you can also send the output of
asset_transformation = client.transformations.retrieve_multiple(external_ids==f'{PREFIX}-assets']))0]
asset_transformation
Output is attached
@Philips thanks, did you regenerate a secret, updated it in the code and reran through the notebook?
What was the output of
print(client_secret_auth.iam.token.inspect())
The problem is coming from the secret according to the error message.
Yes, I updated the secret in the code. The output is attached
And if you can also send the output of
asset_transformation = client.transformations.retrieve_multiple(external_ids==f'{PREFIX}-assets']))0]
asset_transformation
@Philips thanks, did you regenerate a secret, updated it in the code and reran through the notebook?
What was the output of
print(client_secret_auth.iam.token.inspect())
The problem is coming from the secret according to the error message.
You can then test the newly generated secret via
from cognite.client.credentials import OAuthClientCredentials
client_secret_auth = CogniteClient(ClientConfig(
client_name="test-client",
base_url=f"https://{CDF_CLUSTER}.cognitedata.com",
project=COGNITE_PROJECT,
credentials=OAuthClientCredentials(
token_url=f"{AUTHORITY_URI}/oauth2/v2.0/token",
client_id=CLIENT_ID,
client_secret=CLIENT_SECRET,
scopes = " ".join(SCOPES)
),
))
print(client_secret_auth.iam.token.inspect())
@Gaetan Helness , The above code did run without any error.
But when I’m trying to run the transformation (given below), the error remains the same as before:
client.transformations.run(asset_transformation.id, wait=False)
Error:CogniteAPIError: Invalid source/destination credentials: Could not authenticate with the OIDC credentials. Please check your credentials. | code: 403 | X-Request-ID: 42d5a64d-d981-9154-8e9c-e302a2082e38
@Philips let us know when you had a chance to test it out, but generating a new secret should fix the issue
You can then test the newly generated secret via
from cognite.client.credentials import OAuthClientCredentials
client_secret_auth = CogniteClient(ClientConfig(
client_name="test-client",
base_url=f"https://{CDF_CLUSTER}.cognitedata.com",
project=COGNITE_PROJECT,
credentials=OAuthClientCredentials(
token_url=f"{AUTHORITY_URI}/oauth2/v2.0/token",
client_id=CLIENT_ID,
client_secret=CLIENT_SECRET,
scopes = " ".join(SCOPES)
),
))
print(client_secret_auth.iam.token.inspect())
Hello Philips,
Please generate the new secret in the course and use newly generated secret for auth in the code
Regards
Kumar
Hi Gaetan,
On running the code you gave, I’m getting below error.
AttributeError: 'OidcCredentials' object has no attribute 'authorization_header'
The token inspect you printed is actually authenticated with your user, but great that works
Yes I agree with Roman
@Philips under the section Create credentials for use in Transformations, could you try to run this code?
This will try to authenticate with the client secret provided above in the notebook
from cognite.client.data_classes import OidcCredentials
creds = OidcCredentials(
client_id = CLIENT_ID,
client_secret = CLIENT_SECRET,
scopes = " ".join(SCOPES),
token_uri = f"{AUTHORITY_URI}/oauth2/v2.0/token",
cdf_project_name = COGNITE_PROJECT
)
cnf_secret = ClientConfig(
client_name="test-client",
base_url=f"https://{CDF_CLUSTER}.cognitedata.com",
project=COGNITE_PROJECT,
credentials=creds,
)
client_secret_auth = CogniteClient(cnf_secret)
print(client_secret_auth.iam.token.inspect())
It seems like an error with generating secrets from the course. @Rahul Kumar , could you look into it?
Hi @Sofie Haug , @Gaetan Helness
Thanks for the response.
Yes, I was able to authenticate the first cell in the notebook (i.e. I added the secret that i generated in the course).
When this statement is run, print(client.iam.token.inspect().projects)
I encountered no error.
Following the output I got:
>{ "url_name": "cdf-fundamentals", "groups": s 7233047623736400 ] }, { "url_name": "infield-training", "groups": s 1376085367190226 ] }, { "url_name": "publicdata", "groups": s 1701516184810448, 4860375157547584 ] }, { "url_name": "de-transformations", "groups": s 3974891759857484 ] }]
The information I get via below is attached.
print(client.iam.token.inspect())
best regards,
philips
Thanks for your feedback.
It seems like an access right issue.
Did you generate a secret and added it to the code?
Did you encounter any errors earlier in the notebook, like in the first steps on the
print(client.iam.token.inspect().projects)
And what information do you get via
print(client.iam.token.inspect())
Hi @Philips
Could are able to authenticate in the first cell in the notebook?