For Step 5. you could also add change the tenantID/clientID/client secret in any devops tool (CI/CD) that are used to deploy functions/transformations,... I would add a step 6 if you are completely moving away from the previous tenantStep 6: Unregister the previous IdP from the domain. Please reach out to support@cognite.com to unregister the previous IdP for the domain with the following information Previous tenant id.
Great guide, thank you for writing this!cc @Anita Hæhre
From a first look, the configuration looks correct. It could be that some entries in the “START_DATETIME” column are None and therefore it would fail on thatadding a where clause check that “START_DATETIME” is not null for exampleOr it could be that the format of the timestamp is not correct, you can try to cast it to a datetime objectCAST(START_DATETIME as datetime) Another temporary solution is to send the data to RAW where you would see what data is sent or not and from there, you can move to sending the data directly to time_seriesFor example, like the below destination: database: "Test-DB" table: "Test-Table" primary-key: "{UB}:{Well}:{TSOBJ}"
Authentication by API keys is deprecated, thank you, we will update the documentationYou can refer to this page insteadhttps://github.com/cognitedata/cognite-sdk-js/blob/master/guides/authentication.md
Hi, You can use the API https://api-docs.cognite.com/20230101/tag/Extraction-Pipelines-Runs/operation/createRuns or the SDK for that purpose https://cognite-sdk-python.readthedocs-hosted.com/en/latest/data_ingestion.html#cognite.client._api.extractionpipelines.ExtractionPipelineRunsAPI.create If using our python extractor-utils library, you get that out of the boxhttps://cognite-extractor-utils.readthedocs-hosted.com/en/latest/uploader.html#extraction-pipeline-runs Hope that helps, let me know if any additional questions
If you have access to Cognite Academy, I would also recommend going through the courses that go through those steps in details and with hands-on exerciseshttps://learn.cognite.com/page/cognite-learn-catalog#role_data-engineer
Hi, I can guide a bit towards ingesting data It will be fine to login using Google Colab, using the code presented here https://cognite-sdk-python.readthedocs-hosted.com/en/latest/quickstart.html#instantiate-a-new-client Ypu can use this in a previous cell to install the required cognite-sdk !pip install cognite-sdk Instead of having the secret as clear text or environment variable, you can use input in Google Colab to enter the secret at runtime client_secret = input('Enter your secret: ') After you initiated the client, to create assets/asset hierarchy, I refer you to the endpoint in our SDKhttps://cognite-sdk-python.readthedocs-hosted.com/en/latest/core_data_model.html#create-assetshttps://cognite-sdk-python.readthedocs-hosted.com/en/latest/core_data_model.html#create-asset-hierarchyThey should be plenty of examples to get you started Referring upload data to CSV, you can do that using RAWhttps://docs.cognite.com/cdf/integration/guides/extraction/raw_explorer/You can also use the S
Yes, great suggestion @Johannes Hovda , this should work as expected :) Let us know how it goes @Olav Alstad
Hi. We have a specific endpoint to get the latest datapoint, so that should be no problem.However, getting the first one might be trickier, as you mentioned, you might need to get all datapoints and get the first one from the list which is far from ideal indeed. I will check with our engineering team if there is a better solution.
yes, adding service principal (app registration) to the relevant AD groups that are mapping to the CDF groups giving the required capabilities
Hi Chris,Sorry for the late reply. Are you trying to create or update events? When specifying the id, since it is an auto-generated field, the action is going to be UPDATE only, as you see in the endpoint. The error message mentions that when trying to update, those events were not found, they will not be created though since id cannot be used for creation (only external_id)If you want to update only existing events and avoid error, you can do on join on _cdf.events table, which contains existing CDF events
Hello,To connect to a private project, while requesting it, you will need to connect it to an organization/domain name. https://docs.cognite.com/cdf/sign-up/#step-1-choose-the-domain-name-and-cdf-adminThis organization name is connected to an Identity Provider (idP) which will take care of the authentication of the project. When requesting a new (private) project, you are being asked for the IdP details, like admin group ID and tenant ID from Azure AD for example. If it is a new IdP, you should also configure the idP to establish trust between the IdP and CDF. You can read more info here https://docs.cognite.com/cdf/learn/cdf_auth/cdf_auth_idp
Also, a link to our documentation, which hopefully explains the detailshttps://docs.cognite.com/cdf/access/guides/add_service_principal
Hello,The interactive authentication will use your group ownership to give you certain capabilities within CDF.When you use client id + client secret, you are authenticating with an app registration as you said. Most likely the app registration is not part of the same groups in the Azure AD as your user. You should ask the client IT team to add the app registration to all the different groups in the Azure AD
Thanks for your patience and help for troubleshooting. Kudos to @roman.chesnokov , @Sofie Haug and @Rahul Kumar for the help as well
@Philips Could you complete the last step 4 Clean up resources and start from scratch, or at least delete the transformationclient.transformations.delete(external_id=[f"{PREFIX}-assets", f"{PREFIX}-ts", f"{PREFIX}-dp"])and recreate it. The previous secret that was not working has been stored in the transformation and therefore the transformation needs to be recreated
Thanks, everything looks quite good, it is hard to troubleshoot what is wrong. We will have another look through your attachments and get back to you
And if you can also send the output of asset_transformation = client.transformations.retrieve_multiple(external_ids=[f'{PREFIX}-assets'])[0]asset_transformation
@Philips thanks, did you regenerate a secret, updated it in the code and reran through the notebook? What was the output of print(client_secret_auth.iam.token.inspect())The problem is coming from the secret according to the error message.
@Philips let us know when you had a chance to test it out, but generating a new secret should fix the issue
You can then test the newly generated secret via from cognite.client.credentials import OAuthClientCredentialsclient_secret_auth = CogniteClient(ClientConfig( client_name="test-client", base_url=f"https://{CDF_CLUSTER}.cognitedata.com", project=COGNITE_PROJECT, credentials=OAuthClientCredentials( token_url=f"{AUTHORITY_URI}/oauth2/v2.0/token", client_id=CLIENT_ID, client_secret=CLIENT_SECRET, scopes = " ".join(SCOPES)),))print(client_secret_auth.iam.token.inspect())
The token inspect you printed is actually authenticated with your user, but great that works
Yes I agree with Roman@Philips under the section Create credentials for use in Transformations, could you try to run this code?This will try to authenticate with the client secret provided above in the notebookfrom cognite.client.data_classes import OidcCredentialscreds = OidcCredentials( client_id = CLIENT_ID, client_secret = CLIENT_SECRET, scopes = " ".join(SCOPES), token_uri = f"{AUTHORITY_URI}/oauth2/v2.0/token", cdf_project_name = COGNITE_PROJECT)cnf_secret = ClientConfig( client_name="test-client", base_url=f"https://{CDF_CLUSTER}.cognitedata.com", project=COGNITE_PROJECT, credentials=creds,)client_secret_auth = CogniteClient(cnf_secret)print(client_secret_auth.iam.token.inspect())
Thanks for your feedback. It seems like an access right issue. Did you generate a secret and added it to the code? Did you encounter any errors earlier in the notebook, like in the first steps on the print(client.iam.token.inspect().projects)And what information do you get viaprint(client.iam.token.inspect())
Great to hear. Thanks for the feedback
Already have an account? Login
Enter your username or e-mail address. We'll send you an e-mail with instructions to reset your password.
Sorry, we're still checking this file's contents to make sure it's safe to download. Please try again in a few minutes.
Sorry, our virus scanner detected that this file isn't safe to download.