Consent

Cognite Hub is going public. So far, Cognite Hub has required users to log in to access some of its content. Now we have decided to open most parts of the community up to the public. For more information, please read our Privacy Policy and Terms and Conditions.

If you don’t agree with this privacy statement, you can reach out to support@cognite.com

×

COGNITE HUB

Check the documentation

Ask the Community

Take a look at Academy

Cognite Status Page

Contact Cognite Support

Products Cognite Data Fusion

Cognite InField

Cognite InRobot

Cognite Maintain

Cognite Remote

Website cognite.com

Groups
- - Knowledge Base
Product Updates
Product Ideas
Docs
Events
Academy
Our Partners

Impact 2024: The Industrial Data and AI Conference for and by Users | Nominate Speakers Now for a Ch...

(Mon, 14 Oct, 2:00 pm)

Home
Search

Replies posted by thomafred

18 Replies

Newest first

Oldest first

plheideCommitted

Geospatial support in FDM?Planned for development

What is the state of geospatial support in FDM? Since Q3 2023 has come and gone, I am super curious 🤓

thomafredSeasoned

asked in Developer and User Community

CDF Files block writes `ReadError`

Hi Alex and thanks for the input.I can confirm that increasing the block size did not negatively impact the implementation.Furthermore, it seems that there may have been a problem on our end related to how the block IDs were encoded. How this resulted in a transport-error baffles me a bit but overall everything appears to run smoothly now.Thanks for the help to everyone involved :)

thomafredSeasoned

asked in Developer and User Community

CDF Files block writes `ReadError`

I did some more digging and found that it appears to be the last block that is failing. This block is significantly larger than the other blocks at 4772525 bytes, compared to 133120 bytes for the other blocks (4.55 MiB and 0.13 MiB respectively).A quick look at the Azure-docs shows that the maximum block size for service versions prior to 2016-05-31 is 4 MiB: Can you confirm that the maximum block size is 4 MiB?

thomafredSeasoned

asked in Developer and User Community

CDF Files block writes `ReadError`

Apologies for the late response. I have been on holiday and have not had a chance to follow up until now. Here’s a an excerpt from our logs containing the information you need. Note that this is from a separate test-run than the earlier request-ids posted. DEBUG:httpx._client:HTTP Request: PUT https://westeurope-1.cognitedata.com/api/v1/files/storage/cognite/6723545981847516/2188674687693268/wod_gld_2013.00.nc?sig=8OyJ25tydRcTKrHZpUTSl6QAAAqf%2Bqx0Yn9Ll1OmI6A%3D&se=2023-08-22T09%3A40%3A00Z&sv=2019-02-02&rsct=text%2Fplain&sp=cw&sr=b&comp=block&blockid=gAAAAAAAA9Q= "HTTP/1.1 201 Created"DEBUG:odp.odcat.storage.cdffs.async_http_client:status code returned: 201DEBUG:odp.odcat.storage.cdffs.async_files:Uploaded 5242880 bytes to https://westeurope-1.cognitedata.com/api/v1/files/storage/cognite/6723545981847516/2188674687693268/wod_gld_2013.00.nc?sig=8OyJ25tydRcTKrHZpUTSl6QAAAqf%2Bqx0Yn9Ll1OmI6A%3D&se=2023-08-22T09%3A40%3A00Z&sv=2019-02-02&rsct=text%2Fp

thomafredSeasoned

asked in Developer and User Community

CDF Files block writes `ReadError`

Here’s the response-headers of the last fewPUT-requests date x-ms-request-id x-ms-content-crc64 Thu, 20 Jul 2023 07:20:56 GMT 7fe59a18-d01e-0067-34da-ba1b5d000000 KCEynvNwPkg= Thu, 20 Jul 2023 07:20:56 GMT 1c6e3a95-f01e-0065-63da-ba4d59000000 ULboHdXz6Oo= Thu, 20 Jul 2023 07:20:57 GMT 43b3866c-701e-007d-03da-baa56a000000 AwfypHp6avE= Thu, 20 Jul 2023 07:20:57 GMT 7f69797a-d01e-0077-7cda-baab7f000000 1BtDPMJLLWo= Thu, 20 Jul 2023 07:20:58 GMT f4de6d2c-101e-003f-14da-ba207f000000 OOiCTQl/tsg=

thomafredSeasoned

asked in Developer and User Community

CDF Files block writes `ReadError`

My apologies for the confusion. The logs provided are from the logs of our own internal service - the 500 error is generated in response to the user request. We are not receiving any response from the CDF API.

thomafredSeasoned

asked in Developer and User Community

CDF Files block writes `ReadError`

The file is 2.671 GB. You can find it here (named wod_gld_2013.nc): https://www.ncei.noaa.gov/thredds-ocean/catalog/ncei/wod/2013/catalog.html

thomafredSeasoned

asked in Developer and User Community

CDF Files block writes `ReadError`

There is an exponential backoff retry strategy already implemented (double the wait for each try, max 10 seconds). This should also be visible in the top-half of the log.Not sure if it is possible to provide a request ID, as I am not getting any response from the server.

thomafredSeasoned

asked in Developer and User Community

Streaming upload to CDF Files

An update on this.Block updates work quite well. However, after a few minutes (typically around 4 minutes, but appears to be somewhat random), the upload may fail due to a read-error on a block update. There doesn’t appear to be any response from the server.

thomafredSeasoned

asked in Developer and User Community

Streaming upload to CDF Files

Sorry about the delay, I have only now been able to follow up on this.Did a quick proof-of-concept using postman, and using `PUT block` and `PUT blocklist` seems to work, however there are some limitations.First and foremost, it appears that I am not authorized do the `GET blocklist`-operation (https://learn.microsoft.com/en-us/rest/api/storageservices/get-block-list?tabs=azure-ad), and instead getting the following error:<?xml version="1.0" encoding="utf-8"?><Error> <Code>AuthorizationPermissionMismatch</Code> <Message>This request is not authorized to perform this operation using this permission.RequestId:44177ea5-601e-0030-78d2-ada960000000Time:2023-07-03T17:21:35.0950174Z</Message></Error>I was able to do an async Python POC though :)

thomafredSeasoned

asked in Developer and User Community

Streaming upload to CDF Files

We are exclusively running CDF on Azure :p We are also using Azure as idP.In other words - yes to both :p

thomafredSeasoned

asked in Developer and User Community

Streaming upload to CDF Files

Will this work if our CDF tenant is located on Azure?

thomafredSeasoned

posted in Developer and User Community

CDF Client connection string

This is very useful, thank you :)

thomafredSeasoned

asked in Developer and User Community

Python SDK Azure B2C Authentication

FYI - The `token`-argument in `CogniteClient` can also be callable: def get_token(): print("----> Fetching bearer token") response = requests.request("POST", url, headers={ 'Content-Type': 'application/x-www-form-urlencoded' }, data={ "grant_type": 'password', "client_id": client_id, "scope": 'openid https://westeurope-1.cognitedata.com/user_impersonation', "username": username, "password": password, }) creds = response.json() return creds["access_token"]client = CogniteClient( token=get_token, project="oceandata", base_url='https://westeurope-1.cognitedata.com', client_name="cognite-python-dev", debug=True,)print(client.iam.token.inspect())

thomafredSeasoned

posted in Developer and User Community

CDF Modelling Challenge - SpatioTemporal Asset Catalog

Awesome! Thank you for taking this initiative, Susanna!

thomafredSeasoned

posted in Developer and User Community

CDF Modelling Challenge - Open Lineage

Hi @Kjetil Halvorsen,Thank you so much for your response.I really like how you approach this problem in two ways. The easy solution seems particularly relevant to teams who have not set up the necessary infrastructure for data processing.The “hard” solution is however more relevant to Open Lineage. It makes perfect sense to use events to model runs the way you describe, and having the other object-types be assets.I wonder if templates could fit this scheme? Open Lineage does essentially have a GraphQL definition already (link). From what I understand, relationships are not supported by CDF Templates yet, but templates could allow us to create job views and dataset views?

thomafredSeasoned

asked in Developer and User Community

CDF Templates API documentation and itroduction

These API docs are for the template completion API.I am looking for more documentation on the Templates-API, the one used by the Python-SDK linked above.Particularly, I would like to learn more about intended use-cases and what kind of volumes it supports.

thomafredSeasoned

asked in Developer and User Community

CDF Files and cloud optimized storage

Allowing for a range-argument is a nice feature. Is this part of a propriatary protocol, or can I (in python) use something like gcsfs as described here: https://pangeo.io/data.html

Generate client secret for OID

Please generate a client secret once every 180 days.

Choose the application:

Badge winners

Prasanna Kumar Tellapanenihas earned the badge Working with Cognite InField
Geraldinehas earned the badge Cognite Data Fusion Fundamentals
Jay Vansicklehas earned the badge Working with Cognite InField
Emmanuel Eduardo Perezhas earned the badge Cognite Domain Expert Basics
Adam Sharpehas earned the badge Cognite Data Fusion Fundamentals

Show all badges

Powered by Gainsight

Terms & Conditions Cookie Settings

Sign up

Already have an account? Login

Log in with your business account

Log in to the community

Log in with your business account

Enter your username or e-mail address. We'll send you an e-mail with instructions to reset your password.

Username or e-mail

Back to overview

Scanning file for viruses.

Sorry, we're still checking this file's contents to make sure it's safe to download. Please try again in a few minutes.

OK

This file cannot be downloaded

Sorry, our virus scanner detected that this file isn't safe to download.

OK

RESOURCES Learn Documentation Support Status Page
GETTING STARTED Sign up to Cognite Hub Get Started with Cognite Hub About Us Contact Us
FOLLOW US

© 2022

Powered by inSided