Impact 2024: The Industrial Data and AI Conference for and by Users | Nominate Speakers Now for a Ch...
Hello @Sonali Vishal Patil . Have you tried to change the case? You seem to have ‘Name’ in your query and ‘name’ in the target scheme.
Hey Ankita, for me this way works finefor unstructre_insight in client.events(type="Insight", chunk_size=32): print(len(unstructure_insights))Could you please double-check that you are getting all 64 events at once?
@eashwar11 the closest that you probably could get to replicate Excel in Python is pandas. But of course, you can also make your calculations using SQL in CDF Transformations, there you can use a 'with' clause for example to make temporary subqueries if you need them.
Hey, thanks for reaching out, You can definitely use Cognite Functions for that. But please keep in mind the limitations.It makes sense to ingest input data as RAW table/tables, and then you can use them from the CDF functions. For example, get it from CDF as a pandas data frame.Here is the doc about creating a CDF function from a Python script. If you have any questions or need further clarification, please don't hesitate to let me know.
@Asan Arifov the major versions don't guarantee backward compatibility. Recently SDK was updated to version 6. But of course you can specify the version in requirements.
@Asan Arifov it seems that you use the latest version of cognite-sdk, and there you can retrieve datapoints like that: >>> from cognite.client import CogniteClient>>> client = CogniteClient()>>> dps = client.time_series.data.retrieve(id=42, start="2w-ago") To resolve the issue you can either update the code or use the 5th version of SDK.
It seems like an error with generating secrets from the course. @Rahul Kumar , could you look into it?
I see; then probably the best way is to use our REST API directly. You can find the docs here. As far as I know, there are no plans for implementing cursor usage by users via SDK.
Hey Janos, the client.events() method is basically a generator. So you can use something like that for pagination:events_provider = client.events(chunk_size=250)next_chunck = next(events_provider)Then next_chunck is a list of the next 250 events. And request to API happens when you call the next function. So it uses a cursor, but you don’t need to handle it by yourself. Do you think it will work for you?
Sure, you can create a RAW table in Manage staged data menu, and then you can use that table as a target in your transformation:You can change the target afterwards or create a separate transformation that will not be scheduled for debugging purposes.Hope that answers your question.
That’s a limitation of the Preview mode, it’s only meant to give you a view of the data, and then you can run the transformation. But you can use transformation from RAW to RAW, then you will not be limited by 240 seconds, and you can inspect results in the separate RAW table after the completion of the transformation job.
Hey Sonali, if you’re trying to run a preview, then there is a limit in the list:When you run a transformation itself, there will not be that limit.
Example of time for each API call for different chunk_size:t1 = datetime.now()for event_list in client.events(chunk_size=2500): print(len(event_list)) t2 = datetime.now() print(t2 - t1) t1 = t225000:00:02.74440425000:00:01.41413625000:00:02.49859325000:00:01.86544425000:00:02.715027 t1 = datetime.now()for event_list in client.events(chunk_size=250): print(len(event_list)) t2 = datetime.now() print(t2 - t1) t1 = t22500:00:00.3688912500:00:00.2748052500:00:00.2851392500:00:00.3328112500:00:00.309854
The first approach uses a cursor under the hood, it uses a generator and makes an API call for each chunk. Could you explain to me the requirements and why you need a cursor specifically?
Hello Shah, For events specifically you can use that idea:Iterate over chunks of events to reduce memory load:from cognite.client import CogniteClientc = CogniteClient()for event_list in c.events(chunk_size=2500): event_list # do something with the eventsYou can also make batches based on last_updated_time, start_time or end_time and use the timestamp ranges in events.list() method. Another approach is to iterate over assets and get events for each asset.Let me know if it was helpful.
Hi Shah, the Python SDK only supports the cursor parameter to list the transformation schedules, but there are different approaches that you can implement if you need pagination. For example, for datapoints you can use timestamps for pagination. You can also read some additional info about the performance of datapoints retrieval here.
That looks ok. I removed your transformation now, could you try to create it again? The credentials were not updated since we’re not updating the transformation but trying to create it again.
That looks like a result for a normal client with interactive auth workflow. Could you add a cell and run my code from the previous message? It will print not only projects but also capabilities.
Could you try to instantiate a client with the client secret and send here token capabilities:from cognite.client.credentials import OAuthClientCredentialscreds_test = OAuthClientCredentials(token_url=f"{AUTHORITY_URI}/oauth2/v2.0/token", client_id=CLIENT_ID, scopes=" ".join(SCOPES), client_secret=CLIENT_SECRET)cnf_test = ClientConfig( project=COGNITE_PROJECT, base_url=f'https://{CDF_CLUSTER}.cognitedata.com', client_name='cognite-python-dev', credentials=creds_test) client_test = CogniteClient(cnf_test)client_test.iam.token.inspect()
I found out that we have a temporary issue with the secret generator in the course. But I’ve provided you with a working secret for that project in the direct messages. Let me know if you have any other blockers.
Hey Jasar, to instantiate CogniteClient you use your account and interactive authentication workflow. But it doesn’t work for transformations. You should set the source and destination credentials. If you use the same project to read and write data, they could be the same. You need to generate a client secret as described in that lesson and use it for running your transformations. You enter it in one of the first cells:CLIENT_SECRET = "" # Enter secret
Already have an account? Login
Enter your username or e-mail address. We'll send you an e-mail with instructions to reset your password.
Sorry, we're still checking this file's contents to make sure it's safe to download. Please try again in a few minutes.
Sorry, our virus scanner detected that this file isn't safe to download.