Replies posted by roman.chesnokov

49 Replies

jasar
Active
jasarActive
asked in Developer and User Community

Error when run a transformation from the course.

Hey Jasar, to instantiate CogniteClient you use your account and interactive authentication workflow. But it doesn’t work for transformations. You should set the source and destination credentials. If you use the same project to read and write data, they could be the same. You need to generate a client secret as described in that lesson and use it for running your transformations. You enter it in one of the first cells:CLIENT_SECRET = "" # Enter secret

10
smrsgv
Committed
10 months ago
jasar
Active
jasarActive
asked in Developer and User Community

Error when run a transformation from the course.

I found out that we have a temporary issue with the secret generator in the course. But I’ve provided you with a working secret for that project in the direct messages. Let me know if you have any other blockers.

10
smrsgv
Committed
10 months ago
jasar
Active
jasarActive
asked in Developer and User Community

Error when run a transformation from the course.

Could you try to instantiate a client with the client secret and send here token capabilities:from cognite.client.credentials import OAuthClientCredentialscreds_test = OAuthClientCredentials(token_url=f"{AUTHORITY_URI}/oauth2/v2.0/token", client_id=CLIENT_ID, scopes=" ".join(SCOPES), client_secret=CLIENT_SECRET)cnf_test = ClientConfig( project=COGNITE_PROJECT, base_url=f'https://{CDF_CLUSTER}.cognitedata.com', client_name='cognite-python-dev', credentials=creds_test) client_test = CogniteClient(cnf_test)client_test.iam.token.inspect()  

10
smrsgv
Committed
10 months ago
jasar
Active
jasarActive
asked in Developer and User Community

Error when run a transformation from the course.

That looks like a result for a normal client with interactive auth workflow. Could you add a cell and run my code from the previous message? It will print not only projects but also capabilities.

10
smrsgv
Committed
10 months ago
jasar
Active
jasarActive
asked in Developer and User Community

Error when run a transformation from the course.

That looks ok. I removed your transformation now, could you try to create it again? The credentials were not updated since we’re not updating the transformation but trying to create it again.

10
smrsgv
Committed
10 months ago
S
Committed
Shah AyushiCommitted
asked in Developer and User Community

Query for pagination

Hi Shah, the Python SDK only supports the cursor parameter to list the transformation schedules, but there are different approaches that you can implement if you need pagination. For example, for datapoints you can use timestamps for pagination. You can also read some additional info about the performance of datapoints retrieval here.

10
roman.chesnokov
Seasoned Practitioner
1 year ago
S
Committed
Shah AyushiCommitted
asked in Developer and User Community

Query for pagination

Hello Shah,  For events specifically you can use that idea:Iterate over chunks of events to reduce memory load:from cognite.client import CogniteClientc = CogniteClient()for event_list in c.events(chunk_size=2500): event_list # do something with the eventsYou can also make batches based on last_updated_time, start_time or end_time and use the timestamp ranges in events.list() method. Another approach is to iterate over assets and get events for each asset.Let me know if it was helpful. 

10
roman.chesnokov
Seasoned Practitioner
1 year ago
S
Committed
Shah AyushiCommitted
asked in Developer and User Community

Query for pagination

The first approach uses a cursor under the hood, it uses a generator and makes an API call for each chunk. Could you explain to me the requirements and why you need a cursor specifically? 

10
roman.chesnokov
Seasoned Practitioner
1 year ago
S
Committed
Shah AyushiCommitted
asked in Developer and User Community

Query for pagination

Example of time for each API call for different chunk_size:t1 = datetime.now()for event_list in client.events(chunk_size=2500): print(len(event_list)) t2 = datetime.now() print(t2 - t1) t1 = t225000:00:02.74440425000:00:01.41413625000:00:02.49859325000:00:01.86544425000:00:02.715027 t1 = datetime.now()for event_list in client.events(chunk_size=250): print(len(event_list)) t2 = datetime.now() print(t2 - t1) t1 = t22500:00:00.3688912500:00:00.2748052500:00:00.2851392500:00:00.3328112500:00:00.309854 

10
roman.chesnokov
Seasoned Practitioner
1 year ago
S
Seasoned
Sonali Vishal PatilSeasoned
asked in Developer and User Community

Query timed out after running transaformation

Hey Sonali, if you’re trying to run a preview, then there is a limit in the list:When you run a transformation itself, there will not be that limit.

6
Dilini Fernando
Seasoned Practitioner
1 year ago
S
Seasoned
Sonali Vishal PatilSeasoned
asked in Developer and User Community

Query timed out after running transaformation

That’s a limitation of the Preview mode, it’s only meant to give you a view of the data, and then you can run the transformation. But you can use transformation from RAW to RAW, then you will not be limited by 240 seconds, and you can inspect results in the separate RAW table after the completion of the transformation job.

6
Dilini Fernando
Seasoned Practitioner
1 year ago
S
Seasoned
Sonali Vishal PatilSeasoned
asked in Developer and User Community

Query timed out after running transaformation

Sure, you can create a RAW table in Manage staged data menu, and then you can use that table as a target in your transformation:You can change the target afterwards or create a separate transformation that will not be scheduled for debugging purposes.Hope that answers your question.

6
Dilini Fernando
Seasoned Practitioner
1 year ago
S
Committed
Shah AyushiCommitted
asked in Developer and User Community

Query for pagination

Hey Janos, the client.events() method is basically a generator. So you can use something like that for pagination:events_provider = client.events(chunk_size=250)next_chunck = next(events_provider)Then next_chunck is a list of the next 250 events. And request to API happens when you call the next function. So it uses a cursor, but you don’t need to handle it by yourself. Do you think it will work for you? 

10
roman.chesnokov
Seasoned Practitioner
1 year ago
S
Committed
Shah AyushiCommitted
asked in Developer and User Community

Query for pagination

I see; then probably the best way is to use our REST API directly. You can find the docs here. As far as I know, there are no plans for implementing cursor usage by users via SDK. 

10
roman.chesnokov
Seasoned Practitioner
1 year ago
P
Committed
PhilipsCommitted
asked in Developer and User Community

Unable to run the CDF Transformation job created

It seems like an error with generating secrets from the course. @Rahul Kumar , could you look into it? 

19
G
Seasoned Practitioner
1 year ago
Madina Kholmatova
Seasoned Practitioner
Madina KholmatovaCognite Academy Instructor
posted in Academy Discussions

Data Scientist Basics learning path

@Asan Arifov it seems that you use the latest version of cognite-sdk, and there you can retrieve datapoints like that: >>> from cognite.client import CogniteClient>>> client = CogniteClient()>>> dps = client.time_series.data.retrieve(id=42, start="2w-ago") To resolve the issue you can either update the code or use the 5th version of SDK.

14
roman.chesnokov
Seasoned Practitioner
1 year ago
Madina Kholmatova
Seasoned Practitioner
Madina KholmatovaCognite Academy Instructor
posted in Academy Discussions

Data Scientist Basics learning path

@Asan Arifov the major versions don't guarantee backward compatibility. Recently SDK was updated to version 6. But of course you can specify the version in requirements.

14
roman.chesnokov
Seasoned Practitioner
1 year ago
E
Practitioner
eashwar11
asked in Developer and User Community

migrating business logic from excel macros to CDF

Hey, thanks for reaching out, You can definitely use Cognite Functions for that. But please keep in mind the limitations.It makes sense to ingest input data as RAW table/tables, and then you can use them from the CDF functions. For example, get it from CDF as a pandas data frame.Here is the doc about creating a CDF function from a Python script. If you have any questions or need further clarification, please don't hesitate to let me know.

5
E
Practitioner
11 months ago
E
Practitioner
eashwar11
asked in Developer and User Community

migrating business logic from excel macros to CDF

@eashwar11 the closest that you probably could get to replicate Excel in Python is pandas. But of course, you can also make your calculations using SQL in CDF Transformations, there you can use a 'with' clause for example to make temporary subqueries if you need them.

5
E
Practitioner
11 months ago
A
Committed
Ankita ManeCommitted
asked in Developer and User Community

Performance issue for event retrieval

Hey Ankita, for me this way works finefor unstructre_insight in client.events(type="Insight", chunk_size=32):    print(len(unstructure_insights))Could you please double-check that you are getting all 64 events at once? 

4
Dilini Fernando
Seasoned Practitioner
11 months ago
S
Seasoned
Sonali Vishal PatilSeasoned
asked in Developer and User Community

By looking at the message it doesn't seems to be like error even though its a error

Hello @Sonali Vishal Patil . Have you tried to change the case? You seem to have ‘Name’ in your query and ‘name’ in the target scheme.

4
Anita Hæhre
Seasoned Practitioner
11 months ago
E
Practitioner
eashwar11
asked in Developer and User Community

CDF Extractors

Hey @eashwar11, for the PI extractor, you need a Windows Server Machine. Then you can install and configure a few extractors, one for DEV and one for PROD. Extractors are usually running continuously. If you’re going to use a cloud VM, you can configure GH actions to update the config files and restart services, for example, depending on a particular cloud provider. 

11
Dilini Fernando
Seasoned Practitioner
10 months ago
E
Practitioner
eashwar11
asked in Developer and User Community

CDF Extractors

@eashwar11 It strongly depends on the particular architecture and the use case. That’s probably why there are no particular examples of automation in the docs. Basically, you can just install a few instances of extractors on a VM manually and forget about that, that’s how it’s done for many cases. 

11
Dilini Fernando
Seasoned Practitioner
10 months ago
V
Committed
Viswanadha Sai Akhil PujyamCommitted
asked in How-To Guides

.csv file Ingestion into CDF using Python SDK [Community Contributed]

@Viswanadha Sai Akhil Pujyam to insert CSV as a RAW table, you can use the following snippet:import numpy as npimport pandas as pdfrom cognite.client import CogniteClientDATABASE_NAME = "db"TABLE_NAME = "table"CSV_FILENAME = "source.csv"client = CogniteClient()client.raw.databases.create(DATABASE_NAME)client.raw.tables.create(DATABASE_NAME, TABLE_NAME)df = pd.read_csv(CSV_FILENAME, index_col=0).fillna('')client.raw.rows.insert_dataframe(DATABASE_NAME, TABLE_NAME, df) 

4
Dilini Fernando
Seasoned Practitioner
10 months ago
P
Committed
Paramale AratiCommitted
asked in Developer and User Community

Need to co-relate Events with Timeseries

@Paramale Arati It's important to note that in the event you have both a start and end date, so it cannot be displayed as a point marker. So, as @Ankit Kumar  wrote before, you can’t directly display both timeseries and events in one chart in PowerBI. But it allows you to utilise a Line chart along with a custom timeline chart, such as a Gantt chart, for example, one under another, and apply the same filters on intervals. This method can be effective if there are not too many events within the observed interval.And that’s an example of how it looks like in Cognite Charts: 

11
Dilini Fernando
Seasoned Practitioner
11 months ago

Badge winners

Show all badges
Terms & ConditionsCookie Settings