Impact 2024: The Industrial Data and AI Conference for and by Users | Nominate Speakers Now for a Ch...
sure,[ { "externalId": "gb-test-func-1", "type": "function", "name": "1710479974441", "parameters": { "function": { "externalId": "gb-test-func-1", "data": {} } }, "onFailure": "abortWorkflow", "dependsOn": [] }, { "externalId": "gb-test-func-2", "type": "function", "name": "1710482066717", "parameters": { "function": { "externalId": "gb-test-func-2", "data": {} } }, "onFailure": "abortWorkflow", "dependsOn": [ { "externalId": "gb-multiplication-by-2" } ] }, { "externalId": "gb-multiplication-by-2", "type": "function", "name": "1710927303699", "parameters": { "function": { "externalId": "gb-multiplication-by-2", "data": {} } }, "onFailure": "abortWorkflow", "dependsOn": [ { "externalId": "gb-test-func-1" } ] } ]
Hello @Håkon V. Treider ,We tried implementing the given solution in our code as mentioned above. We have an existing multi-threading logic implemented in our code, when we use the del conc._THREAD_POOL_EXECUTOR_SINGLETON with that, we get the performance numbers as expected. But when we remove the multi-threading logic. The performance is not improving.We are using concurrent.futures.ThreadPoolExecutor for implementing multi-threading in our own code.Please suggest a way in which we can use the SDK calls alone with your solution, so that we can get the ideal performance.
Hi @Håkon V. Treider , @Jason Dressel and Anders Daljord Morken,We tried the above solution for fetching data points and we were able to achieve the expected performance number (around 20 seconds) for 1 max worker.Can you please explain what you meant by fetching aggregates of numeric data points? Assuming we have around 6000 time-series and half a million data points what should be the max workers to attain a performance of ~20 sec for full fetch?
Hi @Everton Colling sure. Sharing the code for fetching data. if aggregates: dps = client.time_series.data.retrieve(external_id=external_id, start=int(startParam), end=int(endParam), aggregates=aggregates, granularity=granularity,limit=limit) elif is_latestvalue: dps = client.time_series.data.retrieve_latest(external_id=external_id) else: dps = client.time_series.data.retrieve(external_id=external_id, start=int(startParam), end=int(endParam),limit=limit)
Hello,We tried the approaches suggested and found out that there is not much difference in performance for fetching data from CDF.For capturing the numbers, we used the following data model configuration: Project: slb-test Model: QA-PERF-DATA (QA_SPACE) Version: 1 View: TimeseriesData Properties: externalId, timeseriesId, propertyName, entity, frequency Time Range: 15-11-2023 00:00 to 17-11-2023 23:59 Filters: propertyName: TARGET-WATER-INJECTION-RATE , YEP-GAS-PRODUCTION-RATE, METER-OPENING-GAS-VOLUME ,UC-CONTRACTOR ,SAND-PRODUCTION ,UNIT-INJECTION-COST ,WATER-PRODUCTION-VOLUME,GAS-SALES-VOLUME, GAS-PRODUCTION-RATE,DIFFERENTIAL-PRESSURE-HOLDING-TRIGGERED-VALUE frequency: 15M The time taken by the code which uses API call for fetching data from CDF is 25.945018529891968 seconds Details about both the approaches tried and their results are as follows:Approach 1: Setting max_workers in Global ConfigurationCode change: global_config.default_client_confi
Hi, this is resolved.Thanks!
Thanks @Jason Dressel !
Hi @Jason Dressel, could you please add the following users to the AAD tenant for the appropriate cdf projects?mgandhi3@slb.comspulikonda@slb.comgbhosale2@slb.com we are getting the same error as Divyanshi while accessing the cognite tenant url (Cognite Data Fusion)
Already have an account? Login
Enter your E-mail address. We'll send you an e-mail with instructions to reset your password.
Sorry, we're still checking this file's contents to make sure it's safe to download. Please try again in a few minutes.
Sorry, our virus scanner detected that this file isn't safe to download.
Check the
documentation
Ask the
Community
Take a look
at
Academy
Cognite
Status
Page
Contact
Cognite Support
