Skip to main content
Schlumberger
October 11, 2023
Solved

SDK - Questions around Global Config

  • October 11, 2023
  • 13 replies
  • 337 views

Hello,

  1. What are the best practices around setting the max-workers for a sdk.
  2. If I do not set the Global Config all the default values for workers and retries would be picked correct ?
  3. Can you give any reference on how we can set GlobalConfig, I mean I did not see any way to pass the GlobalConfig to ClientConfig ?
Best answer by Ivar Stangeby

Hello,

When it comes to setting the `max-workers` for an SDK, you have a couple of options:

 

Global Configuration: You can set global configurations that will apply to any instance of `CogniteClient` that is created after the global configurations have been set. Here's how you can set the `GlobalConfig`: 

from cognite.client import global_config, ClientConfig
from cognite.client.credentials import Token

global_config.default_client_config = ClientConfig(
    client_name="my-client", project="myproj", credentials=Token("verysecret")
)
global_config.max_workers = <desired_value>  # Set the desired number for max_workers

Remember, any configurations set in the `global_config` should be made prior to instantiating a `CogniteClient` to ensure they take effect.

Directly in the CogniteClient Constructor: The documentation suggests that you can also pass a `ClientConfig` directly when you create a new `CogniteClient`. So, if you only want to set the `max_workers` for a specific instance, you can do so as:

config = ClientConfig(max_workers=<desired_value>, ...<other_params>)
client = CogniteClient(config)

To address your other questions:

  • If you do not set the `GlobalConfig`, then yes, the SDK would utilize the default values for workers, retries, and other parameters.
  • As for passing the `GlobalConfig` to `ClientConfig`, you don't need to pass it directly. Instead, the `ClientConfig` is set as an attribute of `global_config` (as shown in the example above).

Lastly, if you're working with multiple `CogniteClient` instances and making many concurrent requests, consider increasing the `max_connection_pool_size` in the `global_config` to efficiently reuse connections.

I hope this helps!
Let me know if you have any more questions or need further clarification.

13 replies

Practitioner ⭐️⭐️⭐️
December 18, 2023

(...)

Can you please explain what you meant by fetching aggregates of numeric data points? Assuming we have around 6000 time-series and half a million data points what should be the max workers to attain a performance of ~20 sec for full fetch?

You should be able to run basically the same test as above to figure that out.

My previous comment was about how to measure the performance of fetching aggregates: There are a total of 10 different aggregates you may fetch:

  1. average
  2. continuous_variance
  3. count
  4. discrete_variance
  5. interpolation
  6. max
  7. min
  8. step_interpolation
  9. sum
  10. total_variation

→ The time it takes to fetch a single aggregate is about the same as fetching all.

Seasoned ⭐️⭐️
December 28, 2023

Hello @Håkon V. Treider ,

We tried implementing the given solution in our code as mentioned above. We have an existing multi-threading logic implemented in our code, when we use the del conc._THREAD_POOL_EXECUTOR_SINGLETON with that, we get the performance numbers as expected. But when we remove the multi-threading logic. The performance is not improving.

We are using concurrent.futures.ThreadPoolExecutor for implementing multi-threading in our own code.

Please suggest a way in which we can use the SDK calls alone with your solution, so that we can get the ideal performance.

Practitioner ⭐️⭐️⭐️
January 3, 2024

Hello @Håkon V. Treider ,

We tried implementing the given solution in our code as mentioned above. We have an existing multi-threading logic implemented in our code, when we use the del conc._THREAD_POOL_EXECUTOR_SINGLETON with that, we get the performance numbers as expected. But when we remove the multi-threading logic. The performance is not improving.

We are using concurrent.futures.ThreadPoolExecutor for implementing multi-threading in our own code.

Please suggest a way in which we can use the SDK calls alone with your solution, so that we can get the ideal performance.

Since the SDK already parallelizes its calls for you, I think you should not wrap it inside of another thread pool executor. Maybe you can elaborate on why you use this pattern? If there is a need for this, please share some code and we’ll figure out how to make it as performant as you need together!