Skip to main content
Seasoned ⭐️⭐️
February 17, 2023

Error running the Cognite replicator with Option 5

  • February 17, 2023
  • 16 replies
  • 282 views

I have set up both the environment variables SOURCE_CLIENT_SECRET (Note that the Client secret is generated from the OID widget as per the documentation)and DEST_CLIENT_SECRET and running the replicator with the config file using option 5.I am getting error as per below log

AMAC02Z3123LVCJ:oid-replicator j.subhash.parandekar$ poetry run python3 ./oid_replicator/replicate.py 
2023-02-17 18:46:48,239 root INFO - Config file - Repeat line 5: 

2023-02-17 18:46:48,239 root INFO - Config file - Repeat line 14: 

2023-02-17 18:46:48,239 root INFO - Config file - Repeat line 23: 

Starting replication of resources
Replicating assets...
Traceback (most recent call last):
  File "/Users/j.subhash.parandekar/Library/Caches/pypoetry/virtualenvs/oid-replicator-cwaK-6ym-py3.11/lib/python3.11/site-packages/cognite/client/credentials.py", line 364, in _refresh_access_token
    token_result = self.__oauth.fetch_token(
                   ^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/j.subhash.parandekar/Library/Caches/pypoetry/virtualenvs/oid-replicator-cwaK-6ym-py3.11/lib/python3.11/site-packages/requests_oauthlib/oauth2_session.py", line 366, in fetch_token
    self._client.parse_request_body_response(r.text, scope=self.scope)
  File "/Users/j.subhash.parandekar/Library/Caches/pypoetry/virtualenvs/oid-replicator-cwaK-6ym-py3.11/lib/python3.11/site-packages/oauthlib/oauth2/rfc6749/clients/base.py", line 427, in parse_request_body_response
    self.token = parse_token_response(body, scope=scope)
                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/j.subhash.parandekar/Library/Caches/pypoetry/virtualenvs/oid-replicator-cwaK-6ym-py3.11/lib/python3.11/site-packages/oauthlib/oauth2/rfc6749/parameters.py", line 441, in parse_token_response
    validate_token_parameters(params)
  File "/Users/j.subhash.parandekar/Library/Caches/pypoetry/virtualenvs/oid-replicator-cwaK-6ym-py3.11/lib/python3.11/site-packages/oauthlib/oauth2/rfc6749/parameters.py", line 448, in validate_token_parameters
    raise_from_error(params.get('error'), params)
  File "/Users/j.subhash.parandekar/Library/Caches/pypoetry/virtualenvs/oid-replicator-cwaK-6ym-py3.11/lib/python3.11/site-packages/oauthlib/oauth2/rfc6749/errors.py", line 399, in raise_from_error
    raise cls(**kwargs)
oauthlib.oauth2.rfc6749.errors.InvalidClientError: (invalid_client) AADSTS7000216: 'client_assertion', 'client_secret' or 'request' is required for the 'client_credentials' grant type.
Trace ID: b0528310-0b70-4ee0-9715-daca09e35200
Correlation ID: 5987d7dd-eb61-404c-9525-00954829211c
Timestamp: 2023-02-17 13:16:49Z

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/Users/j.subhash.parandekar/oid-replicator/./oid_replicator/replicate.py", line 8, in <module>
    main()
  File "/Users/j.subhash.parandekar/Library/Caches/pypoetry/virtualenvs/oid-replicator-cwaK-6ym-py3.11/lib/python3.11/site-packages/cognite/replicator/__main__.py", line 401, in main
    cognite.replicator.assets.replicate(
  File "/Users/j.subhash.parandekar/Library/Caches/pypoetry/virtualenvs/oid-replicator-cwaK-6ym-py3.11/lib/python3.11/site-packages/cognite/replicator/assets.py", line 298, in replicate
    assets_src = client_src.assets.list(limit=None)
                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/j.subhash.parandekar/Library/Caches/pypoetry/virtualenvs/oid-replicator-cwaK-6ym-py3.11/lib/python3.11/site-packages/cognite/client/_api/assets.py", line 295, in list
    return self._list(
           ^^^^^^^^^^^
  File "/Users/j.subhash.parandekar/Library/Caches/pypoetry/virtualenvs/oid-replicator-cwaK-6ym-py3.11/lib/python3.11/site-packages/cognite/client/_api_client.py", line 489, in _list
    for resource_list in self._list_generator(
  File "/Users/j.subhash.parandekar/Library/Caches/pypoetry/virtualenvs/oid-replicator-cwaK-6ym-py3.11/lib/python3.11/site-packages/cognite/client/_api_client.py", line 395, in _list_generator
    res = self._post(url_path=url_path or resource_path + "/list", json=body, headers=headers)
          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/j.subhash.parandekar/Library/Caches/pypoetry/virtualenvs/oid-replicator-cwaK-6ym-py3.11/lib/python3.11/site-packages/cognite/client/_api_client.py", line 135, in _post
    return self._do_request(
           ^^^^^^^^^^^^^^^^^
  File "/Users/j.subhash.parandekar/Library/Caches/pypoetry/virtualenvs/oid-replicator-cwaK-6ym-py3.11/lib/python3.11/site-packages/cognite/client/_api_client.py", line 148, in _do_request
    headers = self._configure_headers(accept, additional_headers=self._config.headers.copy())
              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/j.subhash.parandekar/Library/Caches/pypoetry/virtualenvs/oid-replicator-cwaK-6ym-py3.11/lib/python3.11/site-packages/cognite/client/_api_client.py", line 190, in _configure_headers
    auth_header_name, auth_header_value = self._config.credentials.authorization_header()
                                          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/j.subhash.parandekar/Library/Caches/pypoetry/virtualenvs/oid-replicator-cwaK-6ym-py3.11/lib/python3.11/site-packages/cognite/client/credentials.py", line 121, in authorization_header
    self.__access_token, self.__access_token_expires_at = self._refresh_access_token()
                                                          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/j.subhash.parandekar/Library/Caches/pypoetry/virtualenvs/oid-replicator-cwaK-6ym-py3.11/lib/python3.11/site-packages/cognite/client/credentials.py", line 372, in _refresh_access_token
    raise CogniteAuthError(
cognite.client.exceptions.CogniteAuthError: Error generating access token: invalid_client, 401, AADSTS7000216: 'client_assertion', 'client_secret' or 'request' is required for the 'client_credentials' grant type.
Trace ID: b0528310-0b70-4ee0-9715-daca09e35200
Correlation ID: 5987d7dd-eb61-404c-9525-00954829211c
Timestamp: 2023-02-17 13:16:49Z

16 replies

jaydeepAuthor
Seasoned ⭐️⭐️
February 21, 2023

I am trying to replicate timeseries and data points  however I am getting below error

AMAC02Z3123LVCJ:oid-replicator j.subhash.parandekar$ /usr/local/bin/python3 /Users/j.subhash.parandekar/oid-replicator/oid_replicator/replicate.py
2023-02-21 11:31:43,353 root INFO - Config file - Repeat line 5: 

2023-02-21 11:31:43,353 root INFO - Config file - Repeat line 15: 

2023-02-21 11:31:43,353 root INFO - Config file - Repeat line 24: 

Starting replication of resources
Replicating time series...
2023-02-21 11:31:53,330 root INFO - There are 408 existing time series in source (publicdata).
2023-02-21 11:31:53,330 root INFO - There are 10291 existing time series in destination (accenture-tiger-training).
2023-02-21 11:31:57,719 root INFO - If a time series asset id is one of the 1115 assets that have been replicated then it will be linked.
2023-02-21 11:31:57,719 root INFO - These copied/updated time series will have a replicated run time of: 1676959317000.
2023-02-21 11:31:57,719 root INFO - Starting to copy and update 408 time series from source (publicdata) to destination (accenture-tiger-training).
Traceback (most recent call last):
  File "/Users/j.subhash.parandekar/oid-replicator/oid_replicator/replicate.py", line 8, in <module>
    main()
  File "/Library/Frameworks/Python.framework/Versions/3.11/lib/python3.11/site-packages/cognite/replicator/__main__.py", line 427, in main
    cognite.replicator.time_series.replicate(
  File "/Library/Frameworks/Python.framework/Versions/3.11/lib/python3.11/site-packages/cognite/replicator/time_series.py", line 298, in replicate
    if len(ts_src) > batch_size:
       ^^^^^^^^^^^^^^^^^^^^^^^^
TypeError: '>' not supported between instances of 'int' and 'dict'

Gaetan  Helness
MVP ⭐️⭐️⭐️⭐️⭐️
MVP ⭐️⭐️⭐️⭐️⭐️
February 21, 2023

What is the value of  “batch_size”? It looks like it is an dictionary. 
its value is set in the config file for example

len(ts_src) is an int and printed above (408)

batch_size should also be an int, can be set to 10000 for example

Gaetan  Helness
MVP ⭐️⭐️⭐️⭐️⭐️
MVP ⭐️⭐️⭐️⭐️⭐️
February 21, 2023

Ok, apologies, i think you found a bug, we will look into it and fix it today

Gaetan  Helness
MVP ⭐️⭐️⭐️⭐️⭐️
MVP ⭐️⭐️⭐️⭐️⭐️
February 21, 2023

Could you upgrade the package to the latest version

pip install cognite-replicator --upgrade

Version should be 1.2.3
 

jaydeepAuthor
Seasoned ⭐️⭐️
February 21, 2023

yes , after upgrading the cognite-replicator package to 1.2.3 this error is resolved and I am able to execute as per below. Thanks

2023-02-21 16:56:01,725 root INFO - Ext id:  pi:160571 Number of datapoints: 0
2023-02-21 16:56:04,095 root INFO - Ext id:  pi:160699 Number of datapoints: 219394
2023-02-21 16:56:04,138 root INFO - Ext id:  pi:160265 Number of datapoints: 3864
2023-02-21 16:56:04,138 root INFO - Ext id:  pi:161026 Number of datapoints: 0
2023-02-21 16:56:05,471 root INFO - Ext id:  houston.ro.REMOTE_AI[4] Number of datapoints: 122735
2023-02-21 16:56:05,471 root INFO - Ext id:  pi:160551 Number of datapoints: 0
2023-02-21 16:56:05,511 root INFO - Ext id:  houston.ro.REMOTE_AI[75] Number of datapoints: 3590
2023-02-21 16:56:07,950 root INFO - Ext id:  pi:160672 Number of datapoints: 224161
2023-02-21 16:56:07,951 root INFO - Ext id:  pi:160222 Number of datapoints: 0
2023-02-21 16:56:09,931 root INFO - Ext id:  pi:160766 Number of datapoints: 182149
2023-02-21 16:56:09,966 root INFO - Ext id:  pi:160256 Number of datapoints: 3228
Ready to insert datapoints... Tue Feb 21 16:56:09 2023
DATAPOINTS INSERTED AT:  Tue Feb 21 16:57:39 2023

Gaetan  Helness
MVP ⭐️⭐️⭐️⭐️⭐️
MVP ⭐️⭐️⭐️⭐️⭐️
February 21, 2023

Great to hear. Thank you for identifying this bug