Skip to main content
Seasoned ⭐️⭐️
jaydeep
Seasoned ⭐️⭐️
February 17, 2023

Error running the Cognite replicator with Option 5

I have set up both the environment variables SOURCE_CLIENT_SECRET (Note that the Client secret is generated from the OID widget as per the documentation)and DEST_CLIENT_SECRET and running the replicator with the config file using option 5.I am getting error as per below log

AMAC02Z3123LVCJ:oid-replicator j.subhash.parandekar$ poetry run python3 ./oid_replicator/replicate.py 
2023-02-17 18:46:48,239 root INFO - Config file - Repeat line 5: 

2023-02-17 18:46:48,239 root INFO - Config file - Repeat line 14: 

2023-02-17 18:46:48,239 root INFO - Config file - Repeat line 23: 

Starting replication of resources
Replicating assets...
Traceback (most recent call last):
  File "/Users/j.subhash.parandekar/Library/Caches/pypoetry/virtualenvs/oid-replicator-cwaK-6ym-py3.11/lib/python3.11/site-packages/cognite/client/credentials.py", line 364, in _refresh_access_token
    token_result = self.__oauth.fetch_token(
                   ^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/j.subhash.parandekar/Library/Caches/pypoetry/virtualenvs/oid-replicator-cwaK-6ym-py3.11/lib/python3.11/site-packages/requests_oauthlib/oauth2_session.py", line 366, in fetch_token
    self._client.parse_request_body_response(r.text, scope=self.scope)
  File "/Users/j.subhash.parandekar/Library/Caches/pypoetry/virtualenvs/oid-replicator-cwaK-6ym-py3.11/lib/python3.11/site-packages/oauthlib/oauth2/rfc6749/clients/base.py", line 427, in parse_request_body_response
    self.token = parse_token_response(body, scope=scope)
                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/j.subhash.parandekar/Library/Caches/pypoetry/virtualenvs/oid-replicator-cwaK-6ym-py3.11/lib/python3.11/site-packages/oauthlib/oauth2/rfc6749/parameters.py", line 441, in parse_token_response
    validate_token_parameters(params)
  File "/Users/j.subhash.parandekar/Library/Caches/pypoetry/virtualenvs/oid-replicator-cwaK-6ym-py3.11/lib/python3.11/site-packages/oauthlib/oauth2/rfc6749/parameters.py", line 448, in validate_token_parameters
    raise_from_error(params.get('error'), params)
  File "/Users/j.subhash.parandekar/Library/Caches/pypoetry/virtualenvs/oid-replicator-cwaK-6ym-py3.11/lib/python3.11/site-packages/oauthlib/oauth2/rfc6749/errors.py", line 399, in raise_from_error
    raise cls(**kwargs)
oauthlib.oauth2.rfc6749.errors.InvalidClientError: (invalid_client) AADSTS7000216: 'client_assertion', 'client_secret' or 'request' is required for the 'client_credentials' grant type.
Trace ID: b0528310-0b70-4ee0-9715-daca09e35200
Correlation ID: 5987d7dd-eb61-404c-9525-00954829211c
Timestamp: 2023-02-17 13:16:49Z

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/Users/j.subhash.parandekar/oid-replicator/./oid_replicator/replicate.py", line 8, in <module>
    main()
  File "/Users/j.subhash.parandekar/Library/Caches/pypoetry/virtualenvs/oid-replicator-cwaK-6ym-py3.11/lib/python3.11/site-packages/cognite/replicator/__main__.py", line 401, in main
    cognite.replicator.assets.replicate(
  File "/Users/j.subhash.parandekar/Library/Caches/pypoetry/virtualenvs/oid-replicator-cwaK-6ym-py3.11/lib/python3.11/site-packages/cognite/replicator/assets.py", line 298, in replicate
    assets_src = client_src.assets.list(limit=None)
                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/j.subhash.parandekar/Library/Caches/pypoetry/virtualenvs/oid-replicator-cwaK-6ym-py3.11/lib/python3.11/site-packages/cognite/client/_api/assets.py", line 295, in list
    return self._list(
           ^^^^^^^^^^^
  File "/Users/j.subhash.parandekar/Library/Caches/pypoetry/virtualenvs/oid-replicator-cwaK-6ym-py3.11/lib/python3.11/site-packages/cognite/client/_api_client.py", line 489, in _list
    for resource_list in self._list_generator(
  File "/Users/j.subhash.parandekar/Library/Caches/pypoetry/virtualenvs/oid-replicator-cwaK-6ym-py3.11/lib/python3.11/site-packages/cognite/client/_api_client.py", line 395, in _list_generator
    res = self._post(url_path=url_path or resource_path + "/list", json=body, headers=headers)
          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/j.subhash.parandekar/Library/Caches/pypoetry/virtualenvs/oid-replicator-cwaK-6ym-py3.11/lib/python3.11/site-packages/cognite/client/_api_client.py", line 135, in _post
    return self._do_request(
           ^^^^^^^^^^^^^^^^^
  File "/Users/j.subhash.parandekar/Library/Caches/pypoetry/virtualenvs/oid-replicator-cwaK-6ym-py3.11/lib/python3.11/site-packages/cognite/client/_api_client.py", line 148, in _do_request
    headers = self._configure_headers(accept, additional_headers=self._config.headers.copy())
              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/j.subhash.parandekar/Library/Caches/pypoetry/virtualenvs/oid-replicator-cwaK-6ym-py3.11/lib/python3.11/site-packages/cognite/client/_api_client.py", line 190, in _configure_headers
    auth_header_name, auth_header_value = self._config.credentials.authorization_header()
                                          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/j.subhash.parandekar/Library/Caches/pypoetry/virtualenvs/oid-replicator-cwaK-6ym-py3.11/lib/python3.11/site-packages/cognite/client/credentials.py", line 121, in authorization_header
    self.__access_token, self.__access_token_expires_at = self._refresh_access_token()
                                                          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/j.subhash.parandekar/Library/Caches/pypoetry/virtualenvs/oid-replicator-cwaK-6ym-py3.11/lib/python3.11/site-packages/cognite/client/credentials.py", line 372, in _refresh_access_token
    raise CogniteAuthError(
cognite.client.exceptions.CogniteAuthError: Error generating access token: invalid_client, 401, AADSTS7000216: 'client_assertion', 'client_secret' or 'request' is required for the 'client_credentials' grant type.
Trace ID: b0528310-0b70-4ee0-9715-daca09e35200
Correlation ID: 5987d7dd-eb61-404c-9525-00954829211c
Timestamp: 2023-02-17 13:16:49Z

    16 replies

    Gaetan Helness
    MVP ⭐️⭐️⭐️⭐️⭐️
    Gaetan Helness
    MVP ⭐️⭐️⭐️⭐️⭐️
    February 21, 2023

    Great to hear. Thank you for identifying this bug

    Seasoned ⭐️⭐️
    jaydeepAuthor
    Seasoned ⭐️⭐️
    February 21, 2023

    yes , after upgrading the cognite-replicator package to 1.2.3 this error is resolved and I am able to execute as per below. Thanks

    2023-02-21 16:56:01,725 root INFO - Ext id:  pi:160571 Number of datapoints: 0
    2023-02-21 16:56:04,095 root INFO - Ext id:  pi:160699 Number of datapoints: 219394
    2023-02-21 16:56:04,138 root INFO - Ext id:  pi:160265 Number of datapoints: 3864
    2023-02-21 16:56:04,138 root INFO - Ext id:  pi:161026 Number of datapoints: 0
    2023-02-21 16:56:05,471 root INFO - Ext id:  houston.ro.REMOTE_AI[4] Number of datapoints: 122735
    2023-02-21 16:56:05,471 root INFO - Ext id:  pi:160551 Number of datapoints: 0
    2023-02-21 16:56:05,511 root INFO - Ext id:  houston.ro.REMOTE_AI[75] Number of datapoints: 3590
    2023-02-21 16:56:07,950 root INFO - Ext id:  pi:160672 Number of datapoints: 224161
    2023-02-21 16:56:07,951 root INFO - Ext id:  pi:160222 Number of datapoints: 0
    2023-02-21 16:56:09,931 root INFO - Ext id:  pi:160766 Number of datapoints: 182149
    2023-02-21 16:56:09,966 root INFO - Ext id:  pi:160256 Number of datapoints: 3228
    Ready to insert datapoints... Tue Feb 21 16:56:09 2023
    DATAPOINTS INSERTED AT:  Tue Feb 21 16:57:39 2023

    Gaetan Helness
    MVP ⭐️⭐️⭐️⭐️⭐️
    Gaetan Helness
    MVP ⭐️⭐️⭐️⭐️⭐️
    February 21, 2023

    Could you upgrade the package to the latest version

    pip install cognite-replicator --upgrade

    Version should be 1.2.3
     

    Gaetan Helness
    MVP ⭐️⭐️⭐️⭐️⭐️
    Gaetan Helness
    MVP ⭐️⭐️⭐️⭐️⭐️
    February 21, 2023

    Ok, apologies, i think you found a bug, we will look into it and fix it today

    Gaetan Helness
    MVP ⭐️⭐️⭐️⭐️⭐️
    Gaetan Helness
    MVP ⭐️⭐️⭐️⭐️⭐️
    February 21, 2023

    What is the value of  “batch_size”? It looks like it is an dictionary. 
    its value is set in the config file for example

    len(ts_src) is an int and printed above (408)

    batch_size should also be an int, can be set to 10000 for example

    Seasoned ⭐️⭐️
    jaydeepAuthor
    Seasoned ⭐️⭐️
    February 21, 2023

    I am trying to replicate timeseries and data points  however I am getting below error

    AMAC02Z3123LVCJ:oid-replicator j.subhash.parandekar$ /usr/local/bin/python3 /Users/j.subhash.parandekar/oid-replicator/oid_replicator/replicate.py
    2023-02-21 11:31:43,353 root INFO - Config file - Repeat line 5: 

    2023-02-21 11:31:43,353 root INFO - Config file - Repeat line 15: 

    2023-02-21 11:31:43,353 root INFO - Config file - Repeat line 24: 

    Starting replication of resources
    Replicating time series...
    2023-02-21 11:31:53,330 root INFO - There are 408 existing time series in source (publicdata).
    2023-02-21 11:31:53,330 root INFO - There are 10291 existing time series in destination (accenture-tiger-training).
    2023-02-21 11:31:57,719 root INFO - If a time series asset id is one of the 1115 assets that have been replicated then it will be linked.
    2023-02-21 11:31:57,719 root INFO - These copied/updated time series will have a replicated run time of: 1676959317000.
    2023-02-21 11:31:57,719 root INFO - Starting to copy and update 408 time series from source (publicdata) to destination (accenture-tiger-training).
    Traceback (most recent call last):
      File "/Users/j.subhash.parandekar/oid-replicator/oid_replicator/replicate.py", line 8, in <module>
        main()
      File "/Library/Frameworks/Python.framework/Versions/3.11/lib/python3.11/site-packages/cognite/replicator/__main__.py", line 427, in main
        cognite.replicator.time_series.replicate(
      File "/Library/Frameworks/Python.framework/Versions/3.11/lib/python3.11/site-packages/cognite/replicator/time_series.py", line 298, in replicate
        if len(ts_src) > batch_size:
           ^^^^^^^^^^^^^^^^^^^^^^^^
    TypeError: '>' not supported between instances of 'int' and 'dict'

    Mithila Jayalath
    Expert ⭐️⭐️⭐️⭐️
    Mithila Jayalath
    Expert ⭐️⭐️⭐️⭐️
    February 20, 2023

    raise CogniteAPIError(msg, code, x_request_id, missing=missing, duplicated=duplicated, extra=extra)

    cognite.client.exceptions.CogniteAPIError: Unauthorized | code: 401 | X-Request-ID: d838efa4-540a-95cb-bbee-65502e48b41a

    As per the above error that you have shared, it is not an error due to insufficient access rights. It looks like the access token has expired by the time you have tried to replicate the events.
     
    Since there is a huge number of assets, events, time series, etc. It will take some time to replicate them all at once. Which will result in an access token expiration. It would be better if you can replicate one resource type at a time to avoid token expiration.

    Practitioner ⭐️⭐️⭐️
    Viraj Dhanusha
    Practitioner ⭐️⭐️⭐️
    February 20, 2023

    Hi @jaydeep 

     

    I’m Viraj from Cognite support and we have received your ticket related to this issue. We will follow up via the support ticket and I hope its okay for you. 

     

    Best regards, 
    Viraj 

    Seasoned ⭐️⭐️
    jaydeepAuthor
    Seasoned ⭐️⭐️
    February 20, 2023

    created support request to check the access rights on the events resource type for the publicdata project

    https://cognite.zendesk.com/hc/requests/7756

      Expert ⭐️⭐️⭐️⭐️
      Carin Meems
      Expert ⭐️⭐️⭐️⭐️
      February 20, 2023

      Hi @jaydeep, 

      Would you be able to create a ticket for Cognite Support via support@cognite.com, mentioning your issue and sharing your request ID? With that, they should be able to see if you have enough rights.


      Best,

      Carin 

      Show more replies
      Terms & ConditionsAccessibility statement