Impact 2024: The Industrial Data and AI Conference for and by Users | Nominate Speakers Now for a Ch...
Thanks for the reply @matiasholte. Under the hood, the REST-extractor does POST the datapoints to the timeseries.```2023-10-31 13:57:39.081 UTC [DEBUG ] ThreadPoolExecutor-1_0 - HTTP Error 405 POST https://sao.fusion.cognite.com/api/v1/projects/https://sao.fusion.cognite.com/timeseries/data: <html><head><title>405 Not Allowed</title></head><body><center><h1>405 Not Allowed</h1></center><hr><center>nginx</center></body></html>```I’m setting up a notebook to “manually” upload the datapoints to the timeseries (without using the extractor library), and will see if that’s possible with the current setup. If that works, it must be something in the library? Has nobody seen this error type before?
To provide an update, everything works “manually” just using the SDK: ```ts_test_ext_id = ".Pixelite_CCU_C1-C1.Temperature" # pseudo ext_id matching test timeseries ext_id in CDFdatapoints = [(arrow.now().timestamp(), 10), (datetime(2018,1,2), 2)]client.time_series.data.insert(datapoints, external_id=ts_test_ext_id)```I can do a clean environment install of both cogex (python-extractor-utils) and the cogex REST extension, but is there a stable version that works out-of-the-box? E.g.the current version of python-extractor-utils does not include a retry.py, though the current version of the REST extractor does use one: python-extractor-utils-rest/cognite/extractorutils/rest/extractor.py at main · cognitedata/python-extractor-utils-rest (github.com) Also had to make the same changes as mentioned here: Cognite Hub to avoid errors when doing poetry run extractor-name.py config.yaml
Hi again Peter,Thanks for taking the time. Good idea. With the DEBUG logger, it seems the error does not complain about the POST payload, just the Cognite POST path (https://sao.fusion.cognite.com/api/v1/projects/https://sao.fusion.cognite.com/timeseries/data?):```2023-10-31 15:46:12.414 UTC [DEBUG ] ThreadPoolExecutor-1_0 - Starting new HTTPS connection (1): sao.fusion.cognite.com:4432023-10-31 15:46:12.585 UTC [DEBUG ] ThreadPoolExecutor-1_0 - https://sao.fusion.cognite.com:443 "POST /api/v1/projects/https://sao.fusion.cognite.com/timeseries/data HTTP/1.1" 405 1502023-10-31 15:46:12.586 UTC [DEBUG ] ThreadPoolExecutor-1_0 - HTTP Error 405 POST https://sao.fusion.cognite.com/api/v1/projects/https://sao.fusion.cognite.com/timeseries/data: <html><head><title>405 Not Allowed</title></head><body><center><h1>405 Not Allowed</h1></center><hr><center>nginx</center></body></html>```my `paths` varia
Ah I just set up my .env file wrong while debugging, though fixed it now. The address seems plausible, but the error persists:```2023-10-31 16:06:32.233 UTC [DEBUG ] ThreadPoolExecutor-1_0 - https://sao.fusion.cognite.com:443 "POST /api/v1/projects/sao-staging/timeseries/data HTTP/1.1" 405 1502023-10-31 16:06:32.234 UTC [DEBUG ] ThreadPoolExecutor-1_0 - HTTP Error 405 POST https://sao.fusion.cognite.com/api/v1/projects/sao-staging/timeseries/data: <html><head><title>405 Not Allowed</title></head><body><center><h1>405 Not Allowed</h1></center><hr><center>nginx</center></body></html>```We can rule out that it is credentials issue though, as the same Cognite client can be used to upload the datapoints using the python-SDK, instead of the REST extractor.
Ahh, good to know. I’ll try to dive into the library to see which variables to define. Currently, the config and env file looks like:### config.yamllogger: console: level: DEBUGcognite: # Read these from environment variables host: ${COGNITE_BASE_URL} project: ${COGNITE_PROJECT} # extraction_pipeline: # external_id: rest:imenco:ocean_farm_1:ts idp-authentication: token-url: ${COGNITE_TOKEN_URL} client-id: ${COGNITE_CLIENT_ID} secret: ${COGNITE_CLIENT_SECRET} scopes: - ${COGNITE_BASE_URL}/.default# source:# base_url: ${SOURCE_BASE_URL} # auth: # basic: # username: my-user # password: ${SOURCE_PASSWORD}### .env fileCOGNITE_BASE_URL=https://sao.fusion.cognite.comCOGNITE_PROJECT=sao-stagingCOGNITE_TOKEN_URL=https://login.windows.net/(removed tenant)/oauth2/tokenCOGNITE_CLIENT_ID=(removed)COGNITE_CLIENT_SECRET=(removed)SOURCE_BASE_URL=http://127.0.0.1:3000 #not used
Yep you’ve solved it. Saw the discrepancy with how my cognite client was set up in the python SDK notebook example compared to the example used in the REST example_config.yaml.In other words, this works:```COGNITE_BASE_URL=https://westeurope-1.cognitedata.comCOGNITE_PROJECT=sao-stagingCOGNITE_TOKEN_URL=https://login.windows.net/de10159d-2c09-4762-966c-e841d3391feb/oauth2/v2.0/token```Thank you Peter. Saving the project timeline as you do :)```2023-10-31 16:40:00.300 UTC [DEBUG ] ThreadPoolExecutor-1_0 - Starting new HTTPS connection (1): westeurope-1.cognitedata.com:4432023-10-31 16:40:00.564 UTC [DEBUG ] ThreadPoolExecutor-1_0 - https://westeurope-1.cognitedata.com:443 "POST /api/v1/projects/sao-staging/timeseries/data HTTP/1.1" 200 222023-10-31 16:40:00.574 UTC [DEBUG ] ThreadPoolExecutor-1_0 - HTTP/1.1 POST https://westeurope-1.cognitedata.com/api/v1/projects/sao-staging/timeseries/data 2002023-10-31 16:40:00.575 UTC [INFO ] MainThread - Uploaded 84 datapoints```
I’m sure you’re supposed to start with extractors after doing more courses and getting familiar with all CDF endpoints, so my bad :)An addition perhaps:- cognitedata/python-extractor-utils: Framework for developing extractors in Python (github.com)- Defining a config schema — cognite-extractor-utils 5.4.0 documentation (readthedocs-hosted.com) (or Extractor-utils Library for Cognite Python SDK)and hereREST extensionon config file configuration like you showed me can be a huge time saver. I completely missed the developer docs :D Happy halloween to you too!Best Patrick
As a follow-up, if we had the option to choose what monitoring and alerting solution to build/implement, does Cognite have a recommended approach? Would it be to start with the Cognite Charts alerting functionality, or to use AIR, or to be to build one for example using the Cognite SDK, Cognite functions or in other 3rd party software e.g. Grafana?
Great guide! Just one small edit I discovered when following the guide: When trying to run the docker image, you suggest mounting the config folder on root ‘/’ with `$PWD/config:/config`, then passing the path of the config as an arg to the run docker command:docker run --env-file=.env -v $PWD/config:/config rest-extractor config/config.yaml But since the docker application working dir is ‘/usr/src/app’ the entrypoint searches ‘/usr/src/app/config/config.yaml’, which doesn’t exist. So instead specifying the new mounted location of the config folder, ‘/config/config.yaml’ works:docker run --env-file=.env -v $PWD/config:/config rest-extractor /config/config.yaml
Already have an account? Login
Enter your E-mail address. We'll send you an e-mail with instructions to reset your password.
Sorry, we're still checking this file's contents to make sure it's safe to download. Please try again in a few minutes.
Sorry, our virus scanner detected that this file isn't safe to download.
Check the
documentation
Ask the
Community
Take a look
at
Academy
Cognite
Status
Page
Contact
Cognite Support
