Replies posted by patricknitschke

9 Replies

Pierre Pernot
Practitioner
Pierre PernotPractitioner
posted in How-To Guides

How to package a custom extractor as a Docker image [Cognite Official]

Great guide! Just one small edit I discovered when following the guide: When trying to run the docker image, you suggest mounting the config folder on root ‘/’ with `$PWD/config:/config`, then passing the path of the config as an arg to the run docker command:docker run --env-file=.env -v $PWD/config:/config rest-extractor config/config.yaml But since the docker application working dir is ‘/usr/src/app’ the entrypoint searches  ‘/usr/src/app/config/config.yaml’, which doesn’t exist. So instead specifying the new mounted location of the config folder, ‘/config/config.yaml’ works:docker run --env-file=.env -v $PWD/config:/config rest-extractor /config/config.yaml  

3
P
Committed
5 months ago
K
Committed
KjerstiCommitted
posted in Monitoring and Alerting [Early Adopter]

Monitoring CDF in grafana

As a follow-up, if we had the option to choose what monitoring and alerting solution to build/implement, does Cognite have a recommended approach? Would it be to start with the Cognite Charts alerting functionality, or to use AIR, or to be to build one for example using the Cognite SDK, Cognite functions or in other 3rd party software e.g. Grafana?

4
I
Practitioner
6 months ago
P
Committed
patricknitschkeCommitted
asked in Developer and User Community

Uploading InsertDatapoints via REST extractor - Getting a CogniteAPIError code 405 Not Allowed Error

I’m sure you’re supposed to start with extractors after doing more courses and getting familiar with all CDF endpoints, so my bad :)An addition perhaps:- cognitedata/python-extractor-utils: Framework for developing extractors in Python (github.com)- Defining a config schema — cognite-extractor-utils 5.4.0 documentation (readthedocs-hosted.com) (or Extractor-utils Library for Cognite Python SDK)and hereREST extensionon config file configuration like you showed me can be a huge time saver. I completely missed the developer docs :D Happy halloween to you too!Best Patrick

14
P
Committed
6 months ago
P
Committed
patricknitschkeCommitted
asked in Developer and User Community

Uploading InsertDatapoints via REST extractor - Getting a CogniteAPIError code 405 Not Allowed Error

Yep you’ve solved it. Saw the discrepancy with how my cognite client was set up in the python SDK notebook example compared to the example used in the REST example_config.yaml.In other words, this works:```COGNITE_BASE_URL=https://westeurope-1.cognitedata.comCOGNITE_PROJECT=sao-stagingCOGNITE_TOKEN_URL=https://login.windows.net/de10159d-2c09-4762-966c-e841d3391feb/oauth2/v2.0/token```Thank you Peter. Saving the project timeline as you do :)```2023-10-31 16:40:00.300 UTC [DEBUG   ] ThreadPoolExecutor-1_0 - Starting new HTTPS connection (1): westeurope-1.cognitedata.com:4432023-10-31 16:40:00.564 UTC [DEBUG   ] ThreadPoolExecutor-1_0 - https://westeurope-1.cognitedata.com:443 "POST /api/v1/projects/sao-staging/timeseries/data HTTP/1.1" 200 222023-10-31 16:40:00.574 UTC [DEBUG   ] ThreadPoolExecutor-1_0 - HTTP/1.1 POST https://westeurope-1.cognitedata.com/api/v1/projects/sao-staging/timeseries/data 2002023-10-31 16:40:00.575 UTC [INFO    ] MainThread - Uploaded 84 datapoints```

14
P
Committed
6 months ago
P
Committed
patricknitschkeCommitted
asked in Developer and User Community

Uploading InsertDatapoints via REST extractor - Getting a CogniteAPIError code 405 Not Allowed Error

Ahh, good to know. I’ll try to dive into the library to see which variables to define. Currently, the config and env file looks like:### config.yamllogger: console: level: DEBUGcognite: # Read these from environment variables host: ${COGNITE_BASE_URL} project: ${COGNITE_PROJECT} # extraction_pipeline: # external_id: rest:imenco:ocean_farm_1:ts idp-authentication: token-url: ${COGNITE_TOKEN_URL} client-id: ${COGNITE_CLIENT_ID} secret: ${COGNITE_CLIENT_SECRET} scopes: - ${COGNITE_BASE_URL}/.default# source:# base_url: ${SOURCE_BASE_URL} # auth: # basic: # username: my-user # password: ${SOURCE_PASSWORD}### .env fileCOGNITE_BASE_URL=https://sao.fusion.cognite.comCOGNITE_PROJECT=sao-stagingCOGNITE_TOKEN_URL=https://login.windows.net/(removed tenant)/oauth2/tokenCOGNITE_CLIENT_ID=(removed)COGNITE_CLIENT_SECRET=(removed)SOURCE_BASE_URL=http://127.0.0.1:3000 #not used

14
P
Committed
6 months ago
P
Committed
patricknitschkeCommitted
asked in Developer and User Community

Uploading InsertDatapoints via REST extractor - Getting a CogniteAPIError code 405 Not Allowed Error

Ah I just set up my .env file wrong while debugging, though fixed it now. The address seems plausible, but the error persists:```2023-10-31 16:06:32.233 UTC [DEBUG   ] ThreadPoolExecutor-1_0 - https://sao.fusion.cognite.com:443 "POST /api/v1/projects/sao-staging/timeseries/data HTTP/1.1" 405 1502023-10-31 16:06:32.234 UTC [DEBUG   ] ThreadPoolExecutor-1_0 - HTTP Error 405 POST https://sao.fusion.cognite.com/api/v1/projects/sao-staging/timeseries/data: <html><head><title>405 Not Allowed</title></head><body><center><h1>405 Not Allowed</h1></center><hr><center>nginx</center></body></html>```We can rule out that it is credentials issue though, as the same Cognite client can be used to upload the datapoints using the python-SDK, instead of the REST extractor.

14
P
Committed
6 months ago
P
Committed
patricknitschkeCommitted
asked in Developer and User Community

Uploading InsertDatapoints via REST extractor - Getting a CogniteAPIError code 405 Not Allowed Error

Hi again Peter,Thanks for taking the time. Good idea. With the DEBUG logger, it seems the error does not complain about the POST payload, just the Cognite POST path (https://sao.fusion.cognite.com/api/v1/projects/https://sao.fusion.cognite.com/timeseries/data?):```2023-10-31 15:46:12.414 UTC [DEBUG   ] ThreadPoolExecutor-1_0 - Starting new HTTPS connection (1): sao.fusion.cognite.com:4432023-10-31 15:46:12.585 UTC [DEBUG   ] ThreadPoolExecutor-1_0 - https://sao.fusion.cognite.com:443 "POST /api/v1/projects/https://sao.fusion.cognite.com/timeseries/data HTTP/1.1" 405 1502023-10-31 15:46:12.586 UTC [DEBUG   ] ThreadPoolExecutor-1_0 - HTTP Error 405 POST https://sao.fusion.cognite.com/api/v1/projects/https://sao.fusion.cognite.com/timeseries/data: <html><head><title>405 Not Allowed</title></head><body><center><h1>405 Not Allowed</h1></center><hr><center>nginx</center></body></html>```my `paths` varia

14
P
Committed
6 months ago
P
Committed
patricknitschkeCommitted
asked in Developer and User Community

Uploading InsertDatapoints via REST extractor - Getting a CogniteAPIError code 405 Not Allowed Error

To provide an update, everything works “manually” just using the SDK: ```ts_test_ext_id = ".Pixelite_CCU_C1-C1.Temperature" # pseudo ext_id matching test timeseries ext_id in CDFdatapoints = [(arrow.now().timestamp(), 10), (datetime(2018,1,2), 2)]client.time_series.data.insert(datapoints, external_id=ts_test_ext_id)```I can do a clean environment install of both cogex (python-extractor-utils) and the cogex REST extension, but is there a stable version that works out-of-the-box? E.g.the current version of python-extractor-utils does not include a retry.py, though the current version of the REST extractor does use one: python-extractor-utils-rest/cognite/extractorutils/rest/extractor.py at main · cognitedata/python-extractor-utils-rest (github.com) Also had to make the same changes as mentioned here: Cognite Hub to avoid errors when doing poetry run extractor-name.py config.yaml  

14
P
Committed
6 months ago
P
Committed
patricknitschkeCommitted
asked in Developer and User Community

Uploading InsertDatapoints via REST extractor - Getting a CogniteAPIError code 405 Not Allowed Error

Thanks for the reply @matiasholte. Under the hood, the REST-extractor does POST the datapoints to the timeseries.```2023-10-31 13:57:39.081 UTC [DEBUG   ] ThreadPoolExecutor-1_0 - HTTP Error 405 POST https://sao.fusion.cognite.com/api/v1/projects/https://sao.fusion.cognite.com/timeseries/data: <html><head><title>405 Not Allowed</title></head><body><center><h1>405 Not Allowed</h1></center><hr><center>nginx</center></body></html>```I’m setting up a notebook to “manually” upload the datapoints to the timeseries (without using the extractor library), and will see if that’s possible with the current setup. If that works, it must be something in the library? Has nobody seen this error type before?

14
P
Committed
6 months ago

Badge winners

  • Cognite Data Fusion Fundamentals
    Mohammad Mateen Khanhas earned the badge Cognite Data Fusion Fundamentals
  • Working with Cognite InField
    AnnaMariahas earned the badge Working with Cognite InField
  • Cognite Data Fusion Fundamentals
    RERPALMEROhas earned the badge Cognite Data Fusion Fundamentals
  • Cognite Data Fusion Fundamentals
    Yushu Rajhas earned the badge Cognite Data Fusion Fundamentals
  • Cognite Data Fusion Fundamentals
    FredMartinezhas earned the badge Cognite Data Fusion Fundamentals
Show all badges
Terms & ConditionsCookie Settings