SOLVED:
The Cognite client was set up wrong, hence the Cognite API error when uploading data to CDF. See docs posted by Peter for setting up config.yaml for the extractor. Specifically these were the changes:
```
COGNITE_BASE_URL=https://westeurope-1.cognitedata.com (not Fusion URL)
COGNITE_PROJECT= {some project name}
COGNITE_TOKEN_URL= https://login.windows.net/{some_tenant}/oauth2/v2.0/token
(Different from OpenID connect token URL)
```
_______________________________________________
Hello curious and helpful members of Cognite Hub,
I’m building my first REST extractor to GET a response (sensor data) from a camera_sensor API, where I then aim to upload the data to corresponding timeseries. I have multiple cameras with each its URL path, so it seems @extractor.get_multiple(paths) gives me the desired behaviour of essentially looping @extractor.get through the URLs.
However, this error arises when my extractor tries to upload datapoints to the timeseries in CDF.
^CTraceback (most recent call last):
File "/Users/patrick.nitschke/opt/anaconda3/envs/cdf_new/lib/python3.11/site-packages/cognite/extractorutils/util.py", line 344, in _retry_internal
return f()
^^^
File "/Users/patrick.nitschke/opt/anaconda3/envs/cdf_new/lib/python3.11/site-packages/cognite/extractorutils/uploader/time_series.py", line 242, in _upload_batch
self.cdf_client.time_series.data.insert_multiple(upload_this)
File "/Users/patrick.nitschke/opt/anaconda3/envs/cdf_new/lib/python3.11/site-packages/cognite/client/_api/datapoints.py", line 1340, in insert_multiple
dps_poster.insert(datapoints)
File "/Users/patrick.nitschke/opt/anaconda3/envs/cdf_new/lib/python3.11/site-packages/cognite/client/_api/datapoints.py", line 1479, in insert
self._insert_datapoints_concurrently(binned_dps_object_lists)
File "/Users/patrick.nitschke/opt/anaconda3/envs/cdf_new/lib/python3.11/site-packages/cognite/client/_api/datapoints.py", line 1550, in _insert_datapoints_concurrently
summary.raise_compound_exception_if_failed_tasks(
File "/Users/patrick.nitschke/opt/anaconda3/envs/cdf_new/lib/python3.11/site-packages/cognite/client/utils/_concurrency.py", line 64, in raise_compound_exception_if_failed_tasks
collect_exc_info_and_raise(
File "/Users/patrick.nitschke/opt/anaconda3/envs/cdf_new/lib/python3.11/site-packages/cognite/client/utils/_concurrency.py", line 101, in collect_exc_info_and_raise
raise CogniteAPIError(
cognite.client.exceptions.CogniteAPIError: <html>
<head><title>405 Not Allowed</title></head>
<body>
<center><h1>405 Not Allowed</h1></center>
<hr><center>nginx</center>
</body>
</html>
| code: 405 | X-Request-ID: None
The API Failed to process some items.
Successful (2xx): u]
Unknown (5xx): /]
Failed (4xx): {'externalId': 'Pixelite_CCU_C10-C1.Temperature'}, {'externalId': 'Pixelite_CCU_C10-C1.Depth'}, ...(more failed InsertDatapoints)where I traced this to a failure in line 222 (res = f.result()) in the execute_tasks() function of cognite/client/utils/_concurrency.py,
I also checked the variables “being executed”:
tasks:
<(e{'externalId': 'Pixelite_CCU_C10-C1.Temperature', 'datapoints': r(1698690892, 7.7)]}, …(more InsertDatapoints...) , {'externalId': 'Pixelite_CCU_C4-C3.topCameraHeading', 'datapoints': m(1698690892, 150.0)]}, {'externalId': 'Pixelite_CCU_C4-C3.bottomCameraHeading', 'datapoints': m(1698690892, 205.0)]}],)]and func:
<bound method DatapointsPoster._insert_datapoints of <cognite.client._api.datapoints.DatapointsPoster object at 0x10f54a750>>So how can I fix this error?
Looking at this conversation Cognite Hub, everything seems ok.
Code:
This is what I have so far:
### dto.py ###
@dataclass
class SensorData:
Temperature: str
Depth: str
OxygenLevel: float
OxygenLevelPercent: float
internalHeading: int
topCameraHeading: int
bottomCameraHeading: int
@dataclass
class ImencoCameraData:
id: str
OxygenSensorModel: str
CameraSensors: ListrSensorData]
connected : bool
### extractor.py ###
@extractor.get_multiple(paths=paths, response_type=ImencoCameraData, interval=60)
def imenco_response_to_datapoints_handler(imenco_camera_data: ImencoCameraData) -> GeneratornInsertDatapoints, None, None]:
"""
Receive responses of type 'ImencoCameraData' from GET request to decorated function 'paths'.
Transform response to yield a generator of 'InsertDatapoints', which is then uploaded to CDF.
A response has 7 sensor values, which means we yield 7 InsertDatapoints - each with a unique
externalId and a single Datapoint.
Parameters:
imenco_camera_data (ImencoCameraData): Response from GET request to Imenco IP.
Returns:
A generator of 'InsertDatapoints' that will be uploaded to CDF.
Constructor: InsertDatapoints(
external_id,
ListeDatapoint], where Datapoint = TupleoTimeStamp, float]
)'
"""
camera_name = imenco_path_to_camera_name_mapicamera_id_to_path_mapnimenco_camera_data.id]]
ox_sensor_model = imenco_camera_data.OxygenSensorModel
connected_status = imenco_camera_data.id
# list with 1 'SensorData' value
for camera_sensors in imenco_camera_data.CameraSensors:
for data_field in fields(camera_sensors):
timestamp = arrow.now().timestamp()
ts_ext_id = f"{camera_name}.{data_field.name}" #pseudo ext_id for this example
value = float(getattr(camera_sensors, data_field.name))
print(timestamp, ts_ext_id, type(value), value)
yield InsertDatapoints(
# id=imenco_ext_id_to_id_map ts_ext_id],
external_id=ts_ext_id,
datapoints= (timestamp, value)]
)where the raw data from `print(timestamp, ts_ext_id, type(value), value)` looks like:
1698688899.664962 Pixelite_CCU_C1-C2.bottomCameraHeading <class 'float'> 205.0
1698688899.665888 Pixelite_CCU_C7-C3.bottomCameraHeading <class 'float'> 205.0
1698688899.667632 Pixelite_CCU_C4-C1.Temperature <class 'float'> 8.4
1698688899.667749 Pixelite_CCU_C4-C1.Depth <class 'float'> 2.2
1698688899.667816 Pixelite_CCU_C4-C1.OxygenLevel <class 'float'> 10.164129140950386
1698688899.66788 Pixelite_CCU_C4-C1.OxygenLevelPercent <class 'float'> 110.06