Skip to main content
Solved

Cognite - data modelling SDK query instances giving timed out error

  • 19 June 2024
  • 5 replies
  • 58 views

Hello team,

We are trying to use the data modeling query instances SDK method.

client.data_modeling.instances.query(query)

 

our query is:

query {
"with": {
"Min_15_50ts": {
"nodes": {
"filter": {
"and":
{
"hasData":
{
"type": "view",
"space": "QA_Test_Space",
"externalId": "Min_15_50ts",
"version": "3a62f20104e445"
}
]
},
{
"in": {
"property":
"QA_Test_Space",
"Min_15_50ts/3a62f20104e445",
"entity"
],
"values":
"QA_Perf_WELL_1",
"QA_Perf_WELL_2",
"QA_Perf_WELL_3",
"QA_Perf_WELL_4",
"QA_Perf_WELL_5",
"QA_Perf_WELL_6",
"QA_Perf_WELL_7",
"QA_Perf_WELL_8",
"QA_Perf_WELL_9",
"QA_Perf_WELL_10",
"QA_Perf_WELL_11",
"QA_Perf_WELL_12",
"QA_Perf_WELL_13",
"QA_Perf_WELL_14",
"QA_Perf_WELL_15",
"QA_Perf_WELL_16"
]
}
}
]
},
"chain_to": "destination",
"direction": "outwards"
},
"limit": 1000
}
},
"select": {
"Min_15_50ts": {
"sources":
{
"source": {
"space": "QA_Test_Space",
"external_id": "Min_15_50ts",
"version": "3a62f20104e445",
"type": "view"
},
"properties":
"timeseries_1",
"entity"
]
}
]
}
},
"cursors": {
"Min_15_50ts": null
}
}

for this we are getting the following error:

cognite.client.exceptions.CogniteAPIError: Graph query timed out. Reduce load or contention, or optimise your query. | code: 408 | X-Request-ID: 8d863ca0-3605-93ac-a893-a7c80b7fae8b

 

We are using 7.38.2 version of the SDK.

Kindly let us know how we can make this run.

 

 

5 replies

Userlevel 2

Hi @Gargi Bhosale, firstly, you don’t require either the chain_to or direction property here. Chain_to is only needed when you are chaining to another part of your query, and direction for nodes is only required when you specify a direct relation in 'through’ property.

Secondly, have you set up an index on the entity property? If not, could you try adding it, as this should significantly improve query performance here. Could you also mention the total instances you might have in the 'Min_15_50ts’ container/view?

Userlevel 1
Badge

Hello Uzair,

 

Thank you for your prompt response .

We have a few observations:

 

  1. for the chain_to or direction property you were mentioning, even if we change that property, this issue/failure is not constant, it is intermittent. So, we don't think changing that would make a difference, as if this was an issue the query would never run successfully.
  2.  We have added indexing to index property, it is running successfully, but we tried without indexing too it is also running. So, we think this is an intermittent issue which doesn't depend on indexing
  3. Min_15_50ts has 5,25,000 instances.

 

Please let us know how we can avoid this intermittent issue for ensuring that our code runs successfully in every run.

Your help is much appreciated.

Userlevel 2

Hi,

So, to clarify, do you still run into the same issue when you have the entity property indexed?

I understand that the query intermittently fails on the non-indexed scenario with occasional success. This is to be expected with this sort of filtering and data volume, and hence why indexing the properties that you would be querying on is important. If the same issue happens after the property is indexed than this can be a performance issue in the service worth investigating.

Do you mind sharing the container configuration, that shows indexes applied?

Regards

Userlevel 1
Badge

Hi,

 

We are not running into the same issue after indexing the entity property.

 

So, just to clarify and draw a conclusion, we will have to index those properties on which we would be applying filters.

 

Sharing the container configuration below:

[
{
"name": "Min_15_50ts",
"space": "QA_Test_Space",
"externalId": "Min_15_50ts",
"usedFor": "node",
"createdTime": 1690541696938,
"lastUpdatedTime": 1690541696938,
"isGlobal": false,
"constraints": {},
"indexes": {
"propertyNameIndex": {
"properties": [
"propertyName"
],
"indexType": "btree",
"cursorable": false,
"bySpace": false
},
"entityIndex": {
"properties": [
"entity"
],
"indexType": "btree",
"cursorable": false,
"bySpace": false
}
},
"kind": "container",
"fields": [
{
"externalId": "timeseries_1",
"name": "timeseries_1",
"type": {
"name": "TimeSeries",
"list": false,
"nonNull": false,
"custom": false,
"type": "timeseries",
"kind": "scalar"
},
"containerType": {
"type": "timeseries",
"list": false
},
"autoIncrement": false
},
{
"externalId": "timeseries_2",
"name": "timeseries_2",
"type": {
"name": "TimeSeries",
"list": false,
"nonNull": false,
"custom": false,
"type": "timeseries",
"kind": "scalar"
},
"containerType": {
"type": "timeseries",
"list": false
},
"autoIncrement": false
},
{
"externalId": "timeseries_3",
"name": "timeseries_3",
"type": {
"name": "TimeSeries",
"list": false,
"nonNull": false,
"custom": false,
"type": "timeseries",
"kind": "scalar"
},
"containerType": {
"type": "timeseries",
"list": false
},
"autoIncrement": false
},
{
"externalId": "timeseries_4",
"name": "timeseries_4",
"type": {
"name": "TimeSeries",
"list": false,
"nonNull": false,
"custom": false,
"type": "timeseries",
"kind": "scalar"
},
"containerType": {
"type": "timeseries",
"list": false
},
"autoIncrement": false
},
{
"externalId": "timeseries_5",
"name": "timeseries_5",
"type": {
"name": "TimeSeries",
"list": false,
"nonNull": false,
"custom": false,
"type": "timeseries",
"kind": "scalar"
},
"containerType": {
"type": "timeseries",
"list": false
},
"autoIncrement": false
},
{
"externalId": "timeseries_6",
"name": "timeseries_6",
"type": {
"name": "TimeSeries",
"list": false,
"nonNull": false,
"custom": false,
"type": "timeseries",
"kind": "scalar"
},
"containerType": {
"type": "timeseries",
"list": false
},
"autoIncrement": false
},
{
"externalId": "timeseries_7",
"name": "timeseries_7",
"type": {
"name": "TimeSeries",
"list": false,
"nonNull": false,
"custom": false,
"type": "timeseries",
"kind": "scalar"
},
"containerType": {
"type": "timeseries",
"list": false
},
"autoIncrement": false
},
{
"externalId": "timeseries_8",
"name": "timeseries_8",
"type": {
"name": "TimeSeries",
"list": false,
"nonNull": false,
"custom": false,
"type": "timeseries",
"kind": "scalar"
},
"containerType": {
"type": "timeseries",
"list": false
},
"autoIncrement": false
},
{
"externalId": "timeseries_9",
"name": "timeseries_9",
"type": {
"name": "TimeSeries",
"list": false,
"nonNull": false,
"custom": false,
"type": "timeseries",
"kind": "scalar"
},
"containerType": {
"type": "timeseries",
"list": false
},
"autoIncrement": false
},
{
"externalId": "timeseries_10",
"name": "timeseries_10",
"type": {
"name": "TimeSeries",
"list": false,
"nonNull": false,
"custom": false,
"type": "timeseries",
"kind": "scalar"
},
"containerType": {
"type": "timeseries",
"list": false
},
"autoIncrement": false
},
{
"externalId": "timeseries_11",
"name": "timeseries_11",
"type": {
"name": "TimeSeries",
"list": false,
"nonNull": false,
"custom": false,
"type": "timeseries",
"kind": "scalar"
},
"containerType": {
"type": "timeseries",
"list": false
},
"autoIncrement": false
},
{
"externalId": "timeseries_12",
"name": "timeseries_12",
"type": {
"name": "TimeSeries",
"list": false,
"nonNull": false,
"custom": false,
"type": "timeseries",
"kind": "scalar"
},
"containerType": {
"type": "timeseries",
"list": false
},
"autoIncrement": false
},
{
"externalId": "timeseries_13",
"name": "timeseries_13",
"type": {
"name": "TimeSeries",
"list": false,
"nonNull": false,
"custom": false,
"type": "timeseries",
"kind": "scalar"
},
"containerType": {
"type": "timeseries",
"list": false
},
"autoIncrement": false
},
{
"externalId": "timeseries_14",
"name": "timeseries_14",
"type": {
"name": "TimeSeries",
"list": false,
"nonNull": false,
"custom": false,
"type": "timeseries",
"kind": "scalar"
},
"containerType": {
"type": "timeseries",
"list": false
},
"autoIncrement": false
},
{
"externalId": "timeseries_15",
"name": "timeseries_15",
"type": {
"name": "TimeSeries",
"list": false,
"nonNull": false,
"custom": false,
"type": "timeseries",
"kind": "scalar"
},
"containerType": {
"type": "timeseries",
"list": false
},
"autoIncrement": false
},
{
"externalId": "timeseries_16",
"name": "timeseries_16",
"type": {
"name": "TimeSeries",
"list": false,
"nonNull": false,
"custom": false,
"type": "timeseries",
"kind": "scalar"
},
"containerType": {
"type": "timeseries",
"list": false
},
"autoIncrement": false
},
{
"externalId": "timeseries_17",
"name": "timeseries_17",
"type": {
"name": "TimeSeries",
"list": false,
"nonNull": false,
"custom": false,
"type": "timeseries",
"kind": "scalar"
},
"containerType": {
"type": "timeseries",
"list": false
},
"autoIncrement": false
},
{
"externalId": "timeseries_18",
"name": "timeseries_18",
"type": {
"name": "TimeSeries",
"list": false,
"nonNull": false,
"custom": false,
"type": "timeseries",
"kind": "scalar"
},
"containerType": {
"type": "timeseries",
"list": false
},
"autoIncrement": false
},
{
"externalId": "timeseries_19",
"name": "timeseries_19",
"type": {
"name": "TimeSeries",
"list": false,
"nonNull": false,
"custom": false,
"type": "timeseries",
"kind": "scalar"
},
"containerType": {
"type": "timeseries",
"list": false
},
"autoIncrement": false
},
{
"externalId": "timeseries_20",
"name": "timeseries_20",
"type": {
"name": "TimeSeries",
"list": false,
"nonNull": false,
"custom": false,
"type": "timeseries",
"kind": "scalar"
},
"containerType": {
"type": "timeseries",
"list": false
},
"autoIncrement": false
},
{
"externalId": "timeseries_21",
"name": "timeseries_21",
"type": {
"name": "TimeSeries",
"list": false,
"nonNull": false,
"custom": false,
"type": "timeseries",
"kind": "scalar"
},
"containerType": {
"type": "timeseries",
"list": false
},
"autoIncrement": false
},
{
"externalId": "timeseries_22",
"name": "timeseries_22",
"type": {
"name": "TimeSeries",
"list": false,
"nonNull": false,
"custom": false,
"type": "timeseries",
"kind": "scalar"
},
"containerType": {
"type": "timeseries",
"list": false
},
"autoIncrement": false
},
{
"externalId": "timeseries_23",
"name": "timeseries_23",
"type": {
"name": "TimeSeries",
"list": false,
"nonNull": false,
"custom": false,
"type": "timeseries",
"kind": "scalar"
},
"containerType": {
"type": "timeseries",
"list": false
},
"autoIncrement": false
},
{
"externalId": "timeseries_24",
"name": "timeseries_24",
"type": {
"name": "TimeSeries",
"list": false,
"nonNull": false,
"custom": false,
"type": "timeseries",
"kind": "scalar"
},
"containerType": {
"type": "timeseries",
"list": false
},
"autoIncrement": false
},
{
"externalId": "timeseries_25",
"name": "timeseries_25",
"type": {
"name": "TimeSeries",
"list": false,
"nonNull": false,
"custom": false,
"type": "timeseries",
"kind": "scalar"
},
"containerType": {
"type": "timeseries",
"list": false
},
"autoIncrement": false
},
{
"externalId": "timeseries_26",
"name": "timeseries_26",
"type": {
"name": "TimeSeries",
"list": false,
"nonNull": false,
"custom": false,
"type": "timeseries",
"kind": "scalar"
},
"containerType": {
"type": "timeseries",
"list": false
},
"autoIncrement": false
},
{
"externalId": "timeseries_27",
"name": "timeseries_27",
"type": {
"name": "TimeSeries",
"list": false,
"nonNull": false,
"custom": false,
"type": "timeseries",
"kind": "scalar"
},
"containerType": {
"type": "timeseries",
"list": false
},
"autoIncrement": false
},
{
"externalId": "timeseries_28",
"name": "timeseries_28",
"type": {
"name": "TimeSeries",
"list": false,
"nonNull": false,
"custom": false,
"type": "timeseries",
"kind": "scalar"
},
"containerType": {
"type": "timeseries",
"list": false
},
"autoIncrement": false
},
{
"externalId": "timeseries_29",
"name": "timeseries_29",
"type": {
"name": "TimeSeries",
"list": false,
"nonNull": false,
"custom": false,
"type": "timeseries",
"kind": "scalar"
},
"containerType": {
"type": "timeseries",
"list": false
},
"autoIncrement": false
},
{
"externalId": "timeseries_30",
"name": "timeseries_30",
"type": {
"name": "TimeSeries",
"list": false,
"nonNull": false,
"custom": false,
"type": "timeseries",
"kind": "scalar"
},
"containerType": {
"type": "timeseries",
"list": false
},
"autoIncrement": false
},
{
"externalId": "timeseries_31",
"name": "timeseries_31",
"type": {
"name": "TimeSeries",
"list": false,
"nonNull": false,
"custom": false,
"type": "timeseries",
"kind": "scalar"
},
"containerType": {
"type": "timeseries",
"list": false
},
"autoIncrement": false
},
{
"externalId": "timeseries_32",
"name": "timeseries_32",
"type": {
"name": "TimeSeries",
"list": false,
"nonNull": false,
"custom": false,
"type": "timeseries",
"kind": "scalar"
},
"containerType": {
"type": "timeseries",
"list": false
},
"autoIncrement": false
},
{
"externalId": "timeseries_33",
"name": "timeseries_33",
"type": {
"name": "TimeSeries",
"list": false,
"nonNull": false,
"custom": false,
"type": "timeseries",
"kind": "scalar"
},
"containerType": {
"type": "timeseries",
"list": false
},
"autoIncrement": false
},
{
"externalId": "timeseries_34",
"name": "timeseries_34",
"type": {
"name": "TimeSeries",
"list": false,
"nonNull": false,
"custom": false,
"type": "timeseries",
"kind": "scalar"
},
"containerType": {
"type": "timeseries",
"list": false
},
"autoIncrement": false
},
{
"externalId": "timeseries_35",
"name": "timeseries_35",
"type": {
"name": "TimeSeries",
"list": false,
"nonNull": false,
"custom": false,
"type": "timeseries",
"kind": "scalar"
},
"containerType": {
"type": "timeseries",
"list": false
},
"autoIncrement": false
},
{
"externalId": "timeseries_36",
"name": "timeseries_36",
"type": {
"name": "TimeSeries",
"list": false,
"nonNull": false,
"custom": false,
"type": "timeseries",
"kind": "scalar"
},
"containerType": {
"type": "timeseries",
"list": false
},
"autoIncrement": false
},
{
"externalId": "timeseries_37",
"name": "timeseries_37",
"type": {
"name": "TimeSeries",
"list": false,
"nonNull": false,
"custom": false,
"type": "timeseries",
"kind": "scalar"
},
"containerType": {
"type": "timeseries",
"list": false
},
"autoIncrement": false
},
{
"externalId": "timeseries_38",
"name": "timeseries_38",
"type": {
"name": "TimeSeries",
"list": false,
"nonNull": false,
"custom": false,
"type": "timeseries",
"kind": "scalar"
},
"containerType": {
"type": "timeseries",
"list": false
},
"autoIncrement": false
},
{
"externalId": "timeseries_39",
"name": "timeseries_39",
"type": {
"name": "TimeSeries",
"list": false,
"nonNull": false,
"custom": false,
"type": "timeseries",
"kind": "scalar"
},
"containerType": {
"type": "timeseries",
"list": false
},
"autoIncrement": false
},
{
"externalId": "timeseries_40",
"name": "timeseries_40",
"type": {
"name": "TimeSeries",
"list": false,
"nonNull": false,
"custom": false,
"type": "timeseries",
"kind": "scalar"
},
"containerType": {
"type": "timeseries",
"list": false
},
"autoIncrement": false
},
{
"externalId": "timeseries_41",
"name": "timeseries_41",
"type": {
"name": "TimeSeries",
"list": false,
"nonNull": false,
"custom": false,
"type": "timeseries",
"kind": "scalar"
},
"containerType": {
"type": "timeseries",
"list": false
},
"autoIncrement": false
},
{
"externalId": "timeseries_42",
"name": "timeseries_42",
"type": {
"name": "TimeSeries",
"list": false,
"nonNull": false,
"custom": false,
"type": "timeseries",
"kind": "scalar"
},
"containerType": {
"type": "timeseries",
"list": false
},
"autoIncrement": false
},
{
"externalId": "timeseries_43",
"name": "timeseries_43",
"type": {
"name": "TimeSeries",
"list": false,
"nonNull": false,
"custom": false,
"type": "timeseries",
"kind": "scalar"
},
"containerType": {
"type": "timeseries",
"list": false
},
"autoIncrement": false
},
{
"externalId": "timeseries_44",
"name": "timeseries_44",
"type": {
"name": "TimeSeries",
"list": false,
"nonNull": false,
"custom": false,
"type": "timeseries",
"kind": "scalar"
},
"containerType": {
"type": "timeseries",
"list": false
},
"autoIncrement": false
},
{
"externalId": "timeseries_45",
"name": "timeseries_45",
"type": {
"name": "TimeSeries",
"list": false,
"nonNull": false,
"custom": false,
"type": "timeseries",
"kind": "scalar"
},
"containerType": {
"type": "timeseries",
"list": false
},
"autoIncrement": false
},
{
"externalId": "timeseries_46",
"name": "timeseries_46",
"type": {
"name": "TimeSeries",
"list": false,
"nonNull": false,
"custom": false,
"type": "timeseries",
"kind": "scalar"
},
"containerType": {
"type": "timeseries",
"list": false
},
"autoIncrement": false
},
{
"externalId": "timeseries_47",
"name": "timeseries_47",
"type": {
"name": "TimeSeries",
"list": false,
"nonNull": false,
"custom": false,
"type": "timeseries",
"kind": "scalar"
},
"containerType": {
"type": "timeseries",
"list": false
},
"autoIncrement": false
},
{
"externalId": "timeseries_48",
"name": "timeseries_48",
"type": {
"name": "TimeSeries",
"list": false,
"nonNull": false,
"custom": false,
"type": "timeseries",
"kind": "scalar"
},
"containerType": {
"type": "timeseries",
"list": false
},
"autoIncrement": false
},
{
"externalId": "timeseries_49",
"name": "timeseries_49",
"type": {
"name": "TimeSeries",
"list": false,
"nonNull": false,
"custom": false,
"type": "timeseries",
"kind": "scalar"
},
"containerType": {
"type": "timeseries",
"list": false
},
"autoIncrement": false
},
{
"externalId": "timeseries_50",
"name": "timeseries_50",
"type": {
"name": "TimeSeries",
"list": false,
"nonNull": false,
"custom": false,
"type": "timeseries",
"kind": "scalar"
},
"containerType": {
"type": "timeseries",
"list": false
},
"autoIncrement": false
},
{
"externalId": "propertyName",
"name": "propertyName",
"type": {
"name": "String",
"list": false,
"nonNull": true,
"custom": false,
"type": "text",
"kind": "scalar"
},
"containerType": {
"type": "text",
"list": false,
"collation": "ucs_basic"
},
"autoIncrement": false
},
{
"externalId": "entity",
"name": "entity",
"type": {
"name": "String",
"list": false,
"nonNull": true,
"custom": false,
"type": "text",
"kind": "scalar"
},
"containerType": {
"type": "text",
"list": false,
"collation": "ucs_basic"
},
"autoIncrement": false
},
{
"externalId": "frequency",
"name": "frequency",
"type": {
"name": "String",
"list": false,
"nonNull": true,
"custom": false,
"type": "text",
"kind": "scalar"
},
"containerType": {
"type": "text",
"list": false,
"collation": "ucs_basic"
},
"autoIncrement": false
}
]
}
]

 

Userlevel 2

Hi,

Yes. That is the expectation and why we encourage users to index most commonly queried properties, especially for containers where the instance volume is large (i.e. >100,000 perhaps, although not necessarily a threshold, and can depend on the types of queries to be expected as well).

An additional note on indexes. If you expect to retrieve large volumes of data that match your filter criteria, i.e. through cursors, than also set your indexes to be 'cursorable’. This ensures good performance when you would like cursor through the matched results as well. In your case however that does not seem to be the case so it is fine as is.

Regards

Reply