Skip to main content
Solved

Cognite - data modelling SDK query instances giving timed out error

  • 19 June 2024
  • 5 replies
  • 58 views

Hello team,

We are trying to use the data modeling query instances SDK method.

client.data_modeling.instances.query(query)

 

our query is:

query {
    "with": {
        "Min_15_50ts": {
            "nodes": {
                "filter": {
                    "and":  
                        {
                            "hasData":  
                                {
                                    "type": "view",
                                    "space": "QA_Test_Space",
                                    "externalId": "Min_15_50ts",
                                    "version": "3a62f20104e445"
                                }
                            ]
                        },
                        {
                            "in": {
                                "property":  
                                    "QA_Test_Space",
                                    "Min_15_50ts/3a62f20104e445",
                                    "entity"
                                ],
                                "values":  
                                    "QA_Perf_WELL_1",
                                    "QA_Perf_WELL_2",
                                    "QA_Perf_WELL_3",
                                    "QA_Perf_WELL_4",
                                    "QA_Perf_WELL_5",
                                    "QA_Perf_WELL_6",
                                    "QA_Perf_WELL_7",
                                    "QA_Perf_WELL_8",
                                    "QA_Perf_WELL_9",
                                    "QA_Perf_WELL_10",
                                    "QA_Perf_WELL_11",
                                    "QA_Perf_WELL_12",
                                    "QA_Perf_WELL_13",
                                    "QA_Perf_WELL_14",
                                    "QA_Perf_WELL_15",
                                    "QA_Perf_WELL_16"
                                ]
                            }
                        }
                    ]
                },
                "chain_to": "destination",
                "direction": "outwards"
            },
            "limit": 1000
        }
    },
    "select": {
        "Min_15_50ts": {
            "sources":  
                {
                    "source": {
                        "space": "QA_Test_Space",
                        "external_id": "Min_15_50ts",
                        "version": "3a62f20104e445",
                        "type": "view"
                    },
                    "properties":  
                        "timeseries_1",
                        "entity"
                    ]
                }
            ]
        }
    },
    "cursors": {
        "Min_15_50ts": null
    }
}

for this we are getting the following error:

cognite.client.exceptions.CogniteAPIError: Graph query timed out. Reduce load or contention, or optimise your query. | code: 408 | X-Request-ID: 8d863ca0-3605-93ac-a893-a7c80b7fae8b

 

We are using 7.38.2 version of the SDK.

Kindly let us know how we can make this run.

 

 

Hi @Gargi Bhosale, firstly, you don’t require either the chain_to or direction property here. Chain_to is only needed when you are chaining to another part of your query, and direction for nodes is only required when you specify a direct relation in 'through’ property.

Secondly, have you set up an index on the entity property? If not, could you try adding it, as this should significantly improve query performance here. Could you also mention the total instances you might have in the 'Min_15_50ts’ container/view?

Hello Uzair,

 

Thank you for your prompt response .

We have a few observations:

 

  1. for the chain_to or direction property you were mentioning, even if we change that property, this issue/failure is not constant, it is intermittent. So, we don't think changing that would make a difference, as if this was an issue the query would never run successfully.
  2.  We have added indexing to index property, it is running successfully, but we tried without indexing too it is also running. So, we think this is an intermittent issue which doesn't depend on indexing
  3. Min_15_50ts has 5,25,000 instances.

 

Please let us know how we can avoid this intermittent issue for ensuring that our code runs successfully in every run.

Your help is much appreciated.

Hi,

So, to clarify, do you still run into the same issue when you have the entity property indexed?

I understand that the query intermittently fails on the non-indexed scenario with occasional success. This is to be expected with this sort of filtering and data volume, and hence why indexing the properties that you would be querying on is important. If the same issue happens after the property is indexed than this can be a performance issue in the service worth investigating.

Do you mind sharing the container configuration, that shows indexes applied?

Regards

Hi,

 

We are not running into the same issue after indexing the entity property.

 

So, just to clarify and draw a conclusion, we will have to index those properties on which we would be applying filters.

 

Sharing the container configuration below:

d
  {
    "name": "Min_15_50ts",
    "space": "QA_Test_Space",
    "externalId": "Min_15_50ts",
    "usedFor": "node",
    "createdTime": 1690541696938,
    "lastUpdatedTime": 1690541696938,
    "isGlobal": false,
    "constraints": {},
    "indexes": {
      "propertyNameIndex": {
        "properties": "
          "propertyName"
        ],
        "indexType": "btree",
        "cursorable": false,
        "bySpace": false
      },
      "entityIndex": {
        "properties": "
          "entity"
        ],
        "indexType": "btree",
        "cursorable": false,
        "bySpace": false
      }
    },
    "kind": "container",
    "fields": "
      {
        "externalId": "timeseries_1",
        "name": "timeseries_1",
        "type": {
          "name": "TimeSeries",
          "list": false,
          "nonNull": false,
          "custom": false,
          "type": "timeseries",
          "kind": "scalar"
        },
        "containerType": {
          "type": "timeseries",
          "list": false
        },
        "autoIncrement": false
      },
      {
        "externalId": "timeseries_2",
        "name": "timeseries_2",
        "type": {
          "name": "TimeSeries",
          "list": false,
          "nonNull": false,
          "custom": false,
          "type": "timeseries",
          "kind": "scalar"
        },
        "containerType": {
          "type": "timeseries",
          "list": false
        },
        "autoIncrement": false
      },
      {
        "externalId": "timeseries_3",
        "name": "timeseries_3",
        "type": {
          "name": "TimeSeries",
          "list": false,
          "nonNull": false,
          "custom": false,
          "type": "timeseries",
          "kind": "scalar"
        },
        "containerType": {
          "type": "timeseries",
          "list": false
        },
        "autoIncrement": false
      },
      {
        "externalId": "timeseries_4",
        "name": "timeseries_4",
        "type": {
          "name": "TimeSeries",
          "list": false,
          "nonNull": false,
          "custom": false,
          "type": "timeseries",
          "kind": "scalar"
        },
        "containerType": {
          "type": "timeseries",
          "list": false
        },
        "autoIncrement": false
      },
      {
        "externalId": "timeseries_5",
        "name": "timeseries_5",
        "type": {
          "name": "TimeSeries",
          "list": false,
          "nonNull": false,
          "custom": false,
          "type": "timeseries",
          "kind": "scalar"
        },
        "containerType": {
          "type": "timeseries",
          "list": false
        },
        "autoIncrement": false
      },
      {
        "externalId": "timeseries_6",
        "name": "timeseries_6",
        "type": {
          "name": "TimeSeries",
          "list": false,
          "nonNull": false,
          "custom": false,
          "type": "timeseries",
          "kind": "scalar"
        },
        "containerType": {
          "type": "timeseries",
          "list": false
        },
        "autoIncrement": false
      },
      {
        "externalId": "timeseries_7",
        "name": "timeseries_7",
        "type": {
          "name": "TimeSeries",
          "list": false,
          "nonNull": false,
          "custom": false,
          "type": "timeseries",
          "kind": "scalar"
        },
        "containerType": {
          "type": "timeseries",
          "list": false
        },
        "autoIncrement": false
      },
      {
        "externalId": "timeseries_8",
        "name": "timeseries_8",
        "type": {
          "name": "TimeSeries",
          "list": false,
          "nonNull": false,
          "custom": false,
          "type": "timeseries",
          "kind": "scalar"
        },
        "containerType": {
          "type": "timeseries",
          "list": false
        },
        "autoIncrement": false
      },
      {
        "externalId": "timeseries_9",
        "name": "timeseries_9",
        "type": {
          "name": "TimeSeries",
          "list": false,
          "nonNull": false,
          "custom": false,
          "type": "timeseries",
          "kind": "scalar"
        },
        "containerType": {
          "type": "timeseries",
          "list": false
        },
        "autoIncrement": false
      },
      {
        "externalId": "timeseries_10",
        "name": "timeseries_10",
        "type": {
          "name": "TimeSeries",
          "list": false,
          "nonNull": false,
          "custom": false,
          "type": "timeseries",
          "kind": "scalar"
        },
        "containerType": {
          "type": "timeseries",
          "list": false
        },
        "autoIncrement": false
      },
      {
        "externalId": "timeseries_11",
        "name": "timeseries_11",
        "type": {
          "name": "TimeSeries",
          "list": false,
          "nonNull": false,
          "custom": false,
          "type": "timeseries",
          "kind": "scalar"
        },
        "containerType": {
          "type": "timeseries",
          "list": false
        },
        "autoIncrement": false
      },
      {
        "externalId": "timeseries_12",
        "name": "timeseries_12",
        "type": {
          "name": "TimeSeries",
          "list": false,
          "nonNull": false,
          "custom": false,
          "type": "timeseries",
          "kind": "scalar"
        },
        "containerType": {
          "type": "timeseries",
          "list": false
        },
        "autoIncrement": false
      },
      {
        "externalId": "timeseries_13",
        "name": "timeseries_13",
        "type": {
          "name": "TimeSeries",
          "list": false,
          "nonNull": false,
          "custom": false,
          "type": "timeseries",
          "kind": "scalar"
        },
        "containerType": {
          "type": "timeseries",
          "list": false
        },
        "autoIncrement": false
      },
      {
        "externalId": "timeseries_14",
        "name": "timeseries_14",
        "type": {
          "name": "TimeSeries",
          "list": false,
          "nonNull": false,
          "custom": false,
          "type": "timeseries",
          "kind": "scalar"
        },
        "containerType": {
          "type": "timeseries",
          "list": false
        },
        "autoIncrement": false
      },
      {
        "externalId": "timeseries_15",
        "name": "timeseries_15",
        "type": {
          "name": "TimeSeries",
          "list": false,
          "nonNull": false,
          "custom": false,
          "type": "timeseries",
          "kind": "scalar"
        },
        "containerType": {
          "type": "timeseries",
          "list": false
        },
        "autoIncrement": false
      },
      {
        "externalId": "timeseries_16",
        "name": "timeseries_16",
        "type": {
          "name": "TimeSeries",
          "list": false,
          "nonNull": false,
          "custom": false,
          "type": "timeseries",
          "kind": "scalar"
        },
        "containerType": {
          "type": "timeseries",
          "list": false
        },
        "autoIncrement": false
      },
      {
        "externalId": "timeseries_17",
        "name": "timeseries_17",
        "type": {
          "name": "TimeSeries",
          "list": false,
          "nonNull": false,
          "custom": false,
          "type": "timeseries",
          "kind": "scalar"
        },
        "containerType": {
          "type": "timeseries",
          "list": false
        },
        "autoIncrement": false
      },
      {
        "externalId": "timeseries_18",
        "name": "timeseries_18",
        "type": {
          "name": "TimeSeries",
          "list": false,
          "nonNull": false,
          "custom": false,
          "type": "timeseries",
          "kind": "scalar"
        },
        "containerType": {
          "type": "timeseries",
          "list": false
        },
        "autoIncrement": false
      },
      {
        "externalId": "timeseries_19",
        "name": "timeseries_19",
        "type": {
          "name": "TimeSeries",
          "list": false,
          "nonNull": false,
          "custom": false,
          "type": "timeseries",
          "kind": "scalar"
        },
        "containerType": {
          "type": "timeseries",
          "list": false
        },
        "autoIncrement": false
      },
      {
        "externalId": "timeseries_20",
        "name": "timeseries_20",
        "type": {
          "name": "TimeSeries",
          "list": false,
          "nonNull": false,
          "custom": false,
          "type": "timeseries",
          "kind": "scalar"
        },
        "containerType": {
          "type": "timeseries",
          "list": false
        },
        "autoIncrement": false
      },
      {
        "externalId": "timeseries_21",
        "name": "timeseries_21",
        "type": {
          "name": "TimeSeries",
          "list": false,
          "nonNull": false,
          "custom": false,
          "type": "timeseries",
          "kind": "scalar"
        },
        "containerType": {
          "type": "timeseries",
          "list": false
        },
        "autoIncrement": false
      },
      {
        "externalId": "timeseries_22",
        "name": "timeseries_22",
        "type": {
          "name": "TimeSeries",
          "list": false,
          "nonNull": false,
          "custom": false,
          "type": "timeseries",
          "kind": "scalar"
        },
        "containerType": {
          "type": "timeseries",
          "list": false
        },
        "autoIncrement": false
      },
      {
        "externalId": "timeseries_23",
        "name": "timeseries_23",
        "type": {
          "name": "TimeSeries",
          "list": false,
          "nonNull": false,
          "custom": false,
          "type": "timeseries",
          "kind": "scalar"
        },
        "containerType": {
          "type": "timeseries",
          "list": false
        },
        "autoIncrement": false
      },
      {
        "externalId": "timeseries_24",
        "name": "timeseries_24",
        "type": {
          "name": "TimeSeries",
          "list": false,
          "nonNull": false,
          "custom": false,
          "type": "timeseries",
          "kind": "scalar"
        },
        "containerType": {
          "type": "timeseries",
          "list": false
        },
        "autoIncrement": false
      },
      {
        "externalId": "timeseries_25",
        "name": "timeseries_25",
        "type": {
          "name": "TimeSeries",
          "list": false,
          "nonNull": false,
          "custom": false,
          "type": "timeseries",
          "kind": "scalar"
        },
        "containerType": {
          "type": "timeseries",
          "list": false
        },
        "autoIncrement": false
      },
      {
        "externalId": "timeseries_26",
        "name": "timeseries_26",
        "type": {
          "name": "TimeSeries",
          "list": false,
          "nonNull": false,
          "custom": false,
          "type": "timeseries",
          "kind": "scalar"
        },
        "containerType": {
          "type": "timeseries",
          "list": false
        },
        "autoIncrement": false
      },
      {
        "externalId": "timeseries_27",
        "name": "timeseries_27",
        "type": {
          "name": "TimeSeries",
          "list": false,
          "nonNull": false,
          "custom": false,
          "type": "timeseries",
          "kind": "scalar"
        },
        "containerType": {
          "type": "timeseries",
          "list": false
        },
        "autoIncrement": false
      },
      {
        "externalId": "timeseries_28",
        "name": "timeseries_28",
        "type": {
          "name": "TimeSeries",
          "list": false,
          "nonNull": false,
          "custom": false,
          "type": "timeseries",
          "kind": "scalar"
        },
        "containerType": {
          "type": "timeseries",
          "list": false
        },
        "autoIncrement": false
      },
      {
        "externalId": "timeseries_29",
        "name": "timeseries_29",
        "type": {
          "name": "TimeSeries",
          "list": false,
          "nonNull": false,
          "custom": false,
          "type": "timeseries",
          "kind": "scalar"
        },
        "containerType": {
          "type": "timeseries",
          "list": false
        },
        "autoIncrement": false
      },
      {
        "externalId": "timeseries_30",
        "name": "timeseries_30",
        "type": {
          "name": "TimeSeries",
          "list": false,
          "nonNull": false,
          "custom": false,
          "type": "timeseries",
          "kind": "scalar"
        },
        "containerType": {
          "type": "timeseries",
          "list": false
        },
        "autoIncrement": false
      },
      {
        "externalId": "timeseries_31",
        "name": "timeseries_31",
        "type": {
          "name": "TimeSeries",
          "list": false,
          "nonNull": false,
          "custom": false,
          "type": "timeseries",
          "kind": "scalar"
        },
        "containerType": {
          "type": "timeseries",
          "list": false
        },
        "autoIncrement": false
      },
      {
        "externalId": "timeseries_32",
        "name": "timeseries_32",
        "type": {
          "name": "TimeSeries",
          "list": false,
          "nonNull": false,
          "custom": false,
          "type": "timeseries",
          "kind": "scalar"
        },
        "containerType": {
          "type": "timeseries",
          "list": false
        },
        "autoIncrement": false
      },
      {
        "externalId": "timeseries_33",
        "name": "timeseries_33",
        "type": {
          "name": "TimeSeries",
          "list": false,
          "nonNull": false,
          "custom": false,
          "type": "timeseries",
          "kind": "scalar"
        },
        "containerType": {
          "type": "timeseries",
          "list": false
        },
        "autoIncrement": false
      },
      {
        "externalId": "timeseries_34",
        "name": "timeseries_34",
        "type": {
          "name": "TimeSeries",
          "list": false,
          "nonNull": false,
          "custom": false,
          "type": "timeseries",
          "kind": "scalar"
        },
        "containerType": {
          "type": "timeseries",
          "list": false
        },
        "autoIncrement": false
      },
      {
        "externalId": "timeseries_35",
        "name": "timeseries_35",
        "type": {
          "name": "TimeSeries",
          "list": false,
          "nonNull": false,
          "custom": false,
          "type": "timeseries",
          "kind": "scalar"
        },
        "containerType": {
          "type": "timeseries",
          "list": false
        },
        "autoIncrement": false
      },
      {
        "externalId": "timeseries_36",
        "name": "timeseries_36",
        "type": {
          "name": "TimeSeries",
          "list": false,
          "nonNull": false,
          "custom": false,
          "type": "timeseries",
          "kind": "scalar"
        },
        "containerType": {
          "type": "timeseries",
          "list": false
        },
        "autoIncrement": false
      },
      {
        "externalId": "timeseries_37",
        "name": "timeseries_37",
        "type": {
          "name": "TimeSeries",
          "list": false,
          "nonNull": false,
          "custom": false,
          "type": "timeseries",
          "kind": "scalar"
        },
        "containerType": {
          "type": "timeseries",
          "list": false
        },
        "autoIncrement": false
      },
      {
        "externalId": "timeseries_38",
        "name": "timeseries_38",
        "type": {
          "name": "TimeSeries",
          "list": false,
          "nonNull": false,
          "custom": false,
          "type": "timeseries",
          "kind": "scalar"
        },
        "containerType": {
          "type": "timeseries",
          "list": false
        },
        "autoIncrement": false
      },
      {
        "externalId": "timeseries_39",
        "name": "timeseries_39",
        "type": {
          "name": "TimeSeries",
          "list": false,
          "nonNull": false,
          "custom": false,
          "type": "timeseries",
          "kind": "scalar"
        },
        "containerType": {
          "type": "timeseries",
          "list": false
        },
        "autoIncrement": false
      },
      {
        "externalId": "timeseries_40",
        "name": "timeseries_40",
        "type": {
          "name": "TimeSeries",
          "list": false,
          "nonNull": false,
          "custom": false,
          "type": "timeseries",
          "kind": "scalar"
        },
        "containerType": {
          "type": "timeseries",
          "list": false
        },
        "autoIncrement": false
      },
      {
        "externalId": "timeseries_41",
        "name": "timeseries_41",
        "type": {
          "name": "TimeSeries",
          "list": false,
          "nonNull": false,
          "custom": false,
          "type": "timeseries",
          "kind": "scalar"
        },
        "containerType": {
          "type": "timeseries",
          "list": false
        },
        "autoIncrement": false
      },
      {
        "externalId": "timeseries_42",
        "name": "timeseries_42",
        "type": {
          "name": "TimeSeries",
          "list": false,
          "nonNull": false,
          "custom": false,
          "type": "timeseries",
          "kind": "scalar"
        },
        "containerType": {
          "type": "timeseries",
          "list": false
        },
        "autoIncrement": false
      },
      {
        "externalId": "timeseries_43",
        "name": "timeseries_43",
        "type": {
          "name": "TimeSeries",
          "list": false,
          "nonNull": false,
          "custom": false,
          "type": "timeseries",
          "kind": "scalar"
        },
        "containerType": {
          "type": "timeseries",
          "list": false
        },
        "autoIncrement": false
      },
      {
        "externalId": "timeseries_44",
        "name": "timeseries_44",
        "type": {
          "name": "TimeSeries",
          "list": false,
          "nonNull": false,
          "custom": false,
          "type": "timeseries",
          "kind": "scalar"
        },
        "containerType": {
          "type": "timeseries",
          "list": false
        },
        "autoIncrement": false
      },
      {
        "externalId": "timeseries_45",
        "name": "timeseries_45",
        "type": {
          "name": "TimeSeries",
          "list": false,
          "nonNull": false,
          "custom": false,
          "type": "timeseries",
          "kind": "scalar"
        },
        "containerType": {
          "type": "timeseries",
          "list": false
        },
        "autoIncrement": false
      },
      {
        "externalId": "timeseries_46",
        "name": "timeseries_46",
        "type": {
          "name": "TimeSeries",
          "list": false,
          "nonNull": false,
          "custom": false,
          "type": "timeseries",
          "kind": "scalar"
        },
        "containerType": {
          "type": "timeseries",
          "list": false
        },
        "autoIncrement": false
      },
      {
        "externalId": "timeseries_47",
        "name": "timeseries_47",
        "type": {
          "name": "TimeSeries",
          "list": false,
          "nonNull": false,
          "custom": false,
          "type": "timeseries",
          "kind": "scalar"
        },
        "containerType": {
          "type": "timeseries",
          "list": false
        },
        "autoIncrement": false
      },
      {
        "externalId": "timeseries_48",
        "name": "timeseries_48",
        "type": {
          "name": "TimeSeries",
          "list": false,
          "nonNull": false,
          "custom": false,
          "type": "timeseries",
          "kind": "scalar"
        },
        "containerType": {
          "type": "timeseries",
          "list": false
        },
        "autoIncrement": false
      },
      {
        "externalId": "timeseries_49",
        "name": "timeseries_49",
        "type": {
          "name": "TimeSeries",
          "list": false,
          "nonNull": false,
          "custom": false,
          "type": "timeseries",
          "kind": "scalar"
        },
        "containerType": {
          "type": "timeseries",
          "list": false
        },
        "autoIncrement": false
      },
      {
        "externalId": "timeseries_50",
        "name": "timeseries_50",
        "type": {
          "name": "TimeSeries",
          "list": false,
          "nonNull": false,
          "custom": false,
          "type": "timeseries",
          "kind": "scalar"
        },
        "containerType": {
          "type": "timeseries",
          "list": false
        },
        "autoIncrement": false
      },
      {
        "externalId": "propertyName",
        "name": "propertyName",
        "type": {
          "name": "String",
          "list": false,
          "nonNull": true,
          "custom": false,
          "type": "text",
          "kind": "scalar"
        },
        "containerType": {
          "type": "text",
          "list": false,
          "collation": "ucs_basic"
        },
        "autoIncrement": false
      },
      {
        "externalId": "entity",
        "name": "entity",
        "type": {
          "name": "String",
          "list": false,
          "nonNull": true,
          "custom": false,
          "type": "text",
          "kind": "scalar"
        },
        "containerType": {
          "type": "text",
          "list": false,
          "collation": "ucs_basic"
        },
        "autoIncrement": false
      },
      {
        "externalId": "frequency",
        "name": "frequency",
        "type": {
          "name": "String",
          "list": false,
          "nonNull": true,
          "custom": false,
          "type": "text",
          "kind": "scalar"
        },
        "containerType": {
          "type": "text",
          "list": false,
          "collation": "ucs_basic"
        },
        "autoIncrement": false
      }
    ]
  }
]

 

Hi,

Yes. That is the expectation and why we encourage users to index most commonly queried properties, especially for containers where the instance volume is large (i.e. >100,000 perhaps, although not necessarily a threshold, and can depend on the types of queries to be expected as well).

An additional note on indexes. If you expect to retrieve large volumes of data that match your filter criteria, i.e. through cursors, than also set your indexes to be 'cursorable’. This ensures good performance when you would like cursor through the matched results as well. In your case however that does not seem to be the case so it is fine as is.

Regards

Reply


Terms & ConditionsCookie Settings