We are now planning to integrate a large amount of historical data. The context about historical data and our data organization is in this topic:
We create an interface WellState, which contains 2 properties: WellId and Archived, in order to propagate the Archived flag to every object in our data models.
After integrated large historical data in our development env, we found that DataFetchingException always occurred in Daily(550K lines) and Operation(2M lines) in using GraphQL. And if using cdf_nodes to load Daily and Operation, the transformation will be blocked util that your session is closed.
Info: Daily has many relationships and Operation has only a relationship with Daily. But We have also other 3 objects like Operation(big dataset and a relationship with Daily), they can work well.
GraphQL:
query MyQuery {
listOperation {
edges {
node {
externalId
space
}
}
}
listDaily {
edges {
node {
externalId
space
}
}
}
}the result is always like that:
{
"errors": [
{
"message": "Client request(POST https://westeurope-1.cognitedata.com/api/v1/projects/totalenergies-sandbox/models/instances/query) invalid: 408 Request Timeout. Text: \"{\n \"error\": {\n \"code\": 408,\n \"message\": \"Graph query timed out. Reduce load or contention, or optimise your query.\",\n \"notices\": [\n {\n \"code\": \"significantHasDataFiltering\",\n \"category\": \"filtering\",\n \"level\": \"warning\",\n \"hint\": \"The provided `hasData` filters expand to several joins, which can be problematic depending on data distribution. Consider `requires` constraints, which can enable query time join pruning.\",\n \"grade\": \"C\",\n \"resultExpression\": \"0\",\n \"containers\": [\n {\n \"type\": \"container\",\n \"space\": \"cdf_cdm\",\n \"externalId\": \"CogniteDescribable\"\n },\n {\n \"type\": \"container\",\n \"space\": \"sp_dm_dap_knowledge_graph\",\n \"externalId\": \"Daily\"\n },\n {\n \"type\": \"container\",\n \"space\": \"sp_dm_dap_knowledge_graph\",\n \"externalId\": \"WellState\"\n }\n ]\n }\n ]\n }\n}\"",
"locations": [
{
"line": 10,
"column": 3
}
],
"path": [
"listDaily"
],
"extensions": {
"classification": "DataFetchingException"
}
},
{
"message": "Client request(POST https://westeurope-1.cognitedata.com/api/v1/projects/totalenergies-sandbox/models/instances/query) invalid: 408 Request Timeout. Text: \"{\n \"error\": {\n \"code\": 408,\n \"message\": \"Graph query timed out. Reduce load or contention, or optimise your query.\",\n \"notices\": [\n {\n \"code\": \"significantHasDataFiltering\",\n \"category\": \"filtering\",\n \"level\": \"warning\",\n \"hint\": \"The provided `hasData` filters expand to several joins, which can be problematic depending on data distribution. Consider `requires` constraints, which can enable query time join pruning.\",\n \"grade\": \"C\",\n \"resultExpression\": \"0\",\n \"containers\": [\n {\n \"type\": \"container\",\n \"space\": \"cdf_cdm\",\n \"externalId\": \"CogniteDescribable\"\n },\n {\n \"type\": \"container\",\n \"space\": \"sp_dm_dap_knowledge_graph\",\n \"externalId\": \"Operation\"\n },\n {\n \"type\": \"container\",\n \"space\": \"sp_dm_dap_knowledge_graph\",\n \"externalId\": \"WellState\"\n }\n ]\n }\n ]\n }\n}\"",
"locations": [
{
"line": 2,
"column": 3
}
],
"path": [
"listOperation"
],
"extensions": {
"classification": "DataFetchingException"
}
}
],
"data": {
"listOperation": null,
"listDaily": null
},
"extensions": {
"dmsQuery": {
"listOperation": {
"with": {
"0": {
"limit": 50,
"nodes": {
"filter": {
"and": [
{
"matchAll": {}
},
{
"hasData": [
{
"type": "view",
"space": "sp_dm_dap_knowledge_graph",
"externalId": "Operation",
"version": "0.72"
}
]
}
]
}
}
}
},
"select": {
"0": {
"sources": [
{
"source": {
"type": "view",
"space": "sp_dm_dap_knowledge_graph",
"externalId": "Operation",
"version": "0.72"
},
"properties": [
"Phase",
"Daily"
]
}
]
}
},
"includeTyping": true,
"forceCursorsDespitePerformanceHazard": true,
"debug": {}
},
"listDaily": {
"with": {
"0": {
"limit": 50,
"nodes": {
"filter": {
"and": [
{
"matchAll": {}
},
{
"hasData": [
{
"type": "view",
"space": "sp_dm_dap_knowledge_graph",
"externalId": "Daily",
"version": "0.72"
}
]
}
]
}
}
}
},
"select": {
"0": {
"sources": [
{
"source": {
"type": "view",
"space": "sp_dm_dap_knowledge_graph",
"externalId": "Daily",
"version": "0.72"
},
"properties": [
"Phase",
"Activity"
]
}
]
}
},
"includeTyping": true,
"forceCursorsDespitePerformanceHazard": true,
"debug": {}
}
},
"dmsDebug": {}
}
}
Check the
documentation
Ask the
Community
Take a look
at
Academy
Cognite
Status
Page
Contact
Cognite Support