Incremental Load in Hosted Cognite Rest extractor

For incremental load you only have access to the context object you can see on this page. You can do something like this with an `if` expression:

if context.last_run is not null {
    // Something based on last_run
} else {
    1234 // something constant
}

Yes , based on context.last run whether it is null or not I want to update startIndex and endIndex ( Query params) for my outgoing request.I have set to use query params for incremental load in REST extractor. something like this: -

Key Name: Query

(V)Value:
if context.last_run is not null {
{
    "filter": {
        ...context.request.body.filter,
      "query": { 
      
        "startIndex":"context.last_run",
      "endIndex":"2025-04-04T00:00:00Z"
        
      }
    }
  }
}
else {
{
    "filter": {
        ...context.request.body.filter,
      "query": { 
        "startIndex":"2025-02-01T00:00:00Z",
        "endIndex":"2025-03-04T00:00:00Z"
      }
    }
  }
}
But getting error like this : Could not create HTTP request: object. Got null, expected object at 50..220

Q1. What is the exact meaning of key name here?
Q2. How to modify this Value(V)so that it gives startIndex and endIndex a constant value if context.last_run is null or else it update query based on last run according to above code.

query in incremental load refers to the request query, like `my/path?foo=123&bar=345`, so `key name: foo` would mean set `foo` to the result of the query.

It looks to me like what you want is body incremental load, which we do support, if the request method is set to POST.

The expression looks fine for body, but the error you’re getting means that you’re trying to combine filter from the original request body with your updated query. What have you set the initial request body to? (under Request body (Optional) in the create job form)

This is a GET request query where I am trying to fetch data from aveva and push it into CDF after transformation. In this GET Api I need to pass start and end date which should be set to constant value for historical load at first request but afterwards it should update according to the code provided above in incremental load body based on last run (if possible).

If you are saying key name refers query param name which is  startIndex and endIndex here so how will we update both keys at time because we have only text box for key Name so accordingly startDate or end Date could be updated at a time other will need to be remain fixed.
Herea are some screenshots of what I am trying to achieve: -

I think I get it, though I can’t see your screenshots for whatever reason. What you want is probably:

 

if context.last_run is not null {
    format_timestamp(context.last_run, "%Y-%m-%dT%H:%M:%SZ")
} else {
    "2025-02-01T00:00:00Z"
}

with keyName set to “startIndex”.

You cannot set both startIndex and endIndex in incrementalLoad, so you will need to set “endIndex” to a constant value in the job itself, under Query parameters (Optional) in the form.

Yes, Correct I want to achieve this only but the problem here is my startIndex now is based on my last run which is correct, but my end index will remain constant due to which same data will be fetched in every next request with less datapoints as now interval for which we are fetching will be reduced as  start index will shift by some amount to lasr_run timestamp and end index will remain same.
Scenario which I want to achieve for e.g. in second GET request is startIndex = Context.last_run and end index = context.last_run + 100000 MS which will lead to incremental fetching of data in every request

I understand what you’re asking for, but there is currently no way to accomplish this. Does it not work if you specify an `endIndex` very far in the future, or not specify one at all?