Limitations in Incremental refresh in power BI using CDF connector

Follow-up question:

Could I achieve my use case goal (only load the newest Events data into the PBI model) by configuring a Push Dataset and a subscription to our Events? Similar to this guide, but for Events. This would also allow us get a more real-time refresh (say, every 10 minutes) instead of just at-most hourly.

Hi ​@Anders Brakestad!

Let’s me try to respond to your queries in parts:

1. What happens if the StartTime attribute of an Event is a date/time/zone when loaded into the model? The attribute needs to be a date/time type, which means I need to convert the type first. How will the incremental refresh work if the model needs to load the data in order to convert the type, and then apply the RangeStart/RangeEnd filters?
   The timestamps in CDF are indeed timezone aware, so a datetimezone type is the best match for them. That being said the RangeStart, RangeEnd parameters that Microsoft requires to enable incremental refresh are datetime objects. For this reason you cannot use them directly to apply the filtering, you’ll have to define the parameters as specified by Microsoft, but then you need to create derived variables that are of type datetimezone and use theses derived variables to filter the data. This is relevant for OData where the filtering is performed on the column itself with a well defined type, but for REST resources, you’ll have to convert the RangeStart and RangeEnd to either a UTC timestamp in ms since epoch (accepted on most REST endpoint time range filters) or a UTC ISO timestamp (accepted by GraphQL filters). I’ll try to include one example for each type of request, OData, REST and GraphQL in an upcoming documentation page about incremental refresh.
    
2. Can I use Incremental refersh with the new REST API Connector? When I load events with the new connector I get the start times as number of milliseconds since epoch. So the type conversion from above also applies here.
   Short answer is yes. I briefly responded above on the type conversion, as it applies here as well.
    
3. Could I achieve my use case goal (only load the newest Events data into the PBI model) by configuring a Push Dataset and a subscription to our Events? Similar to this guide, but for Events. This would also allow us get a more real-time refresh (say, every 10 minutes) instead of just at-most hourly.
   This is a viable route, but be mindful of the data/retention limits that Power BI imposes on push/streaming datasets. This is the simplest way to get a close to “real-time” refresh in Power BI with CDF data.