Topics started by Ben Brandt

26 Topics

Ben Brandt
Seasoned
Ben BrandtSeasoned

DirectQuery: Add Support for Ad-hoc Querying with Azure Data StudioGathering Interest

I just gained access to the DirectQuery early access so was trying to switch over from the PostgreSQL Gateway to the official DirectQuery solution.  When experimenting with the types of queries I can include in my SaaS application code, I use Azure Data Studio to query Postgres.  Through ADS, I always get an error with DirectQuery:cursor "b07f4a07-c9db-4d5d-9f7c-c9adb01abaf7" does not exist Even for simple queries like this:select "timestamp"from public.datapoint_aggregatesWHERE timeseries_external_id = '419b9b45-8e09-48a9-9912-08da820807fc'AND "timestamp" > '2022-08-23T00:00:00'AND "timestamp" < '2050-08-23T00:00:00'This is likely caused by something the client is doing because using this same login, I am able to run the example PowerBI queries.

11
Anita Hæhre
Seasoned Practitioner
1 year ago
Ben Brandt
Seasoned
Ben BrandtSeasoned
asked in Developer and User Community

Unable to use Transformations to query for a single day

When I run the query below, I am getting an error.  If I increase the end time by 1 second I can get a result, but it gives me 2 days and I only want 1.  All my data is consistently daily at midnight, UTC.  Any ideas how to get past this?SELECT  dp.id,  dp.externalId AS `key`,  dp.timestamp,  dp.valueFROM  `_cdf`.`datapoints` dpWHERE  dp.externalId IN (    'f18ae31cbfe544cb7bca08da91e5245a-SuFMiCo0f7c8eca4ce444208853aedd00ded4fb',    '730619216b764ba77bc708da91e5245a-SuFMiCo6bb90df0efbc461d863faedd00df009b'  )  AND dp.timestamp >= TO_TIMESTAMP('2022-09-01T00:00:00Z')  AND dp.timestamp < TO_TIMESTAMP('2022-09-02T00:00:00Z') Gives me a result, but not what I want:  cc @Torgrim Aas  @Sunil Krishnamoorthy 

9
Sunil Krishnamoorthy
Seasoned Practitioner
1 year ago
Ben Brandt
Seasoned
Ben BrandtSeasoned

Cognite Data Fusion: Support Dates Prior to 1970 in API and Python SDKImplemented

We have a time series that begins on the install date of an asset, which for some assets out in the field have a install date prior to 1970.  When we try to insert these time series data points into CDF via the Python SDK, we are getting this error: cognite.client.exceptions.CogniteAPIError: Timestamp is too low, must be at least 31536000000 Is there already a workaround for storing data points with a date prior to 1970? Is this supported by the HTTP API, but not the Python SDK? Thank you,Ben Brandt

9
Anita Hæhre
Seasoned Practitioner
1 year ago
Ben Brandt
Seasoned
Ben BrandtSeasoned

Cognite Data Fusion: Full Support for Auth0 as Identity ProviderPlanned for development

Really great progress has been made in support for OAuth2 and Open ID Connect, as described here.  Support for Azure AD was the right first move into this area.  Next, I recommend full support be added to CDF (backend and frontend) for using Auth0 as the identity provider.  I discussed with a member of the Cognite identity team in Q4 of 2021, but was told the support was ready with the backend since Auth0 meets the Minimum IdP requirements, but the frontend would not be accessible with Auth0.  Only API access would work.I’ve lost track of this area a bit, so decided a Product Feedback topic would be a great way to keep everyone updated on the status.  Thanks! 

7
Anita Hæhre
Seasoned Practitioner
7 months ago
Ben Brandt
Seasoned
Ben BrandtSeasoned

Cognite Data Fusion: Allow PostgreSQL Gateway Datapoints Query By Timeseries IDPlanned for development

Please allow queries to the datapoints “table” in the PostgreSQL gateway by the Timeseries ID.Not sure if id is intended here to be the id of the datapoint or the timeseries, but something like this: 

7
V
Practitioner
1 year ago
Ben Brandt
Seasoned
Ben BrandtSeasoned

Cognite Data Fusion: Allow Update of Name and External Id of Transformation From New UXGathering Interest

I looked around in all the usual places and couldn’t figure out how to rename/re-externalid a transformation with the new UX.  I went to the old and did it no problem and then switched back to the new UX.

6
Anita Hæhre
Seasoned Practitioner
1 year ago
Ben Brandt
Seasoned
Ben BrandtSeasoned
asked in Developer and User Community

Dashboard Query: Top 10 from Aggregate Sum of ~1000 Tim Series

I need to add a dashboard query that is simple to write with SQL but seems to be complex to write with the CDF API.Input parameters:start and end timestamp range List of time series to queryQuery steps:For each time series in the list, sum all values that are in the user provided range Order time series by the sum Return metadata from the top 10 of this ordered listMaybe something equivalent to this:-- Likely SLOW in a SQL DB with many datapoints-- This is a conceptual example, so simplifying by combining Time Series and -- Data Points to single entity.Select TOP 10 TimeSeries.Name, SUM(TimeSeries.Value)FROM TimeSeriesWHERE TimeSeries.Id IN (@ListOfTimeSeries)AND TimeSeries.Timestamp >= @StartTimestampAND TimeSeries.Timestamp < @EndTimestampGROUP BY TimeSeries.Id, TimeSeries.NameORDER BY SUM(TimeSeries.Value)What’s the simplest way to get to this data with the tools provided to a software developer interacting with CDF?

6
E
Practitioner
1 year ago
Ben Brandt
Seasoned
Ben BrandtSeasoned

Cognite Data Fusion: Data Set Group/Tenant/Company Selector Within a Project To Switch Between Groupings of Data Sets User Has Access ToGathering Interest

Hello!We currently have a multi-tenant application (multiple companies data can be stored within same DB with a tenant Identifier to segment data).  Our business performs services work for these companies, so we have internal employees that are granted access to several of these companies.  We are in the process of migrating some of our functionality over to being backed by CDF.  During this transition, we will have internal employees demoing to customers in both our product and the CDF UI.  For Data Sets, we plan to externalId them prefixed with Company/Tenant Identifier combined with the data source, so that the externalId may be something like “abc-inc-app-that-does-stuff” where “abc-inc” would be the tenant identifier.  When our internal employees are demoing results using both our product and CDF to Company A, the internal employee does not want other companies (Company B) data to be visible.  In our app, a user with access to multiple companies is able to switch between Companies

5
Anita Hæhre
Seasoned Practitioner
2 years ago
Ben Brandt
Seasoned
Ben BrandtSeasoned

Cognite Data Fusion: Document Contract/Expectations of a Cursor Returned From The APIImplemented

Let’s start with a few questions:When a nextCursor value is returned from an API call, is this cursor value valid forever or does it expire? If it expires, how long is it guaranteed to be valid? (This documents part of the contract of the API.) When we call an API subsequent times with the same parameters and provide a cursor value, are we getting data at the time of the current API call or cached data from the first time the API was called with those parameters? If subsequent calls to the API with a cursor are NOT returning results of a cached copy and data has been inserted or deleted in the middle of the list while iterating through the cursors, is there any guarantee that data is not duplicated or skipped in the result sets returned? Please add these tidbits to the docs here:https://docs.cognite.com/dev/concepts/pagination/ Thanks!Ben Brandt

5
C
Seasoned Practitioner
1 year ago
Ben Brandt
Seasoned
Ben BrandtSeasoned

Cognite Data Fusion: Documentation Update: Please Document Behavior of Recursive=false for Asset DeleteImplemented

https://docs.cognite.com/api/v1/#operation/deleteAssetsIt is documented what will happen when recursive=true.  Please document the behavior on this page for the case when children exist and recursive=false. 

5
Anita Hæhre
Seasoned Practitioner
1 year ago
Ben Brandt
Seasoned
Ben BrandtSeasoned
asked in Developer and User Community

Similar to 365d-ago for historical, how do we specify relative time for future?

For the Retrieve data points API and a few other API’s that allow date range input, we are able to specify a string where, “The format is N[timeunit]-ago where timeunit is w,d,h,m,s.”  This is able to specify a time in the past.  I am now trying to specify end as “-365d-ago” to set a date 365 days in the future, but this is giving an error.Is there another syntax available for specifying future dates?N[timeunit]-afterNow?365d-afterNow?

4
Ben Brandt
Seasoned
2 years ago
Ben Brandt
Seasoned
Ben BrandtSeasoned
asked in Developer and User Community

Error Searching On Cognite Hub

403 ERRORThe request could not be satisfied.Request blocked. We can't connect to the server for this app or website at this time. There might be too much traffic or a configuration error. Try again later, or contact the app or website owner.If you provide content to customers through CloudFront, you can find steps to troubleshoot and help prevent this error by reviewing the CloudFront documentation.Generated by cloudfront (CloudFront)Request ID: 5cVGYuZaPs0Tnwt0Mm7an00fCMQ14bwLFy-SDA27JszqfsOdnRdbtg==

4
Ben Brandt
Seasoned
1 year ago
Ben Brandt
Seasoned
Ben BrandtSeasoned

Cognite Data Fusion: Data Set Archive Feature That Also Archives Data Within the Data SetClosed

Hello!After reading the documentation for CDF Data Sets I ran a few experiments with the Python SDK, creating assets, time series, and data points all within a data set.  When I was done, so as not to clutter up the CDF instance we are all sharing while prototyping, I selected my data set within the “Create, view, and manage data sets” and clicked the “Archive” option.I browsed back over to the Assets page, but was surprised to still see my assets from my archived data set visible within the CDF UI.  I thought maybe it was just because of eventual consistency as we all struggle with that with distributed systems.  After a while I check the documentation and saw this:Edit and explore data sets | Cognite documentation You can also edit the data sets to add more documentation to them, or to write-protect them to provide an extra layer of protection to production-critical data. By archiving data sets, you can hide data sets that are no longer used.  Archive data sets to hide them from th

4
Anita Hæhre
Seasoned Practitioner
1 year ago
Ben Brandt
Seasoned
Ben BrandtSeasoned

Cognite Data Fusion: Add .NET 6 Support to Cognite FunctionsGathering Interest

Cognite Functions are looking like a great solution to filling some of the gaps to what the CDF API is able to provide, allowing us to effectively extend the API with custom aggregations not possible with the limitations of  Synthetic Time Series as well as alleviate the need for application vendors to run more complex business logic in their own backend service layer.Currently the native language during the Cognite Functions preview is Python.  As Cognite Functions moves towards being production-ready I propose the next language/ecosystem that official support be added for is .NET 6.  Our company has quite a lot of IP written in a .NET backend and it would be nice to be able to leverage Cognite Functions to port this over and quickly swap out our queries to our Azure SQL instance over to where data will be persisted in CDF.

4
Ben Brandt
Seasoned
2 years ago
Ben Brandt
Seasoned
Ben BrandtSeasoned
asked in Developer and User Community

Nesting of Synthetic Time Series - Combining More than 100 Time Series

Are Synthetic Time Series able to reference other Synthetic Time Series in their formulas specified?   This is the problem I am trying to solve:Synthetic Time Series is limited to referencing 100 time series Sometimes a unit/facility/system/functional location/site may have more than 100 direct child assets I need to calculate a “Synthetic Time Series” that is the daily average of a time series found on all child assets My thought is that, for 300 assets, I would have:STS A: Sum for assets 1-100 STS B: Sum for assets 101-200 STS C: Sum for assets 201-300 STS D: Sum of STS A, B, and C divided by 300Is there a better way to accomplish something like this in CDF?

4
Ben Brandt
Seasoned
2 years ago
Ben Brandt
Seasoned
Ben BrandtSeasoned

Cognite Data Fusion: Promote CDF .NET SDK to Official Supported Instead of Community SupportedGathering Interest

The first line of the readme in cognitedata/cognite-sdk-dotnet: .NET SDK for Cognite Data Fusion (CDF) (github.com) is:Under development, not recommended for production use cases We are currently using this library in production, but we are having to add our own support for some features that are natively built into the Python SDK:Mocking/Testability Concurrency and connection pooling More mature data classes/SDK “glue”Please devote time and effort towards making this library production ready and officially supported.  Thank you!

3
Ben Brandt
Seasoned
2 years ago
Ben Brandt
Seasoned
Ben BrandtSeasoned

Consistency of Field Names for PostreSQL Gateway and Direct QueryGathering Interest

In PostgreSQL gateway the datapoints use the field name “time_stamp”.  In the PostgreSQL connection used by direct query the field is called “timestamp”.  As much as possible, can these field names across these providers be consistent with one another?

3
Ben Brandt
Seasoned
1 year ago
Ben Brandt
Seasoned
Ben BrandtSeasoned
asked in Developer and User Community

GraphQL Federation with Schema Services/Templates

At release, will Schema Services support being federated into a GraphQL schema, like with Hot Chocolate or Apollo?  This is how we are planning to utilize combining what is available in CDF with what we are providing in our GraphQL schemas and I was assuming this is feasible.  But, a dev mentioned that the downstream services have to support being federated for it to be made available at the federation gateway. If you are using Lighthouse for your implementation, then I see it has preview support for being federated, but I need to confirm you will configure it this way or whatever you may be using will support being federated. Thanks!

2
David Liu
Seasoned Practitioner
2 years ago
Ben Brandt
Seasoned
Ben BrandtSeasoned
asked in Developer and User Community

DirectQuery For Future Datapoints

Hello!I am querying for DataPoints using the PostgreSQL Gateway, which I assume will have the same behavior as the DirectQuery feature built on the PostgreSQL Gateway.  What I am noticing is that I am not able to retrieve any DataPoints for future dates.  Is this intended or a bug?  Most of the data we produce and manage is in the future, so this would be an important feature for us if it could be supported.  Thanks! Query results returned in Azure Data Studio from PostgreSQL gateway:In CDF these time series have datapoints representing predictions far out into the future: 

2
Ben Brandt
Seasoned
1 year ago
Ben Brandt
Seasoned
Ben BrandtSeasoned
asked in Developer and User Community

Document Behavior of Create Time Series Status Code 422

Please add better documentation around the behavior of status code 422 for Create Time Series.  Specifically, I’d like these questions answered:When multiple external ids are given and some are duplicate and some are not, will the time series that are not duplicates be inserted and will these created ids be returned in the response? Does this function similar to datapoints where create is actually create or update and the duplicates are updated with the data provided?  If not, what is the recommended pattern for create or update of time series entities through the API?I am aware of advice here around “EAFP vs LBYL”, but I am in a case where 98% of the time the time series will not exist and I’d like to optimize for this case.  Thanks!

2
C
Seasoned Practitioner
1 year ago
Ben Brandt
Seasoned
Ben BrandtSeasoned

Cognite Data Fusion: Move Magic Datapoint Granularity Selection Algorithm From Direct Query to the APIParked

I have been playing around with the excellent DirectQuery preview functionality and really enjoyed that I can simply give it a timestamp range and it figures out how many points I need and what the granularity will be with the assumption that I will use the queried data for some visualization.  Even if someone has an 8K monitor with no border around their chart, with one pixel per point the most I could possibly need would be 7680 datapoints.  Reality is that we need much less since we show a datapoint as more than a pixel width.  But I don’t have to care about that complexity.  Cognite calculates optimal granularity for me. Can this feature also be available via the API for “Retrieve data points”?https://docs.cognite.com/api/v1/#tag/Time-series/operation/getMultiTimeSeriesDatapointsMaybe we would be able to set granularity to “auto” then the magic happens? 

2
Anita Hæhre
Seasoned Practitioner
1 year ago
Ben Brandt
Seasoned
Ben BrandtSeasoned
asked in Developer and User Community

Show Average KPI For Time Series From Today to End That is Read Often

Hello, Thank you for all the great docs and tutorials you’ve put together on CDF.  I would like to get an expert opinion on the best way to accomplish a specific use case I have.We have a daily time series that we calculate for predicted availability of an asset, for a time range that is usually between 5 and 40 years (usually between 1,826 and 14,610 data points per time series).  We have a view were an engineer will want to see this time series graphed, but also will want to see a single KPI value that represents the average of the time series from today to end of the range.  For all of our assets, this KPI could be ready several times a day as people view this KPI.  When we were storing this data in Azure SQL, to speed it up, we pre-calculated and stored this KPI average and stored a new time series (throwing away some of the time series points and linear interpolating).  That way, reading a single data point representing the average of from today to the end of the range and display

2
Ben Brandt
Seasoned
2 years ago
Ben Brandt
Seasoned
Ben BrandtSeasoned
asked in Developer and User Community

Bug: Curl Example Syntax is Invalid For dataModel ACLs

The curl command suggested here needs quotes around all JSON property names, needs to use " instead of ', and needs trailing commas removed before curl will stop reporting invalid JSON payload. 

1
Anders Hafreager
Practitioner
1 year ago
Ben Brandt
Seasoned
Ben BrandtSeasoned

Cognite Data Fusion: Expand short notation for data point retrieval to support both relative past and future datesGathering Interest

More info here:  

1
Anita Hæhre
Seasoned Practitioner
2 years ago
Ben Brandt
Seasoned
Ben BrandtSeasoned

Cognite Data Fusion: Add Dataset and Contextualization Annotation Support to Cognite ReplicatorGathering Interest

Dataset Awareness · Issue #173 · cognitedata/cognite-replicator (github.com) We have discovered that when using Cognite Replicator, our source Project has dataset id's set, but in the target Project all dataset id's are null. Please add support for replicating datasets and setting dataset id's on all entities to the replicator. The workaround we will be operating on in the meantime is to run a script afterward to copy the Datasets and set the dataset id's. Add Support For Mapping of Annotations · Issue #174 · cognitedata/cognite-replicator (github.com) Annotations are stored in Cognite as events of type cognite_annotation. There are sometimes id values in metadata fields CDF_ANNOTATION_resource_id and CDF_ANNOTATION_file_id that we have seen. We depend on these id's being correct to show the list of files applicable to an asset or other entities. When these events are being replicated to the target project, the id's are not being updated with the target id. Please add support

1
Anita Hæhre
Seasoned Practitioner
2 years ago

Badge winners

Show all badges
Terms & ConditionsCookie Settings