Impact 2024: The Industrial Data and AI Conference for and by Users | Nominate Speakers Now for a Ch...
Thanks! I am not blocked or anything, so if there’s any action of closing this thread, it can be done now.
This SQL seems to run perfectly fine on the PostgreSQL gateway from Azure Data Studio on my local dev machine against my CDF project. Any plans to remove the, “Querying data from CDF for analysis and visualization is not supported.” statement from https://docs.cognite.com/cdf/integration/guides/interfaces/postgres_gateway/#when-should-you-use-the-postgresql-gateway and make querying officially supported?
Clarification, this is being developed in a SaaS application running in Azure not within Power BI.
https://hub.cognite.com/search?q=%22order%20by%22 Maybe it thinks its a SQL injection attack and filtering it. Other queries seem to work.
@Thomas Sjølshagen @Anita Hæhre Was this documented as a “Breaking Change” in the SDK for anyone that may be expecting the old behavior?
Confirmed: Minimum TS now seems to be -2208988800000.Date and time (GMT): Monday, January 1, 1900 12:00:00 AM { "error": { "code": 400, "message": "Timestamp cannot be lower than -2208988800000. Was '-2211753600000'" }}
Looks like there’s been progress. Docs now say this:Timestamps are in the range from 00:00, January 1st, 1900 to 23:59, December 31st, 2050. Negative timestamps are used to define dates before 1970 I was able to run a Synthetic Time Series Query in Postman with a negative “start”. Now it’s up to the individual SDK’s to update their validations for start. Any plans to move end out further into the future? The max 64 bit integer, 9,223,372,036,854,775,807, can get us pretty far out there.
@vwrocks8 Is there a known IP that the Cognite Function traffic would be coming from that @Pedersen Jon-Robert would be able to whitelist on his end so that it’s not completely wide open on the internet?I am not aware of what infrastructure you are running on. For example, this is how the firewall rules work with Azure SQL:IP firewall rules - Azure SQL Database and Azure Synapse Analytics | Microsoft Docs
Thank you for the response, Thomas. I have heard from our product owner, @Mark Felder, that dates after 2050 will be needed for us for storing predictive results, but future is not as urgent as historical. Thanks!
I understand that we all have limited capacity, it is quite difficult to hire developers to expand capacity at this point in time, and you have prioritized Python to come first. Thank you for the update.
FYI, the README statement has been updated to:Unofficial: please note that this is an unofficial and community driven SDK. Feel free to open issues, or provide PRs if you want to improve the library.
@DanielJohnLevings Could it be assumed you are talking purely about the time series data points or are you also thinking about sequences and other entity types?
Really great update, @David Liu! I am super impressed with the work that your team has been doing. One thing I am still a bit confused about is how the Template functionality stays in sync with existing String, Long, and Double fields that are in the metadata of existing CDF data model types like assets or events. The impedance mismatch I am seeing is that while CDF pulls data together to be able to query in one place, the shape of the data can vary wildly across different CDF projects or sometimes within the same CDF project when it comes to the metadata fields. Much like how the RAW data can be in any shape, the metadata fields on the CDF data model cannot be expected to fulfill any type of contract that an application layer needs as key names are not standardized (Ex: variation in casing, naming. units, and level of nesting of keys: OrificePlateDiameter vs orifice_plate_diameter vs orifice_plate_diameter_inches vs orifice_plate_diameter_centimeters).That is why I am very hopeful
One more difference that I recall may need to be accounted for is that AAD exposes both a Group Name and Group Id. CDF depends on checking against a Group “Source ID” when managing access. From some experimentation and reading in the Auth0 forum, Group Name is simple to provide in the token, but getting the internal Group Id requires some additional calls to API’s which may be rate limited and not recommended to use in this context. We will have to check if Auth0 adds the Group name to the approriate claim if we can just configure “Source ID” and “Source name” in the CDF Access Management/Groups/Create new group UI.
A few Cognite and Auth0 cross-references I’ve come across:https://github.com/cognitedata/cognite-sdk-js/pull/660#issuecomment-903535822dshub/auth.tsx at 23b7bdfef51d0cebbaaa5c7bad3bd22f74295e24 · andeplane/dshub (github.com) Cognite SDK/API is able to support Auth0:cognite-sdk-js/authentication.md at 21823f6618d6bf11dd738fb022289677259966da · cognitedata/cognite-sdk-js (github.com)https://docs.cognite.com/cdf/access/troubleshooting/troubleshoot_oidc/#using-other-idps-than-azure-adhttps://docs.cognite.com/cdf/access/concepts/minimum_idp_requirements It seems most of CDF supports using Auth0. The frontend may be the only piece lacking?
@AasmundEldhuset As requested, I have added the product idea here:
Hi @Torgrim Aas,This query will be run on demand from a facility/unit dashboard. It will run several times a day for each facility or unit. Yes, the output of the calculation/aggregation is a single time series with one data point per day for a user configurable number of days/years. When implemented in Azure SQL, we did it as a post processing job after each child simulation was completed. I think a CDF transformation might work well for us. I saw where a transformation can be scheduled or manually triggered via the API. But, is there any type of configurable eventing for this where something like a data change event in CDF could trigger the transformation?
Hi @Torgrim Aas,Thank you for the suggestion. The time series we will need to aggregate is data that we are producing ourselves and known to have a consistent start and end date for all children and be at daily resolution. Usually the range is between 5 and 40 years. The average produced would be a new time series with the same start and end as all child data point.Simple example with 3 ts and 4 days: If we have 40 years of daily data for 500 time series, this would be ~7,305,000 data points. Are you suggesting that we should be able to load all data point in memory in a Cognite Function, perform the average for each day and then return the result pretty quickly (< 1sec)? With the maximum memory allocation of 2.5 for a Cognite Function, what would the upper limit of how many data points would fit into memory? We can always chunk it into 5 year increments and make multiple calls if needed. Thank you,Ben
Thank you for the info, Torgrim. That was the other complexity of the situation I had left out. The value not only has to change daily, but each time a new data point arrives. Currently most of our data sources are daily or lower resolution, but this could certainly change to higher resolution in the future. I had discussed with Morten and this is the additional detail he provided:The time series capability in CDF can do this on the fly. Aggregates are automatically created by the backend so that you can in a few milliseconds get averages and other metrics. For python details, see https://cognite-docs.readthedocs-hosted.com/projects/cognite-sdk-python/en/latest/cognite.html#retrieve-datapointsYou can set aggregate to "average" Great to know CDF can provide these aggregates in milliseconds and I no longer need to maintain my own code to pre-aggregate these values. I am going to need to look further at synthetic time series for another case where we want to get the daily average of a
Thank you, Kevin. That sums it up will. One clarification is that I would like the filtering to apply to both the data sets and the other entities attached to those data sets (the data within the data sets). Example Case:Data Set Group: Company ABC Data Set: company-abc-some-calculation-results Data Set: company-abc-some-pi-data Data Set: company-abc-some-events-from-sap Data Set Group: Company DEF Data Set: company-def-some-calculation-results Data Set: company-def-some-pi-data Data Set: company-def-some-events-from-sap As a user with access to both Company ABC and Company DEF, I would like to toggle between which Company’s data is visible in the CDF UI. Feel free to substitute the word “Company” with “Customer” or “Tenant” if that is more clear in CDF’s terminology. Thanks!
My proposed backlog order for Product Owner:Delete of empty data set Archive of data set including data Delete of data set including data
Yes. Deleting an empty data set should be introduced first. After that, I propose that the feature to delete a Data Set “Including Data” be introduced.
To be clear @Anders Willersrud, for this feature request, when you say “delete a data set”, do you mean delete only the data set entity and any assets, time series, or other entities would remain, but have their data set id set to null or would you expect that deleting a dataset would also delete any data associated with that data set?
@Ashutosh Kumar ,Where will the recording of the webinar be posted for later review? Thank you!Ben Brandt
Already have an account? Login
Enter your username or e-mail address. We'll send you an e-mail with instructions to reset your password.
Sorry, we're still checking this file's contents to make sure it's safe to download. Please try again in a few minutes.
Sorry, our virus scanner detected that this file isn't safe to download.