Replies posted by Jørgen Lund

53 Replies

B
Active
Bertil EkernActive

Graphical view of bid matrixPlanned for development

Thanks Bertil, a graphical view of the bid matrix is something we have discussed internally in the team multiple times, so we appreciate this input! Your suggestion sounds like one of the better ways it could be implemented 👍  Adding this feature to the product is not yet on the roadmap, but @Marte Nielsen can elaborate on that.Would be interesting to get more opinions on this topic from others at Hafslund or Lyse (cc @Sindre Tøsse)! 

3
Anita Hæhre
Seasoned Practitioner
1 year ago
B
Practitioner
Benjamin MedbøenPractitioner

PI Extractor - support for writing metadata to RAW tableImplemented

I confirm this will be planned for the December release.

5
Jørgen Lund
Practitioner
8 months ago
N
Committed
Niranjan Madhukar KarvekarCommitted

How to use is_new() function with lastUpdatedTime from FDM type in transformationPlanned for development

Hi, update on this ticket: We originally planned this for the October release, but after looking closer at possible solutions this week we have decided to go for an implementation that is slightly more complex, but that we believe will provide a better experience and performance for the users. The timeline is therefore a bit more unceratin, but we’re working actively on this.

10
N
Committed
5 months ago
N
Committed
Niranjan Madhukar KarvekarCommitted

How to use is_new() function with lastUpdatedTime from FDM type in transformationPlanned for development

@Aditya Kotiyal Thanks for the additonal context, and apologies for the late reply. We’re looking into the implementation details, which brings a follow-up question to be able to answer your question: How frequently are you planning to execute the Transformations that will leverage this functionality once it’s available? Initially, there will be a difference implementation for jobs running more frequently than every 3 days (e.g. hourly), and jobs running less frequently (e.g. weekly). 

10
N
Committed
5 months ago
N
Committed
Niranjan Madhukar KarvekarCommitted

How to use is_new() function with lastUpdatedTime from FDM type in transformationPlanned for development

Hi @Niranjan Madhukar Karvekar @Aditya Kotiyal. Great, thanks for the additional information. The team will prioritize this feature, meaning you do not have to go for a workaround as we will provide the proper solution soon. I will keep you updated, and let you know when you can try it out.

10
N
Committed
5 months ago
A
Seasoned
Akash SoodSeasoned
asked in Data Workflows [Early Adopter]

Workflows use case: Run a function post successful run of previous function

Hi @Akash Sood, thank you for the question.Workflow Orchestration allows you to define the dependencies between Cognite Functions and have them executed accordingly.If a Cognite Funciton “B” depends on the successful completion of Cognite Function “A”, you could define a workflow “A” → “B”, which when executed would first trigger “A”, and on its successful completion, trigger “B”. If “A” fails, the workflow will fail, and “B” will not be triggered. The workflow definition is quite flexible, so more complex dependencies between steps are possible (e.g. in cases where you want to execute a set of Functions, wait for all to complete, before executing another set of Functions).It sounds like you would be able to leverage Workflows to solve your use case, and I encourage you to try it out. @Aditya Kotiyal can help you get set up. Note that the service is currently in Beta and should not yet be relied upon in a production setting. Let me know if there are additional nuances to your use case,

2
A
Seasoned
6 months ago
L
Committed
Leandro SantosCommitted
asked in Data Workflows [Early Adopter]

Triggering the workflow run

Hi @Leandro Santos, thanks for the question! Currently, the way to trigger a worklfow is using the /run endpoint. To execute a worklow on a schedule, we recommend using a scheduled Cognite Function as described in the last part of the example notebook that you can find in this post.Native capabilities for creating schedule- and event-based (e.g. based on new data arrival as you suggest) triggers for workflows is something we are looking into adding at a later stage, so any input on specific functionality you would need is valuable information. 

1
Jørgen Lund
Practitioner
7 months ago
N
Committed
Niranjan Madhukar KarvekarCommitted
asked in Data Workflows [Early Adopter]

workflow orchestration failures for transformations

Hi @Niranjan Madhukar Karvekar, we’ll look into this today and get back to you.

7
Jørgen Lund
Practitioner
5 months ago
N
Committed
Niranjan Madhukar KarvekarCommitted
asked in Data Workflows [Early Adopter]

workflow orchestration failures for transformations

Thanks for the additional information. Our conclusion is indeed that it is an auth related issue. The team is investigating, I’ll keep you posted.

7
Jørgen Lund
Practitioner
5 months ago
A
Seasoned
Aditya KotiyalSeasoned
asked in Data Workflows [Early Adopter]

ML model and training it

@Aditya Kotiyal Cognite Functions resource limits can be found here.

8
A
Seasoned
2 months ago
O
Committed
Oussama ALLALICommitted

Lack of best practices to use CDF (e.g. transformation)Planned for development

Hi @Oussama ALLALI. We are currently working on best practices documentation for Transformations and our staging service (Raw). This will be made available within the end of the year. In addition, work is ongoing to make reference end-to-end implementations, across product capabilities, available for users to refer to as examples and to build on. You are probably already aware, but in the meantime please refer to this page in our documentation for specifics on Transformations.

1
Jørgen Lund
Practitioner
6 months ago
N
Committed
Niranjan Madhukar KarvekarCommitted
asked in Data Workflows [Early Adopter]

workflow orchestration failures for transformations

Hi, for the ‘Unknown reason’ failure, we have identified the problem and are working on implementing a solution. It is related to the IdP used, which why it was not catched earlier. We hope to have this resolved by end of next week. For the other failures you were seeing (initial post) of 'Transformation job is already running and failure mode is set to fail on running' we rolled out a fix yesterday. Please let us know if you keep seeing this type of failure when not expected (concurrency policies for Transformations is a new feature, briefly described here, where failing a workflow task when the Transformation is already running is one of the options). 

7
Jørgen Lund
Practitioner
5 months ago
S
Active
Sverre LofthusActive

Add filter functionality based on column and row valuesParked

Hi @Sverre Lofthus, thank you for submitting this product idea. Would you be able to provide a bit more context and describe your use cases for such a feature? This would help us better understand the need and how it could be addressed.

3
Jørgen Lund
Practitioner
6 months ago
S
Active
Sverre LofthusActive

Add filter functionality based on column and row valuesParked

Got it. Just to make sure I understand: there is some overlap between the the data the 14 different triggers need, but also some differences? I.e. it would not be a solution to separate the data into more granular tables? 

3
Jørgen Lund
Practitioner
6 months ago
N
Committed
Niranjan Madhukar KarvekarCommitted
asked in Data Workflows [Early Adopter]

workflow orchestration failures for transformations

Update: we are still working on resolving this, will get back here once it’s in place.

7
Jørgen Lund
Practitioner
5 months ago
Z
zgchinaNew
asked in Data Workflows [Early Adopter]

Monitoring Functionality and Notification Setup in Workflow

Hi @zgchina. We are working on improved notifications functionaliy that will work across CDF. We want to make sure the notification capabilities for workflows fits in with this to create a more coherent experience across the platform, and combines well with e.g. Transformations.At the moment, I’m not able to provide an accurate timeline, but it is in progress and notifications is one of the main priorities for new features to be added to workflow orchestration. 

1
Jørgen Lund
Practitioner
5 months ago
H
Committed
HarshaCommitted

Extractor Monitoring DashboardPlanned for development

Hi @rsiddha @Harsha , thansk for sharing your insight. A follow-up question: in addition to extractors, would it be relevant to include the other processing/transformation steps that happen within CDF (e.g. Transformations, Functios, various contextualization jobs) in the same dashboard? 

9
R
Seasoned
1 month ago
N
Committed
Niranjan Madhukar KarvekarCommitted
asked in Data Workflows [Early Adopter]

workflow orchestration failures for transformations

Hi @Leandro Santos @Niranjan Madhukar Karvekar We have released a fix for SAuth-related issue. Take a look at this post for an example on how to schedule workflow executions with Cognite Functions without running into issues. You should not experience failures similar to the ones reported in this post, but let us know if you do, or if you run into other issues.Thanks for your patience!FYI @Aditya Kotiyal @Everton Colling  

7
Jørgen Lund
Practitioner
5 months ago
Diego Antonino
Practitioner
Diego AntoninoPractitioner

Enhance Extraction Pipeline Monitoring ViewPlanned for development

Thanks for the insight @Diego Antonino! We are planning improvements to the monitoring capabilities across extractors and other data onboarding processes in CDF. They might not come in the exact shape and form outlined in your suggestion, but it is great input that we’ll bring into the work on shaping a better monitoring experience. We might reach out to you for additional input if needed. 

2
Jørgen Lund
Practitioner
4 months ago
H
Committed
HarshaCommitted

Extractor Monitoring DashboardPlanned for development

Thank you all for the great input. We are planning improvements to the monitoring capabilities across extractors and other data onboarding processes in CDF, so I’m changing the status of this Product Idea. If you have more details or related input you’d like to share, please don’t hesitate to add more to the comments. We’ll make sure to bring all of it into the process of shaping a better monitoring experience in CDF.

9
R
Seasoned
1 month ago
Christian Alvim
Practitioner
Christian AlvimPractitioner

Show credentials in transformation run historyGathering Interest

Hi @Christian Alvim, thanks for sharing. Would the need for the user mainly be to see which runs were started by the user vs. which ones were started by others? Or also to have visibility of which user/client started each job?

2
Christian Alvim
Practitioner
4 months ago
I
Seasoned
ibrahim.alsyedSeasoned

Improve CDF Raw StoreGathering Interest

Hi @ibrahim.alsyed - thanks for this feedback and input.As you observe, both Raw and Transformations have certain limitations. We are working on making these services more robust and performant. We will also bring the input you proivde here into our efforts to evolve the stage and transform layer of CDF going forward. A few questions:Would it be fair to say that transform jobs requiring multiple joins, leading to long run times and aggravating the problem of job failures, are the main pain points you are currently facing?  For the possible workarounds you mention: are you currently doign piecemeal data ingestion (e.g. per Site)? If so, what are the main pain points of having to take this approach? 

1
Jørgen Lund
Practitioner
4 months ago
HanishSharma
Practitioner
HanishSharmaPractitioner
asked in Data Workflows [Early Adopter]

Issues in Running a Flow in a Project

Hi @HanishSharma, thanks for reporting.From our logs we see that it’s an issue related to the IdP you’re using (SAuth) and Cognite’s Sessions API. We’ll follow up on this as soon as possible, and keep you posted on any updates here. 

11
Dilini Fernando
Seasoned Practitioner
1 month ago
HanishSharma
Practitioner
HanishSharmaPractitioner
asked in Data Workflows [Early Adopter]

Issues in Running a Flow

Hi @HanishSharma, looks like a duplicate post, let’s follow up in the other thread 🙂

2
Dilini Fernando
Seasoned Practitioner
3 months ago
HanishSharma
Practitioner
HanishSharmaPractitioner
asked in Data Workflows [Early Adopter]

While running the Flow it does not considers the updated version

Hi @HanishSharma.The graphical user interface is unfortunately in an alpha state, and has certain shortcomings currently. From the screenshots, it looks like the workflow definition did not get updated properly (version number should change to “v2” if you make a change and publish it successfully), like this: Could you try updating the workflow definition again, and see if that leads to a version update? Otherwise, you could try recreating the workflow. I’ve tested this on my side now, and unable to reproduce the issue. 

3
Jørgen Lund
Practitioner
3 months ago

Badge winners

Show all badges
Terms & ConditionsCookie Settings