Replies posted by Jørgen Lund

17 Replies

R
Committed
Rajendra PasupuletiCommitted

OData connectivity for Extractors and Transformations for Power BIParked

Hi @Rajendra Pasupuleti, you can fetch information about Transformations and Extraction Pipelines from our APIs and use your preferred framework to create a dashboard assuming it allows you to use custom code to fetch data.I’d recommend using Python, as it will allow you to leverage our Python SDK. You can also take a look at using our Streamlit capabiltiy in Fusion (currently Alpha) as the framework to implement the dashboard.

7
Jørgen Lund
Practitioner
28 days ago
M
Active
MatiasRamslandActive

"Many-to-Many" relationship between "Extraction Pipelines" and datasetsGathering Interest

Thanks @MatiasRamsland and @Øystein Aspøy.Just for clarifiation: is the extractor in this example writing to multiple datasets directly, or to a single dataset that is subsequently processed/moved into a number of other datasets?In general I think this sounds like a good idea, and we’ll consider it as part of the improvements we are planning for monitoring. 

3
M
Active
2 months ago
Øystein Aspøy
Committed
Øystein AspøyCommitted

Add tenant and project to Extraction pipeline alertsGathering Interest

Thanks @Simon Bay Andersen - we’ll look into adding project information to the email for extraction pipelines. 

3
Jørgen Lund
Practitioner
2 months ago
Øystein Aspøy
Committed
Øystein AspøyCommitted

Add tenant and project to Extraction pipeline alertsGathering Interest

Hi @Øystein Aspøy, thanks for the feedback! Attempting to break it down to clarify and ask follow-up questions: Get more metadata in the email alerts. We could consider adding:   The CDF project of the extraction pipeline that generated the alert. Other extraction pipeline metadata. Here (under “Responses”) you can see a list of all the metadata that will be available for a particular extraction pipeline. From this list, is there anything in particular that you would like to see as part of the email notification?  Edit the information sent by email. Would this be covered if the email notifications by default contain more metadata about the extraction pipeline? You could then e.g. edit the custom metadata fields, and that information would be included in the email. Automation on alerts. Could you elaborate on this a bit? Are you looking to integrate alerts with an external system? I assume this would be separate from email alerts? 

3
Jørgen Lund
Practitioner
2 months ago
I
Seasoned
ibrahim.alsyedSeasoned

Improve CDF Raw StoreGathering Interest

Hi @ibrahim.alsyed - thanks for this feedback and input.As you observe, both Raw and Transformations have certain limitations. We are working on making these services more robust and performant. We will also bring the input you proivde here into our efforts to evolve the stage and transform layer of CDF going forward. A few questions:Would it be fair to say that transform jobs requiring multiple joins, leading to long run times and aggravating the problem of job failures, are the main pain points you are currently facing?  For the possible workarounds you mention: are you currently doign piecemeal data ingestion (e.g. per Site)? If so, what are the main pain points of having to take this approach? 

1
Jørgen Lund
Practitioner
3 months ago
Christian Alvim
Practitioner
Christian AlvimPractitioner

Show credentials in transformation run historyGathering Interest

Hi @Christian Alvim, thanks for sharing. Would the need for the user mainly be to see which runs were started by the user vs. which ones were started by others? Or also to have visibility of which user/client started each job?

2
Christian Alvim
Practitioner
3 months ago
H
Committed
HarshaCommitted

Extractor Monitoring DashboardPlanned for development

Thank you all for the great input. We are planning improvements to the monitoring capabilities across extractors and other data onboarding processes in CDF, so I’m changing the status of this Product Idea. If you have more details or related input you’d like to share, please don’t hesitate to add more to the comments. We’ll make sure to bring all of it into the process of shaping a better monitoring experience in CDF.

9
R
Seasoned
24 days ago
Diego Antonino
Practitioner
Diego AntoninoPractitioner

Enhance Extraction Pipeline Monitoring ViewPlanned for development

Thanks for the insight @Diego Antonino! We are planning improvements to the monitoring capabilities across extractors and other data onboarding processes in CDF. They might not come in the exact shape and form outlined in your suggestion, but it is great input that we’ll bring into the work on shaping a better monitoring experience. We might reach out to you for additional input if needed. 

2
Jørgen Lund
Practitioner
4 months ago
H
Committed
HarshaCommitted

Extractor Monitoring DashboardPlanned for development

Hi @rsiddha @Harsha , thansk for sharing your insight. A follow-up question: in addition to extractors, would it be relevant to include the other processing/transformation steps that happen within CDF (e.g. Transformations, Functios, various contextualization jobs) in the same dashboard? 

9
R
Seasoned
24 days ago
S
Active
Sverre LofthusActive

Add filter functionality based on column and row valuesParked

Got it. Just to make sure I understand: there is some overlap between the the data the 14 different triggers need, but also some differences? I.e. it would not be a solution to separate the data into more granular tables? 

3
Jørgen Lund
Practitioner
5 months ago
S
Active
Sverre LofthusActive

Add filter functionality based on column and row valuesParked

Hi @Sverre Lofthus, thank you for submitting this product idea. Would you be able to provide a bit more context and describe your use cases for such a feature? This would help us better understand the need and how it could be addressed.

3
Jørgen Lund
Practitioner
5 months ago
O
Committed
Oussama ALLALICommitted

Lack of best practices to use CDF (e.g. transformation)Planned for development

Hi @Oussama ALLALI. We are currently working on best practices documentation for Transformations and our staging service (Raw). This will be made available within the end of the year. In addition, work is ongoing to make reference end-to-end implementations, across product capabilities, available for users to refer to as examples and to build on. You are probably already aware, but in the meantime please refer to this page in our documentation for specifics on Transformations.

1
Jørgen Lund
Practitioner
5 months ago
N
Committed
Niranjan Madhukar KarvekarCommitted

How to use is_new() function with lastUpdatedTime from FDM type in transformationPlanned for development

Hi @Niranjan Madhukar Karvekar @Aditya Kotiyal. Great, thanks for the additional information. The team will prioritize this feature, meaning you do not have to go for a workaround as we will provide the proper solution soon. I will keep you updated, and let you know when you can try it out.

10
N
Committed
5 months ago
N
Committed
Niranjan Madhukar KarvekarCommitted

How to use is_new() function with lastUpdatedTime from FDM type in transformationPlanned for development

@Aditya Kotiyal Thanks for the additonal context, and apologies for the late reply. We’re looking into the implementation details, which brings a follow-up question to be able to answer your question: How frequently are you planning to execute the Transformations that will leverage this functionality once it’s available? Initially, there will be a difference implementation for jobs running more frequently than every 3 days (e.g. hourly), and jobs running less frequently (e.g. weekly). 

10
N
Committed
5 months ago
N
Committed
Niranjan Madhukar KarvekarCommitted

How to use is_new() function with lastUpdatedTime from FDM type in transformationPlanned for development

Hi, update on this ticket: We originally planned this for the October release, but after looking closer at possible solutions this week we have decided to go for an implementation that is slightly more complex, but that we believe will provide a better experience and performance for the users. The timeline is therefore a bit more unceratin, but we’re working actively on this.

10
N
Committed
5 months ago
B
Practitioner
Benjamin MedbøenPractitioner

PI Extractor - support for writing metadata to RAW tableImplemented

I confirm this will be planned for the December release.

5
Jørgen Lund
Practitioner
7 months ago
B
Active
Bertil EkernActive

Graphical view of bid matrixPlanned for development

Thanks Bertil, a graphical view of the bid matrix is something we have discussed internally in the team multiple times, so we appreciate this input! Your suggestion sounds like one of the better ways it could be implemented 👍  Adding this feature to the product is not yet on the roadmap, but @Marte Nielsen can elaborate on that.Would be interesting to get more opinions on this topic from others at Hafslund or Lyse (cc @Sindre Tøsse)! 

3
Anita Hæhre
Seasoned Practitioner
1 year ago

Badge winners

Show all badges
Terms & ConditionsCookie Settings