Topics started by MatiasRamsland

2 Topics

M
Active
MatiasRamslandActive

"Many-to-Many" relationship between "Extraction Pipelines" and datasetsGathering Interest

Hi, I am not sure if this is already on your roadmap, however I wanted to share an idea that could potentially improve the current data management processes in CDF. Our team utilizes for some data integrations a single extractor to fetch data that multiple datasets uses. This is done to minimize the load on our source enterprise databases, and reduce the maintenance of having a single larger extraction rather than many smaller extractions. This approach, presents a challenge in monitoring and lineage visibility, as each extraction can only be linked to only one dataset.The core of the proposal is to consider a "many-to-many" relationship between Extraction Pipelines and Datasets. This would allow us to maintain our integrations easier while significantly improving lineage visibility for all related datasets. Implementing such a feature could provide a clearer overview of data dependencies, that I believe other firms than AkerBP would also find useful. 

3
M
Active
2 months ago
M
Active
MatiasRamslandActive

Proposal for Enhanced Dataset Overview by displaying the Integration OwnerGathering Interest

Hi,I'm reaching out to suggest an enhancement to the CDF's dataset overview functionality that could significantly improve user experience and efficiency.Currently, while reviewing a dataset in the CDF platform, users can see the data owner but not the integration owner or the specific developer/team responsible for the dataset's integration. This gap in the documentation requires users to spend considerable time identifying the individual or team who developed the pipeline and is accountable for its integration.To address this challenge and enhance our workflow, I propose adding an "Integration Owner" field to the summary box within the dataset's overview tab. This addition would provide a clear, immediate reference to the responsible party, streamlining communication and troubleshooting processes.

1
Armin Gharakhanloo
Practitioner
2 months ago

Badge winners

Show all badges
Terms & ConditionsCookie Settings