Skip to main content
Gathering Interest

Ability to download/upload a config file in extraction pipeline

Related products:Extractors
HanishSharma
Practitioner

Background

Currently CDF allows

  • uploading a remote extraction config file via python sdk/api - for deployment engineers
  • editing in the UI - for Fusion UI users

We have noticed in several projects some extraction config files become huge (> 5000 lines). The UI becomes unresponsive in handling such large files. Technical users like deployment engineers are able to edit/update this config through their code but it becomes hard of non-technical users to update stuff in config files (like adding new queries to the ODBC extractor config file).

 

Requirement

If we allow users with a download and an upload option in the UI, it will be helpful for the users to edit/update large config files within the UI. Users can use their local file editor which would be easier to handle.

 

Jørgen Lund
Seasoned Practitioner

Hi @HanishSharma.

Thank you for suggesting this product idea. 

When editing the config file, you’re able to copy the entire config and paste it into a local file editor. Would following this pattern be sufficient to support the users and workflow that you’re suggesting? Or is it important to be able to download to retain the file format of the config? 

We’re also planning an improved experience for extractor configuration. We’ll likely be able to alleviate this problem directly in the next version of the config editor in the UI, so taking your feedback as input into that work.

 

HanishSharma
Practitioner
Forum|alt.badge.img+3

Hey @Jørgen Lund - I missed this among notifications!

 

Yes, we can copy and paste the full file, but when the config becomes huge (>5000 lines) because of large queries, copying and pasting it from an external editor makes it prone to some unknown formatting issues. I have experienced this a couple of times where the yaml gives an error when I copy and paste it VS code, but when I paste the same to notepad and then into the UI - it works! 

 

Looking forward for the upcoming update like you mentioned!

T
Seasoned Practitioner
Forum|alt.badge.img

Hi Hanish,

Is the problem you’re raising more of a problem with the cumbersomeness of managing configurations with large queries and include/exclude lists through a web UI, or that you’d prefer to use a CI/CD process to deploy and manage the config file containing complex queries and include/exclude lists?

Also, is this about a specific on-premises extractor, or across multiple (which ones)?


Terms & ConditionsCookie Settings

Cookie Policy

We use cookies to enhance and personalize your experience. If you accept you agree to our full cookie policy. Learn more about our cookies.

 
Cookie Settings