Skip to main content
Solved

DB Extractor: Multiple query execution via single .ymal file


Forum|alt.badge.img+5

How can I execute two or more different queries extracting the data from the source and inserting the data in the same CDF RAW table?
Currently only the last query available in the config.yaml file gets executed.
 

Best answer by Pierre Pernot

In the logs, you should see if the config was loaded correctly or not. Could you check? If you edited your file, it could be that some formatting is wrong now, and the extractor is still using the old version of the config 

View original
Did this topic help you find an answer to your question?

7 replies

Pierre  Pernot
Seasoned Practitioner
  • Seasoned Practitioner
  • 15 replies
  • January 6, 2023

Hi,
The DB extractor allows for running many queries, did you have a look at the logs ? Maybe there is something wrong with the first queries ? (setting your logger to DEBUG will probably give you more insights)


Forum|alt.badge.img+5

Hi @Pierre Pernot ,

I’m keeping the destination of both queries as some. Even setting the logger to DEBUG, I’m unable to get any other information.

 


Pierre  Pernot
Seasoned Practitioner
  • Seasoned Practitioner
  • 15 replies
  • Answer
  • January 6, 2023

In the logs, you should see if the config was loaded correctly or not. Could you check? If you edited your file, it could be that some formatting is wrong now, and the extractor is still using the old version of the config 


  • Seasoned Practitioner
  • 223 replies
  • January 13, 2023

Hi @Prashant Chauhan, just wondering if you were able to sort out your issue?


Forum|alt.badge.img+5

Hi @Carin Meems ,

I’m still unable to resolve this.
Only the last-mentioned query is getting executed.

 


Pierre  Pernot
Seasoned Practitioner
  • Seasoned Practitioner
  • 15 replies
  • January 24, 2023

Hi,
In the configuration file you shared, I can’t see any dash ( “-” ). Can you please add a dash in front of each “name” argument ? 

In a YAML file, each list element must start with a dash. (https://docs.ansible.com/ansible/latest/reference_appendices/YAMLSyntax.html

queries:
  - name: query_one
    database: postgres
    query: >
      SELECT * FROM ITEM_PROPERTY WHERE PROPERTY_TYPE = "NAME"
    destination:
      database: "query_multiple"
      table: "raw_one"
    primary-key: "{ITEM_ID}-{PROPERTY_STRING}"
  - name: query_two
    database: postgres
    query: >
      SELECT * FROM ITEM_PROPERTY WHERE PROPERTY_TYPE = "PRODUCT"
    destination:
      database: "query_multiple"
      table: "raw_one"
    primary-key: "{ITEM_ID}-{PROPERTY_STRING}"

 

 

 

Please try and let us know.

Pierre


Forum|alt.badge.img+5

Hi @Pierre Pernot,

Thanks for your help.


Reply


Cookie Policy

We use cookies to enhance and personalize your experience. If you accept you agree to our full cookie policy. Learn more about our cookies.

 
Cookie Settings