Skip to main content

using map and distinct in query

  • February 14, 2023
  • 2 replies
  • 133 views
M
Active

Hi, 

I am trying to transform data from Raw to timeseries in CDF. I made a query and everything seems ok, but when I use distinct to check the external id uniqueness, I get error message:

"Cannot have map type columns in DataFrame which calls set operations(intersect, except, etc.), but the type of column metadata is map<string,string>;

 

I don’t know why I get this message, Isn’t it possible to use distinct when wi do MAP or To_Metadata mapping? 

2 replies

V
Practitioner
Forum|alt.badge.img
  • Vu Hai Nguyen
  • Practitioner
  • February 15, 2023

Hi,

Unfortunately distinct can’t work on map.
I don’t see your complete query but a workaround that you can be inspired on is:

select map(...) as metadata, .... from (select distinct * from my.table)

 

K
M
M
Active
  • maryamgholami
  • Author
  • Active
  • February 16, 2023

Thanks a lot for your answer, I also found instead of map I can use cast_to_string with distinct  first for testing and then change to map. 

V
Terms & ConditionsCookie SettingsAccessibility statement