Skip to main content

using map and distinct in query


Hi, 

I am trying to transform data from Raw to timeseries in CDF. I made a query and everything seems ok, but when I use distinct to check the external id uniqueness, I get error message:

"Cannot have map type columns in DataFrame which calls set operations(intersect, except, etc.), but the type of column metadata is map<string,string>;

 

I don’t know why I get this message, Isn’t it possible to use distinct when wi do MAP or To_Metadata mapping? 

2 replies

Forum|alt.badge.img

Hi,

Unfortunately distinct can’t work on map.
I don’t see your complete query but a workaround that you can be inspired on is:

select map(...) as metadata, .... from (select distinct * from my.table)

 


Thanks a lot for your answer, I also found instead of map I can use cast_to_string with distinct  first for testing and then change to map


Reply


Cookie Policy

We use cookies to enhance and personalize your experience. If you accept you agree to our full cookie policy. Learn more about our cookies.

 
Cookie Settings