Skip to main content

Hi, 

I am trying to transform data from Raw to timeseries in CDF. I made a query and everything seems ok, but when I use distinct to check the external id uniqueness, I get error message:

"Cannot have map type columns in DataFrame which calls set operations(intersect, except, etc.), but the type of column metadata is map<string,string>;

 

I don’t know why I get this message, Isn’t it possible to use distinct when wi do MAP or To_Metadata mapping? 

Hi,

Unfortunately distinct can’t work on map.
I don’t see your complete query but a workaround that you can be inspired on is:

select map(...) as metadata, .... from (select distinct * from my.table)

 


Thanks a lot for your answer, I also found instead of map I can use cast_to_string with distinct  first for testing and then change to map


Reply