using map and distinct in query

  • 14 February 2023
  • 2 replies
  • 64 views
M
Rank

Hi, 

I am trying to transform data from Raw to timeseries in CDF. I made a query and everything seems ok, but when I use distinct to check the external id uniqueness, I get error message:

"Cannot have map type columns in DataFrame which calls set operations(intersect, except, etc.), but the type of column metadata is map<string,string>;

 

I don’t know why I get this message, Isn’t it possible to use distinct when wi do MAP or To_Metadata mapping? 

2 replies

V
Rank
Userlevel 1
Badge

Hi,

Unfortunately distinct can’t work on map.
I don’t see your complete query but a workaround that you can be inspired on is:

select map(...) as metadata, .... from (select distinct * from my.table)

 

M
Rank

Thanks a lot for your answer, I also found instead of map I can use cast_to_string with distinct  first for testing and then change to map. 

Reply


Terms & ConditionsCookie Settings