Solved

Retrieve dataframe from RAW table

  • 30 July 2023
  • 3 replies
  • 75 views
E
Rank
Badge +2

I uploaded a CSV into a raw table in cognite data fusion. Now I have seen the raw table and see 200 numeric columns and 2 text columns are present. When i come to SDK and retrieve the rows from the same table as dataframe, it is showing 199 columns as numeric and 3 text columns. there was one specific column '0NA' which was originally a numeric column now shows the datatype as 'object' instead of 'float64'. I checked and there are no null values and no other NaN characters and it's just decimal values all across.

 

When I fetch all numeric columns directly from CSV, I see the column as float-64 but after importing that CSV into RAW table, it is doing something else. 

I tried to see the unique values in the column and found these two as string,I suppose. 

In CSV , they were stored as number but when uploading to RAW table, it does convert into something like this. 

Please help if RAW table cannot accommodate values with more than 10E142 values such as these?. 

How to handle this scenario.

'24100000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000.000000000000000000000000000000' '3950000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000.000000000000000000000000000000'

 

 

icon

Best answer by Dilesha Kodithuwakku 31 July 2023, 13:02

View original

3 replies

C
Rank
Userlevel 1
Badge

Hi @eashwar11,

If I understood your issue correctly, you want to upload values more than 10E142 to the raw table.

If so, could you please try to format the cell to text field and upload it to raw table.

Thank you, and have a nice day.


Best regards,
Chyrus

E
Rank
Badge +2

Thanks @Chyrus Ramesh for the response. CDF automatically takes the field as numeric only (#) in the raw table. At least that is what i observed after I uploaded from CSV.

 

But when i run this command and fetch it as a data frame and try to see the total sum of this column, it says it has str values. Ideally, if it is stored as # columns then it should be a numeric column in dataframe too, why is the retrieve-dataframe converts the long numeric value into an object in a dataframe. 

 

Attaching the csv for your reference. You can try in any CDF project and check it out and do retrieve as a dataframe in a notebook and see the result. 

from cognite.client import CogniteClient
c = CogniteClient()
df = c.raw.rows.retrieve_dataframe("database-name", "table-name", limit=None)

when you try to check the following, it should not give any error if it was properly retrieved as a numeric column. 

df[‘0NA’].sum()

 

D
Rank
Userlevel 1
Badge +1

Hi Eashwar,

We have created a support ticket for this issue. We will further follow up and continue the communication on this issue in the support ticket.

Have a nice day!

Best regards,

Dilesha

Reply


Terms & ConditionsCookie Settings