Skip to main content
Solved

Retrieve pandas dataframe - RAW tables


Forum|alt.badge.img+2

There is a problem in fetching raw tables records through SDK. This was working till last week and something has changed now. 

 

client.raw.rows.retrieve_dataframe(db_name="Eashwar_MOTDB-db",table_name="lp_input",limit=None)

Now getting this error below. Please advise.

--------------------------------------------------------------------------
MissingSchema                             Traceback (most recent call last)
Cell In[6], line 1
----> 1 client.raw.rows.retrieve_dataframe(db_name="Eashwar_MOTDB-db",table_name="lp_input",limit=None,columns=None)

File /lib/python3.11/site-packages/cognite/client/_api/raw.py:538, in RawRowsAPI.retrieve_dataframe(self, db_name, table_name, min_last_updated_time, max_last_updated_time, columns, limit)
    514 """`Retrieve rows in a table as a pandas dataframe. <https://developer.cognite.com/api#tag/Raw/operation/getRows>`_
    515 
    516 Rowkeys are used as the index.
   (...)
    535         >>> df = c.raw.rows.retrieve_dataframe("db1", "t1", limit=5)
    536 """
    537 pd = cast(Any, local_import("pandas"))
--> 538 rows = self.list(db_name, table_name, min_last_updated_time, max_last_updated_time, columns, limit)
    539 idx = [r.key for r in rows]
    540 cols = [r.columns for r in rows]

File /lib/python3.11/site-packages/cognite/client/_api/raw.py:588, in RawRowsAPI.list(self, db_name, table_name, min_last_updated_time, max_last_updated_time, columns, limit)
    552 """`List rows in a table. <https://developer.cognite.com/api#tag/Raw/operation/getRows>`_
    553 
    554 Args:
   (...)
    585         ...     row_list # do something with the rows
    586 """
    587 if is_unlimited(limit):
--> 588     cursors = self._get(
    589         url_path=utils._auxiliary.interpolate_and_url_encode(
    590             "/raw/dbs/{}/tables/{}/cursors", db_name, table_name
    591         ),
    592         params={
    593             "minLastUpdatedTime": min_last_updated_time,
    594             "maxLastUpdatedTime": max_last_updated_time,
    595             "numberOfCursors": self._config.max_workers,
    596         },
    597     ).json()["items"]
    598 else:
    599     cursors = [None]

File /lib/python3.11/site-packages/cognite/client/_api_client.py:134, in APIClient._get(self, url_path, params, headers)
    131 def _get(
    132     self, url_path: str, params: dict[str, Any] | None = None, headers: dict[str, Any] | None = None
    133 ) -> Response:
--> 134     return self._do_request("GET", url_path, params=params, headers=headers, timeout=self._config.timeout)

File /lib/python3.11/site-packages/cognite/client/_api_client.py:185, in APIClient._do_request(self, method, url_path, accept, **kwargs)
    182 kwargs.setdefault("allow_redirects", False)
    184 if is_retryable:
--> 185     res = self._http_client_with_retry.request(method=method, url=full_url, **kwargs)
    186 else:
    187     res = self._http_client.request(method=method, url=full_url, **kwargs)

File /lib/python3.11/site-packages/cognite/client/_http_client.py:113, in HTTPClient.request(self, method, url, **kwargs)
    111 while True:
    112     try:
--> 113         res = self._do_request(method=method, url=url, **kwargs)
    114         last_status = res.status_code
    115         retry_tracker.status += 1

File /lib/python3.11/site-packages/cognite/client/_http_client.py:168, in HTTPClient._do_request(self, method, url, **kwargs)
    166         raise CogniteConnectionRefused from e
    167     raise CogniteConnectionError from e
--> 168 raise e

File /lib/python3.11/site-packages/cognite/client/_http_client.py:149, in HTTPClient._do_request(self, method, url, **kwargs)
    136 """requests/urllib3 adds 2 or 3 layers of exceptions on top of built-in networking exceptions.
    137 
    138 Sometimes the appropriate built-in networking exception is not in the context, sometimes the requests
   (...)
    146 Returns:
    147     requests.Response: No description."""
    148 try:
--> 149     res = self.session.request(method=method, url=url, **kwargs)
    150     return res
    151 except Exception as e:

File /lib/python3.11/site-packages/requests/sessions.py:575, in Session.request(self, method, url, params, data, headers, cookies, files, auth, timeout, allow_redirects, proxies, hooks, stream, verify, cert, json)
    562 # Create the Request.
    563 req = Request(
    564     method=method.upper(),
    565     url=url,
   (...)
    573     hooks=hooks,
    574 )
--> 575 prep = self.prepare_request(req)
    577 proxies = proxies or {}
    579 settings = self.merge_environment_settings(
    580     prep.url, proxies, stream, verify, cert
    581 )

File /lib/python3.11/site-packages/requests/sessions.py:486, in Session.prepare_request(self, request)
    483     auth = get_netrc_auth(request.url)
    485 p = PreparedRequest()
--> 486 p.prepare(
    487     method=request.method.upper(),
    488     url=request.url,
    489     files=request.files,
    490     data=request.data,
    491     json=request.json,
    492     headers=merge_setting(
    493         request.headers, self.headers, dict_class=CaseInsensitiveDict
    494     ),
    495     params=merge_setting(request.params, self.params),
    496     auth=merge_setting(auth, self.auth),
    497     cookies=merged_cookies,
    498     hooks=merge_hooks(request.hooks, self.hooks),
    499 )
    500 return p

File /lib/python3.11/site-packages/requests/models.py:368, in PreparedRequest.prepare(self, method, url, headers, files, data, params, auth, cookies, hooks, json)
    365 """Prepares the entire request with the given parameters."""
    367 self.prepare_method(method)
--> 368 self.prepare_url(url, params)
    369 self.prepare_headers(headers)
    370 self.prepare_cookies(cookies)

File /lib/python3.11/site-packages/requests/models.py:439, in PreparedRequest.prepare_url(self, url, params)
    436     raise InvalidURL(*e.args)
    438 if not scheme:
--> 439     raise MissingSchema(
    440         f"Invalid URL {url!r}: No scheme supplied. "
    441         f"Perhaps you meant https://{url}?"
    442     )
    444 if not host:
    445     raise InvalidURL(f"Invalid URL {url!r}: No host supplied")

MissingSchema: Invalid URL '/api/v1/projects/undefined/raw/dbs/Eashwar_MOTDB-db/tables/lp_input/cursors': No scheme supplied. Perhaps you meant https:///api/v1/projects/undefined/raw/dbs/Eashwar_MOTDB-db/tables/lp_input/cursors?

 

Best answer by HaydenH

Hi @eashwar11, it appears you asked a very similar question on this issue here. As can be seen in the error you quoted, the part that sticks out is the “undefined” bit (as Håkon wrote, one would expect this to be something like company-prod, company-dev, etc):

MissingSchema: Invalid URL '/api/v1/projects/undefined/raw/dbs/Eashwar_MOTDB-db/tables/lp_input/cursors': No scheme supplied. Perhaps you meant https:///api/v1/projects/undefined/raw/dbs/Eashwar_MOTDB-db/tables/lp_input/cursors?

Can you check your project details and if you are reading environment variables, can you check those are being read correctly? This might happen if the variables are None/not being read properly. Situation could be that you are using a library for reading .env files but not specifying a specific version and so libraries can be subject to regression in behaviour or new bugs are introduced. So maybe checking your requirements.txt file (if running this in Cognite Functions) is worth taking a look at too. If you know which versions were installed when it was working, then it might be worth trying to use that specific version.

View original

  • Practitioner
  • September 14, 2023

Hi @eashwar11, it appears you asked a very similar question on this issue here. As can be seen in the error you quoted, the part that sticks out is the “undefined” bit (as Håkon wrote, one would expect this to be something like company-prod, company-dev, etc):

MissingSchema: Invalid URL '/api/v1/projects/undefined/raw/dbs/Eashwar_MOTDB-db/tables/lp_input/cursors': No scheme supplied. Perhaps you meant https:///api/v1/projects/undefined/raw/dbs/Eashwar_MOTDB-db/tables/lp_input/cursors?

Can you check your project details and if you are reading environment variables, can you check those are being read correctly? This might happen if the variables are None/not being read properly. Situation could be that you are using a library for reading .env files but not specifying a specific version and so libraries can be subject to regression in behaviour or new bugs are introduced. So maybe checking your requirements.txt file (if running this in Cognite Functions) is worth taking a look at too. If you know which versions were installed when it was working, then it might be worth trying to use that specific version.


Forum|alt.badge.img+2

Hi @HaydenH 

When I run the same piece of code in my VS code and then it runs successfully. 

This fails when I run the code in Jupyter notebooks in CDF online. I think there is a problem in the CDF Jupyter notebooks.

If there is a problem in APIs or variables, then it must fail and throw the same error in VS code as well.

In local VS code, I pass the parameters such as client secrets, tenantid etc and create the CogniteClient object. 

There seems to be a problem in the Jupyter notebook where it doesnt create the client object in the right manner for the CDF project it is associated with.


  • Practitioner
  • September 14, 2023

Can you please print/report back the versions of the Cognite Python SDK for each of those environments @eashwar11? If I recall correctly, the Jupyter Notebook environment in Fusion always has the latest version of the SDK installed.


  • Seasoned Practitioner
  • September 26, 2023

Hi @eashwar11, do you need any further help?


Reply


Cookie Policy

We use cookies to enhance and personalize your experience. If you accept you agree to our full cookie policy. Learn more about our cookies.

 
Cookie Settings