Retrieve pandas dataframe - RAW tables

  • 14 September 2023
  • 4 replies

Badge +2

There is a problem in fetching raw tables records through SDK. This was working till last week and something has changed now. 



Now getting this error below. Please advise.

MissingSchema Traceback (most recent call last)
Cell In[6], line 1
----> 1 client.raw.rows.retrieve_dataframe(db_name="Eashwar_MOTDB-db",table_name="lp_input",limit=None,columns=None)

File /lib/python3.11/site-packages/cognite/client/_api/, in RawRowsAPI.retrieve_dataframe(self, db_name, table_name, min_last_updated_time, max_last_updated_time, columns, limit)
514 """`Retrieve rows in a table as a pandas dataframe. <>`_
516 Rowkeys are used as the index.
535 >>> df = c.raw.rows.retrieve_dataframe("db1", "t1", limit=5)
536 """
537 pd = cast(Any, local_import("pandas"))
--> 538 rows = self.list(db_name, table_name, min_last_updated_time, max_last_updated_time, columns, limit)
539 idx = [r.key for r in rows]
540 cols = [r.columns for r in rows]

File /lib/python3.11/site-packages/cognite/client/_api/, in RawRowsAPI.list(self, db_name, table_name, min_last_updated_time, max_last_updated_time, columns, limit)
552 """`List rows in a table. <>`_
554 Args:
585 ... row_list # do something with the rows
586 """
587 if is_unlimited(limit):
--> 588 cursors = self._get(
589 url_path=utils._auxiliary.interpolate_and_url_encode(
590 "/raw/dbs/{}/tables/{}/cursors", db_name, table_name
591 ),
592 params={
593 "minLastUpdatedTime": min_last_updated_time,
594 "maxLastUpdatedTime": max_last_updated_time,
595 "numberOfCursors": self._config.max_workers,
596 },
597 ).json()["items"]
598 else:
599 cursors = [None]

File /lib/python3.11/site-packages/cognite/client/, in APIClient._get(self, url_path, params, headers)
131 def _get(
132 self, url_path: str, params: dict[str, Any] | None = None, headers: dict[str, Any] | None = None
133 ) -> Response:
--> 134 return self._do_request("GET", url_path, params=params, headers=headers, timeout=self._config.timeout)

File /lib/python3.11/site-packages/cognite/client/, in APIClient._do_request(self, method, url_path, accept, **kwargs)
182 kwargs.setdefault("allow_redirects", False)
184 if is_retryable:
--> 185 res = self._http_client_with_retry.request(method=method, url=full_url, **kwargs)
186 else:
187 res = self._http_client.request(method=method, url=full_url, **kwargs)

File /lib/python3.11/site-packages/cognite/client/, in HTTPClient.request(self, method, url, **kwargs)
111 while True:
112 try:
--> 113 res = self._do_request(method=method, url=url, **kwargs)
114 last_status = res.status_code
115 retry_tracker.status += 1

File /lib/python3.11/site-packages/cognite/client/, in HTTPClient._do_request(self, method, url, **kwargs)
166 raise CogniteConnectionRefused from e
167 raise CogniteConnectionError from e
--> 168 raise e

File /lib/python3.11/site-packages/cognite/client/, in HTTPClient._do_request(self, method, url, **kwargs)
136 """requests/urllib3 adds 2 or 3 layers of exceptions on top of built-in networking exceptions.
138 Sometimes the appropriate built-in networking exception is not in the context, sometimes the requests
146 Returns:
147 requests.Response: No description."""
148 try:
--> 149 res = self.session.request(method=method, url=url, **kwargs)
150 return res
151 except Exception as e:

File /lib/python3.11/site-packages/requests/, in Session.request(self, method, url, params, data, headers, cookies, files, auth, timeout, allow_redirects, proxies, hooks, stream, verify, cert, json)
562 # Create the Request.
563 req = Request(
564 method=method.upper(),
565 url=url,
573 hooks=hooks,
574 )
--> 575 prep = self.prepare_request(req)
577 proxies = proxies or {}
579 settings = self.merge_environment_settings(
580 prep.url, proxies, stream, verify, cert
581 )

File /lib/python3.11/site-packages/requests/, in Session.prepare_request(self, request)
483 auth = get_netrc_auth(request.url)
485 p = PreparedRequest()
--> 486 p.prepare(
487 method=request.method.upper(),
488 url=request.url,
489 files=request.files,
491 json=request.json,
492 headers=merge_setting(
493 request.headers, self.headers, dict_class=CaseInsensitiveDict
494 ),
495 params=merge_setting(request.params, self.params),
496 auth=merge_setting(auth, self.auth),
497 cookies=merged_cookies,
498 hooks=merge_hooks(request.hooks, self.hooks),
499 )
500 return p

File /lib/python3.11/site-packages/requests/, in PreparedRequest.prepare(self, method, url, headers, files, data, params, auth, cookies, hooks, json)
365 """Prepares the entire request with the given parameters."""
367 self.prepare_method(method)
--> 368 self.prepare_url(url, params)
369 self.prepare_headers(headers)
370 self.prepare_cookies(cookies)

File /lib/python3.11/site-packages/requests/, in PreparedRequest.prepare_url(self, url, params)
436 raise InvalidURL(*e.args)
438 if not scheme:
--> 439 raise MissingSchema(
440 f"Invalid URL {url!r}: No scheme supplied. "
441 f"Perhaps you meant https://{url}?"
442 )
444 if not host:
445 raise InvalidURL(f"Invalid URL {url!r}: No host supplied")

MissingSchema: Invalid URL '/api/v1/projects/undefined/raw/dbs/Eashwar_MOTDB-db/tables/lp_input/cursors': No scheme supplied. Perhaps you meant https:///api/v1/projects/undefined/raw/dbs/Eashwar_MOTDB-db/tables/lp_input/cursors?


4 replies

Hi @eashwar11, it appears you asked a very similar question on this issue here. As can be seen in the error you quoted, the part that sticks out is the “undefined” bit (as Håkon wrote, one would expect this to be something like company-prod, company-dev, etc):

MissingSchema: Invalid URL '/api/v1/projects/undefined/raw/dbs/Eashwar_MOTDB-db/tables/lp_input/cursors': No scheme supplied. Perhaps you meant https:///api/v1/projects/undefined/raw/dbs/Eashwar_MOTDB-db/tables/lp_input/cursors?

Can you check your project details and if you are reading environment variables, can you check those are being read correctly? This might happen if the variables are None/not being read properly. Situation could be that you are using a library for reading .env files but not specifying a specific version and so libraries can be subject to regression in behaviour or new bugs are introduced. So maybe checking your requirements.txt file (if running this in Cognite Functions) is worth taking a look at too. If you know which versions were installed when it was working, then it might be worth trying to use that specific version.

Badge +2

Hi @HaydenH 

When I run the same piece of code in my VS code and then it runs successfully. 

This fails when I run the code in Jupyter notebooks in CDF online. I think there is a problem in the CDF Jupyter notebooks.

If there is a problem in APIs or variables, then it must fail and throw the same error in VS code as well.

In local VS code, I pass the parameters such as client secrets, tenantid etc and create the CogniteClient object. 

There seems to be a problem in the Jupyter notebook where it doesnt create the client object in the right manner for the CDF project it is associated with.

Can you please print/report back the versions of the Cognite Python SDK for each of those environments @eashwar11? If I recall correctly, the Jupyter Notebook environment in Fusion always has the latest version of the SDK installed.

Userlevel 3

Hi @eashwar11, do you need any further help?