Data Engineer Basics - Integrations

Hi @ikequarsh 

After you

pip install cognite-extractor-manager

and initialize the extractor project

cogex init

During this process poetry sets up a virtual environment and installs python modules into the venv. But you need to set / activate that virtual environment in your IDE, if it happens automatically or not I'm not sure. But just check that the python interpreter for your IDE’s project is set to: “your-extractor-directory”/venv/bin/python 

more native Poetry support was added to Pycharm last year(2021) so there can be some differences there now than what the video from what the course says; if it is pycharm you are using, but I'm not that familiar with Pycharm, cause I mainly use VS code or Spyder, But it should be simmilar. 

Then I hope you can find and import all modules and objects inside your pycharm project, that poetry installed in to the venv during the “cogex init” process. 

If you are not familiar with poetry I suggest looking it up. because it can be seen as the npm of python. that both sets up virtual environments (isolation) and handles package/module installing and versioning (dependency resolver). which previous needed several applications like venv or virtualenv and  pip. but poetry does all this service. 

Hope this helps you solve your issue. 

Hi Toye and Miguel, 

We have fixed the problem at our end and everything is working fine at my end.

Please can you run the code again and check if the problem is solved at your end.

Regards 
Kumar

 

Hi Toye, 
Thanks for posting the question here.
 Please can you mention in which lesson of the course you are facing the problem, so I will test the code at my end and investigate in details. 

Regards 
Kumar

Hi @Miguel Rosado and @Toye ,

Are you still having problems creating assets? 

Hi again, very strange I must say, we have checked permissions on our end and can not find the issue. Can you run the notebook 1_Authentication.ipynb until the cell with the code:

client.iam.token.inspect()

Then paste all the output into a simple txt file and post it here? Thank you for your patience and cooperation.

Hi @Shaileshkumar Od 

Could you please refer to this code here and try to authenticate accordingly and let us know if this sorts your issue here? Also for further information, please refer to the latest python SDK documentation here.

Best regards,

Dilesha

Hi @Mirtunjay Kumar,

Can you try again now? The assessment should be unlocked now and the PostegreSQL course completed. 

Let us know if you have any more questions. 

Best regards,

Maritsa

Hi @ikequarsh ,
I’m Enikö from Cognite Academy. First, congratulations on your Cognite Date Fusion Fundamentals certificate!

I see you’re working through the Extractor-utils Library for Cognite Python SDK course. I suggest you watch the Defining a Code Schema video that talks about defining the destination.

Let me know if the video has answered your question!
Happy learning!

Enikö

Hi @ikequarsh,

You will find the ‘Defining a Code Schema’ video in the ‘Extractor-utils Library for Cognite Python SDK’ course. As you are inside the course, you can go to the ‘Build Your Own Python-based Extractor - Part (1 of 2)’ lesson, where you need to click on the ‘Defining a Code Schema' tab; on doing so, it will take you to the video. 🙂
 

Please find the link below for easy navigation:
https://learn.cognite.com/path/data-engineer-basics-integrations/extractor-utils-library-for-cognite-python-sdk/114686

Let us know if you have any queries. Happy learning!
Nimesh Madandas

Hello @Stig Harald Gustavsen,

Thanks for your response. I can use VS Code and is it possible to guide on how to setup with Cogex? 

I think your training materials need to be revisited since it expects user to have knowledge with Cogex.

Let me hear from you. 

Isaac

Hi @ikequarsh 

Cogex is a command line tool that you get when you pip install cognite-extractor-manager. 
Hence CogEx short for Cognite Extractor. 

It should be the same way of setting it up.

Cogex init, prefills a directory with files, and makes virtual environment, and installs into the virtual enviroment the python modules specific to the extractors base project. and you just activate / set up the python interpreter inside your IDE to the virtual environment created  by poetry (“your-extractor-directory”/venv/bin/python ), that was ran when you ran cogex init. but it would be the same way if you manually setup a virtual environment and added the files and ran pip install requirement.txt as when you work on previous python project templates / boilerplates / forks before more and more python projects went over to poetry for the dependency and virtual environment manager. but now the python modules / libraries are in pyproject.toml file instead of being pip freezed into requirements.txt

As they say in the course video on “importing the new extractor project” you don’t have to use Pycharm for the extractor projects, but you use any IDE/text editor you want for this project. just set up your environment. I personally would use VS code and Windows Subsystem for Linux (WSL), but that is my preference. Sometimes working with virtual environments in windows can be a hassle, but that is where poetry comes in and try to help us.

Are you familiar with working with virtual environment in Python? 

Hope we get one step closer to solving your issue Isaac :)

Hi Miguel, 

Thanks for posting you question here.
Please can add the snapshot of the error/problem and then it will be easy for us to track down the problem.

Waiting for your reply.

Regards 
Kumar

 

As part of  “Learn to Use the Cognite Python SDK” course in Data integration , while running “test_Authentication.ipynb” notebook , I am getting below error on line “c.login.status()”

--> 703 httplib_response = self._make_request( 704 conn, 705 method, 706 url, 707 timeout=timeout_obj, 708 body=body, 709 headers=headers, 710 chunked=chunked, 711 ) 713 # If we're going to release the connection in ``finally:``, then 714 # the response doesn't need to know about the connection. Otherwise 715 # it will also try to release it and we'll have a double-release 716 # mess. File c:\Users\j.subhash.parandekar\AppData\Local\Programs\Python\Python38\lib\site-packages\urllib3\connectionpool.py:386, in HTTPConnectionPool._make_request(self, conn, method, url, timeout, chunked, **httplib_request_kw) 385 try: --> 386 self._validate_conn(conn) 387 except (SocketTimeout, BaseSSLError) as e: 388 # Py2 raises this as a BaseSSLError, Py3 raises it as socket timeout. File c:\Users\j.subhash.parandekar\AppData\Local\Programs\Python\Python38\lib\site-packages\urllib3\connectionpool.py:1042, in HTTPSConnectionPool._validate_conn(self, conn)

...

150 raise CogniteConnectionRefused from e --> 151 raise CogniteConnectionError from e 152 raise e CogniteConnectionError: