Data Engineer Basics - Integrations


Userlevel 2
Badge +5

Welcome to the Data Engineer Basics - Integrations!

This discussion is dedicated to help learners of the Data Engineer Basics – Integrations learning path succeed. If you’re struggling any of the courses in this learning path, post a comment with the challenge you’re facing. You can also post your own tips and respond to fellow learners’ questions. Cognite Academy’s instructors are also here to help.


43 replies

Userlevel 3
Badge +6

Hi @sunnyfufu! Can you send me a link to the course you are doing and which step you get this message?

 

 

Badge +3

Hi @Miguel Rosado and @Toye ,

Are you still having problems creating assets? 

Hello!
As Toye mentioned a couple of days ago, that is correct.
WHen Creating an asset, it would show the same permissions error. If i am to guess, this probably applies to creating any kind of item from the SDK (Since the same happened with data sets)
Here are the screenshots after running my code

The error reads:

CogniteAPIError: Resource not found. This may also be due to insufficient access rights. | code: 403 | X-Request-ID: ec318c97-4c11-94e3-b268-1522f4c88b22The API Failed to process some items.Successful (2xx): []Unknown (5xx): []Failed (4xx): ['worldMRosado_dataset']

 

I have run into this problem today. any update on how to fix it?

Hi! I’m currently following the Extractor-utils Library for Cognite Python SDK course and encountered an error during the ‘Main Function’ video. I ran the command ‘poetry run csv_extractor example_config.yaml’ as instructed, but got an error that no .env file was found to handle the variables in the example_config.yaml file (screenshot attached).

Two things to note that may be related: 

  1. When I followed the first video for starting a new project and ran the pip install cognite-extractor-manager command, I got errors that the pre-commit, black, and isort commands could not be found (screenshot attached), so I redid the steps and pip installed those 3 packages first before running installing the cogex manager and was then able to proceed with the cogex init process.
  2. I am using PyCharm but the video uses an older version, so it’s possible I may have settings incorrectly configured when I created the new project.

 

Thank you!

Badge +1

This is the one I created: 5060425428066370

Userlevel 3
Badge +6

Thank you so much for letting us know @Lixiang Sun ! I’ll have to forward this to one of our engineers to dig into this, do you know the data set id of the data set you created?

Badge +1

@Sofie Haug I found that if I use this dataset id: 1399248851345249, then this line will be successful.

root_asset = client.assets.create(Asset(name='world', data_set_id=data_set_id))

But if I use a dataset id that is created by me (res = c.data_sets.create(…...)), then the same line will give me the error (the same as I reported in the first place):

cognite.client.exceptions.CogniteAPIError: Resource not found. This may also be due to insufficient access rights. | code: 403 | X-Request-ID: 0d29c074-6166-9fa8-ba5e-cd38944ec5aa

How come I cannot create assets underneath the dataset I just created?

 

 

 

Userlevel 3
Badge +6

@Lixiang Sun if the line above was successful, then the next line

c.data_sets.list()

should show all data sets

Userlevel 3
Badge +6

@Lixiang Sun are you getting a successful log in for the first part of notebook number 2 (2_List_Search_Retrieve)?
 

c = cauth.create_cognite_client() 
Badge +1

@Sofie Haug I tried both create_update_insert and solution - Hands on Exercise notebooks. The create asset statement always fail with the same error..

 

Userlevel 3
Badge +6

Hi @Lixiang Sun!

In which notebook are you? Could you try to clear the history and re-run?

 

Sofie

Badge +1

Looks there is a permission issue completing the hands on lab in da-basics project using Python SDK.

This call failed with permission error:

root_asset = client.assets.create(Asset(name='world', data_set_id=data_set_id))

cognite.client.exceptions.CogniteAPIError: Resource not found. This may also be due to insufficient access rights. | code: 403 | X-Request-ID: 0d29c074-6166-9fa8-ba5e-cd38944ec5aa

I saw other people faced the same problem. Please help?

Thanls,

Lixiang

 

Userlevel 1
Badge +1

Hi @Shaileshkumar Od ,

I created a ticket for this issue to get more details and to troubleshoot further into this. Please 7451 check your email and respond to the questions asked in this email thread. Please let me know if you haven’t received or are unable to respond to the ticket created.

Regards,
Shashan

Userlevel 3

Hi @Shaileshkumar Od

I have created a ticket for our Support team for the issue you are facing. They’ll help you further. 

Best,

Carin

Badge +1

 

 

Hi Delisha

Code worked but got following errors

 

Request Id: ce98e14a-d3f3-4a3f-8ed6-c664ed182500
Correlation Id: 923287ac-ed8b-40be-af39-56c0afc44133
Timestamp: 2023-01-14T04:50:51Z
Message: AADSTS650057: Invalid resource. The client has requested access to a resource which is not listed in the requested permissions in the client's application registration. Client app ID: 1b90ede3-271e-401b-81a0-a4d52bea3273(OID-Api). Resource value from request: https://westeurope-1.cognitedata.com. Resource app ID: 209cbd1f-df92-4fb1-8e30-054812586bc9. List of valid resources from app registration: 00000003-0000-0000-c000-000000000000, 7cabb796-bb66-4af2-b8e4-17a5c3f68ac4.

 

Request Id: 0844c0ea-024f-45c8-acb9-12da07bb4a00
Correlation Id: 7a828b43-02f3-422a-a036-f130e47c89eb
Timestamp: 2023-01-14T04:40:42Z
Message: AADSTS500011: The resource principal named https://europe2.cognitedata.com was not found in the tenant named Cognite Hub. This can happen if the application has not been installed by the administrator of the tenant or consented to by any user in the tenant. You might have sent your authentication request to the wrong tenant.

 

Thank You

Shailesh Od

Badge +1

Hello Delisha

Thank you for the help, one more question,  how to get variables, I am struggling to get correct  variables here, I tried different variables  but they are not working. 

 

Thank You

Shailesh od

Userlevel 1
Badge +1

Hi @Shaileshkumar Od 

Could you please refer to this code here and try to authenticate accordingly and let us know if this sorts your issue here? Also for further information, please refer to the latest python SDK documentation here.

Best regards,

Dilesha

Badge +1

Hi 

I am new to Data engineer topic, i tried to connect database but unable to do that. using following variable. Please help me  connecting data source.

 

TENANT_ID = '48d5043c-cf70-4c49-881c-c638f5796997'CLIENT_ID = '1b90ede3-271e-401b-81a0-a4d52bea3273'CDF_CLUSTER = 'westeurope-1' COGNITE_PROJECT = 'publicdata'SCOPES = [f'https://{CDF_CLUSTER}.cognitedata.com/.default']AUTHORITY_HOST_URI = 'https://login.microsoftonline.com'AUTHORITY_URI = AUTHORITY_HOST_URI + '/' + TENANT_ID
Userlevel 2

Hi again, very strange I must say, we have checked permissions on our end and can not find the issue. Can you run the notebook 1_Authentication.ipynb until the cell with the code:

client.iam.token.inspect()

Then paste all the output into a simple txt file and post it here? Thank you for your patience and cooperation.

Badge +1

Hi @Miguel Rosado and @Toye ,

Are you still having problems creating assets? 

Hello!
As Toye mentioned a couple of days ago, that is correct.
WHen Creating an asset, it would show the same permissions error. If i am to guess, this probably applies to creating any kind of item from the SDK (Since the same happened with data sets)
Here are the screenshots after running my code

The error reads:

CogniteAPIError: Resource not found. This may also be due to insufficient access rights. | code: 403 | X-Request-ID: ec318c97-4c11-94e3-b268-1522f4c88b22The API Failed to process some items.Successful (2xx): []Unknown (5xx): []Failed (4xx): ['worldMRosado_dataset']

 

Userlevel 2

Hi @Miguel Rosado and @Toye ,

Are you still having problems creating assets? 

Badge +1

Hi Toye, 
Thanks for posting the question here.
 Please can you mention in which lesson of the course you are facing the problem, so I will test the code at my end and investigate in details. 

Regards 
Kumar

Hello Rahul. This is for the ‘Learn to Use the Cognite Python SDK’. The Github repo given is the following:
https://github.com/cognitedata/using-cognite-python-sdk

and the problem ocurrs when trying to complete the ‘Hands on excercise’.

Userlevel 2

Hi @jaydeep ,

Are you using poetry to create the venv? I recommend trying to uninstall the venv and do a reinstall. 

Userlevel 2
Badge +6

Hi Jaydeep,

I have forward your problem to the our data engineers team and they are investigating the  problem and you will get back to you.

Regards 
Kumar

 

Badge +6

As part of  “Learn to Use the Cognite Python SDK” course in Data integration , while running “test_Authentication.ipynb” notebook , I am getting below error on line “c.login.status()”

 

--> 703 httplib_response = self._make_request( 704 conn, 705 method, 706 url, 707 timeout=timeout_obj, 708 body=body, 709 headers=headers, 710 chunked=chunked, 711 ) 713 # If we're going to release the connection in ``finally:``, then 714 # the response doesn't need to know about the connection. Otherwise 715 # it will also try to release it and we'll have a double-release 716 # mess. File c:\Users\j.subhash.parandekar\AppData\Local\Programs\Python\Python38\lib\site-packages\urllib3\connectionpool.py:386, in HTTPConnectionPool._make_request(self, conn, method, url, timeout, chunked, **httplib_request_kw) 385 try: --> 386 self._validate_conn(conn) 387 except (SocketTimeout, BaseSSLError) as e: 388 # Py2 raises this as a BaseSSLError, Py3 raises it as socket timeout. File c:\Users\j.subhash.parandekar\AppData\Local\Programs\Python\Python38\lib\site-packages\urllib3\connectionpool.py:1042, in HTTPSConnectionPool._validate_conn(self, conn)

...

150 raise CogniteConnectionRefused from e --> 151 raise CogniteConnectionError from e 152 raise e CogniteConnectionError:

Userlevel 2
Badge +6

Hi Toye, 
Thanks for posting the question here.
 Please can you mention in which lesson of the course you are facing the problem, so I will test the code at my end and investigate in details. 

Regards 
Kumar

Reply