Data Engineer Basics - Integrations

Hi again, very strange I must say, we have checked permissions on our end and can not find the issue. Can you run the notebook 1_Authentication.ipynb until the cell with the code:

client.iam.token.inspect()

Then paste all the output into a simple txt file and post it here? Thank you for your patience and cooperation.

Hi 

I am new to Data engineer topic, i tried to connect database but unable to do that. using following variable. Please help me  connecting data source.

TENANT_ID = '48d5043c-cf70-4c49-881c-c638f5796997'CLIENT_ID = '1b90ede3-271e-401b-81a0-a4d52bea3273'CDF_CLUSTER = 'westeurope-1' COGNITE_PROJECT = 'publicdata'SCOPES = [f'https://{CDF_CLUSTER}.cognitedata.com/.default']AUTHORITY_HOST_URI = 'https://login.microsoftonline.com'AUTHORITY_URI = AUTHORITY_HOST_URI + '/' + TENANT_ID

Hi @Shaileshkumar Od 

Could you please refer to this code here and try to authenticate accordingly and let us know if this sorts your issue here? Also for further information, please refer to the latest python SDK documentation here.

Best regards,

Dilesha

Hello Delisha

Thank you for the help, one more question,  how to get variables, I am struggling to get correct  variables here, I tried different variables  but they are not working. 

Thank You

Shailesh od

Hi Delisha

Code worked but got following errors

Request Id: ce98e14a-d3f3-4a3f-8ed6-c664ed182500
Correlation Id: 923287ac-ed8b-40be-af39-56c0afc44133
Timestamp: 2023-01-14T04:50:51Z
Message: AADSTS650057: Invalid resource. The client has requested access to a resource which is not listed in the requested permissions in the client's application registration. Client app ID: 1b90ede3-271e-401b-81a0-a4d52bea3273(OID-Api). Resource value from request: https://westeurope-1.cognitedata.com. Resource app ID: 209cbd1f-df92-4fb1-8e30-054812586bc9. List of valid resources from app registration: 00000003-0000-0000-c000-000000000000, 7cabb796-bb66-4af2-b8e4-17a5c3f68ac4.

Request Id: 0844c0ea-024f-45c8-acb9-12da07bb4a00
Correlation Id: 7a828b43-02f3-422a-a036-f130e47c89eb
Timestamp: 2023-01-14T04:40:42Z
Message: AADSTS500011: The resource principal named https://europe2.cognitedata.com was not found in the tenant named Cognite Hub. This can happen if the application has not been installed by the administrator of the tenant or consented to by any user in the tenant. You might have sent your authentication request to the wrong tenant.

Thank You

Shailesh Od

Hi @Shaileshkumar Od, 

I have created a ticket for our Support team for the issue you are facing. They’ll help you further. 

Best,

Carin

Hi @Shaileshkumar Od ,

I created a ticket for this issue to get more details and to troubleshoot further into this. Please 7451 check your email and respond to the questions asked in this email thread. Please let me know if you haven’t received or are unable to respond to the ticket created.

Regards,
Shashan

Looks there is a permission issue completing the hands on lab in da-basics project using Python SDK.

This call failed with permission error:

root_asset = client.assets.create(Asset(name='world', data_set_id=data_set_id))

cognite.client.exceptions.CogniteAPIError: Resource not found. This may also be due to insufficient access rights. | code: 403 | X-Request-ID: 0d29c074-6166-9fa8-ba5e-cd38944ec5aa

I saw other people faced the same problem. Please help?

Thanls,

Lixiang

Hi @Lixiang Sun!

In which notebook are you? Could you try to clear the history and re-run?

Sofie

@Sofie Haug I tried both create_update_insert and solution - Hands on Exercise notebooks. The create asset statement always fail with the same error..

@Lixiang Sun are you getting a successful log in for the first part of notebook number 2 (2_List_Search_Retrieve)?
 

c = cauth.create_cognite_client() 

@Lixiang Sun if the line above was successful, then the next line

c.data_sets.list()

should show all data sets

@Sofie Haug I found that if I use this dataset id: 1399248851345249, then this line will be successful.

root_asset = client.assets.create(Asset(name='world', data_set_id=data_set_id))

But if I use a dataset id that is created by me (res = c.data_sets.create(…...)), then the same line will give me the error (the same as I reported in the first place):

cognite.client.exceptions.CogniteAPIError: Resource not found. This may also be due to insufficient access rights. | code: 403 | X-Request-ID: 0d29c074-6166-9fa8-ba5e-cd38944ec5aa

How come I cannot create assets underneath the dataset I just created?

Thank you so much for letting us know @Lixiang Sun ! I’ll have to forward this to one of our engineers to dig into this, do you know the data set id of the data set you created?

This is the one I created: 5060425428066370

Hi! I’m currently following the Extractor-utils Library for Cognite Python SDK course and encountered an error during the ‘Main Function’ video. I ran the command ‘poetry run csv_extractor example_config.yaml’ as instructed, but got an error that no .env file was found to handle the variables in the example_config.yaml file (screenshot attached).

Two things to note that may be related: 

1. When I followed the first video for starting a new project and ran the pip install cognite-extractor-manager command, I got errors that the pre-commit, black, and isort commands could not be found (screenshot attached), so I redid the steps and pip installed those 3 packages first before running installing the cogex manager and was then able to proceed with the cogex init process.
2. I am using PyCharm but the video uses an older version, so it’s possible I may have settings incorrectly configured when I created the new project.

Thank you!

Hi @Miguel Rosado and @Toye ,

Are you still having problems creating assets? 

Hello!
As Toye mentioned a couple of days ago, that is correct.
WHen Creating an asset, it would show the same permissions error. If i am to guess, this probably applies to creating any kind of item from the SDK (Since the same happened with data sets)
Here are the screenshots after running my code

The error reads:

CogniteAPIError: Resource not found. This may also be due to insufficient access rights. | code: 403 | X-Request-ID: ec318c97-4c11-94e3-b268-1522f4c88b22The API Failed to process some items.Successful (2xx): []Unknown (5xx): []Failed (4xx): ['worldMRosado_dataset']

 

I have run into this problem today. any update on how to fix it?

Hi @sunnyfufu! Can you send me a link to the course you are doing and which step you get this message?

Hi I’m new at python and I’m realizing that I’m a little bit lost with this course right now. 

Is there something to understand better what I need to do correctly maybe it’s because there’s to many ways to do the same thing but presently not sure what I need to do exactly.  I don’t have poetry install but have Visual Studio 2022.  I was able to download a copy of the repository to my pc locally and it’s showing in my visual studio.

Now I think I should be trying to do a authentification to connect to the Cognite Data Fusion.

But that’s where I’m starting to get lost.  it says in the readme.md :Use the Authentication script file `utils/auth.py` in this repo…. but in the repo it’s cognite_auth.py under the utils, which I think it’s the same but I’m really peaky at details when I’m not to much familiarize with something.

Also I’m confused with what Jupity’s notebook…...

How do I connect to execute the first thing like authentification….

Thank you for helping me out as a beginner in the python sdk world !

Steve

I installed the latest version of python (3.13.1) but when trying to run the command for poetry install I get the error Unable to find installation candidates for pywin32 (306) . After searching, I managed to find that it is because there is no suitable distribution of that version of pywin32 for my python version. Which version should I use?

​@SteveDelVecchio The following Academy courses and microlearnings might help you to get a better understanding about the Python SDK and the Jupitor notebook. If you have further questions, please create a seprate topic so we can further assist you.

• Python SDK
• Jupitor Notebook

​@Romulo Barbosa Ribeiro Try to install Python 3.12.x (or an earlier version) that pywin32 officially supports. You can check the PyPI page for pywin32 to find the latest supported Python versions.

I'm now having issues with the pyzmq module. I was able to install further when downgrading from Python 3.13 to 3.12 but it is unable to install pyzmq.

Creating virtualenv using-cognite-python-sdk in C:\Users\abc.def\Documents\Projetos\Cognite\using-cognite-python-sdk\.venv
Installing dependencies from lock file

Package operations: 101 installs, 0 updates, 0 removals

- Installing six (1.16.0)
- Installing attrs (23.1.0)
- Installing platformdirs (3.2.0)
- Installing pyrsistent (0.19.3)
- Installing python-dateutil (2.8.2)
- Installing pywin32 (306)
- Installing traitlets (5.9.0)
- Installing arrow (1.2.3)
- Installing fastjsonschema (2.16.3)
- Installing jsonschema (4.17.3)
- Installing jupyter-core (5.3.0)
- Installing pycparser (2.21)
- Installing pyzmq (25.0.2): Failed

ChefBuildError

Backend subprocess exited when trying to invoke get_requires_for_build_wheel

C:\Users\abc~1.def\AppData\Local\Temp\tmp5wm2thjv\.venv\Lib\site-packages\setuptools\_distutils\dist.py:270: UserWarning: Unknown distdefution option: 'cffi_modules'
warnings.warn(msg)
running egg_info
writing pyzmq.egg-info\PKG-INFO
writing dependency_links to pyzmq.egg-info\dependency_links.txt
writing requirements to pyzmq.egg-info\requires.txt
writing top-level names to pyzmq.egg-info\top_level.txt
running configure
Traceback (most recent call last):
File "C:\Users\abc.def\pipx\venvs\poetry\Lib\site-packages\pyproject_hooks\_in_process\_in_process.py", line 389, in
main()
File "C:\Users\abc.def\pipx\venvs\poetry\Lib\site-packages\pyproject_hooks\_in_process\_in_process.py", line 373, in main
json_out["return_val"] = hook(**hook_input["kwargs"])
^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "C:\Users\abc.def\pipx\venvs\poetry\Lib\site-packages\pyproject_hooks\_in_process\_in_process.py", line 143, in get_requires_for_build_wheel
return hook(config_settings)
^^^^^^^^^^^^^^^^^^^^^
File "C:\Users\abc~1.def\AppData\Local\Temp\tmp5wm2thjv\.venv\Lib\site-packages\setuptools\build_meta.py", line 334, in get_requires_for_build_wheel
return self._get_build_requires(config_settings, requirements=[])
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "C:\Users\abc~1.def\AppData\Local\Temp\tmp5wm2thjv\.venv\Lib\site-packages\setuptools\build_meta.py", line 304, in _get_build_requires
self.run_setup()
File "C:\Users\abc~1.def\AppData\Local\Temp\tmp5wm2thjv\.venv\Lib\site-packages\setuptools\build_meta.py", line 320, in run_setup
exec(code, locals())
File "", line 1372, in
File "C:\Users\abc~1.def\AppData\Local\Temp\tmp5wm2thjv\.venv\Lib\site-packages\setuptools\__init__.py", line 117, in setup
return distutils.core.setup(**attrs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "C:\Users\abc~1.def\AppData\Local\Temp\tmp5wm2thjv\.venv\Lib\site-packages\setuptools\_distutils\core.py", line 186, in setup
return run_commands(dist)
^^^^^^^^^^^^^^^^^^
File "C:\Users\abc~1.def\AppData\Local\Temp\tmp5wm2thjv\.venv\Lib\site-packages\setuptools\_distutils\core.py", line 202, in run_commands
dist.run_commands()
File "C:\Users\abc~1.def\AppData\Local\Temp\tmp5wm2thjv\.venv\Lib\site-packages\setuptools\_distutils\dist.py", line 983, in run_commands
self.run_command(cmd)
File "C:\Users\abc~1.def\AppData\Local\Temp\tmp5wm2thjv\.venv\Lib\site-packages\setuptools\dist.py", line 999, in run_command
super().run_command(command)
File "C:\Users\abc~1.def\AppData\Local\Temp\tmp5wm2thjv\.venv\Lib\site-packages\setuptools\_distutils\dist.py", line 1002, in run_command
cmd_obj.run()
File "C:\Users\abc~1.def\AppData\Local\Temp\tmp5wm2thjv\.venv\Lib\site-packages\setuptools\command\egg_info.py", line 312, in run
self.find_sources()
File "C:\Users\abc~1.def\AppData\Local\Temp\tmp5wm2thjv\.venv\Lib\site-packages\setuptools\command\egg_info.py", line 320, in find_sources
mm.run()
File "C:\Users\abc~1.def\AppData\Local\Temp\tmp5wm2thjv\.venv\Lib\site-packages\setuptools\command\egg_info.py", line 543, in run
self.add_defaults()
File "C:\Users\abc~1.def\AppData\Local\Temp\tmp5wm2thjv\.venv\Lib\site-packages\setuptools\command\egg_info.py", line 581, in add_defaults
sdist.add_defaults(self)
File "C:\Users\abc~1.def\AppData\Local\Temp\tmp5wm2thjv\.venv\Lib\site-packages\setuptools\command\sdist.py", line 109, in add_defaults
super().add_defaults()
File "C:\Users\abc~1.def\AppData\Local\Temp\tmp5wm2thjv\.venv\Lib\site-packages\setuptools\_distutils\command\sdist.py", line 239, in add_defaults
self._add_defaults_ext()
File "C:\Users\abc~1.def\AppData\Local\Temp\tmp5wm2thjv\.venv\Lib\site-packages\setuptools\_distutils\command\sdist.py", line 323, in _add_defaults_ext
build_ext = self.get_finalized_command('build_ext')
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "C:\Users\abc~1.def\AppData\Local\Temp\tmp5wm2thjv\.venv\Lib\site-packages\setuptools\_distutils\cmd.py", line 316, in get_finalized_command
cmd_obj.ensure_finalized()
File "C:\Users\abc~1.def\AppData\Local\Temp\tmp5wm2thjv\.venv\Lib\site-packages\setuptools\_distutils\cmd.py", line 124, in ensure_finalized
self.finalize_options()
File "", line 1111, in finalize_options
File "C:\Users\abc~1.def\AppData\Local\Temp\tmp5wm2thjv\.venv\Lib\site-packages\setuptools\dist.py", line 999, in run_command
super().run_command(command)
File "C:\Users\abc~1.def\AppData\Local\Temp\tmp5wm2thjv\.venv\Lib\site-packages\setuptools\_distutils\dist.py", line 1001, in run_command
cmd_obj.ensure_finalized()
File "C:\Users\abc~1.def\AppData\Local\Temp\tmp5wm2thjv\.venv\Lib\site-packages\setuptools\_distutils\cmd.py", line 124, in ensure_finalized
self.finalize_options()
File "", line 359, in finalize_options
File "C:\Users\abc~1.def\AppData\Local\Temp\tmpkr4ytwzi\pyzmq-25.0.2\buildutils\misc.py", line 71, in locate_vcredist_dir
vcvars = msvc.msvc14_get_vc_env(plat)
^^^^^^^^^^^^^^^^^^^^^^
AttdefuteError: module 'setuptools.msvc' has no attdefute 'msvc14_get_vc_env'


at ~\pipx\venvs\poetry\Lib\site-packages\poetry\installation\chef.py:164 in _prepare
160│
161│ error = ChefBuildError("\n\n".join(message_parts))
162│
163│ if error is not None:
→ 164│ raise error from None
165│
166│ return path
167│
168│ def _prepare_sdist(self, archive: Path, destination: Path | None = None) -> Path:

Note: This error originates from the build backend, and is likely not a problem with poetry but with pyzmq (25.0.2) not supporting PEP 517 builds. You can verify this by running 'pip wheel --no-cache-dir --use-pep517 "pyzmq (==25.0.2)"'.

​@Håkon V. Treider any idea regarding the issue that ​@Romulo Barbosa Ribeiro is observing?

Hi,

I’m facing a permission issue when i try create asset in dataset that i’ve created during “Hands on” part of “Learn to Use the cognite Python SDK” course.

You can see the parameters for authentication Below:

I’ve received following error by run these code root_asset = Asset(name="world", description="Asset root of countries", data_set_id=ds_id)
client.assets.create(root_asset)

Can someone help me?