Solved

Client id for the Python-SDK


Userlevel 1

Hello,
I have a Python project that uses publicdata project of Open Industrial Data. I am trying to refactor the authentication step to use Python-SDK with OAuth since the API keys are (being) deprecated.
To do so, I need the parameters token_url, client_id, client_secret, scopes.
I am setting the parameters as such:

cognite_tenant_id = "48d5043c-cf70-4c49-881c-c638f5796997"
token_url=f"https://login.microsoftonline.com/{cognite_tenant_id}/oauth2/v2.0/authorize"

client_id=???

client_secret=<client secret generated from Open Industrial Data https://hub.cognite.com/open-industrial-data-211>

scopes=["https://api.cognitedata.com/user_impersonation"]

I would like to know if we can use the already defined Cognite tenant in AAD with Tenant Id “48d5043c-cf70-4c49-881c-c638f5796997” ? Are you providing the client id for that then? Similar to the ones publicly provided for Postman and JS-SDK? Please also note that I cannot use interactive login since it is not a one-off script.
Moreover, I cannot register an app in AAD as the documentation says, since I don’t have an active Azure subscription, or a Microsoft product/service that entitles me to use free/paid AAD.
Regards,
Arian

icon

Best answer by Maria Fonseca 6 December 2022, 09:31

View original

13 replies

Userlevel 2
Badge +6

Hi Arian, 

Thanks for posting your question here. 
For the client id,  take a look of How to connect to OIDC and please also take a look of our academy course Learn to use python SDK course that will guide in the right direction.

If the interactive login is not working for you,  please other login method such as device code, client secret.

Please let us know, if you have any other questions.

Regards
Kumar

Userlevel 1

Hello Rahul,

Thanks for answering. The topic you shared on How to connect to OIDC only provides client-id for the Postman and JavaScript applications; not for the Python-SDK. 
Moreover, using device code is not a viable option for me as it requires to “get a code you can pass to a web page and authenticate there”. So the only option for me would be to use client-id, client-secret.

Regards,
Arian

Userlevel 3

Hi @arian, did using the client-id, client-secret option work for you?

Userlevel 1

Hi @Carin Meems ,

I am still struggling with getting the client-id. I restate my question:
Do you publicly provide a client id for the Python-SDK for the publicdata project, the way you do for Postman and JS-SDK?

Regards,
Arian

I’m fighting a similar problem. I have used the js ClientId successfully but none of the authentication flows allow for a non interactive login. 

 

Here’s my working code for the Device Flow inside a django management command (should be easy to re-write for any other app). Even working out what values to use for these various parts was difficult.

import asyncio
import atexit
import datetime
import os
import time

from channels.layers import get_channel_layer
from cognite.client import ClientConfig, CogniteClient
from cognite.client.credentials import OAuthClientCredentials, Token
from django.core.management.base import BaseCommand
from msal import (
ConfidentialClientApplication,
PublicClientApplication,
SerializableTokenCache,
)

# Contact Project Administrator to get these
TENANT_ID = "48d5043c-cf70-4c49-881c-c638f5796997"
CLIENT_ID = "dea6bb8d-0f48-4bf0-a469-176fc19edc14"
CLIENT_SECRET = "my_secret"

CDF_CLUSTER = "api" # api, westeurope-1 etc
COGNITE_PROJECT = "publicdata"
BASE_URL = f"https://{CDF_CLUSTER}.cognitedata.com"

CACHE_FILENAME = "cache.bin"
SCOPES = [f"https://{CDF_CLUSTER}.cognitedata.com/.default"]


AUTHORITY_HOST_URI = "https://login.microsoftonline.com"
AUTHORITY_URI = AUTHORITY_HOST_URI + "/" + TENANT_ID

TOKEN_URL = f"https://login.microsoftonline.com/{TENANT_ID}/oauth2/v2.0/token"

PORT = 53000


def create_cache():
cache = SerializableTokenCache()
if os.path.exists(CACHE_FILENAME):
cache.deserialize(open(CACHE_FILENAME, "r").read())
atexit.register(
lambda: open(CACHE_FILENAME, "w").write(cache.serialize())
if cache.has_state_changed
else None
)
return cache


def authenticate(app: PublicClientApplication):

accounts = app.get_accounts()
if accounts:
creds = app.acquire_token_silent(SCOPES, account=accounts[0])
else:

device_flow = app.initiate_device_flow(scopes=SCOPES)
print(device_flow["message"]) # print device code to screen
creds = app.acquire_token_by_device_flow(flow=device_flow)

return creds


class Command(BaseCommand):
group_name = "cognite"
channel_name = "thedata"
channel_layer = get_channel_layer()

def add_arguments(self, parser):
pass

def handle(self, *args, **options):
try:

# creds = OAuthClientCredentials(
# token_url=TOKEN_URL,
# client_id=CLIENT_ID,
# client_secret=CLIENT_SECRET,
# scopes=SCOPES,
# )

# self.cnf = ClientConfig(
# client_name="astar-tanar",
# project=COGNITE_PROJECT,
# credentials=creds,
# base_url=BASE_URL,
# )

# self.client = CogniteClient(self.cnf)

self.app = PublicClientApplication(
client_id=CLIENT_ID, authority=AUTHORITY_URI
)
self.app.acquire_token_by_device_flow
cnf = ClientConfig(
client_name="astar-tanar",
base_url=BASE_URL,
project=COGNITE_PROJECT,
credentials=Token(authenticate(self.app)["access_token"]),
)
self.client = CogniteClient(cnf)

print(self.client.assets.list())

while True:
val = self.client.datapoints.retrieve_latest(id=52336799167961)[0]
theData = val.dump()
loop = asyncio.get_event_loop()
coroutine = self.channel_layer.group_send(
self.group_name,
{
"type": "receive",
"thedata": theData,
},
)
loop.run_until_complete(coroutine)

time.sleep(5)

except Exception as e:
print("ERROR")
print(e)

return "Done"

You will notice that I have been trying to use the OAuthClientCredentials approach. I used the js ClientID and generated a new ClientSecret from the portal but the secret and client id do not match up. Essentially without the OAuth login working I can’t see how anyone can use the publicdata project to run extractors in a service. Using the device flow the token doesn’t last very long and is not useful for long running tasks.

 

Maybe this is an intended limitation for the publicdata?

Userlevel 2
Badge +3

Hello, I have proceed to clarify on this article which details you should use for Python SDK 

 

Details to create the Cognite client when using Postman and Python SDK

  • Tenant ID - 48d5043c-cf70-4c49-881c-c638f5796997,

  • Client ID - 1b90ede3-271e-401b-81a0-a4d52bea3273,

  • project=publicdata,

  • CDF_CLUSTER - api

  • App name: OID-Api

 

You can get the client secret by using the Widget in Open Industrial Data and selecting “Others”. This is explained with more detail in 

Please let me know if this resolve your concerns.

I have made a couple of convenience classes based on the OIDC examples given here. Perhaps it can be useful to you. 

import atexit
from pathlib import Path

from cognite.client import CogniteClient, ClientConfig
from cognite.client.credentials import Token
from msal import PublicClientApplication, SerializableTokenCache

from publicdata_credentials import credentials as creds


class OIDCInteractiveRefresh:
"""Convenience class for authentication toward CDF with Open ID Connect interactive refresh.
Based on the following exmaple: https://github.com/cognitedata/python-oidc-authentication"""
def __init__(
self,
tenant_id: str=None,
client_id: str=None,
cdf_cluster: str="api",
cognite_project: str=None,
client_name: str=""
):
"""
Initialize authenticator class.

Args
tenant_id : The ID for the CDF tenant. Type: str
client_id : The ID for the CDF client. Type: str
cdf_cluster : The name if the CDF cluster. Type: str. Default: api
cognite_project: The name of the cognite project. Type: str
client_name : Optional name of the CDF client. Type: str. Default:
"""
self.tenant_id = tenant_id
self.client_id = client_id
self.cdf_cluster = cdf_cluster
self.cognite_project = cognite_project
self.client_name = client_name

self.cache_filename = Path("cache.bin")
self.base_url = f"https://{self.cdf_cluster}.cognitedata.com"
self.scopes = [f"https://{self.cdf_cluster}.cognitedata.com/.default"]

self.authority_host_uri = "https://login.microsoftonline.com"
self.authority_uri = self.authority_host_uri + "/" + self.tenant_id
self.port = 53000

self.app = PublicClientApplication(
client_id=self.client_id,
authority=self.authority_uri,
token_cache=self._create_cache()
)
self.token = self._get_token()

self.config = ClientConfig(
client_name=self.client_name,
project=self.cognite_project,
credentials=Token(self.token),
base_url=self.base_url
)
self.client = CogniteClient(self.config)

def _create_cache(self):
cache = SerializableTokenCache()
if self.cache_filename.exists():
cache.deserialize(self.cache_filename.open().read())
atexit.register(
lambda:
self.cache_filename.open("w").write(cache.serialize()) if cache.has_state_changed else None
)
return cache

def _authenticate_azure(self, app):
accounts = app.get_accounts()
if accounts:
creds = app.acquire_token_silent(self.scopes, account=accounts[0])
else:
creds = app.acquire_token_interactive(scopes=self.scopes, port=self.port)
return creds

def _get_token(self):
return self._authenticate_azure(self.app)["access_token"]


if __name__ == "__main__":
aut = OIDCInteractiveRefresh(
tenant_id=creds.tenant_id,
client_id=creds.client_id,
cdf_cluster=creds.cdf_cluster,
cognite_project=creds.cognite_project
)

print(aut.client.assets.list(limit=5).to_pandas().externalId)

And the contents of `publicdata_credentials.py`:

 

from types import SimpleNamespace

credentials = SimpleNamespace(**{
"tenant_id": "48d5043c-cf70-4c49-881c-c638f5796997",
"client_id": "1b90ede3-271e-401b-81a0-a4d52bea3273",
"cdf_cluster": "api",
"cognite_project": "publicdata"
})

Example usage of the class for the  public data project is given in the “if __name__ == …….” statement.

And a similar class based on the device code workflow:

from cognite.client import CogniteClient, ClientConfig
from cognite.client.credentials import Token
from msal import PublicClientApplication

from publicdata_credentials import credentials as creds


class OIDCDeviceCode:
"""Convenience class for authentication toward CDF with Open ID Connect device code.
Based on the following exmaple: https://github.com/cognitedata/python-oidc-authentication"""
def __init__(
self,
tenant_id: str=None,
client_id: str=None,
cdf_cluster: str="api",
cognite_project: str=None,
client_name: str=""
):
"""
Initialize authenticator class.

Args
tenant_id : The ID for the CDF tenant. Type: str
client_id : The ID for the CDF client. Type: str
cdf_cluster : The name if the CDF cluster. Type: str. Default: api
cognite_project: The name of the cognite project. Type: str
client_name : Optional name of the CDF client. Type: str. Default:
"""
self.tenant_id = tenant_id
self.client_id = client_id
self.cdf_cluster = cdf_cluster
self.cognite_project = cognite_project
self.client_name = client_name

self.base_url = f"https://{self.cdf_cluster}.cognitedata.com"
self.scopes = [f"https://{self.cdf_cluster}.cognitedata.com/.default"]

self.authority_host_uri = "https://login.microsoftonline.com"
self.authority_uri = self.authority_host_uri + "/" + self.tenant_id

self.creds = self._authenticate_device_code()
self.config = ClientConfig(
client_name=self.client_name,
project=self.cognite_project,
credentials=Token(self.creds["access_token"]),
base_url=self.base_url
)
self.client = CogniteClient(self.config)

def _authenticate_device_code(self):
app = PublicClientApplication(client_id=self.client_id, authority=self.authority_uri)
flow = app.initiate_device_flow(scopes=self.scopes)
print(flow["message"])
creds = app.acquire_token_by_device_flow(flow=flow)
return creds


if __name__ == "__main__":
aut = OIDCDeviceCode(
tenant_id=creds.tenant_id,
client_id=creds.client_id,
cdf_cluster=creds.cdf_cluster,
cognite_project=creds.cognite_project
)

print(aut.client.assets.list(limit=5).to_pandas().externalId)

 

Userlevel 1

Hello, I have proceed to clarify on this article which details you should use for Python SDK 

 

Details to create the Cognite client when using Postman and Python SDK

  • Tenant ID - 48d5043c-cf70-4c49-881c-c638f5796997,

  • Client ID - 1b90ede3-271e-401b-81a0-a4d52bea3273,

  • project=publicdata,

  • CDF_CLUSTER - api

  • App name: OID-Api

 

You can get the client secret by using the Widget in Open Industrial Data and selecting “Others”. This is explained with more detail in 

Please let me know if this resolve your concerns.

Hello @majofoal ,
This doesn’t work. Here is a trace that might be helpful (Checked with both CogniteClient from the SDK and MSAL’s ConfidentialClientApplication authentication ):

Error generating access token: invalid_resource, 400, AADSTS500011: The resource principal named https://api.cognitedata.com/user_impersonation was not found in the tenant named Cognite Hub. This can happen if the application has not been installed by the administrator of the tenant or consented to by any user in the tenant. You might have sent your authentication request to the wrong tenant.
Trace ID: abee9008-11fa-438a-a036-aa77f172e201
Correlation ID: 3555db05-b89a-4a1f-b742-aef25393c2c4
Timestamp: 2022-12-13 15:22:23Z

The fact that it doesn’t throw an “Invalid client secret” exception means that the provided client secret is correct; moreover, it correctly identifies tenant since it finds “Cognite Hub” in AAD.
As of the scope, I tested both impersonation and read modes. None worked.

Here is a minimal code to reproduce it:
 

# Using Cognite-SDK
cognite_base_url = 'https://api.cognitedata.com'
cognite_tenant_id = "48d5043c-cf70-4c49-881c-c638f5796997"
python_client_id = "1b90ede3-271e-401b-81a0-a4d52bea3273"
client_secret= <Client id from OID secret generator's "other" application>

creds = OAuthClientCredentials(token_url=f"https://login.microsoftonline.com/{cognite_tenant_id}/oauth2/v2.0/token",
client_id=python_client_id,
client_secret=client_secret,
scopes=[f"{cognite_base_url}/user_impersonation/.default"])
cnf = ClientConfig(client_name="OID-Api", base_url=cognite_base_url, project="publicdata", credentials=creds)
c = CogniteClient(cnf)
c.login.status()

# Directly using MSAL
app = ConfidentialClientApplication(
python_client_id,
authority=f"https://login.microsoftonline.com/{cognite_tenant_id}/",
client_credential=client_secret)
scope = app.get_authorization_request_url([cognite_base_url]).split('scope=')[1] + '/.default'
auth= app.acquire_token_for_client([scope])

I don’t know if client_name=”OID-Api” is correct or not but it probably doesn’t matter since it is apparently not used by the SDK’s LoginAPI.

Userlevel 1

I have made a couple of convenience classes based on the OIDC examples given here. Perhaps it can be useful to you. 

import atexit
from pathlib import Path

from cognite.client import CogniteClient, ClientConfig
from cognite.client.credentials import Token
from msal import PublicClientApplication, SerializableTokenCache

from publicdata_credentials import credentials as creds


class OIDCInteractiveRefresh:
"""Convenience class for authentication toward CDF with Open ID Connect interactive refresh.
Based on the following exmaple: https://github.com/cognitedata/python-oidc-authentication"""
def __init__(
self,
tenant_id: str=None,
client_id: str=None,
cdf_cluster: str="api",
cognite_project: str=None,
client_name: str=""
):
"""
Initialize authenticator class.

Args
tenant_id : The ID for the CDF tenant. Type: str
client_id : The ID for the CDF client. Type: str
cdf_cluster : The name if the CDF cluster. Type: str. Default: api
cognite_project: The name of the cognite project. Type: str
client_name : Optional name of the CDF client. Type: str. Default:
"""
self.tenant_id = tenant_id
self.client_id = client_id
self.cdf_cluster = cdf_cluster
self.cognite_project = cognite_project
self.client_name = client_name

self.cache_filename = Path("cache.bin")
self.base_url = f"https://{self.cdf_cluster}.cognitedata.com"
self.scopes = [f"https://{self.cdf_cluster}.cognitedata.com/.default"]

self.authority_host_uri = "https://login.microsoftonline.com"
self.authority_uri = self.authority_host_uri + "/" + self.tenant_id
self.port = 53000

self.app = PublicClientApplication(
client_id=self.client_id,
authority=self.authority_uri,
token_cache=self._create_cache()
)
self.token = self._get_token()

self.config = ClientConfig(
client_name=self.client_name,
project=self.cognite_project,
credentials=Token(self.token),
base_url=self.base_url
)
self.client = CogniteClient(self.config)

def _create_cache(self):
cache = SerializableTokenCache()
if self.cache_filename.exists():
cache.deserialize(self.cache_filename.open().read())
atexit.register(
lambda:
self.cache_filename.open("w").write(cache.serialize()) if cache.has_state_changed else None
)
return cache

def _authenticate_azure(self, app):
accounts = app.get_accounts()
if accounts:
creds = app.acquire_token_silent(self.scopes, account=accounts[0])
else:
creds = app.acquire_token_interactive(scopes=self.scopes, port=self.port)
return creds

def _get_token(self):
return self._authenticate_azure(self.app)["access_token"]


if __name__ == "__main__":
aut = OIDCInteractiveRefresh(
tenant_id=creds.tenant_id,
client_id=creds.client_id,
cdf_cluster=creds.cdf_cluster,
cognite_project=creds.cognite_project
)

print(aut.client.assets.list(limit=5).to_pandas().externalId)

And the contents of `publicdata_credentials.py`:

 

from types import SimpleNamespace

credentials = SimpleNamespace(**{
"tenant_id": "48d5043c-cf70-4c49-881c-c638f5796997",
"client_id": "1b90ede3-271e-401b-81a0-a4d52bea3273",
"cdf_cluster": "api",
"cognite_project": "publicdata"
})

Example usage of the class for the  public data project is given in the “if __name__ == …….” statement.

@Anders Brakestad  Thanks. This is a useful script to exploit token caching if you could authenticate interactively the first time (to get the account), and from then on to acquire token for that account with the specified scope.  I can’t use any interactive login.
Also, as I mentioned earlier, using device code is not an option for me.

When I use this an interactive log in window pops up - is this what you refer to? Then it caches after. 

Userlevel 1

When I use this an interactive log in window pops up - is this what you refer to? Then it caches after. 

Yes exactly 🙂.  My script is deployed remotely so I can’t do an interactive login at all. Unless I mimic a browser on a headless server 🤓

Userlevel 2

Looking at your scopes parameter it’s a bit wrong, it should only contain .default and not user_impersonation:

scopes=[f"{cognite_base_url}/.default"]

 

Reply