Replies posted by Peter Arwanitis

18 Replies

Sverre Dorheim
Committed
Sverre DorheimCommitted
posted in Developer and User Community

Manage data governance at scale in CDF

Hi @Sverre Dorheim , thanks for writing and sharing!It was a team effort the last months, but it needs one to step up and talk about it :)As governance (and access-control & data-lineage as part of it) is a highly complex topic, I fully agree that we must have configuration-driven tooling to stay on top from day one and during the operational life-cycle of a project.As Cognite Principal Solution Architect and author of the mentioned “inso-bootstrap-cli” I’m looking forward for questions and issues how to use this approach and make it part of every CDF project. Please checkout or ask for other tools available.For issues (feature-requests, bugs, improvements, clarifications, ..) the GitHub Issues tracking is available, which allows our team to follow up.🤖let’s automate!

2
K
Practitioner
1 year ago
A
Seasoned
Adrian DamsSeasoned
asked in Academy Discussions

inso-bootstrap-cli is not creating CDF groups

Hi Adrian,Just confirming to Sverre's reply.Diagram really helps, before you get used to the naming schema of the created cdf-groupsIn your case some -- but not all -- group-names generated by your config are:test:all:ownertest:all:readtest:src:001:fteg:ownertest:src:001:fteg:read

9
Peter Arwanitis
Practitioner
1 year ago
A
Seasoned
Adrian DamsSeasoned
asked in Academy Discussions

inso-bootstrap-cli is not creating CDF groups

Hi Adrian,> Thanks. I changed the groupnames to all:owner and all:read and it works like a charmGlad to hear :) > I still find it “disappointing” that you’re limited to those names but they’re usable so that’ll workCould you please explain what is the “disappointing” part?What would be your ideas for an ideal naming-scheme? ---Some of the constraints which lead to this naming-scheme and example-configuration:supporting an alpha-num sortable list in CDF Group, RAW DB or Data Set listings (the only reason that three-digit numbers are suggested on node-level, which are not mandatory) allowing an easy filtering in Fusion UI by a known namespace or node-name reflect the hierarchical namespace in the name / externalId keep the naming scheme consistent across CDF Group, RAW Database names and Data Sets we have to keep the name short as for example RAW DB names are max 32 characters (a validation step help and alerts if names are getting too long) where “descriptions” are possible you ca

9
Peter Arwanitis
Practitioner
1 year ago
A
Seasoned
Adrian DamsSeasoned
asked in Academy Discussions

inso-bootstrap-cli is not creating CDF groups

Hi Ardian and thank you for taking your time to explain your case, which I have now (hopefully) understood.Looking forward to your feedback, as I got a bit carried away with explaining and evangelizing the approach :) > let me describe my setup that I was trying to map over to CDF Migrating an existing setup to a CDF “bootstrap-cli” one, is for sure not an easy task.Why? Because “bootstrap-cli” comes with an opinionated naming-scheme -- as you are aware now too.Migrating requires now either to seek for a a compromise or to follow a new approach.The compromise (or alignment) can happen in my opinion on:naming your ns/nodes level freedom of mapping in idp-cdf-mapping (where in an ideal world the AAD Group names follow a similar naming pattern as on the CDF side) creating specific namespace/nodes only reflecting end user-roles, using the `shared-access` feature to define shared dataset access-control by using the READ/OWNER semantic. (I only refer to CDF Dataset in my explanations, but

9
Peter Arwanitis
Practitioner
1 year ago
A
Seasoned
Adrian DamsSeasoned
asked in Academy Discussions

inso-bootstrap-cli is not creating CDF groups

> You would have to manually configure the SourceId to map the group to an associated AAD group. Yes, keeping AAD and CDF groups in sync is a manual job.CDF and AAD are seen here as two independent systems, with a lose-coupling using idp-cdf-mapping, as it is not feasible to dictate or automate customer side IdP (AAD) from bootstrap-cli end.

9
Peter Arwanitis
Practitioner
1 year ago
A
Seasoned
Adrian DamsSeasoned
asked in Academy Discussions

inso-bootstrap-cli is not creating CDF groups

> What I suppose I am saying I would like to be able to do (notwithstanding the constraints you describe) is to be able to define groupnames in a custom fashion and not be limited to “owner” and “read”. To answer this explicit: Only on the AAD side you can name groups in a custom way, on CDF side you can control some parts of the naming, but `:owner` and `:read` are hardcoded and automatically provided for *every* configured node-name.

9
Peter Arwanitis
Practitioner
1 year ago
B
Active
Basudeba MandalActive
asked in Developer and User Community

Getting AttributeError while using "bootstrap-cli prepare" command

Hi Basudeba, thank you for reaching out for help on our Cognite Hub! My name is Peter from Cognite and I wrote the `bootstrap-cli` tool. The AttributeError states that bootstrap-cli expects in your provided config.yaml a `logger` section. You can simply copy and add this `logger` section (from bootstrap-cli documentation) and I expect it to work. logger: file: path: ./logs/test-deploy.log level: INFO console: level: INFO Hopefully this is fixing your issue and you can proceed :)best regardsPeter(=PA=)

7
B
Active
11 months ago
B
Active
Basudeba MandalActive
asked in Developer and User Community

Getting AttributeError while using "bootstrap-cli prepare" command

Hi Basudeba, just for your confirmation:the `prepare` command requires to access Cognite with a service-account (an AAD App) which requires the `cognite` section available too as documented here it is using a `.env` file to separate the secrets from the configuration cognite: host: ${BOOTSTRAP_CDF_HOST} project: ${BOOTSTRAP_CDF_PROJECT} # # IdP login: # idp-authentication: client-id: ${BOOTSTRAP_IDP_CLIENT_ID} secret: ${BOOTSTRAP_IDP_CLIENT_SECRET} scopes: - ${BOOTSTRAP_IDP_SCOPES} token_url: ${BOOTSTRAP_IDP_TOKEN_URL}Which requires -- beside others -- a `BOOTSTRAP_IDP_CLIENT_ID / BOOTSTRAP_IDP_CLIENT_SECRET` (client/secrets) of an AAD Application for service-level authentication.which you had to create first in your AAD tenant because this (first) AAD App is used for root access to Cognite, a typical name could be “CDF_DEV_ROOT_CLIENT” (“DEV” or “TEST”  or “PROD” in case SLB has staging projects connected to the same SLB AAD) which must be a member of the same A

7
B
Active
11 months ago
B
Active
Basudeba MandalActive
asked in Developer and User Community

Getting AttributeError while using "bootstrap-cli prepare" command

Hi Basudeba, > I had missed .env file.Not mandatory, just the recommended practice not storing secrets in config files, which maybe get added to a version-control-system (like git) The permission problem of your AAD App should not happen, as long as you have added the app to the same AAD Group which you used for requesting the CDF Project. “Typically” the very first CDF Group setup and linked to this AAD Group is named “oidc-admin-group” in Cognite, and as long as your user has the permissions, you open it from “Manage > Access Management” select the group, go to edit and copy the “Source ID” (which is the AAD Group object-id)? This (screenshot) are the required capabilities in Cognite, the AAD App requires (through its AAD Group > Cognite linking) I’m sorry if this sounds over-complicated, but I try to provide enough information and context, hopefully to help you navigate and solving your problem ASAP. Best regardsPeter(=PA=)

7
B
Active
11 months ago
S
Seasoned
Snehal-JagtapSeasoned
asked in Academy Discussions

Bootstrap CLI - Space support + Custom group support

Hi @Snehal-Jagtap,-- 1. part about ‘spaces’ support available with bootstrap-cli --- You can get full ”spaces” support (equal to “datasets”) from a bootstrap-cli pre-release (release candidate):https://github.com/cognitedata/inso-bootstrap-cli/releases/tag/v3.0.0-beta.1As noted in release notes it is already ok to use it in production environment, and I expect no configuration changes for the official v3 release :)In case you used bootstrap-cli before with v2, please read through the linked migration guide and the provided config-example for v3.Please let me know if you run into problem, which I can help you to fix.Bets regardsPeter(=PA=)

5
Peter Arwanitis
Practitioner
10 months ago
S
Seasoned
Snehal-JagtapSeasoned
asked in Academy Discussions

Bootstrap CLI - Space support + Custom group support

-- 2. part about ‘custom groups or granular groups’ --- I’m not sure that I fully understood your idea?Using “bootstrap-cli” provides you a fully templated-setup, following the namespaces and node-names. By design it avoids customization (which typically harms transparency). Instead you are encouraged to configure “namespaces & nodes”, let’s say:`src:001:weather` for managing your RAW and cleansed weather data in CDF RAW, CDF Time Series and CDF Events (as an example) `uc:004:forecast` for managing you source and domain data-models Using “templates” means you get more than you asked for, simply ignore everything which you don’t need.only use provided RAWDb `src:001:weather:db` and Dataset `src:001:weather:ds` to manage your raw-tables, time-series and events only use provided Space `src-001-weather-spc` to manage your data-models and data-model-instancesWith this approach, a lot of Datasets, RAWDbs or Spaces are created by bootstrap-cli according to your configuration, but not nec

5
Peter Arwanitis
Practitioner
10 months ago
S
Seasoned
Snehal-JagtapSeasoned
asked in Academy Discussions

Bootstrap CLI - Space support + Custom group support

Hi @Snehal-Jagtap glad to here that v3.0.0-beta.1 is working for you :)Please follow the releases, to catch up with new ones. > Will there be any cost impact if the bootstrap-cli creates some Datasets, RAWDbs or Spaces and if some of those are unused.? The answer is no, there is no impact. > Also, are there any plans to include groups for granular level access in bootstrap-cli, like access on specific instances or timeseries objects, etc.  No, because this is -- kind of -- already supported:for each access-control requirement => you have to establish an `idp-cdf-mapping` which means, linking one of your AAD Groups to one (or more) of the CDF Groups available based on your bootstrap-cli configuration creating the right amount of namespaces/nodes allows you to express your project semantics (terminology)The “template approach” of bootstrap-cli is creating all scopes for you, but -- as pointed out in last post -- with only two templates of capabilities: OWNER and READso you get m

5
Peter Arwanitis
Practitioner
10 months ago
P
Committed
patricknitschkeCommitted
asked in Developer and User Community

Uploading InsertDatapoints via REST extractor - Getting a CogniteAPIError code 405 Not Allowed Error

Hi Patrick,two things could help debugging:understanding the exact CDF API call sent from your code. Given that you are using the latest Cognite Python SDK, you can activate debug logging with `client.config.debug = True` Not all the parameters of your decorator are visible, what is the value going in `paths` here? (paths=paths, .. An Error 405 is very uncommon using the Cognite Python SDK, so hopefully we understand it better with more details.best regardsPeter(=PA=)

14
P
Committed
6 months ago
P
Committed
patricknitschkeCommitted
asked in Developer and User Community

Uploading InsertDatapoints via REST extractor - Getting a CogniteAPIError code 405 Not Allowed Error

and on a 2nd look, you posted this```2023-10-31 13:57:39.081 UTC [DEBUG   ] ThreadPoolExecutor-1_0 - HTTP Error 405 POST https://sao.fusion.cognite.com/api/v1/projects/https://sao.fusion.cognite.com/timeseries/data: <html> the API URL looks wrong, as at the place where you expect the “cdf-project” value (after the `/projects` there is the base-url again? Can you check your Cognite client configuration and envvars used?(=PA=)

14
P
Committed
6 months ago
P
Committed
patricknitschkeCommitted
asked in Developer and User Community

Uploading InsertDatapoints via REST extractor - Getting a CogniteAPIError code 405 Not Allowed Error

Still the API URL looks wrong> https://sao.fusion.cognite.com/api/v1/projects/sao-staging/timeseries/data Now I see it starts with https://sao.fusion.., but it is supposed to start with something like`http://{cluster}.cognitedata.com/...`Where cluster is `westeurope-1` or `api` or else.You see it in Fusion URL, like `?env=westeurope-1&cluster=westeurope-1.cognitedata.com` Can you pls check again?Or share your config (w/o secrets :))(=PA=) 

14
P
Committed
6 months ago
P
Committed
patricknitschkeCommitted
asked in Developer and User Community

Uploading InsertDatapoints via REST extractor - Getting a CogniteAPIError code 405 Not Allowed Error

yourCOGNITE_BASE_URLis wrong and doesn’t point to our CDF API endpoint, but Fusion UI endpoint. If you don’t know, which cluster your cdf-project is running on, you can see it in Fusion UI URL at the `?cluster=` http-parameter!Just add `https://` in front and set it as your `COGNITE_BASE_URL` Most likely correct, but I’ve seenCOGNITE_TOKEN_URLwith endings `/oauth2/token` or `/oauth2/v2.0/token` (=PA=) 

14
P
Committed
6 months ago
P
Committed
patricknitschkeCommitted
asked in Developer and User Community

Uploading InsertDatapoints via REST extractor - Getting a CogniteAPIError code 405 Not Allowed Error

Tried to find the documentation about the `baseUrl`Looks like you can find it inhttps://cognite-sdk-python.readthedocs-hosted.com/en/latest/quickstart.html#instantiate-a-new-client # This value will depend on the cluster your CDF project runs oncluster = "api"base_url = f"https://{cluster}.cognitedata.com"tenant_id = "my-tenant-id"client_id = "my-client-id"Or herehttps://developer.cognite.com/dev/use_the_APIOr herehttps://developer.cognite.com/dev/quickstart/#step-2-set-up-environment-variablesbaseURL: You can find your baseURL from the CDF project. Navigate to your CDF project. Under Manage & Configure > Manage access, select Open ID connect tab. The URL in the audience field is the baseURL. I’ve to admit it is not 100% crystal clear(?) Let me try to follow up internally, and check how we can improve documentation especially for onboarding new developers!(=PA=)

14
P
Committed
6 months ago
P
Committed
patricknitschkeCommitted
asked in Developer and User Community

Uploading InsertDatapoints via REST extractor - Getting a CogniteAPIError code 405 Not Allowed Error

Yep you’ve solved it. Glad you made it into a successful evening :)Happy Halloween!(=PA=)

14
P
Committed
6 months ago

Badge winners

Show all badges
Terms & ConditionsCookie Settings