Solved

Getting AttributeError while using "bootstrap-cli prepare" command

  • 31 May 2023
  • 7 replies
  • 72 views
B
Rank

Hi Team,

 

I am trying to use bootstrap-cli to create raw DB, CDF groups and datasets. 

As per the documentation, I am executing “bootstrap-cli prepare” command before applying “bootstrap-cli deploy”. However I am encountering “AttributeError” .  Could you please help me what can be done to fix this? Following are the few details of the issue that is encountered.

 

  • Command that I used to run the bootstrap-cli prepare.

poetry run bootstrap-cli prepare --idp-source-id f2e5b1ef-e601-46ec-87ec-25f62810ad1f

  • Error Snippet:

Prepare CDF Project ...
Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "C:\Users\BMandal2\bootstrap\inso-bootstrap-cli\incubator\bootstrap_cli\__main__.py", line 2365, in main
    bootstrap_cli()
  File "C:\Users\BMandal2\AppData\Local\pypoetry\Cache\virtualenvs\inso-bootstrap-cli-Jo_B4J-G-py3.10\lib\site-packages\click\core.py", line 1128, in __call__
    return self.main(*args, **kwargs)
  File "C:\Users\BMandal2\AppData\Local\pypoetry\Cache\virtualenvs\inso-bootstrap-cli-Jo_B4J-G-py3.10\lib\site-packages\click\core.py", line 1053, in main
    rv = self.invoke(ctx)
  File "C:\Users\BMandal2\AppData\Local\pypoetry\Cache\virtualenvs\inso-bootstrap-cli-Jo_B4J-G-py3.10\lib\site-packages\click\core.py", line 1659, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "C:\Users\BMandal2\AppData\Local\pypoetry\Cache\virtualenvs\inso-bootstrap-cli-Jo_B4J-G-py3.10\lib\site-packages\click\core.py", line 1395, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "C:\Users\BMandal2\AppData\Local\pypoetry\Cache\virtualenvs\inso-bootstrap-cli-Jo_B4J-G-py3.10\lib\site-packages\click\core.py", line 754, in invoke
    return __callback(*args, **kwargs)
  File "C:\Users\BMandal2\AppData\Local\pypoetry\Cache\virtualenvs\inso-bootstrap-cli-Jo_B4J-G-py3.10\lib\site-packages\click\decorators.py", line 38, in new_func
    return f(get_current_context().obj, *args, **kwargs)
  File "C:\Users\BMandal2\bootstrap\inso-bootstrap-cli\incubator\bootstrap_cli\__main__.py", line 2252, in prepare
    BootstrapCore(config_file, command=CommandMode.PREPARE, debug=obj["debug"])
  File "C:\Users\BMandal2\bootstrap\inso-bootstrap-cli\incubator\bootstrap_cli\__main__.py", line 494, in __init__
    if self.config.logger.file:
AttributeError: 'NoneType' object has no attribute 'file'

 

  • config-bootstrap.yml file content:

bootstrap:
  features:
    aggregated-level-name: all
    dataset-suffix: ds
    rawdb-suffix: db
  idp-cdf-mappings:
    - cdf-project: slb-pdf
      mappings:
        - cdf-group: cdf:all:owner
          idp-source-id: f2e5b1ef-e601-46ec-87ec-25f62810ad1f
          idp-source-name: CDF_SLBPDF_ALLPROJECTS_OWNER
        - cdf-group: cdf:all:viewer
          idp-source-id: f2e5b1ef-e601-46ec-87ec-25f62810ad1f
          idp-source-name: CDF_SLBPDF_ALLPROJECTS_VIEWER
  namespaces:
    - ns-name: src
      description: Customer source-systems
      ns-nodes:
        - node-name: src:001:basu
          description: Sources 001; from basu
          external-id: src:001:basu
        - node-name: src:001:mandal
          description: Sources 002; from mandal
          # external-id will be auto generated in this case

    - ns-name: in
      description: End user data-input provided through deployed CDF solutions
      ns-nodes:
        - node-name: in:001:trade
          description: Description about user inputs related to name
          # external_id: in:001:trade

    - ns-name: uc
      description: Use Cases representing the data-products
      ns-nodes:
        - node-name: uc:001:demand
          description: Use Case 001; Demand Side
          metadata:
            created: 220427
            generated: by cdf-config-hub script
          shared-access:
            read:
              - node-name: src:001:basu
              - node-name: src:001:mandal
            owner:
              - node-name: in:001:trade

icon

Best answer by Basudeba Mandal 2 June 2023, 11:04

View original

7 replies

Peter Arwanitis
Rank
Userlevel 2
Badge

Hi Basudeba,

 

thank you for reaching out for help on our Cognite Hub!

 

My name is Peter from Cognite and I wrote the `bootstrap-cli` tool.

 

The AttributeError states that bootstrap-cli expects in your provided config.yaml a `logger` section.

 

You can simply copy and add this `logger` section (from bootstrap-cli documentation) and I expect it to work.

 

logger:
  file:
    path: ./logs/test-deploy.log
    level: INFO
  console:
    level: INFO

 

Hopefully this is fixing your issue and you can proceed :)

best regards

Peter

(=PA=)

B
Rank

Hi Peter,

 

Thanks for looking into it, much appreciated.

As per the input, I have added “logger” section in the config yaml and with that experienced another AttributeError. I have also tried by adding “cognite” section along with “logger” section, and faced “access token” issue.

I am further exploring and looking into it. Will keep you posted by EOD if I figured out anything, otherwise I would like your assistance to get over this.

 

Error when added logger section without cognite section:

 

Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "C:\Users\BMandal2\bootstrap\inso-bootstrap-cli\incubator\bootstrap_cli\__main__.py", line 2365, in main
    bootstrap_cli()
  File "C:\Users\BMandal2\AppData\Local\pypoetry\Cache\virtualenvs\inso-bootstrap-cli-Jo_B4J-G-py3.10\lib\site-packages\click\core.py", line 1128, in __call__
    return self.main(*args, **kwargs)
  File "C:\Users\BMandal2\AppData\Local\pypoetry\Cache\virtualenvs\inso-bootstrap-cli-Jo_B4J-G-py3.10\lib\site-packages\click\core.py", line 1053, in main
    rv = self.invoke(ctx)
  File "C:\Users\BMandal2\AppData\Local\pypoetry\Cache\virtualenvs\inso-bootstrap-cli-Jo_B4J-G-py3.10\lib\site-packages\click\core.py", line 1659, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "C:\Users\BMandal2\AppData\Local\pypoetry\Cache\virtualenvs\inso-bootstrap-cli-Jo_B4J-G-py3.10\lib\site-packages\click\core.py", line 1395, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "C:\Users\BMandal2\AppData\Local\pypoetry\Cache\virtualenvs\inso-bootstrap-cli-Jo_B4J-G-py3.10\lib\site-packages\click\core.py", line 754, in invoke
    return __callback(*args, **kwargs)
  File "C:\Users\BMandal2\AppData\Local\pypoetry\Cache\virtualenvs\inso-bootstrap-cli-Jo_B4J-G-py3.10\lib\site-packages\click\decorators.py", line 38, in new_func
    return f(get_current_context().obj, *args, **kwargs)
  File "C:\Users\BMandal2\bootstrap\inso-bootstrap-cli\incubator\bootstrap_cli\__main__.py", line 2252, in prepare
    BootstrapCore(config_file, command=CommandMode.PREPARE, debug=obj["debug"])
  File "C:\Users\BMandal2\bootstrap\inso-bootstrap-cli\incubator\bootstrap_cli\__main__.py", line 1377, in prepare
    _logger.debug(f"GROUPS in CDF:\n{self.deployed.groups}")
AttributeError: 'NoneType' object has no attribute 'groups'
 

 

Error when added logger section along with cognite section:

 

Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "C:\Users\BMandal2\bootstrap\inso-bootstrap-cli\incubator\bootstrap_cli\__main__.py", line 2365, in main
    bootstrap_cli()
  File "C:\Users\BMandal2\AppData\Local\pypoetry\Cache\virtualenvs\inso-bootstrap-cli-Jo_B4J-G-py3.10\lib\site-packages\click\core.py", line 1128, in __call__
    return self.main(*args, **kwargs)
  File "C:\Users\BMandal2\AppData\Local\pypoetry\Cache\virtualenvs\inso-bootstrap-cli-Jo_B4J-G-py3.10\lib\site-packages\click\core.py", line 1053, in main
    rv = self.invoke(ctx)
  File "C:\Users\BMandal2\AppData\Local\pypoetry\Cache\virtualenvs\inso-bootstrap-cli-Jo_B4J-G-py3.10\lib\site-packages\click\core.py", line 1659, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "C:\Users\BMandal2\AppData\Local\pypoetry\Cache\virtualenvs\inso-bootstrap-cli-Jo_B4J-G-py3.10\lib\site-packages\click\core.py", line 1395, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "C:\Users\BMandal2\AppData\Local\pypoetry\Cache\virtualenvs\inso-bootstrap-cli-Jo_B4J-G-py3.10\lib\site-packages\click\core.py", line 754, in invoke
    return __callback(*args, **kwargs)
  File "C:\Users\BMandal2\AppData\Local\pypoetry\Cache\virtualenvs\inso-bootstrap-cli-Jo_B4J-G-py3.10\lib\site-packages\click\decorators.py", line 38, in new_func
    return f(get_current_context().obj, *args, **kwargs)
  File "C:\Users\BMandal2\bootstrap\inso-bootstrap-cli\incubator\bootstrap_cli\__main__.py", line 2252, in prepare
    BootstrapCore(config_file, command=CommandMode.PREPARE, debug=obj["debug"])
  File "C:\Users\BMandal2\bootstrap\inso-bootstrap-cli\incubator\bootstrap_cli\__main__.py", line 525, in __init__
    self.client: CogniteClient = self.config.cognite.get_cognite_client(  # noqa
  File "C:\Users\BMandal2\AppData\Local\pypoetry\Cache\virtualenvs\inso-bootstrap-cli-Jo_B4J-G-py3.10\lib\site-packages\cognite\extractorutils\configtools.py", line 222, in get_cognite_client
    return CogniteClient(
  File "C:\Users\BMandal2\AppData\Local\pypoetry\Cache\virtualenvs\inso-bootstrap-cli-Jo_B4J-G-py3.10\lib\site-packages\cognite\client\_cognite_client.py", line 85, in __init__
    self._config = ClientConfig(
  File "C:\Users\BMandal2\AppData\Local\pypoetry\Cache\virtualenvs\inso-bootstrap-cli-Jo_B4J-G-py3.10\lib\site-packages\cognite\client\utils\_client_config.py", line 94, in __init__
    token_generator = utils._token_generator.TokenGenerator(
  File "C:\Users\BMandal2\AppData\Local\pypoetry\Cache\virtualenvs\inso-bootstrap-cli-Jo_B4J-G-py3.10\lib\site-packages\cognite\client\utils\_token_generator.py", line 21, in __init__
    self._generate_access_token()
  File "C:\Users\BMandal2\AppData\Local\pypoetry\Cache\virtualenvs\inso-bootstrap-cli-Jo_B4J-G-py3.10\lib\site-packages\cognite\client\utils\_token_generator.py", line 49, in _generate_access_token
    raise CogniteAPIKeyError(
cognite.client.exceptions.CogniteAPIKeyError: Error generating access token: missing_token, 400, Missing access token parameter.
 

Peter Arwanitis
Rank
Userlevel 2
Badge

Hi Basudeba,

 

just for your confirmation:

  • the `prepare` command requires to access Cognite with a service-account (an AAD App)
  • which requires the `cognite` section available too
  • as documented here
    • it is using a `.env` file to separate the secrets from the configuration
cognite:
  host: ${BOOTSTRAP_CDF_HOST}
  project: ${BOOTSTRAP_CDF_PROJECT}
  #
  # IdP login:
  #
  idp-authentication:
    client-id: ${BOOTSTRAP_IDP_CLIENT_ID}
    secret: ${BOOTSTRAP_IDP_CLIENT_SECRET}
    scopes:
      - ${BOOTSTRAP_IDP_SCOPES}
    token_url: ${BOOTSTRAP_IDP_TOKEN_URL}

Which requires -- beside others -- a `BOOTSTRAP_IDP_CLIENT_ID / BOOTSTRAP_IDP_CLIENT_SECRET` (client/secrets) of an AAD Application for service-level authentication.

  • which you had to create first in your AAD tenant
  • because this (first) AAD App is used for root access to Cognite, a typical name could be “CDF_DEV_ROOT_CLIENT” (“DEV” or “TEST”  or “PROD” in case SLB has staging projects connected to the same SLB AAD)
  • which must be a member of the same AAD Group which was used requesting the CDF Project => “Admin group source ID *”

I hope this makes sense, and I’m happy to follow up and improve the documentation after our session.

 

Bets regards

Peter

(=PA=)

B
Rank

Thanks Peter!

I had missed .env file.  Now I can connect to the slb-pdf project. However I still see “API Failed to process some items” error with 403 unauthorized error code. As per the error code, It looks like the client id/secrets that I use does not have enough permission to perform the actions. 

 

Error Snippet:

 

Prepare CDF Project ...
2023-06-01 08:30:48.821 UTC [INFO    ] MainThread - Starting CDF Bootstrap configuration for command: <prepare>
2023-06-01 08:30:49.555 UTC [INFO    ] MainThread - Successful connection to CDF client to project: 'slb-pdf'
2023-06-01 08:30:50.114 UTC [INFO    ] MainThread - Deployed CDF Resource counts:
            RAW Dbs(n/a with this command)
            Data Sets(n/a with this command)
            CDF Groups(6)
Traceback (most recent call last):
  File "<string>", line 1, in <module>


  File "C:\Users\BMandal2\AppData\Local\pypoetry\Cache\virtualenvs\inso-bootstrap-cli-Jo_B4J-G-py3.10\lib\site-packages\cognite\client\utils\_concurrency.py", line 85, in collect_exc_info_and_raise
    raise CogniteAPIError(
cognite.client.exceptions.CogniteAPIError: Unauthorized | code: 403 | X-Request-ID: 94abaa61-3e37-93c4-9082-08c7d9a35207
The API Failed to process some items.
Successful (2xx): []
Unknown (5xx): []
Failed (4xx): [{'name': 'cdf:bootstrap', 'source_id': 'f2e5b1ef-e601-46ec-87ec-25f62810ad1f', 'capabilities': [{'datasetsAcl': {'actions': ['READ', 'WRITE', 'OWNER'], 'scope': {'all': {}}}}, {'rawAcl': {'actions': ['READ', 'WRITE', 'LIST'], 'scope': {'all': {}}}}, {'groupsAcl': {'actions': ['LIST', 'READ', 'CREATE', 'UPDATE', 'DELETE'], 'scope': {'all': {}}}}, {'projectsAcl': {'actions': ['READ', 'UPDATE'], 'scope': {'all': {}}}}]}]
 

I am using a set of client id/secrets in the .env file along with other environment variables. Do we need to provide additional roles/permissions so that It can perform the actions without 403 ?

 

 

Peter Arwanitis
Rank
Userlevel 2
Badge

Hi Basudeba,

 

> I had missed .env file.

Not mandatory, just the recommended practice not storing secrets in config files, which maybe get added to a version-control-system (like git)

 

The permission problem of your AAD App should not happen, as long as you have added the app to the same AAD Group which you used for requesting the CDF Project.

 

“Typically” the very first CDF Group setup and linked to this AAD Group is named “oidc-admin-group” in Cognite, and as long as your user has the permissions, you open it from “Manage > Access Management” select the group, go to edit and copy the “Source ID” (which is the AAD Group object-id)?

 

This (screenshot) are the required capabilities in Cognite, the AAD App requires (through its AAD Group > Cognite linking)

 


I’m sorry if this sounds over-complicated, but I try to provide enough information and context, hopefully to help you navigate and solving your problem ASAP.

 

Best regards

Peter

(=PA=)

J
Rank
Userlevel 4
Badge

@Basudeba Mandal ,
I can help administer the clientId and secret you need to to access slb-pdf.  I’ve reached out on Teams to verify and share with you the correct credentials.


Jason

B
Rank

Thanks @Jason Dressel, appreciate your help. I was able to run “bootstrap-cli preparae” command successfully. Will progress further and reach out to you guys in case of any further issues. We can move this to closure.

 

Regards,

Basu

Reply


Terms & ConditionsCookie Settings