Solved

Cognite Custom Extractor Util Config

  • 13 February 2024
  • 7 replies
  • 62 views

Userlevel 1

How can we use hybrid configs (local and remote) in custom extractor utils?

Edit: while trying to consume combination of local and remote configs (published in extraction pipeline mentioned in local config), it just loads local configs.

 


example configs:


type: local
cognite:
    extraction_pipeline:
        external_id: extraction-pipeline-id
custom:
    property1: some-value


type: remote
custom:
    property2: some-value


 

icon

Best answer by Murad Sæter 14 February 2024, 12:26

View original

7 replies

Userlevel 2

It’s same as with the Cognite relased extractors. You have a local file with type:remote which refers to the extraction pipeline which contains the config.

 

Sample local config:

type: remote

cognite:

    # Read these from environment variables

    host: ${COGNITE_BASE_URL}

    project: ${COGNITE_PROJECT}

    idp-authentication:

        token-url: ${COGNITE_TOKEN_URL}

        client-id: ${COGNITE_CLIENT_ID}

        secret: ${COGNITE_CLIENT_SECRET}

        scopes:

            - ${COGNITE_BASE_URL}/.default

    extraction-pipeline:

        external-id: ${COGNITE_EXTRACTION_PIPELINE}

Userlevel 1

@Murad Sæter I’ve added more details in post, is the expectation valid from remote and local configs?

Userlevel 2

What are you trying to achieve here? You cannot have local and remote in the same config.

If you have a local config with the extractor it will only read the local config file.

If you have type:remote with the extractor it should only contain details around connecting to CDF (cognite section). The extractor will then connect to CDF and read all config from the extraction pipeline.

Userlevel 1

I want to merge custom config entries from local config and remote

I understand that ‘type: remote’ allows only ‘cognite:’ configs from host and rest from extraction pipeline. Is there a way to get custom config-entries from local config file?

 

host config file:

type: remote
cognite:
    extraction_pipeline:
        external_id: extraction-pipeline-id
custom:
    property1: some-value

 

config in extraction pipeline:
custom:
    property2: some-value

Userlevel 2

No, that is not a supported use case.

If you need different values per extractor-machine locally you could use environment variables in the config stored in the extraction pipeline.

Userlevel 1

Thank @Murad Sæter

Badge

It’s same as with the Cognite relased extractors. You have a local file with type:remote which refers to the extraction pipeline which contains the config.

 

Sample local config:

type: remote cognite:     # Read these from environment variables     host: ${COGNITE_BASE_URL}     project: ${COGNITE_PROJECT}

    idp-authentication:         token-url: ${COGNITE_TOKEN_URL}

        client-id: ${COGNITE_CLIENT_ID}         secret: ${COGNITE_CLIENT_SECRET}         scopes:             - ${COGNITE_BASE_URL}/.default

    extraction-pipeline:         external-id: ${COGNITE_EXTRACTION_PIPELINE}

Hi

Im wondering with this approach, where does the actual variables value ${} is stored?

 

Thank you

Reply