Solved

Cognite Custom Extractor Util Config

  • 13 February 2024
  • 7 replies
  • 63 views
M
Rank
Userlevel 1

How can we use hybrid configs (local and remote) in custom extractor utils?

Edit: while trying to consume combination of local and remote configs (published in extraction pipeline mentioned in local config), it just loads local configs.

 

example configs:

type: local
cognite:
    extraction_pipeline:
        external_id: extraction-pipeline-id
custom:
    property1: some-value

type: remote
custom:
    property2: some-value

 

icon

Best answer by Murad Sæter 14 February 2024, 12:26

View original

7 replies

Murad Sæter
Rank
Userlevel 2

It’s same as with the Cognite relased extractors. You have a local file with type:remote which refers to the extraction pipeline which contains the config.

 

Sample local config:

type: remote

cognite:

    # Read these from environment variables

    host: ${COGNITE_BASE_URL}

    project: ${COGNITE_PROJECT}

    idp-authentication:

        token-url: ${COGNITE_TOKEN_URL}

        client-id: ${COGNITE_CLIENT_ID}

        secret: ${COGNITE_CLIENT_SECRET}

        scopes:

            - ${COGNITE_BASE_URL}/.default

    extraction-pipeline:

        external-id: ${COGNITE_EXTRACTION_PIPELINE}

M
Rank
Userlevel 1

@Murad Sæter I’ve added more details in post, is the expectation valid from remote and local configs?

Murad Sæter
Rank
Userlevel 2

What are you trying to achieve here? You cannot have local and remote in the same config.

If you have a local config with the extractor it will only read the local config file.

If you have type:remote with the extractor it should only contain details around connecting to CDF (cognite section). The extractor will then connect to CDF and read all config from the extraction pipeline.

M
Rank
Userlevel 1

I want to merge custom config entries from local config and remote

I understand that ‘type: remote’ allows only ‘cognite:’ configs from host and rest from extraction pipeline. Is there a way to get custom config-entries from local config file?

 

host config file:

type: remote
cognite:
    extraction_pipeline:
        external_id: extraction-pipeline-id
custom:
    property1: some-value

 

config in extraction pipeline:
custom:
    property2: some-value

Murad Sæter
Rank
Userlevel 2

No, that is not a supported use case.

If you need different values per extractor-machine locally you could use environment variables in the config stored in the extraction pipeline.

M
Rank
Userlevel 1

Thank @Murad Sæter

S
Rank
Badge

It’s same as with the Cognite relased extractors. You have a local file with type:remote which refers to the extraction pipeline which contains the config.

 

Sample local config:

type: remote cognite:     # Read these from environment variables     host: ${COGNITE_BASE_URL}     project: ${COGNITE_PROJECT}

    idp-authentication:         token-url: ${COGNITE_TOKEN_URL}

        client-id: ${COGNITE_CLIENT_ID}         secret: ${COGNITE_CLIENT_SECRET}         scopes:             - ${COGNITE_BASE_URL}/.default

    extraction-pipeline:         external-id: ${COGNITE_EXTRACTION_PIPELINE}

Hi

Im wondering with this approach, where does the actual variables value ${} is stored?

 

Thank you

Reply


Terms & ConditionsCookie Settings