Cognite Custom Extractor Util Config

It’s same as with the Cognite relased extractors. You have a local file with type:remote which refers to the extraction pipeline which contains the config.

Sample local config:

type: remote

cognite:

    # Read these from environment variables

    host: ${COGNITE_BASE_URL}

    project: ${COGNITE_PROJECT}

    idp-authentication:

        token-url: ${COGNITE_TOKEN_URL}

        client-id: ${COGNITE_CLIENT_ID}

        secret: ${COGNITE_CLIENT_SECRET}

        scopes:

            - ${COGNITE_BASE_URL}/.default

    extraction-pipeline:

        external-id: ${COGNITE_EXTRACTION_PIPELINE}

@Murad Sæter I’ve added more details in post, is the expectation valid from remote and local configs?

What are you trying to achieve here? You cannot have local and remote in the same config.

If you have a local config with the extractor it will only read the local config file.

If you have type:remote with the extractor it should only contain details around connecting to CDF (cognite section). The extractor will then connect to CDF and read all config from the extraction pipeline.

I want to merge custom config entries from local config and remote

I understand that ‘type: remote’ allows only ‘cognite:’ configs from host and rest from extraction pipeline. Is there a way to get custom config-entries from local config file?

host config file:

type: remote
cognite:
    extraction_pipeline:
        external_id: extraction-pipeline-id
custom:
    property1: some-value

config in extraction pipeline:
custom:
    property2: some-value

No, that is not a supported use case.

If you need different values per extractor-machine locally you could use environment variables in the config stored in the extraction pipeline.

Thank @Murad Sæter

Hi

Im wondering with this approach, where does the actual variables value ${} is stored?

Thank you