Skip to main content
Solved

Cognite Custom Extractor Util Config


How can we use hybrid configs (local and remote) in custom extractor utils?

Edit: while trying to consume combination of local and remote configs (published in extraction pipeline mentioned in local config), it just loads local configs.

 


example configs:


type: local
cognite:
    extraction_pipeline:
        external_id: extraction-pipeline-id
custom:
    property1: some-value


type: remote
custom:
    property2: some-value


 

Best answer by Murad Sæter

No, that is not a supported use case.

If you need different values per extractor-machine locally you could use environment variables in the config stored in the extraction pipeline.

View original
Did this topic help you find an answer to your question?

7 replies

Murad Sæter
Practitioner
  • Practitioner
  • 48 replies
  • February 13, 2024

It’s same as with the Cognite relased extractors. You have a local file with type:remote which refers to the extraction pipeline which contains the config.

 

Sample local config:

type: remote

cognite:

    # Read these from environment variables

    host: ${COGNITE_BASE_URL}

    project: ${COGNITE_PROJECT}

    idp-authentication:

        token-url: ${COGNITE_TOKEN_URL}

        client-id: ${COGNITE_CLIENT_ID}

        secret: ${COGNITE_CLIENT_SECRET}

        scopes:

            - ${COGNITE_BASE_URL}/.default

    extraction-pipeline:

        external-id: ${COGNITE_EXTRACTION_PIPELINE}


  • Author
  • Committed
  • 12 replies
  • February 14, 2024

@Murad Sæter I’ve added more details in post, is the expectation valid from remote and local configs?


Murad Sæter
Practitioner
  • Practitioner
  • 48 replies
  • February 14, 2024

What are you trying to achieve here? You cannot have local and remote in the same config.

If you have a local config with the extractor it will only read the local config file.

If you have type:remote with the extractor it should only contain details around connecting to CDF (cognite section). The extractor will then connect to CDF and read all config from the extraction pipeline.


  • Author
  • Committed
  • 12 replies
  • February 14, 2024

I want to merge custom config entries from local config and remote

I understand that ‘type: remote’ allows only ‘cognite:’ configs from host and rest from extraction pipeline. Is there a way to get custom config-entries from local config file?

 

host config file:

type: remote
cognite:
    extraction_pipeline:
        external_id: extraction-pipeline-id
custom:
    property1: some-value

 

config in extraction pipeline:
custom:
    property2: some-value


Murad Sæter
Practitioner
  • Practitioner
  • 48 replies
  • Answer
  • February 14, 2024

No, that is not a supported use case.

If you need different values per extractor-machine locally you could use environment variables in the config stored in the extraction pipeline.


  • Author
  • Committed
  • 12 replies
  • February 14, 2024

Forum|alt.badge.img
Murad Sæter wrote:

It’s same as with the Cognite relased extractors. You have a local file with type:remote which refers to the extraction pipeline which contains the config.

 

Sample local config:

type: remote cognite:     # Read these from environment variables     host: ${COGNITE_BASE_URL}     project: ${COGNITE_PROJECT}

    idp-authentication:         token-url: ${COGNITE_TOKEN_URL}

        client-id: ${COGNITE_CLIENT_ID}         secret: ${COGNITE_CLIENT_SECRET}         scopes:             - ${COGNITE_BASE_URL}/.default

    extraction-pipeline:         external-id: ${COGNITE_EXTRACTION_PIPELINE}

Hi

Im wondering with this approach, where does the actual variables value ${} is stored?

 

Thank you


Reply


Cookie Policy

We use cookies to enhance and personalize your experience. If you accept you agree to our full cookie policy. Learn more about our cookies.

 
Cookie Settings