Skip to main content
Seasoned ⭐️⭐️
Shweta
Seasoned ⭐️⭐️
Jan 10, 2024 11:43 AM UTC
Solved

Enable logging to be accessible from Extraction Pipeline UI

Hi Team, I want to change the message for the custom extractor’s run history message by python sdk. Right now, message says “successful shutdown” after successfully running the extractor, but I want to change it with how many datapoints are uploading and in how much time.
I tried to do it by following methods:

  1. By using abstract class Extractor in our code to overwrite the _report_success method’s message.

       2.By using this method 

 cognite.extractorutils.util.add_extraction_pipeline(extraction_pipeline_ext_id: str, cognite_client: CogniteClient, heartbeat_waiting_time: int = 600, added_message: str = '')

       3. By using below class too:

cclient.data_classes.extractionpipelines.ExtractionPipeline(id: int | None = None, external_id: str | None = None, last_message: str | None = None)

 

 

There is no way of passing the custom message here in my knowledge if it’s there let me know!

And is there any way to pass logs in extraction pipeline?

    Jason DresselAnswer
    Expert ⭐️⭐️⭐️⭐️
    Feb 22, 2024 6:38 PM UTC

    @Shweta 
    Sorry for the delay.  It appears there may have been a bug in the SDK that is now fixed.  Please try with the following code snippet:

    run = ExtractionPipelineRunWrite(extpipe_external_id="csv:oee:datapoints", status="failure", message="custom error message")
    result = client.extraction_pipelines.runs.create(run)
    result

    -Jason

    View in thread

    6 replies

    Seasoned ⭐️⭐️
    ShwetaAuthor
    Seasoned ⭐️⭐️
    Jan 10, 2024 6:18 PM UTC

    Hi @Jason Dressel thank you so much for your reply but I can’t find  this class ExtractionPipelineRunWrite in data_classes library! that’s why I am unable to import it for my use. 
     

    Expert ⭐️⭐️⭐️⭐️
    Jason Dressel
    Expert ⭐️⭐️⭐️⭐️
    Jan 10, 2024 6:32 PM UTC

    @Shweta what version of the python sdk are you using?  You many need to upgrade.
    Try just using ExtractionPipelineRun

    -Jason

    Seasoned ⭐️⭐️
    ShwetaAuthor
    Seasoned ⭐️⭐️
    Jan 11, 2024 10:15 AM UTC

           

    Yes @Jason Dressel  upgraded the version and used the above classes like this and I am getting below error!
    creds = OAuthClientCredentials(

            token_url=clientConfig.token_url, 

            client_id=clientConfig.client_id, 

            client_secret=clientConfig.client_secret, 

            scopes=clientConfig.scope

            )

     

        cnf = ClientConfig(client_name=clientConfig.project, project=clientConfig.project_name, credentials=creds, base_url=clientConfig.base_url)

        client = CogniteClient(cnf)


    class usage:
        newSuccessRun = client.extraction_pipelines.create(ExtractionPipelineRunWrite( extpipe_external_id="your_ext_id",status="success", message="Error message1"))

     

    and same below error with ExtractionPipelineRun.

    Error:

    TypeError: 'extraction_pipeline' must be one of types [<class 'cognite.client.data_classes.extractionpipelines.ExtractionPipelineCore'>, typing.Sequence], not <class 'cognite.client.data_classes.extractionpipelines.ExtractionPipelineRunWrite'>
     

    can you help me with this?

    Expert ⭐️⭐️⭐️⭐️
    Jason DresselAnswer
    Expert ⭐️⭐️⭐️⭐️
    Feb 22, 2024 6:38 PM UTC

    @Shweta 
    Sorry for the delay.  It appears there may have been a bug in the SDK that is now fixed.  Please try with the following code snippet:

    run = ExtractionPipelineRunWrite(extpipe_external_id="csv:oee:datapoints", status="failure", message="custom error message")
    result = client.extraction_pipelines.runs.create(run)
    result

    -Jason

    Seasoned ⭐️⭐️
    ShwetaAuthor
    Seasoned ⭐️⭐️
    Feb 23, 2024 2:03 PM UTC

    Yes, thank you very much for you help, issue is resolved.

    Terms & ConditionsAccessibility statement