Solved

Enable logging to be accessible from Extraction Pipeline UI

  • 10 January 2024
  • 6 replies
  • 103 views
S
Rank
Badge

Hi Team, I want to change the message for the custom extractor’s run history message by python sdk. Right now, message says “successful shutdown” after successfully running the extractor, but I want to change it with how many datapoints are uploading and in how much time.
I tried to do it by following methods:

  1. By using abstract class Extractor in our code to overwrite the _report_success method’s message.

       2.By using this method 

 cognite.extractorutils.util.add_extraction_pipeline(extraction_pipeline_ext_id: str, cognite_client: CogniteClient, heartbeat_waiting_time: int = 600, added_message: str = '')

       3. By using below class too:

cclient.data_classes.extractionpipelines.ExtractionPipeline(id: int | None = None, external_id: str | None = None, last_message: str | None = None)

 

 

There is no way of passing the custom message here in my knowledge if it’s there let me know!

And is there any way to pass logs in extraction pipeline?

icon

Best answer by Jason Dressel 22 February 2024, 19:38

View original

6 replies

J
Rank
Userlevel 4
Badge

@Shweta 

Each extraction run can have it’s own message: https://cognite-sdk-python.readthedocs-hosted.com/en/latest/data_ingestion.html#cognite.client._api.extractionpipelines.ExtractionPipelineRunsAPI.create

https://cognite-sdk-python.readthedocs-hosted.com/en/latest/data_ingestion.html#cognite.client.data_classes.extractionpipelines.ExtractionPipelineRunWrite
 

Is this what you need?

-Jason

S
Rank
Badge

Hi @Jason Dressel thank you so much for your reply but I can’t find  this class ExtractionPipelineRunWrite in data_classes library! that’s why I am unable to import it for my use. 
 

J
Rank
Userlevel 4
Badge

@Shweta what version of the python sdk are you using?  You many need to upgrade.
Try just using ExtractionPipelineRun

-Jason

S
Rank
Badge

       

Yes @Jason Dressel  upgraded the version and used the above classes like this and I am getting below error!
creds = OAuthClientCredentials(

        token_url=clientConfig.token_url, 

        client_id=clientConfig.client_id, 

        client_secret=clientConfig.client_secret, 

        scopes=clientConfig.scope

        )

 

    cnf = ClientConfig(client_name=clientConfig.project, project=clientConfig.project_name, credentials=creds, base_url=clientConfig.base_url)

    client = CogniteClient(cnf)


class usage:
    newSuccessRun = client.extraction_pipelines.create(ExtractionPipelineRunWrite( extpipe_external_id="your_ext_id",status="success", message="Error message1"))

 

and same below error with ExtractionPipelineRun.

Error:

TypeError: 'extraction_pipeline' must be one of types [<class 'cognite.client.data_classes.extractionpipelines.ExtractionPipelineCore'>, typing.Sequence], not <class 'cognite.client.data_classes.extractionpipelines.ExtractionPipelineRunWrite'>
 

can you help me with this?

J
Rank
Userlevel 4
Badge

@Shweta 
Sorry for the delay.  It appears there may have been a bug in the SDK that is now fixed.  Please try with the following code snippet:

run = ExtractionPipelineRunWrite(extpipe_external_id="csv:oee:datapoints", status="failure", message="custom error message")
result = client.extraction_pipelines.runs.create(run)
result

-Jason

S
Rank
Badge

Yes, thank you very much for you help, issue is resolved.

Reply


Terms & ConditionsCookie Settings