Skip to main content
Solved

Enable logging to be accessible from Extraction Pipeline UI

S
Committed
Forum|alt.badge.img

Hi Team, I want to change the message for the custom extractor’s run history message by python sdk. Right now, message says “successful shutdown” after successfully running the extractor, but I want to change it with how many datapoints are uploading and in how much time.
I tried to do it by following methods:

  1. By using abstract class Extractor in our code to overwrite the _report_success method’s message.

       2.By using this method 

 cognite.extractorutils.util.add_extraction_pipeline(extraction_pipeline_ext_id: str, cognite_client: CogniteClient, heartbeat_waiting_time: int = 600, added_message: str = '')

       3. By using below class too:

cclient.data_classes.extractionpipelines.ExtractionPipeline(id: int | None = None, external_id: str | None = None, last_message: str | None = None)

 

 

There is no way of passing the custom message here in my knowledge if it’s there let me know!

And is there any way to pass logs in extraction pipeline?

Best answer by Jason Dressel

@Shweta 
Sorry for the delay.  It appears there may have been a bug in the SDK that is now fixed.  Please try with the following code snippet:

run = ExtractionPipelineRunWrite(extpipe_external_id="csv:oee:datapoints", status="failure", message="custom error message")
result = client.extraction_pipelines.runs.create(run)
result

-Jason

View original
Anita Hæhre
1 person likes this
  • Anita Hæhre
    Anita Hæhre
Did this topic help you find an answer to your question?

6 replies

J
Seasoned Practitioner
Forum|alt.badge.img
  • Jason Dressel
  • Seasoned Practitioner
  • 285 replies
  • January 10, 2024

@Shweta 

Each extraction run can have it’s own message: https://cognite-sdk-python.readthedocs-hosted.com/en/latest/data_ingestion.html#cognite.client._api.extractionpipelines.ExtractionPipelineRunsAPI.create

https://cognite-sdk-python.readthedocs-hosted.com/en/latest/data_ingestion.html#cognite.client.data_classes.extractionpipelines.ExtractionPipelineRunWrite
 

Is this what you need?

-Jason

S
Committed
Forum|alt.badge.img
  • Shweta
  • Author
  • Committed
  • 6 replies
  • January 10, 2024

Hi @Jason Dressel thank you so much for your reply but I can’t find  this class ExtractionPipelineRunWrite in data_classes library! that’s why I am unable to import it for my use. 
 

J
Seasoned Practitioner
Forum|alt.badge.img
  • Jason Dressel
  • Seasoned Practitioner
  • 285 replies
  • January 10, 2024

@Shweta what version of the python sdk are you using?  You many need to upgrade.
Try just using ExtractionPipelineRun

-Jason

S
Committed
Forum|alt.badge.img
  • Shweta
  • Author
  • Committed
  • 6 replies
  • January 11, 2024

       

Yes @Jason Dressel  upgraded the version and used the above classes like this and I am getting below error!
creds = OAuthClientCredentials(

        token_url=clientConfig.token_url, 

        client_id=clientConfig.client_id, 

        client_secret=clientConfig.client_secret, 

        scopes=clientConfig.scope

        )

 

    cnf = ClientConfig(client_name=clientConfig.project, project=clientConfig.project_name, credentials=creds, base_url=clientConfig.base_url)

    client = CogniteClient(cnf)


class usage:
    newSuccessRun = client.extraction_pipelines.create(ExtractionPipelineRunWrite( extpipe_external_id="your_ext_id",status="success", message="Error message1"))

 

and same below error with ExtractionPipelineRun.

Error:

TypeError: 'extraction_pipeline' must be one of types [<class 'cognite.client.data_classes.extractionpipelines.ExtractionPipelineCore'>, typing.Sequence], not <class 'cognite.client.data_classes.extractionpipelines.ExtractionPipelineRunWrite'>
 

can you help me with this?

J
Seasoned Practitioner
Forum|alt.badge.img
  • Jason Dressel
  • Seasoned Practitioner
  • 285 replies
  • Answer
  • February 22, 2024

@Shweta 
Sorry for the delay.  It appears there may have been a bug in the SDK that is now fixed.  Please try with the following code snippet:

run = ExtractionPipelineRunWrite(extpipe_external_id="csv:oee:datapoints", status="failure", message="custom error message")
result = client.extraction_pipelines.runs.create(run)
result

-Jason

S
Committed
Forum|alt.badge.img
  • Shweta
  • Author
  • Committed
  • 6 replies
  • February 23, 2024

Yes, thank you very much for you help, issue is resolved.

Reply


Terms & ConditionsCookie Settings

Cookie Policy

We use cookies to enhance and personalize your experience. If you accept you agree to our full cookie policy. Learn more about our cookies.

 
Cookie Settings