Creating timeseries external Ids by extractor

Hi Amit,

It’s Viraj from Cognite Support.

I need more information about the extractor and the config file to address your issue.

So I will create a support ticket for your issue and contact you via it.

I hope it's okay for you going forward on this issue.

Best regards, 
Viraj 

 Hi @Viraj Dhanusha,

I am currently using Cognite DB Extractor (dbextractor-winservice-v2.6.0)

with version - v2.6.0

and config file:

version: 2
logger:
 console:
  level: DEBUG
 file:
  level: DEBUG
  path: "file_2023.log"
cognite:
 host: "https://westeurope-1.cognitedata.com"
 project: slb-pdf
 # (Either this or api-key is required) OIDC setup for CDF auth
 idp-authentication:
# OIDC client ID
  client-id: ${COGNITE_CLIENT_ID}
# OIDC client secret
  secret: ${COGNITE_CLIENT_SECRET}
# URL to fetch OIDC tokens from
  token-url: "https://login.microsoftonline.com/45472a8e-6a12-4531-97d8-1f38f5d553a4/oauth2/v2.0/token"
# List of OIDC scopes to request
  scopes:
    #- "https://api.cognitedata.com/.default"
    - "https://westeurope-1.cognitedata.com/.default"

 data-set:
   id: 1409796585286854
 #extraction-pipeline:
 #  external-id: "Historical_Test_TS_pipeline-ExtID"

extractor:
# (Optional) Number of rows to fetch from sources before uploading
  upload-queue-size: 10000
# (Optional) Where to store extraction states (progress) between runs.
# Required for incremental load to work.
  state-store:
# Uncomment to use a local json file for state storage
  #local:
    #path: C:\Users\ARane3\Desktop\odf\ingestion\cognite-db-extractor\state.json
# (Optional) Save interval (in seconds) for intermediate file saves.
# A final save will also be made on extractor shutdown.
    #save-interval: 30
# Uncomment to use a RAW table for state storage
    raw:
# RAW database and table to use
      database: "src:001:avocet:pdm:state"
      table: "src:001:avocet:pdm:state:table"
# (Optional) Upload interval (pin seconds) for intermediate uploads.
# A final upload will also be made on extractor shutdown.
      upload-interval: 300

  #upload-queue-size: 2

#metrics:
#  push-gateways:
#    - host: http://localhost:9091
#      job-name: db-extractor-avocet-metrics-job-1

    # Multiple Prometheus PushGateway destinations:
#  cognite:
#    external-id-prefix: "dbextractor-avocet-metrics-prefix:"
#    push-interval: 5

databases:
 -
  name: CDF_SOURCE_DATA #this is your connection identifier name
  connection-string: "DSN=incremental-data;Uid=Uid;Pwd=Pwd" # dsn+user_id+password
  timeout: 0
queries:

 -
  name: WELL_READ_STRING
# Name of database to use (as specified in the databases section)
  database: CDF_SOURCE_DATA # same as above database

# Query to execute. Supports interpolation with {incremental_field}
# and{start_at} to facilitate incremental load between runs
  query: SELECT distinct CONCAT (ITEM_ID, ':', ITEM_TYPE, ':', PROPERTY COLLATE DATABASE_DEFAULT) AS externalId,START_DATETIME AS timestamp, PROPERTY_VALUE AS value FROM
        (SELECT T1.ITEM_ID, START_DATETIME, LAST_UPDT_DATE, T1.ITEM_TYPE, ITEM_NAME
        FROM VT_WELL_READ_en_US as T1 inner JOIN
        (select I_P.ITEM_ID, I_L.FROM_ITEM_ID, I_L.TO_ITEM_ID, I_P.ITEM_TYPE AS ITEM_TYPE, I_P.PROPERTY_STRING AS NAME
        FROM ITEM_PROPERTY AS I_P
        inner join ITEM_LINK AS I_L on I_P.ITEM_ID =I_L.FROM_ITEM_ID  WHERE I_L.LINK_TYPE IN ('BATTERY_ITEM', 'ORG_ITEM')) as t2
        on T1.ITEM_ID=t2.ITEM_ID WHERE ITEM_NAME IS NOT NULL) AS WELL_PROP_DATA
        UNPIVOT
        (PROPERTY_VALUE FOR PROPERTY IN (ITEM_NAME))
        AS UNPIVOT_PROPERTY

  destination-type: TIME_SERIES
  incremental-field: timestamp
  schedule: "*/10 * * * *"
  initial-start: '1900-01-01'

correct time series using currently - fb517df6da3143418087eea39b2ef2bd:WELL_READ:TUBING_PRESS

external id of a created time-series by the extractor which not need in the dataset - fb9fe3a9d6c64418a394d3452b967e61:WELL_READ:TUBING_TEMP

I have taken only one extrenalId there are many extrenalIds

Please let me know anything else needed from my end.

Thanks,

Amit

Hi Amit,

Thank you for the information.

I have already created a support ticket and started communication via it.

Can you please reply to it with this information so that we can track down the content and work with engineering if needed. 

Best regards,

Viraj 

Hi @Amit Rane,

I hope Viraj’s reply was helpful. Let us know if you need further assistance.

Best regards,
Dilini