Skip to main content
Solved

transformations-cli with FDM


Hi,

I am trying to run transformations-cli locally with below command:

transformations-cli deploy . 

(The current directory has manifest.yaml and transformation.sql files)

 

I am getting below error:

Deploying transformations...
Failed to parse transformation config, please check that you conform required fields and format: Invalid config: can not match type "dict" to any type of "destination" union: typing.Union[cognite.transformations_cli.commands.deploy.transformation_types.DestinationType, cognite.transformations_cli.commands.deploy.transformation_types.DestinationConfig, cognite.transformations_cli.commands.deploy.transformation_types.RawDestinationConfig, cognite.transformations_cli.commands.deploy.transformation_types.SequenceRowsDestinationConfig, cognite.transformations_cli.commands.deploy.transformation_types.AlphaDMIDestinationConfig]

 

Here is destination section of manifest file:

destination:
  viewSpaceExternalId: my-model-space-id
  viewExternalId: CDFTimeSeries
  viewVersion: 0_2
  instanceSpaceExternalId: my-model-space-id
  type: instances

 

Please let me know if I am missing anything.

 

Thanks

 

Best answer by Dilini Fernando

Hi @Niranjan Madhukar Karvekar,

I hope Jason’s reply has helped you to solve the issue. As of now, I will close this thread. Please feel free to create a new post if you still experience problems.

Br,
Dilini

View original
Did this topic help you find an answer to your question?

Forum|alt.badge.img

Hi Niranjan,

Can you share with me your entire transformation config yaml?  Here or via Teams :).


Jason


Forum|alt.badge.img

Hi, it looks like you did not use the latest version of the cli, since we already renamed AlphaDMIDestinationConfig not DMIDestinationConfig , and the new type of FDM is InstancesDestinationConfig


Hi,

Thanks for your reply. I tried with latest transformations-cli (2.3.0) but I see the same error. I generated manifest file from CDF UI. Can you please provide me the example of manifest file with destination as FDM model? 

Thanks.

 


Forum|alt.badge.img

Hi,
Ah right you need to use the syntax with _ instead of camel case I think :)
https://github.com/cognitedata/transformations-cli/blob/main/tests/test_deploy.py#L185-L190

        destination:
            type: instances
            view_external_id: test_view
            view_version: test_view_version
            view_space_external_id: test_space
            instance_space_external_id: test_space

 


Hi,

 

Thanks for your reply. Looks like I am still missing something: 

 

transformations-cli deploy .


Deploying transformations...
Failed to parse transformation config, please check that you conform required fields and format: Invalid config: Wrong type for field "destination" - got "{'view_space_external_id': 'test-data-model-space', 'view_external_id': 'Entity', 'view_version': 2, 'instance_space_external_id': 'test-data-model-space', 'type': 'instances'}" of type dict instead of DestinationType, DestinationConfig, RawDestinationConfig, SequenceRowsDestinationConfig, DMIDestinationConfig, InstancesDestinationConfig, RawDestinationAlternativeConfig

 

 

Here is the entire contents of manifest file:

 


externalId: tr-cli-test-transformation
name: cli-test-transformation
query: >-
  file: transformation.sql
destination:
  type: instances
  view_space_external_id: test-data-model-space
  view_external_id: Test
  view_version: 0_2
  instance_space_external_id: test-data-model-space
ignoreNullFields: true
shared: true
action: upsert
authentication:
  clientId: ${COGNITE_CLIENT_ID}
  clientSecret: ${COGNITE_CLIENT_SECRET}
  tokenUrl: https://login.microsoftonline.com/${TENANT}/oauth2/v2.0/token
  # Optional: If idP requires providing the scopes
  cdfProjectName: ${CDF_PROJECT_NAME}
  scopes:
    - ${SCOPES}

 

 


transformations_cli  version is 2.3.0

 

 

 


Forum|alt.badge.img

Hi, I think it saw the view version as a number but not string according to your log
'view_version': 2

Can you force it to string by doing:
  view_version: “0_2”
In the manifest instead?


Hi, 

I tried that, now I see different error (unauthorized error), looks like some permission issue:

 

  return Instances(
Credentials for cli-test-transformation write failed to validate: Unauthorized | code: 401 

 

 


Forum|alt.badge.img

Hi,

I don’t think it’s permission issue (You would get 403 instead in that case). Here it’s 401 so you are not authorized, looks like your credentials is not correct. Can you double check it again, also is it possible to get the full error message, may be I can get a hint which endpoint returns that 401.


Hi,

 

I have verified the credentials again and I use the same credentials to call APIs

 

Here is the complete error:

 

> transformations-cli deploy .

Deploying transformations...
..\Python311\Lib\site-packages\cognite\transformations_cli\commands\deploy\transformations_api.py:95: UserWarning: Feature DataModelStorage is in beta and still in development. Breaking changes can happen in between patch versions.
  return Instances(
Credentials for cli-test-transformation write failed to validate: Unauthorized | code: 401 | X-Request-ID: xxxxx

 

 

 


Forum|alt.badge.img

Can you try to replace all env variable by raw string with the real value for the authentication section in the .yaml file to see?

authentication:
  clientId: "your ClientId"
  clientSecret: "yourClientSecret"
  tokenUrl: "https://login.microsoftonline.com/yourTenantId/oauth2/v2.0/token"
  scopes: 
    - "yourScope" # should be smthing like https://....
  cdfProjectName: "yourProjectName"



Aslo can you give me the request ID from the error above ? I can ask auth team to check the 401
401 | X-Request-ID: xxxxx
But I still think something is wrong with your credentials. If it’s a parsing value issue we should have seen it with other users who used the cli as well


I tried that as well, but I still the same issue.

 

 

 

 


Forum|alt.badge.img

Can you give me the uuid string of the request Id?
Should be the one with xxxx in your error log.
I can try to see with auth team why that request is rejected


Hi,

Here is the request id : 099cefef-6bc9-98c0-8122-68076440353c

 

 


Forum|alt.badge.img

Hello Niranjan,
I created a working sample I hope helps. I created an FDM targeted transformation in Fusion (version as of the date I wrote this) and used the export CLI button in the upper right. This created the following manifest.  I updated authentication section with the credentials that are required when the transformation itself is going to run.  This auth

# Manifest file downloaded from fusion
externalId: tr-BSEEWell-Header-Asset-FDM-cli
name: BSEE Well Header Asset - FDM - CLI
query: >-
  select
    cast(`API_WELL_NUMBER` as STRING) as externalId,
    cast(`WELL_NAME` as STRING) as name,
    cast(`API_WELL_NUMBER` as STRING) as apiNumber
  from
    `bsee`.`well_header`;
destination:
  view:
    space: space-well-model-simple
    externalId: Well
    version: "1"
  instanceSpace: space-well-model-simple
  type: nodes
ignoreNullFields: true
shared: true
action: upsert
dataSetExternalId: ds-bsee

# this is authentication credentials for Fusion, not the transformation-cli
authentication:
  clientId: mycliendid
  clientSecret: myclientsecret
  tokenUrl: https://login.microsoftonline.com/mytenantid/oauth2/v2.0/token
  # Optional: If idP requires providing the scopes
  cdfProjectName: cdf_project
  scopes:
   - https://api.cognitedata.com/.default

Regarding cognite-transformation-cli itself:

 

  • I”m using a virtual python environment (version 3.10.5)
  • I ran pip install cognite-transformations-cli in the above environment and confirmed that the version (transformations-cli --version) of the cli is  2.3.3
  • I sourced the following shell script my path:
    #!/bin/sh
    
    export TRANSFORMATIONS_CLIENT_ID=myclientid
    export TRANSFORMATIONS_CLIENT_SECRET=myclientsecret
    export TRANSFORMATIONS_TOKEN_URL=https://login.microsoftonline.com/mytentantid/oauth2/v2.0/token
    export TRANSFORMATIONS_PROJECT=mycdfproject
    export TRANSFORMATIONS_CLUSTER=api
    export TRANSFORMATIONS_SCOPES=https://api.cognitedata.com/.default
  • Run env at the command line to ensure you have the correct environment variables set:
  • Run transformations-cli list.  This will list all transformations in the target project and will confirm you are correctly authenticated.
  • Lastly, run transformations-cli deploy ./my-transformations to deploy your transformations to CDF.  The my-transformation folder contains the .yaml file containing the manifest above.
  • Confirm the transformation has been deployed to your CDF project.

Hope this helps!

Jason


Dilini Fernando
Seasoned Practitioner
Forum|alt.badge.img+2

Hi @Niranjan Madhukar Karvekar,

Please let us know if you are able to try out Jason’s solution and if you are able to solve the issue. 

Br,
Dilini 


Dilini Fernando
Seasoned Practitioner
Forum|alt.badge.img+2

Hi @Niranjan Madhukar Karvekar,

I hope Jason’s reply has helped you to solve the issue. As of now, I will close this thread. Please feel free to create a new post if you still experience problems.

Br,
Dilini


Forum|alt.badge.img

Hi @Dilini Fernando


We have tried the steps mentioned in the Jason’s reply yet getting 401 error.


Sharing X-Request-ID for reference.
 

 Unauthorized | code: 401 | X-Request-ID: 2fc9c103-6de0-9ae2-bb38-f65110f66fed

Regards,

Sonali


Reply


Cookie Policy

We use cookies to enhance and personalize your experience. If you accept you agree to our full cookie policy. Learn more about our cookies.

 
Cookie Settings