Solved

transformations-cli with FDM

  • 22 February 2023
  • 17 replies
  • 301 views

Userlevel 1

Hi,

I am trying to run transformations-cli locally with below command:

transformations-cli deploy . 

(The current directory has manifest.yaml and transformation.sql files)

 

I am getting below error:

Deploying transformations...
Failed to parse transformation config, please check that you conform required fields and format: Invalid config: can not match type "dict" to any type of "destination" union: typing.Union[cognite.transformations_cli.commands.deploy.transformation_types.DestinationType, cognite.transformations_cli.commands.deploy.transformation_types.DestinationConfig, cognite.transformations_cli.commands.deploy.transformation_types.RawDestinationConfig, cognite.transformations_cli.commands.deploy.transformation_types.SequenceRowsDestinationConfig, cognite.transformations_cli.commands.deploy.transformation_types.AlphaDMIDestinationConfig]

 

Here is destination section of manifest file:

destination:
  viewSpaceExternalId: my-model-space-id
  viewExternalId: CDFTimeSeries
  viewVersion: 0_2
  instanceSpaceExternalId: my-model-space-id
  type: instances

 

Please let me know if I am missing anything.

 

Thanks

 

icon

Best answer by Dilini Fernando 18 April 2023, 16:49

View original

17 replies

Userlevel 1
Badge

Can you try to replace all env variable by raw string with the real value for the authentication section in the .yaml file to see?

authentication:
clientId: "your ClientId"
clientSecret: "yourClientSecret"
tokenUrl: "https://login.microsoftonline.com/yourTenantId/oauth2/v2.0/token"
scopes:
- "yourScope" # should be smthing like https://....
cdfProjectName: "yourProjectName"



Aslo can you give me the request ID from the error above ? I can ask auth team to check the 401
401 | X-Request-ID: xxxxx
But I still think something is wrong with your credentials. If it’s a parsing value issue we should have seen it with other users who used the cli as well

Userlevel 1

I tried that as well, but I still the same issue.

 

 

 

 

Userlevel 4
Badge +2

Hi @Niranjan Madhukar Karvekar,

Please let us know if you are able to try out Jason’s solution and if you are able to solve the issue. 

Br,
Dilini 

Userlevel 1
Badge

Can you give me the uuid string of the request Id?
Should be the one with xxxx in your error log.
I can try to see with auth team why that request is rejected

Userlevel 1

Hi,

Here is the request id : 099cefef-6bc9-98c0-8122-68076440353c

 

 

Userlevel 4
Badge +2

Hi @Niranjan Madhukar Karvekar,

I hope Jason’s reply has helped you to solve the issue. As of now, I will close this thread. Please feel free to create a new post if you still experience problems.

Br,
Dilini

Userlevel 2
Badge

Hi @Dilini Fernando


We have tried the steps mentioned in the Jason’s reply yet getting 401 error.


Sharing X-Request-ID for reference.
 

 Unauthorized | code: 401 | X-Request-ID: 2fc9c103-6de0-9ae2-bb38-f65110f66fed

Regards,

Sonali

Userlevel 4
Badge

Hello Niranjan,
I created a working sample I hope helps. I created an FDM targeted transformation in Fusion (version as of the date I wrote this) and used the export CLI button in the upper right. This created the following manifest.  I updated authentication section with the credentials that are required when the transformation itself is going to run.  This auth

# Manifest file downloaded from fusion
externalId: tr-BSEEWell-Header-Asset-FDM-cli
name: BSEE Well Header Asset - FDM - CLI
query: >-
select
cast(`API_WELL_NUMBER` as STRING) as externalId,
cast(`WELL_NAME` as STRING) as name,
cast(`API_WELL_NUMBER` as STRING) as apiNumber
from
`bsee`.`well_header`;
destination:
view:
space: space-well-model-simple
externalId: Well
version: "1"
instanceSpace: space-well-model-simple
type: nodes
ignoreNullFields: true
shared: true
action: upsert
dataSetExternalId: ds-bsee

# this is authentication credentials for Fusion, not the transformation-cli
authentication:
clientId: mycliendid
clientSecret: myclientsecret
tokenUrl: https://login.microsoftonline.com/mytenantid/oauth2/v2.0/token
# Optional: If idP requires providing the scopes
cdfProjectName: cdf_project
scopes:
- https://api.cognitedata.com/.default

Regarding cognite-transformation-cli itself:

 

  • I”m using a virtual python environment (version 3.10.5)
  • I ran pip install cognite-transformations-cli in the above environment and confirmed that the version (transformations-cli --version) of the cli is  2.3.3
  • I sourced the following shell script my path:
    #!/bin/sh

    export TRANSFORMATIONS_CLIENT_ID=myclientid
    export TRANSFORMATIONS_CLIENT_SECRET=myclientsecret
    export TRANSFORMATIONS_TOKEN_URL=https://login.microsoftonline.com/mytentantid/oauth2/v2.0/token
    export TRANSFORMATIONS_PROJECT=mycdfproject
    export TRANSFORMATIONS_CLUSTER=api
    export TRANSFORMATIONS_SCOPES=https://api.cognitedata.com/.default
  • Run env at the command line to ensure you have the correct environment variables set:
  • Run transformations-cli list.  This will list all transformations in the target project and will confirm you are correctly authenticated.
  • Lastly, run transformations-cli deploy ./my-transformations to deploy your transformations to CDF.  The my-transformation folder contains the .yaml file containing the manifest above.
  • Confirm the transformation has been deployed to your CDF project.

Hope this helps!

Jason

Userlevel 4
Badge

Hi Niranjan,

Can you share with me your entire transformation config yaml?  Here or via Teams :).


Jason

Userlevel 1
Badge

Hi, it looks like you did not use the latest version of the cli, since we already renamed AlphaDMIDestinationConfig not DMIDestinationConfig , and the new type of FDM is InstancesDestinationConfig

Userlevel 1

Hi,

Thanks for your reply. I tried with latest transformations-cli (2.3.0) but I see the same error. I generated manifest file from CDF UI. Can you please provide me the example of manifest file with destination as FDM model? 

Thanks.

 

Userlevel 1
Badge

Hi,
Ah right you need to use the syntax with _ instead of camel case I think :)
https://github.com/cognitedata/transformations-cli/blob/main/tests/test_deploy.py#L185-L190

        destination:
type: instances
view_external_id: test_view
view_version: test_view_version
view_space_external_id: test_space
instance_space_external_id: test_space

 

Userlevel 1

Hi,

 

Thanks for your reply. Looks like I am still missing something: 

 

transformations-cli deploy .


Deploying transformations...
Failed to parse transformation config, please check that you conform required fields and format: Invalid config: Wrong type for field "destination" - got "{'view_space_external_id': 'test-data-model-space', 'view_external_id': 'Entity', 'view_version': 2, 'instance_space_external_id': 'test-data-model-space', 'type': 'instances'}" of type dict instead of DestinationType, DestinationConfig, RawDestinationConfig, SequenceRowsDestinationConfig, DMIDestinationConfig, InstancesDestinationConfig, RawDestinationAlternativeConfig

 

 

Here is the entire contents of manifest file:

 


externalId: tr-cli-test-transformation
name: cli-test-transformation
query: >-
  file: transformation.sql
destination:
  type: instances
  view_space_external_id: test-data-model-space
  view_external_id: Test
  view_version: 0_2
  instance_space_external_id: test-data-model-space
ignoreNullFields: true
shared: true
action: upsert
authentication:
  clientId: ${COGNITE_CLIENT_ID}
  clientSecret: ${COGNITE_CLIENT_SECRET}
  tokenUrl: https://login.microsoftonline.com/${TENANT}/oauth2/v2.0/token
  # Optional: If idP requires providing the scopes
  cdfProjectName: ${CDF_PROJECT_NAME}
  scopes:
    - ${SCOPES}

 

 


transformations_cli  version is 2.3.0

 

 

 

Userlevel 1
Badge

Hi, I think it saw the view version as a number but not string according to your log
'view_version': 2

Can you force it to string by doing:
  view_version: “0_2”
In the manifest instead?

Userlevel 1

Hi, 

I tried that, now I see different error (unauthorized error), looks like some permission issue:

 

  return Instances(
Credentials for cli-test-transformation write failed to validate: Unauthorized | code: 401 

 

 

Userlevel 1
Badge

Hi,

I don’t think it’s permission issue (You would get 403 instead in that case). Here it’s 401 so you are not authorized, looks like your credentials is not correct. Can you double check it again, also is it possible to get the full error message, may be I can get a hint which endpoint returns that 401.

Userlevel 1

Hi,

 

I have verified the credentials again and I use the same credentials to call APIs

 

Here is the complete error:

 

> transformations-cli deploy .

Deploying transformations...
..\Python311\Lib\site-packages\cognite\transformations_cli\commands\deploy\transformations_api.py:95: UserWarning: Feature DataModelStorage is in beta and still in development. Breaking changes can happen in between patch versions.
  return Instances(
Credentials for cli-test-transformation write failed to validate: Unauthorized | code: 401 | X-Request-ID: xxxxx

 

 

 

Reply