Solved

Cannot import Schema from PostgreSQL in Azure Data Factory

  • 15 May 2023
  • 7 replies
  • 78 views

Userlevel 2
Badge +3

Course: PostgreSQL Gateway
Learning Path: https://learn.cognite.com/path/data-engineer-basics-integrations/postgresql-gateway

 

I get an error when trying to import the Schema, as described here: Step 5: Map the source and destination tables

“Test Connection” succeeds, and I can list out all the tables, such as public.assets, but I get an error when trying to import the schema:

 

icon

Best answer by Johannes Hovda 20 May 2023, 00:15

View original

7 replies

Userlevel 2
Badge +6

Hi Johannes,
Thanks for posting your query. 
I wouldn’t able to re-generate the same issue as you are facing now.
It would be easy for us to investigate more in details, if you can mention the steps you are following before the mapping.

Thanks & Regards
Kumar
 

 

Userlevel 2
Badge +3

Hi,

I recorded a screen share of the problem:

https://drive.google.com/file/d/1RhpjGzKrmE-d22fLX-AClll6KruSULjg/view

 

Userlevel 2

Most likely something went wrong when you created the user for the gateway. It looks like the user is lacking CDF credentials.

Userlevel 2
Badge +3

Most likely something went wrong when you created the user for the gateway. It looks like the user is lacking CDF credentials.

Strange, I have even tried recreating the user, but still the same error.

Userlevel 2
Badge

Hei Johannes,

Have you made sure that the user is part of a group that has the right access?

Check out substep 5 from the guide.

https://docs.cognite.com/cdf/integration/guides/interfaces/postgres_gateway/#step-3-create-a-group-in-cdf-and-link-to-the-azure-ad-group

Add relevant capabilities. The minimum requirement is to add the project:list and groups:list capabilities. You also need to add read and write capabilities for the CDF resources you want to ingest data into. For instance, if you're ingesting assets, add asset:read and asset:write.

 

Hope this helps.

 

Userlevel 2
Badge +3

Hei Johannes,

Have you made sure that the user is part of a group that has the right access?
 

Yes, I have. I have the group in CDF set up with permissions, and the source ID points to the AAD group. The AAD Group contains the App Registration as a member.

 

Note,  in contrast with e.g. the Postman App Registrations, the doc doesn't mention any API permissions need to be set. E.g. Impersonate, or similar. 

 

Userlevel 2
Badge +3

Ok, so I finally got this to work.

One of the problems was related to this:

 

when in fact I had to use the App Registration application ID as Client ID in the Postman body, not the AAD Group.

Another problem was related to parameter resolution in Postman, where I had populated the Token URL and Project with variables that weren't properly resolved, but there was no warning or error when creating the user and password. Also, it still resulted in a Test Connection OK inside Azure Data Factory. 

Reply