Skip to main content

I need to use a private package from our Azure DevOps Feed. We use Cognite UI to upload zip file which contains handler.py and requirements.txt. If I deploy using cognite-sdk then I get additional arguments like in this link : https://docs.cognite.com/cdf/functions/#additional-arguments

which I can use in client.functions.create()

So I have options:

extra_index_urls (list): A list of URLs pointing to additional python package repositories. Supports basic HTTP authentication.

Using any of these approach I can point to my azure feed(however I am not sure about authentication part. how to provide PAT)

However, when I am using Cognite UI to deploy functions via zip files I dont get options to provide index_urls
 

So I tried including whl file of our private package within zipped folder and deploy the function. But I got this error:

 

Message: Function deployment failed.

Trace:
Failed to publish code:

Source directory : /tmp/zipdeploy/extracted

Destination directory: /home/site/wwwroot

Python Version: /tmp/oryx/platforms/python/3.11.11/bin/python3.11

Creating directory for command manifest file if it does not exist

Removing existing manifest file

Running pip install...

Done in 0 sec(s).

WARNING: Requirement './my_sdk-0.0.1rc140425-py3-none-any.whl' looks like a filename, but the file does not exist

s12:54:43+0000] Processing ./my_sdk-0.0.1rc140425-py3-none-any.whl

ERROR: Could not install packages due to an OSError: OErrno 2] No such file or directory: '/tmp/zipdeploy/extracted/my_sdk-0.0.1rc140425-py3-none-any.whl'

WARNING: You are using pip version 21.2.4; however, version 25.0.1 is available.

You should consider upgrading via the '/tmp/oryx/platforms/python/3.11.11/bin/python3.11 -m pip install --upgrade pip' command.

"2025-04-14 12:54:43"|ERROR|WARNING: Requirement './my_sdk-0.0.1rc140425-py3-none-any.whl' looks like a filename, but the file does not exist

s12:54:43+0000] Processing ./my_sdk-0.0.1rc140425-py3-none-any.whl

ERROR: Could not install packages due to an OSError: OErrno 2] No such file or directory: '/tmp/zipdeploy/extracted/my_sdk-0.0.1rc140425-py3-none-any.whl'

WARNING: You are using pip version 21.2.4; however, version 25.0.1 is available.

You should consider upgrading via the '/tmp/oryx/platforms/python/3.11.11/bin/python3.11 -m pip install --upgrade pip' command. | Exit code: 1 | Please review your requirements.txt | More information: https://aka.ms/troubleshoot-python

\n/opt/Kudu/Scripts/starter.sh oryx build /tmp/zipdeploy/extracted -o /home/site/wwwroot --platform python --platform-version 3.11 -p packagedir=.python_packages/lib/site-packages

Generating summary of Oryx build

Deployment Log file does not exist in /tmp/oryx-build.log

The logfile at /tmp/oryx-build.log is empty. Unable to fetch the summary of build

Deployment Failed. deployer = Push-Deployer deploymentPath = Functions App ZipDeploy. Ex

I have tried following references:
Course: https://learn.cognite.com/cognite-functions
Documentation: https://docs.cognite.com/cdf/functions
Github: https://github.com/cognitedata/learn-cognite-functions/blob/main/notebooks/Part%201%20-%20Calculating%20the%20thermal%20resistance.ipynb

However, I did not get good references to deploy function.

 

my zip file contains:

handler.py # this has handle method

requirements.txt # this has path to whl file, in this case ./my_sdk-0.0.1rc140425-py3-none-any.whl

./my_sdk-0.0.1rc140425-py3-none-any.whl # whl file itself

 

 

 

@Khilesh Sahu I assume you use Toolkit and not the UI for deployment of functions*. To include private packages as wheels your requirements.txt has to look like this:

cognite-sdk>=7.74
...# Any other public dependencies you have
function/my_sdk-0.0.1rc140425-py3-none-any.whl

Note that the word ‘function/’ is not a placeholder. 

The zip folder should look like you are describing (the wheel should not be placed inside a function folder).

handler.py
requirements.txt
my_sdk-0.0.1rc140425-py3-none-any.whl

Thanks so much for reporting, let me know if that solved the issue for you. 

 

Extra details:

*If you are not using Toolkit I strongly recommend you use it over the UI for governing functions, see here in the docs if you are not familiar with Toolkit and Cognite Functions.

Due to the issues with index URL and extra index URL, see docs, I recommend you instead you include the wheel in the zip file as it seems you are doing.

I have reported the missing parameters (index URL and extra index URL) to the team. 

Finally, this is a poorly documented feature. I will make sure it is better documented. 

 


@Anders Albert - Thank you for your reply. Also thanks for taking report on missing parameters - index URL and extra index url.

I am not using toolkit for function deployment. I am not even sure if it’s possible. I know 2 main approaches - 1. By UI and 2. By Cognite SDK. Please let me know recommended approach along with reasons.

I am aware about your mentioned security issue from docs, whereby the package dependency resolver incorrectly installs a public package with the same name as the intended private package. I found that if I mention my private azure devops feed url as index url and pypi.org url as extra index url then preference will always be given to my feed url. Also, other mentioned approach in docs can also work in my favour.

You solved my problem where I can use whl file directly, still I am curious to know about index url and extra index url authentication. I want to get my sdk from azure feed therefore I have to provide pat token with it in the url itself while deploying function via cognite sdk or put --index-url in requirements.txt itself.

For example: https://pat:<my-token>@pkgs.dev.azure.com/<my-org>/_packaging/<my-feed>/pypi/simple/

I am not sure if it’s secured way to do it. May this can be treated as Secrets and that could be more secured approach.


I like your solution of switching your private as index URL and pypi.org as the extra. 

I think setting the index-url like you suggest with token is the secure way to do it:  https://pat:<my-token>@pkgs.dev.azure.com/<my-org>/_packaging/<my-feed>/pypi/simple/

When it comes to UI vs PySDK vs Toolkit. My arguments are as follows

  • For development, one-off deployment, both UI and PySDK are fine. It is a simple way to test something quickly. However, it can be hard to reproduce for another team member. 
  • Toolkit is a way to run CI/CD when you want to have the function code checked into version control and have a reproducible pipeline for deployment. 

But I have not found any reference of creating cognite functions using toolkit. Can you please share it with me?


It is this one: https://docs.cognite.com/cdf/deploy/cdf_toolkit/references/resource_library#functions

 


Thank you ​@Anders Albert . I missed it completely. However, in this example as well we dont have index url and extra index url like in case of UI. Also it would be great if we can set pat as other secrets


Will note it as a task. Thanks for feedback 🙂


Reply