Both Catalyst and pcore clusters use virtual python environments (venvs) to run all pipelines. Once a venv is created it can be linked with a pipeline when uploaded. You can create as many venvs as you like and can attach one venv to multiple pipelines.
- Requirements - python packages to be included in your virtual env
To create a new venv on Catalyst or a pcore cluster you import the create function from the pipeline-ai library (replace
<username> with your username):
from pipeline.cloud.environments import create_environment env_id = create_environment(name="<username>/numpy", python_requirements=["numpy==1.25.2"])
This environment can then be tagged with a pipeline when uploading like so:
from pipeline.cloud.pipelines import upload_pipeline ... upload_pipeline( "<username>/env-example", environment_id_or_name="<username>/numpy", )
We highly reccomend pointing to specific package versions as defined on pypi.org
Once an environment is created its requirements cannot be changed, you must create a new one if you need to modify the requirements.
It is possible to pass in a specific git repo as a package, we strongly recommend pointing to a specific commit as follows:
from pipeline.cloud.environments import create_environment env_id = create_environment( name="<username>/numpy", python_requirements=[ "git+https://github.com/numpy/numpy.git@ea677928332c37e8052b4d599bf6ee52cf363cf9", ], )
- We do not currently support OS level packages, no
apt installbut will be coming in a future release. The only package you can expect at the moment is
ffmpeg- please reach out to the team on discord if you have an urgent need for an OS level package: Discord.
python==3.10venvs are currently supported, and you must upload new pipelines from
- The only supported source language is python for uploading and running the venv.
Updated 15 days ago