lxm3.xm_cluster.PythonPackage

class lxm3.xm_cluster.PythonPackage(entrypoint: CommandList | ModuleName, path: str = '.', resources=NOTHING, extra_packages=NOTHING, pip_args=NOTHING)

Python package describes an executable that can be packaged by pip install.

PythonPackage describes a python distribution that can be installed by pip install. This is the recommended way to package python projects. For an introduction to python packaging, see Packaging Python Projects.

Note

lxm3 uses pip install to install the package into a temporary directory that will be subsequently packaged into a zip archive that will be deployed and unzipped when jobs run on the cluster. However, this should be considered an implementation detail.

lxm3 uses pip install --no-deps by default to install the package. This means that lxm3 will not install the dependencies required by the project. The reason is that typical ML dependencies are large. (e.g. TensorFlow is around 400MB) Therefore, you should install your dependencies first on the cluster (better, if you are using singularity, install them in the container).

The root directory of the deployed archive will be added to PYTHONPATH, so you can import your package as usual. However, if that interferes with package imports, please open an issue and let us know.

entrypoint

Entrypoint for the built executable.

Currently, only ModuleName and CommandList are supported.

For ModuleName, this corresponds to running python3 -m <module_name>.

For CommandList, this corresponds to running a shell script including the commands.

Type:

lxm3.xm_cluster.executable_specs.CommandList | lxm3.xm_cluster.executable_specs.ModuleName

path

Path to the python project. This should be the path to a directory path that can be installed as a python package by pip install <path>. If it’s a relative path, this will be resolved relative to the launcher’s working directory.

Type:

str

resources

List of resources to package. Currently, only Fileset is supported.

Type:

List[lxm3.xm_cluster.executable_specs.Fileset]

extra_packages

List of paths to additional packages that will be install by pip.

extra_packages is useful if you also want to include a dependency that you are also working on locally and you want to use your development version which are not installed in the runtime environment.

Type:

List[str]

pip_args

Additional options that will be passed to pip install. Defaults to --no-deps --no-compile.

Type:

List[str]

__init__(entrypoint: CommandList | ModuleName, path: str = '.', resources=NOTHING, extra_packages=NOTHING, pip_args=NOTHING) → None

Method generated by attrs for class PythonPackage.

Methods

__init__(entrypoint[, path, resources, ...])

Method generated by attrs for class PythonPackage.

Attributes

name
entrypoint
path
resources
extra_packages
pip_args