Python dependencies are missing in release binaries

[ShahanaFarooqui]

Currently, we are not packaging Python plugins (clnrest & wssproxy) dependencies with the release binaries. To resolve this, we can consider the following options:

Option 1: Use PyInstaller

Convert the plugin into a standalone executable using PyInstaller. This approach bundles the Python interpreter and all required libraries into a single folder or executable file, making distribution easier.

Option 2: Bundle Python Libraries with Binaries

Bundle dist-packages for CLN with the binaries as they are already being installed with && pip install -r requirements.txt \ command in Dockerfile. files.**

Option 3: Bundle requirements.txt with Binaries

Include the requirements.txt with the binaries and install the dependencies after extraction. The installation process would look something like this:

sudo tar -xvf clightning-v24.08-Ubuntu-24.04.tar.xz -C /usr/local --strip-components=2 \
&& sudo pip3 install -r /usr/local/python/requirements.txt \
&& sudo rm -rf /usr/local/python

For more complex future installation steps, we could convert it into a shell script (e.g., install.sh).

#!/bin/bash

# Extract the tar.xz file and strip components
sudo tar -xvf clightning-v24.08-Ubuntu-24.04.tar.xz -C /usr/local --strip-components=2

# Install Python dependencies
sudo pip3 install -r /usr/local/python/requirements.txt

# Delete the python directory after installation
sudo rm -rf /usr/local/python

Note to self: If we choose this option, remember to remove && pip install -r requirements.txt \ from Dockerfile. files.

Open to Other Suggestions:

Feel free to share your thoughts if there is a better solution.

[vincenzopalazzo]

Due that we already have rust as a build dependencies, why not use the alternative rust plugin that @daywalker90 made for cln rest?

IMHO this should be a good alternative, and we stop required python dependencies to just run vanilla cln

[dfgjknhgcrj]

pip install pyinstaller

[dfgjknhgcrj]

&& pip install -r requirements.txt \henry

[king-11]

I also agree with @vincenzopalazzo packaging and dependency management in python seems like a mess a rust binary plugin just might be a great solution.

From the provided options option 1 seems fair but bundling python interpreter won't that bind the interpreter selection instead of allowing user to choose which one he wants in his system a might just be an extra bloat for lean docker containers.

option 2 & 3 utilize requirements.txt and it seems to be the case that they are discouraged nowadays and was the reason for removing them from cln plugins as well.

[cdecker]

Quick idea to get the python-based plugins to behave a bit better again: uv inline dependency declarations.

I have been using uv for a while, and it's much simpler to use than poetry and all the rest, since it a) ignores the system python installation if any, b) it manages its own python binaries, and c) it is incredibly fast at building virtualenvs on the fly, obviating the need to even keep them around.

A plugin with the shebang #!/usr/bin/env -S uv run only requires that the uv executable is somewhere on the path, and then we can specify dependencies in the script itself like this:

# /// script
# requires-python = ">=3.12"
# dependencies = [
#     "flask==3.*",
# ]
# ///

Being invoked with uv run from the env this will be parsed, and uv will spin up a virtualenv with the exact dependency versions specified in the script (and transitive dependencies), and then invoke the script from that virtualenv.

So the management of dependencies is done at startup every time, and there is no drift. Only open question is whether we can make uv silent enough so it doesn't interfere with the stdout / stdin based protocol between lightningd and the plugins

[rustyrussell]

Switching to uv seems sensible (as does switching to the rust cln-rest, but that's independent). Using uv for reckless as well would be nice, rather than the current venv wrapper hack.

For this milestone, we should simply include the requirements.txt in the tarfile. It should be in something like $(pkglibexecdir)/python-requirements.txt. As long as the plugins fail gracefully if their dependencies are not installed (and they do, I believe), we're good.