Use Specific Python Virtual Environment Dependency

[MattePalte]

System

• Codeql version: 2.11.2.
• Ubuntu 20.04
• system wide installation in folder (/home/user/.local/lib/): python3.6, python3.8, python3.10.
• local python version (virtual environment): venv38/lib/python3.8/

What

I have a bash script that generates the CodeQL database from a specific folder:

...
# enter in the python3.8 environment with my libraries installed
source venv38/bin/activate
...
# create the db
codeql database create $dataset_path/codeql \
    --language=python \
    --overwrite \
    --source-root=$dataset_raw_files_path
...

Output Log:

...
[2023-09-12 13:54:00] [build-stdout] [INFO] [2] Extracted file /usr/lib/python3.6/tarfile.py in 2036ms
[2023-09-12 13:54:00] [build-stdout] [INFO] [6] Extracted module _md5 in 13ms
[2023-09-12 13:54:00] [build-stdout] [INFO] [3] Extracted file /home/username/.local/lib/python3.6/site-packages/my_fancy_dependency/folder/dependency_file_version_36.py in 352ms
..

Expectation

I would expect codeql to create the db with the custom path to my virtual environment venv38/lib/python3.8/site-packages/.. and not the system wide python installation: .local/lib/python3.6.

Question: How can I achieve that? Is there a specific flag or procedure? I cannot find it in the manual codeql database create

Extra Info

1. I need this because the package I need (e.g., in the example the ficticious my_fancy_dependency) has a different version for python3.6 (systemwide python installation) and python3.8.
2. I noted that I have other databases which correctly use the right version of python, but it is unclear how they have been created since I have a similar script to generate them from the same folder.

Let me know if you need more info.

[RasmusWL]

@MattePalte that's indeed very strange, and I am not able to reproduce. With your venv active, does which python3 resolve to the python3 binary in the venv? If not, that's the problem.

If it does, can you create a test-setup in a new folder in the following way:

# <new-test-folder>/src/foo.py
import tty

# <new-test-folder>/script.sh

set -x

# run with system wide python
codeql database create --overwrite --language python --source-root src/ db

# create python 3.8 venv
/home/user/.local/bin/python3.8 -m venv venv

source venv/bin/activate

codeql database create --overwrite --language python --source-root src/ db

and provide the output from running the script above? (for me, it shows different paths for the tty module being extracted)

Finally, I want to note that we're moving towards not extracting installed packages at all. See this changelog post for more details.

[MattePalte]

Thanks for the fast and precise reply @RasmusWL

Indeed I also noticed that something was wrong with the virtual environment, now I created one with conda and python 3.10 and it seems to work (aka the db includes python files from the conda folder path).

But, I have couple of followup questions regarding the recent change:

1. does that mean that the API Graph here will stop working because there no knowledge about the external libraries?
2. If yes, how to force the CodeQL CLI to model the dependencies as well (I am not interested in the flag on the github action, I need to run it locally from the CLI).

Thanks in advance for all your help

[RasmusWL]

Hi @MattePalte, no API graphs will not stop working. They were designed for usage without the source code of dependencies available.

Disclaimer, if the library implementation looks like below, and you use API::moduleImport("my_lib").getMember("Foo").getMember("method").getACall(), with the dependency installed we would also find a result for the usage of AlsoFoo in my_cool_app.py, but without dependency installation you would need to write API::moduleImport("my_lib").getMember(["Foo", "AlsoFoo"]).getMember("method").getACall(). We are working on recovering the most useful cases of this for security related queries (such as FastAPI, Django, Flask) -- this looks to give better results overall, since getting the dependencies right just wasn't possible in many cases.

# my_lib.py
class Foo: ...
class AlsoFoo = Foo

# my_cool_app.py
AlsoFoo.method()

PS. Sorry for the late response, getting back to you got lost in a busy schedule.

[RasmusWL]

Oh yes, I also wanted to mention that if you are looking into getting more serious with CodeQL queries, you can join the slack channel mentioned here: https://securitylab.github.com/get-involved/ 😊