bug: Offline operation, sqlite checker attempts internet access and results in error

[bufferattack]

Description

completely offline installation errors on sqlite.py file line 34 even when using the --offline argument. Even when trying to skip the checker.

NOTE 1, this is a completely offline installation. cve-bin-tool is temperamental by insisting on having internet access across the installation process even after attempting to use offline downloaded packages.
NOTE 2: After installation, nonexistent cache/directories prohibit offline database import (another bug) unless tool is run initially first to build the cache/directories, then import is possible. If you've installed cve-bin-tool online, then you will not be able to duplicate this issue.
NOTE 3: Attempting to skip the "sqlite" checker and the error in this submission is still happening. It is probable that the checker is not actually being skipped; this may be another possible bug?

To reproduce

cve-bin-tool --offline -s sqlite /usr/bin/7z

Expected behaviour: no access to internet should be attempted by tool.
Actual behaviour: access query is attempted AND error is not handled.

Version/platform info

Version of CVE-bin-tool( e.g. output of cve-bin-tool --version): 3.3
Installed from pypi or github? pip3
Operating system: Linux/Windows (other platforms are unsupported but feel free to report issues anyhow)

• On Linux (or Windows Subsystem for Linux) you can run uname -a kali 2024.2
• On Windows you can run systeminfo | findstr /B /C:"OS Name" /C:"OS Version"
  Python version (e.g. python3 --version): 3.11.9
  Running in any particular CI environment we should know about? (e.g. Github Actions)

Anything else?

┌──(user㉿kali-purple)-[~/…/lib/python3.11/site-packages/cve_bin_tool]
└─$ /home/user/.local/bin/cve-bin-tool --offline -s sqlite /usr/bin/7z
[14:40:22] INFO cve_bin_tool - CVE Binary Tool v3.3 cli.py:571
INFO cve_bin_tool - This product uses the NVD API but is not endorsed or certified by the NVD. cli.py:572
WARNING cve_bin_tool - Not verifying CVE DB cache cli.py:810
INFO cve_bin_tool.CVEDB - There are 278373 CVE entries in the database cvedb.py:362
INFO cve_bin_tool.CVEDB - There are 260087 CVE entries from NVD in the database cvedb.py:364
INFO cve_bin_tool.CVEDB - There are 18284 CVE entries from REDHAT in the database cvedb.py:364
INFO cve_bin_tool.CVEDB - There are 2 CVE entries from GAD in the database cvedb.py:364
INFO cve_bin_tool.CVEDB - There are 278373 CVE entries in the database cvedb.py:362
INFO cve_bin_tool.CVEDB - There are 260087 CVE entries from NVD in the database cvedb.py:364
INFO cve_bin_tool.CVEDB - There are 18284 CVE entries from REDHAT in the database cvedb.py:364
INFO cve_bin_tool.CVEDB - There are 2 CVE entries from GAD in the database cvedb.py:364
INFO cve_bin_tool - CVE database contains CVEs from National Vulnerability Database (NVD), Open Source Vulnerability cli.py:832
Database (OSV), Gitlab Advisory Database (GAD) and RedHat
INFO cve_bin_tool - CVE database last updated on 16 August 2024 at 14:09:50 cli.py:835
ERROR cve_bin_tool - An error occurred while fetching https://www.sqlite.org/changes.html: util.py:281
HTTPSConnectionPool(host='www.sqlite.org', port=443): Max retries exceeded with url: /changes.html (Caused by
NameResolutionError("<urllib3.connection.HTTPSConnection object at 0x7ff81fc36bd0>: Failed to resolve
'www.sqlite.org' ([Errno -3] Temporary failure in name resolution)"))
╭─────────────────────────────── Traceback (most recent call last) ────────────────────────────────╮
│ /home/user/.local/bin/cve-bin-tool:8 in │
│ │
│ 5 from cve_bin_tool.cli import main │
│ 6 if name == 'main': │
│ 7 │ sys.argv[0] = re.sub(r'(-script.pyw|.exe)?$', '', sys.argv[0]) │
│ ❱ 8 │ sys.exit(main()) │
│ 9 │
│ │
│ /home/user/.local/lib/python3.11/site-packages/cve_bin_tool/cli.py:1012 in main │
│ │
│ 1009 │ │ │ │ │ cve_scanner.get_cves(product_info, triage_data) │
│ 1010 │ │ if args["directory"]: │
│ 1011 │ │ │ sbom_root = args["directory"] │
│ ❱ 1012 │ │ │ version_scanner = VersionScanner( │
│ 1013 │ │ │ │ should_extract=args["extract"], │
│ 1014 │ │ │ │ exclude_folders=args["exclude"], │
│ 1015 │ │ │ │ error_mode=error_mode, │
│ │
│ /home/user/.local/lib/python3.11/site-packages/cve_bin_tool/version_scanner.py:58 in init │
│ │
│ 55 │ │ │ update_egg() │
│ 56 │ │ │
│ 57 │ │ # Load checkers if not given │
│ ❱ 58 │ │ self.checkers = checkers or self.load_checkers() │
│ 59 │ │ self.score = score │
│ 60 │ │ self.total_scanned_files = 0 │
│ 61 │ │ self.exclude_folders = exclude_folders + [".git"] │
│ │
│ /home/user/.local/lib/python3.11/site-packages/cve_bin_tool/version_scanner.py:80 in │
│ load_checkers │
│ │
│ 77 │ @classmethod │
│ 78 │ def load_checkers(cls) -> dict[str, type[Checker]]: │
│ 79 │ │ """Loads CVE checkers""" │
│ ❱ 80 │ │ checkers = dict( │
│ 81 │ │ │ map( │
│ 82 │ │ │ │ lambda checker: (checker.name, checker.load()), │
│ 83 │ │ │ │ importlib_metadata.entry_points().select(group=cls.CHECKER_ENTRYPOINT), │
│ │
│ /home/user/.local/lib/python3.11/site-packages/cve_bin_tool/version_scanner.py:82 in │
│ │
│ 79 │ │ """Loads CVE checkers""" │
│ 80 │ │ checkers = dict( │
│ 81 │ │ │ map( │
│ ❱ 82 │ │ │ │ lambda checker: (checker.name, checker.load()), │
│ 83 │ │ │ │ importlib_metadata.entry_points().select(group=cls.CHECKER_ENTRYPOINT), │
│ 84 │ │ │ ) │
│ 85 │ │ ) │
│ │
│ /usr/lib/python3.11/importlib/metadata/init.py:202 in load │
│ │
│ 199 │ │ return the named object. │
│ 200 │ │ """ │
│ 201 │ │ match = self.pattern.match(self.value) │
│ ❱ 202 │ │ module = import_module(match.group('module')) │
│ 203 │ │ attrs = filter(None, (match.group('attr') or '').split('.')) │
│ 204 │ │ return functools.reduce(getattr, attrs, module) │
│ 205 │
│ │
│ /usr/lib/python3.11/importlib/init.py:126 in import_module │
│ │
│ 123 │ │ │ if character != '.': │
│ 124 │ │ │ │ break │
│ 125 │ │ │ level += 1 │
│ ❱ 126 │ return _bootstrap._gcd_import(name[level:], package, level) │
│ 127 │
│ 128 │
│ 129 _RELOADING = {} │
│ in _gcd_import:1204 │
│ in _find_and_load:1176 │
│ in _find_and_load_unlocked:1147 │
│ in _load_unlocked:690 │
│ in exec_module:940 │
│ in _call_with_frames_removed:241 │
│ │
│ /home/user/.local/lib/python3.11/site-packages/cve_bin_tool/checkers/sqlite.py:50 in │
│ │
│ 47 │ return version_map │
│ 48 │
│ 49 │
│ ❱ 50 class SqliteChecker(Checker): │
│ 51 │ CONTAINS_PATTERNS = [ │
│ 52 │ │ r"unable to open a temporary database file for storing temporary tables", │
│ 53 │ │ r"json_object() requires an even number of arguments", │
│ │
│ /home/user/.local/lib/python3.11/site-packages/cve_bin_tool/checkers/sqlite.py:66 in │
│ SqliteChecker │
│ │
│ 63 │ │
│ 64 │ mapdb = VersionSignatureDb("sqlite", get_version_map, 30) │
│ 65 │ with mapdb: │
│ ❱ 66 │ │ VERSION_MAP = mapdb.get_mapping_data() │
│ 67 │ │
│ 68 │ def guess_contains(self, lines): │
│ 69 │ │ """Tries to determine if a file includes sqlite""" │
│ │
│ /home/user/.local/lib/python3.11/site-packages/cve_bin_tool/version_signature.py:100 in │
│ get_mapping_data │
│ │
│ 97 │ │ │ │ (time.time(),), │
│ 98 │ │ │ ) │
│ 99 │ │ │ │
│ ❱ 100 │ │ │ for mapping in self.mapping_function(): │
│ 101 │ │ │ │ self.cursor.execute( │
│ 102 │ │ │ │ │ f"INSERT INTO {self.table_name} (version, sourceId) VALUES (?, ?)", │
│ 103 │ │ │ │ │ (mapping[0], mapping[1]), │
│ │
│ /home/user/.local/lib/python3.11/site-packages/cve_bin_tool/checkers/sqlite.py:34 in │
│ get_version_map │
│ │
│ 31 │ │
│ 32 │ # timeout = 300s = 5minutes. This is a guess. │
│ 33 │ response = make_http_requests("text", url=changeurl, timeout=300) │
│ ❱ 34 │ lines = response.splitlines() │
│ 35 │ │
│ 36 │ last_version = "UNKNOWN" │
│ 37 │ for line in lines: │
╰──────────────────────────────────────────────────────────────────────────────────────────────────╯

[terriko]

Ugh, this is definitely a bug. I'm guessing since the sqlite checker predates the offline flag it may not be doing checks appropriately, but I'll need to dig into things to figure out for sure. Thanks for reporting it!

[bufferattack]

No problem reporting the issue as it's the least I can do for those who built and maintain an awesome tool. It may be needed to submit two more separate issues I found related to this effort (as I noted in the original submission). The fact that a check (sqlite) cannot be skipped is of concern. The other is the fact that importing offline database fails unless the tool is run first (to build the cache), then imports are possible.

[terriko]

We've been talking about the cache issue relative to doing scans without the database at all #4110 but please report it so hopefully someone who isn't me will get a chance to work on it while I'm dealing with release stuff! I'm pretty sure you can fix just the cache initialization without having to fix the whole architecture. And yeah, sqlite definitely is an issue. I think we should resolve it by converting it to behave like the other data sources, but it'll take some thinking to do it correctly.

[Enriquerzrz]

Hello there, I am having the same issue when trying to start the tool on a completely disconnected machine. I don't know if the problem is already fixed or some kind of temporary fix has been found to prevent the sqlite connection attempt.

[terriko]

I don't think we have a fix yet, sorry! I'll put a note in our gitter chat saying that you're particularly interested in a fix and hopefully someone will get a chance to look at it, but it likely won't be me.

[Enriquerzrz]

Hi, just a couple of days ago I managed to fix the problem just by adding some error checking inside the sqlite.py file. I'll leave it here if it helps, I was thinking of creating a pull request and uploading it to fix the problem when I had some time to do so.

sqlite.py.txt

[terriko]

A pull request would be great!