Presence of a plugin causes doctests a shared namespace to fail traceback with assertion

[jaraco]

In jaraco/jaraco.test#3, I've described an issue that appears to implicate some of the most edgy features of pytest and Python, in particular:

• assert rewrite
• doctests
• namespace packages
• plugins

Oh, joy.

In summary, the issue seems to be that:

The presence of an empty module as a plugin in a namespace shared by the package under test causes assert exceptions in doctests to be re-written, causing the output from those doctests to fail where they wouldn't otherwise, even if the plugin is disabled with -p no:....

As you can see from the tox execution in the downstream report, I'm using pytest 6.1.2 on Python 3.9.0 with jaraco.test 4.0.1 (the implicated plugin) on macOS 11.0.1 (latest stable everything as of this writing).

I don't yet have a minimal example, though you can check out jaraco/jaraco.itertools@e9f23a4 and run tox -- -k assert to see the error.

I expect (though haven't verified) that any of the following remedies might be possible:

• Move the plugin to a package not sharing the jaraco namespace. I was considering creating pytest-enabler anyway.
• Replace the bare assert with a raise AssertionError. I'll test that soon.
• Update the test expectation to expect the rewritten assert output.
• Replace the doctest with a unit test.

None of these workarounds would address the root cause, but instead avoid the confounding factors leading to the failed expectation.

I recognize this issue is probably too obscure to expect it to be fixed upstream, but I could really use some advice on how to proceed with an investigation. In particular, can you explain or point me to the implementation where doctests get special treatment for assert-rewrite (or vice-versa), especially any code that might be relevant for selectively affecting the modules-under-test or plugins?

[jaraco]

In the referenced commit, I've confirmed and committed the second workaround, strengthening my suspicion that assert rewrite is implicated.

[jaraco]

I took a quick look at _pytest.assertion.rewrite, I'm reminded that rewrites are done through a special importer, so it's probably not a surprise that if a namespace (jaraco) is initialized during plugin setup might affect the timing of rewrites for that package (and maybe modules/packages in that namespace).

I notice #2371 and #2419 may be related, though neither of those reference doctests.

Probably the most helpful question to be answered: how is it that in the simple, general case, doctests that trigger assertions don't stumble on rewritten assertions?

[nicoddemus]

Hi @jaraco,

Sorry for the delay.

The assertion rewriter is triggered automatically in test modules (normal test_*.py files) and all python files of a plugin. The rationale for the latter is that plugins often use assert themselves as part of their functionality, so we want to rewrite those.

The code that marks plugin files for rewriting is found here:

pytest/src/_pytest/config/__init__.py

Lines 1115 to 1133 in 775ba63

	def _mark_plugins_for_rewrite(self, hook) -> None:
	"""Given an importhook, mark for rewrite any top-level
	modules or packages in the distribution package for
	all pytest plugins."""
	self.pluginmanager.rewrite_hook = hook
	if os.environ.get("PYTEST_DISABLE_PLUGIN_AUTOLOAD"):
	# We don't autoload from setuptools entry points, no need to continue.
	return
	package_files = (
	str(file)
	for dist in importlib_metadata.distributions()
	if any(ep.group == "pytest11" for ep in dist.entry_points)
	for file in dist.files or []
	)
	for name in _iter_rewritable_modules(package_files):
	hook.mark_rewrite(name)

This iterates over all the files in a plugin package and marks their asserts for later rewriting on import.

As an example, pytest-qt has the entry-point pytest11 = pytestqt.plugin, and the asserts of all files inside the package are rewritten, even if they are not below the entry point file (for example pytestqt.wait_signal).

So your assessment is correct: this mechanism is rewriting all asserts inside the jaraco.* namespace, because the pytest11 entry point is jaraco.test.pytest.enabler, so all files found in the jaraco distribution get rewritten.

I suspect your case is not isolated, the rewriter is probably being triggered in other packages which include a pytest plugin, although it should only cause problems if someone expects assertion errors in a specific format (like doctests), as the rewriter only changes the assertion message/traceback.

Probably the most helpful question to be answered: how is it that in the simple, general case, doctests that trigger assertions don't stumble on rewritten assertions?

In the general case doctests are not part of a plugin, but normal user code, so they don't get rewritten.

As for workarounds, your two first suggestions are on-point, as you confirmed by using raise AssertionError. Another would be to disable assertion rewriting in that module explicitly.

About fixing this in pytest, I'm not sure how we could detect that part of a plugin is normal code, which shouldn't be changed by the assertion rewriter, and other part is "testing" code, which should have their assertions rewritten.