Skip to content

Support PEP420 (implicit namespace packages) as --pyargs target. #13426

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 3 commits into from
Jul 12, 2025
Merged

Support PEP420 (implicit namespace packages) as --pyargs target. #13426

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
3 commits
Select commit Hold shift + click to select a range
864079f
Support PEP420 (implicit namespace packages) as `--pyargs` target.
karlicoss May 16, 2025
9dc1210
address review
karlicoss Jul 9, 2025
45c03cf
Update changelog/478.feature.rst
nicoddemus Jul 10, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
3 changes: 3 additions & 0 deletions changelog/478.feature.rst
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,3 @@
Support PEP420 (implicit namespace packages) as `--pyargs` target when :confval:`consider_namespace_packages` is `true` in the config.

Previously, this option only impacted package names, now it also impacts tests discovery.
1 change: 1 addition & 0 deletions doc/en/reference/reference.rst
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -1384,6 +1384,7 @@ passed multiple times. The expected format is ``name=value``. For example::
when collecting Python modules. Default is ``False``.

Set to ``True`` if the package you are testing is part of a namespace package.
Namespace packages are also supported as ``--pyargs`` target.

Only `native namespace packages <https://packaging.python.org/en/latest/guides/packaging-namespace-packages/#native-namespace-packages>`__
are supported, with no plans to support `legacy namespace packages <https://packaging.python.org/en/latest/guides/packaging-namespace-packages/#legacy-namespace-packages>`__.
Expand Down
42 changes: 35 additions & 7 deletions src/_pytest/main.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -774,6 +774,9 @@ def perform_collect(
self._collection_cache = {}
self.items = []
items: Sequence[nodes.Item | nodes.Collector] = self.items
consider_namespace_packages: bool = self.config.getini(
"consider_namespace_packages"
)
try:
initialpaths: list[Path] = []
initialpaths_with_parents: list[Path] = []
Expand All @@ -782,6 +785,7 @@ def perform_collect(
self.config.invocation_params.dir,
arg,
as_pypath=self.config.option.pyargs,
consider_namespace_packages=consider_namespace_packages,
)
self._initial_parts.append(collection_argument)
initialpaths.append(collection_argument.path)
Expand Down Expand Up @@ -981,7 +985,9 @@ def genitems(self, node: nodes.Item | nodes.Collector) -> Iterator[nodes.Item]:
node.ihook.pytest_collectreport(report=rep)


def search_pypath(module_name: str) -> str | None:
def search_pypath(
module_name: str, *, consider_namespace_packages: bool = False
) -> str | None:
"""Search sys.path for the given a dotted module name, and return its file
system path if found."""
try:
Expand All @@ -991,13 +997,29 @@ def search_pypath(module_name: str) -> str | None:
# ValueError: not a module name
except (AttributeError, ImportError, ValueError):
return None
if spec is None or spec.origin is None or spec.origin == "namespace":

if spec is None:
return None
elif spec.submodule_search_locations:
return os.path.dirname(spec.origin)
else:

if (
spec.submodule_search_locations is None
or len(spec.submodule_search_locations) == 0
):
# Must be a simple module.
return spec.origin

if consider_namespace_packages:
# If submodule_search_locations is set, it's a package (regular or namespace).
# Typically there is a single entry, but documentation claims it can be empty too
# (e.g. if the package has no physical location).
return spec.submodule_search_locations[0]
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Perhaps instead of just returning it blindly, we should check if this is a directory before returning?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

So I looked at it:

  • in my experience of working with namespace packages, this is always a directory. I guess it's possible for it not to be if some sort of obscure package loader is used, but I'm not sure pytest would work with such packages anyway?
  • either way even if it's not there isn't much we can do.
    If spec.submodule_search_locations is set at all, then spec.origin will be None, so we can't even fall back onto os.path.dirname(spec.origin).

IMO it's fine to keep it as is, especially considering now this is hidden behind consider_namespace_packages

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I agree... the docs might say "a path" or "a directory", but given it is a string, who knows what some loader/package might be returning...


if spec.origin is None:
# This is only the case for namespace packages
return None

return os.path.dirname(spec.origin)


@dataclasses.dataclass(frozen=True)
class CollectionArgument:
Expand All @@ -1009,7 +1031,11 @@ class CollectionArgument:


def resolve_collection_argument(
invocation_path: Path, arg: str, *, as_pypath: bool = False
invocation_path: Path,
arg: str,
*,
as_pypath: bool = False,
consider_namespace_packages: bool = False,
) -> CollectionArgument:
"""Parse path arguments optionally containing selection parts and return (fspath, names).

Expand Down Expand Up @@ -1049,7 +1075,9 @@ def resolve_collection_argument(
parts[-1] = f"{parts[-1]}{squacket}{rest}"
module_name = None
if as_pypath:
pyarg_strpath = search_pypath(strpath)
pyarg_strpath = search_pypath(
strpath, consider_namespace_packages=consider_namespace_packages
)
if pyarg_strpath is not None:
module_name = strpath
strpath = pyarg_strpath
Expand Down
61 changes: 61 additions & 0 deletions testing/test_collection.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -1935,3 +1935,64 @@ def test_func():
result = pytester.runpytest()
assert result.ret == 0
result.stdout.fnmatch_lines(["*1 passed*"])


@pytest.mark.parametrize("import_mode", ["prepend", "importlib", "append"])
def test_namespace_packages(pytester: Pytester, import_mode: str):
pytester.makeini(
f"""
[pytest]
consider_namespace_packages = true
pythonpath = .
python_files = *.py
addopts = --import-mode {import_mode}
"""
)
pytester.makepyfile(
**{
"pkg/module1.py": "def test_module1(): pass",
"pkg/subpkg_namespace/module2.py": "def test_module1(): pass",
"pkg/subpkg_regular/__init__.py": "",
"pkg/subpkg_regular/module3": "def test_module3(): pass",
}
)

# should collect when called with top-level package correctly
result = pytester.runpytest("--collect-only", "--pyargs", "pkg")
result.stdout.fnmatch_lines(
[
"collected 3 items",
"<Dir pkg>",
" <Module module1.py>",
" <Function test_module1>",
" <Dir subpkg_namespace>",
" <Module module2.py>",
" <Function test_module1>",
" <Package subpkg_regular>",
" <Module module3.py>",
" <Function test_module3>",
]
)

# should also work when called against a more specific subpackage/module
result = pytester.runpytest("--collect-only", "--pyargs", "pkg.subpkg_namespace")
result.stdout.fnmatch_lines(
[
"collected 1 item",
"<Dir pkg>",
" <Dir subpkg_namespace>",
" <Module module2.py>",
" <Function test_module1>",
]
)

result = pytester.runpytest("--collect-only", "--pyargs", "pkg.subpkg_regular")
result.stdout.fnmatch_lines(
[
"collected 1 item",
"<Dir pkg>",
" <Package subpkg_regular>",
" <Module module3.py>",
" <Function test_module3>",
]
)
17 changes: 14 additions & 3 deletions testing/test_main.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -169,8 +169,13 @@ def test_dir(self, invocation_path: Path) -> None:
):
resolve_collection_argument(invocation_path, "src/pkg::foo::bar")

def test_pypath(self, invocation_path: Path) -> None:
@pytest.mark.parametrize("namespace_package", [False, True])
def test_pypath(self, namespace_package: bool, invocation_path: Path) -> None:
"""Dotted name and parts."""
if namespace_package:
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I kind of piggybacked on the existing test that covers this functionality, but not sure whether it's a bit dirty!
Please let me know if you'd prefer me to do it in some other way, e.g. create a new fixture specifically for this case. Or perhaps this should be tested in some different place rather than test_main?

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hi @karlicoss,

I think it is perfectly fine to reuse an existing test like this one, thanks!

However functionality related to collection and packages tends to break complex test suites in subtle ways... been there done that.

But to be honest not sure how we can even foresee how this will be handled in the wild, unless we put it in the wild.

I'm considering adding this behind a feature flag, so if it causes havoc, we can optionally revert it...

On the other hand, we can just bite the bullet and see. If this causes massive breakage, we can always revert the patch and make a hot fix.

Just thinking aloud here.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

On the other hand, we can just bite the bullet and see. If this causes massive breakage, we can always revert the patch and make a hot fix.

No, agree, it makes a lot of sense! From experience, such random breakages are pretty annoying. Happy to implement a feature flag and add documentation for it.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I've been thinking about the flag name & documentation.. and actually, perhaps it makes sense to simply reuse consider_namespace_packages?

From my experience with it, right now only helps with package/module names. E.g. if there is a test inside test_pkg.py

src/
    pkg/
        test_pkg.py

And we run PYTHONPATH=src pytest --pyargs pkg.test_pkg

It correctly discovers tests in both cases, but.

  • with consider_namespace_packages, it has __package == "pkg" and __module__ == "pkg.test_pkg" (as one might expect)

  • without consider_namespace_packages, it has __package == "" and __module__ == "test_pkg"

    If we add pkg/__init__.py, this has correct __package__ and __module__ though.

This is consistent with --help:

Consider namespace packages when resolving module names during import

However, according to https://docs.pytest.org/en/stable/reference/reference.html#confval-consider_namespace_packages

Controls if pytest should attempt to identify namespace packages when collecting Python modules. Default is False.
Set to True if the package you are testing is part of a namespace package.

So the docs say it "should attempt to idenfity when collecting", but I'm not sure what this actually means here.

E.g. if you have the following hierarchy:

src/
    pkg/
        __init__.py
        subpkg/
            test_subpkg.py

pkg is a regular package with __init__.py, but pkg.subpkg is a namespace subpackage.
Then if you run PYTHONPATH=src pytest --pyargs pkg (with upstream pytest), pytest correctly discovers the test in test_subpkg.py

The only difference is

  • with consider_namespace_packages it correctly sets __package__ and __module__ in test_subpkg.py
  • without, it sets __package__ = ""

From a lurk in pytest code, to me it feels that indeed it has to do more with naming than test discovery -- e.g. I think stuff in this file is mostly after we already collected candidate files to run?

pytest/src/_pytest/pathlib.py

Line 494 in 83536b4

consider_namespace_packages: bool,

So perhaps if I just reuse consider_namespace_packages, this makes it closer to the original intent? So it could both discover tests under namespace packages (this PR), and they also will have correct __package__/__module__ attributes (already working in upstream pytest).

This also limits the potential impact from the change -- there are "only" 400ish matches for this setting on github https://github.com/search?q=consider_namespace_packages&type=code (tiny amount comparing to total pytest users), and I'd imagine people who use this setting know what they are doing anyway.

What do you think?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hey @nicoddemus --- any thoughts on this? :) ^

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hi @karlicoss, sorry I missed your comment.

Yeah I think reusing the option makes sense, and the impact is has a small radius as you mention.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks! Reused the option, feels like it's much more consistent now!

I also added an extra 'end-to-end' test for consider_namespace_packages = true in testing/test_collection.py, felt like worth it since there is a configuration option involved.
Also it kind of describes what the option does rather than test_pypath.

# Namespace package doesn't have to contain __init__py
(invocation_path / "src/pkg/__init__.py").unlink()

assert resolve_collection_argument(
invocation_path, "pkg.test", as_pypath=True
) == CollectionArgument(
Expand All @@ -186,7 +191,10 @@ def test_pypath(self, invocation_path: Path) -> None:
module_name="pkg.test",
)
assert resolve_collection_argument(
invocation_path, "pkg", as_pypath=True
invocation_path,
"pkg",
as_pypath=True,
consider_namespace_packages=namespace_package,
) == CollectionArgument(
path=invocation_path / "src/pkg",
parts=[],
Expand All @@ -197,7 +205,10 @@ def test_pypath(self, invocation_path: Path) -> None:
UsageError, match=r"package argument cannot contain :: selection parts"
):
resolve_collection_argument(
invocation_path, "pkg::foo::bar", as_pypath=True
invocation_path,
"pkg::foo::bar",
as_pypath=True,
consider_namespace_packages=namespace_package,
)

def test_parametrized_name_with_colons(self, invocation_path: Path) -> None:
Expand Down