Inline script metadata for zipapp archives and directory execution (original) (raw)

Inline script metadata is formally defined only for single-file Python scripts.

There’s a relatively straightforward potential extension to also handle zipapp archives and filesystem directories: put the inline script metadata in __main__.py, since that’s the equivalent of a single file script in these cases.

However, the specification doesn’t actually say that tools should support that. Given that this would affect multiple tool runners, would we need a PEP to change that?

Or could we just add a MAY to the spec page without needing a PEP?

(This question occurred to me because I’m considering adding inline script metadata support to the launch modules in venvstacks, and I allow those to be folders with a __main__.py file in addition to supporting single file scripts)

pf_moore (Paul Moore) May 6, 2025, 3:32pm 2

Given that the most important (IMO) feature of script metadata is automatic dependency management, and a key feature of zipapps is the ability to bundle dependencies with the code, I think the set of use cases that would benefit from script metadata in a zipapp is relatively small. What do you see as the use cases here?

But use cases aside, from the spec:

This specification defines a metadata format that can be embedded in single-file Python scripts

(emphasis mine). Apart from that statement, though, the spec doesn’t really prohibit putting metadata in the __main__.py file of a zipapp - it says:

Any Python script may have top-level comment blocks that MUST start with the line # /// TYPE where TYPE determines how to process the content.

and I don’t see why “any Python script” wouldn’t include __main__.py.

Having said all of this, I’d be uncomfortable with updating the specification to more clearly support __main__.py (zipapps and directories) unless the “standard” script runners (such as pipx, pip-run and uv) supported that. But if you’re willing to go with the existing spec and support __main__.py in venvstacks on the basis that “it doesn’t say we can’t”, then that seems OK.

One edge case that should probably be clarified - if a zipapp bundles requests, and includes dependencies = ["requests"] in its metadata, should the runner install requests (my view: yes, because it’s unreasonable to demand that the runner checks this), and which copy of requests should take priority (my view: I don’t know, but it could be important).

brettcannon (Brett Cannon) May 6, 2025, 8:03pm 3

I’ve been thinking the same thing (same goes for when you execute a directory).

If you have extension modules then the bundling case doesn’t work. So supporting inline script metadata for zip files would let them act more as a way to distribute a whole project that contains multiple files that work together while having some CLI. One interesting possibility this opens up (I think) is runnable wheel files: include a __main__.py in the wheel with inline script metadata and suddenly the wheel is “self-contained” as a CLI tool.

pf_moore (Paul Moore) May 6, 2025, 8:37pm 4

Oh, cool. Yes, that makes sense. But I do think we would need to clarify what happens if a zipapp bundles some dependencies and puts others in metadata:

I do think it’s a nice idea, though, so I’m not against it. Maybe the existing script runners have feedback from their users on what might be the best design here? (If no-one has asked for it, that’s information as well, of course :wink:)

steve.dower (Steve Dower) May 6, 2025, 8:43pm 5

This should already be resolved by sys.path ordering, yeah? (Potentially in either direction, though I’m pretty sure that the ZIP would appear before a site directory, so you’d get the bundled one.)

Otherwise, yeah, seems like a good idea overall. Natural extension of the existing idea.

pf_moore (Paul Moore) May 6, 2025, 9:37pm 6

Zipapps tend (in my experience) to manipulate sys.path at runtime, because they put vendored libraries in a subdirectory. But my point isn’t so much about runtime, as about the resolver, which can’t see the vendored libraries, and so won’t consider them when trying to install the dependencies declared in metadata.

To give an example:

Try to run this, the script runner sees a need for B, and installs it. Because it can’t see the vendored A, it also installs A. Whether the vendored A or the installed A is used depends on the way the zipapp manipulates sys.path. Worse still, the runner will probably install B 2.0, as it’s the latest. It doesn’t know that A 1.0 is vendored, so it doesn’t spot the incompatibility. If the zipapp puts vendored libs before site-packages (likely), then the resulting environment is broken.

Ideally, the resolver would install B 1.0, and not install A (as it’s already there). But I don’t see how a resolver could do that, as that would involve knowing what sys.path manipulations will happen at runtime.

Of course, this is in some sense user error, so the response is “well, don’t do that”. But it’s subtle, and I think it’s likely to be something people will get wrong without realising. If we were talking about a PEP, I’d want this covered in a “How do we teach this?” section. As we’re not, I’m simply pointing out that we need to think about the problem.

oscarbenjamin (Oscar Benjamin) May 6, 2025, 10:00pm 7

Is there anything that turns a single file script with inline metadata into a zipapp?

ncoghlan (Alyssa Coghlan) May 6, 2025, 10:02pm 8

Yeah, I’m happy to implement this for venvstacks either way (the launch modules there are executed via -m anyway, so even the single file case is already a variation on the way the inline script metadata is normally used).

As far as use cases go, one of the main ones I was thinking of was using this as an input format for build tools that support bundling dependencies into a zip archive.

I definitely don’t think there is any urgency to the question/suggestion, so letting tooling authors make their own decisions without trying to push a particular outcome seems fine to me.

bwoodsend (Brénainn Woodsend) May 6, 2025, 10:30pm 9

Given that a project + its dependency metadata put in a zip file is already what a wheel is, could you achieve the same goals more easily if pipx/uv run supported pipx run ./package.whl?

DavidCEllis (David Ellis) May 6, 2025, 11:06pm 10

Technically I guess I do this with ducktools-env, but all that does is bundle its script runner into the zipapp with __main__.py extracting the runner and executing the bundled script. Mostly intended for creating something that would run without needing a script runner already, or for bundling some additional data with the script.

I had intended to add support for bundling the dependencies in the zipapp (essentially including a wheelhouse that could be extracted and installed) but haven’t ended up needing that myself.

ncoghlan (Alyssa Coghlan) May 7, 2025, 5:35am 11

Wheels are more than that - they’re effectively definitions for “plugin libraries” that are designed to be unpacked into a target Python environment, and then made available to all Python code that runs in that environment. From the original wheel PEP:

… while some Python software is written to support running directly from a zip archive, it is still common for code to be written assuming it has been fully installed. When that assumption is broken by trying to run the software from a zip archive, the failures can often be obscure and hard to diagnose (especially when they occur in third party libraries)

Hence zip-import-compatible-but-not-a-wheel formats like zipapp and shiv archives, that are more explicit about the fact that they do support direct execution as Python “scripts” (and the publisher has taken care of ensuring that those archives actually work when used that way).

ncoghlan (Alyssa Coghlan) May 7, 2025, 5:39am 12

I’m not aware of anything that currently does this, but checking showed that there’s an open PR proposing to add the capability to shiv: Adding inline script metadata (pep-723) support by clayrosenthal · Pull Request #267 · linkedin/shiv · GitHub