[Python-Dev] Packaging and binary distributions for Python 3.3 (original) (raw)

Vinay Sajip vinay_sajip at yahoo.co.uk
Mon Oct 17 11:15:55 CEST 2011

Paul Moore <p.f.moore gmail.com> writes:

Interesting. That's not a use case that I have encountered, but I can see it could be an issue. I have been working on the basis that a bdistsimple format that matches the functionality of bdistwininst is at least good enough for those projects that currently use bdistwininst. The only potential issue right now is with postinstall scripts, which bdistsimple doesn't support. It would be easy enough to add, and indeed it may be possible to use existing packaging functionality already (I haven't looked into this area).

The packaging machinery contains a reasonably good set of hooks which a setup.cfg can plug into, which is IMO more flexible than just a post- installation script (e.g. sometimes you need your code to run before installation, to determine where to install things).

I don't disagree, but I'm struggling to see how that would be done.

See example below.

Can you give an example of a setup.cfg, that would work like this? Suppose I have two files, foo.py and bar.pyd, which are a pure-python module and a compiled C extension. How would I write a setup.cfg and lay out the directory structure to allow pysetup install to do the right thing? I tried to do this myself, but couldn't get it to work the way I expected. It may be I was just hitting bugs, but it felt to me like I was going at the problem the wrong way.

It may not work for you, because in the default branch, packaging is currently missing some functionality or has bugs (I've raised about a dozen issues since trying to get packaging working with virtual environments).

In the pythonv branch (which is pretty up to date with default), I've added the missing functionality/fixed some of the issues. Here's an example: I've created an empty virtual environment. Here are the contents of it at the moment:

C:\TEMP\VENV │ env.cfg │ ├───Include ├───Lib │ └───site-packages └───Scripts activate.bat deactivate.bat pysetup3-script.py pysetup3.exe [various stock Python binaries omitted]

Here's an example setup.cfg:

[global] setup_hooks = hooks.setup

[install_data] pre-hook.win32 = hooks.pre_install_data

[metadata] name = dory version = 0.1 requires_python = >= 3.3 [other metadata omitted]

[files] modules = dory packages = apackage apackage.asubpackage

scripts = dory

extra_files = hooks.py

resources = data/data.bin = {data} compiled/ spam.pyd = {compiled}

#[extension: spam] #language = c #sources = spammodule.c

The extension section is commented out because we're not building the extension at installation time. This setup.cfg works hand-in-hand with some hooks, in file hooks.py below:

def setup(config): # Can do various checks here. For example, platform # compatibility checks, or set up different binaries # to install for x86 vs. x64, etc. # Do this by setting up config['files']['resources'] # appropriately based on installation time environment. pass

def pre_install_data(cmd): # assumes os.name == 'nt' for simplicity of this example cmd.categories['compiled'] = '%s/Lib/site-packages' % cmd.install_dir

Notice how in setup.cfg, file 'spam.pyd' in 'compiled' is expected to be copied to category 'compiled', whose path is set in hooks.pre_install_data.

Here's the project directory:

C:\USERS\VINAY\PROJECTS\DORY │ dory │ dory.py │ hooks.py │ setup.cfg │ ├───apackage │ │ init.py │ │ │ └───asubpackage │ init.py │ ├───compiled │ spam.pyd │ └───data data.bin

At the moment, the "spam" module is of course not installed:

(venv) c:\Users\Vinay\Projects\dory>python -c "import spam" Traceback (most recent call last): File "", line 1, in ImportError: No module named 'spam'

Now, we install the project:

(venv) c:\Users\Vinay\Projects\dory>pysetup3 install . Installing from source directory: c:\Users\Vinay\Projects\dory running install_dist running build running build_py running build_scripts copying and adjusting dory -> build\scripts-3.3 running install_lib creating C:\temp\venv\Lib\site-packages\apackage byte-compiling C:\temp\venv\Lib\site-packages\apackage_init_.py to init.c python-33.pyc byte-compiling C:\temp\venv\Lib\site-packages\dory.py to dory.cpython-33.pyc running install_scripts running pre_hook hooks.pre_install_data for command install_data running install_data running install_distinfo creating C:\temp\venv\Lib\site-packages\dory-0.1.dist-info creating C:\temp\venv\Lib\site-packages\dory-0.1.dist-info\METADATA creating C:\temp\venv\Lib\site-packages\dory-0.1.dist-info\INSTALLER creating C:\temp\venv\Lib\site-packages\dory-0.1.dist-info\REQUESTED creating C:\temp\venv\Lib\site-packages\dory-0.1.dist-info\RESOURCES creating C:\temp\venv\Lib\site-packages\dory-0.1.dist-info\RECORD

Now, the virtualenv looks like this:

C:\TEMP\VENV │ env.cfg │ ├───data │ data.bin │ ├───Include ├───Lib │ └───site-packages │ │ dory.py │ │ spam.pyd │ │ │ ├───apackage │ │ │ init.py │ │ │ │ │ ├───asubpackage │ │ │ │ init.py │ │ │ │ │ │ │ └───pycache │ │ │ init.cpython-33.pyc │ │ │ │ │ └───pycache │ │ init.cpython-33.pyc │ │ │ ├───dory-0.1.dist-info │ │ INSTALLER │ │ METADATA │ │ RECORD │ │ REQUESTED │ │ RESOURCES │ │ │ └───pycache │ dory.cpython-33.pyc │ └───Scripts activate.bat deactivate.bat dory-script.py dory.exe pysetup3-script.py pysetup3.exe [various stock Python binaries omitted]

Now we can import "spam" and run something from it:

(venv) c:\Users\Vinay\Projects\dory>python -c "import spam; spam.system('cd')" c:\Users\Vinay\Projects\dory

I'd like to see a bdistxxx command to do the build step as you describe, if only to make it trivially simple for developers to produce binary distributions. Having to package stuff up manually is bound to put at least some developers off. If you can give me the example I mentioned above, I could work on modifying the bdistsimple code I posted to the tracker today to produce that format rather than my custom format based on bdistwininst.

Example as above, though you may need to use the pythonv branch to actually get it working. I can zip up the directory and send it to you, but at the moment there's missing functionality in pythonv in terms of the link step when building the extension. (I overcame this by linking manually .) If you want, I can zip all the project files up and send them to you.

In the more general case, one might want an arrangement with a directory structure like compiled/x86/..., compiled/x64/... in the built zip, with the hooks.py code setting up the resources appropriately based on the target environment as determined at installation time.

For the installation step, you shouldn't even need to unzip, as pysetup3 can do the unpacking for you.

Indeed, with pythonv I could just zip the whole "dory" project directory and install with e.g. "pysetup3 install dory-1.0-win32-py3.3.zip".

Agreed. Personally, as I've said, I'm happy not to use Add/Remove even for system installations - pysetup list and pysetup remove do what I need without slowing down the Add/Remove list. But I accept that's not likely to be the view of many Windows users. Anyone using virtual envs, though, is probably by definition comfortable enough with command line tools to be willing to use pysetup3.

A fair subset of those who must have ARP integration will probably also want to install using MSI, so that would be taken care of by having a good bdist_simple -> bdist_msi conversion.

Regards,

Vinay Sajip

More information about the Python-Dev mailing list