Issue 5311: bdist_msi generates version number for pure Python packages (original) (raw)
Created on 2009-02-19 04:16 by bethard, last changed 2022-04-11 14:56 by admin. This issue is now closed.
Messages (26)
Author: Steven Bethard (bethard) * 
Date: 2009-02-19 04:16
I just ran "setup.py bdist_msi" with NLTK which is a pure Python package. You can see the setup.py here: http://code.google.com/p/nltk/source/browse/trunk/nltk/setup.py. Despite the fact that NLTK is pure Python, the generated .msi's look like nltk-0.9.8.win32-py2.5.msi and nltk-0.9.8.win32-py2.6.msi (built with Python 2.5 and 2.6 respectively).
So, two questions: (1) are the version numbers supposed to be there? and (2) if so, does that mean a .msi for a pure Python package built by Python 2.6 won't work on any other version?
Author: Martin v. Löwis (loewis) *
Date: 2009-02-19 06:45
So, two questions: (1) are the version numbers supposed to be there?
Yes.
(2) if so, does that mean a .msi for a pure Python package built by Python 2.6 won't work on any other version?
Yes.
Author: Steven Bethard (bethard) *
Date: 2009-02-19 07:02
Mostly out of curiosity, why is that? With bdist_wininst, a pure Python package would generate a version-less installer that could then be used with any Python version.
Author: Martin v. Löwis (loewis) *
Date: 2009-02-19 07:15
Mostly out of curiosity, why is that?
Primarily because it's not implemented. To implement it, you would need to collect all Python installations on the system from the registry, then create a UI to let the user select a specific installation, then use that. Collecting all Python versions is fairly difficult to do with standard MSI actions.
In addition, a common use case is that MSI installation works unattended (no UI), in which case you would also have to make a choice of default version to install to (e.g. with highest version number).
Contributions are welcome.
Author: Steven Bethard (bethard) *
Date: 2009-02-19 07:20
I'm certainly no Windows API expert, but if no one takes a stab at it sooner, maybe I can spend some time looking at this during PyCon.
I'm switching the ticket type to a feature request.
Author: Steven Bethard (bethard) *
Date: 2009-03-31 08:05
Ok, so here's what needs to happen to make this work. Note that all of the following needs to happen at runtime, not at the time at which the .msi is created:
(1) Find all sub-keys of SOFTWARE\Python\PythonCore in the registry. These are the versions, e.g. 2.5, 2.6, 3.0, etc.
(2) For each version, get the Python installation dir from SOFTWARE\Python\PythonCore%(version)s\InstallPath
(3) Populate the ListView table with entries containing these installation paths, e.g. something like:
TARGETDIR 1 C:\Python24
TARGETDIR 2 C:\Python25
...(4) Modify Control/SelectDirectoryDlg so that it uses a ListView (filled with the above values) instead of a DirectoryCombo.
(5) Make a couple minor edits to bdist_msi.py to stop it from inserting the version into the .msi name for Python-only modules.
I looked into a couple ways of doing this. Ideally, we should avoid using a CustomAction, which would require maintaining some additional C or VBScript code, and instead do everything through the database tables that are built into all .msi files. Some problems I've run into with this approach:
The only way to read the registry AFAICT is through the RegLocator table: http://msdn.microsoft.com/en-us/library/aa371171(VS.85).aspx. But RegLocator can only look up single key values, and cannot look up the sub-keys under a key (which is what we need to get the Python versions).
We could hard code all the possible versions, and stick all the corresponding SOFTWARE...%(version)s\InstallPath keys into RegLocator. Then these can be read into properties using the AppSearch table (http://msdn.microsoft.com/en-us/library/aa371559(VS.85).aspx), and we could then fill in the ListView table with the install paths. But AFAICT, there is no way to keep from putting one row into the ListView table for each version of Python we statically define. Which means a bunch of inappropriate rows at runtime (e.g. there'd be a row for Python 2.3 even if there was no Python 2.3 on your system).
Basically, the problem is that we'd like to determine what goes into the ListView table at runtime, but I can only figure out how to put things into it at the time at which the .msi is built.
I'm going to continue to look into this, and I'd welcome suggestions anyone has.
The last resort would be to create a CustomAction using a DLL or VBScript, but I'm really trying to avoid that, both because maintaining and debugging such code is painful, and because Michael Foord suggested that some virus checkers will complain about .msi files with embedded VBScript (probably forcing me to maintain a DLL - ugh!).
Author: Martin v. Löwis (loewis) *
Date: 2009-03-31 14:56
In
http://www.installsite.org/pages/en/msi/articles/MultiListBox/index.htm
there is a demo how to modify the listbox contents dynamically using VBScript.
Author: Steven Bethard (bethard) *
Date: 2009-04-01 11:30
Thanks for the link. I did play around with that code for quite a while, and while I'm convinced there's a way to get it to work, I'm still struggling with it. I believe the attached .vbs file does about what we need to do, but I think I'm attaching it at the wrong location in the .msi file because things seem to die at the "Session.Database.OpenView" line with an error saying the object doesn't have that method (or no error message but still at that line, depending on how I've attached the action.)
Anyway, I'll probably keep playing around with this, but some food for thought on why we may not want to implement this as a .vbs: http://blogs.msdn.com/robmen/archive/2004/05/20/136530.aspx In particular, some anti-virus products will silently keep them from working. That said, I probably need to figure out how to write the .vbs code in order to (eventually) write the C or whatever else code.
Author: Steven Bethard (bethard) *
Date: 2009-04-01 12:26
My OpenView bug was just a missing "Set". The CustomAction does seem to be correctly gathering the Python paths from the registry and filling the ListView now. I've still got a couple of errors showing up later in the process, but I expect I'll probably have a patch within a week or two.
Author: Steven Bethard (bethard) *
Date: 2009-04-08 00:56
Ok, I've made some progress on this. The attached patch now generates MSIs which are version agnostic and look up the appropriate Python version in the registry. Some things still remaining to do:
The ProductName needs to be modified at runtime to prefix the "Python X.Y" for Add/Remove Programs (ARP). I have a custom action making the appropriate modifications, but this doesn't seem to affect the name in ARP. Not entirely sure what the problem is, but it seems I may only be setting the property on the client, and not on the server. I'll continue looking into this.
I need to thoroughly test that it does the right thing when a particular version of Python is required. The code should basically be there now to do that, but I haven't tested it at all. Probably I need to play around both with target_version and actually building an extension module.
Speaking of tests, I have no idea how I would write a unittest for any of these things. To check some of them, it seems like you'd actually have to install the .msi onto a Windows machine. So at the moment, all my tests are by hand. Any better suggestions for testing these kinds of things would be greatly appreciated.
Author: Steven Bethard (bethard) *
Date: 2009-04-30 19:09
I'm still stuck on getting the right name to show up in ARP.
Another problem: it seems like we have to update the ProductCode at runtime as well - otherwise, you can only have one module installed for all the versions of Python you have on your machine. But if we generate a new GUID at runtime, how do we make sure that the GUID for moduleXXX + Python 2.6 is the same every time (as it should be for the MSI to recognize when a product is being removed, reinstalled, etc.)
Thoughts? Is there a way to deterministically generate a GUID for each Python version? (Note that it doesn't make sense to do this ahead of time - an installer created for Python 3.1 should work with Python 3.2 as well.)
Author: Steven Bethard (bethard) *
Date: 2009-04-30 19:20
Updated the patch to make sure ProductName is set before ValidateProductID.
Author: Martin v. Löwis (loewis) *
Date: 2009-04-30 19:23
I'm still stuck on getting the right name to show up in ARP.
It may that indeed Installer blocks the property from being passed onto the server side. Three things to try:
- inspect the log file, to see whether it is passed, and then whether it gets set. You do logging with "msiexec /i foo.msi /l*v foo.log".
- add the property to SecureCustomProperties, to have it passed to server mode; by default, only properties in UPPER_CASE get passed.
- alternatively, add an upper-case property, and make ProductName computed.
Another problem: it seems like we have to update the ProductCode at runtime as well
I knew that would cause problems some day :-) I expected you to desire this only for a single installation. Installing multiple copies is much more difficult.
IIUC, a common approach is to use transforms, although I'm not sure how precisely that would work.
A hacky approach might be to use computed uuids (if that can work at all: I'm skeptical that you can change the productcode at runtime): have a fixed ProductCode in the MSI, and then add the minor Python version to the last digit of uuid. See Tools/msi/msi.py for how the uuid of the Win64 installer changes from the one for the 32-bit installer.
I think this is really is a question to ask on some MSI channels; most likely, the answer is that this cannot possibly work.
Author: Steven Bethard (bethard) *
Date: 2009-05-02 18:54
Ok, I've been chatting with folks on microsoft.public.platformsdk.msi, and I think the right approach here is to define a Feature for each version of Python. Each Feature would install the exact same files, but to a different Python directory.
One of the nice things about this approach would be that you could install a module for multiple Python versions all at the same time. It also should avoid the need for a VBScript or C-based CustomAction - we should be able to do everything with the database tables.
One of the downsides is that we'll have to hard-code in all possible versions of Python. My current plan is just to declare all versions as 2.0, 2.1, ..., 3.9. That should be good enough for a while, but at some point this will have to get updated to work with Python 4.0. ;-)
I'll post back here when I've made some progress on this.
Author: Steven Bethard (bethard) *
Date: 2009-05-02 20:54
Ok, that was actually easier than I thought it would be. The new patch introduces properties for each Python version (e.g. TARGETDIR2.4, PYTHON.MACHINE.2.4, etc.), and disables and hides the features for any Python versions that aren't found in the registry.
The one remaining issue: What should we do about Python installations that are missing the appropriate keys in the registry? I imagine this could happen if, say, you build Python from source. My first thought was to add an "Other Python Installation" Feature that is disabled (but visible) by default, and allow the path for that Feature to be filled in by hand. Does that make sense?
Author: Steven Bethard (bethard) *
Date: 2009-05-03 01:04
Ok, I added one final Feature that allows the user to specify an alternate Python directory. (The PathEdit for specifying the directory will only display if this Feature is set to be installed.)
I think this patch is pretty much ready to go in now. It could use a review and some testing from folks on other kinds of machines, but I think all the functionality is there now. (And no need for CustomAction scripts! Yay!)
If at all possible, I'd like to get this into the Python 3.1 beta so that we can have as many people as possible test this out on other Windows machines.
Author: Steven Bethard (bethard) *
Date: 2009-05-03 19:28
A slightly improved patch, using DuplicateFile instead of storing a copy of each file for each Python version. Should keep the size of the resulting MSI similar to the size of the currently generated MSIs.
Author: Steven Bethard (bethard) *
Date: 2009-05-03 20:12
Ok, one last tiny update that makes sure TARGETDIR is always set to one of the TARGETDIRX.Ys from a Feature that is actually selected.
I swear I'm done with this now. ;-)
Author: Martin v. Löwis (loewis) *
Date: 2009-05-03 20:23
Can you kindly attach a demo MSI, to simplify review?
Author: Steven Bethard (bethard) *
Date: 2009-05-03 20:27
Here's an MSI generated for the argparse module.
Author: Martin v. Löwis (loewis) *
Date: 2009-05-04 16:18
The patch looks fine so far, please apply to trunk and 3k. As this is a new feature, I think backporting it is not appropriate.
I believe that the support for (pre)install scripts is incorrect; I would expect that each such script should be executed once per version for which the package is installed. However, this can be fixed later; for the moment, it would be good enough to warn at packaging time that scripts may not work for multi-version packages.
You can also consider providing "Change" installation, allowing the user to selectively add a package to a python version after that python version is installed (or to remove it before the python version gets removed). To do so, just add an option to the MaintenanceTypeDlg that forks into the feature selection dialog, with the standard INSTALL action.
Author: Anthony Tuininga (atuining) *
Date: 2009-05-04 18:22
One additional suggestion: allow the packager to specify what the minimum Python version is. Otherwise, you might have a package that enables installation for Python 2.3 and 2.4 when the maintainer has already stated that Python 2.5 is the minimum version supported. Otherwise, looks great.
Author: Steven Bethard (bethard) *
Date: 2009-05-04 18:53
@martin: Thanks! Do I need to do something special to make the merging work right? Or do I just apply the patch separately to the trunk and the py3k branche?
Good point about the install script - I think the condition needs to be ("&Python%s=3" % ver) instead of "NOT Installed". I'll use this approach when I apply the patch, but clearly this needs some testing by folks that use install scripts either way. I didn't worry much about the pre-install script, since finalize_options() already throws an exception saying that isn't supported
I'll open a new issue for support for a "Change" installation. This is one of the nice things that treating Python versions as features allows, I just haven't had time to look into it yet.
@anthony: Allowing a minimum version to be set sounds like a nice feature. Could you open a new feature request?
Author: Anthony Tuininga (atuining) *
Date: 2009-05-04 19:31
I've created another feature request as requested for supplying a minimum Python version when creating pure Python packages.
http://bugs.python.org/issue5926
Author: Martin v. Löwis (loewis) *
Date: 2009-05-04 20:14
Thanks! Do I need to do something special to make the merging work right? Or do I just apply the patch separately to the trunk and the py3k branche?
You commit to the trunk, then you do "svnmerge merge -r" in the 3k branch, then "svn commit -F svnmerge-something.txt" (in case of conflicts, you fix them first, of course).
Good point about the install script - I think the condition needs to be ("&Python%s=3" % ver) instead of "NOT Installed".
I'm not sure - I think the install script must run several times actually, so there must be several custom actions, each with its own condition.
I'll open a new issue for support for a "Change" installation. This is one of the nice things that treating Python versions as features allows, I just haven't had time to look into it yet.
Well, if there had been separate packages per version, you could install them independently, anyway.
Author: Steven Bethard (bethard) *
Date: 2009-05-05 02:23
You commit to the trunk, then you do "svnmerge merge -r" in the 3k branch, then "svn commit -F svnmerge-something.txt" (in case of conflicts, you fix them first, of course).
Done in r72306 for trunk and r72309 for 3k. Thanks.
I'm not sure - I think the install script must run several times actually, so there must be several custom actions, each with its own condition.
Yep. That's what the patch does - adds a CustomAction and an entry in InstallExecuteSequence for each version.
Well, if there had been separate packages per version, you could install them independently, anyway.
For what it's worth, you still have this option if anyone wants to distribute multiple MSIs. Simply pass --target-version to bdist_msi, and you'll get an MSI for only the selected version of Python that you can install independently of other MSIs generated with --target-version.
History
Date
User
Action
Args
2022-04-11 14:56:45
admin
set
github: 49561
2009-05-05 02:23:30
bethard
set
status: open -> closed
resolution: accepted -> fixed
messages: +
2009-05-04 20:14:56
loewis
set
messages: +
2009-05-04 19:31:06
atuining
set
messages: +
2009-05-04 18:53:58
bethard
set
messages: +
2009-05-04 18:22:31
atuining
set
nosy: + atuining
messages: +
2009-05-04 16🔞41
loewis
set
resolution: accepted
messages: +
2009-05-03 20:27:31
bethard
set
files: + argparse-0.9.1.win32.msi
messages: +
2009-05-03 20:23:32
loewis
set
messages: +
2009-05-03 20:12:20
bethard
set
files: - bdist_msi.patch
2009-05-03 20:12:05
bethard
set
files: + bdist_msi.patch
messages: +
2009-05-03 19:28:49
bethard
set
files: - bdist_msi.patch
2009-05-03 19:28:34
bethard
set
files: + bdist_msi.patch
messages: +
2009-05-03 01:04:58
bethard
set
files: + bdist_msi.patch
messages: +
2009-05-02 20:54:14
bethard
set
files: + bdist_msi.patch
messages: +
2009-05-02 18:54:26
bethard
set
messages: +
2009-04-30 19:23:49
loewis
set
messages: +
2009-04-30 19:20:20
bethard
set
files: + bdist_msi.patch
messages: +
2009-04-30 19:19:08
bethard
set
files: - bdist_msi.patch
2009-04-30 19:09:36
bethard
set
messages: +
2009-04-08 00:56:39
bethard
set
files: + bdist_msi.patch
keywords: + patch
messages: +
2009-04-08 00:56:03
bethard
set
files: - PythonVersions.vbs
2009-04-01 12:26:14
bethard
set
assignee: tarek -> bethard
messages: +
2009-04-01 11:30:47
bethard
set
files: + PythonVersions.vbs
messages: +
2009-03-31 14:56:32
loewis
set
messages: +
2009-03-31 08:05:49
bethard
set
messages: +
2009-02-23 05:36:46
tarek
set
versions: + Python 3.1, Python 2.7, - Python 2.6, Python 2.5
2009-02-19 07:20:18
bethard
set
type: behavior -> enhancement
messages: +
2009-02-19 07:15:11
loewis
set
messages: +
2009-02-19 07:02:17
bethard
set
messages: +
2009-02-19 06:45:23
loewis
set
nosy: + loewis
messages: +
2009-02-19 04:16:05
bethard
create