[Python-Dev] PEP 376 proposed changes for basic plugins support (original) (raw)

M.-A. Lemburg mal at egenix.com
Mon Aug 2 22:37:39 CEST 2010

Michael Foord wrote:

On 02/08/2010 20:36, M.-A. Lemburg wrote:

Michael Foord wrote:

On 02/08/2010 13:31, exarkun at twistedmatrix.com wrote:

On 12:21 pm, mal at egenix.com wrote:

Tarek Ziad� wrote:

On Mon, Aug 2, 2010 at 3:06 AM, P.J. Eby<pje at telecommunity.com> wrote: ..

So without specific examples of why this is a problem, it's hard to see why a special Python-specific set of configuration files is needed to resolve it, vs. say, encouraging application authors to use the available alternatives for doing plugin directories, config files, etc. I don't have a specific example in mind, and I must admit that if an application does the right thing (provide the right configuration file), this activate feature is not useful at all. So it seems to be a bad idea. I propose that we drop the PLUGINS file idea and we add a new metadata field called Provides-Plugin in PEP 345, which will contain the info I've described minus the state field. This will allow us to expose plugins at PyPI. IOW, have entry points like setuptools provides, but in a metadata field instead of a entrypoints.txt file. Do we really need to make Python packaging even more complicated by adding support for application-specific plugin mechanisms ? Packages can already work as application plugins by simply defining a plugins namespace package and then placing the plugin packages into that namespace. See Zope for an example of how well this simply mechanism works out in practice: it simply scans the "Products" namespace for sub-packages and then loads each sub-package it finds to have it register itself with Zope. This is also roughly how Twisted's plugin system works. One drawback, though, is that it means potentially executing a large amount of Python in order to load plugins. This can build up to a significant performance issue as more and more plugins are installed. unittest will solve this problem by having plugins explicitly enabled in its own configuration system, and possibly managed through a separate tool like a plugins subcommand. The full package list will only need to be scanned when managing plugins, not during normal execution. Having this distutils2 supported "plugin declaration and discovery" will be extremely useful for the unittest plugin system. Given that plugins may need configuring after installation, and tools that handle both activation and configuration can be provided, it doesn't seem a heavy cost. The downside to this is that installing and activating plugins are two separate steps. Given that each project can have a different set of plugins enabled I don't see a way round it. You might want to take a look at the Trac plugin system which works in more or less the same way: http://trac.edgewall.org/wiki/TracPlugins Ouch. I really don't want to emulate that system. For installing a plugin for a single project the recommended technique is: * Unpack the source. It should provide a setup.py. * Run: $ python setup.py bdistegg Then you will have a *.egg file. Examine the output of running python to find where this was created. Once you have the plugin archive, you need to copy it into the plugins directory of the project environment For global plugins it just uses entry points, which is similar to the functionality we are suggesting adding... However note: Unlike plugins installed per-environment, you'll have to explicitly enable globally installed plugins via trac.ini. Really this sounds astonishingly like the system we are proposing. :-) (Global discovery with per-application choice about whether or not installed plugins are actually used).

The difference being that Trac is usually hosted using a separate Python installation, so the pre-application choice is really a per-Trac instance choice.

But yes, the system you are proposing does sound a lot like what Trac uses and it works well - for that one application.

Since applications tend to have a rather diverse set of needs for plugins, I don't think we should add plugins support to PEP 376. We are really just suggesting adding entry points.

Tarek's email sounded a lot like the attempt to come up with a universal plugin system, both in terms of managing installed plugins and their configuration.

Perhaps I've just missed some twist in the thread :-)

Users of applications will not want to edit a single configuration file to maintain plugins of many different applications This we are not proposing. Nor were we ever proposing it. The single file that was proposed (and in my understanding is no longer proposed) was to be maintained by distutils2 anyway.

Sorry, I was refering to the plugins.cfg file used for enabling the plugins, not the PLUGINS file used by installers.

http://mail.python.org/pipermail/python-dev/2010-August/102627.html

(they might break some other application doing so) and sys admins will have trouble with such a setup as well (they usually want to have control over which plugins get used for various reasons).

In the end, you'd have a system wide plugin configuration (maintained by the sys admin), a per user one (with local customizations) and a per application one (providing application-specific defaults) - which only increases complexity and doesn't really solve anything. We simply provide information about the availability of plugins. System administrators or users can control the use of this information (and the plugins) as per their own policies. Instead, I'd suggest to let each application do its own little thing to manage plugins, in a complex or simple way, with or without configuration, and have them all live happily side-by-side. The stdlib should really only provide tools to applications and make useful suggestions, not try to enforce application design choices. I think that's simply out of scope for the stdlib

Well, a tool for application developers is pretty much all that is being proposed.

Right, but one which has consequences for users of applications relying on the feature.

setuptools was also "just" a tool for application developers, but one which had some serious side-effects for users.

Let's please not play the same trick again and be more careful about the user side of things, e.g. plugin configuration should not be part of a Python packaging system.

Plugin discovery is useful, but doesn't really require yet another lookup file. The few bits of extra information could easily be placed into the distribution meta-data of PEP 345.

Perhaps the main motivation behind adding a new PLUGINS file is to reduce the overhead of having to scan dozens of meta-data .dist-info directories ?!

If that's the case, then it would be better to come up with an idea of how to make access to that meta-data available in a less I/O intense way, e.g. by having pip or other package managers update a central SQLite database cache of the data found on disk.

-- Marc-Andre Lemburg eGenix.com

Professional Python Services directly from the Source (#1, Aug 02 2010)

Python/Zope Consulting and Support ... http://www.egenix.com/ mxODBC.Zope.Database.Adapter ... http://zope.egenix.com/ mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/

::: Try our new mxODBC.Connect Python Database Interface for free ! ::::

eGenix.com Software, Skills and Services GmbH Pastor-Loeh-Str.48 D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg Registered at Amtsgericht Duesseldorf: HRB 46611 http://www.egenix.com/company/contact/

More information about the Python-Dev mailing list