Issue 13501: Make libedit support more generic; port readline / libedit to FreeBSD (original) (raw)
Created on 2011-11-29 15:19 by ngie, last changed 2022-04-11 14:57 by admin. This issue is now closed.
Messages (39)
Author: Enji Cooper (ngie) *
Date: 2011-11-29 15:18
The attached patch enables libedit support in a generic way via configure.in, so I can pass in --with-readline=editline, --with-readline=readline, --with-readline=yes, or --with[out]-readline[=no] and it will do one of the following:
- Enable editline support.
- Enable readline support.
- Enable default OS support (editline on OSX, readline otherwise).
- Explicitly disable readline support.
Tested functional via cmd.py and with basic poking around via the readline module.
The attached patch was created against the 2.7 branch, but I'll produce a patch against 'trunk' sometime later on this week.
PS. The only quirk I found was the fact that FreeBSD 9's libedit lied when it reports the number of available history items. Hrmmm..
Author: Éric Araujo (eric.araujo) * 
Date: 2011-12-03 14:53
ISTM --with-readline=yes should just be --with-readline, and the =no forms should just be --without-readline. That would be more in line with other options and less confusing (--without-readline=no ?!).
There is no trunk anymore now that we’ve switched away from Subversion; you should probably work on the Mercurial default branch (see devguide).
Author: Éric Araujo (eric.araujo) *
Date: 2011-12-03 14:54
And if this is really two different requests (port readline module to FreeBSD i.e. change if APPLE to if HAVE_EDITLINE and give more control about readline vs. editline in the configure script), then two reports should be opened.
Author: Ned Deily (ned.deily) *
Date: 2011-12-03 18:05
Without having yet done a detailed review of the patch and the configure options, I don't see a need to open a second issue. The scope of this one is fine: generalizing the support of libedit to other platforms.
Author: Martin Panter (martin.panter) *
Date: 2015-05-24 09:46
Garrett Cooper: If you are still around, you might want to review the following three changes in this new patch. I suspect they were mistakes in your version, but I cannot be 100 percent sure.
- Restoring length - 1 subtraction in call_readline() at line 1239
- Change #if APPLE back to #ifdef
- Gnu Readline requires <stdio.h> to be included first, so I fixed the configure.ac script
Other changes in this new patch:
- Merged with current code
- Remove outdated comment about using a runtime library check
- Couple minor corrections to comments, conditional variable declarations, etc
- Fixed wrong HIST_ENTRY pointer type used in a Gnu Readline conditional branch
- Changed double single quote sign back to single single quote. I am no Autocrap expert, but it screwed up the generated comment in pyconfig.h.in.
- Add ability to include with <editline/readline.h>
Tested on Arch Linux with both Gnu Readline 6.3.008 and Editline 20150325-3.1 available:
No readline argument: Uses Gnu Readline --with-readline=editline: Uses Editline --with-readline=readline: Gnu Readline --with-readline=yes: Gnu Readline --with-readline: Gnu Readline --with-readline=no: No “readline” module --without-readline: No “readline” module --without-readline=no: error: invalid package name: readline=no
However after successfully compiling with Editline, there are a couple bugs with keystrokes or output not being synchronized. I do not think I will spent much more effort on it. But perhaps other people are interested in taking this further.
Author: Ned Deily (ned.deily) *
Date: 2015-06-06 01:58
See also suggested patch in Issue24388.
Author: Ed Maste (emaste)
Date: 2015-06-06 02:02
This issue causes the LLDB debugger to crash on FreeBSD (it uses Python as its embedded script interpreter).
What needs to be done to make some progress on this issue?
Author: Ed Maste (emaste)
Date: 2015-06-06 02:07
Note that the patch in Issue24388 is more a proof of concept. I'm not sure it's the "right" fix.
LLDB is a bit of a special case: LLDB links against libedit, but the Python libedit module is built as if readline is in use. It turns out this "magically" works out, presumably due to the runtime workaround detection. As far as I know this issue would affect Linux as well, but perhaps the version of libedit on common Linux distributions is one with the 0-based vs 1-based history fix?
Author: Ned Deily (ned.deily) *
Date: 2015-06-06 02:24
The suggested change to generalize support for libedit to other platforms is a new feature so, by default, it would first appear in a new feature release, e.g. 3.6. There would probably have to be an exception granted to add it to 2.7.x or other 3.x releases, which are in maintenance mode. I think the quickest workarounds are to either link Python with GNU readline or, if that is not acceptable, persuade FreeBSD to carry a patch similar to the one you suggested for their current versions of Python. WRT Linux, it looks like Debian has both a libeditline0 package (presumably the old ABI) and a libedit2 package and their lldb package is linked with the latter. (The Debian Pythons are always linked with GNU readline.)
Author: Ed Maste (emaste)
Date: 2015-06-15 17:14
It looks like rust developers hit the issue in Issue24388 with lldb on Ubuntu 15.04 as well: https://github.com/rust-lang/rust/issues/26297
Author: Ed Maste (emaste)
Date: 2015-06-15 17:23
Actually, in I should claim the issue is with libedit / GNU readline compatibility and/or the workarounds in Python's readline module, not that it's specifically Issue24388.
Author: Martin Panter (martin.panter) *
Date: 2015-06-15 23:13
Maybe I am missing something, but is it possible to use a newer version of Editline (libedit) that fixes the compatibility bug, as mentioned in Issue 18458?
Author: Ed Maste (emaste)
Date: 2015-06-16 13:23
I believe the 0-based vs 1-based history is only one of a few different inconsistencies between libedit and readline. Workarounds will be necessary until a fixed libedit is deployed on all operating systems / distros of interest, but yes I agree that eventually they should not be needed.
Author: Ned Deily (ned.deily) *
Date: 2016-02-11 02:12
Martin, thanks for updating the patch. I've left some review comments on Rietveld. After reviewing it, I think Garrett's original specification is correct: there is a need for four options to preserve current expected behavior although the default is slightly more complicated than stated. The current behavior is: enable building readline with GNU readline if found on the search paths, possibly modified by CPPFLAGS and LDFLAGS, and, if not found and on OS X, use editline if found on the search paths. Some build scripts depend on that behavior.
Author: Ned Deily (ned.deily) *
Date: 2016-02-11 02:17
That said, it might be OK to change the default behavior to just remove the "and on OS X" condition: enable building readline with GNU readline if found on the search paths, possibly modified by CPPFLAGS and LDFLAGS, and, if not found, use editline if found on the search paths. That's probably the desired behavior on FreeBSD systems where, AFAIK, GNU readline isn't shipped by default but BSD editline is.
Author: Martin Panter (martin.panter) *
Date: 2016-02-12 23:34
Thanks for the review Ned. I haven’t looked too closely but your comments sound reasonable in general. I don’t have a personal need to update this patch (Editline doesn’t work well for me on Linux), but I can try to update it if there is demand for this.
Regarding the leftover #ifdef APPLE bits, I would have to double-check, but I presume I didn’t need to enable those for my version of Editline, so I presumed they were Apple-specific bugs.
Author: Martin Panter (martin.panter) *
Date: 2016-03-20 07:40
Quoting Ned: “The current behavior is: enable building readline with GNU readline if found on the search paths, . . . and, if not found and on OS X, use editline if found on the search paths. Some build scripts depend on that behavior.”
Can you point out any logic (e.g. in configure.ac) that handles this? I suspect you might be mistaken. My understanding is that we always include <readline/readline.h> and always link against -lreadline, regardless of platform. My guess is that on OS X, the include file and library are actually pointers to Editline code, rather than Gnu code.
This is in contrast to my situation on Arch Linux, where I can install both the Gnu “readline” and the “libedit” packages without conflict. Gnu Readline is accessible via <readline/readline.h> and -lreadline, and Editline’s Readline compatibility is accessible via <editline/readline.h> and -ledit.
So I suspect there is no special on-OS-X condition to change or remove (if we ignore those #ifdef APPLE bug workarounds for the moment). Maybe I should drop that DEFAULT_LIBREADLINE business from the patch.
Author: Martin Panter (martin.panter) *
Date: 2016-03-22 05:57
Patch v3 is an update taking into account Ned’s comments, and my own now deeper understanding:
- Reworked the --with-readline handling. Suggested options are --with-readline=editline, and --without-readline. The current behaviour (use -lreadline if possible) should still work by default.
- Adjust the style of the configure.ac syntax to match surrounding code.
- Restore rl_completion_suppress_append check, independent of rl_completion_append_character.
- Remove completion_matches() #ifdef madness, originally spawned by Issue 1703270, which makes no sense to me.
- Restored the runtime Editline checks protected by APPLE, but if WITH_EDITLINE is specified (e.g. on other platforms) the Editline support is always enabled and there are no runtime checks.
Author: Martin Panter (martin.panter) *
Date: 2016-06-02 06:31
Patch v4 resolves conflicts against the current code. I also used AC_CHECK_DECL invocations instead of AC_COMPILE_IFELSE in the configure script.
FTR I think my problems with the prompt and the terminal settings are caused by a line of code that is commented out in the Apple version. See <https://gnats.netbsd.org/48957> and <http://cvsweb.netbsd.org/bsdweb.cgi/src/lib/libedit/readline.c.diff?r1=1.110&r2=1.111>. Also, the following test fails for me when using Editline:
====================================================================== ERROR: testHistoryUpdates (test.test_readline.TestHistoryManipulation)
Traceback (most recent call last): File "/home/proj/python/cpython/Lib/test/test_readline.py", line 38, in testHistoryUpdates readline.replace_history_item(0, "replaced line") ValueError: No history item at position 0
Third, test_zipimport passes, but messes up the terminal (staircase effect). Probably related to the Apple patch mentioned above.
Author: Mark Harfouche (Mark Harfouche)
Date: 2018-08-19 16:29
Is there still interest in this?
Author: Romain Geissler (RomainGeissler1A)
Date: 2018-10-28 18:40
Hi,
Yes this bug report is still valid. I just tried compiling Python 3.7.1 on linux with libedit installed (https://thrysoee.dk/editline/) but not libreadline, and indeed the configure script fails to detect I have it installed and falls back on the python readline emulation.
Note: I am using Linux, not Mac OS.
Cheers, Romain
Author: Mark Harfouche (Mark Harfouche)
Date: 2018-10-28 18:42
My workaround has been to use neither readline nor libedit.
On Sun, Oct 28, 2018 at 2:40 PM Romain Geissler <report@bugs.python.org> wrote:
Romain Geissler <romain.geissler@amadeus.com> added the comment:
Hi,
Yes this bug report is still valid. I just tried compiling Python 3.7.1 on linux with libedit installed (https://thrysoee.dk/editline/) but not libreadline, and indeed the configure script fails to detect I have it installed and falls back on the python readline emulation.
Note: I am using Linux, not Mac OS.
Cheers, Romain
nosy: +RomainGeissler1A
Python tracker <report@bugs.python.org> <https://bugs.python.org/issue13501>
Author: Romain Geissler (RomainGeissler1A)
Date: 2018-10-28 20:12
Yes in my case the build works, but then I have a resulting python interpreter with limited readline functionality. That's why I have some interest in having this patch rebased and included into trunk.
Author: Mark Harfouche (Mark Harfouche)
Date: 2018-10-28 23:57
Right, but when do you actually use the bare python interpreter? I'm always on ipython or something that uses that as the backend.
On Sun, Oct 28, 2018 at 4:12 PM Romain Geissler <report@bugs.python.org> wrote:
Romain Geissler <romain.geissler@amadeus.com> added the comment:
Yes in my case the build works, but then I have a resulting python interpreter with limited readline functionality. That's why I have some interest in having this patch rebased and included into trunk.
Python tracker <report@bugs.python.org> <https://bugs.python.org/issue13501>
Author: Enji Cooper (ngie) *
Date: 2019-02-26 16:56
I'll try to rebase Martin's changes, as they don't apply to the master branch on GitHub today.
Author: Robert Marshall (Robert Marshall) *
Date: 2019-02-27 21:04
I have rebased this to recent master hash 53b9e1a1c1 and submitted a pull request.
It built in my CI environment, so seems to be OK as far as compiling.
I'll run the test suite against it later this evening when I have time to stand up a quick build enviroment with the full dependency list.
Author: Robert Marshall (Robert Marshall) *
Date: 2019-04-01 19:08
The pull request is now ready for a review.
Apologies for the delay - took me a bit longer to circle back on this than I would have liked.
Author: Gregory P. Smith (gregory.p.smith) *
Date: 2019-06-07 18:16
In PR it you refer to -ledit and to editline. libedit and libeditline appear to be two different libraries.
I was under the impression that libedit is the desirable one (it's what *BSD uses - coming originally from NetBSD if I understand correctly - which is how it wound up being macOS's choice). I could also be wrong.
Is there a relationship between libedit and libeditline? They both look like useful non-GPL readline alternatives.
We're building CPython internally at Google with libedit instead of readline of late and our initial patch to do this looks similar (though simpler) than your PR's readline.c changes. The #ifdef APPLE checks are replaced with a #ifdef HAVE_LIBEDIT check which we set in our pyconfig.h.
I'm interested in seeing this stuff get in one way or another, there's no reason we should tie people to GNU readline. :)
Author: egaudry (egaudry) *
Date: 2019-08-13 15:48
Hi, I think that there are some issues in the last patch (0001-Build-or-disable-readline-module-with-Editline.patch) made available.
For instance, the using_libedit_emulation variable should be protected by DETECT_EDITLINE ifdef (and not SUPPORT_EDITLINE). And the setup.py file mentioned a 'readline_lib' string instead of the readline_lib variable.
I am interested by being able to build python without readline but libedit instead too.
Author: Enji Cooper (ngie) *
Date: 2020-10-23 04:40
Is there something I can do to help move this change along? 8 years is a long time for an issue to be open :(.
Author: Gregory P. Smith (gregory.p.smith) *
Date: 2020-11-01 07:59
at a glance, it looks like the PR needs updating.
Author: Roland Hieber (rhi) *
Date: 2020-12-14 00:00
What's the status of this patch? Is it still needed after 68669ef7883e, which went into v3.8.1?
Author: Roland Hieber (rhi) *
Date: 2020-12-14 00:02
(That was meant to be 68669ef7883 for the autolinking feature)
Author: Roland Hieber (rhi) *
Date: 2021-01-10 21:30
Okay, I've had a look at this, and it seems that all the heavy lifting is already included in 68669ef7883 (via #38634, PR 16986). The selection of libedit vs. readline still needs to be ported. I've prepared (and tested) PR 24189 for the rebase of this patch to the current master.
Author: Gregory P. Smith (gregory.p.smith) *
Date: 2021-02-09 01:06
New changeset e1f77695132e814728cda982f11342a2e3c7272c by Roland Hieber in branch 'master': bpo-13501: allow choosing between readline and libedit (GH-24189) https://github.com/python/cpython/commit/e1f77695132e814728cda982f11342a2e3c7272c
Author: Gregory P. Smith (gregory.p.smith) *
Date: 2021-02-09 01:08
I'm closing this as I believe everything we need done is done at this point. Open new issues if there are remaining libedit vs libreadline things to take care of.
Thanks everyone!
Author: Kubilay Kocak (koobs) 
Date: 2021-02-09 01:12
Would we like the freebsd worker CI configurations to be modified to use ports/package versions of libedit (or readline, or whatever) to ensure converage?
Right now the defaults are used (whatever edit library that entails)
Author: Gregory P. Smith (gregory.p.smith) *
Date: 2021-02-09 01:44
That seems like a good idea to prevent regressions if anyone knows how to do that.
For python.org/dev/'s buildbot fleet, the configuration of what configure and make flags happens via https://github.com/python/buildmaster-config/tree/master/master/custom - note the 'def setup' overriding that needs to be done in factories.py so that older branches {'3.7', '3.8', '3.9'} are not configured using the flag.
Coordinate with the owner of a particular buildbot to make sure the system has the requisite installed libraries and headers.
I manually tested a Linux build against libedit, it builds and works. But test_readline has failures. tracking those in https://bugs.python.org/issue43172
Author: Kubilay Kocak (koobs)
Date: 2021-02-09 06:37
@Gregory I'm the worker owner and can coordinate buildbot worker changes with Zach or Victor
FreeBSD has:
editline (in FreeBSD base, installed in /usr/ ) libedit in ports/packages (installs in /usr/local) readline in ports/packages (installs in /usr/local)
Which combination would get us the best coverage?
Also, if both libedit and readline are installed from ports, does configure have a preference/default or will it barf?
Primary motivation behind the questions is that the Python build system is notoriously flaky with respect to third party libs in various locations (we've had issues with gettext, ssl, expat, among a few others)
I'm happy to install whatever you'd like on the workers at your convenience to the tests can be run as a custom build (if they haven't been already)
History
Date
User
Action
Args
2022-04-11 14:57:24
admin
set
github: 57710
2021-04-15 06:36:36
tnir
set
nosy: + tnir
pull_requests: + <pull%5Frequest24151>
2021-02-09 06:37:28
koobs
set
messages: +
2021-02-09 01:44:00
gregory.p.smith
set
messages: +
2021-02-09 01:12:14
koobs
set
messages: +
2021-02-09 01:08:37
gregory.p.smith
set
status: open -> closed
resolution: fixed
messages: +
stage: patch review -> commit review
2021-02-09 01:06:01
gregory.p.smith
set
messages: +
2021-01-10 21:30:03
rhi
set
messages: +
2021-01-10 20:58:40
rhi
set
pull_requests: + <pull%5Frequest23016>
2020-12-14 00:02:10
rhi
set
messages: +
2020-12-14 00:00:35
rhi
set
nosy: + rhi
messages: +
2020-11-01 07:59:15
gregory.p.smith
set
messages: +
2020-10-23 04:40:35
ngie
set
messages: +
2020-10-20 00:33:39
gregory.p.smith
set
versions: + Python 3.10, - Python 3.8
2019-12-09 03:17:20
gregory.p.smith
set
versions: + Python 3.9, - Python 2.7, Python 3.4, Python 3.5, Python 3.6, Python 3.7
2019-10-29 19:40:16
serge-sans-paille
set
pull_requests: + <pull%5Frequest16513>
2019-08-13 15:48:45
egaudry
set
nosy: + egaudry
messages: +
2019-06-07 18:16:49
gregory.p.smith
set
assignee: gregory.p.smith
messages: +
2019-06-07 18:08:49
gregory.p.smith
set
nosy: + gregory.p.smith
2019-04-01 19:08:39
Robert Marshall
set
messages: +
2019-02-27 21:04:26
Robert Marshall
set
files: + 0001-Build-or-disable-readline-module-with-Editline.patch
nosy: + Robert Marshall
messages: +
pull_requests: + <pull%5Frequest12095>
2019-02-26 16:56:50
ngie
set
versions: + Python 2.7, Python 3.4, Python 3.5, Python 3.7, Python 3.8
2019-02-26 16:56:16
ngie
set
messages: +
2018-10-29 11:48:54
vstinner
set
nosy: - vstinner
2018-10-28 23:57:18
Mark Harfouche
set
messages: +
2018-10-28 20:12:29
RomainGeissler1A
set
messages: +
2018-10-28 18:42:48
Mark Harfouche
set
messages: +
2018-10-28 18:40:46
RomainGeissler1A
set
nosy: + RomainGeissler1A
messages: +
2018-08-19 16:29:44
Mark Harfouche
set
nosy: + Mark Harfouche
messages: +
2017-03-23 11:24:13
martin.panter
link
2016-06-02 06:31:11
martin.panter
set
files: + editline.v4.patch
messages: +
2016-03-22 05:57:57
martin.panter
set
files: + editline.v3.patch
messages: +
2016-03-20 07:40:49
martin.panter
set
messages: +
2016-02-12 23:34:11
martin.panter
set
messages: +
2016-02-11 02:17:37
ned.deily
set
messages: +
2016-02-11 02:12:47
ned.deily
set
assignee: ned.deily -> (no value)
messages: +
2016-02-04 02:56:50
berker.peksag
set
nosy: + berker.peksag
2015-06-19 11:59:48
donmez
set
nosy: + donmez
2015-06-16 13:23:08
emaste
set
messages: +
2015-06-15 23:13:02
martin.panter
set
messages: +
2015-06-15 17:26:45
koobs
set
nosy: + koobs
2015-06-15 17:23:10
emaste
set
messages: +
2015-06-15 17:14:11
emaste
set
messages: +
2015-06-06 02:24:45
ned.deily
set
messages: +
2015-06-06 02:07:53
emaste
set
messages: +
2015-06-06 02:02:12
emaste
set
messages: +
2015-06-06 01:58:45
ned.deily
set
nosy: + emaste
messages: +
stage: needs patch -> patch review
2015-06-06 01:57:22
ned.deily
link
2015-05-24 09:47:31
martin.panter
link
2015-05-24 09:46:19
martin.panter
set
files: + editline.v2.patch
components: + Extension Modules, - Library (Lib)
versions: + Python 3.6, - Python 3.3
nosy: + martin.panter
messages: +
stage: patch review -> needs patch
2011-12-03 18:05:08
ned.deily
set
messages: +
2011-12-03 14:54:46
eric.araujo
set
messages: +
2011-12-03 14:53:32
eric.araujo
set
nosy: + eric.araujo
messages: +
2011-11-29 18:48:15
vstinner
set
nosy: + vstinner
2011-11-29 18:35:59
ned.deily
set
assignee: ned.deily
2011-11-29 15:21:11
pitrou
set
nosy: + ronaldoussoren, ned.deily
stage: patch review
components: + Build
versions: + Python 3.3, - Python 2.7, Python 3.4
2011-11-29 15:19:01
ngie
create