Issue 10224: Build 3.x documentation using python3.x (original) (raw)
Created on 2010-10-28 22:16 by belopolsky, last changed 2022-04-11 14:57 by admin. This issue is now closed.
Messages (31)
Author: Alexander Belopolsky (belopolsky) * 
Date: 2010-10-28 22:16
Opening a ticket to track the progress of getting python 3.x build its own documentation.
So far I have installed Sphinx from http://bitbucket.org/birkenfeld/sphinx, but I get the following error:
$ sphinx-build -b html -d Doc/build3/doctrees -D latex_paper_size= Doc Doc/build3/html
There is a syntax error in your configuration file: invalid syntax (patchlevel.py, line 71) Did you change the syntax from 2.x to 3.x?
Author: Alexander Belopolsky (belopolsky) *
Date: 2010-10-28 22:39
I had some success after running 2to3 on Doc/conf.py and Doc/tools. The build succeeded, but I have not compared the output yet. Running doctest, however, still reported lots of errors and hang in turtle.rst.
$ sphinx-build -b doctest -d Doc/build3/doctrees -D latex_paper_size= Doc Doc/build3/html
Author: Alexander Belopolsky (belopolsky) *
Date: 2010-10-29 04:07
Incremental build was not that successful. After editing Doc/faq/programming.rst, I get the following in the error log:
Sphinx version: 1.1pre
Python version: 3.2a3+.0
Docutils version: 0.7 release
Jinja2 version: 2.5.5
Traceback (most recent call last):
File "/Users/sasha/Apps/lib/python3.2/site-packages/Sphinx-1.1predev_20101028-py3.2.egg/sphinx/cmdline.py", line 173, in main
app.build(force_all, filenames)
File "/Users/sasha/Apps/lib/python3.2/site-packages/Sphinx-1.1predev_20101028-py3.2.egg/sphinx/application.py", line 203, in build
self.builder.build_update()
File "/Users/sasha/Apps/lib/python3.2/site-packages/Sphinx-1.1predev_20101028-py3.2.egg/sphinx/builders/init.py", line 193, in build_update
'out of date' % len(to_build))
File "/Users/sasha/Apps/lib/python3.2/site-packages/Sphinx-1.1predev_20101028-py3.2.egg/sphinx/builders/init.py", line 249, in build
self.write(docnames, list(updated_docnames), method)
File "/Users/sasha/Apps/lib/python3.2/site-packages/Sphinx-1.1predev_20101028-py3.2.egg/sphinx/builders/init.py", line 279, in write
self.prepare_writing(docnames)
File "/Users/sasha/Apps/lib/python3.2/site-packages/Sphinx-1.1predev_20101028-py3.2.egg/sphinx/builders/html.py", line 231, in prepare_writing
self.load_indexer(docnames)
File "/Users/sasha/Apps/lib/python3.2/site-packages/Sphinx-1.1predev_20101028-py3.2.egg/sphinx/builders/html.py", line 619, in load_indexer
self.indexer.load(f, self.indexer_format)
File "/Users/sasha/Apps/lib/python3.2/site-packages/Sphinx-1.1predev_20101028-py3.2.egg/sphinx/search.py", line 131, in load
frozen = format.load(stream)
File "/Users/sasha/Apps/lib/python3.2/site-packages/Sphinx-1.1predev_20101028-py3.2.egg/sphinx/search.py", line 64, in load
return self.loads(f.read())
File "/Users/sasha/Apps/lib/python3.2/site-packages/Sphinx-1.1predev_20101028-py3.2.egg/sphinx/search.py", line 55, in loads
if not data or not s.startswith(self.PREFIX) or not
TypeError: expected an object with the buffer interface
Author: Georg Brandl (georg.brandl) *
Date: 2010-10-29 05:19
I fixed that bug in Sphinx rev 49747f5b0c70 (which I will push as soon as bitbucket is up again); incremental build now works for me.
Author: Georg Brandl (georg.brandl) *
Date: 2010-10-29 05:31
In r85910, I ported the Python-specific modules in tools/ so that they don't need to be 2to3-converted.
Now you can basically easy_install Sphinx on 3.1 and run its sphinx-build without touching the Doc/ tree.
Author: Martin v. Löwis (loewis) *
Date: 2010-10-29 06:40
Did you know break building the 3.x documentation with Python 2? If so, please revert that change.
Author: Georg Brandl (georg.brandl) *
Date: 2010-10-29 06:55
Nope, these files run just as fine in Python 2.
Author: Georg Brandl (georg.brandl) *
Date: 2010-10-29 06:56
(The usual build process via Makefile still uses Python 2, and that won't change for 3.2.)
Author: Alexander Belopolsky (belopolsky) *
Date: 2010-10-29 15:50
The usual build process via Makefile still uses Python 2, and that won't change for 3.2.
Would you consider changing "doctest" make target to check out sphinx trunk (or 3.x compatible release) and run it with the py3k python?
This should not affect regular doc builds and running doctest under anything other than the current version of python makes very little sense.
Note that I am still targeting documentation fixes for 3.2. See issue 10225.
Author: Alexander Belopolsky (belopolsky) *
Date: 2010-11-08 16:12
As discussed on #python-dev, building 3.x documentation using python3.x will require Sphinx 1.1 which will not be released in time for 3.2.
I am adding #10225 as a dependency because I feel that fixing actual errors in ReST doctests is more important than upgrading documentation tool chain. Hopefully this will bring more eyes to that issue.
This said, I believe it is important for python-dev to "eat their own dogfood" and make 3.2 self-hosting. In other words, if we want to convince users that 3.x is ready, we should not require 2.x in the build process.
Fortunately, Doc/tools is not shipped with python releases, so the transition can be made at any time regardless of python release schedule.
Author: Georg Brandl (georg.brandl) *
Date: 2010-11-08 16:20
I agree that 3.2 should eventually be independent of a 2.x Python. Since the "port to 3.x" part of Sphinx 1.1 is basically done, I might just do an early alpha release and use that for Doc/tools before 3.2 final.
Author: Sye van der Veen (syeberman) *
Date: 2011-05-23 13:15
I ran smack into this while setting up my Doc build for the first time. In trying to fix my build problems, it took me about an hour to find that this was a known issue.
I've attached a patch to warn others of the issue, in the documentation and the build itself. Also, I've updated the version of Sphinx in building.rst, and made a note reminding others to do the same. I've only tested this patch on Windows 7.
Author: Éric Araujo (eric.araujo) *
Date: 2011-05-23 13:57
I don’t see how the comment “As such, you will need to specify PYTHON explicitly” logically follows from the fact that Python 2.x is required. IMO, building the docs require Python 2.x, so you’ll need a “python” executable on your PATH that is 2.x.
Author: Sye van der Veen (syeberman) *
Date: 2011-05-25 00:24
I had trouble wording that sentence. Under Unix, Makefile is used, in which you specify "PYTHON=
building.rst doesn't mention any of this, though; README.txt does. I wanted to keep the patch small, as I suspect this issue will be properly fixed soon, but I could create a new patch that better explains how to force that 2.x be used.
Author: Éric Araujo (eric.araujo) *
Date: 2011-06-09 14:33
In either case, there's a high likelihood the PYTHON default will resolve to 3.3a0 for any Python developer.
Ah, I understand the reason for the phrasing in your patch: UNIX systems typically install 3.x as python3, not python, so the high likelihood is only for Windows and a few UNIX systems like Arch.
Author: Martin v. Löwis (loewis) *
Date: 2011-06-14 06:26
Actually, on Windows, PYTHON is typically not set at all. So the likelihood of it being set to Python 3 is very low, unless you are trying to build Python documentation from time to time.
Sye: I fail to see the point of your patch. sphinx-build.py already issues an error message - why do you need more error messages?
Author: Sye van der Veen (syeberman) *
Date: 2011-07-13 14:14
When I built the documentation on Win7, it failed with a SyntaxError, and it took some digging to find the reason why. I was hoping that issuing this warning would save others the trouble.
I agree: on Windows, PYTHON is rarely set. However, look in make.bat: you'll see that when PYTHON is not already set, it's set to ..\pcbuild\python, which is Python 3.
Author: Brett Cannon (brett.cannon) *
Date: 2012-11-17 16:21
What exactly is blocking this? From looking on PyPI it seems Jinja2, Pygments, and Sphinx have all been ported to Python 3 at this point. Is it a matter of porting any custom code in Doc/ to Python 3 and then updating the Makefile to grab Python 3 versions of the required libraries?
Author: Terry J. Reedy (terry.reedy) *
Date: 2013-03-07 21:54
Doc/make.bat assumes that this issue has already been resolved: if "%PYTHON%" EQU "" set PYTHON=..\pcbuild\python Even with '_d' appended, as it should be, that does not work for 3.x. I think that line should be @rem'ed out until it does work. Does anyone mind if I do so?
I already suggested on #15605 that a workaround be given until it is no longer needed.
Author: Zachary Ware (zach.ware) *
Date: 2013-03-09 19:06
Doc/make.bat assumes that this issue has already been resolved: if "%PYTHON%" EQU "" set PYTHON=..\pcbuild\python Even with '_d' appended, as it should be, that does not work for 3.x. I think that line should be @rem'ed out until it does work. Does anyone mind if I do so?
I have offered an alternative fix for that issue as a part of .
Author: Cherniavsky Beni (cben) *
Date: 2013-03-21 03:17
After a lot of experimentation, this worked for me to build AND doctest using the local python:
cd Doc/ ../python -m venv toolsenv curl http://python-distribute.org/distribute_setup.py | toolsenv/bin/python toolsenv/bin/easy_install pip toolsenv/bin/pip install Sphinx
EDIT conf.py: add 'toolsenv' to exclude_trees.
toolsenv/bin/sphinx-build -b html -d build/doctrees -D latex_paper_size= . build/html toolsenv/bin/sphinx-build -b doctest -d build/doctrees . build/doctest
The tool versions I got:
$ toolsenv/bin/pip list distribute (0.6.35) docutils (0.10) Jinja2 (2.6) Pygments (1.6) Sphinx (1.1.3)
HTML: http://dl.dropbox.com/u/132894668/python3-Doc-build/html/index.html The HTML has no significant differences vs the output of "make html". It's mostly sphinx html tweaks. The only user-visible diff I noticed is s/v3.4.0a0/3.4.0a0/.
The doctests are pretty horrible - 559 failed out of 1954! (attached) But it's better than 900+ failed with python 2 ("make doctest") and 700+ with python 3.2 that I originally tried via "apt-get install python3-sphinx"...
So let's update the tools ASAP, and start using PYTHON=../python. The doctests won't get any love unless "make doctest" does the right thing out of the box.
Should I write a Makefile patch using the venv/install from pypi approach? Or could somebody upload new versions to http://svn.python.org/projects/external/? (Lack of access there was the reason I went the venv way. IMHO the ease of future upgrades outweighs the concern of not depending on external sites, but I don't feel strongly about it.)
Author: Ezio Melotti (ezio.melotti) *
Date: 2013-03-21 14:36
Should I write a Makefile patch using the venv/install from pypi approach? Or could somebody upload new versions to http://svn.python.org/projects/external/?
I looked at this issue over the past few days and discussed briefly with Georg about it. The 3 main issues are:
- Sphinx and its dependency have a 2to3-able source that needs to be converted to Python 3 before being used. Should this be done every time or should we provide the already converted source somewhere?
- If we go for the latter, should it be on externals/ (which still uses SVN) or on a new hg repo?
- since we don't actually need all the history, should we still just clone these repos or is it better to download the source only?
IMHO for 1) it would be better to provide Python3-ready code. For 2) it would also be nice to get rid of the SVN dependency for the docs so if we are going to use a repo it might be better to make a new one on HG. For 3), I'm not sure it makes sense to use repos at all if we don't want the history. If we don't, we would have to put the files somewhere and write some code to download and extract the zip/tarball. OTOH, if we do, we will be able to use hg to clone and update the repo easily, and, if we still decide not to download the history, we can just use the zip/tarballs already provided by hg.python.org.
Author: Georg Brandl (georg.brandl) *
Date: 2013-03-21 15:13
A problem with using automatic tarballs from hg.python.org is that they take server load to generate, every time someone builds the docs from a new checkout.
Author: Ezio Melotti (ezio.melotti) *
Date: 2013-03-21 16:30
Apparently they are cached, and unless the cache expires often, the server doesn't need to regenerate the tarball too often. However I don't know the details of the caching, and if they can be tweaked.
Author: Georg Brandl (georg.brandl) *
Date: 2013-03-21 17:29
Ok, that sounds good then.
BTW, for 3.3+ would it make sense instead to create a virtualenv and directly install the required versions from pypi?
Author: Brett Cannon (brett.cannon) *
Date: 2013-03-21 17:38
I guess if we are willing to write a mini-pip to download the latest release directly from PyPI we could. Since there are no C extensions we don't have to care about the lack of header files in Include if we are going to build from the repo build directly. Or are you more thinking that we just require 3.3 externally to build and not rely on the repo build unless absolutely necessary? Just wondering about when the venv environment copies instead of symlinks the binary.
Author: Cherniavsky Beni (cben) *
Date: 2013-05-31 09:41
I was only thinking of 3.4, which will have venv and a pip bootstrapper. Is changing the doc build / doctest in scope for minor releases of 3.3 (or even earlier)?
The commands I listed (using setup_distribute.py) also work with 3.3. (But they're unsecure -- https://bitbucket.org/tarek/distribute/issue/374/) 3.2 is harder as it doesn't even have builtin venv.
Author: Ned Deily (ned.deily) *
Date: 2014-04-14 18:34
For 3.4.1 and 3.5, the Doc/Makefile no longer explicitly vendors sphinx and its dependencies (via svn), rather it assumes that an externally-supplied sphinx-build is available, for example, via a venv and pip. And the current versions of sphinx and its dependencies are all supported on current Py3's. So can this issue be closed, Georg?
Author: Georg Brandl (georg.brandl) *
Date: 2014-04-14 19:24
Yep, thanks for the reminder.
Author: Éric Araujo (eric.araujo) *
Date: 2014-04-14 19:36
Do we need to change Doc/Makefile to have “PYTHON = python3” or even “PYTHON = ./python” (so doctests run with the development version)?
Author: Ezio Melotti (ezio.melotti) *
Date: 2014-07-02 03:49
Éric, I think you should open a separate issue for this.
History
Date
User
Action
Args
2022-04-11 14:57:08
admin
set
github: 54433
2014-07-02 03:49:21
ezio.melotti
set
messages: +
2014-04-14 19:36:13
eric.araujo
set
messages: +
2014-04-14 19:24:45
georg.brandl
set
status: open -> closed
resolution: out of date
messages: +
2014-04-14 18:34:37
ned.deily
set
versions: + Python 3.5
nosy: + ned.deily
messages: +
assignee: docs@python ->
2013-10-13 19:11:21
ezio.melotti
link
2013-05-31 09:41:45
cben
set
messages: +
2013-03-21 17:38:24
brett.cannon
set
messages: +
2013-03-21 17:29:55
georg.brandl
set
messages: +
2013-03-21 16:48:04
r.david.murray
set
nosy: + r.david.murray
2013-03-21 16:30:49
ezio.melotti
set
messages: +
2013-03-21 15:13:23
georg.brandl
set
messages: +
2013-03-21 14:36:01
ezio.melotti
set
messages: +
2013-03-21 03:17:12
cben
set
files: + output.txt
nosy: + cben
messages: +
2013-03-15 01:10:23
terry.reedy
link
2013-03-09 19:06:02
zach.ware
set
nosy: + zach.ware
messages: +
2013-03-07 22:39:56
tshepang
set
nosy: + tshepang
2013-03-07 21:54:09
terry.reedy
set
nosy: + terry.reedy
messages: +
2012-11-17 17:42:38
ezio.melotti
set
dependencies: + upgrade to sphinx 1.1
2012-11-17 16:21:11
brett.cannon
set
dependencies: - Fix doctest runable examples in python manual
messages: +
2012-09-12 21:47:54
chris.jerdonek
set
nosy: + chris.jerdonek
2012-09-12 21:36:20
brett.cannon
set
versions: + Python 3.4, - Python 3.3
2012-09-08 16:27:04
brett.cannon
set
nosy: + brett.cannon
2012-08-13 13:54:55
asvetlov
set
nosy: + asvetlov
2011-07-13 16:06:26
eric.snow
set
nosy: + eric.snow
2011-07-13 14:14:49
syeberman
set
messages: +
2011-06-14 06:26:11
loewis
set
messages: +
2011-06-09 14:33:27
eric.araujo
set
messages: +
2011-05-28 10:05:53
ezio.melotti
set
nosy: + ezio.melotti
2011-05-25 00:24:32
syeberman
set
messages: +
2011-05-23 13:57:44
eric.araujo
set
messages: +
2011-05-23 13:15:46
syeberman
set
files: + Issue10224Warnings.patch
nosy: + syeberman
messages: +
keywords: + patch
2011-01-20 23:34:37
brett.cannon
link
2010-11-08 16:20:13
georg.brandl
set
messages: +
2010-11-08 16:12:44
belopolsky
set
type: enhancement
dependencies: + Fix doctest runable examples in python manual
messages: +
stage: needs patch
2010-10-30 09:37:21
zbysz
set
nosy: + zbysz
2010-10-29 16:50:14
daniel.urban
set
nosy: + daniel.urban
2010-10-29 15:50:45
belopolsky
set
messages: +
2010-10-29 06:56:25
georg.brandl
set
messages: +
2010-10-29 06:55:35
georg.brandl
set
messages: +
2010-10-29 06:40:29
loewis
set
nosy: + loewis
messages: +
2010-10-29 05:31:25
georg.brandl
set
messages: +
2010-10-29 05:20:00
georg.brandl
set
messages: +
versions: + Python 3.3, - Python 3.2
2010-10-29 04:07:12
belopolsky
set
messages: +
2010-10-29 02:19:06
belopolsky
link
2010-10-28 22:39:05
belopolsky
set
messages: +
2010-10-28 22:29:53
belopolsky
set
assignee: docs@python
nosy: + docs@python
components: + Build, Documentation
versions: + Python 3.2
2010-10-28 22:29:14
belopolsky
set
nosy: + georg.brandl, eric.araujo
2010-10-28 22:16:37
belopolsky
create