[Patches] [Patch #103252] Startup optimize: read *.pyc as string, not with getc() (original) (raw)

noreply@sourceforge.net noreply@sourceforge.net
Wed, 17 Jan 2001 21:39:24 -0800

Patch #103252 has been updated.

Project: python Category: core (C code) Status: Closed Submitted by: pj99 Assigned to : tim_one Summary: Startup optimize: read *.pyc as string, not with getc()

Follow-Ups:

Date: 2001-Jan-17 21:39 By: tim_one

Comment: file.readlines() already uses a stack buffer at least 8Kb, so I wasn't worried much about that. But worried enough to make sure this got into the alpha release. We get away with it or we don't; not a big deal either way.

Date: 2001-Jan-17 21:23 By: pj99

Comment: You're welcome, Tim. Thanks for finishing it off!

I was about to post saying that I was skeptical of the 16 Kb stack array, but since that's done (and since it was Guido himself suggesting it, and you agreeing to it), I won't go into that. Lets just say that I spent a past life working in a 4 Kb fixed length kernel stack.

Good work.

Date: 2001-Jan-17 20:41 By: tim_one

Comment: Thanks for pushing this, Paul! Checked in a variant as discussed here, marshal.c ver 1.59 and ACKS ver 1.71. The stat/fstat fiddling was done in a previous checkin.

Date: 2001-Jan-17 13:06 By: tim_one

Comment: Brrrr. HAVE_STAT and HAVE_FSTAT aren't defined in a usable way -- they're defined by (and local to) the files in question, if symbols like DONT_HAVE_STAT are not defined. So I'll rip that stuff out, get rid of the duplication, and move the HAVE_[F]STAT stuff into pyport.h first.

Assuming that doesn't break the build for anyone else, and since we agree on what ought to be done next, I'll then rework the patch accordingly and check it in.

Date: 2001-Jan-17 05:56 By: gvanrossum

Comment: Re the stat syscall: you seem to be using fstat(), and HAVE_FSTAT does exist. (It is used in fileobject.c.) And if you need stat(): HAVE_STAT is also defined (used in import.c).

So I think you're in good shape there.

I agree with Tim's other recommendations!

Date: 2001-Jan-16 22:15 By: tim_one

Comment: I'm afraid this patch needs major work before we could use it. The big thing is that stat isn't std C, and as far as I can tell we don't even define a config symbol saying whether it's available. Even when it is available, on at least one oddball platform it apparently needs oddball declarations (see the STAT/FSTAT/etc macros in posixfile.c). And the Macintosh code isn't in the CVS tree (Jack Jansen owns it), so no telling how it's done there. Standard C doesn't have a good way to determine the size of a file, either. While Python's os.path.getsize() is supported on all platforms, that's done in platform-dependent ways in platform-dependent files, and there's no Python C API for it. So the first step would be to define and implement a PyOS_GetFileSize() C API, akin to the existing PyOS_GetLastModificationTime() (in getmtime.c).

Beyond that, there are small things that need fixing. Like:

Once we get beyond all that , I think a three-tiered strategy would be worth implementing: also declare a 16K stack buffer. If a file is small enough, read into that; else if less than 256K (as Guido asked) malloc a buffer as is done now; else give up. 75% of the .pyc files on my box are under 16K (90% of IDLE's), so that would save the bulk of the malloc/free calls due to this.

Date: 2001-Jan-16 13:36 By: pj99

Comment: No problem, Tim (on the alledged abruptness). In any case, it was useful - in that it helped motivate /f into proposing a fix, me into coding it, and you to consider applying it. All in all a useful result. Most thanks, Tim.

Date: 2001-Jan-15 20:26 By: tim_one

Comment: Indeed, IDLE takes more than a full second to come up on my box, much longer than Notepad. Now that we have a clear goal, I assigned this patch to me . (OK, it's really that I was abrupt with Paul when this came up on c.l.py, and left him hanging there -- I owe him on this one).

Date: 2001-Jan-15 20:18 By: gvanrossum

Comment: Oh, but I do care about the load time of Tkinter. :-) Anything to keep IDLE ahead of Komodo in the race. :-)

Date: 2001-Jan-15 20:05 By: tim_one

Comment: I like the idea, but gimmicks like this make porting to tiny platforms harder when pushed "too far".

People concerned about startup time seem mostly to deal with apps that e.g. fire up Python 1000s of times from shell scripts or cgi. They're not going to care about monster modules (like Tkinter and grail). So I think Paul's original 64K is a good compromise.

Date: 2001-Jan-15 19:39 By: gvanrossum

Comment: Not a bad idea.

I'd up the limit to 256K so Tkinter.pyc (comfortably) fits.

For more info, visit:

http://sourceforge.net/patch/?func=detailpatch&patch_id=103252&group_id=5470