A turning point for GNU libc [LWN.net] (original) (raw)

We're bad at marketing

We can admit it, marketing is not our strong suit. Our strength is writing the kind of articles that developers, administrators, and free-software supporters depend on to know what is going on in the Linux world. Please subscribe today to help us keep doing that, and so we don’t have to get good at marketing.

The kernel may be the core of a Linux system, but neither users nor applications deal with the kernel directly. Instead, almost all interactions with the kernel are moderated through the C library, which is charged with providing a standards-compliant interface to the kernel's functionality. There are a number of C library implementations available, but, outside of the embedded sphere, most Linux systems use the GNU C library, often just called "glibc." The development project behind glibc has a long and interesting history which took a new turn with the dissolution of its steering committee on March 26.

In its early days, the GNU project was forced to focus on a small number of absolutely crucial projects; that is why the first program released under the GNU umbrella was Emacs. Once the core was in place, though, the developers realized they would need a few other components to build their new system; a C library featured prominently on that list. So, back in 1987, Roland McGrath started development on the GNU C library; by 1988, it was seen as being sufficiently far along that systems could be built on top of it.

By the time the Linux kernel came around in 1991, GNU libc was, like many other GNU components, the obvious choice for those working to build the first distributions. But, while GNU libc was a good starting point, it quickly became clear that (1) it still needed a lot of work, and (2) the GNU project's management style was, in typical fashion, making it hard to get that work done and merged. So, early on in the history of Linux, the GNU C library was forked and a Linux-specific version ("Linux libc") was maintained and shipped with Linux distributions.

The GNU project was rather slow to come around to the idea that Linux was the system that they had been trying to build all these years; they continued to develop their C library on their own, despite the fact that Linux was not using it. Early on most of the work came from Roland, but, slowly, other contributors appeared; the first contribution from Ulrich "Uli" Drepper appears to have been made in September, 1995. Early 1997 saw the first glibc 2.0 release; by then, most commits were made by Ulrich (though the code sometimes came from others).

While Linux libc had been adequately maintained, it had not seen vast amounts of new development work. There came a point where it became clear that, in fact, glibc 2 was a better C library in many ways. Distributors of Linux came to the conclusion that the time had come to switch back to the GNU library and, for all practical purposes, simply abandon the Linux libc project. Those of us who lived through the subsequent transition (as seen in the Red Hat 5.0 and Debian 2.0 releases) still tend to shudder; it was a time when programs would not build and bugs abounded. But things eventually settled down and glibc has been the standard C library for most Linux distributions since.

Glibc has continued to evolve up to the present at varying speeds; during most of this time, the bulk of the work has been done by Ulrich Drepper. Of the nearly 19,000 commits found in the project's git repository (which contains changes back to 1995), over 12,000 were made by Ulrich. Never known for a willingness to defer to others, Ulrich quickly became_the_ decision maker for glibc. The project's steering committee wasformedby the FSF in 2001 as a way of asserting some control over the project, but, to a great extent, that control remained in Ulrich's hands. In 2008, Roland described the situation this way:

drepper, roland have blanket discretion to commit. (This means that Uli gets to go upside my head after I commit something he doesn't like. It doesn't mean you get to lobby me to commit something that Uli has refused.)

Four other developers (Andreas Jaeger, Jakub Jelinek, Richard Henderson, and Andreas Schwab) had the ability to commit "after approval". But that approval, in the end, had to come from Ulrich.

One need not look far to find complaints about how Ulrich managed the glibc project. He was never one to suffer fools, and, evidently, he saw himself surrounded by a lot of fools. Attempts to get changes into glibc often resulted in serious flaming or simply being ignored. Developers often despaired of getting things done and simply gave up trying. It is easy to criticize Ulrich's attitude at times, but one should also not lose track of all the work he got done. This includes steady improvements to glibc, big jumps in functionality (as with the Native POSIX Thread Library work done with Ingo Molnar) and lots of fixes. Ulrich's work has made all of our systems better.

That work notwithstanding, Ulrich is widely seen as, at best, an unpleasant maintainer who has turned the glibc project into an unwelcoming place. One need not challenge that characterization to point out that glibc actually had a number of problems associated with highly cathedral-style projects: a well-ensconced priesthood, a lack of interest in or awareness of the wider world's needs, etc. At some point, it seemingly became clear to a number of developers associated with glibc that it needed to become a more open project with more than a single point of control. A number of changes have been made to prod it in that direction.

The switch to Git for source code management in May, 2009 was one such change. At that time, the repository was made slightly more open so that non-core maintainers could keep branches there. Ulrich opposed that decision, but Roland responded:

Please don't stand in the way of minor matters the rest of the community wants to improve collaboration, on vague grounds of conservatism or a default of mistrust.

The project started creating separate release branches and putting maintainers other than Ulrich in charge of them. The restricted-access "libc-hackers" mailing list fell into disuse and has been closed down. The use of Fedora as a sort of private testing ground has been halted after things broke one too many times. And a strong effort has been made to encourage more developers and to merge more code. The result is a release history that looks like this:

Release	Date	Changes
2.3	2002-10-03	3204
2.4	2006-03-06	5276
2.5	2006-09-29	348
2.6	2007-05-15	357
2.7	2007-10-18	446
2.8	2008-04-12	281
2.9	2008-11-13	293
2.10	2009-05-09	344
2.11	2009-10-30	408
2.12	2010-05-03	389
2.13	2011-01-17	261
2.14	2011-05-31	256
2.15	2011-12-23	582

(The 2.15 date is when the release was tagged; the announcement was, for whatever reason, not sent out for a few months). The 2.15 release suggests that development activity is on the increase after a sustained low point. Indeed, it is at its highest point since semi-regular releases began in 2006. Activity since 2.15 (483 commits, as of this writing) continues to be high by recent historical standards; perhaps more tellingly, only about 25% of those commits came from Ulrich. Given that he accounts for over 60% of the changes from 2.10 to 2.14, that suggests that a significant shift has taken place.

Also significant is the fact that Ulrich left Red Hat in September, 2010; according to his LinkedIn page, he is now "VP, Technology Division at Goldman Sachs." One can easily imagine that his new position might just have requirements that are unrelated to glibc maintenance. So a drop in participation is not entirely surprising; what is significant is that a growing community is more than taking up the slack.

The culmination of these trends came on March 26, when Roland announced that there was no longer any need for a glibc steering committee:

With the recent reinvigoration of volunteer efforts, the community of developers is now able to govern itself. Therefore, the Steering Committee has decided unanimously to dissolve. The direction and policies of the project will now be governed directly by the consensus of the people active in glibc development.

Glibc will remain a project under the GNU umbrella; Roland, along with Joseph Myers and Carlos O'Donell, will be the official GNU maintainers. But that role will give them no special decision-making powers; those powers are meant to be exercised by the project's development community as a whole. David Miller (an active glibc contributor) was quick to celebratethis change:

If you've tried to contribute to glibc in the past and were rudely treated and therefore gave up, I encourage you to come and try again, things are much different now. I promise.

David also hinted that one other possible obstacle to contribution - the requirement to assign copyrights to the Free Software Foundation - is also being worked on, though the form the solution might take is unclear. Other developers are already talking about working to merge at least some of the changes from the EGLIBC fork back into the glibc mainline and reopening old bugs and change requests.

So the GNU C library project has moved into a new phase, one where, for the first time in its long history, it will not be under the control of a single developer. It will be interesting to watch where things go from here. If the project succeeds in building robust community governance and shedding its hostile-to-contributors image, the pace of development could pick up considerably. Given the crucial position occupied by glibc on so many systems, these changes are almost certainly a good thing.