The rise and fall of the Gopher protocol (original) (raw)

[raw shortcodes=1]

.byline {display: none;} .service-links {display: none;} .page-title {display: none;}

.mp { font-family: Georgia, serif; }

.mp .text-block p { line-height: 2em; }

.mp .text-block p.first { font-size: 1.4em; }

.mp .component-label { font-family: georgia; }

.mp .image { max-width: 960px; margin: auto; margin-top: 2em; margin-bottom:2em; }

.mp .image iframe { margin: auto; }

.mp .credit { margin: 0; font-size: .8em; }

.mp .float-left { margin: 0 3em 1em 0; }

.mp .float-right { margin: 0 0 1em 3em; }

.mp .four-up { display: block; max-width: 80%; margin: auto; margin-top: 2em; margin-bottom:2em; } .mp .four-up img { float: left; width: 45%; margin-bottom: 1em; }

.mp .four-up :nth-child(even) { margin-right: 1em; }

.mp .four-up div { clear: both; }

@media (max-width: 600px) { .mp .four-up { width:100%; }

.mp .four-up img { width: 95%; margin: 0 auto 1em; } }

.mp .video-container { position: relative; padding-bottom: 56.25%; padding-top: 30px; height: 0; overflow: hidden; }

.mp .video-container iframe, .mp .video-container object, .mp .video-container embed { position: absolute; top: 0; left: 0; width: 100%; height: 100%; }

.mp .section-break { text-align: center; margin: 4em 0 4em 0; }

It was mid-March 1992, and Mark McCahill had never been to San Diego before. Back home in Minneapolis, the skies had been dumping snow for six months, and would keep at it for several more weeks. McCahill checked into the Hyatt Islandia, an 18-story high-rise hotel overlooking Mission Bay. “There were palm trees,” he recalls. “Boy, was it nice.”

McCahill was then in his mid-30s and managing the Microcomputer Center at the University of Minnesota–Twin Cities, which facilitated the emerging use of personal computers on campus. He and Farhad Anklesaria, a programmer in the center, had been invited to address the 23rd Internet Engineering Task Force (IETF), an elite convocation of academics and government officials from around the world who were literally deciding how the internet should work.

Mark P. McCahill Papers, Charles Babbage Institute, University of Minnesota

“The gods of the internet,” McCahill says, though in other circles they would have gone unnoticed. In his memoirs, another internet pioneer, Tim Berners-Lee, describes these gatherings as “people in T-shirts and jeans, and at times no footwear. They would meet in different small rooms and talk excitedly.”

The internet then, as now, was a vast array of information stored in random computers around the world, only there was no easy or consistent access. It was difficult even to discover what was out there — there were no good search engines. The most popular protocol, or method of retrieving information from another computer, was FTP (file transfer protocol), the primitive, labor-intensive equivalent of knocking on someone’s door and asking if you could carry away his piano.

The IETF had been convening since 1986 to iron out these issues, which had prevented the internet from becoming the “Intergalactic Network” its originators had foreseen, instead remaining the limited domain of physicists and the military. But this meeting felt different. For the first time, the internet seemed on the verge of going public.

On March 18, in a conference room of the hotel, Berners-Lee presented one possible breakthrough: the World Wide Web. It was evening. Many of the 530 conference attendees had already gone to the bar or to dinner. To the curious who stayed behind, Berners-Lee explained that the Web could be used to connect all the information on the internet through hyperlinks. You could click on a word or a phrase in a document and immediately retrieve a related document, click again on a phrase in that document, and so on. It acted like a web laid over the internet, so you could spider from one source of information to another on nearly invisible threads.

Two other programs with the potential to expand access to the internet — WAIS and Prospero — were discussed in the same session. In the reports of people who saw the presentation, the Web did not come across as the best of them, or even as particularly promising.

The next day, in the light of the afternoon, McCahill and Anklesaria presented the Internet Gopher. It was simple enough to explain: With minimal computer knowledge, you could download an interface — the Gopher — and begin searching the internet, retrieving information linked to it from anywhere in the world. It was like the Web but more straightforward, and it was already working.

In fact, most attendees needed little introduction to Gopher — the software had been out for months. It was the developers they were curious about, the Minnesotans who had created the first popular means of accessing the internet. “People we’d never met were telling us how they were using our stuff and adding things to it,” McCahill says. “We had no idea how big Gopher was going to be until we experienced this firsthand and realized that growth could be exponential for a while.”

In the years that followed, the future seemed obvious. The number of Gopher users expanded at orders of magnitude more than the World Wide Web. Gopher developers held gatherings around the country, called GopherCons, and issued a Gopher T-shirt — worn by MTV veejay Adam Curry when he announced the network’s Gopher site. The White House revealed its Gopher site on Good Morning America. In the race to rule the internet, one observer noted, “Gopher seems to have won out.”

McCahill’s father was an executive for Conoco, the oil company, which moved him around the country about every two years. McCahill was in junior high when the family finally settled in the Twin Cities. He graduated from the U in 1979 with a BA in chemistry, spent a year studying effluent in rivers, and realized he liked the computer analysis (“heavy number smashing”) more than the chemistry itself (“kind of dirty”). So he took a job in the U’s Microcomputer Center, programming some of Apple’s first personal computers.

The Twin Cities were a proto-Silicon Valley then, with a long history of producing some of the world’s most powerful computers at UNIVAC, Control Data Corporation, and Engineering Research Associates, which supported the work of Honeywell, IBM, and other local tech firms. The Minnesota Educational Computing Consortium, or MECC, formed in 1973 to get computers in schools and created software for them — most famously The Oregon Trail. By the early 1980s, when only a fraction of schools in neighboring states had computers, there were about three or four in every public school in Minnesota.

When McCahill began working in the Microcomputer Center, a turf battle was heating up — “a religious war,” McCahill calls it — between “the high priests of computing” who oversaw the U’s venerable mainframes, the enormous machines that once occupied entire rooms, and the growing cadre of personal-computer converts. “Microcomputer guys were as far out of the mainstream as you could get and still be a part of the U’s computer center,” McCahill says. Anklesaria, who had earned a doctorate in genetics before gravitating to computer science, says the Microcomputer Center was “a splinter group” when he joined in late 1987. “The mainframe was still the only thing — the Mac was considered a toy.”

McCahill sided with PCs. “The idea of democratizing access to computing, putting computers in the hands of everyday people — that resonated with me, and that was part of all the early PC stuff,” he says. “If you were interested in PCs, you just absorbed that attitude by osmosis. It was in the air and the water.”

Slides from a 1992 Internet Gopher presentation explaining the protocol:

Mark P. McCahill Papers, Charles Babbage Institute, University of Minnesota

McCahill, who had long hair then that he now pulls into a ponytail, spent his free time wind-surfing on Lake Calhoun, and says the PC revolution “looked like a good wave to ride … it would enable me to do what I’ve tried to do ever since: take technology that is cutting edge and get it to the point that it’s palatable to mom and dad and English majors.”

In the late 1980s, McCahill and Anklesaria developed the first popular internet email system, called POPMail. “It was partly for selfish reasons,” McCahill says. “I wanted more people using email so I didn’t have to walk down the hall to my mailbox to collect my phone messages on little slips of paper.” Instead, secretaries could send an email. For that matter, so could callers.

At the same time, the U was determined to network its computers on the internet in a so-called campus-wide information system, or CWIS, and the schism was delaying development. By early 1991, a committee of more than 20 department heads and computing specialists had been meeting for months, producing a long list of demands — including the use of mainframe computers — but zero code. “They had some complicated shit for doing searches that I didn’t want to do,” McCahill says. “Farhad didn’t want to write it at all.”

“But I had to show something,” Anklesaria says. So he stripped the program down to its simplest parts — a basic protocol for making information in one place available somewhere else. He cobbled it together on a Mac, writing a server (a program enabling a computer to “serve up” requested files) and a client (how most of our computers are programmed, enabled to search for and request those files). “And I said, ‘This stuff kind of works.’ Since we had nothing else, we went for it.”

Mark P. McCahill Papers, Charles Babbage Institute, University of Minnesota

A 1994 University of Minnesota alumni magazine spread featuring the Gopher protocol architects.

McCahill pushed for a full-text search engine — something we now take for granted — and borrowed the gist of one from a computer system called NeXT, which had recently been invented by Steve Jobs. “We had this marriage of Farhad’s super-simple protocol for saying give me a list of items, a menu,” says McCahill, “and my thing of having a way of searching, and we glued those two together.”

It was plain text — no pictures, given that modem speeds were so slow. And it was organized like the one information source most people alive in the early 1990s were familiar with: a library, with similar subjects grouped together. You just pointed your gopher, as the lingo went, to any Gopher site you wanted to explore, and there you were, burrowing through the internet. It was so simple that just about anyone could make it work, even an English major.

“It was one of the rare times when we both looked at each other and said, ‘Holy shit, we’ve got a really good idea here,’” McCahill recalls. Anklesaria called it the Internet Gopher, a triple play on words: the U’s mascot, a critter that digs, and a go-fer — one who fetches. “We figured that if we called it Gopher, the committee couldn’t complain,” McCahill says. “It’s the school mascot!”

Sir Tim Berners-Lee — he was knighted in 2004 — was born in London in 1955, less than a year before McCahill. His parents were both mathematicians. While McCahill’s father was being shipped from Colorado to Oklahoma to Minnesota, Sir Tim’s parents were developing the world’s first commercially available computer.

Berners-Lee grew up with the internet, or at least the concept of it. As a child, he built mock computers out of cardboard boxes. He came home from high school one day to find his father writing a speech on how computers might someday make intuitive connections, linking information the way the brain uses random associations to link thoughts.

Creative Commons/John S. and James L. Knight Foundation

Tim Berners-Lee in a 2008 photo.

He was a teenager when the internet began as the ARPAnet, connecting a handful of computers at Stanford and other universities. It was designed as a defense, a secure means of communication should the Soviets destroy the American telephone system, though its practical purpose was to allow scientists to use the computing power at another facility for massive calculations.

Berners-Lee earned a degree in physics from Oxford — where he built his first computer with parts from an old television, a calculator, an electronics kit, and a car battery — then worked as a software engineer for a few years before taking a job with CERN, the famous particle-physics lab in Geneva, Switzerland. By then, in 1984, the internet had gone global; CERN was the largest internet node in Europe.

Yet using the internet was problematic even within the CERN lab. When you wanted information, Berners-Lee later recalled, “often it was just easier to go ask people while they were having coffee.” He decided he could do better. He had never forgotten his father’s research on the brain, and when he wove the Web, it was based on this holistic, serendipitous, strangely rewarding experience of surfing from one vaguely related idea to the next.

He finished a model of the Web in 1989, but for years it went nowhere. The concept was too abstract and it only worked on NeXT computers. But in early 1991, just as McCahill and Anklesaria were conceiving the Internet Gopher in Minnesota, the first Web servers outside of CERN were switched on.

After outlining the Gopher protocol, McCahill and Anklesaria pulled together four programmers who worked in the Microcomputer Center to write the software. McCahill wanted it in a hurry. Not because he was racing Berners-Lee, but because he wanted it over with. He wanted it before the next CWIS committee meeting, in a month.

The programmers were young guys, mostly in their 20s and, like McCahill, mostly huge Nirvana fans. Paul Lindner, a coding wunderkind from northern Minnesota who was dubbed the Gopher Dude for his evangelism, had long metal-head hair and signed Gopher emails with lyrics like “You have to spit to see the shine” from Babes in Toyland. Early Gopher servers were named Mudhoney, Danzig, and Anthrax. The sole outlier in the microcomputer mosh pit was Bob Alberti, a programmer who named a server Indigo, as in the Indigo Girls.

MinnPost photo by Corey Anderson

Shepherd Labs on the University of Minnesota campus.

The center was in Shepherd Labs, a hulking cement building built like a tank in 1968 on the U’s Minneapolis campus, with concrete floors and no windows. Early on, it was used for NASA materials research. “There were pipes with strange fluids running through them,” Lindner recalls.

The microcomputer team, in addition to developing software for the U, ran a showroom for students and faculty interested in buying a Mac, taught computer training classes, tested software, and served as a help center for people with PC problems — walk-in and call-in. “Everyone would answer phones at least one day a week,” Lindner says, “even if you were programming. That way you were close to the pain you were inflicting on people — if programmers today still took calls, we’d have more user-friendly software.”

Gopher, however, was claiming more and more of their time. “It became infectious,” Lindner says. “How can we build this into the science fiction of our dreams, access to all the information in the world, the library of everything?”

The team, in 36-hour sessions fueled by beer, pizza, and speed metal, finished writing Gopher in about three weeks. They installed the first computer running a Gopher server — a Mac SE/30, a little droid of a computer with an iPad-size monitor built in — in a narrow hallway between their offices and the showroom, in a closet with metal shelves. It became known as the Mother Gopher.

The committee meeting where the team first presented the Gopher protocol was a disaster, “literally the worst meeting I’ve ever seen,” says Alberti. “I still remember a woman in pumps jumping up and down and shouting, ‘You can’t do that!’ ”

Among the team’s offenses: Gopher didn’t use a mainframe computer and its server-client setup empowered anyone with a PC, not a central authority. While it did everything the U required and then some, to the committee it felt like a middle finger. “You’re not supposed to have written this!” Alberti says of the group’s reaction. “This is some lark, never do this again!” The Gopher team was forbidden from further work on the protocol.

After the meeting, McCahill leaned on the director of the computer center, a Chinese-American man named Shih Pau Yen, who had supported Gopher all along. “I said I would quit before I stopped working on the coolest thing we’d ever created,” McCahill says. Yen ran interference, and the Gopher team kept working on it in their own time.

“In this bureaucracy of little fiefdoms, where everyone had a hard time working together, the one thing that could unify us was telling everyone else off,” Lindner says. “That was our rallying cry.”

Finally, in April 1991, still unable to persuade the U to take on Gopher, Lindner released it into the wild. He made the Gopher software available via FTP — the most popular way to share information on the internet at the time — and wrote a brief, quiet announcement on an internet mailing list: Hey, we’ve got this thing, come and get it.

Within months, the team was hearing from Gopher users around the country. “It was the first viral software,” Alberti says. “All these people started calling the U and pestering the president and other administrators, saying, ‘This Gopher thing is great, when are you going to release a new version?’ And the administrators said, ‘What are you talking about?’”

The ban was lifted. Never numbering more than six core members, the team fanned out to conferences to spread the Gopher gospel, while continuing to improve and diversify the protocol. In the process they ended up laying the foundation for much of how we navigate the internet. The first hyperlinks. The first bookmarks. McCahill, thinking of windsurfing, even coined the term “surf the internet.”

A video of veejay Adam Curry wearing an “Internet Gopher World Tour” T-shirt on MTV.

Within a year, there were hundreds of Gopher servers. Berners-Lee, who had publicly introduced the World Wide Web a few months after Gopher’s debut, used Gopher to do it. “People look at the World Wide Web today and think it sprang out of Tim Berners-Lee’s forehead,” Alberti says. “But the fact is, the only way he was able to spread the word about the Web is because the Internet Gopher was there to allow people to download his files, find a discussion group, and talk about it.”

“We had the right product at the right time,” McCahill says. “People were looking to expand the internet beyond physicists’ stuff. Gopher could do that. It was simple to use, it could network lots and lots of computers. It gave people a reason to say, hey, this internet is good.”

Lindner was solicited for side gigs — twice he went to Ecuador to set up Gopher for the country’s fledgling internet. “That’s when I knew we were really onto something,” he says, “when I was helping wire a whole country.”

Al Gore, then a U.S. senator, came to visit. Four GopherCons, held between 1992 and 1995, drew reps from the New York Times, the World Bank, Microsoft, and other global heavyweights. The Gopher T-shirt, black and scribbly, listed the names of places with Gopher servers on the back, in the style of rock tour shirts. It was an apt metaphor, as Gopher team member Daniel Torrey told the Pioneer Press in 1996: “We thought we were rock stars.”

Some team members dreamed of fortune to go with their fame. But the internet was not yet open for business. It had been built on dot-mil and dot-edu, on public funds. Programmers shared source code; if you needed something, someone gave it to you. A dot-com address was considered crass. It was “as though all of TV was PBS,” Lindner says. “No commercials.”

Still, Alberti raised the profit potential of Gopher with Shih Pau Yen. Before coming to work in the Microcomputer Center, Alberti had helped create the first online multi-player role-playing game — called Scepter of Goth, an ancestor of World of Warcraft and the like. “I said we should take this private, we should make a business of this and make some money off it,” Alberti recalls. “He looked at me like I’d just grown another head.”

Mark P. McCahill Papers, Charles Babbage Institute, University of Minnesota

Eventually, though, the U did want some money — for itself. At GopherCon ’93, Yen announced that for-profit Gopher users would need to pay the U a licensing fee: hundreds or thousands of dollars, depending on the size and nature of their business. Many users felt betrayed. In the open-source computing spirit of the day, they had contributed code to Gopher, helping the team keep up with the times. Now they were being asked to pony up.

The reaction deflated the team. As hard as they were expected to work on Gopher, they were never relieved of other duties, including answering the U’s help line. “We never got additional funding, we were going broke,” Alberti says. “We had the whole internet yelling at us, when are you going to update your software, when are you going to put images on Gopher pages, and make this smoother and better? And we’re like: ‘We’re six guys!’”

For a while, the U threatened to get rid of them altogether, a bid to outsource the university’s computer work, provoking one programmer to bug a computer in Morrill Hall, the U’s administrative center, so the team could listen in on discussions.

Asking for a contribution seemed reasonable. When it backfired, the team posted a defensive letter to Gopher users in March 1993: “… There has been a lot of hysteria, misinformation, and rumor floating around. … In a time where we are having budgets slashed, it is impossible to justify continued (increasing) resources being allocated to Gopher development unless some good things result for the University of Minnesota. This is a fact of life. … Before you go off and flame once more, ask yourself if you want to get YOUR particular server going with as little fuss and expense as possible … or if you just want to stir up the soup.”

“That socially killed Gopher,” Alberti says of the licensing fiasco.

Yet it wasn’t the end. In 1993, Gopher was still far more popular than the World Wide Web, and Gopher traffic grew by 997 percent. But the Web was starting to catch up — that year, it grew by 341,634 percent.

At the San Diego internet conference, in 1992, Berners-Lee had pulled McCahill and Anklesaria aside on the last day of meetings and asked if they wanted to collaborate on a Web/Gopher hybrid. Some melding of their different designs — an internet super-system — though it wasn’t clear how that might work.

“Tim is a great guy, but he’s a little odd, a little scattered,” McCahill says. “Talking to him is like a ball of twine experience. The Web you see now, that’s how he thinks.”

McCahill told Berners-Lee that he would need to look at the Web more closely. But it wasn’t much to look at when McCahill went back to Minnesota and examined it. There were no graphics yet. It was still only running on NeXT computers. “I wasn’t feeling it,” McCahill says. I told him, ‘Tim, I don’t think so.’ Of course, I look back and say, ‘I might have been wrong.’ ”

Soon enough, the Web did have pictures and was available on more platforms. In 1993, the first popular Web browser, Mosaic, was introduced for sale, breaking the commercial taboo of the internet and suggesting — to McCahill at least — that tech investors had taken sides. “The fix was in,” he says.

In 1994, modem speeds doubled, and the interminable rendering of images on the Web — once dubbed the World Wide Wait — greatly accelerated. PCs began to be sold with these faster modems built in. To anyone looking for a simple, even crude explanation for the Web’s rise, this is it: the ability to view a reasonable facsimile of a naked woman in the privacy of your own home. “That’s what came to drive a lot of the internet,” Alberti says. “Porn.”

The Internet Gopher, with his text-only menu and gloss-less, institutional mien, couldn’t keep up. He had fallen off the wave, and almost overnight was revealed as a buck-toothed square, ignored by the girls on the beach, his surfboard held together by duct tape. “Obsolesced,” as one observer put it in 1994. A has-been.

“I remember the exact moment I knew I was no longer on the right track,” says Lindner. “It was September 9, 1993. I was invited to give a talk about Gopher at Princeton, and I had my slides all printed up on my little university-budget black-and-white foils. The person presenting before me was talking about the future of the Web, with full-color LCD projection. I said, ‘I think I see where things are going.’ ”

Courtesy of Bob Alberti

Mark McCahill’s entry on the University of Minnesota’s “Wall of Discovery,” which celebrates the accomplishments of students and faculty.

For McCahill, the realization happened on the street. “I saw a URL on the side of a bus,” he says. “That’s when I knew the Web was all about advertising. Gopher was not good for advertising. I knew it would start winding down.”

In the spring of 1994, Web traffic overtook Gopher traffic for the first time. Gopher, within a few months, began to decline. Its reign had lasted three years.

The coup de grâce came from the U itself. In 1995, Dr. John Najarian, a renowned transplant surgeon at the university, was indicted for tax evasion, embezzlement, and fraud. Although Najarian was later cleared of all charges, the National Institutes of Health (NIH) put the U on probational status, threatening tens of millions of dollars in grants and the school’s viability as a top research institution. “It was an all hands on deck situation,” McCahill says.

The issue was essentially simple: bad accounting. Paperwork was literally on paper, and transaction records sat around for weeks before making it into a ledger, if they made it in at all. The Gopher programming team was diverted to creating a more accountable accounting system for the U, what turned out to be the first Web-based transaction program. “That’s what my kick-ass development shop did for a year and a half,” McCahill says, “to show the NIH that we were cleaning things up.” By the time they finished, the Internet Gopher was dead.

In the beginning, when the Mother Gopher was new and there were no other Gopher servers to link to, Gopherspace was empty. The Gopher team, to demonstrate the usefulness of their invention, dumped in a cookbook and searched for eggplant. As late as 1993, the most popular information in Gopherspace included recipes, weather, phone books, and movies of chemical reactions.

“Gopher represented a simpler, more naïve time,” says one of its modern-day fans. Others call it “a purer way” of navigating the internet, of “making structure out of chaos.”

Today, there are about 140 Gopher servers still out there, many of them relatively new. The tech world is not a sentimental place, but it does appreciate simplicity and irony in equal measure — many of these servers were set up on April Fool’s Day.

MinnPost photo by Bill Kelley

Farhad Anklesaria: “You have to be at the right time and place to have your technology take off and become popular.”

Cameron Kaiser, perhaps the Internet Gopher’s greatest advocate these days (he runs a Gopher support site called the Overbite Project), figures he spends about 25 percent of his internet time in Gopherspace. “It’s actually rather nice to have a small ecosystem because no one’s running annoying ads in Gopherspace or trying to track your browsing habits,” he says. “The protocol makes the former hard and the latter almost impossible.”

Courtesy of Paul Lindner

Paul Lindner

As the Web has become synonymous with the internet, and we conduct more of our lives online, we’ve learned to abide the misinformation, solicitations, and scams that thrive in its chaos — though it has changed us and society in ways we are only beginning to understand. If Gopher had won out, who knows how things would be different.

The Gopher team, who might understandably be disappointed, seem genuinely OK with how things turned out. More than OK, actually, as though they expected it. They were first, after all. And in the march of technology, the first are eventually last. “This is the natural order of things,” Anklesaria says. “You have some building blocks — the natural thing to do is build on them. That’s how civilization bootstraps and how we progress.”

If they are surprised by anything, it’s that the Web was the system that surpassed them and has outlasted everything else. There was a protocol that was better than either the Web or Gopher, they say, called Hyper-G, developed in Austria, which never got off the ground. “You have to be at the right time and place to have your technology take off and become popular,” says Anklesaria. “If you’re at the base of a wave, you have a chance to rise with it. But if you’re already on top of the wave and the wave has broken, that’s not good — if you’re too early, you won’t last.”

MinnPost photo by Bill Kelley

Bob Alberti: “The only way [Tim Berners-Lee] was able to spread the word about the Web is because the Internet Gopher was there to allow people to download his files, find a discussion group, and talk about it.”

On July 1, 1997, or maybe in 2009, most likely in the 2000s, someone walked into the closet in Shepherd Labs and unplugged the Mother Gopher. No one can agree on the date. No one from the Gopher team was around.

“While I was at the U,” McCahill says, “it was a point of pride that we’d keep running the server.” But McCahill left in 2007 to become a systems architect at Duke University, developing instructional and research computing technology. Lindner left in 1996, having followed a Gopher side gig to Geneva, Switzerland, where he worked for the United Nations, and is now a software engineer for Google. Anklesaria retired last spring. Of the core Gopher team members, Alberti is the only one still at the U, as an information security architect.

At its peak, the Mother Gopher consisted of 10 Apple IIci computers. But when it was finally euthanized, who knows what shape it was in. There was no ceremony. Nothing was carted off to a museum. Gopherspace simply became emptier, and the world without the Web became harder to imagine.

[/raw]