Chapter 8: Power of X

BILL FULTON ENTERED CLAREMONT MCKENNA College to study mathematics with psychology as a second major. Understanding how people think, and how he himself thought, could prove more useful than a background in advanced mathematical concepts.

"I was always good at math but terrible at humans. In the end, you should always study what you don't know," Fulton says.

Fulton was fascinated by how little he understood of human behavior despite having spent nearly two decades of his life being one. He also suffered a bit of existential crisis. He angled his navigation through psychology toward the field of behavioral economics--then known as judgment decision-making--to better comprehend his thought processes. There were times, he knew, when his brain told him to take shortcuts that crossed societal lines.

"Stereotypes, for example," he says. "A shortcut might say, 'Oh, that's a man. Men do X.' How often is that incorrect? And how often am I unaware of that when it's happening? That guided to that particular field of psychology, but mostly, I just couldn't understand people very well. I wanted to get a handle on it because it was causing distress in my personal life."

To him, psychology boiled down to empirical philosophy. One asked good questions, and one received answers. Those answers became data one could sort through to arrive at informed opinions. "In order to answer a question, in my opinion, you need data. That's how the math part reflected. I became a research methodologist in how to quantity human experiences and those kinds of things."

Fulton graduated from Claremont and entered University of Washington's graduate program in psychology. Over three years, he noticed a disturbing trend. His peers were dropping out of school. One of the first to leave was a friend.

"Why'd you leave?" Fulton asked the next time they met up.

Microsoft, his friend explained, had taken a greater interest in the usability of their products. Hiring people who understood why potential customers might love or bounce off a new version of DOS or the next update to Excel would help them develop more user-friendly software. In turn, Microsoft would generate greater profits.

Fulton still failed to grok his friend's motivation for walking away from his doctorate. His friend looked him straight in the eye. "Half the work for three times the pay," he said.

"It was actually a pretty serious problem for the department," Fulton admits. "A chunk of my friends were suddenly working at Microsoft instead of completing their PhDs."

Fulton respected their decision but disagreed. Before jumping ship to Microsoft, they had lived the notoriously frugal existences of graduate students. That was their past, but remained Fulton's present and, until he graduated, his future. "You have to recognize that graduate students live very close to the poverty line. I remember doing calculations on cost per meal between beans and rice. The not-living-in-poverty thing I could kind of get. It wasn't for me, but it started me being more open to the notion of why smart, motivated people would leave for Microsoft as opposed to staying in academia."

Two events shook Fulton's determination to finish his education. First, his advisor went on sabbatical. That posed a problem. As sure of his abilities as he felt, he realized his field was so specialized: There were five, maybe 10 jobs available. He was confident in his abilities, but he was also self-aware enough to know that other respected schools would produce doctorates as knowledgeable as he was, if not more. Heading into graduation, he'd have to fight to land one of those positions.

Second, his advisor, who had helped to guide Fulton through his coursework, went on a year-long sabbatical to Princeton. "I figured, He's in Jersey and I'm at UW, so I might as well work at Microsoft as a contractor for a year, make a lot of money, and finish my dissertation when he comes back. Well, he never came back."

At the time Fulton learned of his now-former advisor's plans, his dissertation topic had been approved. All he had to do was run the appropriate experiments and write them up. He had a choice to make: find a new advisor in his field, a virtual impossibility since no one else was comparable, and a process that, combined with the effort involved in finishing his dissertation, would span another 18 months; or go full-time at Microsoft and continue what, to his astonishment, was deeply fascinating work.

"I went with that," he says. "I think that was one of the best decisions I've made."

Fulton became a salaried employee of Microsoft's usability group, a department founded in May 1988. He and a few others worked in a lab suite composed of four laboratories where usability experts conducted tests to monitor how users interacted with Microsoft products and reported results to managers in charge of that software. The labs were in a building with long hallways that twisted in a figure-eight shape. "Nobody could figure out where they were going. I still can't navigate that building," Fulton admits. "It had little rooms. You might be able to fit in a bed and a nightstand, and that was it. You could refer to it as a cell and not be unfair. It was just a desk, bad LED lighting, and a computer. You were there to produce work."

When Fulton joined the group in the summer of 1996 as a graduate student, he had trouble believing he was working for Microsoft. Not because he was in awe of the MS-DOS company's reputation. Quite the opposite. He was a diehard Mac user and made a habit of referring to his employer as "the evil Empire" in conversation. More troubling, though, was that he viewed everything about the environment--from how the company used psychologists to the testing process--as a waste of time.

"I was told these people were running five or eight participants and then recommending changes to multimillion-dollar projects and was like, 'That seems... inadvisable.' I was a quantitative psychologist. I believe you have to run very large numbers of people to understand patterns and tease out nuance. So I went in there with a bad attitude."

His first assignment was NetShow, a media player launched in 1996 designed to broadcast audio and video media to the Internet. Users could view or listen to media live, an alternative to waiting hours for audio or video hours to download over dial-up connections, and an improvement over Microsoft's outmoded Media Player application, which was better suited to playing audio from CD-ROM. The competition, RealPlayer, was more popular than Microsoft's solution, and the company wanted to know why.

Fulton started by reading documents that detailed what NetShow was supposed to do. One afternoon, a department-wide email appeared in his inbox bearing good news: The usability group had hit its milestone. If they could stay on track and hit their next milestones, money allocated for air mattresses could go toward beer. "I started laughing, thinking, That's pretty funny. Then I realized they weren't joking. There were people who had air mattresses so they wouldn't have to go home."

The job clicked with Fulton. It was a lot like grad school, only in an office setting, and for creating and improving products. It came with the added benefit of enabling him to continue his journey of learning to be a better human. "Engineers are told to make a thing where it's possible to do X, Y, and Z. They make that thing, but as anyone who's used software knows, 'possible' versus 'I can actually figure it out and do it, and it's not so inconvenient that I'll do it again' is separated by a massive gulf. In order to make sure that what you make gets used--not 'is possible' to use--you need to understand the human side of things, not just the software side. The goal is to make it so that what gets made, gets used, and for people to be happy in the usage of it."

Working on NetShow bore out Fulton's way of thinking. Microsoft needed to know how companies that produced video and audio content would use the software's tools. The authors of that content were too busy to spend hours or days in Microsoft's usability labs, so Fulton drove around Seattle and visited worksites to sit beside creators and observe how they performed their jobs. "I said, 'Show me what you do.' I would just take notes. They would say, 'Oh, I can't do this here. On Real Media, it's easy to do.' Or, 'I have 50 videos to do, and you make me run one at a time. Why can't I just batch-process them?' I'd listen to them complain, watch them do certain repetitive tasks and think, That took seven steps? It doesn't have to. It could be done reasonably in three steps."

Fulton and other usability researchers had their hands full. While more consumers and businesses were switching to Windows 95, MS-DOS remained the PC's dominant operating system for years afterwards. Usability researchers helped establish conventions for Windows programs so switching from one to another felt seamless. The "X" icon to close a program or folder, for instance, should always be in the upper-right corner. When a program offered a prompt with "OK" and "Cancel" options, OK should always be on the left.

NetShow became another example of an industry-wide overreliance on MS-DOS, even within Microsoft. Compression algorithms for video and audio had to be efficient for optimal playback while streaming. Through testing, NetShow's engineers determined that the algorithms for compressing video had to be unique from those written for audio compression. The problem was compression threw the visuals out of sync with audio, making the video look like a bad dub job. To re-sync, developers had to run a DOS program that asked them to enter a timestamp. Then the program forced them into alignment.

Fulton was so annoyed and disgusted by the solution that he walked into his project manager's office and said, "It's 1996. DOS?" then walked out.

"I was young and stupid, and acted inappropriately. Two days later, they had a GUI interface for syncing video and audio. I was just a contractor. I'd only been there a month and already giving flak to full-timers. That was just stupid."

In 1997, Ed Fries extended the usability lab to include games in development within his group. Fries had been a proponent of user research since his time on the Excel and Word teams. Unable to convince the programmers he managed that Word's user-friendliness needed major improvements, he asked the researchers to assemble a highlight video showing users interacting with the program. The video was a testament to the conventions Fulton and others were struggling to enact in Windows programs.

"I showed it to the programmers," Fries recalls, "and I think they'll remember it as the most traumatic half-hour in their lives, watching people try to do a simple thing using the product they'd built and failing over and over again. Programmers were literally yelling at the screen as somebody dragged their mouse over and were about to click the right button, then they changed their mind and dragged the cursor off somewhere else."

When the freelance researcher assigned to test PC games went on maternity leave, a request went out for any researcher interested in games to go to the games group and lend a hand. Fulton, still a contractor himself, submitted his resume within an hour. He loved video games and was eager to put his knowledge and burgeoning skill set to the test in an industry he enjoyed.

Fulton was assigned to CART Precision Racing, an upcoming Windows game from Terminal Reality where users race Formula One cars in simulations meant to mirror real-life F1 races as closely as possible. Fulton arrived in the games area to find a desk and a computer waiting for him. The hard drive was bare; employees had to install Windows, Office, and any other programs their job required.

CART clearly needed a lot of work. The first time Fulton ran the game, it crashed. He rebooted his computer only for Windows to halt. "I wouldn't run that version," a developer told him. "When it crashes, it corrupts the OS."

Fulton wiped the hard drive and installed everything again. "That's when I learned, wow, I'm going to learn a lot more. I'm also going to need a second computer because I couldn't even send an email to this guy, and if you don't have someone's email, you don't know what office they're in. Developers used to turn off Windows so they could run in DOS mode and not give any of their precious resources to Windows."

Upon joining Microsoft as a salaried employee, Fulton worked exclusively in the games group as its manager of research. He arrived for his first day to find the lab under his purview completely unsuited to research methods. At first, he kept his cool. He had resolved to spend the day observing and making a list of improvements to enact. By that night, he was so aghast with what he viewed as slipshod testing that he called a meeting first thing the next morning. Except for two researchers, he told the team, no one knew the proper procedures for carrying out tests and experiments. He revoked permission to work overtime in his lab from everyone save the two researchers he singled out.

The manager above him reprimanded him harshly, and the experience humbled Fulton. "I was not in my authority to do that. I just thought that if anyone thought this was okay, they weren't going to make it. That was wrong on a human level, on an organizational level, on every level. I got reprimanded on my second day of work. I still was not good at being a human. I went from being a contractor on Friday to a manager of around five people on Monday, and I was not prepared for that."

MICHAEL MEDLOCK WAS TIRED OF watching rats have sex in the dark. He did, however, enjoy studying behavior.

As a teenager, he'd volunteered at an aquarium where he'd given talks to visitors on sea mammals, sharks, the touch tank, and marine-birds, among other areas. Enrolling in college, he thought he'd become a marine biologist only to develop an interest in behavioral psychology. Even more surprising to him, he loved his statistics course, and combined the two. "It was the first type of math that, to me, was practical. What I didn't like about math was it was all theoretical. Statistics was like, 'Oh, I answer questions with statistics. This is incredible.' I just kept surprising myself with how much I liked psychology."

For his master's thesis, he trained male rats to abstain from sexual reinforcement based on the common understanding that humans and animals can be taught self-control by waiting for larger rewards later. After two years spent watching rats indulge eagerly in their rewards, he tired of the work involved in PhD programs and looked for a simpler job like tending bar before he completed his degree. While mowing lawns one summer, a friend who interned at Microsoft extended an offer to visit the office.

"It was like a different world," Medlock says. "I think we all forget now how completely unique the culture of work was at Microsoft. Nobody wore any suits. There weren't work hours; you could come and go as you pleased. There were free drinks. This was deeply strange at the time, and we forget that because technology has ushered that atmosphere into many, many jobs."

Medlock's friend told him about the company's usability engineers. He, Medlock, made a connection. He could do at Microsoft the same tests he'd done as a grad student, but in exchange for a paycheck instead of taking notes while rats copulated in front of him.

When Medlock asked where he should submit a resume, his friend said to send it to him directly. Two days later, Medlock heard from Kevin Keeker, one of the first usability engineers at Microsoft. The day after that, Medlock passed his interview and joined Bill Fulton, Marshall McClintock, and a few others on the scrappy team. His first assignment was Internet Explorer 4.0, then in a heated battle with Netscape Navigator.

When Fulton, who was working on NetShow, asked Medlock if he'd like to play Blizzard Entertainment's WarCraft II as a way to unwind, Medlock was intrigued. He'd never played a PC game over a network. "We all got together with our giant machines—we had no laptops—and all the cords, spent an hour figuring out how to network them, finally did, and we played a multiplayer game, and it was amazing," Medlock remembers. "I thought I was good at WarCraft. I was awful. I got my ass kicked so badly, but I learned a ton about, 'Oh, wow, how'd they do that?' and became a much better player."

Within a year, Medlock, Fulton, and the others had made the jump to testing games. Medlock, who was amazed that his professional life revolved around computer games, was in his element. He reached a point where he could turn reports around within two weeks. He soon developed a rapport with Fulton, who worked on the playtest side while Medlock carried out usability testing centered on behavioral studies. At first, playtesting and usability tests were carried out separately. When the groups merged, Medlock and Fulton collaborated to plan out tests and attend one another's debriefing meetings.

They made a perfect team. Fulton was the logician, whereas Medlock understood human behavior better. Ahead of one meeting, they worked out how to ask for the testing parameters they wanted, and received permission from the manager before the meeting adjourned. Fulton walked alongside his friend after the meeting broke up. "They just agreed to do what we said they should do," he said slowly, "because they like you."

Medlock waited for more.

"That's the only reason they did that," Fulton added.

"Yes, Bill," Medlock said. "That's what humans do."

As the games group signed more development partners, the usability lab had to winnow their areas of focus. Fulton, who had been promoted to a management position, ran tests on single-player modes after tests on games that required Internet Gaming Zone, Microsoft's free multiplayer service, forced him to stretch resources thin. "I made some decisions that were terrible to have to make, but I felt I made the right ones," Fulton says. "In order to test multiplayer gaming, you need a massively different set of facilities and people."

Ensemble's Age of Empires II, for instance, supported up to eight players online. That was a nightmare from a research perspective: Eight testers would play the game in a lab, but that one game made up one test. The researcher assigned to the project would need to run dozens more to track down every issue related to connectivity besides gameplay balance.

"We said, 'No multiplayer, and first hour of the game only.' We didn't test all the way to the end of games for quite some time," Fulton adds. "We tested the first hour because the first hour is the most important, so we said, 'Let's make sure people can get into the game, and then we'll just have to trust the team to do a great job on everything past the first hour.'"

Eventually, Fulton and other managers coordinated weekend shifts where one or more researchers would come in for up to 16 hours to go deep into single-player modes.

From the outside, processes for testing and research may seem interchangeable with the function of a quality assurance department. "QA is technically there for bugs," explains Fulton. "User research is free of that. It's an attempt to test what was intended and to question whether that's the right intent. In other words, my goal when I encounter a bug is to move past it: 'It's not my job to fix it, so I'm going to assume it worked and will work. I need to test the experience past the bug.' A good QA tester would log that bug and then go look for other bugs."

Whatever the usability team's methods, most within Microsoft agreed on two points: Their methods produced results, and usability changed the way developers approached software. "Most of the people there had been brought over from other parts of Microsoft and were pretty new to gaming, but we figured a few things out," says Stuart Moulder of the games group. "Bill Fulton and his team were specialists in: You put somebody down in front of the game cold turkey, and what's their initial experience? What are their friction points?"

In 1998, the usability group channeled every lesson they learned and achieved their greatest accomplishment yet: Age of Empires II's tutorial.

CIVILIZATION HAD PROGRESSED EXPONENTIALLY BETWEEN the release of 1997's Age of Empires and the anticipated sequel in 1999. Ensemble's team of developers rewrote the AI from scratch so the default difficulty level posed a challenge--maybe too much of a challenge, lead designer Tony Goodman admitted to the producers at Microsoft.

"Tony Goodman and I both shared a philosophy that if you sell people a $50 game, you want them to feel good about that, and that means you want them to win," says Stuart Moulder. "A lot of designers did not understand that. They felt they'd accomplished something if they defeated their customer. We'd figured out that one way to mass-market appeal was you let the customer win, but you let them win in such a way that they feel like they earned it. Like, 'Wow, that was really hard, but I won! I'm really good!' Whether or not they were, you'd done your job."

Playing Age of Empires II made even its developers feel hopeless. Moulder worried it would be one of those games players got frustrated with and uninstalled forever. Microsoft and Ensemble debated over what to do. They could poke holes in the AI for players to exploit, or dial down its progression so it developed at a pace equal to or slower than most players.

As the ship date approached, no decision had been reached. The game was debugged and ready, save for its untenable difficulty. Moulder hit on an idea. "My brilliant contribution was, 'You know what we're going to do? We're going to label 'Easy' to 'Normal,' and 'Normal' as 'Hard.'' We just changed the names."

It was a brilliant fix. No code would have to be written, no algorithms adjusted. "Ensemble did all the heavy lifting, but there were little things like that where we could help make a difference," Moulder adds of Microsoft's contributions. "That's my proudest game-design moment: Realizing the right thing to do was just change names."

None of the game's features, such as 13 civilizations with their own language, female units to diversify warfare, and the Town Bell, a structure in town that sends production units running for cover when rang, would have resonated with players if not for the usability department's efforts in teaching them how to play.

Tim Znamenacek, producer on the first and second titles, gave the researchers a mission. With Age of Empires II, he wanted to do more than satisfy fans of real-time strategy games. He wanted to grow the genre. To do that, Microsoft and Ensemble needed to broaden their target demographic. "Tim said his target demographic was his mother," remembers Fulton. "His mother knew about computer games, had a computer, but had never played a computer game other than Solitaire. He wanted her capable of playing an RTS game by the end of the tutorial. He set a really, insanely high bar."

Fulton and other researchers knew the importance of Age 2. The first game had been Microsoft's breakout hit under its publishing label, and expectations for the sequel were that much greater. Toward the end of growing the genre, researchers watched testers who had experience in RTS games and those who played games but knew little about titles like Age 1 and Command & Conquer. They widened their pool of testers to users who rarely played games but owned a copy of Windows. "We learned so much about what people did not understand about the game. Like, 'You need to put your hand on the mouse to play this game,'" Fulton says.

Age 2's AI could handle much of the heavy lifting. Combat units, for instance, will automatically attack hostile units within a certain radius, a feature missing from the original. However, players would never know that until they knew how to reach the point where they could build, group, and move combat units. The game's tutorial had to teach that, along with fundamentals such as how to harvest resources, construct buildings, and scout around for opponents.

Testing for the tutorial proceeded to one final playtest. Thirty testers would enter one of Microsoft's labs on a Saturday, and Fulton would monitor them as they played. The group scheduled for the test comprised consumers age 55 and older, and those unfamiliar with computers. It was Fulton's first time overseeing neophytes in the lab environment. He went to work early to Xerox paper surveys and place one beside each station. At certain points during the test, Age 2 would pause and instruct users to answer a block of questions on their survey.

Surveys were a vital tool in a researcher's toolbox, provided they were applied correctly. "The thing about a survey is you're trying to do it at scale," Michael Medlock says. "It's impossible to watch 25 people at once, but it has the advantage of you can run 25 people at once and get data instantly. Usability tests have the advantage of seeing what they can do deeply, but you have to run them one at a time. Your bottleneck is the human doing the observation because the human is the data collector, not the system."

Breaking up a test by pausing the software and asking users to answer questions may throw off their rhythm. However, it also increases the likelihood that the opinions they formed based on recent experiences will be fresh in their minds. That, and researchers acknowledge it as the only practical method of gathering information during tests. "Nobody has cracked the code--although many have tried--of getting continual feedback," Medlock admits. "Some magical brain-scanning metric that tells you how they're feeling at every moment. Anybody who tells you it does--and some will, with galvanic skin response and stuff like that--they're full of shit. It's not that it measures nothing; it's that it's not a magical brain-reading machine."

Fulton looked up at a knock. A little old lady waited on the other side. She was short, in her mid-70s, and dressed in a white sweater with floral patterns knitted on the front. "Are you here for the playtest?" he asked.

"Yes," she replied.

"You're very early," Fulton said.

"I'm always early to my appointments," she said.

She stepped inside and waited in the lobby for an hour. Once the others showed up, the older woman joined them as they filtered into the lab and took their seats at computers divided by cubicle walls. Although Fulton maintained his habit of observing everyone, he took an interest in the early bird. By the end of the tutorial, he was pleased with her progress. She'd trained villagers, set them to harvest resources, and placed a barracks. When barbarians stormed her village, she kept a level head and rang the Town Bell. As her workers ran for cover, she clicked on the barracks and trained combat units.

Fulton knew she was doomed. She lacked the resources to build enough troops to turn back the attack. But that wasn't the purpose of the tutorial. In that, she had succeeded. "She did actions that would claim she understood how to play the game. She just wasn't good at it and was way too slow to meet the challenge bar that had been set."

After the others left, Fulton waited for his new friend and escorted her to the door. "Well," he said, "what did you think?"

She paused. "It's not my cup of tea, but my grandson might like it."

As he cleaned up, Fulton couldn't help feeling impressed with her progress. She had done things far outside the bounds of most casual computer users. Even though she wasn't good at Age of Empires II, she had learned the underpinnings and put them into practice.

"I realized we really had a chance of growing the genre," Fulton says. "Anyone who has an interest could get to the game and play it, whereas before, you had to have a certain base knowledge. I knew we had a real chance of breaking the gates open with this one. To this day, every once in a while people learn I worked on Age 2, and they'll say stuff like, 'Oh, yeah, my 12-year-old daughter got into video games playing that because she was able to learn how to play.' It was all because we set the bar so ludicrously high that anyone who didn't understand going in would be told to use the mouse, would be told about left-click and right-click, and could go from there."

Experiences like that Saturday afternoon playtest reinforced to Fulton that he had made the right choice leaving his grad program for his job at Microsoft. There would be limitations, he knew. He wants to teach one day and knows many programs only accept candidates with PhDs.

"But honestly," he adds, "it's provided me a fabulous life filled with interesting people and fun challenges, and I wouldn't trade that for finishing my PhD and entering the same work even a year later. I got there at the right moment to do some very interesting things right before Xbox hit us. I got to do very interesting work, and then they threw a complete monkey wrench into everything, and I had to do very interesting work again."