Salting Out the Genius - Daybreak's Jennifer Chan on Production in EverQuest
EverQuest's technical director talks the challenge of introducing veteran players to new systems, and discloses a personal story that opened her eyes to how developing video games can change the world.
Making a game is hard work. Making and evolving a game that’s lasted over 20 years is Herculean.
How is it done? How does a company where developers come and go, where player demands and needs change, where technology advances and competitors rise up, keep one game going for decades? By salting out.
“Salting out” is the process of adding salt to a solution to exhaust water molecules, leaving only desired molecules behind. In game development-ese, teams add a feature, see if it works, and adapt from there. Programmers, artists, and designers are on the front lines of this process, but they’re not the only ones coming up with and implementing plans.
As EverQuest’s technical director, Jennifer Chan is one of many developers salting out the best elements and systems in the game’s persistent world. Her background in data analytics, software development, and theater made her a well-rounded addition to the Sony Online Entertainment—now Daybreak Game Company—team when she joined in 2008.
Chan and I discussed the process of maintaining a game that lives and breathes around the clock, whether or not her expertise in data analytics proved beneficial or inconsequential to her work on EverQuest, and the satisfaction she never expected to glean from a career making video games.
David Craddock: How did you enter the games industry?
Jennifer Chan: Beforehand, I was working in telecommunications. It was a lot of trying to predict the future based off of data. So, predictive analytics. EverQuest was actually my first job in the industry. On the side, I just made little games as a personal hobby. My first game was completely text-based. I did that when I was six. It was written in PASCAL. No, not Pascal. Wait... QBasic! It became PASCAL later.
David Craddock: I started programming with QBasic.
Jennifer Chan: Yeah, yeah. This was just a little game, completely ASCII art. You have your little character, the little 'a' running around. It was based oft the kids' game, capture-the-flag. It was my first game ever, and my first language. I wanted to make games [for a living], but I didn't know how to get in. So it was like, Okay, well, I have a background in theater—Shakespeare and all that stuff. My degrees are in computer science and theater. All this time, I was like, "How do I get in?" Okay, well, I found the best way was to just do things.
I did a little thing here, a little thing there. Somebody I went to college with said, "Hey, I know you've really wanted to get into the industry as a professional game developer. Do you have a resume?" It was like, say no more! I went home, sent in my latest resume, and from there on out, here I am.
David Craddock: So, you wanted to be a game developer, but how did you end up working in data analytics?
Chan, Jennifer: Yeah, I've wanted to get into gaming since I was six. I grew up with the Sierra On-Line adventures: King's Quest, Quest for Glory, Space Quest. All those. They were my favorite games growing up. I was asking, "How do I make this? How do I get in?" Data analytics came about because I couldn't get into the industry yet, but I needed to eat. I needed to make money, so I needed to have a job.
I've been doing data analytics for telecom, and before that I did general IT and programming work: making a payroll system, all that stuff.
David Craddock: You have a lot of interests: theater, data, programming. The cool thing about a CS degree is that it itself is broad, too.
Chan, Jennifer: Yeah, it is.
Craddock: It's kind of like data analytics: it's broad enough that you can apply it almost anywhere. Did you find that your concentration in data analytics allowed you to be diverse in terms of abilities throughout your career? Or would you say it's more your natural inclination to be curious and try different things?
Chan, Jennifer: Analytics definitely has helped, especially on EverQuest, because it's so much data. But when I came on to the team, I was like, "Oh. Well, I've processed [this much data] in an hour at my old job." I do have a natural curiosity: How does this work? How does that work?
Combining [computer science] with theater actually helped even more than data analytics to get here, because when I got here, the artists were using the exact same 3D modeling tool I'd been using in theater. Exact same version, same software. I said, "Oh! I know this! I can do all these things!" Someone actually said, "We hired you as a software engineer, right? Not as an artist?" [laughs]
Craddock: That fits, though, because you were introduced to playing games during an era of the industry when so many developers I spoke to said they wore so many hats. Nobody did just one thing.
Chan, Jennifer: Absolutely. The Sierra On-Line era, it was like, "This game was made by five people."
Craddock: It's funny. I played adventure games back then, too, and it seemed like most fans of the genre fell into one of two camps: those who were introduced to it through LucasArts, or through Sierra. King's Quest 6 was my first. Then I played Gabriel Knight, and I was just hooked. I still play through those games regularly.
Chan, Jennifer: Yeah, yeah. I got them on GOG. It was so cool: "Oh, yeah! I remember these!"
Craddock: Before we dig into EverQuest, I'd like to hear more about your career in data analytics so I can better understand that slice of your abilities and how it has applied to the game.
Chan, Jennifer: When I was doing data analytics, my division was a subset of the quality assurance division. But we did a lot of project management as well. I primarily worked with the finance people and project management people, to take all their algorithms and formulas that they had developed over decades of, okay, we're going to predict this project. It's going to have these technologies, and we want to get a rough estimate of where should we run this project? It was an international company, and we had studios all over the place: Shanghai, Tokyo, Australia, Israel, India—just all over the place.
It was a big company. We basically split off of AT&T. We had this motto of, "We're a young company, but we have the data of 60 years' worth of software development." My primary role was to take all this data and try to build new models, and supplement subject matter expert project managers with additional data, so they could extract information from, say, 1982. And then, based off of that, try to build a budget plan for projects: Where's it going to be? How much? How long is it going to take? What technologies do you have to have? What technologies are typically associated [with these types of projects]?
I crunched all those numbers and turned them into charts that executives could bring up and go, "I think it's going to be this because of all that information."
Craddock: So, take me to your job with Daybreak. How would you define the role of "technical director," and did your expertise in data analytics facilitate that role?
Chan, Jennifer: Actually, it hasn't really as much as you would think. When I came in, we were very much, "We make games. We're not analytics people. We're not Zynga. We don't make financial models projecting how much a game is going to make." When I came in, I was trying to bring in more of a business aspect, more corporate software development practices, procedures, and policies.
It was a blend of, I'm learning professional game development and trying to impart, "This is a pretty standard practice in software development. Where is our peer review system? How do we assign work?" At the time, we were still doing a very pseudo-waterfall method [author’s note: progress flowing in a single direction], and even then it wasn't formal. It was, "Go talk to your lead. Your lead will give you a task. Go do the things." There were never any planning meetings. We had a weekly meeting where people would say, "I did this thing this week," but it was definitely a culture change coming here.
Craddock: That sort of change doesn't happen overnight, right? Especially for a team working on a game that's been out almost 20 years?
Chan, Jennifer: Right.
Craddock: What was your process for implementing all these changes?
Chan, Jennifer: Well, the first week I was here, they welcomed me to the team and had me play EverQuest a little bit. They also helped me get my machine set up for development. I said, "What do I need to know?" and they said, "Here's an [introductory] email we've been sending around for years." I said, "Why isn't this in an [easily accessible] location? Why is it buried in emails?"
I found a wiki site that we use internally. It's basically Wikipedia, but internal. I said, "I'm going to [take these processes from the email and] put them here." It's been bit by bit: "Okay, I'm learning how to do this, so let me write up something, and I'll put something on there." Over time, it became a process. We've been building into that, little by little, going, "Okay, what just happened? We released this gigantic expansion. What failed? What did we forget to do? Wait, why isn't this a checklist?"
Usually after every project, every big release, I [take stock of] what happened, what we can improve on, and what should happen every single time. I just make lists. Over time, we have a lot of processes to help that process. For instance, we're setting up two new servers, progression servers for the anniversaries. Engineers can just go to the [wiki] page and follow detailed instructions because there are 50 different things you have to change when you add a new server.
Craddock: At previous jobs, you talked about working with current and past data to predict what may happen in the future. EverQuest is kind of like that, because it's a living game with two decades of history, and continues to evolve. The development team is trying to think ahead. Do all of these processes you institute have to wait until after a product release, or are there processes you can implement by keeping your finger on the game's pulse, so to speak, and do some on the fly?
Chan, Jennifer: Yeah. We don't have a lot of data-tracking in EverQuest still, because it's a 20-year-old game. So a lot of it is, we listen to our communities, we read fan sites, we have community forums. And a lot of it is, "Okay, I've played this, too, and experienced this, too. Let's try it again." We just try to listen to our players as much as we can to get a sense of things, especially things that are brand-new and haven't been in the game before. We're looking for, "How does this feel?"
We also have EverQuest 2 to lean on, and DCUO [DC Universe Online], a lot of similar games. We also have Lord of the Rings Online and Dungeons & Dragons Online that we work with closely. Well, kind of. Our player demographics and our businesses are run similarly, and our players are very similar people.
Craddock: You've raised many great points. One is that this game is only 20, which means there are probably only so many new processes you can shoehorn into a mature project before it starts looking like the Winchester House in California.
Chan, Jennifer: [laughs] Right, right.
Craddock: What are some initiatives you've wanted to add on, but due to the maturity of the game, you either could not, or had to rethink in terms of implementation so they could be made to work?
Chan, Jennifer: Ah. So, a great example of this is the advanced loot system we've put in. Before we put this in, the only way to loot things was you had to literally go to the corpse, you had to lock the corpse down so nobody else could touch it, and then you had to go through every item and say, "Do I want this? Do I not want that?" Our players had actually developed the whole master looter system and death-kill points, DKP system, on their own because we had nothing for them for that. It was, "Okay, you, you're the master looter. We trust you to take things [and distribute them fairly] and not just run away with the loot."
With the advanced loot system, we made that a little bit easier. We've allowed you to add in filters to say, "I always want this," or "No, I never want this." Putting that in was tricky because we couldn't get rid of the loot system. If you don't have the [advanced] system turned on, you can still go to a corpse, lock down that corpse, and nobody else can touch it while you're standing at it.
Allowing for both systems to work internally was [difficult]. It was, there's this one situation over here, and we've solved it in the new system. How do we solve that situation in the old system so it doesn't break?
Craddock: Can you not, say, gut the old system and embed the new system because EverQuest is persistent? Or is it just so embedded?
Chan, Jennifer: It was mainly a player [decision]. It was absolutely something we could have ripped out, but we know that change, especially large change, is scary. We wanted players to use the new system and find that, yes, it is a better system. It's faster, and allows you to play the game instead of standing around looking at items for hours. But we also said, "Everybody's going to freak out if we just take out the old system and give them the new system. No one's going to know how to use it off the bat, and people won't like it for one reason or another." Like, "I don't like this pixel because it's too far to the right."
Our players have voiced concerns about, hey, you moved that five pixels! Well, yes, because it wasn't aligned. But you moved it five pixels! Well... yes.
Craddock: That's another good point. I've been playing the Resident Evil 2 remake, and the original game is over 20 years old. But that's a game Capcom can change dramatically because no one plays the original anymore. Whereas in EverQuest, you have players who have been playing since March 16, 1999. Community input seems to be more important than anything you and the development team may want to do with the game.
Chan, Jennifer: Right, absolutely. Even with these new progression servers we announced, we definitely had some fair negative feedback. It was fair, so we said, "Let's change that. Let's make that better." That came from the community. We said, "You're making fair and reasoned arguments. We understand, and we agree, so let's change it."
Craddock: What are some of the new processes you've learned working on EverQuest that you weren't familiar with in data analytics?
Chan, Jennifer: I really got used to change management and, again, with players. In my old job, you worked for the company, and they made internal tools. Tough. We took the changes and liked them because that's what the company was doing. Obviously, we can't do that with players. Really figuring out and trying to estimate how much change is too much, and how much can we get away with—slowly salting out the genius.
Craddock: Since you've had experience in programming and using the same art package as EverQuest's developers, has that experience helped you either integrate content yourself, or gain an understanding of what the developers have to do on your planning and scheduling side?
Chan, Jennifer: Being familiar with the program allowed me to understand some of the technology the artists were using, but the pipeline was [established] by the time I'd joined the team. However, it's helped in that when we need to do software upgrades, I can do that now. We don't have a technical artist on the team right now, and when we need to upgrade software, people ask, "Who can make sure our tools work?"
I'm going in there and upgrading the art tools, something typically reserved for and done by technical artists.
Craddock: I have to imagine that someone in your position is constantly thinking ahead about what to implement, and how. Have you thought beyond the content for the upcoming 20th anniversary?
Chan, Jennifer: Oh, absolutely. In fact, we were having a discussion this morning about things we want to roll out next year. To get us to that position, we had to ask, "What do we need to get into place?" All of these other smaller projects have to be in place or else [our plan for next year] just will not work. We just couldn't do it.
I'm always thinking about new features, quality-of-life changes. If I really like a feature, I may think, "I also think players would really like this, but we can't do it right now. Let's try to slate that for December, or maybe even next year."
Craddock: What does being part of EverQuest's 20th anniversary mean for you, considering most games come and go within weeks or months?
Chan, Jennifer: It's humbling to be a part of something so big, and it's an honor to get to expand and improve and make things happen for people. Humbling and a lot of respect.
Craddock: Since you've wanted to develop games for a living since you were a kid, it sounds like this is something of a dream job for you. What have you enjoyed most about it that you never suspected would be part of the job?
Chan, Jennifer: The players. I did not expect the players to be so... vocal? [laughs] So impassioned about what I would normally consider pretty minute things. That's something I definitely didn't expect. But on that same train of that, they are so impassioned and vocal because EverQuest is such a big part of their lives.
I didn't realize how much I could actually change the world for the better by [developing] a game. I didn't expect that before coming into the games industry. I had no idea this would change lives. That it would literally help people beat diseases. I always get a little teary when I think about one of our players who was told by her doctor that she had a terminal illness, and, yeah: "The best I can give you is six months." She attributes EverQuest to allowing her to live... I think it's eight years, now. Eight years past that six-month diagnosis.
Stories like that make me choke up. Wow. I'm actually getting to do something meaningful for people.
-
Shacknews
replyDavid Craddock posted a new article, Salting Out the Genius - Daybreak's Jennifer Chan on Production in EverQuest
