Opinion: NPD's Digital Tracking Leads to New Inaccuracies

It's an open secret that game sales data from market research firm The NPD Group hasn't given an altogether accurate or complete look at the video game industry for a long time. The advent of digital sales and increasing influence of marketplaces like Steam, PlayStation Network, and Xbox Live have made its brick-and-mortar model not quite obsolete, but not exactly a reliable snapshot either. In an announcement this morning, the firm announced it would be including digital sales with data straight from publishers. However, this move is trading one kind of neutral, inadvertent inaccuracy for another, consciously selected type.

For years, NPD has held an essentially uncontested role as the arbiter of video game sales data. And in an industry especially rife with fans who make a sport out of corporate competition, this has raised its profile significantly. Media and the community alike pore over data to learn which team is "winning" the hardware war, whether a new franchise had a strong debut month, or if a long-standing series is starting to slip. It's as much a part of video game culture as Billboard is to music.

But it isn't immune to the ravages of time, and digital sales have taken their toll. Starting this month, NPD is attempting to address that by including digital data in its reports. The data is provided by the publishers and covers three of the most popular distribution services: Steam, PlayStation Network, and Xbox Live. To its credit, the NPD Group made sure to have some of the biggest publishers locked down before rolling out this new service, including Activision Blizzard, Electronic Arts, and Ubisoft, among others.

However, the system as implemented leaves several gaps. Despite the inclusion of Activision Blizzard, Battle.net is not included, which means we'll have no data on the digital PC sales of games like Overwatch. Similarly, Electronic Arts isn't including Origin, and doesn't release PC games on Steam, which means we'll have no insight whatsoever into EA's PC sales. Microsoft and Sony aren't providing data on their own first-party games. Nintendo's eShop storefront isn't being included at all. The move to dollar sales instead of unit sales means free-to-play games will never make the cut. And since the data is self-selected from established publishers, indie games that explode in popularity like Minecraft did at first inception are going to be counted out by default.

The result is likely to be even more inaccurate than before, at least for the time being. While NPD's previous data only counted physical sales, it was at least equal in that every publisher and video game was playing by the same set of rules. Those versed enough in NPD's methodology all understood that this was a reasonably accurate snapshot of a particular kind of data--that of physical sales. The new data sets will mix physical and digital together, but not all digital, leading to a hodge-podge of inconsistent information. A mock-up spreadsheet attempted to stem this problem by marking a game in the top 10 that isn't from a participating publisher, but that does nothing to serve the countless non-participating publisher games that may rightly belong in the top 10, or belong at a different rank.

NPD, for its part, seems to recognize the inaccuracies. Representatives told Shacknews that the firm has been working on this initiative for some time, and reasoned it would be better to launch now to get the ball rolling than to wait. It's hard to blame the group, given the multitude of different deals and special access it would need to be granted to get a truly full accounting of digital sales. Even capturing half of the digital space is a feat, and getting the biggest publishers and biggest storefronts on-board is certainly an accomplishment.

Understandable as it may be, though, the fact remains that right now, and for an uncertain amount of time in the future, these numbers will be inaccurate. We don't know when they'll stop being inaccurate. And as more publishers and storefronts join, they'll be inaccurate in varying ways and to varying degrees that will make month-to-month comparisons nearly impossible.

I can't help but think this is partly strategically motivated. By introducing a system that rewards participation, it inversely punishes sitting out. Publishers and storefronts that don't get on-board will have their games counted for less than they may actually be. It's a subtle, clever, and even slightly mischevious way of encouraging more companies to join ranks. 

At Shacknews, we will continue to report on the NPD Group's monthly sales data. We will feel an obligation, though, to remind readers to take the findings with an extra grain of salt. This new methodology of sales tracking means that from now on, rather than getting an accurate picture at an incomplete segment of the industry, we'll be seeing an incomplete picture of the entire industry.