SteamProphet
Limitations

Estimate

First and foremost, the SteamProphet scores posted on this site are estimates, not actual known, proven figures. They have low precision, but reasonable accuracy, so long as they are properly interpreted.

Lower Bound

If a game has a matured SteamProphet score of X, we make of it the single claim: We do not make any of the following claims: A lower bound tells you the least something has (probably) made, it does not tell you how much it actually made. In practice we find that SteamProphet scores almost always underestimate actual results, which is in fact the point -- we want to be sure that our lower bounds are true lower bounds, and in the face of uncertainty that requires being extremely conservative.

Extremely Conservative and Aggressively Pessimistic

Our chief goal with the default SteamProphet scoring system is to arrive at results that are very likely to be true, meaning that the real-world result is higher than the score. We do not particularly care how much we underestimate the true value by, so long as we do not overestimate. We therefore stack together four sources of pessimism:

Sourced from SteamSpy

We scrape both the Steam storefront and SteamSpy.com for information, but rely entirely on SteamSpy for our scoring metrics. SteamSpy uses a 98% confidence interval for our its player estimates, which means that on rare ocassions (2%), its lower bound will in fact be innacurate. SteamProphet also inherits all of the limitations that SteamSpy is subject to.

Accuracy and Precision

A lot of people use the word "accurate" when they mean "precise." Accurate, in this context, simply means, "the claim is true," or "the true value is within the specified range." Precision, on the other hand, is about having a value with low uncertainty.

Example 1: "I am on Earth right now." A 100% accurate statement that is very low-precision. It's true, but the precision is so low that it's not very useful.

Example 2: "I am in Texas right now." Another 100% accurate statement, but this one is more precise. It's still not very useful but now you have a general idea of where to go if you want to find me.

Example 3: "I am at latitude 48.858093, longitude 2.294694" is a very precise statement, but it's not accurate, because I am not in fact currently standing beneath the Eiffel Tower in Paris, France.

Example 4: "As of May 24, 2017, What Remains of Edith Finch had 18,308 players, plus or minus 4,028." This is a statement where we have 98% confidence in its accuracy -- that the true value lies within that range. It's not very precise, however -- the plus or minus range is 22% the size of the median estimated value.

Not (yet) Empirically Validated

You'll notice we have not attached a confidence figure to the SteamProphet score. We are pretty sure that our pessimistic estimate is accurate, but we're not sure how accurate it is. In order to come up with a confidence figure we would need to test it against real-world results, and I doubt that we can get a complete enough sample for that. We would be interested in doing some spot checks, though, so if you're a developer willing to share your data we would be grateful.

I would wager that it's probably about ~90% accurate, but that's just a guess for now.

Does not track activity off of Steam

SteamProphet has no idea what is happening on PSN, Nintendo eShop, XBox Live, GOG, Itch.io, iTunes, Google Play, or any other number or alternative stores developers can and do release their games on. A game that did not earn much on Steam could have earned more in other places.

Gross, not Net

SteamProphet scores are estimates of gross earnings -- that is, the total amount customers paid for the game, not how much the developers actually received, or how much money was left over after the store cut, taxes, and expenses are accounted for. A game that grossed $10M could very well still leave its developers in debt, whereas a game that earned $10 could represent a profit.

Does not directly measure "success"

Success is relative, and is most easily defined as, "did this meet your expectations?" Many developers have financial expectations for their games, but not all. Above and beyond all the financial limitations we've pointed out, SteamProphet has no current way of measuring whether a game was able to reach its developers' subjective expectations or not.

Not a reliable proof of "Failure"

Take all these limitations added together, along with the salient fact that SteamProphet only claims to measure a lower bound, not an upper bound or "ceiling" on gross earnings -- and it should be stated loud and clear that SteamProphet scores make for extremely poor "failure" detectors. If anyone ever points to a SteamProphet score as proof that some game has "failed", just link them to this sentence and kindly let them know they are mistaken.