EstimateFirst and foremost, the SteamProphet scores posted on this site are estimates, not actual known, proven figures. They have low precision, but reasonable accuracy, so long as they are properly interpreted.
Lower BoundIf a game has a matured SteamProphet score of X, we make of it the single claim:
- The game has probably grossed at least $X USD, or more.
- The game has grossed exactly $X USD
- The game has grossed less than $X USD
- The game has not grossed some larger amount
- The game is a "good" game
- The game is a "bad" game
- The game is financially successful
- The game is financially unsuccessful
- The game has met its developer's expectations
- The game has not met its developer's expectations
Extremely Conservative and Aggressively PessimisticOur chief goal with the default SteamProphet scoring system is to arrive at results that are very likely to be true, meaning that the real-world result is higher than the score. We do not particularly care how much we underestimate the true value by, so long as we do not overestimate. We therefore stack together four sources of pessimism:
- Use "players" metric rather than "owners" (players is always < owners)
- Use lower bound of players rather than the median value
- Use lowest price the game has ever been for sale at in its launch window
- Round down to the nearest thousand. Yes, even for values like "999.99".
Sourced from SteamSpyWe scrape both the Steam storefront and SteamSpy.com for information, but rely entirely on SteamSpy for our scoring metrics. SteamSpy uses a 98% confidence interval for our its player estimates, which means that on rare ocassions (2%), its lower bound will in fact be innacurate. SteamProphet also inherits all of the limitations that SteamSpy is subject to.
Accuracy and PrecisionA lot of people use the word "accurate" when they mean "precise." Accurate, in this context, simply means, "the claim is true," or "the true value is within the specified range." Precision, on the other hand, is about having a value with low uncertainty.
Example 1: "I am on Earth right now." A 100% accurate statement that is very low-precision. It's true, but the precision is so low that it's not very useful.
Example 2: "I am in Texas right now." Another 100% accurate statement, but this one is more precise. It's still not very useful but now you have a general idea of where to go if you want to find me.
Example 3: "I am at latitude 48.858093, longitude 2.294694" is a very precise statement, but it's not accurate, because I am not in fact currently standing beneath the Eiffel Tower in Paris, France.
Example 4: "As of May 24, 2017, What Remains of Edith Finch had 18,308 players, plus or minus 4,028." This is a statement where we have 98% confidence in its accuracy -- that the true value lies within that range. It's not very precise, however -- the plus or minus range is 22% the size of the median estimated value.
Not (yet) Empirically ValidatedYou'll notice we have not attached a confidence figure to the SteamProphet score. We are pretty sure that our pessimistic estimate is accurate, but we're not sure how accurate it is. In order to come up with a confidence figure we would need to test it against real-world results, and I doubt that we can get a complete enough sample for that. We would be interested in doing some spot checks, though, so if you're a developer willing to share your data we would be grateful.
I would wager that it's probably about ~90% accurate, but that's just a guess for now.