How to run a SteamProphet Group
There's a couple big things and then a bunch of little details.
Please consult the FAQ for more details.
- Find some participants
Get a group of friends and colleagues together. Appoint someone to run the group (presumably you). I've found people don't have to make predictions every week to still be a useful part of the group. It's interesting just to join in the resultant discussions, and a lot of people just like to observe. You do need at least a few people regularly making picks to get things going, though. You can run your group on something as low-tech as a private email list.
Pick a schedule
For convenience sake, my SteamProphet groups are run weekly. Every Sunday I send out the list of eligible titles for the next week based on upcoming Steam games scheduled for release in the next seven days. Everybody votes before noon on Monday (I'm not too strict about enforcing the deadline). I've found this is regular enough to be interesting but not so often as to be overwhelming.
Nothing is stopping you from running once a month, or even every day -- it's up to you. That said, most games don't schedule their release dates more than about a week out. In fact, some games don't specify concrete release dates at all before they launch, and some don't commit until later in the week, so you're bound to miss a few. (We're OK with this).
Generate a list of games to pick from
We provide a list of upcoming releases. This is nothing fancy -- it's a simple scrape of the Steam storefront and SteamSpy data, but it is customized for this project's purposes. By default it filters out all games that are either Free 2 Play, VR-exclusive, or not tagged with "indie." We are looking into expanding the feature set in the future.
If you prefer to generate a list by hand, go to the upcoming games section on Steam:
Then, use the filters to narrow the choices. (I always filter by checking the "indie" tag). Here's a link to the feed I use each week. Personally, I always go through and manually remove all F2P and VR games from my list, but the choice is up to you. (I haven't yet figured an easier way to sort those out yet automatically.)
If you're the one running a group, I want to emphasize that you'll get more participation if you make it REALLY EASY for your participatns to go through the feed. I used to post a link to the filtered upcoming games page along with a text list of eligible games, but that was kind of a pain for my participants. Later I started creating a list of games with embedded links to each one. Remember! Convenience = participation!
Measure results when the picks mature
Set a "maturation period" for when you'll go back and score the games people picked. We've found 4 weeks to be the sweet spot. That's enough time to get a good feel for how well the game has launched out of the gate, but it's a short enough time that the game hasn't gone on any extra discounts, sales, promotions, bundles, free weekends, etc, that can make it really difficult to interpret SteamSpy stats.
The scoring method we use is this:
Get the SteamSpy "players" stat and take the lower bound.
If you have (3,074 +/- 1,606), do (3,074-1,606) = 1468. If it goes negative, floor to zero.
- Get the lowest price the game has ever been available at since launch.
- Multiply the lower bound of the players stat by the lowest price.
- Round the resulting figure down to the nearest 1,000.
Some quick examples:
- (1,074 +/- 2,606) is 0 -- the uncertainty is larger than the base estimate!
- A preliminary score of 940 rounds down to a final score of 0.
- A preliminary score of 34901 rounds down to a final score of 34,000.
- Get the SteamSpy "players" stat and take the lower bound.
Keep it respectful -- no Gambling, no Roasting
You're running what amounts to a fancy discussion group here, so you have all the normal human social dynamics, with the added burden of some special wrinkles unique to SteamProphet. For one, these are real people's actual games, and will often be games developed by fellow members of your own group. Criticism is inescapable and in fact necessary, but rudeness, derision, and mockery are unecessary, harmful, and discourage honest and open participation. I do not tolerate any of this behavior in my own groups and I strongly warn against it in those who choose to run their own. I want to make a similar note about gambling and bets -- it is expressly forbidden in my groups, as I explain in the FAQ. I can't control what third parties do in their own groups, but gambling can cause a lot of problems, and I feel it undermines the purpose of the project by turning it into a zero-sum competitive game rather than a collaborative and sobering learning exercise.
As you play you'll notice certain games will crop up in your list of picks week after week. This is normal – these are just games that have slipped their release date. If a game gets delayed, anyone who picked that game that week should score a 0 for it. One popular rule is to let people pick one or two "runner up" picks in addition to their regular five as alternates in case of something like this. If you let people pick more than one alternate, players should specify the replacement order so they can't pick whichever one did better in the event of a substitution.
Abnormal Owners:Players ratio
On rare occassions you'll notice a game has a really abnormal owners to players ratio on SteamSpy. This is usually a sign that the game has been in a bundle or other mass giveaway. A common source of this for games is titles that debut in the Humble Monthly subscription service. You might choose to exclude these games when you detect them; the groups I've been running are split on that issue. This is where alternate picks can be useful.
Generally, we treat Early Access games exactly the same as "fully launched" games on Steam for the purposes of SteamProphet. The key thing is that we just care about whether the game is debuting on Steam, as in it wasn't available for purchase last week, but it will be this coming week. You can choose to exclude E.A. games from your group if you want, but I find them useful. (Northgard famously launched as E.A. and exceeded all expectations, for instance). The most important thing is that you don't treat an Early Access → Full Release transition as a "launch." That game's already been available for sale on Steam for a while, so the SteamProphet calculations won't really work.
The scoring method is not designed to precisely pinpoint how much money a game has made – that's basically impossible without direct data from the devs. What we want to do is to accurately – but not precisely – make valid comparisons between different games to some degree of confidence. Originally we just compared lower bounds of the "players" metric, but we found that low-priced games were routinely out-scoring high-priced ones because it's obviously easier to move a bunch of copies at $2.99, but the $39.99 games were probably earning more overall. Multiplying by the lowest price serves to "normalize" differently priced games' scores and make them more of an apples to apples comparison. The SteamProphet score is a FLOOR on a game's earnings -- it has, we're pretty sure, made at least this much money, or more. Some developers have shared true figures with me and it looks like the SteamProphet score can underestimate the true value by at least as much as half. That's fine, I just care that it's almost always less than the real value. That's because a reliable floor can serve as a "hit detector" – if the floor is pretty high, that means the game did really well.
This is not a be-all end-all scoring method, and we encourage and seek to develop alternate scoring methods for those interested in testing other sorts of predictions. The default scoring method, is, however, a fairly reliable quantifiable method for evaluating games against a common baseline. It is subject to many limitations and these should be carefully considered before drawing any conclusions.