TechCast vs SciCast
TechCast vs SciCast
A comparative study of two great forecasting systems
Editor's Note: Anamaria Berea, PhD, conducted this study while working for SciCast. She is now on the faculty at the Smith School of Business, U. Maryland, and a TechCast Expert.
TechCast vs SciCast
SciCast is a crowdsourced forecasting platform for science and technology run by George Mason University. Essentially, it is a combinatorial prediction market, based on the idea that the collective wisdom of an informed and diverse group is often a better predictor than the judgment of a single expert. SciCast is probably the largest science and technology forecasting effort we know of, crowdsourcing in real-time from a pool of thousands of scientists and enthusiasts.
On another hand, TechCast Global is the oldest science and technology platform available, and also the only one that continuously forecasts long term, strategic, high-impact questions. TechCast pools the collective intelligence of empirical background data and the knowledge of 130 experts worldwide to forecast breakthroughs in technology, social trends and wild cards, thereby covering the entire strategic landscape.
If the present level of uncertainty is defined as 100 percent, â??TechCast fâ??indâ??sâ?? their process of collective intelligence reduces uncertainty to about 20 to 30 percent. TechCast research, such as annual validation studies, find that the average error of forecasts over time is roughly +1/-3 years at ten years out. The -3 error is the average number of years the forecasts tend to fall short of arrival. This confirms the well-known tendency toward optimism. TechCast calls it "forecast creep."
SciCast calculates forecasting accuracy using the Brier score for each question and does not evaluate the forecasting accuracy of the entire prediction market. The Brier score is a measurement of the accuracy of probabilistic forecasts, where observations are either 0 (no occurrence) or 1 (occurrence); the Brier score calculates the mean square error of probabilistic forecasts. In general, SciCast scored low on the Brier score (closer to 0), showing the power of aggregated probabilistic forecasting.
In a research effort run at George Mason University in early 2014, SciCast imported and adapted 7 questions from TechCast in an online experiment to compare the forecasting accuracy and methods of these two large systems. The experiment showed that the two systems are largely different and complementary, making it hard to make useful comparisons. But we learned many lessons. Perhaps the most important lesson is that both systems have a great impact and represent amazing advances in forecasting science and technology.
For example, some of the questions forecasted on both systems were: "Will 30% of newspapers, magazines, scientific journals, and books be delivered electronically by end of March 2014?"; "Will 30% (by value) of music, movies, (non-gambling) games, and other entertainment be enjoyed online by end of March 2014?" or "Will 10% of the value of retail consumer goods and services be sold online by the end of March 2014?" In order to resolve these questions, the researchers had to draw from many research publications or technological reports published by other third parties. Overall, the TechCast questions on SciCast performed well, but not as well as the more specific questions that had a more comprehensive resolution language and shorter deadlines. This points again to the idea that each forecasting question is very embedded in its' forecasting platform and they should be addressed accordingly.
One of the first difficulties in comparing them came from the fact that SciCast is designed for short term, very specific questions, while TechCast is designed for broad areas of inquiry. That was also one of the main reasons that only 7 TechCast questions were implementable and adaptable to the SciCast platform.
Another difficulty came from the methodology itself. SciCast is a probabilistic-based forecasting effort that answers primarily questions formulated around the "happening" or "non-happening" of an event before a deadline, while TechCast forecasts an absolute year "When event X is most likely to happen." Basically, the unit of analysis in TechCast is the Most Likely Year, while in SciCast, it is the probability of a forecast happening. SciCast is based on aggregating individual probabilities, while TechCast is based on expert judgments of a combination of qualitative assessments and technology adoption curves.
In TechCast, confidence in forecasts is represented by the confidence of the experts regarding their assessment; this is a subjective measure of the expert and not a statistical measure. In SciCast, this subjectivity is intrinsically incorporated in the individual probabilities.
The method used to resolve the forecast question also differs between the two platforms. SciCast requires very specific language and a priori defines the resolution terms of a question, defining each word in the question. TechCast targets are usually adoption levels, so various sources must be used to find data confirming when those adoption levels are reached.
In the end, it was very difficult to compare the two systems, as there were few comparable measurements or methodologies against which we can hold both accountable. But we have learned some very important lessons, and the most striking one is perhaps the fact that the two systems are complementary but not similar. The two systems can learn a lot from each other, and users can benefit from using both TechCast and SciCast.
All Public Comments
© 2013-2017 TechCast Global Inc Printed By: Dec 10, 2018 For personal use only