The "CAPS" Prediction System and Stock Market Returns
Faculty Research Working Paper Series Christopher Avery
Harvard Kennedy School and NBER

Judith Chevalier
Yale School of Management and NBER

Richard J. Zeckhauser
Harvard Kennedy School

July 2011 RWP11-028: Revision of RWP09-011, April 2009

The views expressed in the HKS Faculty Research Working Paper Series are those of the author(s) and do not necessarily reflect those of the John F. Kennedy School of Government or of Harvard University. Faculty Research Working Papers have not undergone formal review and approval. Such papers are included in this series to elicit feedback and to encourage debate on important public policy challenges. Copyright belongs to the author(s). Papers may be downloaded for personal use only.
www.hks.harvard.edu

The “CAPS” Prediction System and Stock Market Returns

Christopher Avery, Judith Chevalier, Richard Zeckhauser

July 2011

Acknowledgements: We appreciate the efforts of the Motley Fool Company in making data available to us. Avery and Zeckhauser gratefully ackhowledge support from the Alfred P. Sloan Foundation.

Abstract: We study the predictive power of approximately 2.5 million stock picks submitted by individual users to the “CAPS” website run by the Motley Fool company (www.caps.fool.com). These picks prove to be surprisingly informative about future stock prices. Indeed, a strategy of shorting stocks with a disproportionate number of negative picks on the site and buying stocks with a disproportionate number of positive picks produces a return of over nine percent per annum over the sample period. These results are mostly driven by the fact that negative picks on the site strongly predict future stock price declines; positive picks on the site produce returns that are statistically indistinguishable from the market. A Fama French decomposition suggests that these results are largely due to stock-picking rather than style factors or market timing.

Christopher Avery, Harvard University John F. Kennedy School of Government and NBER

Judith A Chevalier, Yale School of Management and NBER

Richard Zeckhauser, Harvard University John F. Kennedy School of Government

1 INTRODUCTION

3

1

Introduction

“Social investing” or “crowdsourcing” websites attempt to forecast stock price performance by aggregating predictions from individual website participants. We analyze the informational content of these predictions for the future price movements of individual stocks. Our data consists of more than 2.5 million stock picks provided by more than 60,000 individuals from November 1, 2006 to December 31, 2008, a period with significant swings in stock market performance. These individuals made predictions through the CAPS open access website created and operated by the Motley Fool company (www.caps.fool.com). The Motley Fool prediction system is motivated by a hypothesis, contrary to the efficient markets hypothesis, that posits that many individuals—each with limited information—can provide very accurate assessments of future movements in individual stock prices, if their information is elicited and aggregated in an appropriate fashion. While collaborative filtering has been demonstrated to be useful in a wide variety of contexts (such as the Ebay user rating system or the Amazon recommendor system), it has not been demonstrated to be of value in the challenging context of predicting stock price movements. It is natural to ask how collaborative filtering could possibly elicit information that is not already incorporated into stock prices: why not trade on information rather than volunteer it without prospect of financial gain to a public website? The idea behind collaborative filtering of stock opinions is that individuals have high trading costs and imprecise information. Individuals whose trading costs exceed the expected value of their imprecise information will rationally choose not to trade to the point where this information is fully reflected in prices. Thus, the idea behind collaborative filtering is that it aggregates the unutilized information of such individuals.1 . We analyze the informational content of stock market picks submitted to CAPS by tracking the performance of portfolios formed on the basis of those picks. We intentionally use simple algorithms to create portfolios based on positive and negative picks (predictions of increases and decreases in the prices of individual stocks, respectively). We show that buying positive picks and shorting negative picks produces annual returns of over nine percent over our sample period. Using a Fama French decomposition, we show that these excess returns are largely due to stock-picking rather than style factors or market timing.
Why individuals bother to post their information is another interesting question. As discussed below, Motley Fool labels and promotes top performers in the CAPS system. This may be a source of utility in it of itself
1

1 INTRODUCTION

4

This paper builds on several separate strains of academic literature within finance. First, many studies assess the ability of institutional investors to surpass market profits without taking on excessive risk. Studies of investment professionals, such as mutual fund managers (Chevalier and Ellison, 1999, Wermers, 2000, Pastor and Stambaugh 2002, and Baker, Litov, Wachter, and Wurgler, 2006, among others), newsletter publishers (Metrick, 1999) and analysts (Mikhail, Walther, Willis, 2003) generally conclude that a small percentage of people consistently “beat the market”. However, the results of these papers suggest that there are relatively few people who possess either the information or the ability to successfully pick individual stocks. A second set of studies supports the conventional wisdom that individual investors perform poorly as stock market investors. Odean and coauthors, in a series of papers, exploit a large dataset of individual customer accounts at a major discount brokerage firm to analyze the results for individuals as traders. Odean (1999) finds that individual investors’ purchases tend to underperform their sales by a significant margin. Barber and Odean (2000) show that, on average, the stock choices of individual investors underperform market indices, and that this underperformance is particularly acute for active traders. Barber and Odean (2001a) find that men are more likely to be active traders than women, and that this trading hurts their portfolio returns. They link this evidence to survey results showing that men are more confident in their investing abilities, and conclude that overconfidence in stock-picking ability leads to underperformance. Finally, Barber, Odean, and Zhu (2009) show that stocks purchased heavily by individual investors in a given week tend to outperform other stocks for the next two weeks, but then underperform the market in the subsequent months. Despite these findings, there is growing evidence that some individual investors have superior information about some assets. For example, Coval, Hirshleifer, and Shumway (2005) demonstrate that individual investors’ trades show strong persistence in performance. This suggests that some individuals may be able to earn abnormal profits. Furthermore, a number of papers suggest that the excess performance may result because those individuals concentrate their portfolios in stocks for which they have an informational advantage. For example, Ivkovic and Weisbenner (2005) and Massa and Simonov (2006) using U.S. and Swedish data, respectively, find that investments in local stocks outperform non-local investments and in the Swedish case, outperform market benchmarks. Similarly, Ivkovic, Sialm, and Weisbenner (2008), using data on the investments of a large number of individual investors made through a discount broker from 1991 to 1996, find

1 INTRODUCTION

5

that among households with account balances greater than $100,000, those that hold only 1 or 2 stocks outperform those that hold 3 or more stocks by 41 basis points per month. They also show that the excess performance is concentrated in non-S&P500 stocks that receive little analyst coverage. These findings are consistent with some investors concentrating their holdings in securities for which they have a true informational advantage. Such an informational advantage would be harder to achieve for stocks that are widely followed. Another set of related studies considers the effect of the internet on stock market trading and prices. The internet has lowered the cost of stock trading, but has also made it possible for individuals to participate in the stock market as commentators on message boards. One common theme of prior research is that the internet may exacerbate behavioral biases that lead to suboptimal investments (Barber and Odean, 2001b), and possibly even create new methods for stock manipulation (Frieder and Zittrain, 2007). A series of recent papers assesses the informational content of postings on message boards such as Yahoo! and Raging Bull as well as the effect of these messages on stock trading and prices. The most robust finding in this literature identifies a connection between the volume of messages about a stock and future trading of that stock—a high volume of messages tends to predict higher future trading volumes and pricing volatility (Antweiler and Frank, 2004). In terms of information, message board postings overlap in content with forthcoming news stories (Antweiler and Frank, 2004), and earnings announcements (Bagnoli, Daniel and Watts, 1999), but message boards promulgate them sooner than traditional media sources. Yet there is at best limited evidence that the informational content of message board postings predicts future price movements for individual stocks. Even though message board postings may predict future news articles, the news articles themselves have limited and short-lived predictive power on future stock prices (Tetlock, 2007). Similarly, the assessed correlation between message board content and stock price movements is generally small and short-lived (Das and Chen, 2007, Tumarkin and Whitelaw, 2001, Antweiler and Frank, 2004), though very unusual volumes of message board activity correlate with substantial next-day price movements for thinly traded microcap stocks (Sabherwal, Sarkar, and Zhang, 2008) and negative future returns for a broader set of stocks (Antweiler and Frank, 2006). This paper departs substantially from the previous literature on message board content because of the nature of the data compiled by the Motley Fool CAPS website. CAPS differs from stock message boards in three im-

1 INTRODUCTION

6

portant ways that facilitate our research. First, CAPS users make specific predictions about the future price of a particular stock. In contrast, analysis of message board postings requires a systematic language-extraction algorithm to classify each message imperfectly as (say) “Buy/Sell/Hold”. Second, CAPS is designed to promote the reputations of its participants. Each player is rated based on the performance of previous picks, and each player’s past history of picks and performance can be viewed by others. Third, CAPS synthesizes the history of past picks to produce a rating for each stock (from the worst rating of “One Star” to the best rating of “Five Stars”) that provides a single prediction for each stock at each point in time. A final set of related papers concerns online prediction markets such as Intrade. These websites host competitive prediction markets for trade of shares that will pay off if a particular event occurs (e.g. Rick Santorum receives the Republican presidential nomination). Recent work such as Wolfers and Zitzewitz (2004) and Wolfers (2004) examine the functioning of markets. In many ways, CAPS can be thought of as a hybrid of a message board and a prediction market. First, like a prediction market but unlike a message board, CAPS users make very specific predictions. In the case of CAPs, these predictions are always about the future price of a particular stock. In contrast, analysis of message board postings requires a systematic language-extraction algorithm to classify each message imperfectly as (say) “Buy/Sell/Hold”. Second, CAPS synthesizes the history of past picks to produce a rating for each stock (from the worst rating of ”One Star” to the best rating of ”Five Stars”. Prediction markets aggregate predictions by displaying a market price that clears the market. Message boards do not generally attempt to aggregate predictions. Third, message boards do not generally attempt to incentivize participants to produce high quality predictions. Prediction markets are automatically incentivized because participants receive financial payouts if their predictions are correct. In contrast, incentives in Motley Fool exist, but are less explicit. Participants have no direct financial incentives to participate. Instead, participants receive reputation scores and the best players with the best reputation scores are highlighted on the site. The paper proceeds as follows. Section 2 describes the data. Section 3 provides descriptive statistics based only on absolute returns. Section 4 analyzes portfolio returns using a Four-Factor decomposition. Section 5 concludes.

2 DATA

7

2

Data

The data for this study was provided by the Motley Fool company under a license agreement with Harvard University. The data contains all stock market picks from the CAPS website from November 1, 2006 through December 31, 2008. The Motley Fool compiles information on each participant’s picks, and uses this information to rate both players and stocks. CAPS only allows picks for stocks that have a price of at least $1.50 per share and a market cap of at least $100 Million at any given time. CAPS allows participants to make predictions about the future movements of individual stocks from their current prices. Many websites, including CAPS, Amazon.com, and eBay use some kind of reputation system to measure past performance of website participants.2 . CAPS assigns a rating to each participant as a function of the objective performance of the stocks that she/he picked. Similar to Amazon.com’s “top reviewers” program (but in contrast to eBay), it is not clear what material benefit a participant gains from garnering a positive reputation on CAPS, though CAPS includes a number of features that promote interest in participant reputation. Each participant’s player rating is publicly available and represents that participant’s current percentile ranking (from 0 to 100) based on the market performance of past picks.3 Participants with ratings of 80 or above, representing those in the top fifth of player ratings, are labelled as “CAPS All Stars” and their picks are highlighted throughout the website. The data provided by Motley Fool for the study includes 2,704,719 distinct picks encompassing 7,287 different stocks and 113,154 different usernames.4 As part of the data for the study, Motley Fool also provided the daily CAPS rankings of these stocks from “1-Star” (the worst) to“5-Stars” (the best) for this one-year period. The CAPS website states that these stock rankings are based on a proprietary algorithm, aggregating individual player positive and negative picks using a methodology that gives higher
On eBay, for example, participants are invited to rate individual transactions with others as buyers and sellers (although they receive no direct rewards for doing so). The resulting scores for participants (number of rankings and percent positive) have been shown to be of sufficient credibility to participants that the sellers ratings have been show to affect both the price and the probability that a listed good is sold on eBay (Resnick and Zeckhauser, 2002; Resnick, Zeckhauser, Swanson, and Lockwood, 2006; Cabral and Hortacsu, 2010. 3 Participants with fewer than seven CAPS picks are not given player ratings. 4 An individual person can register more than once on the CAPS website and make picks using multiple “player names”. The exact number of distinct participants who have made picks on the CAPS website is unknown.
2

2 DATA

8

weight to players with higher rankings (i.e. better past performance) and to more recent picks. 5 We note that the CAPS methodology contrasts with that used in other stock market-oriented crowdsourcing websites. For example, Piqqem.com weights all participants opinions equally to form overall predictions, consistent with their stated philosophy that “every individual has some unique knowledge that is relevant to at least some set of stocks”. Each star rating corresponds to a quintile of stocks based on an underlying (unreported) CAPS cardinal rating for those stocks. That is, “1 Star” stocks consist of those stocks at the 20th percentile or below in cardinal ranking, whereas “5 Star” stocks consist of those stocks at the 80th percentile or above in cardinal ranking based on past CAPS picks. We were given no information about the proprietary system used to generate these rankings and made no attempt to identify its properties. We compiled stock price data from the Center for Research and Security Prices (CRSP) for November 1, 2006 (the official launch date for CAPS) to June 30, 2009 (six months after the end of our sample of CAPS stock picks). Since CRSP and CAPS use different identification numbers for stocks, we matched stocks across the two databases by ticker symbol and by name. We were able to match 6,429 stocks using this methodology. We used Yahoo Finance for historical pricing data for 385 stocks that do not appear in CRSP but that received at least ten CAPS picks during at least one of the five trading periods identified below in Table 1.6 Our matched database of stock picks from CAPS and stock prices from CRSP/Yahoo includes incorporates 2,695,044 CAPS picks, comprising 99.6% of the picks in the original data set. The remaining unmatched picks are primarily for small and often de-listed stocks. We will present results for stocks in different size classes; we classify companies with market caps of more than $5 Billion as “Large Caps”, companies with market caps between $1 and $5 Billion as “Medium Caps” and companies with market caps less than $1 Billion as “Small Caps”.7
See http://caps.fool.com/help.aspx for publicly available information on the CAPS rating algorithm. 6 One distinction between the CRSP data and the Yahoo Finance data is that stock prices listed in Yahoo Finance have already been adjusted to account for dividends and splits. We used a standard method to adjust the CRSP prices for dividends and splits, essentially assuming that dividend distributions are reinvested in the given stock. We made one adjustment to the Yahoo data, assigning the closing price on a day where a stock was not traded to be the opening price on the next that the stock is traded. By contrast, Yahoo uses a default procedure of listing the closing price to equal the previous day’s closing price on a day that a stock is not traded. 7 CRSP provides the number of shares outstanding for each stock, so we compute
5

2 DATA

9

200000 175000 150000

Number of Picks

125000 100000 75000 50000 25000 0

Month

Figure 1: Total CAPS Picks by Month Figure 1 graphs the number of picks submitted to the CAPS website for each month during the sample period. CAPS enjoyed a steady increase in popularity over time from its launch through the middle of 2008, but the number of picks submitted per month fell in the last half of 2008. This pattern suggests that some users may have lost interest in participating in CAPS over the course of the financial crisis and coincident downturn in stock prices in late 2008.

2.1

Trading Periods in the Sample

This study focuses attention on stock returns in five separate six month trading periods: (1) January 1 to June 30, 2007; (2) July 1 to December 31, 2007; (3) January 1 to June 30, 2008; (2) July 1 to December 31, 2008; (5) January 1 to June 30, 2009. We assess the predictive power of CAPS ratings
the market cap for stocks listed in CRSP for each trading day in the sample by simply multiplying this number of shares by (unadjusted) closing price. We were unable to find similar information in Yahoo Finance for most of the 170 stocks where we used Yahoo to compile historical price information. Therefore, we estimated the historical market caps for this subset of stocks based on the current market caps listed for these companies on the CAPS website as of February, 2009 and February, 2010.

3 DESCRIPTIVE STATISTICS FOR CAPS PICKS

10

Table 1: Sample Periods for CAPS Picks and Stock Market Returns
Trading Period 1 2 3 4 5 Stock Market Trading Days Jan 1 2007 to June 30 2007 July 1 2007 to Dec 31 2007 Jan 1 2008 to June 30 2008 July 1 2008 to Dec 31 2008 Jan 1 2009 to June 30 2009 Dates for Relevant CAPS Picks Nov 1 2006 to Dec 31 2006 Jan 1 2007 to June 30 2007 July 1 2007 to Dec 31 2007 Jan 1 2008 to June 30 2008 July 1 2008 to Dec 31 2008

in each trading period as a function of the CAPS rating for each stock at the beginning of that period. Similarly, we assess the predictive power of individual CAPS picks as a function of the performance of stocks in the subsequent trading period after each pick was submitted. Since CAPS was launched in November 2006, Trading Period 1 is associated with only two months of prior CAPS picks; every other Trading Period is associated with six months of prior CAPS picks. Table 1 summarizes this information. Table 2 presents summary statistics for the number of picks submitted to the CAPS website. On average, CAPS participants, like most stock market analysts, were relatively bullish, producing a ratio of about five positive picks per negative pick. Further, this ratio remained fairly constant over time— suggesting that CAPS users were not particularly successful at timing the market or anticipating the onset of the financial crisis. CAPS participants were significantly more likely to submit negative picks for Small Cap stocks than for Medium or Large Cap stocks. For the analysis in this study, we will use different aggregated measures of player picks in order to assess the “crowd wisdom.” We first use Motley Fool’s published rankings of individual stocks. We also use measures of the extent to which stocks have been picked as positive picks and the extent to which stocks have been picked as negative picks.

3

Descriptive Statistics for CAPS Picks and Subsequent Six-Month Returns

In this section, we take a preliminary look at the relationship between individual picks in the CAPS system and subsequent stock market returns. We start by examining returns where the unit of observation is the individual pick. Table 3 compares the six-month returns for positive and negative picks submitted to the CAPS website. To standardize the comparisons across cat-

3 DESCRIPTIVE STATISTICS FOR CAPS PICKS

11

Table 2: Positive and Negative Picks Submitted to the CAPS Website
Number of Picks 1. Whole Sample 2. Large Cap ($above $5B) 3. Medium Cap ($1B–$5B) 4. Small Cap ($100M–$1B) 5. 6. 7. 8. 9. Trading Trading Trading Trading Trading Period Period Period Period Period 1 2 3 4 5 2,684,733∗ 985,632 678,792 860,874 157,696 486,268 612,778 834,053 593,938 Percentage of Picks 100% 39.1% 26.8% 34.2% 5.9% 18.1% 22.8% 31.1% 22.1% Percent Positive Picks 80.8% 86.2% 82.6% 77.3% 84.9% 82.3% 80.0% 81.9% 78.6%

∗ Tabulations

in row 1 include picks for stocks with market caps too small to be included in analysis. egories in this table, we compute six month returns for each stock in each of our five trading periods based on the periods listed in Table 1. That is, we compute the returns associated with each pick based on the six-month returns for that stock in the next trading period—so a pick submitted in August 2007 is associated with stock market returns in Trading Period 3 beginning in January 2008. The overall return reported below, then, is the average return of all positive picks during the relevant time period and all negative picks over the relevant time period.8 Since we impose an artificial lag between the submission of most picks and the calculation of market returns associated with that pick, the summary results in Table 3 likely underestimate the information contained in each pick. Several systematic patterns stand out in Table 3. First, the average sixmonth return in most periods was negative in most cases. Trading Periods 1 and 5 produced nominally positive returns, but these were more than offset by the huge negative returns in Trading Period 4. More interestingly, Positive Picks systematically outperformed Negative
8 Thus, consider the example where only two stocks existed in the universe. Assume Stock A was the subject of 5 negative picks and 10 positive picks in a trading period and Stock B was the subject of 2 negative picks and 20 positive picks. The “return on negative picks” reported above is then the weighted average of Stock A and Stock B’s returns in the trading period, with the weights being 5/7 and 2/7, respectively. The “return on positive picks” reported above is the weighted average of Stock A and Stock B’s returns in the trading period, with the weights being 1/3 and 2/3, respectively.

3 DESCRIPTIVE STATISTICS FOR CAPS PICKS

12

Table 3: Returns for Positive and Negative CAPS Picks Return on Positive Picks Whole Sample Large Cap Medium Cap Small Cap Trading Trading Trading Trading Trading Period Period Period Period Period 1 2 3 4 5 -11.8% -11.6% -11.0% -12.5% 9.8% -1.6% -11.6% -40.2% 16.7% Return on Negative Picks -16.0% -16.2% -13.3% -17.5% 6.5% -10.1% -23.0% -37.1% 10.9% Return on Equal-Weight Mkt Index -11.2% -12.5% -11.1% -10.9% 7.1% -10.9% -13.7% -30.8% 15.9% Return on Value-Weight Mkt Index -10.7% -11.2% -10.6% -10.3% 6.9% -0.1% -11.8% -28.1% 3.2%

Picks, with simple differences of at least 2.5% in average returns in every period except for Trading Period 4. These differences are strikingly large in magnitude— across all five trading periods, a pick-weighted portfolio (equal investment for each pick) based on buying positive picks produced a sixmonth return of −11% whereas a pick-weighted portfolio based on buying negative picks produced a six-month return of −15.4%. Since each trading period is six months, this difference of 4.4 percentage points translates into an enormous difference of approximately nine percentage points in returns per year. Furthermore, Table 3 shows that the relationship between returns for Positive vs. Negative Picks varies very little by market cap. In particular, Positive Picks outperformed Negative Picks by about five percentage points in six-month returns for both Large and Small Cap stocks, though by approximately half as much for Medium Cap Stocks. The positive and negative picks submitted by the individual players in CAPS form the basis for the ranking assigned by Motley Fool to each stock— 1 star to 5 stars. Figure 2 graphs simple average returns for stocks with each Motley Fool ranking for each of our five six month trading periods. In each case, we classify stocks by their Motley Fool (quintile) rankings from the day before the start of the trading period. The return for a given stock in each period is simply the percentage change in stock price from the beginning

3 DESCRIPTIVE STATISTICS FOR CAPS PICKS

13

to the end of that trading period. So, for example, stocks with a 1-star rating on December 31, 2006 had an average return of 3.7% in the first trading period (Jan 2007 to June 2007), while stocks with a 5-star rating on December 31, 2006 had an average return of 13.7% during that same trading period. This difference of 11 percentage points in returns translates into a difference of (approximately) 22 percentage points in annualized returns.
30%

20% 4 5

3

10%
1 0%

2

1 Star
-10%

2 Stars
-20%

3 Stars 4 Stars

-30%

5 Stars

-40%

Jan 07 - June 07

Jul 07 - Dec 07

Jan 08 - June 08

Jul 08 - Dec 08

Jan 09 - Dec 09

Figure 2: Six Month Return by Motley Fool Rating Consistent with the patterns in Table 3, there is a clear monotonic relationship between Motley Fool rating and average stock returns in each of the first three trading periods. Averaging across these three periods, 5-star stocks produced returns 14.6 percentage points higher than 1-star stocks in six-month returns (29.2 percent annualized). The major reversal occurred in period 4, during the height of the financial crisis. During this period, 1 star stocks performed best and 5 star stocks performed worst. Averaging across the five half-year periods, 5 star stocks outperformed 1 star stocks by 9 percentage points.

3 DESCRIPTIVE STATISTICS FOR CAPS PICKS

14

3.1

Alternate Rankings Based on Positive and Negative Picks

We also used the data for individual picks to compute our own stock rankings in order to separately assess the predictive power of positive and negative CAPS picks. For each trading period, we tabulated the number of active positive picks and the number of active negative picks for each stock submitted to CAPS in the six months just prior to the start of that trading period.9 We then grouped stocks by market cap and classified each stock with at least ten active picks from the previous six months into quintiles. One classification was based entirely on the number of positive picks that the stock received; the second classification was based solely on the number of negative picks that the stock received. For our Positive Pick Ranking, we assigned a ranking of 5 to the top quintile of stocks and a ranking of 1 to the bottom quintile of stocks based on a simple count of active positive picks within each subgroup of stocks by market cap. That is, the stocks with Positive Pick Ranking of 5 in any trading period were those stocks that were in the top 20% among “Large Cap”, “Medium Cap” or “Small Cap” stocks in the count of active positive picks on the first trading day of the period.10 Similarly, for our Negative Pick Ranking, we assigned a ranking of 1 to the least frequently picked quintile of stocks and a ranking of 5 to the most frequently picked quintile of stocks based on a simple count of active negative picks within each subgroup of stocks by market cap. That is, we code both our Negative Pick Ranking and our Positive Pick Ranking so that the ranking of 5 in each case corresponds to the group of stocks about which players are most optimistic. Clearly, the Motley Fool ranking is some aggregation of Negative Picks and Positive Picks. However, depending on the pattern of picks submitted by different players, in principle, a stock’s ranking in the Negative Pick Ranking and a stock’s ranking in the Positive Pick Ranking may be positively correlated, negatively correlated or uncorrelated. Figure 3 provides a summary of the positive pick portfolio, constructed
CAPS participants have the opportunity to “close” a pick at any time, “locking in” the market returns associated with that pick. Once a pick is closed, subsequent changes in prices associated with that pick have no effect on the ranking of the participant who submitted that pick. For the purposes of creating “Positive Pick” and “Negative Pick” rankings in each Trading Period, we exclude picks that were both submitted and closed by a participant in the previous six month period. 10 We first divide stocks into groups by Market Cap and then order them by number of positive picks because there is an empirical positive relationship between cap size and number of picks (both positive and negative) submitted to CAPS.
9

3 DESCRIPTIVE STATISTICS FOR CAPS PICKS

15

identically to Figure 2, except that the returns are shown for unweighted Positive Pick Portfolio quintiles. Figure 4 provides a summary of the Negative Pick Portfolio.
30%

20%

10%

0%

1

2

3

4

5 1 Star

-10%

2 Stars 3 Stars 4 Stars 5 Stars

-20%

-30%

-40%

Jan 07 - June 07

Jul 07 - Dec 07

Jan 08 - June 08

Jul 08 - Dec 08

Jan 09 - Dec 09

Figure 3: Six Month Return by Positive Pick Rating The results for the Positive Pick Ratings are similar to those for the overall Motley Fool rating, but smaller. The 5 star portfolio outperforms the 1 star portfolio by an average of 2 percentage points across the five sixmonth periods (with dramatic underperformance in the fourth period). The Negative Pick Ratings 5 star portfolio outperforms the 1 star portfolio by 8 percentage points over the time period. The 5 star Negative Pick Portfolio outperforms the 1 star portfolio in every one of the 5 periods, even during the Period 4 financial crisis. Interestingly, Motley Fool stockpickers make Negative Picks much less frequently than Positive Picks, yet these results suggest that Negative Picks are, on average, more reliable during our sample period.

4 PERFORMANCE DECOMPOSITION

16

30%

20%

10%

0%

1

2 3

4

5

Add 1 Star 2 Stars

-10%

-20%

3 Stars 4 Stars

-30%

5 Stars

-40%

Jan 07 - June 07

Jul 07 - Dec 07

Jan 08 - June 08

Jul 08 - Dec 08

Jan 09 - Dec 09

Figure 4: Six Month Return by Negative Pick Rating

4

Performance Decomposition

We have demonstrated that the stocks ranked highly by Motley Fool participants earn higher subsequent raw returns than the stocks ranked poorly by Motley Fool participants during our sample period. The portfolios of stocks favored by Motley Fool participants may have different returns from the portfolios of stocks disfavored by Motley Fool participants because they have different risk or style factors on average, because they have time-varying differences in risk or style factors, or the differences may be independent of risk or style factors. Thus, we decompose differences that can be attributed to market or style factors, and differences that cannot be attributed to market or style factors. We use the classic style/risk factors identified by Fama and French (1996) and Carhart (1997) to decompose performance. Fama and French (1996) identified three measures that have been demonstrated to predict future stock returns. The first of these factors is the “Market Return” less the risk free rate, RMRF, which can be used to control for correlation between the returns in a portfolio of individual stocks and the returns on the market

4 PERFORMANCE DECOMPOSITION

17

portfolio. (For example, this correlation would be high for a portfolio of high-beta stocks, indicating a high-risk portfolio.) The second factor is a value/growth factor, HML, which can be used to control for the composition of a portfolio of stocks in terms of book value relative to market value. The third factor is a small stock factor, SMB, which controls for composition of a portfolio of stocks in terms of market cap value. We use the three factors identified by Fama and French in our analysis along with a fourth factor, Momentum, Mom, identified by Carhart (1997). The Momentum factor can be used to control for the composition of a portfolio of stocks in terms of previous year’s stock market performance. We observe Motley Fool participants in the aggregate over 5 quarters. We separately examine the Motley Fool CAPS-ranked quintile portfolios, the Positive Pick quintile portfolios and the Negative Pick quintile portfolios. For each of these portfolio types, we focus on the investment strategy of buying the 5th quintile portfolio (highest, or 5 star) and shorting the 1st quintile portfolio (lowest, or 1 star). We compute the one-day returns for each of the 5 rating portfolios for each trading day from January 1, 2007 to June 30, 2009. There are 628 trading days under study. Our decomposition strategy isolates the unexplained alpha after controlling for various factors. To understand the decomposition strategy, consider the mean daily raw return net of the risk-free rate: αA =
t (rij

− rf t )

T

allof

(A)

Where T is the number of trading days. Of course, since the risk free rate is fairly stable, the behavior of αA is very similar to the raw returns examined above. We also consider the constant term from two regressions: rij − rf t = αB + RMRFt β1 + SMBt β2 + HMLt β3 + Momt β4 + eit (B) rij − rf t = αC + (RMRFt xIt )B1 + (SMBt xIt )B2 + (HMLt xIt )B3 + (Momt xIt )B4 + eit (C)

Where RMRF, SMB, HML and Mom are the four as described above and It is a vector of indicator variables for each of the 5 trading periods. Here αB is the return net of loadings on the Fama French factors and momentum factor and αC is the return net of the four factors when the weights on the four factors have been allowed to vary by trading period. Following Wermers (2000), we can decompose the average return net of the risk free rate in (A) into three components of interest. First, αC is

4 PERFORMANCE DECOMPOSITION

18

the return net of time-varying four factors. This is the return that cannot be explained at all by market timing or style factors and thus, this is the “stock-picking” return earned by the strategy. Next, αB − αC is the “market-timing” return. If Motley Fool participants systematically favored stocks that covaried with risk or style portfolios that were going to earn high raw returns, but picked average-returning stocks within each style factor, we could observe positive market timing returns but zero “stock-picking” return. Finally, αA − αB is the portion of the return attributable to the average style or risk weightings of Motley Fool participants. Thus, for example, if Motley Fool participants chose stocks that had a high correlation with the returns of small cap stocks minus large cap stocks, and the SMB portfolio had significant returns relative to the market over our entire sample period, then a significant part of the return would be attributable to style factors. Note then, that this decomposition is complete: the overall return αA equals the “stock-picking” return (αC ) plus the “market timing” return (αB − αC ) plus the “average style” return (αA − αB ). The results for the estimated αs in specifications (A), (B), and (C) and the resulting decompositions are shown in Table 4. We show the results using the Motley Fool CAPS-ranked quintile portfolios, the Positive Pick quintile portfolios, and the Negative Pick quintile portfolios. The results are quite striking. Both the Negative Pick quintile portfolio strategy and the MF quintile portfolio strategy earn significantly positive returns, on average, net of (constant) Fama French factors, as shown in (B). The Positive Pick quintile portfolio strategy fails to earn significantly positive returns. Indeed, for all three portfolios, controlling for the average style or risk factors has little impact on the magnitude of the estimated excess returns. When allowing for time-varying Fama French and momentum factors, however, we see that only the Negative Pick quintile portfolio continues to show significantly positive alpha. The decomposition suggests that the Negative Pick quintile portfolio strategy has positive and statistically significant returns attributable to stock picking and small negative returns associated with average style and market timing. In contrast, the Positive Pick quintile portfolio has strongly negative return attributable to stock picking and positive returns attributable to market timing which nets out to almost zero. Unsurprisingly, since the Motley Fool portfolio must be some combination of the positive and negative picks, it nets out to small positive stock picking and market timing factors. We also illustrate how the estimated time-varying 4-factor alphas evolve over time for all five of the quintile portfolios in the following figures. In

4 PERFORMANCE DECOMPOSITION

19

Table 4: Return Decomposition.
MF (A) (B) (C) overall controlling for 4 factors controlling for time-varying four factors Performance Decomposition return attributable to stock pricing (C) return attributable to market-timing (B)–(C) return attributable to style (A)–(B) 0.066 (1.42) 0.062 (2.16) 0.029 (1.22) 0.029 0.033 Positive 0.012 (0.35) 0.009 (0.31) −0.055 (2.23) −0.055 0.064 Negative 0.056 (1.63) 0.057 (2.84) 0.069 (3.86) 0.069 −0.012

0.004

0.003

−0.001

Table 4: Regressions using daily returns from buying the 5th quintile portfolio and shorting the 1st quintile portfolio in each category (in percents—e.g. ×100). For the both the Negative Pick Portfolio and the Positive Pick portfolio, the 5th quintile is defined as the set of stocks about which CAPS participants are most optimistic. Results in (A), (B), and (C) show constant term (α) from regressions. t-statistics are in parentheses.

4 PERFORMANCE DECOMPOSITION

20

each case, we estimated the Alpha coefficient for each portfolio relative to Portfolio 1 for each Trading Period.
0.2

0.1

0

2

3

4

5

2 Stars 3 Stars 4 Stars

-0.1

-0.2

5 Stars

-0.3

-0.4

Jan 07 - June 07

Jul 07 - Dec 07

Jan 08 - June 08

Jul 08 - Dec 08

Jan 09 - Dec 09

Figure 5: Estimated Alpha After Four-Factor Filtering: Motley Fool Ranking. The figure shows alpha for each star quintile portfolio minus the alpha of the 1 star quintile portfolio. Consistent with the results above, the estimated alphas in Figure 5 are on net positive for all four of the quintile portfolios relative to the first quintile portfolio. Furthermore, there is a systematically monotonic pattern for the estimated alphas in each of the first two trading periods. That is, within each portfolio of stocks with a given ranking higher rated stocks performed better than each portfolio of stocks with a lower ranking during each of the first three periods period after four-factor filtering. However, four-factor filtering produces qualitatively different results from the raw returns data for the last two trading periods. In each of those periods, 5-star stocks perform worse than stocks of all other rankings and have negative estimated alpha values. The positive estimated alphas for 5-star stocks is statistically significant at the 5% level in each of the first two periods, but neither of the negative estimated alphas for 5-star stocks for the last two periods is

4 PERFORMANCE DECOMPOSITION

21

statistically significant.11 Figures 6 and 7 present Estimated alphas for each period for our alternative Positive Pick and Negative Pick Rankings quintile portfolios. Comparing Figures 5 and 6, rankings based on Positive Picks have estimated alpha values that are systematically more negative than the estimated alpha values for the overall CAPS ratings. The estimated alphas for 5-star stocks based on Positive Picks are negative in every period and are both large in magnitude and statistically significant in trading periods 4 and 5, suggesting that 5-star stocks based on Positive Picks performed significantly worse than 1-star stocks based on Positive Picks in these periods after four-factor filtering.
0.1 0.05 0 2 -0.05 -0.1 -0.15 -0.2 -0.25 -0.3 -0.35 -0.4 3 4 5

2 Stars 3 Stars 4 Stars 5 Stars

Jan 07 - June 07

Jul 07 - Dec 07

Jan 08 - June 08

Jul 08 - Dec 08

Jan 09 - Dec 09

Figure 6: Estimated Alpha After Four-Factor Filtering: Positive Pick Ranking. The figure shows alpha for each star quintile portfolio minus the alpha of the 1 star quintile portfolio.
The estimated negative alpha in trading period 4 is of the same magnitude as the estimated positive alphas for the previous three periods, but interestingly, the standard error on this estimated value is about twice as large as the standard errors in previous periods—consistent with the observation of unusual variations in daily movements in stock prices during the last half of 2008.
11

4 PERFORMANCE DECOMPOSITION
0.2

22

0.1

0

2

3

4

5

-0.1

2 Stars 3 Stars 4 Stars

-0.2

-0.3

5 Stars

-0.4

Jan 07 - June 07

Jul 07 - Dec 07

Jan 08 - June 08

Jul 08 - Dec 08

Jan 09 - Dec 09

Figure 7: Estimated Alpha After Four-Factor Filtering: Negative Pick Ranking, The figure shows alpha for each star quintile portfolio minus the alpha of the 1 star quintile portfolio. By contrast, comparing Figures 5 and 7, rankings based on Negative Picks have estimated alpha values that are systematically more positive than those for the overall CAPS ratings. The estimated alphas for Portfolio 5 (once again, these estimated alphas are relative to Portfolio 1) are positive in every period and are statistically significantly different than 0 in every period except for trading period 3.

4.1

Robustness

An immediate concern with our analysis is that the exceptional performance of the Negative Pick portfolio may derive from thinly traded Small Cap stocks. Recall that, at the time of our study, CAPS only allowed picks for stocks that have a price of at least $1.50 per share and a market cap of at least $100 Million at any given time. Furthermore, we construct our pick rankings by subdividing twenty percent of each of the Small Cap Stocks, the Medium Cap Stocks, and the Large Cap stocks into each pick ranking. Thus, by construction, there are as many Small Cap stocks in the 5th quintile of

4 PERFORMANCE DECOMPOSITION

23

the Negative Pick Portfolio as there are in the 1st quintile. Nonetheless, we undertake a robustness check by examining the performance of the Motley Fool ratings, our Positive Pick Portfolio ratings, and our Negative Pick Portfolio ratings by market capitalization grouping. Table 5 reports timevarying Four Factor alphas, as in Figures 4 through 6 for each Star category for each capitalization grouping. Table 5: Estimated alphas from Four Factor Model for Quintile Portfolios by Market Capitalization. 1 2 Star 3 Star 4 Star 5 Star Ranking Cap Size N Obs R-Sq 0.047 (0.027) 0.047 (0.027) 0.052 (0.027) 0.041 (0.027) MF Large 628 0.96 2 -0.035 (0.022) -0.019 (0.022) -0.009 (0.022) -0.022 (0.022) MF Medium 628 0.97 3 0.04 (0.021) 0.037 (0.021) 0.031 (0.021) 0.04 (0.021) MF Small 628 0.97 4 0.022 (0.018) 0.024 (0.018) 0.01 (0.018) -0.004 (0.018) Pos Large 628 0.98 5 -0.014 (0.022) -0.003 (0.022) -0.038 (0.022) -0.069 (0.022) Pos Medium 628 0.97 6 -0.001 (0.022) -0.005 (0.022) -0.009 (0.022) -0.064 (0.022) Pos Small 628 0.97 7 0.02 (0.018) 0.022 (0.018) 0.034 (0.018) 0.029 (0.018) Neg Large 628 0.98 8 0.026 (0.018) 0.04 (0.018) 0.061 (0.018) 0.052 (0.018) Neg Med 628 0.98 9 0.061 (0.02) 0.077 (0.02) 0.084 (0.02) 0.092 (0.02) Neg Small 628 0.98

We also consider one possible concern about the way that pick ratings are calculated. We produce Positive and Negative Pick ratings for all stocks that have at least ten active picks from the past six months at the start of each Trading Period. This creates the following measurement issue: if there are two stocks with no active negative picks from the past six months, stock 1 would be assigned a Negative Pick rating if it had ten active positive picks, but not if it only had eight active positive picks from the past six months at the start of a trading period. To assess this possible contamination between the Positive and Negative Pick ratings, we varied the requirement of 10 active picks to 20 and 50 active picks from the past six months to be included in the rankings. Table 6 lists the number of stocks that receive Positive and Negative pick ratings in each period with these varying requirements. Increasing the requirement from 10 to 50 Active Picks reduces the number of rated stocks by about half in most

4 PERFORMANCE DECOMPOSITION

24

Table 6: Number of Rated Stocks as Function of Active Picks Required for Rating
Active Picks Required for Rating 10 20 50 Period 1 Period 2 Period 3 Period 4 Period 5

2,413 1,512 633

4,046 3,090 1,623

4,177 3,292 1,949

4,138 3,344 2,104

3,284 2.507 1,431

periods. Table 7 compares the estimated alpha values for three different rankings— Motley Fool, Positive Pick and Negative Pick ratings—as we vary the number of active picks required for a stock to be rated. The estimated alphas for each ranking scheme change relatively little with these variations in the required number of active picks. With the most rigorous requirement of 50 active picks for a stock to be rated, the estimated alpha for the 5-Star Positive Pick ratings remains negative but loses statistical significance. But our more important finding—that the Negative Pick ratings have positive and significant estimated alphas, continues to hold with these more rigorous ranking rules.

Conclusion
Our investigation of the Motley Fool CAPS system shows that CAPS predictions are surprisingly informative about future stock prices. In particular, while we don’t find much predictive capability of positive stock picks, we do find that negative picks do predict future stock price declines. Our Fama French decomposition suggests that these are results are due to stock picking rather than style factors or market timing. It may not be surprising that the collaborative filtering technology is more successful at predicting abnormally negative future stock performance than it is at predicting abnormally positive future stock performance. The literature surrounding short sales (for example, Jones and Lamont, 2002 and Boehme, Danielsen, and Sorescu, 2006), suggests that acting on negative information about the prospects for a stock can be more costly and difficult than acting on positive information about the prospects for a stock. These papers find support for the hypothesis of Miller (1977) that dispersion of investor opinion in the presence of short-sale constraints may lead to stock

4 PERFORMANCE DECOMPOSITION

25

Table 7: Estimated Alphas for Pick Rankings with Varying Numbers of Picks Required for Ranking Using Time-Varying Four Factor Alphas
(1) 2-Star Stock 3-Star Stock 4-Star Stock 5-Star Stock Ranking System Active Picks for Rating N obs R-squared .025 (.017) .030 (.017) .031 (.017) .029 (.017) MF 10 628 0.98 (2) .040 (.019) .050 (.019) .052 (.019) .053 (.019) MF 20 628 0.98 (3) .040 (.021) .059 (.021) .060 (.021) .066 (.021) MF 50 628 0.97 (4) −.000 (.017) .002 (.017) −.013 (.017) −.055 (.017) Pos Pick 10 628 0.98 (5) .014 (.019) .016 (.019) −.009 (.019) −.045 (.019) Pos Pick 20 628 0.98 (6) .017 (.023) −.008 (.023) −.013 (.023) −.046 (.023) Pos Pick 50 628 0.97 (7) .044 (.015) .057 (.015) .069 (.015) .069 (.015) Neg Pick 10 628 .99 (8) .047 (.017) .057 (.017) .070 (.017) .087 (.017) Neg Pick 20 628 .98 (9) .054 (.023) .054 (.023) .065 (.023) .080 (.023) Neg Pick 50 628 .97

price overvaluation. This may be particularly true for smaller traders. Interesting questions remain about what factors lead the CAPS system to predict future stock returns. In future work, we hope to analyze the text written by participants as part of the pick submission process. More generally, we will examine whether there are ways to judge ex ante which participants picks have relatively more predictive power.

4 PERFORMANCE DECOMPOSITION

26

References
Antweiler, Werner and Murray Z. Frank, 2004, “Is All That Talk Just Noise? The Information Content of Internet Stock Message Boards,” Journal of Finance 59(3), 1259-1294. Antweiler, Werner and Murray Z. Frank, 2006, “Does Talk Matter? Evidence from a Broad Cross Section of Stocks,” working paper, University of Minnesota. Bagnoli, Mark, Messod Daniel and Susan G. Watts, 1999, “Whisper Forecasts of Quarterly Earnings per Share, Journal of Accounting and Economics 28(1), 27-50. Baker, Malcolm, Lubomir Litov, Jessica A. Wachter and Jeffrey Wurgler, 2010, “Can Mutual Fund Managers Pick Stocks? Evidence from their Trades Prior to Earnings Announcements”, Journal of Financial and Quantitative Analysis” 45, 1111-1131. Barber, Brad M., Terrance Odean,, and Ning Zhu, 2009, “Do retail trades move markets?”, Review of Financial Studies 22, 151- 186. Barber, Brad M. and Terrance Odean, 2001a, “Boys will be Boys: Gender, Overconfidence, and Common Stock Investment” Quarterly Journal of Economics 116(1), 261-292. Barber, Brad M. and Terrance Odean, 2001b, “The Internet and the Investor,” Journal of Economic Perspectives 15(1), 41-54. Barber, Brad M. and Terrance Odean, 2000, “Trading is Hazardous to Your Wealth: The Common Stock Investment Performance of Individual Investors” Journal of Finance 55(2), 773-806. Boehme, Rodney, Bartley R. Danielson, and Sorin M. Sorescu, 2006, “Short-sale constraints, differences of opinion, and overvaluation,” Journal of Financial and Quantitative Analysis 41,455-487. Cabral, Luis and Ali Hortacsu, 2010, “The Dynamics of Seller Reputation: Evidence from eBay,” The Journal of Industrial Economics 58 (1),

4 PERFORMANCE DECOMPOSITION 54-78.

27

Chevalier, Judith A. and Glenn Ellison, 1999, “Are Some Mutual Fund Managers Better than Others? Cross-Sectional Patterns in Behavior and Performance,” Journal of Finance 54(3), 875-899. Coval, Joshua, David Hirshleifer, and Tyler Shumway, 2005, “Can Individual Investors Beat the Market?” working paper, Harvard University. Das, Sanjiv R. and Mike Y. Chen, 2007, “Yahoo! for Amazon: Sentiment Extraction from Small Talk on the Web”, Management Science 53(9), 1375-1388. Frieder, Laura and Jonathan Zittrain, 2007, “Spam Works: Evidence from Stock Touts and Corresponding Market Activity,” Hastings Communications and Entertainment Law Journal 30, 479-520. Hogarth, Robin, 1977, “Methods for Aggregating Opinions,” in Helmut Jungerman and Gerald de Zeeuw (eds.), Decision Making and Change in Human Affairs, D. Reidel Publishing Co., Dordrecht, Holland, 231-255. Ivkovic, Zoran and Scott Weisbenner, 2005, “Local Does as Local Is: Information Content of the Geography of Individual Investors’ Common Stock Investments”, Journal of Finance 60(1), 267-306. Ivkovic, Zoran, Clemens Sialm, and Scott Weisbenner, 2008, “Portfolio Concentration and the Performance of Individual Investors”, Journal of Financial and Quantitative Analysis, 43(3), 613-656. Jones, Charles M. and Owen A. Lamont, 2002, “Short-sale constraints and stock returns”, Journal of Financial Economics 66, 207-239. Linstone, Harold A. and Murray Turoff (eds.), 1975, The Delphi Method: Techniques and Applications, Addison-Wesley, Reading, Ma. Metrick, Andrew, 1999, “Performance evaluation with transactions data: The stock selection of investment newsletters,” Journal of Finance 54, 17431775. Mikhail, Michael B., Beverly R. Walther, and Richard H. Willis, 2004, “Do security analysts exhibit persistent differences in stock picking ability?”,

4 PERFORMANCE DECOMPOSITION Journal of Financial Economics 74, 67-91.

28

Odean, Terrence, 1999, “Do investors trade too much?”, American Economic Review 89, 1279-1298. Pastor, Lubos, and Robert F. Stambaugh, 2002, “Investing in equity mutual funds”, Journal of Financial Economics 63, 351-380. Resnick, Paul and Richard Zeckhauser, 2002, “Trust Among Strangers in Internet Transactions: Empirical Analysis of eBay’s Reputation System,” in The Economics of the Internet and E-Commerce, M. R. Baye (ed.), Amsterdam: Elsevier Science, 127-157. Resnick, Paul, Richard Zeckhauser, John Swanson, and Kate Lockwood, 2006, “The value of reputation on eBay: A controlled experiment”, Experimental Economics 9, 79-101. Sabherwal, Sanjiv, Salil Sarkar and Ying Zhang, 2008, “Online Talk: Does It Matter?” Managerial Finance 34(6), 423-426. Sunstein, Cass, 2006, “Deliberation and Information Markets”, in Robert Hahn and Paul Tetlock (eds.), Information Markets: A New Way of Making Decisions, AEI Press, Washington, D.C., 67-99. Tetlock, Paul, 2007, ”Giving Content to Investor Sentiment: The Role of Media in the Stock Market,” Journal of Finance 62(3), 1139-1168. Wermers, Russell, 2000, “Mutual fund performanc: An empirical decomposition into stock-picking talent, style, transactions costs, and expenses,” Journal of Finance 55, 1655-1695. Tumarkin, Robert and Robert Whitelaw, 2001, “News or Noise? Internet Postings and Stock Prices?”, Financial Analysts Journal 57, 41-51. Wolfers, Justin and Eric Zitzewitz, 2004,“Prediction markets”, The Journal of Economic Perspectives 18, 107-126.