In my countless client visits over the last 10 years, the most popular risk question has easily been some form of, “Really how different are the various risk model providers?” I have always responded by jumping into the differences between the models. Usually, this starts with named factor vs. principal component vs. hybrid models. From there, I delve into how the factors are defined. But, I don’t know that I ever really answered the question.

So, I want to share a pretty straightforward analysis to address the risk practitioner’s question, “How different are APT, Barra, and Northfield?” Over the next few weeks, I will share additional variations of our core question.

But first, let’s begin with a comparison of the tracking error.

Using FactSet’s LionShares database of mutual fund holdings, I’ll focus on 300 U.S. equity funds. The funds represent the 30 fund constituents of Lipper’s mutual fund indices for the nine standard style boxes and equity income because I wanted to use real portfolios and didn’t want to arbitrarily select the portfolios. I focus on tracking error (Active Risk in the language of Aegis) as of 12/31/2008 using the Russell style index as the benchmark for each style box and the S&P 500 as the benchmark for Equity Income.

I compare the tracking error of APT U.S. Long, Barra USE3L, and Northfield U.S. fundamental. First, I compare the average tracking error and then judge whether the difference is statistically significant by whether the Welch’s T-test is greater than two. While it is reasonable to debate whether this is the perfect theoretical test, I believe that this test best represents the perspective of an investment manager or plan sponsor.

For our purposes, we will refer to the three models as X, Y, and Z. My task isn’t to suggest which model is “best.” Frankly, my analysis doesn’t offer genuine insight on that question and including the names of model providers might mislead the reader to conclusions about the “best” concept.

Let’s review the results:

Looking at Table 1, Models X and Y appear similar. The overall difference is quite small, and we don’t see either model consistently larger. In fact, if anything, we would say that Model X suggests slightly higher tracking error for Large Cap and Mid Cap strategies while Model Y predicts slightly higher tracking error for Small Cap strategies. But, are the differences significant?

On the other hand, Model Z looks to suggest far lower tracking error than either of the other models. This is true across all fund categories.

Turning our attention to the statistical significance:

The test of statistical significance seems to really only bolster the intuitive conclusions drawn from reviewing the averages. The shaky observation that models X and Y might be similar in aggregate, but different for large, mid and small caps lacks support. We also see that the differences between model Z and either X or Y is virtually always significant.

So, how different are APT, Barra, and Northfield? This analysis suggests two of the three models are similar and the other predicts significantly lower tracking error. Though this is just US equity, the conclusions are quite consistent across size and style.

Coming soon: In my next entry, I will remove the benchmark to determine how (or if) our conclusions change when we change from Tracking Error/Active Risk to Absolute Risk/Portfolio Risk.

To receive future posts by e-mail, subscribe to this blog.