In the eight years I have been discussing risk with our clients in the Asia Pacific region, the dominant question I continue to receive is typically phrased: “Are there really any differences in the risk model providers?” Historically this has led to a discussion on the differences in each of the risk model provider’s methodologies. Last year, my colleague Chris Ellis put together a series of blog entries (part 1, part 2, part 3) answering that exact question, which I have since pointed my clients to for answers to their queries. However, inevitably, a couple of days later, I hear back from them with a rebuttal along the lines of: “For the U.S. that makes sense, but what about Asia?”

So I will build off of my colleague’s previous analysis and see if the same results hold for the Asia Pacific region. My analysis is going to be considerably simpler than what Chris put together given the relative size of the market in the Asia Pacific region. Style indices are few and far between; i.e., there are not many Asia Pacific ex-Japan mid-cap growth funds to speak of. For the purposes of this blog entry, I will focus my attention on the three major markets in the region: Australia, Asia ex-Japan, and Japan, and I will perform the same analysis previously conducted using the Northfield, APT, and Axioma risk models. (At their request, Barra will no longer be included in comparisons of the different vendor risk models.)

Using the FactSet LionShares database, I retrieved fund constituents for the 30 largest funds in each market as of June 30, 2010. My initial focus is tracking error (active risk) using the MSCI All Country Asia, S&P ASX 300, and TOPIX as the benchmarks for the three respective markets (Asia ex-Japan, Australia, and Japan). For the Asia Pacific region, I used the respective global models for each of the three providers, and for the Japan and Australia analysis, I used the country-specific models for the analysis. I will refer to the three models simply as A, B, and C. The purpose of this analysis is not to suggest which model is the “best,” as this analysis is not attempting to answer that question and may mislead the reader to conclusions about the “best” concept. I am trying to assess only whether there are differences between the providers.

Let’s first take a look at the average tracking error:



It’s fairly easy to visually see that Model C is suggesting a lower tracking error in all three markets with Model B portending the higher tracking error for all three markets. Model A seems to sit squarely in the middle, although leaning more towards Model C, especially in Australia. But the key question is, are the differences we see here statistically significant?

I used the same statistical technique used in the previous blog entry: the Welch’s T-test. I will deem any value greater than two to be statistically significant.

This test further bolsters the conclusions we were able to see fairly easily from reviewing the average tracking errors. While there is a discernable difference between Models A and C, we can see the differences are not significant. However, the significance between Models B and C is consistent for all three regions.

This was a very simple exercise; I would most likely want to include more funds and a more robust set of portfolios and benchmarks before making any firm conclusions, but we can observe from this brief analysis that two of the three risk models I used in this exercise do appear similar while the third appears to suggest a consistently higher tracking error. This builds upon my colleague’s previous analysis that there are indeed differences between the different risk model providers.

Please share your thoughts in the comments.

Subscribe by e-mail to receive new Taking Risk posts as they are published and follow @FactSet on Twitter.