Significant Testing

When you are looking at money management techniques, you run into all sorts of statistical terms and concepts. One of these is the idea of significant testing, the question posed is what do you have to do to know that the conclusions you come to matter?

This idea is a fundamental one in the world of statistics. You may have heard about something being “statistically significant”, another way of saying that it is likely to matter beyond a reasonable chance. When pollsters tell you that their survey shows that “56% favour no increase in taxes, and 44% say that taxes should be reduced” there is always the “fine print” that says these numbers are +/- 4% say, and this shows the significance of their finding.

It is similar when you have a number of tests, each with a certain outcome, and you take the results and make conclusions. Say you are testing the idea that a share that trades at a higher price than one year ago will go up in the next week, and therefore would be a good bet. If you do one test, and it is true, then obviously you cannot claim that you have done significant testing and that you expect the principle to hold in future. You need to try out your idea on many more cases to get a reasonable picture of the truth of the notion.

Generally speaking, the more tests or results you analyze, the “better” the result, meaning it is more reliable or statistically significant. Statisticians will be able to tell you the likelihood that you would get this result by pure chance is less than a certain amount, making the outcome surer. A large test size is the key to achieving a significant result.

What does this mean with respect to money management techniques? If we are looking for significant testing and comparison of particular techniques, such as fixed ratio position sizing v. fixed fractional position sizing, then it is necessary to ensure that the results of the testing can be statistically determined to be reliable and beyond question. The general way to do so is to make sure that the largest sample size possible is being used, but then that requires more computer power and time.

To conduct this analysis, you must first determine a critical value that you will accept, the amount of certainty required. With this in mind, you can then run the data tests and check what the statistical significance is. This may have two outcomes – the critical value may be met by the data analysis, or the critical value may not be reached.

If the critical value is met, then by definition the result is significant, which means that the probability is very small that the outcome occurred by chance. In this case, you can go ahead, basing your money management on the end result. If the critical value is not reached, then the relationship is not proven by significant testing, and there is a reasonable chance that it occurred by accident.