Thread: Math question for LD experiment

The method Xei outlined should be sufficient to integrate the unique probability for trial 3 with the other trials. The probability it yields corresponds to the ubiquitous p-value, or the probability of obtaining a result at least as extreme as that observed, assuming that the null hypothesis is true (in this case, assuming that you cannot reliably predict the dice color). It is common practice to reject the null hypothesis if this probability is lower than 5%.

Note that this is doing things the long way, which is necessary if you want to retain the observation from trial 3. Widely available software programs (including web-based ones) make it trivially easy to compute the final probability if the expected probability of success under the null hypothesis is constant. However, satisfying this assumption would require excluding trial 3. (I wouldn't recommend revising the observation so as to be either white or black.)

One more thing about the 5% rejection thing. Do not simply stop collecting data as soon as your p-value up to that point falls below .05 (assuming that it does). For example, if your very next attempt turns up a hit, your cumulative p value will probably be below .05 at that point. However, it is unlikely that a sample of only 4 trials is representative of the potential population from which it comes (check out this article; it's a classic). You should decide in advance how many trials you will run. A good rule of thumb is to have at least 30 -- although the more trials you have, the greater are your chances of observing an effect if there really is one.