Recently, I have heard a lot about the disadvantages of frequentist statistics, including the complain about p value, which is a hot topic due to the God particle.
Professor Kruschke, J.K. gave a talk on Doing Bayesian Data Analysis @ Michigan State University on September. He mentioned a concept “Intention“, including intended hypothesis, intended experiments, intended sampling. Basically he explained lots of frequentist procedure for doing statistics are intended procedure, which is not science, since everything depends on people’s intention. If you want to know more about this, please refer to the paper.
Today I came across the following a blog post from Statistical Modeling, Causal Inference, and Social Science, which is also about the intention issue about frequentist statistics:
Sometimes the problem is that the frequentist criterion being used is not of applied relevance. Consider a simple problem such as estimating a proportion p, given y successes out of n trials, where n=100 and y=0. The best estimate of p will be different if I tell you that p is the probability of a rare disease, compared to if I tell you that p is the proportion of African Americans who plan to vote for Mitt Romney.
I do need some frequentist people to explain this intention issue, since I think it’s kind of reasonable questioning. Any comments?
Update:
The following cartoon caused a fight between Frequentist and Bayesian:
- A post from Andrew: I don’t like this cartoon
- A post from Normal Deviate: anti xkcd
And the following is really a point:
Suppose I had a medical test with a 1/6 false positive rate and a 0% false negative rate. That is, if administered to someone without the disease it has a 1/6 chance of reporting positive. The protocol is to administer the test and, if positive, to administer it again. Assuming independence, the probability of two consecutive false positives is 1/36. Some statisticians would reject the null hypothesis (that the patient is disease free) given 2/2 positive tests. That is ridiculous for the same reason the xkcd example is ridiculous (it ignores prior or base rate information) but is is indeed the practice in some circles, I’m told.—–Phil
Also refer to the explanation from Andrew:
In the context of probability mathematics, textbooks carefully explain that p(A|B) != p(B|A), and how a test with a low error rate can have a high rate of errors conditional on a positive finding, if the underlying rate of positives is low, but the textbooks typically confine this problem to the probability chapters and don’t explain its relevance to accept/reject decisions in statistical hypothesis testing.
Update: (Two videos from Professor Kruschke, J.K.)
Bayesian estimation supersedes the t test in 14 minutes of video.+ Bayesian Methods Interpret Data Better
Update:
Examples of Bayesian and frequentist approach giving different answers
3 comments
Comments feed for this article
November 4, 2012 at 11:28 am
John Kruschke
Thanks for attending the workshop at MSU and for posting a link to the article (see updates here: http://www.indiana.edu/~kruschke/BEST/ ).
There are various different issues alluded to in your post. First, the quote from Gelman’s blog is referring to the use of prior knowledge in estimation. If you get 0 heads in 100 flips, the frequentist best estimate of the probability of heads is zero, but the Bayesian estimate is a (posterior) distribution over all possible probabilities, with a mean near zero, but not at zero. How close to zero depends on the prior, which is influenced by whether it’s a disease known to be rare or a vote known to be near 50-50.
The Bayesian prior is different than the stopping intention that defines a frequentist sampling distribution. To determine a p value, the frequentist needs to specify the stopping intention for what it means to take a sample. The usual convention –and it’s only a convention– is to assume a threshold sample size. The Bayesian, on the other hand, doesn’t use p values as the criterion for making decisions, and therefore does not rely on specifying a stopping intention.
November 9, 2012 at 10:25 am
Matt Sigl (@mattsigl)
I have been trying to articulate for myself, in laymen’s terms, the difference between Frequentist and Bayesian models. Is it simply that Bayesians give all primes their own probability distribution while Frequentists use determinate values? Is there a simple, catchy way to understand the difference for a person not immersed in the minutiae of statistical analysis?
Maybe a little off-topic but, I can’t help but think that this debate relates to human epistemology in some deep, but hard-to-define way. Certainly all human beliefs end in a belief (or beliefs) that have the character of certainty (or assumed certainty), beliefs whose Bayesian probability is valued at 100%, like Descartes’ Cogito Ergo Sum. It’s the ultimate stopping intention. Perhaps it is here where Bayesian and Frequentist analysis meet “existentially,” as it were.
November 9, 2012 at 4:16 pm
Honglang Wang
Also refer to this: http://xkcd.com/1132/
which vividly explain the difference between frequentist and bayesian.