p-value and Bayes are the two hottest words in Statistics. Actually I still can not get why the debate between frequentist statistics and Bayesian statistics can last so long. What is the essence arguments behind it? (Any one can help me with this?) In my point of view, they are just two ways for solving practical problems. Frequentist people are using the random version of proof-by-contradiction argument (i.e. small p-value indicates less likeliness for the null hypothesis to be true), while Bayesian people are using learning argument to update their believes through data. Besides, mathematician are using partial differential equations (PDE) to model the real underlying process for the analysis. These are just different methodologies for dealing with practical problems. What’s the point for the long-last debate between frequentist statistics and Bayesian statistics then?
Although my current research area is mostly in frequentist statistics domain, I am becoming more and more Bayesian lover, since it’s so natural. When I was teaching introductory statistics courses for undergraduate students at Michigan State University, I divided the whole course into three parts: Exploratory Data Analysis (EDA) by using R software, Bayesian Reasoning and Frequentist Statistics. I found that at the end of the semester, the most impressive example in my students mind was the one from the second section (Bayesian Reasoning). That is the Monty Hall problem, which was mentioned in the article that just came out in the NYT. (Note that about the argument from Professor Andrew Gelman, please also check out the response from Professor Gelman.) “Mr. Hall, longtime host of the game show “Let’s Make a Deal,” hides a car behind one of three doors and a goat behind each of the other two. The contestant picks Door No. 1, but before opening it, Mr. Hall opens Door No. 2 to reveal a goat. Should the contestant stick with No. 1 or switch to No. 3, or does it matter?” And the Bayesian approach to this problem “would start with one-third odds that any given door hides the car, then update that knowledge with the new data: Door No. 2 had a goat. The odds that the contestant guessed right — that the car is behind No. 1 — remain one in three. Thus, the odds that she guessed wrong are two in three. And if she guessed wrong, the car must be behind Door No. 3. So she should indeed switch.” What a natural argument! Bayesian babies and Google untrained search for youtube cats (the methods of deep learning) are all excellent examples proving that Bayesian Statistics IS a remarkable way for solving problems.
What about the p-values? This random version of proof-by-contradiction argument is also a great way for solving problems from the fact that it have been helping solve so many problems from various scientific areas, especially in bio-world. Check out today’s post from Simply Statistics: “You think P-values are bad? I say show me the data,” and also the early one: On the scalability of statistical procedures: why the p-value bashers just don’t get it.
The classical p-value does exactly what it says. But it is a statement about what would happen if there were no true effect. That can’t tell you about your long-term probability of making a fool of yourself, simply because sometimes there really is an effect. You make a fool of yourself if you declare that you have discovered something, when all you are observing is random chance. From this point of view, what matters is the probability that, when you find that a result is “statistically significant”, there is actually a real effect. If you find a “significant” result when there is nothing but chance at play, your result is a false positive, and the chance of getting a false positive is often alarmingly high. This probability will be called “false discovery rate” (or error rate), which is different with the concept in the multiple comparison. One possible misinterpretation of p-value is regarding p-value as the false discovery rate, which may be much higher than p-value. Think about the Bayes formula and the tree diagram you learned in introductory course to statistics to figure out the relationship between p-value and the “false discovery rate”.
5 comments
Comments feed for this article
September 30, 2014 at 4:31 pm
Richard D. Morey
The issue is that the probabilistic version of falsificationist logic inherent in the use of p values is logically invalid. The conclusion (“The null is unlikely to be true”) does not follow from the premises (“A significant effect would be unlikely if H0 were true” and “A significant effect was observed”).
October 2, 2014 at 10:53 pm
Mayo
Richard: Well statistical significance tests never instruct you to follow such invalid reasoning (although using the technical term of likely, it’s quite correct, I assume you’re alleging we assign a posterior to H.) Statistical inference, by the way, is inductive–at least if it is actually reaching an inference not already merely in the premises. Now the invalid argument that Bayesians depend on is statistical affirming the consequent, i.e., from (If H then E, and E, to H):
So if you have data E and dream up H that implies E, you get “support”, or confirmation or a Bayes boost to H since
P(H|E) > P(H)
The frequentist would demand low error probabilities for the method; the Bayesian posterior is just given as the report. An error probability is NOT attached to it.
——
I was really going to respond to the author of the post: No contrast between Bayesian and frequentist inference can come from contrasting a simple case of the use of Bayes theorem with significance test reasoning.
For my comment on the Flam post (which mentions me, by the way) see:
The deepest difference turns on the relevance of outcomes other than the one observed. If the test statistic is d(X), a tester will compute P(d(X) > d(x); Ho). This probability comes from the sampling distribution of the statistic D(X). The Bayesian conditions on the actual data and so no consideration of error probabilities are allowed. But these are central from sampling theorists, resampling, etc. For instance, stopping rules are irrelevant to the Bayesian.
look up stopping rules, or likelihood principle on my blog: error statistics.com
October 2, 2014 at 11:38 pm
Honglang Wang
Thanks, Professor Mayo. According to your suggestion, I found your following post very interesting:
Thanks a lot.
November 18, 2014 at 11:34 am
Empirical Likelihood meets Bayesian Analysis | Honglang Wang's Blog
[…] Analysis is a very popular and useful method in applications. As we discussed in the last post, it’s essentially an belief updating procedure through data, which is very natural in […]
November 26, 2014 at 2:24 pm
waterproof duffel bags at walmart
Hi there colleagues, nice paragraph and pleasant arguments
commented here, I am genuinely enjoying by these.