Volunteer Response Bias

I've said it before and I'll say it again.  Statistics can be no better than your source of data.  Go to any statistics teacher at any school and ask about volunteer response surveys.  They are among the most common surveys yet have among the worst bias of all surveys.  Bias, in statistics, means that the sample does not accurately represent the population that is being studied.  A volunteer response survey asks people to volunteer to be a part of the survey.  However, only people who feel strongly are likely to respond, and they will encourage friends to respond, many of whom share similar views, making their opinions vastly over-representative.

For example, if CNN ran a survey on their website on whether or not GMO food was good for you, natural-food-nuts would quickly pounce on it and get their friends to do so as well, while most of the population would not bother.  The results would be very skewed and not at all representative of the population's view, even if it got huge numbers of respondents.

With that in mind, the results of the gay Mormon survey are out.  It was a classic case of a volunteer response survey.  I saw it circulating among a certain circle of friends, and I warned those who asked me about it that it was going to produce skewed results because of the way it was being run.

Imagine that you are a former member of the LDS church who was in a so-called mixed orientation marriage, but you got divorced and then excommunicated, or possibly had your name removed from the records of the church.  You are already public about it, and relish the opportunity to make yourself heard.  Not only that, but you have friends in similar situations so when you get the survey, you forward it to all these friends and you end up with huge participation from this portion of the population.

Now imagine that you are a faithful member of the church in one of these mixed orientation marriages.  You aren't public about it because you are unsure how people in your ward would respond.  You don't have lots of friends within the gay community.  Firstly, you are much less likely to come across the survey; and secondly, even if you do, you are less likely to fill it out.  You definitely won't be forwarding it to all your friends in mixed orientation marriages, because you don't know who they are.  You are in the closet.  You aren't making noise.  This segment of the population gets vastly underrepresented due to the nature of volunteer response surveys.

To me it was inevitable that the survey results would show that the participants predominantly left the church and failed at these mixed orientation marriages.  The bias was clearly pointed that way, regardless of the size of the survey or the actual divorce rates (or excommunication rates) of the whole population in question.  The sample does not accurately represent the population being studied.