For those who’ve been following along, you’re about to witness the most important step of the research process.
Step eight: learn from feedback on your work.
Between the comments on my post and emails I received, I realized two important things about the study.
First, I wasn’t clear enough about what the study actually showed. It showed that there was no significant effect of gender on the rate of RPG publication, given this data. That isn’t the same as showing that there is no effect. It just means I didn’t find one!
This turns out to be important because of the second thing: I neglected to do a power analysis as part of my work. A power analysis tells me whether I would have found a gender difference in publication rates, if there were one. Ordinarily I’d have done one as a matter of course, but I was trying to be quick and dirty – and ended up being a little too hasty.
So, was my study powerful enough to find out the truth? Well, it turns out the power of my study is only .159. If there is a gender difference, I’ll only detect it 15.9% of the time. That means we shouldn’t be very confident in the answer we got. There very well might be a gender difference in RPG publication rates that this study didn’t pick up!
This means that my next step isn’t to move on to qualitative analysis, but to do another quantitative analysis that either confirms or refutes this one. Unfortunately, I think the reason the power of the study was so low is that there were so few women who published more than one game. I’m not sure where else to look for data, or if there’s another way to get the power of the study up. If anyone’s got ideas about other sources or approaches I could use, I welcome input in the comments.
On the other hand, I still feel comfortable saying that the difference in rates of first game publication is both significant and problematic. Watch this space – many of us here at Gaming As Women are thinking about this issue!