Hostname: page-component-745bb68f8f-hvd4g Total loading time: 0 Render date: 2025-02-04T10:55:11.575Z Has data issue: false hasContentIssue false
  • English
  • Français

Guarding Against False Positives in Qualitative Comparative Analysis

Published online by Cambridge University Press:  04 January 2017

Bear F. Braumoeller
Bear F. Braumoeller*
Affiliation:
Department of Political Science, The Ohio State University, Columbus, OH 43210
*
e-mail: braumoeller.1@osu.edu (corresponding author)
Get access Rights & Permissions [Opens in a new window]

Abstract

The various methodological techniques that fall under the umbrella description of qualitative comparative analysis (QCA) are increasingly popular for modeling causal complexity and necessary or sufficient conditions in medium-N settings. Because QCA methods are not designed as statistical techniques, however, there is no way to assess the probability that the patterns they uncover are the result of chance. Moreover, the implications of the multiple hypothesis tests inherent in these techniques for the false positive rate of the results are not widely understood. This article fills both gaps by tailoring a simple permutation test to the needs of QCA users and adjusting the Type I error rate of the test to take into account the multiple hypothesis tests inherent in QCA. An empirical application–a reexamination of a study of protest-movement success in the Arab Spring–highlights the need for such a test by showing that even very strong QCA results may plausibly be the result of chance.

Type
Articles
Information
Political Analysis , Volume 23 , Issue 4 , Autumn 2015 , pp. 471 - 487
Copyright
Copyright © The Author 2015. Published by Oxford University Press on behalf of the Society for Political Methodology 

Access options

Get access to the full version of this content by using one of the access options below. (Log in options will check for institutional or personal access. Content may require purchase if you do not have access.)

Footnotes

Author's note: Thanks to Christopher Achen, David Collier, Jirka Lewandowski, the scholars who attended the 2014 summer seminars on Boolean logit at WZB Berlin, and those who attended my sessions at the 2014 IQMR Summer Institute in Syracuse, New York, for valuable feedback, and to Andrew Rosenberg and Austin Knuppe for invaluable research assistance. Replication Data are available on the Dataverse site for this article, http://dx.doi.org/10.7910/DVN/GY6P9I.

References

Ansani, Andrea, and Daniele, Vittorio. 2012. About a revolution: The economic motivations of the Arab Spring. International Journal of Development and Conflict 02(03): 1–24.CrossRefGoogle Scholar
Benjamini, Yoav, and Hochberg, Yosef. 1995. Controlling the false discovery rate: A practical and powerful approach to multiple testing. Journal of the Royal Statistical Society Series B 57(1): 289–300.Google Scholar
Braumoeller, Bear. 2015. Replication data for: Guarding against false positives in qualitative comparative analysis. Harvard Dataverse, V2 [Version], May 18, 2015. http://dx.doi.org/10.7910/DVN/GY6P9I.CrossRefGoogle Scholar
Braumoeller, Bear, and Goertz, Gary. 2000. The methodology of necessary conditions. American Journal of Political Science 44(4): 844–58.CrossRefGoogle Scholar
Cronqvist, Lasse, and Berg-Schlosser, Dirk. 2009. Multi-value QCA (mvQCA). In Configurational comparative methods: Qualitative comparative analysis (QCA) and related techniques, eds. Rihoux, Benoît and Ragin, Charles C., 69–86. Thousand Oaks, CA: Sage Publications, Inc.CrossRefGoogle Scholar
Dion, Douglas. 1998. Evidence and inference in the comparative case study. Comparative Politics 30(2): 127–45.CrossRefGoogle Scholar
Doyle, Michael W. 1983a. Kant, liberal legacies, and foreign affairs. Philosophy and Public Affairs 12(3): 205–35.Google Scholar
Doyle, Michael W 1983b. Kant, liberal legacies, and foreign affairs, part 2. Philosophy and Public Affairs 12(4): 323–53.Google Scholar
Eliason, S. R., and Stryker, R. 2009. Goodness-of-fit tests and descriptive measures in fuzzy-set analysis. Sociological Methods & Research 38(1): 102–46.CrossRefGoogle Scholar
Goertz, Gary, Hak, Tony, and Dul, Jan. 2012. Ceilings and floors: Where are there no observations? Sociological Methods & Research 42(1): 3–40.CrossRefGoogle Scholar
Good, Phillip I. 2005. Permutation, parametric and bootstrap tests of hypotheses, Springer Series in Statistics. 3rd ed. New York: Springer.Google Scholar
Holm, Sture. 1979. A simple sequentially rejective multiple test procedure. Scandinavian Journal of Statistics 6(2): 65–70.Google Scholar
Hug, S. 2013. Qualitative comparative analysis: How inductive use and measurement error lead to problematic inference. Political Analysis 21(2): 252–65.CrossRefGoogle Scholar
Hussain, Muzammil M., and Howard, Philip N. 2013. What best explains successful protest cascades? ICTs and the fuzzy causes of the Arab Spring. International Studies Review 15(1): 48–66.CrossRefGoogle Scholar
Krogslund, C., Choi, D. D., and Poertner, M. 2015. Fuzzy sets on shaky ground: Parameter sensitivity and confirmation bias in fsQCA. Political Analysis 23(1): 21–41.CrossRefGoogle Scholar
LaGraffe, Dan. 2012. The youth bulge in Egypt: An intersection of demographics, security, and the Arab Spring. Journal of Strategic Security 5(2): 65–80.CrossRefGoogle Scholar
Ludbrook, John, and Dudley, Hugh. 1998. Why permutation tests are superior to t and F tests in biomedical research. American Statistician 52(2): 127–32.Google Scholar
Marx, Axel, and Dusa, Adrian. 2011. Crisp-set qualitative comparative analysis (csQCA): Contradictions and consistency benchmarks for model specification. Methodological Innovations Online 6(2): 103–48.CrossRefGoogle Scholar
Monroe, Burt, and Gold, Suzanne. 2004. A close look at the qualitative comparative analysis (QCA) family of methodologies. Paper presented at the Annual Meeting of the Midwest Political Science Association, Palmer House Hilton, Chicago, IL.Google Scholar
Paul, Christopher, and Clarke, Colin P. 2014. A broad approach to countering the Islamic state. Washington Post. Google Scholar
Ragin, Charles C. 1987. The comparative method: Moving beyond qualitative and quantitative strategies. Berkeley: University of California Press.Google Scholar
Ragin, Charles C 2000. Fuzzy-set social science. Chicago: University of Chicago Press.Google Scholar
Ragin, Charles C 2004. Between complexity and parsimony: Limited diversity, counterfactual cases, and comparative analysis. University of California. http://www.sscnet.ucla.edu/soc/soc237/papers/ragin.pdf (accessed on July 10, 2015).Google Scholar
Ragin, Charles C 2008. Redesigning social inquiry: Fuzzy sets and beyond. Chicago: University of Chicago Press.CrossRefGoogle Scholar
Ragin, Charles C., Ilene Strand, Sarah, and Rubinson, Claude. 2008. User's guide to fuzzy-set/qualitative comparative analysis. Manuscript, University of Arizona. http://www.u.arizona.edu/∼cragin/fsQCA/download/fsQCAManual.pdf (accessed March 10, 2015).Google Scholar
Rihoux, Benoît, Alamos-Concha, Damien Bol, Axel Marx, Priscilla, and Rezsöhazy, Ilona. 2013. From niche to mainstream method? A comprehensive mapping of QCA applications in journal articles from 1984 to 2011. Political Research Quarterly 66(1): 175–84.Google Scholar
Schneider, Carsten Q., and Wagemann, Claudius. 2012. Set-theoretic methods for the social sciences: A guide to qualitative comparative analysis. Cambridge, UK: Cambridge University Press.CrossRefGoogle Scholar
Schneider, Carsten Q., and Wagemann, Claudius. 2013. Doing justice to logical remainders in QCA: Moving beyond the standard analysis. Political Research Quarterly 66(1): 211–20.Google Scholar
Thygeson, N. Marcus, Peikes, Deborah, and Zutshi, Aparajita. 2013. Fuzzy-set qualitative comparative analysis: A configurational comparative method to identify multiple pathways to improve patient-centered medical home models. Technical report AHRQ, Publication No. 13-0026-EF, Agency for Healthcare Research and Quality, Rockville, MD.Google Scholar
Welkowitz, Joan, Cohen, Barry H., and Brooke Lea, R. 2012. Introductory statistics for the behavioral sciences. 7th ed. Hoboken, NJ: Wiley.Google Scholar