Hostname: page-component-745bb68f8f-kw2vx Total loading time: 0 Render date: 2025-01-27T04:16:46.692Z Has data issue: false hasContentIssue false
  • English
  • Français

EFFECT SIZE–DRIVEN SAMPLE-SIZE PLANNING, RANDOMIZATION, AND MULTISITE USE IN L2 INSTRUCTED VOCABULARY ACQUISITION EXPERIMENTAL SAMPLES

Published online by Cambridge University Press:  06 September 2021

Joseph P. Vitta Open the ORCID record for Joseph P. Vitta [Opens in a new window] ,
Christopher Nicklin Open the ORCID record for Christopher Nicklin [Opens in a new window]  and
Stuart McLean Open the ORCID record for Stuart McLean [Opens in a new window]
Joseph P. Vitta*
Affiliation:
Kyushu University
Christopher Nicklin
Affiliation:
Rikkyo University
Stuart McLean
Affiliation:
Momoyama Gakuin University
*
*Correspondence concerning this article should be addressed to Joseph P. Vitta, Kyushu University, Faculty of Languages and Cultures, Fukuoka, Japan. E-mail: vittajp@flc.kyushu-u.ac.jp and vittajp@gmail.com
Get access Rights & Permissions [Opens in a new window]

Abstract

In this focused methodological synthesis, the sample construction procedures of 110 second language (L2) instructed vocabulary interventions were assessed in relation to effect size–driven sample-size planning, randomization, and multisite usage. These three areas were investigated because inferential testing makes better generalizations when researchers consider them during the sample construction process. Only nine reports used effect sizes to plan or justify sample sizes in any fashion, with only one engaging in an a priori power procedure referencing vocabulary-centric effect sizes from previous research. Randomized assignment was observed in 56% of the reports while no report involved randomized sampling. Approximately 15% of the samples observed were constructed from multiple sites and none of these empirically investigated the effect of site clustering. Leveraging the synthesized findings, we conclude by offering suggestions for future L2 instructed vocabulary researchers to consider a priori effect size–driven sample planning processes, randomization, and multisite usage when constructing samples.

Type
Methods Forum
Information
Studies in Second Language Acquisition , Volume 44 , Issue 5 , December 2022 , pp. 1424 - 1448
Check for updates
Open Practices
Open data Open materials
Copyright
© The Author(s), 2021. Published by Cambridge University Press

Access options

Get access to the full version of this content by using one of the access options below. (Log in options will check for institutional or personal access. Content may require purchase if you do not have access.)

Footnotes

The research synthesis analyses in this article earned Open Materials and Open Data badges for transparent practices. The materials and data are available at https://www.iris-database.org/iris/app/home/detail?id=york%3A939467&ref=search

The authors wish to acknowledge Joy Egbert, Luke Plonsky, Ali H. Al-Hoorie, Phil Hiver, and Dayna Jost for the expert reviews regarding the report pool creation process and for their external consultations throughout the project. The authors also wish to thank the SSLA editors and the two external reviewers who provided useful and constructive feedback on multiple versions of this manuscript.

†This article has been updated since its original publication. See https://doi.org/10.1017/S0272263121000814

References

REFERENCES

Al-Hoorie, A. H., & Vitta, J. P. (2019). The seven sins of L2 research: A review of 30 journals’ statistical quality and their CiteScore, SJR, SNIP, JCR impact factors. Language Teaching Research, 23, 727–744. https://doi.org/10.1177/1362168818767191.CrossRefGoogle Scholar
Andringa, S., & Godfroid, A. (2020). Sampling bias and the problem of generalizability in applied linguistics. Annual Review of Applied Linguistics, 40, 134–142. https://doi.org/10.1017/S0267190520000033.CrossRefGoogle Scholar
Bryfonski, L., & Mckay, T. H. (2019). TBLT implementation and evaluation: A meta-analysis. Language Teaching Research, 23, 603–632. https://doi.org/10.1177/1362168817744389.CrossRefGoogle Scholar
Brysbaert, M. (2019). How many participants do we have to include in properly powered experiments? A tutorial of power analysis with reference tables. Journal of Cognition, 2, 1–38. https://doi.org/10.5334/joc.72.CrossRefGoogle ScholarPubMed
Brysbaert, M., & Stevens, M. (2018). Power analysis and effect size in mixed effects models: A tutorial. Journal of Cognition, 1, 9. https://doi.org/10.5334/joc.10.CrossRefGoogle ScholarPubMed
Busse, V., Cenoz, J., Dalmann, N., & Rogge, F. (2020). Addressing linguistic diversity in the language classroom in a resource-oriented way: An intervention study with primary school children. Language Learning, 70, 382–419. https://doi.org/10.1111/lang.12382.CrossRefGoogle Scholar
Choi, S. (2017). Processing and learning of enhanced English collocations: An eye movement study. Language Teaching Research, 21, 403–426. https://doi.org/10.1177/1362168816653271.CrossRefGoogle Scholar
Cisco, B. K., & Padrón, Y. (2012). Investigating vocabulary and reading strategies with middle grades English language learners: A research synthesis. Research in Middle Level Education, 36, 1–23. https://doi.org/10.1080/19404476.2012.11462097.Google Scholar
Cohen, J. (1988). Statistical power analysis for the behavioral sciences. Routledge Academic. https://doi.org/10.4324/9780203771587.Google Scholar
Cumming, G. (2012). Understanding the new statistics: Effect sizes, confidence intervals, and meta-analysis. Routledge. https://doi.org/10.4324/9780203807002.Google Scholar
de Carvalho, M. (2018). spearmanCI: Jackknife Euclidean/empirical likelihood inference for Spearman Rho. R package version 1.0. https://cran.rproject.org/web/packages/spearmanCI/spearmanCI.pdf.Google Scholar
Dellinger, J. (2017). Correlation, Spearman. In Allen, M. (Ed.), The SAGE Encyclopedia of Communication Research Methods (pp. 274–275). Sage.Google Scholar
Egbert, J. (2007). Quality analysis of journals in TESOL and applied linguistics. TESOL Quarterly, 41, 157–171. https://doi.org/10.1002/j.1545-7249.2007.tb00044.x.CrossRefGoogle Scholar
Elgort, I. (2017). Incorrect inferences and contextual word learning in English as a second language. Journal of the European Second Language Association, 1, 1–11. http://doi.org/10.22599/jesla.3.CrossRefGoogle Scholar
Elgort, I. (2018). Technology-mediated second language vocabulary development: A review of trends in research methodology. CALICO Journal, 35, 1–29. https://doi.org/10.1558/cj.34554.CrossRefGoogle Scholar
Elgort, I., Beliaeva, N., & Boers, F. (2020). Contextual word learning in the first and second language: Definition placement and inference error effects on declarative and nondeclarative knowledge. Studies in Second Language Acquisition, 42, 7–32. https://doi.org/10.1017/S0272263119000561.CrossRefGoogle Scholar
Fanelli, D. (2010). Do pressures to publish increase scientists’ bias? An empirical support from US States data. PLoS ONE, 5, e10271. https://doi.org/10.1371/journal.pone.0010271.CrossRefGoogle ScholarPubMed
Farsani, M. A., & Babaii, E. (2020). Applied linguistics research in three decades: A methodological synthesis of graduate theses in an EFL context. Quality & Quantity, 54, 1257–1283. https://doi.org/10.1007/s11135-020-00984-w.CrossRefGoogle Scholar
Faul, F., Erdfelder, E., Lang, A. G., & Buchner, A. (2007). G*Power 3: A flexible statistical power analysis program for the social, behavioral, and biomedical sciences. Behavior Research Methods, 39, 175–191. https://doi.org/10.3758/BF03193146.CrossRefGoogle ScholarPubMed
Field, A. (2018). Discovering statistics using IBM SPSS statistics. Sage Publications.Google Scholar
Fisher, R. A. (1935). The design of experiments. Oliver and Boyd.Google Scholar
Folse, K. S. (2006). The effect of type of written exercise on L2 vocabulary retention. TESOL Quarterly, 40, 273–293. https://doi.org/10.2307/40264523.CrossRefGoogle Scholar
Fraley, R. C., & Vazire, S. (2014). The N-pact factor: Evaluating the quality of empirical journals with respect to sample size and statistical power. PloS One, 9, e109019. https://doi.org/10.1371/journal.pone.0109019.CrossRefGoogle ScholarPubMed
Gass, S., Loewen, S., & Plonsky, L. (2021). Coming of age: The past, present, and future of quantitative SLA research. Language Teaching, 54, 245–258. https://doi.org/10.1017/s0261444819000430.CrossRefGoogle Scholar
Glass, G. V. (1965). A ranking variable analogue of biserial correlation: Implications for short-cut item analysis. Journal of Educational Measurement, 2, 91–95. https://doi.org/10.1111/j.1745-3984.1965.tb00396.x.CrossRefGoogle Scholar
Gravetter, F. J., & Wallnau, L. B. (2015). Statistics for the behavioral sciences. Cengage Learning.Google Scholar
Green, P., & , MacLeod (2016). SIMR: An R package for power analysis of generalized linear mixed models by simulation. Methods in Ecology and Evolution, 7, 493–498. https://doi.org/10.1111/2041-210X.12504.CrossRefGoogle Scholar
Harter, R. (2008). Random sampling. In Lavrakas, P. (Ed.), Encyclopedia of Survey Research Methods (pp. 683–684). SAGE Publications. https://doi.org/10.4135/9781412963947.n440.Google Scholar
Hiver, P., & Al‐Hoorie, A. H. (2020). Reexamining the role of vision in second language motivation: A preregistered conceptual replication of You, Dörnyei, and Csizér (2016). Language Learning, 70, 48–102. https://doi.org/10.1111/lang.12371.CrossRefGoogle Scholar
Hulstijn, J. H., & Laufer, B. (2001). Some empirical evidence for the involvement load hypothesis in vocabulary acquisition. Language Learning, 51, 539–558. https://doi.org/10.1111/0023-8333.00164.CrossRefGoogle Scholar
Hulstijn, J. H., Schoonen, R., de Jong, N. H., Steinel, M. P., & Florijn, A. (2012). Linguistic competences of learners of Dutch as a second language at the B1 and B2 levels of speaking proficiency of the Common European Framework of Reference for Languages (CEFR). Language Testing, 29, 203–221. https://doi.org/10.1177/0265532211419826.CrossRefGoogle Scholar
Kim, Y. (2008). The role of task-induced involvement and learner proficiency in L2 vocabulary acquisition. Language Learning, 58, 285–325. https://doi.org/10.1111/j.1467-9922.2008.00442.x.CrossRefGoogle Scholar
Kuehl, R. O. (2000). Design of experiments: Statistical principles in research design and analysis. Duxbury.Google Scholar
Lakens, D. (2013). Calculating and reporting effect sizes to facilitate cumulative science: A practical primer for t-tests and ANOVAs. Frontiers in Psychology, 4, 1–12. https://doi.org/10.3389/fpsyg.2013.00863.CrossRefGoogle Scholar
Lakens, D., Adolfi, F. G., Albers, C. J., Anvari, F., Apps, M.A. J., Argamon, S. E., … Zwaan, R. A. (2018). Justify your alpha. Nature Human Behaviour, 2, 168–171. https://doi.org/10.1038/s41562-018-0311-x.CrossRefGoogle Scholar
Lakens, D., & Caldwell, A. R. (2019). Simulation-based power-analysis for factorial ANOVA designs. PsyArXiv. https://doi.org/10.31234/osf.io/baxsf.CrossRefGoogle Scholar
Laufer, B., & Nation, P. (1995). Vocabulary size and use: Lexical richness in L2 written production. Applied Linguistics, 16, 307–322. https://doi.org/10.1093/applin/16.3.307.CrossRefGoogle Scholar
Laufer, B., & Nation, P. (1999). A vocabulary-size test of controlled productive ability. Language Testing, 16, 33–51. https://doi.org/10.1177/026553229901600103.CrossRefGoogle Scholar
Lewis, M. (1993). The lexical approach. Longman Teaching Publications.Google Scholar
Lin, J., & Lin, H. (2019). Mobile-assisted ESL/EFL vocabulary learning: A systematic review and meta-analysis. Computer Assisted Language Learning, 32, 878–919. https://doi.org/10.1080/09588221.2018.1541359.CrossRefGoogle Scholar
Lindstromberg, S. (2016). Inferential statistics in Language Teaching Research: A review and ways forward. Language Teaching Research, 20, 741–768. https://doi.org/10.1177/1362168816649979.CrossRefGoogle Scholar
Loewen, S. (2015). Instructed second language acquisition. Routledge.Google Scholar
Loewen, S., & Sato, M. (2017). Instructed second language acquisition (ISLA): An overview. In Loewen, S. & Sato, M. (Eds.), The Routledge handbook of instructed second language acquisition (pp. 1–12). Routledge.CrossRefGoogle Scholar
McHugh, M. L. (2012). Interrater reliability: The kappa statistic. Biochemia Medica, 22, 276–282. https://doi.org/10.11613/bm.2012.031.CrossRefGoogle ScholarPubMed
McLean, S., Stewart, J., & Batty, A. O. (2020). Predicting L2 reading proficiency with modalities of vocabulary knowledge: A bootstrapping approach. Language Testing, 37, 389–411. https://doi.org/10.1177/0265532219898380.CrossRefGoogle Scholar
Mehring, J. (2018). The flipped classroom. In Mehring, J. and Leis, A. (Eds.), Innovations in flipping the language classroom: Theories and practices (pp. 1–10). Springer.CrossRefGoogle Scholar
Moranski, K., & Ziegler, N. (2021). A case for multisite second language acquisition research: Challenges, risks, and rewards. Language Learning, 71, 204–242. https://doi.org/10.1111/lang.12434.CrossRefGoogle Scholar
Morgan‐Short, K., Marsden, E., Heil, J., Issa, II, B.I., Leow, R.P., Mikhaylova, A., Mikołajczak, S., Moreno, N., Slabakova, R. and Szudarski, P. (2018), Multisite replication in second language acquisition research: Attention to form during listening and reading comprehension. Language Learning, 68, 392–437. https://doi.org/10.1111/lang.12292.CrossRefGoogle Scholar
Nicklin, C., & Plonsky, L. (2020). Outliers in L2 research in applied linguistics: A synthesis and data re-analysis. Annual Review of Applied Linguistics, 40, 25–55. https://doi.org/10.1017/S0267190520000057.CrossRefGoogle Scholar
Nicklin, C., & Vitta, J. P. (2021). Effect‐driven sample sizes in second language instructed vocabulary acquisition research. The Modern Language Journal, 105, 218–236. https://doi.org/10.1111/modl.12692.CrossRefGoogle Scholar
Norouzian, R. (2020). Sample size planning in quantitative L2 research: A pragmatic approach. Studies in Second Language Acquisition, 41, 849–870. https://doi.org/10.1017/S0272263120000017.CrossRefGoogle Scholar
Norouzian, R. (2021). Interrater reliability in second language meta-analyses: The case of categorical moderators. Studies in Second Language Acquisition. Advance online publication. https://doi.org/10.1017/s0272263121000061.CrossRefGoogle Scholar
O’Connor, D., Green, S., & Higgins, J. P. (2008). Defining the review question and developing criteria for including studies. In Higgins, J. P. & Green, S. (Eds.), Cochrane handbook for systematic reviews of interventions (pp. 81–94). Wiley-Blackwell.CrossRefGoogle Scholar
Osborne, J. W, & Waters, E. (2002). Four assumptions of multiple regression that researchers should always test. Practical Assessment, Research, & Evaluation, 8, 1–5. https://doi.org/10.7275/r222-hv23.Google Scholar
Perugini, M., Gallucci, M., & Costantini, G. (2018). A practical primer to power analysis for simple experimental designs. International Review of Social Psychology, 31, 1–20. http://doi.org/10.5334/irsp.181.CrossRefGoogle Scholar
Peters, E. (2019). The effect of imagery and on-screen text on foreign language vocabulary learning from audiovisual input. TESOL Quarterly, 53, 1008–1032. https://doi.org/10.1002/tesq.531.CrossRefGoogle Scholar
Plonsky, L. (2013). Study quality in SLA: An assessment of designs, analyses, and reporting practices in quantitative L2 research. Studies in Second Language Acquisition, 35, 655–687. https://doi.org/10.1017/S0272263113000399.CrossRefGoogle Scholar
Plonsky, L. (2014). Study quality in quantitative L2 research (1990–2010): A methodological synthesis and call for reform. Modern Language Journal, 98, 450–470. https://doi:10.1111/j.1540-4781.2014.12058.x.CrossRefGoogle Scholar
Plonsky, L. (2015). Statistical power, p values, descriptive statistics, and effect sizes: A “back-to-basics” approach to advancing quantitative methods in L2 research. In Plonsky, L. (Ed.), Advancing quantitative methods in second language research (pp. 23–45). Routledge. https://doi.org/10.4324/9781315870908.CrossRefGoogle Scholar
Plonsky, L., & Gass, S. (2011). Quantitative research methods, study quality, and outcomes: The case of interaction research. Language Learning, 61, 325–366. https://doi.org/10.1111/j.1467-9922.2011.00640.x.CrossRefGoogle Scholar
Plonsky, L., and Gonulal, T. (2015). Methodological synthesis in quantitative L2 research: A review of reviews and a case study of exploratory factor analysis. Language Learning, 65, 9–36. https://doi.org/10.1111/lang.12111.CrossRefGoogle Scholar
Plonsky, L., & Oswald, F. L. (2014). How big is “big”? Interpreting effect sizes in L2 research. Language Learning, 64, 878–912. https://doi.org/10.1111/lang.12079.CrossRefGoogle Scholar
Plonsky, L., Sudina, E., & Hu, Y. (2021). Applying meta-analysis to research on bilingualism: An introduction. Bilingualism: Language and Cognition. Advance online publication. https://doi.org/10.1017/S1366728920000760.CrossRefGoogle Scholar
R Core Team (2020). R: A language and environment for statistical computing. R Foundation for Statistical Computing, Vienna, Austria. https://www.R-project.org/.Google Scholar
Rogers, J., & Révész, A. (2020). Experimental and quasi-experimental designs. In McKinley, J. & Rose, H. (Eds.), The Routledge handbook of research methods in applied linguistics (pp. 133–143). Routledge.Google Scholar
Schmitt, N. (2008). Instructed second language vocabulary learning. Language Teaching Research, 12, 329–363. https://doi.org/10.1177/1362168808089921.CrossRefGoogle Scholar
Shintani, N. (2013). The effect of focus on form and focus on forms instruction on the acquisition of productive knowledge of L2 vocabulary by young beginning-level learners. TESOL Quarterly, 47, 36–62. https://doi.org/10.1002/tesq.54.CrossRefGoogle Scholar
Siddaway, A. P., Wood, A. M., & Hedges, L. V. (2019). How to do a systematic review: A best practice guide for conducting and reporting narrative reviews, meta-analyses, and meta-syntheses. Annual Review of Psychology, 70, 747–770. https://doi.org/10.1146/annurev-psych-010418-102803.CrossRefGoogle Scholar
Traxler, R. E., & Nakatsukasa, K. (2020). The effectiveness of voice-on and voice-off instruction on ASL vocabulary acquisition. Language Teaching Research, 24, 273–286. https://doi.org/10.1177/1362168818791601.CrossRefGoogle Scholar
Trochim, W. M., Donnelly, J. P., & Arora, K. (2016). Research methods: The essential knowledge base. Cengage Learning.Google Scholar
Uchihara, T., Webb, S., & Yanagisawa, A. (2019). The effects of repetition on incidental vocabulary learning: A meta‐analysis of correlational studies. Language Learning, 69, 559–599. https://doi.org/10.1111/lang.12343.CrossRefGoogle Scholar
Vafaee, P., & Suzuki, Y. (2020). The relative significance of syntactic knowledge and vocabulary knowledge in second language listening ability. Studies in Second Language Acquisition, 42, 383–410. https://doi.org/10.1017/S0272263119000676.CrossRefGoogle Scholar
Vitta, J. P., & Al-Hoorie, A. H. (2020). The flipped classroom in second language learning: A meta-analysis. Language Teaching Research. Advance online publication. https://doi.org/10.1177/1362168820981403.CrossRefGoogle Scholar
Vitta, J. P., & Al-Hoorie, A. H. (2021). Measurement and sampling recommendations for L2 flipped learning experiments: A bottom-up methodological synthesis. The Journal of Asia TEFL, 18, 682–692. https://doi.org/10.18823/asiatefl.2021.18.2.23.682.Google Scholar
Vo, S. (2019). Use of lexical features in non-native academic writing. Journal of Second Language Writing, 44, 1–12. https://doi.org/10.1016/j.jslw.2018.11.002.CrossRefGoogle Scholar
Webb, S., & Kagimoto, E. (2009). The effects of vocabulary learning on collocation and meaning. TESOL Quarterly, 43, 55–77. https://doi.org/10.1002/j.1545-7249.2009.tb00227.x.CrossRefGoogle Scholar
Webb, S., Yanagisawa, A., & Uchihara, T. (2020). How effective are intentional vocabulary‐learning activities? A meta‐analysis. The Modern Language Journal, 104, 715–738. https://doi.org/10.1111/modl.12671.CrossRefGoogle Scholar
Willis, D., & Willis, J. (2007). Doing task-based teaching. Oxford University Press.Google Scholar
Yang, Y., Shintani, N., Li, S., & Zhang, Y. (2017). The effectiveness of post-reading word-focused activities and their associations with working memory. System, 70, 38–49. https://doi.org/10.1016/j.system.2017.09.012.CrossRefGoogle Scholar
Yanagisawa, A., Webb, S., & Uchihara, T. (2020). How do different forms of glossing contribute to L2 vocabulary learning from reading? A meta-regression analysis. Studies in Second Language Acquisition, 42, 411–438. https://doi.org/10.1017/so272263119000688.CrossRefGoogle Scholar
You, C., Dörnyei, Z., & Csizér, K. (2016). Motivation, vision, and gender: A survey of learners of English in China. Language Learning, 66, 94–123. https://doi.org/10.1111/lang.12140.CrossRefGoogle Scholar
Zhang, S., & Zhang, X. (2020). The relationship between vocabulary knowledge and L2 reading/listening comprehension: A meta-analysis. Language Teaching Research. Advance online publication. https://doi.org/10.1177/1362168820913998.CrossRefGoogle Scholar
Supplementary material: File

Vitta et al. supplementary material

Vitta et al. supplementary material

Download Vitta et al. supplementary material(File)
File 101.8 KB