2.1 Pragmatic Production

Reference Kasper and RoseKasper and Rose (2002: 117) pointed out that L2 pragmatics was “heavily outweighed by the proliferation of studies on pragmatic production.” This trend remains and reflects the preference for production in pragmatics in general. The skew towards production is understandable, since production can provide more nuanced and easily observed data than perception, and in many cases production involves perception of the other interlocutor’s communication.

Production studies often investigate learners’ production across different influential factors, such as L2 proficiency, learning environment (study abroad vs. study at home), and length of stay abroad. They have yielded controversial and sometimes contradictory findings with respect to the impact of the factors on learners’ pragmatic production, so that our understanding of the effects of these influential factors is incomplete. For instance, partially because different pragmatic targets are examined in different studies, the findings relating to the association between pragmatic competence and linguistic proficiency are often inconsistent. Although most researchers agree that learners’ pragmatic production and their linguistic proficiency do not show a linear correlation, generalization becomes difficult when studies do not employ standardized tests for determining learners’ proficiency levels. Another example is the common finding that learners perform differently from native speakers. However, since a study can only focus on a small cohort of participants, it cannot investigate multiple factors that may influence learners’ pragmatic learning. Given that the number of influencing factors and the way in which they may interact are often different for different learners, it is very complex to compare the findings. For example, Reference Takahashi and BeebeTakahashi and Beebe (1987) found that learners who had studied abroad for an average of four years had a broader range of expressions and more flexibility to adjust their level of directness according to different situations than those who had studied abroad for an average of seven months. Reference BellaBella (2011) demonstrated that even learners with a 4.5-year residence-abroad experience still displayed an underdeveloped pragmatic ability in relation to mitigation devices. Reference RenRen (2019b) found that with respect to request strategies, learners with two years of study-abroad experience resembled native speakers more, while learners with longer study-abroad experience of two to four years showed more non-target-like development. Therefore, this section focuses on what L2 pragmatics production research has investigated, rather than detailed findings in particular areas.

2.2 Pragmatic Perception

Studies on learner pragmatic perception can be roughly divided into two categories, namely pragmatic awareness and pragmatic comprehension, which will be reviewed in the following two subsections respectively.

2.3 Research Modality

Second language pragmatics learning has predominantly investigated learners’ pragmatic competence in face-to-face oral communication, although many studies collect data using written-for-oral tasks (Reference Bardovi-HarligBardovi-Harlig, 2018) – for example, using written DCTs to explore oral pragmatic competence. Only a couple of studies have examined learners’ written pragmatic competence; Reference YounYoun (2014) investigated the relationships among learners’ L2 pragmatics in writing tasks, syntactic complexity, and L2 proficiency. Syntactic complexity was analyzed in three aspects: mean length of T-unit, mean length of clause, and mean number of clauses per T-unit. The results showed that the three measures of syntactic complexity were not strongly related to increases in L2 proficiency and pragmatic competence. Comparatively, the increases in mean length of T-unit were more related to different levels of proficiency and pragmatics than the other two measures. Similarly, Reference ChenChen (2020) analyzed Chinese EFL writers’ use of single-author in academic writing, while Reference Taguchi, Fernández and JiangTaguchi et al. (2021) examined Chinese EFL learners’ advice-giving in written texts.

The fact that learners’ writing pragmatic competence is under-researched may be due in part to two reasons. On the one hand, pragmatists, including L2 pragmatists, still value oral communication over written texts. Thus, except for historical pragmatics, pragmatics research prefers oral communication over written. On the other hand, many scholars investigate L2 learners’ writing such as formulaic expressions (e.g., Reference Pérez-LlantadaPérez-Llantada, 2014) and meta-discourse (e.g., Reference HylandHyland, 2019; Reference Qin and UccelliQin and Uccelli, 2019), but they conduct research under the perspectives of L2 writing and academic discourse rather than L2 pragmatics.

Compared with written communication, L2 pragmatics has paid more attention to learners’ pragmatic competence in internet-mediated communication. This stream of work can be traced at least to the beginning of this century; for example, Reference Belz and KingingerBelz and Kinginger (2002) showcased the learning and use of informal vs. formal T-/V-form second person pronouns by two American students, one studying French and one studying German, in web-based contact including email and synchronous chat with their key pals in France and Germany, respectively.

However, most of the L2 pragmatics studies on internet-mediated communication have been conducted in this decade. Email is a frequently explored communicative genre (e.g., Reference Chen and LiuChen et al., 2016; Reference Economidou-Kogetsidis, Savic and HalenkoEconomidou-Kogetsidis et al., 2021), in both pragmatic production (e.g., Reference Barón and OrtegaBarón and Ortega, 2018; Reference Economidou-KogetsidisEconomidou-Kogetsidis, 2018; Reference Savić, Đorđević, Economidou-Kogetsidis, Savić and HalenkoSavić and Đorđević, 2021) and perception (e.g., Reference Economidou-KogetsidisEconomidou-Kogetsidis, 2016; Reference Ren, Liu, Economidou-Kogetsidis, Savic and HalenkoRen and Liu, 2021; Reference WinansWinans, 2020). Researchers often employ electronic DCTs (EDCTs) to investigate email literacy, and this is more legitimate than using written DCTs for exploring oral communication; although email is a hybrid of oral and written, it is essentially more like a written genre.

Researchers have also investigated learners’ pragmatic competence in various types of internet-mediated communication, including digital stories (Reference García-PastorGarcía-Pastor, 2020), web conferences (Reference CunninghamCunningham, 2017), gameplay (Reference Tang and TaguchiTang and Taguchi, 2021), and social media such as Facebook (Reference Blattner and FioriBlattner and Fiori, 2012), Reddit (Reference Yeh and SwinehartYeh and Swinehart, 2020), Skype (Reference Barron and BlackBarron and Black, 2015), and WhatsApp (Reference García-GómezGarcía-Gómez, 2020). Most research has documented the benefit of internet-mediated communication in helping learners to develop their L2 pragmatic competence. However, Reference García-GómezGarcía-Gómez (2020) found that without careful design or training, social media platforms such as WhatsApp may also be detrimental for learners’ L2 pragmatic development because they may encounter more cases of misunderstanding on WhatsApp.

As the above reviews show, L2 pragmatics learning has investigated oral, written, and internet-mediated communication. It is vital for researchers to “converge on modality” (Reference Bardovi-HarligBardovi-Harlig, 2018: 16) in L2 pragmatics research. That is, if researchers wish to investigate learners’ pragmatic competence in oral communication, it must be investigated by oral tasks; in written communication, it must be investigated by written tasks; and in internet-mediated communication, it must be investigated in the context of online platforms. However, there were some studies that used written tasks to measure learners’ oral production, which posed a serious threat to the validity of the findings (see Section 4.2 for more discussion on validity).

2.4 Summary

This section reviewed studies investigating learners’ acquisition in pragmatic production and perception. Some studies on conventional expressions examined both facets. In contrast, learners’ production and perception were often analyzed separately in other pragmatic features. Future studies are encouraged to investigate the two facets of pragmatic competence in the same groups of learners to explore the relation between productive and receptive pragmatic competences. In addition, except for studies on perception of contextual cues, the research only examines the linguistic resource in pragmatic competence. The semiotic and multimodal resources have not attracted attention, even in the internet-mediated communication where such resources are used pervasively.

Furthermore, although increasingly researchers have investigated learners’ pragmatic learning, there are still a large number of studies only examining learners’ pragmatic use. They investigate learners’ pragmatic performance at the time of data collection, and thus they can be classified as single-moment studies (Reference Kasper and RoseKasper and Rose, 1999). The focus on pragmatic use rather than learning is a partially cause of the peripheral position of L2 pragmatics in general L2 research (Reference RenRen, 2015). Future studies should stop treating learners as a homogenous group and simply comparing their pragmatic performance with that of native speakers. It is more insightful to investigate the pattern of learners’ pragmatic development and the association between pragmatic learning and linguistic learning (i.e., vocabulary, grammar) for uncovering the underlying mechanisms of pragmatic learning and suggesting useful implications for pragmatic instruction. The next section reviews the role of pragmatic teaching.

3.1 Effects of Teaching Second Language Pragmatics

Literature on L2 pragmatics learning has documented that mere exposure is not sufficient for pragmatic development in an L2, even after a long stay in the target language community (e.g., Reference BarronBarron, 2003; Reference RenRen, 2019b; Reference SchauerSchauer, 2009), because “pragmatic functions and relevant contextual factors are often not salient to learners and so not likely to be noticed despite prolonged exposure” (Reference Kasper and RoseKasper and Rose, 2002: 237). Therefore, much of the literature on pragmatics learning calls for pragmatics intervention to assist learners to develop their L2 pragmatic competence (e.g., Reference RenRen, 2013; Reference TaguchiTaguchi, 2015b), which is generally neglected in L2 classrooms or textbooks (Reference Ren and HanRen and Han, 2016).

Early studies on L2 pragmatics teaching were stimulated by Reference KasperKasper’s (1997) seminal talk, “Can Pragmatic competence be taught?”. After summarizing a dozen prior studies on teaching pragmatics, Kasper’s answer to this question was affirmative. The next few years saw increasing numbers of empirical studies on L2 pragmatics teaching (see Section 3.2) and synthetic works, including narrative reviews (Reference Kasper and RoseKasper and Rose, 1999; Reference RoseRose, 2005; Reference TaguchiTaguchi, 2011c), syntheses (Reference TaguchiTaguchi, 2015b; Reference Takahashi and TrosborgTakahashi, 2010), and meta-analyses (Reference Badjadi, Tang and LogonnathanBadjadi, 2016; Reference Jeon, Kaya, Norris and OrtegaJeon and Kaya, 2006; Reference Plonsky, Zhuang and TaguchiPlonsky and Zhuang, 2019; Reference Yousefi and NassajiYousefi and Nassaji, 2019), all of which showed that pragmatics is teachable and pragmatic instruction can facilitate learners’ L2 pragmatic development. It is fair to say that L2 pragmatics scholars have reached a consensus regarding the question of whether teaching is effective in pragmatics; therefore, readers who are interested in L2 pragmatics instruction should shift their attention from exploring whether teaching is helpful in pragmatics to how to teach pragmatics effectively. This is the topic of the next section.

3.2 Approaches in Second Language Pragmatics Teaching

To date, the most investigated teaching approaches in L2 pragmatics have been explicit and implicit teaching. Researchers adopt definitions given by Reference Kasper, Rose and KasperKasper (2001), in which explicit teaching involves metapragmatic explanations of pragmalinguistic uses and/or sociopragmatic rules of target pragmatic features, while implicit teaching does not include such metapragmatic explanation but rather employs various methods to facilitate learners’ unconscious reflection on pragmatic uses of target pragmatic features by themselves.

Implicit teaching consists of a range of different approaches, of which some focus on input, some on output, and others on interaction in completing tasks. This section starts with studies on interventions in terms of input enhancement, which refers to the manipulation of input to make specific target features more salient to learners (Reference Sharwood-SmithSharwood-Smith, 1993), either phonologically or typographically. In L2 pragmatics teaching, input enhancement is manipulated through various forms such as input flood, textual enhancement, and consciousness raising, which are at different positions along the implicit–explicit continuum. As many studies combine different teaching approaches in various ways, it may be more beneficial to investigate the instructional methods separately. Therefore, I will begin with the most implicit tactic and move on to consider the less implicit ones. If two teaching interventions are combined, I will review them in terms of the less implicit teaching method.

3.3 Summary

The above sections demonstrate that pragmatics teaching contributes to learners’ pragmatic development, whether in more salient aspects such as pragmatic routines and speech act strategies or less salient aspects such as internal modifiers of speech acts. Therefore, the first question about whether pragmatic teaching is effective is often treated as a prerequisite for further investigation in L2 pragmatics teaching. As reviewed in this section, there are now fairly well-established teaching frameworks that have been found to be effective and can be used in bringing pragmatics to the classroom. Researchers now spend their efforts on exploring which approach and methods can bring about better and stronger effects in terms of developing learners’ pragmatic competence.

This line of research reveals a much more complex picture. Existing narrative reviews (Reference Kasper and RoseKasper and Rose, 1999; Reference RoseRose, 2005; Reference TaguchiTaguchi, 2011c), syntheses (Reference TaguchiTaguchi, 2015b; Reference Takahashi and TrosborgTakahashi, 2010) and meta-analyses (Reference Jeon, Kaya, Norris and OrtegaJeon and Kaya, 2006; Reference Plonsky, Zhuang and TaguchiPlonsky and Zhuang, 2019) all point to the superiority of explicit teaching interventions over implicit teaching. However, they also acknowledge that factors such as target pragmatic features, learner characteristics, and different degrees of the teaching method, along implicitness–explicitness and length of instruction, all have impacts on the efficacy of L2 pragmatics instruction. In addition, the lack of delayed tests in empirical studies fails to document the sustainability or longevity of instructional effects. Finally, as illustrated by the reviews in this section, many studies combine a wide variety of teaching activities. Quite often the explicit teaching group also receives various implicit teaching activities, particularly aiming to raise learners’ awareness of target features. The increasingly popular task-based pragmatics teaching also often includes metapragmatic explanation prior to task activities. Therefore, the findings regarding the efficacy of teaching approaches are far from conclusive, and more research is warranted to further explore this issue.

4.1 Pragmatic Aspects Assessed

To date, few L2 pragmatics studies explicitly state that they are assessing learners’ pragmatic competence. This may be partially due to the fact that assessing learners’ pragmatic competence at a certain point is close to measuring their pragmatic production and/or perception, which is the focus of studies on pragmatic learning (see Section 2). There is considerable overlap between these two fields in this sense. Reference Hudson, Detmer and BrownHudson et al. (1995) is one of the earliest studies to examine L2 English learners’ pragmatic knowledge of apology, request, and refusal using six measures, including open-ended written DCTs, oral DCTs, roleplays, multiple-choice DCTs, and self-assessment rating scales for the DCTs and roleplays. They operationalized the difficulty levels of items according to three sociopragmatic variables from Reference Brown and LevinsonBrown and Levinson’s (1987) politeness theory: social status, social distance, and ranking of imposition. A situation with higher social status difference and social distance between the speaker and hearer and a higher ranking for the imposition of a speech act was considered more difficult than a situation with lower levels of these three factors. This way of distinguishing the difficulty levels of items is very influential and has frequently been used in later studies (e.g., Reference LiuLiu, 2007; Reference TaguchiTaguchi, 2007b; Reference YounYoun, 2020a).

Compared with speech acts, more studies assess learners’ receptive pragmatic knowledge such as their comprehension of conversation implicature. For example, Reference RoeverRoever (2005) examined learners’ knowledge of implicature and routines using multiple-choice DCTs, with the implicature items adopted from Reference BoutonBouton (1994). Reference TaguchiTaguchi (2009b) illustrated how she designed multiple-choice items to evaluate learners’ comprehension of more and less implicatures in English based on two corpora. She found that comprehension of indirect refusals was easier than comprehension of routines. Questioning a lack of correspondence with empirical conversational data, Reference WaltersWalters (2009) showcased the development of a multiple-choice listening test to assess learners’ pragmatic comprehension in English, employing findings from conversation analysis. He operationalized learners’ pragmatic competence as the ability to understand three types of pragmatic actions: assessment responses, compliment responses, and pre-sequence responses. Walters acknowledged that the three actions did not represent learners’ overall pragmatic competence; the reasons for using these three types of actions were simply because they were well documented in the existing conversation analysis literature.

The above approach explicitly assesses learners’ pragmatic knowledge as an individual trait, but this has been criticized on the grounds that it fails to uncover learners’ pragmatic performance in real interaction. In recent years, many researchers have paid attention to assessing learners’ abilities to use language in interaction, often conceptualized as interactional competence (see a collection of studies in Reference Ross and KasperRoss and Kasper, 2013). Reference YounYoun (2020a) investigated how to analyze learners’ interactional competence quantitatively. Three interactional features were identified—length of interaction, engaging with interaction, and sequential organization—each consisting of various indices. Although the interactional features functioned differently across different roleplays, they were shown to be useful in operationalizing the construct of L2 pragmatic interaction. In another qualitative study, Reference YounYoun (2020b) showcased how learners jointly accomplished proposal sequences in roleplays. It was found that lower-level learners often initiated a proposal sequence without establishing a shared understanding relevant to upcoming actions, while higher-level learners typically employed various shift markers and stepwise transition when initiating and shifting actions. When responding to a proposal the higher-level learners recurrently used coherent topic organizations, but in terms of closing a proposal sequence learners across all proficiency levels were able to perform well.

The associations between high-stakes exams and real communication contexts have also been explored. Reference Seedhouse, Ross and KasperSeedhouse (2013) compared interaction, in terms of repair, sequence organization, and turn taking, in the IELTS Speaking Test (IST) and in L2 classrooms and academic classes at universities. The findings showed that the interaction in the three settings was organized to meet the specific institutional goals. Contrasting with the rigid structure of the IST, the interactional organization of L2 classrooms and university classes varied widely. For example, interviewers in the IST limited repair to clear up test takers’ misunderstanding of interview questions, while teachers and peers frequently provided repair, corrections, and evaluation in L2 classrooms and academic classes. Reference Roever and IkedaRoever and Ikeda (2022) investigated the extent to which the speaking section of the TOEFL iBT could provide information about the test-takers’ interactional competence. They asked two raters to evaluate participants’ roleplays against four criteria: language use for mitigation, social actions, engagement in interaction, and turn organization. The results suggested that the TOEFL iBT speaking section could only predict testees’ ability to use language in interaction to a limited degree. The authors suggested that the absence of recipient design, one of the crucial components of interactional competence, in TOEFL iBT might account for the dissociation between the two measures.

In line with the status quo in L2 pragmatics learning, learners’ interaction has rarely been investigated in the written modality (see Section 2.3). Reference Huth, Betz, Salaberry and KunitzHuth and Betz (2019) set out to assess learners’ interactional knowledge by testing generic practices of interaction in written formats, for instance, using an unscramble-the-sequence task for testing sequence organization in conversation closings, a multiple-choice completion task for projection in interactional quoting, and data analysis/peer discovery of connectors for repair. The authors argued that testing formats for learners’ interactional competence may not necessarily encourage learners to follow prescriptive normativity. Assessing L2 interaction did “not aim for the right answers, but highlight[ed] systematicity, context sensitivity and diversification” (Reference Huth, Betz, Salaberry and KunitzHuth and Betz, 2019: 348).

Dynamic assessment has also been employed to evaluate learners’ pragmatic production and comprehension. For example, combining conversation analysis and dynamic assessment, Reference NicholasNicholas (2020) used dynamically administered strategic interaction scenarios (D-SIS) to assess Japanese EFL learners’ difficulties and development when requesting during interaction. In a D-SIS the researcher performs one of the roles in a roleplay with the learner. Either the learner or the researcher can pause the task whenever they feel there is a challenging issue and begin a mediation procedure, an insertion sequence initiated after pausing the roleplay. Nicholas distinguished two types of learners’ mediation: requiring implicit assistance, indicating that the learner had efficiently oriented to or resolved an object, or extensive explicit mediation, indicating that the object was a particular problem for the learner (Reference Van Compernolle, Ross and Kaspervan Compernolle, 2013). Thus, by analyzing mediation in the D-SIS, researchers can identify learners’ pragmatic difficulties and trace their development in interaction. Reference Qin and Van CompernolleQin and van Compernolle (2021) reported on the design and administration of an online dynamic assessment of implicature comprehension in L2 Chinese; learners were classified into beginner, intermediate, and advanced levels, and the assessment was centered on indirect acceptance, refusals, and opinions. The results showed that while all the learners could improve with mediation, individual variation existed in their responsiveness to assistance. The authors claimed that the learners’ performances with mediation were more informative than independent performance in terms of assessing their pragmatic competence.

4.2 Validity

Validity is an important aspect of assessment, whether the research targets learners’ pragmatic knowledge or interaction. Validation studies aim to systematically investigate empirical evidence that “provides insights into the extent to which a test measures what it is supposed to measure, relative to its purpose and use” (Reference Timpe-Laughlin and ChoiTimpe-Laughlin and Choi, 2017: 21). Reference HaastrupHaastrup (1986) carried out one of the earliest studies to address this, dealing with the validity issues of assessing pragmatics in a structured oral interview in English with Danish children. Haastrup argued that teachers can engage children in a conversation or a simulation (similar to a roleplay) to assess their pragmatic competence, which was operationalized as the ability to carry out a range of speech acts. The validity of roleplays was analyzed in Reference Youn, Taguchi and KimYoun (2018a, Reference Youn2020a) based on conversation analysis, showing that carefully designed roleplays can elicit valid interactional data that reflects learners’ pragmatic competence.

The validity of different measurements has been compared. Reference Brown, Rose and KasperBrown (2001) investigated six types of measures in an EFL setting and a Japanese as L2 setting: a written DCT, a multiple-choice DCT, an oral DCT, a roleplay, a discourse self-assessment task, and a roleplay self-assessment. The results showed that the original English versions of the six pragmatics tests generally did not work as well as the Japanese translation versions, in terms of their reliability and the amounts of variance produced. The EFL tests were subject to stronger method effects than the JSL (Japanese as a Second Language) tests. The author discussed the practical advantages and disadvantages of the six types of tests and ranked the EFL and JSL results separately according to each of the following characteristics: easiness to complete, ease of administration and scoring, validity, reliability, variance, encourages oral language and self-reflection, and suitability for high-stakes decision.

Reference Hudson, Rose and KasperHudson (2001) examined the method effects of written DCTs, oral DCTs, and roleplays on three speech acts: requests, refusals, and apologies. The results showed that oral DCTs were slightly more difficult than roleplays and written DCTs, while roleplays seemed to measure different pragmatic constructs than the two DCTs. This discrepancy between DCTs and more interactive oral measurement techniques was echoed by Reference Walters, Ross and KasperWalters (2013), who suggested that the oral conversation analysis-informed tests and DCTs did not measure the same construct. Reference Brown, Alcón Soler and Martínez-FlorBrown (2008) reported reasonably high reliability estimates for the test instruments used in his earlier study (Reference Brown, Rose and KasperBrown, 2001; Reference Hudson, Detmer and BrownHudson et al., 1995) except for multiple-choice DCTs.

Reference HinkelHinkel (1997) argued that to increase the validity of a multiple-choice task, it must be constructed around actual situations in which the speech act was performed, supplemented by previous relevant research findings on the speech act. In addition, production responses from pilot DCTs should serve as the options for multiple-choice test design and development. These suggestions were adopted by Reference LiuLiu (2007), who developed a multiple-choice DCT to assess Chinese English learners’ apologies in five steps. The first was exemplar generation, in which Chinese students were asked to identify situations where apologies were needed. The second step was to check situation likelihood, which explored how likely it was that the situation would occur in the students’ life. The third step, metapragmatic assessment, asked the students to assess the sociopragmatic variables in each scenario. In the fourth step the scenarios were validated. Finally, multiple-choice options were developed by including alternative responses collected from students’ DCTs. The results of both Rasch analysis and learners’ verbal reports showed that the multiple-choice DCT was reliable and valid.

The validity of DCTs has been examined and challenged for a long time. They have been criticized for having low construct validity, since they cannot collect data representing the oral features of authentic discourse (Reference GolatoGolato, 2003; Reference Johnston, Kasper and RossJohnston et al., 1998; Reference RoseRose, 1994). For example, examining advice among English native speakers and Chinese EFL learners, Reference HinkelHinkel (1997) found that DCTs failed to elicit responses approximating actual speech acts. However, Reference Beebe, Cummings, Gass and NeuBeebe and Cummings (1996) documented that data elicited with DCTs were consistent with naturally occurring data in terms of main patterns and formulas. Reference Economidou-KogetsidisEconomidou-Kogetsidis (2013) compared written DCT and naturally occurring requests, which found that the written DCT requests approximated natural data in terms of directness and lexical modification. In addition, Reference Billmyer and VargheseBillmyer and Varghese (2000) examined the effects of enriching the content of situation prompts in DCTs on L1 and L2 speakers’ output. The results showed that the content-enriched prompts elicited more robust external modification and elaboration than content-poor prompts, “in much the same way as speech in natural spontaneous interaction” (Reference Billmyer and VargheseBillmyer and Varghese, 2000: 533). This is in line with Reference Beebe, Cummings, Gass and NeuBeebe and Cummings (1996). However, enriched DCT prompts are often long, which means that learners may find them cognitively demanding and difficult (Reference Roever, Boxer and CohenRoever, 2004).

Cognitive validity in an L2 pragmatics test is occasionally explored by examining what test takers do when completing the test, and whether the cognitive mechanisms involved in their test performances match the construct. Reference LabbenLabben (2016) examined the cognitive validity of DCTs and argued that written DCTs should be treated as a language test rather than a questionnaire, and should undergo a rigorous developmental process. Reference Youn and BiYoun and Bi (2019) analyzed learners’ reported strategy use in monologic speaking tasks and roleplays to investigate the cognitive validity of L2 pragmatic assessment. They found that higher-level learners were likely to report different types of strategies more frequently than lower-level learners. However, regardless of L2 proficiency levels, learners were aware of the importance of being polite and pragmatically appropriate. Distinct strategies were used depending on the different characteristics of monologic versus dialogic tasks, thus demonstrating that the pragmatic assessment tasks measured the intended constructs and echoing the findings in previous research (e.g., Reference Hudson, Rose and KasperHudson, 2001).

The above studies focus on assessing learners’ production, albeit through measurements that require learners to choose an option for production. Reference Timpe-Laughlin and ChoiTimpe-Laughlin and Choi (2017) aimed to collect evidence of construct validity for a test of receptive pragmatic competence, using a web-based assessment measuring learners’ pragmatic comprehension. The test consisted of tasks targeting learners’ knowledge of the speech acts of requests and offers, routine formulae, and culture-dependent phrases and idioms. The results were quite positive, indicating that the test scores could reliably measure university students’ pragmatic knowledge and provide instructors with useful and meaningful information about the students’ receptive pragmatic competence.

4.3 Rating

It is challenging but crucial to develop criteria that measure learners’ pragmatic competence. Since appropriateness varies according to contexts and potentially in different raters’ reactions, rater disagreement is likely and even expected in pragmatics assessment. Therefore, it is necessary to develop a reliable and valid rating scale to objectively evaluate learners’ L2 pragmatic performance.

Several studies developed their rating scales a priori, sometimes with reference to previous literature or a needs analysis (Reference YounYoun, 2018b). For example, Reference Hudson, Detmer and BrownHudson et al. (1995) used five-point scales on six dimensions, asking participants to evaluate learners’ production collected by DCTs from very unsatisfactory to completely appropriate; the dimensions included the ability to use the correct speech act, typical expressions, amount of speech and information, and levels of formality, directness, and politeness. Reference SasakiSasaki (1998) asked two trained native speakers to evaluate learners’ pragmatic production in DCTs and roleplays in terms of appropriateness and grammar/structure. Although she also used fluency for roleplays, this dimension was not reported. Appropriateness had four levels, ranging from least appropriate/acceptable (scores 1–2), not really appropriate but acceptable (scores 3–5), appropriate to some degree (scores 6–8), and most appropriate for the given situation (scores 9–10) (Reference SasakiSasaki, 1998: 465).

Meanwhile, Reference TaguchiTaguchi (2007b: 121) used a six-point simplified appropriateness rating scale to ask six native speakers to evaluate learners’ performance on a pragmatic speaking task. Specifically, the zero point indicated no performance; a rating of 1 indicated very poor performance, where responses were very difficult to understand and no intended speech acts were performed; a rating of 2 indicated a poor performance, where appropriateness was difficult to judge; a score of 3 indicated a fair performance, where expressions were somewhat appropriate; a score of 4 showed a good performance, where expressions were mostly appropriate; and a score of 5 points indicated an excellent performance, where expressions were fully appropriate. This rating scale has frequently been adopted or adapted in later studies to measure learners’ pragmatic performance (e.g., Reference Levkina, Taguchi and KimLevkina, 2018; Reference LiLi, 2014; Reference TaguchiTaguchi, 2011b). Reference Kuiken and VedderKuiken and Vedder (2017) developed a six-point rating scale to assess to what extent a text fulfilled the communicative function intended. The rating scale consisted of four dimensions, including content, task requirements, comprehensibility, and coherence and cohesion. Finally, Reference Kley, Salaberry and KunitzKley (2019: 318) proposed a rubric for evaluating learners’ interactional competence using a three-point rating scale with four criteria: initiating new topics, reciprocating the interlocutor’s topic-initiating questions, shifting between topics, and expanding on topics.

Unlike the above ratings analyzing learner responses on different dimensions, Reference WaltersWalters (2007) deliberately designed a four-point holistic rating scale to evaluate learners’ oral production of assessment responses, compliment responses, and pre-sequence responses. The rubric was vague, with a score of 1 indicating no evidence of control of assessment responses, 2 indicating more evidence of no control, 3 indicating more evidence of control, and a score of 4 indicating evidence of control (Reference WaltersWalters, 2007: 168). It is worth noting that Walters stated that the vagueness of the rubric was intentional, given the exploratory nature of applying conversation analysis to rating pragmatic performance.

A couple of studies generated their rating scales based on analyses of learners’ pragmatic performance, thereby increasing the validity of the rating scales. Reference YounYoun (2015) developed five rating categories for interactive open roleplays to be used in classroom assessment in an academic English context, including content delivery, language use, sensitivity to situation, engaging with interaction, and turn organization. Content delivery measures learners’ ability to speak appropriately and fluently in a turn; language use measures a range of pragmalinguistic strategies; sensitivity to situation measures the learners’ sociopragmatic perceptions and awareness; engaging with interaction measures the degree of learners’ engagement and coordination with the interlocutor in roleplays; and turn organization measures learners’ orientation to normative turn-taking conventions. Quantitative analyses confirmed the unique contribution of each of the five categories in measuring learners’ pragmatic competence in interaction.

Reference LiuChen and Liu (2016) developed and validated a rating scale to evaluate the speech act performance of apology and request by intermediate Chinese EFL learners in written DCTs. In the scale development stage, initially native speaker raters without training were asked to judge the content and form of DCT productions by American English native speakers and Chinese EFL learners across five hypothesized ability levels, ranging from very poor, poor, fair, good, to excellent along a 10-point Likert scale with a 2-point margin between each level, and write comments for each response. Next, the raters’ comments were coded and the DCTs were classified into five sub-corpora consisting of scripts banded into the five hypothesized levels. A detailed analyses of the rater comments and the DCT productions resulted in a rating scale consisting of two 10-point subscales (for a detailed description, see Reference LiuChen and Liu, 2016: 239). In the validation stage, seven trained raters were asked to rate another group of Chinese EFL learner’ email productions, and a FACETS analysis was conducted to examine the validity and reliability of the rating scale. The results showed that the rating scale could reliably differentiate the learners’ email production. In terms of validity, the results showed that some levels of the 10-point scale were disordered, and the 2-point interval for each ability level confused the raters because it was difficult for them to tell the difference between the two scores at the same level.

In addition to developing rating scales, researchers have also investigated rater variation in evaluating learners’ pragmatic performance. Reference WaltersWalters (2007) found that a native English rater and a non-native English rater interpreted learners’ performance differently, with the non-native rater paying more attention to fluency and pronunciation than the native speaker rater. Reference Tajeddin and AlemiTajeddin and Alemi (2014) investigated native English teachers’ ratings of EFL learners’ apology DCTs, and whether there was rater bias in their ratings. A content analysis of the raters’ descriptions revealed that they mostly paid attention to expressions of apology, explanations, offers of repair, promises for the future, and politeness. Quantitative analyses via FACETS indicated that the raters showed different degrees of severity and tolerance in their ratings.

The influences of raters’ cultural and linguistic backgrounds on their ratings have also been explored. Reference TaguchiTaguchi (2011b) investigated rater variation in the judgment of speech acts by analyzing judges’ verbal interview data. Four native English speakers from different cultural backgrounds – one African American, one Asian American and two Australians – assessed the appropriateness of requests and opinions produced by Japanese EFL learners. The findings showed that the raters evaluated the appropriateness of speech acts differently; some focused more on pragmalinguistic aspects such as the directness level of head acts and the use of politeness markers, while others paid more attention to non-linguistic aspects such as the content of politeness strategies. Even when raters focused on the same dimensions of pragmatic performance, they varied in their degree of tolerance – for example, in their level of acceptance of length of low-imposition requests.

Reference Sonnenburg-Winkler, Eslami and DerakhshanSonnenburg-Winkler et al. (2020) carried out one of the few studies exploring the relationship between linguistic background and pragmatics ratings. This study investigated variability among raters from different L1s by asking ten participants to evaluate written request DCTs produced by their peers and themselves using the rating scale developed by Reference TaguchiTaguchi (2011b). The raters were also asked to provide an explanation for their rating. The findings showed that the raters considered a variety of dimensions when evaluating the speech acts, with some raters focusing more on the directness levels of expressions, while others based their decisions on non-linguistic aspects such as the content of reasons for requests. This finding echoed the rater variation documented in Reference TaguchiTaguchi (2011b). In addition, raters from the same L1 tended to rate productions produced by speakers of the same L1 similarly.

4.4 Summary

Second language pragmatic assessment is an emerging and crucial field, particularly concerning the validity and reliability in assessing and measuring learners’ pragmatic competence in the learning and teaching studies. Validity is indeed one of the key topics in pragmatic assessing research, and the reliability of rating is also an often-examined issue. However, the L2 pragmatic assessing literature should be expanded to explore more issues that are well examined in language assessment but are much neglected in pragmatic assessment, such as washback, assessment literacy, theoretical models, and computer assisted assessment. In addition, learners’ cognitive processes during completing tasks are key to understanding their test performance and the cognitive validity of assessment tools. In the next section, the role of cognitive process is discussed as related to pragmatic learning, pragmatic teaching, and pragmatic assessing.

5.1 Instruments to Measure Cognitive Processes in Pragmatics

Verbal reports as a form of introspection solicit verbalization data about the thought processes of participants during the completion of a task (Reference Cohen, Félix-Brasdefer and KoikeCohen, 2012; Reference Gass and MackeyGass and Mackey, 2016). Verbal reports aim to provide insights into the cognitive processes behind learners’ written or spoken behaviors during linguistic performance, which otherwise would have to be investigated indirectly (Reference CohenCohen, 1998). There are two types of verbal reports: concurrent verbal reports (CVR), also referred to as think aloud protocols, and retrospective verbal reports (RVR), which are implemented subsequent to a task and prompt learners to report on the thoughts they had during task completion. In one of the earliest studies examining learners’ cognitive processes in L2 pragmatics, Reference Robinson and KasperRobinson (1992) showed that when designed and executed with care, particularly in combination with other data collection methods, both CVR and RVR can provide in-depth insights into learners’ pragmatic knowledge and their pragmatic difficulties in the planning and execution of speech acts.

Since then a few studies have used CVR, but the majority of the studies on cognitive process in L2 pragmatics have employed RVR. This is understandable as most L2 pragmatics studies examine learners’ oral performance, which means they cannot perform CVR simultaneously. RVRs have most often been combined with roleplays (e.g., Reference Cohen and OlshtainCohen and Olshtain, 1993; Reference Woodfield, Felix-Brasdefer and KoikeWoodfield, 2012; Reference Ying and RenYing and Ren, 2021), but other data collection methods that have been combined with RVRs include conversation elicitation tasks (Reference NguyenNguyen, 2017), comprehension questionnaires (Reference TaguchiTaguchi, 2008b), written DCTs (Reference BellaBella, 2014a), and multimedia elicitation tasks (Reference RenRen, 2014, Reference Ren2015). In contrast, CVRs have been combined with various types of questionnaires, including yes/no questions (Reference TaguchiTaguchi, 2002), appropriateness judgment questionnaires (Reference Timpe-Laughlin, Green and OhTimpe-Laughlin et al., 2021), and multiple-choice questionnaires (Reference Chen and LinChen and Lin, 2021).

Three studies employed both CVR and RVR. Reference Robinson and KasperRobinson (1992) combined both types of verbal report with DCTs to investigate Japanese English learners’ pragmatic knowledge in producing refusals. Reference WoodfieldWoodfield (2010) innovatively explored learners’ cognitive processes when they were engaged in DCTs by asking them to conduct verbal reports in pairs. The results showed that paired verbal reports were able to provide insights into learners’ cognitive processes and pragmatic knowledge. In addition, paired CVRs were easier and more natural for learners than single-subject CVRs. Following Reference WoodfieldWoodfield (2010), Reference ChenChen (2015) also combined the two types of verbal report to explore learners’ cognitive processes while they were completing an email task.

5.2 Pragmatic Aspects Examined with Cognitive Processes

Research has employed RVRs to investigate learners’ cognitive processes during their production of speech acts, among which requests (e.g., Reference ChenChen, 2015; Reference Cohen and OlshtainCohen and Olshtain, 1993; Reference WoodfieldWoodfield, 2010, Reference Woodfield, Felix-Brasdefer and Koike2012) and refusals (e.g., Reference BellaBella, 2014b; Reference Félix-BrasdeferFélix-Brasdefer, 2008a; Reference RenRen, 2014) have been the most frequently examined. Other speech acts that have been examined include apology (Reference BellaBella, 2014a; Reference Cohen and OlshtainCohen and Olshtain, 1993), complaint (Reference Cohen and OlshtainCohen and Olshtain, 1993; Reference Hassall, Alcon Soler and Martinez FlorHassall, 2008), criticism and responses (Reference NguyenNguyen, 2017), feedback (Reference Youn and BiYoun and Bi, 2019), greeting responses (Reference Ying and RenYing and Ren, 2021), and offers (Reference BellaBella, 2016).

A couple of studies have employed CVRs to explore learners’ cognitive processes while they are performing comprehension tasks involving conversational implicature (Reference Chen and LinChen and Lin, 2021; Reference TaguchiTaguchi, 2008b). Reference Timpe-Laughlin and ChoTimpe-Laughlin and Cho (2021) used CVRs to investigate the effectiveness of a learning platform in developing EFL learners’ awareness of requests and offers.

5.3 Verbal Report Data Examined

Most L2 pragmatics research on cognitive processes has collected and analyzed data based on Reference Newell and SimonNewell and Simon’s (2019 [Reference Newell and Simon1972]) theory of problem-solving (e.g., Reference RenRen, 2015; Reference WoodfieldWoodfield, 2010) in order to explore the three types of problem-solving processing posited by the model: orient, solve, and evaluation. The first of these, orientation, is often investigated by asking about learners’ cognition or attention through questions such as: What did you notice about the situation? What were you paying attention to? What were you focusing on? (e.g., Reference Alcón SolerAlcón Soler, 2012; Reference RenRen, 2014; Reference WoodfieldWoodfield, 2010). Second, solution is often investigated by asking about learners’ intentions or planning through questions such as: What did you intend to convey? (Reference ChenChen, 2015) What were you trying to achieve? (Reference BellaBella, 2016) Why did you choose the answer? (Reference TaguchiTaguchi, 2008b) What made you reply in this manner? (Reference RenRen, 2014) To what extent was it important for you to be direct or indirect when you replied in this situation? (Reference RenRen, 2015) Can you explain your decision-making process? (e.g., Reference Alcón SolerAlcón Soler, 2012; Reference Félix-BrasdeferFélix-Brasdefer, 2008a). The third stage, evaluation, is often investigated by asking learners to evaluate the appropriateness of their own pragmatic performance through questions such as: Were you satisfied with your answer? (Reference Alcón SolerAlcón Soler, 2012; Reference WoodfieldWoodfield, 2010) How would you evaluate your performance in … ? (Reference ChenChen, 2015; Reference Ying and RenYing and Ren, 2021).

Some studies have been interested in learners’ language of thought, asking learners to report on the language they used in planning and executing their performance (e.g., Reference Félix-BrasdeferFélix-Brasdefer, 2008a; Reference RenRen, 2015; Reference WoodfieldWoodfield, 2010). In addition, studies have explored learners’ difficulties and strategy use (e.g., Reference RenRen, 2014; Reference Youn and BiYoun and Bi, 2019) in completing pragmatic tasks based on the collected data.

5.4 Procedures of Verbal Report

When learners are asked to report their cognitive processes, either by CVR or RVR, they must be offered training and allowed to practice. Researchers should ask learners specific questions and avoid asking them cognitively complex questions. For example, questions asking learners to explain complicated issues or to compare abstract concepts such as “Which social parameter carries more weight?” should be avoided.

The time span between RVR and pragmatic tasks, whether they are production or perception tasks, should be kept as short as possible. The majority of L2 pragmatics studies conduct RVR immediately after the pragmatic task, which is highly recommended. In cases where RVR cannot be conducted immediately, it is suggested that it should be carried out within two days (Reference Ericsson and SimonEricsson and Simon, 1993). In addition, learners’ performance in the pragmatic task should be replayed to them to help them recall what they did during the task. Researchers should avoid checking learners’ memories, since questions asking learners to recall what they were thinking during data collection in the past can cause a serious validity threat. If researchers wish to investigate changes in learners’ cognitive processes, RVRs should be conducted after each session, as in Reference RenRen (2014). It is not recommended to conduct RVR only in the last phase of data collection if the intention is to explore pragmatic variation across different phases, as Reference WoodfieldWoodfield (2010) admits in her discussion of limitations.

With respect to the mode of RVR, Reference Hernández and BoeroHernández and Boero (2018) carried out the only study to date that asked learners to write down their thoughts while they were completing pragmatic tasks. As writing is usually much slower than speaking and only offers learners the chance to access their conscious thoughts, it may involve further threats to validity. Future research could investigate the effects of modality on the data collected by verbal report.

5.5 Summary

Verbalization of learners’ cognitive processes can provide insights into the reasoning and thought processes behind their performance (e.g., production of certain pragmatic features, awareness of speech acts, and comprehension of implicature). When carefully designed, verbal report data on learners’ cognitive process can reveal useful information including the pragmatic difficulties they encounter, the source of knowledge they have, the perception of different instruction approaches and measurement tools, and whether their performance resulted from the lack of pragmatic development or their pragmatic agency. Therefore, investigating learners’ cognitive process during pragmatic activities can provide added value for advancing our understanding of L2 pragmatic development. In sum, cognitive process research can integrate studies on learning, teaching, and assessing different aspects of learners’ pragmatic production and perception to provide a comprehensive picture of L2 pragmatic competence.

6.1 Case Study 1: Second Language Self-praise on Social Media

As discussed in Section 2.3, L2 pragmatics has mainly focused on oral communication, with communication in other modalities being examined less frequently. The field needs to expand its research scope to include more pragmatic features, particularly concerning semiotic and multimodal resources. It is important to examine whether L2 learners can successfully achieve meaning making on social media with the increasingly universal digital affordance. Against this background, I designed a project to investigate instances of self-praise posted on social media by advanced learners of Chinese. Self-praise is a speech act through which interlocutors present positive content of themselves such as appearance, possession, and skills (Reference Ren and GuoRen and Guo, 2020). The reason why self-praise was chosen as the research target is that the phenomenon is pervasive in individuals’ daily lives as well as on social media, but it has not been examined from an L2 pragmatics perspective.

6.2 Case Study 2: Lingua Franca Pragmatics

Lingua franca pragmatics refers to the pragmatic practice of ethnic minority people in their national lingua franca. It is different from pragmatics in English as a lingua franca (e.g., Reference RenRen, 2018b). As pointed out earlier, L2 pragmatics has predominantly examined adult learners. Children’s pragmatic competence, particularly that of ethnic minority children in a multilingual country or area, has rarely been explored. Therefore, I developed a project to investigate Chinese ethnic minority children’s pragmatic competence in Mandarin, the national lingua franca in China. I collected data from local children in primary schools, junior high schools, and senior high schools in ethnic minority areas in China including Xinjiang, Tibet, and Inner Mongolia. Due to space limitations, here I only report on one study examining Uyghur-Mandarin children’s pragmatic competence when giving advice in written communication, to provide more data to the under-researched written pragmatic competence (see Section 2.3).