1. Introduction
Despite decades of efforts to increase diversity, women and gender minorities remain underrepresented in many STEM fields. In 2021, women made up about half (48%) of the U.S. total workforce but only 27% of the STEM workforce. In engineering, this number is generally even lower, with women receiving only 22% of bachelor’s degrees (National Center for Education Statistics, 2022) and making up 10% of the workforce (U.S. Census Bureau, 2022). Similarly, the presence of LGBTQ+ minorities, which includes other gender minorities, is 17-21% lower in STEM than in the general populous (Reference FreemanFreeman, 2020). These figures are concerning, particularly because many women show interest in scientific careers during their early collegiate years. The declining percentage of women throughout the STEM pipeline has been attributed to two primary factors: first, the presence of societal stereotypes that create systemic barriers or affect sense of belonging (Reference Henderson, Bloodhart, Adams, Barnes, Burt, Clinton, Godfrey, Pollack, Fischer and HernandezHenderson et al., 2022; Reference Ryan and MorgenrothRyan and Morgenroth, 2024; Reference Terenzini, Springer, Yaeger, Pascarella and NoraTerenzini et al., 1996; Reference BroadleyBroadley, 2015; Reference Fussy, Iddy, Amani and MkimbiliFussy et al., 2023); and second, a perception that STEM fields do not align with values of helping others and contributing to the community (Reference Diekman, Steinberg, Brown, Belanger and ClarkDiekman et al., 2017, Reference Diekman, Brown, Johnston and Clark2010; Reference PhillipsPhillips, 2024).
Social role theory helps explain this phenomenon, since gendered social roles influence individual goals and motivations (Reference Eagly and WoodEagly and Wood, 2012). Women are traditionally socialized to focus on caregiving and family (community), which permeates into professional life: women gravitate toward careers that they perceive as having communal goals such as social impact, whereas men tend to be socialized to focus on personal achievement or career attainment, which are agentic goals. This alignment between personal goals and social goals is referred to as goal congruence. Since some STEM fields (including mechanical engineering) are perceived to be lacking communal opportunities, women tend to display lower interest and engagement in these careers (Reference Diekman, Brown, Johnston and ClarkDiekman et al., 2010). Other STEM fields, such as environmental engineering (32% women) and medicine (37% women) (U.S. Census Bureau, 2022; Reference Young, Pei, Arnhart, Carter and ChaudhryYoung et al., 2023), tend to have a more balanced gender representation (Reference Cheryan, Ziegler, Montoya and JiangCheryan et al., 2017). Even within the field of mechanical engineering, the proportion of women participating in professional development and research-intensive activities is not uniform. Recent work in this space demonstrated that in one design engineering research conference, more women than men participated in design theory tracks (59% women), while mechanisms and robotics tracks had a participation of women of about 9% (Reference Running, Parkinson, Magleby and TohRunning et al., 2024). More work is needed to understand the nature and distribution of socially relevant research in the engineering design discipline and to investigate how this type of research is operationalized in the artifacts that are produced (e.g., sxresearch papers and conference presentations).
One opportunity for investigating the affordance for pursuing socially relevant goals in a discipline is through analyzing peer-reviewed research articles published by relevant venues. Peer-reviewed conference papers and journal articles are an expression of the outcomes of researchers’ efforts and have been vetted by other members of the community as having value and contributing a significant intellectual product to the discipline. Researchers in other fields have previously used bibliometric analysis to understand trends and dynamics in publishing norms and their potential relationship to persistent underrepresentation of women and other minorities broadly (Reference Van Bommel, Hubers and MaasVan Bommel et al., 2024) as well as in specific scientific disciplines, including computer science (Reference De Nicola and D’AgostinoDe Nicola and D’Agostino, 2021) and anesthesiology (Reference Duggan, Atwood, Sanford, Tsai, Egbaria, Carmichael-Tanaka and OutlandDuggan et al., 2024). The advantage of using bibliometric analysis to this domain is the ability to track changes in research topic activity across time and to aggregate results from disparate research communities to understand broader phenomena. Design researchers have begun to apply this technique to great effect (Reference Delen and YukselDelen and Yuksel, 2022; Reference Chai and XiaoChai and Xiao, 2012; Reference Perry and PereiraPerry and Pereira, 2023). However, a barrier to performing a systematic investigation of the quantity and type of socially relevant research in the engineering design discipline is the lack of a framework and method for analysis that can be used across publication venues. Thus, the goal of this paper is to present a framework and protocol for systematically identifying the presence of socially relevant and communal-oriented research in engineering design disciplines through bibliometric analysis. This can serve as a first step toward understanding affordances to pursue socially relevant and communal research across disciplines. This goal is addressed through the following objectives:
-
1. Develop and present a framework for analyzing scholarly work through the lenses of social relevance, goal type (agentic vs. communal), and goal function (advancing vs. fortifying).
-
2. Provide a protocol for systematically identifying social relevance, goal type, and goal function within published research papers in engineering design communities. The protocol is demonstrated using two example peer-reviewed research papers.
This work provides a foundation for understanding the complex social and disciplinary forces that influence the participation of women and other minorities in STEM disciplines that have lagged behind those in the social, medical, and behavioral sciences in terms of gender parity.
2. Methods and analysis
This framework was iteratively developed over the course of bibliometric analysis of two peer-reviewed paper corpora. Using the principles of action research, the procedural details and methodological approaches were developed by the authors of this paper via bibliometric analysis and deep reflection of this process. The main goal of understanding the participation of women and other minorities in engineering design research communities guided the development of this analytical framework.
First, the social relevance of peer-reviewed research papers published in the conference proceedings of the American Society of Mechanical Engineering (ASME)’s International Design Engineering Technical Conferences-Computers and Information in Engineering Conference (IDETC-CIE) in 2022 was analyzed (N = 151). IDETC-CIE contains several tracks, including Mechanisms and Robotics (MR) and Design Theory and Methodology (DTM). Analysis of these two communities revealed vastly different gender demographics and research interests represented in the proceedings of the conference tracks (Reference Running, Parkinson, Magleby and TohRunning et al., 2024). The iterative process described in this prior work provided a foundation for the understanding of coding for social relevance.
Second, to extend the analytical framework to include not only social relevance but also goal type and function, papers published in ASME’s Journal of Mechanical Design (JMD) in 2023 were analyzed (N = 145). The JMD experiences some overlap with authors presenting their work at IDETC. The analysis process for the JMD corpus is described in Figure 1. Only the research objectives and conclusion sections were used for this analysis, similar to what had been done in previous work (Reference Running, Parkinson, Magleby and TohRunning et al., 2024). First, papers were coded collaboratively to develop a shared understanding of the codes. This understanding was further developed during separate, individual coding, where papers that did not clearly fit into the initial code definitions were coded by the authors collaboratively and definitions, operational details, and examples of each coding category were formalized into a codebook. In addition, decision-making aids were created to establish a shared understanding of the coding categories between authors. Once the codebook and decision-making aids were developed, the authors used them to guide the analysis of remaining papers.
Figure 1. The iterative coding process used to develop the bibliometric analysis framework
Finally, inter-rater reliability was measured using Cohen’s Kappa. A random selection of 30 papers (20.8% of the full set) were cross-reviewed. The author agreement for whether the paper had social relevance was 90%. A Cohen’s Kappa (Reference Hsu and FieldHsu and Field, 2003) analysis of these codes gives κ = 0.651, which indicates moderate agreement. For the papers that both authors coded as having social relevance (n = 1), the authors also agreed on the category and whether a minoritized group was mentioned with a rate of 100%. The author agreement for whether one of the thirty papers had an agentic goal, communal goal, or neither, was 100% (κ = 1.00: almost perfect agreement). For the papers that both authors coded as having an agentic or communal goal (n = 10), the authors also agreed on the category with a rate of 100% (κ = 1.00: almost perfect agreement). The author agreement for whether the goal was advancing or fortifying for these 10 papers was 80% (κ = 0.583: moderate agreement). This inter-rater reliability process indicates that the most inconsistent codes were for the advancing versus fortifying code. This code requires the deepest understanding of the content of the paper and a firm grasp of the current work in related spaces. The second-lowest agreement value was for social relevance, since there is variability in how to conclude implications for social relevance based on the results of the work. From our experience, coding a single paper might take anywhere from one to ten minutes depending on the amount of understanding required to make the code.
3. Framework for analyzing research goals
The result of the iterative process outlined in the Section 2 is a framework for understanding research goals and their function in engineering design research papers, that consists of the following two components: 1) The type of social relevance of the research, and 2) The goal type and function addressed in the paper. In the next sections, we discuss the definitions of these characteristics and their relationship to one other in the context of understanding the perceived affordance of pursuing pro-social goals in engineering research communities.
3.1. Social relevance of research
Our findings show that the social relevance of research papers in the engineering design discipline can vary significantly in focus and intended outcomes. Our categorization scheme is adapted from one first presented by Rainock et al. Reference Rainock, Everett, Pack, Dahlin and Mattson(2018) for understanding the social impact of engineering products. Our framework adapts this categorization scheme to be applied to peer-reviewed research artifacts since applied engineering work is often documented in publications. However, the use of these existing categories may exclude research artifacts not included in this list but which display an unlisted type of relevance. This approach can also exclude projects that have social relevance that is not described in the published paper or include papers that have social relevance stated in the paper but not in application. These are risks inherent with the limits of bibliometric analysis that will be mitigated through careful and reflective coding of papers from different venues. With this in mind, the social relevance of an article is assessed by analyzing its alignment with the following codes/themes:
-
Social Relevance: Whether an article’s research goals or contributions have explicitly addressed their impact on society (yes/no).
-
Area of Focus: Each paper is screened for a specific area of focus of social relevance. These areas of focus along with their definitions are adapted from Rainock et al. Reference Rainock, Everett, Pack, Dahlin and Mattson(2018) and shown in Table 1.
-
Explicit Mention of a Minoritized Group: The specific minoritized group that the authors explicitly invoke in their writing as beneficiaries of their socially relevant research goals, such as the elderly, differently-abled people, or populations from developing countries. If no specific minoritized group is invoked, then “N/A” is indicated.
Table 1. Categories used to determine social relevance, adapted from Rainock et al. Reference Rainock, Everett, Pack, Dahlin and Mattson(2018) and Running et al. Reference Running, Parkinson, Magleby and Toh(2024)
3.2. Coding goal congruence
The types and function of goals addressed by each article is assessed by analyzing its alignment with the following codes/themes:
-
Agentic/Communal Goal: Each paper is examined to determine whether it pursues an agentic (selffocused) or a communal (pro-social) goal, with categories adapted from Diekman et al. Reference Diekman, Brown, Johnston and Clark(2010). Table 2 and Table 3 provide the goal congruence categories as well as our added definitions.
-
Advancing/Fortifying Function: Each paper is examined to determine whether the goal of the research is to introduce new concepts, methods, or theories to explain currently observed phenomena (advancing) or to improve existing methods, advance current theories, and deepen understanding of current phenomena (fortifying).
Table 2. Communal goal congruence categories drawn from Diekman et al. Reference Diekman, Brown, Johnston and Clark(2010). The definitions were developed by the authors over the course of coding
Table 3. Agentic goal congruence categories drawn from Diekman et al. Reference Diekman, Brown, Johnston and Clark(2010). The definitions were developed by the authors over the course of coding
For the coding of goal type, categories were adapted from Diekman et al. Reference Diekman, Brown, Johnston and Clark(2010). Adjustments to goal type categories were made to increase the distinction between similar goal types (e.g., serving community vs serving humanity). It is possible that the papers analyzed included goal types not captured in the original list, or that the application of the research indicated a goal not represented in the paper.
The advancing and fortifying functions were developed by the authors to further interpret agentic and communal goals expressed in papers. Since individuals belonging to different social groups may choose to pursue their goals using different strategies and paths-to-power, this additional dimension allows for further granularity in the analysis. For the purposes of our bibliometric analysis, advancing research is adapted from Bernstein Reference Bernstein(1999)‘s “Horizontal” discourse, which looks to other disciplines or uses new concepts or approaches to explain existing phenomena. The work focuses on applying these novel concepts, typically in an attempt to explain something previously unexplainable using existing concepts in that discipline, or is in response to shifting contexts/requirements in that field. Classic examples in our corpus were the development of new computational techniques to solve engineering problems, application of theories from other disciplines to improve process, or the explicit shifting of priorities to different goals. Similarly, fortifying research is adapted from “Hierarchical” discourse, where the work uses existing concepts, theories, or methods and adds to our understanding of how they work, refines them to be more effective, or increases formalization of established processes. Classic examples in our corpus were research papers that sought to explore a particular cognitive reasoning process in existing design methods or work that focused on increasing the performance of certain algorithms or manufacturing processes.
Figure 2 shows a graphical view of advancing vs fortifying functions and their relationship to agentic vs communal goals. The placement of each quote in its respective quadrant can be understood as follows:
-
Agentic-Fortifying (Quadrant I): This has an agentic focus (independence) because it attempts to understand how much designers trust AI tools versus their own ideas in the design process. It is fortifying because it seeks to improve understanding of an existing concept: AI-assisted design.
-
Agentic-Advancing (Quadrant II): This has an agentic focus because it seeks to help non-experts succeed in solving the path synthesis problem. It is advancing because it introduces a new method for the solution.
-
Communal-Advancing (Quadrant III): This has a communal focus (working with people) because it seeks to better understand the empathy link when users and designers work with one another. It is advancing because it uses a new method to accomplish this.
-
Communal-Fortifying (Quadrant IV): This has a communal focus for the same reasons as Quadrant III. In this paper, however, the researchers use existing methods to try to better understand the user-designer empathy link, making it a fortifying approach.
Figure 2. A demonstration of how agentic and communal goals might be categorized as advancing or fortifying
4. Coding protocol
In this section, we present the procedure and operational details related to enacting this framework using peer-reviewed research articles as the unit of analysis. Once a research article has been retrieved and the appropriate analysis platform has been selected, the following steps are performed in order:
-
Step 1: Extract the research objectives and conclusion of the articles.
Note: This step was performed manually for our corpora but can be automated using an LLM or a keyword search for “objectives”, “in this paper”,“purpose”,“conclusion”, or “summary”.
-
Step 2: Assess social relevance category using Table 1.
-
a. Does the text explicitly mention social relevance as defined in the table?
-
b. What category does it mention?
-
c. Does the text explicitly mention a minoritized group?
-
-
Step 3: Assess goal type using Tables 2 and 3.
-
a. Does the text focus on an individual or type of individual?
-
b. Does the text mention an agentic goal as defined in the table?
-
c. Does the text focus on a community?
-
d. Does the text mention a communal goal as defined in the table?
-
e. Does the text indicate that the authors had an advancing or fortifying goal in mind?
-
We present this procedure using examples of two research articles with differing focus and disciplinary areas. Lastly, we develop a decision-making aid for conducting this analysis in a consistent and systematic manner using multiple analysts (referred to as coders).
4.1. Protocol example 1: Sun et al. Reference Sun, Li, Zi and Chen(2023)
-
Step 1: Extract the research objectives and conclusion of the articles.
-
a. Research objectives: “The primary purpose of this study is to develop a KAO to realize lower limb knee flexion/extension and ankle dorsiflexion/plantar flexion. The KAO has a variety of working modes, which can provide driving force and reverse damping force during patient rehabilitation training.”
-
b. Conclusion: “In this paper, a KAO driven by the SMA actuators and DC motor is designed according to the needs of passive rehabilitation and muscle enhancement training for lower limb rehabilitation patients.”
-
-
Step 2: Assess social relevance.
-
a. Does the text explicitly mention social relevance as defined in the table? Yes. The objectives and conclusion explicitly state social relevance of the research: to improve patient rehabilitation training. There is therefore social relevance.
-
b. What category does it mention? Health and safety. Patient rehabilitation training falls in the “Health and Safety” category of social relevance: the goal is to improve the health or well-being of an individual.
-
c. Does the text explicitly mention a minoritized group? Yes: patients. Patients by definition have a medical need and are thus considered a minoritized group.
-
-
Step 3: Assess goal type.
-
a. Does the text focus on an individual or type of individual? Yes: patients.
-
b. Does the text mention an agentic goal as defined in the table? No. The individual mentioned—patients—do not demonstrate any of the agentic goals. This indicates that there is no agentic goal present.
-
c. Does the text focus on a community? Yes: patients.
-
d. Does the text mention a communal goal as defined in the table? Yes: helping others. The goal of the authors is to help people within the community of patients.
-
e. Does the text indicate that the authors had an advancing or fortifying goal in mind? Advancing: the authors develop a new method to solve an existing problem: improving lower limb rehabilitation for patients.
-
4.2. Protocol example 2: da Silva Vieira et al. Reference da Silva Vieira, Gero, Delmoral, Gattol, Fernandes and Fernandes(2019)
-
Step 1: Extract the research objectives and conclusion of the articles.
-
a. Research objectives: “The objectives of the study reported here are to: [first] investigate the use of the EEG technique to distinguish design from problem-solving [and second] identify neurocognitive similarities and differences across design domains. We postulate… [that] design neurocognition of mechanical engineers and architects are different.”
-
b. Conclusion: “The results of this preliminary analysis… provide initial support for [hypotheses 1-3] … On a qualitative level the current study shows evidence of a distinct characteristic of increased Pow and TRP of design tasks from the reference problem solving task across mechanical engineers and architects.”
-
-
Step 2: Assess social relevance.
-
a. Does the text explicitly mention social relevance as defined in the table? No. None of the social relevance categories are explicitly discussed in the copied text. Skip to Step 3.
-
-
Step 3: Assess goal congruence.
-
a. Does the text focus on an individual or type of individual? Yes: mechanical engineers and architects.
-
b. Does the text mention an agentic goal as defined in the table? No: the subjects (mechanical engineers and architects) do not display any agentic goals.
-
c. Does the text focus on a community? Yes: mechanical engineers and architects.
-
d. Does the text mention a communal goal as defined in the table? No: the copied text does not indicate any communal goals.
-
e. Does the text indicate that the authors had an advancing or fortifying goal in mind? Skip: No mention of agentic or communal goals.
-
The flowchart we created as a decision-making aid for coding each paper is shown in Figure 3. After following the protocol, we recommend cross-checking between reviewers to ensure inter-rater reliability.
Figure 3. A decision-making aid for coding goal type and social relevance of research papers. Decision boxes are in light yellow and code boxes are in darker cyan
4.3. Positionality and interpretation
The development of a systematic analysis protocol as described in this paper helps to maintain reliability across multiple coders. However, individual researchers bring their own interpretive lens to this process, applying their unique perspectives and understanding of context to the analysis. We therefore recommend that researchers seeking to apply this framework and protocol disclose their own positionality so that readers can understand how these positions can influence the interpretation of the findings (Reference Hampton, Reeping and OzkanHampton et al., 2021). To that end, we disclose our own positions relative to the research communities and work analyzed in this paper. Each of the authors of this paper is a woman in STEM, with some representing mechanical engineering and others in information and technology. Our involvement in the field ranges from five years to three decades. Authors 3 and 4 have been involved in various leadership and service roles within the ASME and JMD communities. We also represent a variety of other minorities as defined by the demographics of our country of residence, universities, and departments.
5. Preliminary results and discussion
While the focus of this paper is on the framework itself, we also provide some brief results for demonstration purposes. Results and discussion on the social relevance coding of papers from IDETC are available in Running et al. Reference Running, Parkinson, Magleby and Toh(2024). Results of social relevance, goal type, and goal function from the JMD corpora may be summarized as follows:
-
The JMD reports that out of 2,230 authors who submitted to JMD in 2023, 368 (16%) of them selected “female”, “non-binary”, or that they “prefer to self-describe” their gender. This is a slightly higher percentage of gender minorities than observed by Running et al. Reference Running, Parkinson, Magleby and Toh(2024) in the mechanisms and robotics community at IDETC (9%), but much less than observed in the design theory community (59%).
-
Out of N = 145 2023 JMD papers, 14 of them (approximately 10%) had social relevance. This is lower than observed in either IDETC community, each of which was coded as having approximately 30% of papers with social relevance. This indicates perhaps a topical shift from the conference to the journal or a shift in the coding process by the authors between the two papers.
-
Out of n = 14 papers with social relevance, 4 mentioned benefiting a minoritized group (29%). This is more consistent with values seen in the mechanisms and robotics community (also 29%) than in the design theory community (53%), affirming initial hypotheses of a positive trend in mentioning minoritized group in communities with more women (Reference Running, Parkinson, Magleby and TohRunning et al., 2024).
-
Out of N = 145 JMD papers, 32 mentioned agentic goals and 19 mentioned communal goals. Agentic goals seemed to be more likely to be advancing (n = 21) and communal seemed more likely to be fortifying (n = 12). Since goal type and function were not studied by Running et al. Reference Running, Parkinson, Magleby and Toh(2024), conclusions relating these values to gender representation in the community will be drawn after a further corpora has been coded.
6. Conclusion
This paper provides a first step in formalizing processes for investigating the relationship between social relevance and gender-related socialization of the researchers engineering design disciplines. Without systematic analysis procedures like the one described in this paper, it is challenging to develop and assess interventions for increasing the participation of women in STEM research. Through cycles of interpretation, reflection, and building shared understanding, we developed a systematic framework for performing this analysis as a large scale (≈ 300 research papers analyzed to date) and also as a tool for other researchers to perform similar analysis in other STEM research venues.
Through the process of creating this framework and analysis protocol, several key insights have emerged which highlight the interplay between social relevance, research motivations, and goal alignment. First, it can be difficult to distinguish between author motivation and subject motivation. Author motivation is often difficult to infer from published works since many unstated factors such as publishing incentive structures, personal background, ties to specific communities, and career stage play an important role in motivating researchers to pursue goals. To address this, future work will involve analysis of interviews of authors to better understand their motivation. Second, coding for goal type and function may be more of a spectrum than a binary code, with some papers showing greater degrees of purusing agentic or communal goals than others. There may also be some papers which display both agentic and communal goals simultaneously, although to different extents.
In summary, socially relevant STEM research is crucial for addressing global challenges. By understanding how gender differences influence the pursuit of socially relevant communal goals, this study contributes to the growing body of research that aims to promote diversity, equity and inclusion in STEM fields. By applying the framework proposed in this research, researchers can identify trends in goal alignment and social relevance, which can ultimately inform the development of future interventions focused on increasing not only the number of women and gender minorities entering STEM fields but also their retention rate after graduation and in research-intensive communities. We encourage members of engineering and other STEM communities, including those participating in venues such as International Conference on Engineering Design (ICED), to apply this framework to a wider body of research.
Acknowledgements
Bethany Parkinson acknowledges support from the National Science Foundation under Grant No. CMMI-2401218 as well as the ASME BPart fellowship. This research was supported in early stages by Ivyann Running and Dr. Spencer Magleby. We also recognize the assistance of the Editor-in-Chief of the Journal of Mechanical Design, Dr. Carolyn Seepersad.
Open access
