2.1. Generative LLMs in engineering tasks

The impact of AI on the success of engineering tasks depends not only on the task itself but also on the type of AI tool being used and users’ AI-related knowledge and skills, such as prompt engineering skills. Understanding this distinction is important, as task-specific AI systems are designed to be utilised in narrowly defined domains related to the specific task, thus leveraging task-specific knowledge. Significant research has shown that using task-specific or custom-made LLMs in engineering tasks can outperform traditional approaches for the same tasks. For example, studies related to the CAD domain show that LLMs utilising historical knowledge from multiple modalities (CAD models, text, and images) enhance the efficiency of the assembly process and overall manufacturing efficiency (Reference Hu, Tian, Nagato, Nakao and LiuHu et al., 2023). They also outperform traditional approaches when searching design repositories and creating engineering knowledge bases (Reference Meltzer, Lambourne and GrandiMeltzer et al., 2024).

In contrast, general-purpose LLMs, such as ChatGPT, are trained on broad datasets encompassing knowledge from various domains. That makes them suitable for general knowledge acquisition (Reference Ritala, Ruokonen and RamaulRitala et al., 2024), including engineering knowledge (Reference Xu, Kotecha and McAdamsXu et al., 2024). Unlike task-specific AI tools that are limited to well-defined tasks, general-purpose LLMs can address both open-ended and well-defined tasks (Reference Wang, Anwer, Dai and LiuWang et al., 2023). For example, Urban et al. (Reference Urban, Děchtěrenko, Lukavský, Hrabalová, Svacha, Brom and Urban2024) found that designers produced higher-quality solutions in open-ended tasks like concept generation with ChatGPT support. Moreover, participants who had prior experience with ChatGPT generated even higher-quality solutions. However, while ChatGPT, on average, outperforms humans in providing creative solutions to a problem, the best human solutions exceeded those of ChatGPT (Reference Koivisto and GrassiniKoivisto & Grassini, 2023). Research also shows that individuals with low creative abilities benefit from ChatGPT support in brainstorming (Reference Memmert, Mies and BittnerMemmert et al., 2024). However, when focusing only on the ideas generated independently by humans, excluding ChatGPT’s, both low- and high-creativity individuals produced fewer ideas. When observing how individuals engage with the ChatGPT, low-creativity individuals request and accept more ChatGPT suggestions than high-creativity individuals. Furthermore, the output quality in open-ended engineering tasks, such as brainstorming, is highly sensitive to the prompts provided by ChatGPT (Reference Memmert, Mies and BittnerMemmert et al., 2024). Refining prompt engineering could enhance output quality (Reference Memmert and TavanapourMemmert & Tavanapour, 2023). In prototyping tasks, design teams supported with ChatGPT yielded outcomes similar to human teams without ChatGPT support, but faced challenges such as ChatGPT’s tendency to abandon concepts too early, unnecessary complexity, providing vague responses with forgetting previous information or prompts and design fixation (Ege et al., 2024). For well-defined engineering tasks, particularly in programming, ChatGPT yielded lower output quality compared to task-specific generative LLMs that receive code-based prompts (Reference Mnguni, Nkomo, Maguraushe and MutangaMnguni et al., 2024). However, among general-purpose LLMs, ChatGPT demonstrated the highest output quality (Reference Coello, Alimam and KouatlyCoello et al., 2024). Despite that, the role of user feedback in a manner that the models can understand, which is proven to be time- and resource-intensive, remains important. Furthermore, the LLMs’ programming style can affect the output quality, which may differ from that of the users. Engineering expertise is, therefore, essential for providing contextual understanding, particularly when dealing with ambiguities associated with problems that require domain-specific engineering knowledge (Reference Wang, Anwer, Dai and LiuWang et al., 2023), especially since ChatGPT may generate articulate, accurately sounding solutions that are nonetheless incorrect (Reference Mnguni, Nkomo, Maguraushe and MutangaMnguni et al., 2024). That aligns with the notion that domain-specific engineering knowledge and hard skills are crucial in the interaction of user and ChatGPT (Reference Giordano, Spada, Chiarello and FantoniGiordano et al., 2024). Ultimately, this suggests a complementary relationship between human intelligence and AI in solving both open-ended and well-defined engineering tasks (Reference Wang, Anwer, Dai and LiuWang et al., 2023). Thus, ChatGPT also demonstrates potential in supporting CAD tasks since CAD encompasses both open-ended tasks like concept generation and well-defined detailed design engineering tasks. That highlights the need to explore which types of CAD tasks are best suited for ChatGPT support, particularly as these tasks require varying types engineering knowledge.

2.2. LLMs for Computer-Aided Design (CAD)

CAD has long been an essential tool used throughout various design phases. Designers utilise CAD tools to create and test digital representations of their ideas, such as virtual product concepts or prototypes, prior to manufacturing (Reference Azemi, Mehmeti and MalokuAzemi et al., 2018). With the integration of LLMs like ChatGPT into the design process, researchers are also exploring how ChatGPT can support various CAD tasks. Those endeavours mainly focus on automating the generation of 3D objects or CAD models and variations of those models based on different ChatGPT input modalities, including text- and image-based prompts (Reference Makatura, Foshey, Wang, Hähnlein, Ma, Deng, Tjandrasuwita, Spielberg, Owens, Chen, Zhao, Zhu, Norton, Gu, Jacob, Li, Schulz and MatusikMakatura et al., 2024). Text inputs being explored involve interpreting human language instructions, while others focus on utilising CAD programming languages such as CADQuery and OpenSCAD (Reference Makatura, Foshey, Wang, HähnLein, Ma, Deng, Tjandrasuwita, Spielberg, Owens, Chen, Zhao, Zhu, Norton, Gu, Jacob, Li, Schulz and MatusikMakatura et al., 2023). The findings suggest the potential benefits of using ChatGPT, such as generating CAD models representing standard mechanical engineering elements and their variations. However, it struggles with creating complex CAD models or those not representing standard mechanical engineering elements solely through text. Additionally, using ChatGPT in CAD workflows can be time-consuming and requires multiple iterations, as it generates flawed CAD models, requiring user interactions to correct them. While most studies have focused on CAD tasks that primarily require CAD knowledge, specifically the creation or modification of CAD models, there has been limited research into how LLMs like ChatGPT can assist with other types of CAD tasks. These tasks may require other types of expertise and knowledge, such as tasks that include optimisation of models based on various design requirements, including material or manufacturing technology selection.

This gap is even more pronounced in collaborative CAD, which has gained considerable research focus due to the shift from standalone CAD. This transition addresses common challenges associated with collaboration, such as limitations in cloud-based synchronous editing, issues with seamless file sharing, and visibility of design changes (Reference Cheng, Davis, Zhang, Zhou and OlechowskiCheng et al., 2023). Researchers have focused on various collaborative CAD tasks that require different types of knowledge, including concept creation (domain-specific engineering knowledge) (Reference Deng, Mueller, Rogers and OlechowskiDeng et al., 2022) or modification of existing CAD models (CAD knowledge) (Reference Phadnis, Arshad, Wallace and OlechowskiPhadnis et al., 2021). The success of these tasks has been evaluated based on several criteria, particularly the outcome quality (Reference Sadeghi, Dargon, Rivest and PernotSadeghi et al., 2016), with task completeness being the important dimension (Reference Company, Contero, Otey and PlumedCompany et al., 2015). Findings suggest that the quality of collaborative CAD outcomes depends on the context, including team size and the nature of tasks. However, there remains a research gap regarding the role of generative LLMs, such as ChatGPT, in solving collaborative CAD tasks with different types of knowledge, especially concerning the quality and completeness of such tasks.

3.1. Experimental sample and tasks

The study involved 44 mechanical engineering students (11 females and 33 males) from the Faculty of Mechanical Engineering and Naval Architecture from the University of Zagreb, spanning undergraduate and graduate levels. Participation required students to have completed both basic (first year) and advanced (third year) CAD courses at the university, ensuring they possessed both declarative and procedural CAD knowledge (Reference ChesterChester, 2007), along with a prior understanding of design for manufacturability principles. They were divided into 22 pairs. Each pair was assigned to one of two experimental conditions: with or without ChatGPT support.

The experiment consisted of three consecutive collaborative CAD tasks aimed at integrating different types of knowledge - CAD knowledge (declarative and procedural) and knowledge from the engineering design domain, such as design for manufacturability. The first task, referred to as Task 1, involved both the declarative and procedural CAD knowledge. Participants were tasked with creating a 3D CAD model of a crankshaft by measuring and replicating a provided model in STEP format (Figure 2). Task 2 required both CAD and domain-specific knowledge. Participants were tasked with creating a functional assembly from the five models provided previously by researchers that were not dimensionally adequate: the housing, along with crankshaft, upper and lower parts of connecting rod and a piston. Unlike creating a single part, assembly creation requires understanding how components fit together and interact in a functional assembly. Domain-specific knowledge was required to maintain appropriate clearances and ensure the parts’ correct alignment and interaction. Participants also needed CAD knowledge on the declarative level to select the appropriate CAD assembly operations to complete the assembly successfully. Among the five provided parts, four had visible feature trees (a crankshaft, connecting rod parts and a piston), while housing was in STEP format, meaning its feature tree was not accessible and, consequently, not modifiable, requiring the other parts to be adjusted accordingly. The third task, Task 3, required more domain-specific knowledge than the previous tasks by involving redesigning a CAD model of a crankshaft based on design-for-manufacturability principles specific to forging technology. Participants needed to consider the shape and dimensions of the model as it would be forged before any machining occurred. The model to be modified represented the same crankshaft geometry as in Task 1 (Figure 2) and included a visible feature tree, enabling participants to modify the CAD model.

3.2. Experimental setup and procedure

The experimental setup for both conditions consisted of one room with two working places facing each other, as shown in Figure 1. Each working place was equipped with an office chair and table, a high-performance computer, two monitor screens (22” with a resolution of 1920x1080 pixels), a keyboard and a mouse. In both conditions, participants utilised Onshape, a cloud-based CAD software accessed via web browser on each monitor screen. Onshape enables synchronous collaboration on a single CAD model, allowing participants to simultaneously interact with the same CAD model, follow each other’s work and share views. Like other CAD software, Onshape, in addition to creating or modifying a CAD model (single part or assembly), also provides basic viewing functionalities like measuring, sectioning, moving, and hiding parts. Each monitor screen had a different setup. The left screen displayed an Onshape working document. In Task 1, this document was blank; Task 2 included an empty assembly document alongside five CAD models of parts to be assembled; and Task 3 contained a CAD model of a crankshaft that needed modification. The right screen presented another Onshape document with detailed task explanations (imported as a PDF) and the crankshaft CAD model, which participants used to create their models. Task 3 also provided the original crankshaft model in case participants needed it while modifying the model on the left screen. Pairs with the ChatGPT support had an additional web browser window on the right monitor screen with the ChatGPT application (Figure 2). To facilitate synchronous collaboration during the execution of CAD tasks, the content of the right monitor of one team member was shared on the right monitor of the other one screen mirroring. This setup enabled team members to follow each other’s work in real-time, whether it involved measuring the part in Task 1 or interacting with the ChatGPT. Participants utilizing ChatGPT received support from ChatGPT-4 model, from the customized GPT Onshape Usage and Collaboration, developed specifically for the experiment by the authors of this study. To prevent the information from being reused for other pairs’ tasks, the training option for the ChatGPT model was disabled, and its memory was cleared after each pair’s session by the researcher. Additionally, all screen content was recorded for the entire duration of the experiment using OBS Studio. Video and audio recordings of the tasks were captured using a conference camera, and small video cameras were placed on each monitor screen. The experimental procedure consisted of 5 steps: 1) Pre-experiment preparation, 2) Introductory collaborative CAD session, 3) First collaborative CAD session, 4) Second collaborative CAD session, and 5) Third collaborative CAD session.

Figure 1. Experimental setup

Figure 2. Left and right monitor screen set-up (Task 1 - experimental condition)

In the first step, participants received an information package via email a few days before the experiment. The package included detailed tutorials on using Onshape, although all participants were already familiar with the software from their CAD courses at the university. Additional tutorials were provided on ChatGPT to participants in the experimental condition, focusing on the user interface, application usage and prompt engineering techniques. Consent forms were also signed in the first step. Furthermore, steps two through five took place on the same day as the experiment for each team, beginning with a 15-minute introductory session. Participants worked on a simple CAD task during this session, collaborating constantly and synchronously. Participants in the experimental condition were also required to consult ChatGPT to make every decision, the same as they would consult each other. After the introductory session, the first, second, and third collaborative CAD sessions were conducted, corresponding to the tasks mentioned earlier. Teams were allocated 40 minutes, 25 minutes, and 20 minutes for Task 1, Task 2, and Task 3, respectively. Before starting and upon finishing each of the four sessions, the researcher provided each team with details about the next steps. In addition to audio and video recordings, the researcher collected the final CAD outputs after each task session for data analysis.

3.3. Data analysis

The collaborative CAD task completeness was measured via a CAD model completeness, one of the CAD model’s quality dimensions (Reference Company, Contero, Otey and PlumedCompany et al., 2015). According to that, a task is considered complete if it includes all the geometry aspects relevant to the design it represents, specifically regarding its sub-dimensions of shape and size. Therefore, the dependent variable in this study is collaborative CAD task completeness, measured by two specific sub-variables: shape replication and size replication (Reference Company, Contero, Otey and PlumedCompany et al., 2015). Shape replication refers to the accuracy of the geometric features constituting the resulting design that the CAD model represents (e.g., a cylinder, a slot, a fillet, a constituting part of an assembly…). Size refers to the accuracy of geometric dimensions, which quantify the physical aspects of the design represented by a CAD model (e.g., length, width, height, radius, and angles). Even though the completeness variable is usually defined as a dichotomous variable, indicating that it is fully accomplished, herein we used intermediate scoring values for each sub-variable, namely both shape and size replication. This approach allowed us to capture partial and nuanced levels of task completeness.

In Tasks 1 and 3, shape replication refers to the accuracy of the geometric features that make up the resulting CAD model. In contrast, size pertains to the accuracy of geometric dimensions that quantify the physical characteristics of the CAD model. In Task 1, the analysis focuses on evaluating whether all the geometric features present in the provided CAD model are also included in the CAD model created by the pairs participating in the experiment (shape), and whether they are dimensionally accurate (size). Task 3 has a similar focus but relates specifically to design for manufacturability principles. That includes modifying geometric features (such as defining parting lines, adding drafted surfaces and rounded edges) (shape) and ensuring that the dimensions for these features are appropriate (size) (Reference BrallaBralla, 1998). Task 2 is inherently different since its output is a functional assembly. Therefore, the evaluation of shape and size follows the definition of assembly (Reference Lupinetti, Giannini, Monti and PernotLupinetti et al., 2018), which considers the parts and their relationships while considering different levels of information (Chen, 2012): topological structure and relationships between assembly components (shape), and geometric information (size). In our case, the first two aspects involve verifying whether the constituent parts are included in the assembly, whether the parts are correctly placed in relation to each other, and whether the appropriate mates are applied (mates that mimic the intended movements). The geometric information concerns the dimensions between parts, as four of them are not dimensionally suitable for the provided housing. Additionally, functionality dictates certain dimensional relationships, such as those required for a simple engine mechanism represented in the assembly of Task 2.

The following number of measurement elements for sub-variables of shape and size is established for each task: Task 1 has 18 elements for shape and 32 for size; Task 2 has 16 for shape and 10 for size, and Task 3 has 14 elements for both shape and size. The researcher evaluated each element in the resulting CAD models by determining if the element was correctly addressed in the task, receiving a score of 1 for correct and 0 for incorrect. Shape and size scores were calculated by summing the corresponding elements for each sub-variable. These scores were then normalized as percentages and averaged for each pair to provide an overall task completeness. Statistical analyses were conducted using Python.

4.1. Comparing completeness within each condition

To assess whether task completeness differs significantly across tasks within the same experimental condition, the Friedman test was conducted. This non-parametric test is suitable for analysing repeated measures data when the assumption of normality is violated. Additionally, pairwise comparisons were conducted using Wilcoxon signed-rank tests with Bonferroni correction to identify differences between tasks when a significance is found.

The Friedman test revealed a statistically significant effect of task on task completeness in the condition without ChatGPT support (χ²=17.610; p=0.002), with a large effect size (W=0.800). Pairwise comparisons using Wilcoxon signed-rank tests with Bonferroni correction, furthermore, indicated that completeness of Task 1 were significantly higher than Task 2 (W=0.0; p=0.020) and Task 3 (W=0.0; p=0.015). Additionally, completeness of Task 2 was significantly higher than Task 3 (W=1.0; p=0.006). In contrast, in the condition with ChatGPT support, the Friedman test did not reveal a significant difference (χ²=3.818; p=0.148) in completeness across tasks. The corresponding effect size is small (W=0.174). The summary of results is shown in Table 2.

Table 2. Analysis of collaborative CAD task completeness across experimental conditions

*Significant value (p<0.05)

4.2. Comparison of completeness within each task

To compare task completeness between conditions with ChatGPT and without ChatGPT support for each task, different statistical tests were used based on the characteristics of the data. For Task 1, Welch’s t-test was used due to violations of the equal variances. For Task 2, the Mann-Whitney U test was applied due to violations of the normality assumption, as it is a non-parametric alternative suitable for comparing medians between two independent groups. For Task 3, the standard t-test was appropriate as both normality and homogeneity of variance assumptions were met.

For Task 1, Welch’s t-test indicated a statistically significant difference between conditions (t=-3.728, p=0.004), with a large effect size (d=1.589). In Task 2, the Mann-Whitney U test also indicated a statistically significant difference (U=21.5, p=0.011), with a large effect size (r=0.645). In Task 3, however, the standard t-test did not reveal a statistically significant difference (t=0.815, p=0.425) of completeness scores between pair with ChatGPT and without ChatGPT support. The effect size for Task 3 is small to medium (d=0.347). The results are summarised in Table 3.

Table 3. Analysis of difference in task completeness across collaborative CAD tasks

*Significant value (p<0.05)

Figure 4. Comparison of completeness across tasks