1. Motivation and background
Prototyping is often used to test and improve designs, and it is important that we identify effective prototyping strategies and teach those strategies to undergraduate engineering students. Generally, prototyping is considered to be an iterative process used to test and validate design concepts (Dieter & Schmidt, Reference Dieter and Schmidt2013; Hartmann et al., Reference Hartmann, Klemmer, Bernstein, Abdulla, Burr, Robinson-Mosher and Gee2006; Marks & Chase, Reference Marks and Chase2019). Meriam-Webster Dictionary, Cambridge Dictionary, and Oxford Languages all use the word “first” in their definition of a prototype (Cambridge Dictionary, Reference McIntosh2013; Merriam-Webster, 2024; Oxford English Dictionary, 2024). In practice, prototypes are made throughout the design process and not just as a preliminary model of a design concept. Engineering designers use prototypes to learn something about their design concept. Leifer and Steinert present three learning loops to capture the kinds of conceptual change that occurs during iterative prototyping (Reference Leifer and SteinertLeifer & Steinert, 2011). Erichsen et al. expanded this idea by applying the learning loops to an automotive context and used an existing Socialization, Externalization, Combination and Internalization (SECI) model (Reference Nonaka, Toyama and KonnoNonaka et al., 2000) to showcase how learning and knowledge transfer occur during iterative prototyping (Reference Erichsen, Pedersen, Steinert and WeloErichsen et al., 2016).
While a lot of focus has been placed on iteration, other approaches to prototyping beyond iteration on a single concept exist. For example, Dahan and Mendelson present four approaches: One-Shot, Sequential, Parallel, and Hybrid (Reference Dahan and MendelsonDahan & Mendelson, 1998). One-shot refers to a single prototype that is also the final product, sequential refers to a traditional iterative approach on a single design, parallel refers to multiple prototypes being produced simultaneously with different designs, and hybrid refers to a process that blends iterative and parallel prototyping together (Reference Dahan and MendelsonDahan & Mendelson, 1998). Menold et al. proposed the Prototype for X (PFX) prototyping strategy that leveraged Design for X principles through a Frame, Build, Test framework to improve design outcomes (Reference Menold, Jablokow and SimpsonMenold et al., 2017). This framework was shown to improve user satisfaction, perceived artifact value, and manufacturability (Reference Menold, Simpson and JablokowMenold et al., 2016). Through a case study with 31 engineering design professionals, Kirjavainen et al. showed that iteration doesn’t occur as often as expected in industry settings, which might be related to high costs and inflexible budgets during early stages of the design process (Reference Kirjavainen, Lahdenne and BjörklundKirjavainen et al., 2023).
A literature review on design theory principles and prototyping strategies found that research on parallel prototyping is far less common than iterative prototyping (Reference Murphy, Watson, Tomko, Hilton and LinseyMurphy, Watson, et al., 2021). Despite the apparent scarcity of research on parallel prototyping, parallel prototyping has been shown to have advantages over an iterative process through experimental studies (Dow et al., Reference Dow, Glassco, Kass, Schwarz and Klemmer2009, Reference Dow, Glassco, Kass, Schwarz, Schwartz and Klemmer2010). Dow et al. showed that a parallel approach led to increased task confidence and higher user engagement with the final artifact (Reference Dow, Glassco, Kass, Schwarz, Schwartz and KlemmerDow et al., 2010). By observing an industry design-cycle, it was shown that parallel prototyping often occurs at design periods with high uncertainty (Reference Hansen and ÖzkilHansen & Özkil, 2020). The application of parallel prototyping has also been explored in other domains such as signal processing of seismic data (Reference Adair, Rodero, Parashar and MelgarAdair et al., 2023) or the rapid development of software (Reference Nielsen and FaberNielsen & Faber, 1996). In general, research on prototyping is motivated by evidence that designers are often inconsistent and not intentional when applying a prototyping strategy, if a specific strategy is used at all (Reference Camburn, Jensen, Crawford, Otto and WoodB. A. Camburn et al., 2015). Given the scarcity of research on parallel prototyping in engineering design applications, it is critical to continue exploring how designers can leverage a parallel approach in various contexts to improve design outcomes.
In prior work, a parallel prototyping strategy was examined through an experimental study that involved a physical design competition to capture differences in prototyping process (Reference Murphy, Floresca, Fu and LinseyMurphy et al., 2022). The prototyping process constrained participants to either use a parallel or iterative process. Results showed clear advantages for parallel prototyping over iterative prototyping in terms of design success and self-efficacy (Reference Murphy, Floresca, Fu and LinseyMurphy et al., 2022). It was also shown that despite the advantages of a parallel approach, participants reported a strong preference for an iterative approach, though intended to use parallel prototyping in future projects (Reference Murphy, Saracino, Akgun, Fu and LinseyMurphy, Saracino, et al., 2021). When CAD models were examined from the competition, a parallel prototyping approach led to models with fewer features than those created through an iterative approach (Reference Murphy, Whittle, Fu and LinseyMurphy et al., 2023). It was also shown that designs with fewer CAD features were more successful in the design competition (Reference Murphy, Whittle, Fu and LinseyMurphy et al., 2023). Taken together, there is a mismatch between the advantages of a parallel prototyping process and participants’ preference for an iterative approach. In essence, students were seen fixate on a single concept, potentially due to a sunk cost effect, and would produce design iterations of increasing complexity instead of broadening their scope and considering alternative approaches. Those students were then observed to be content with their approach despite poor design performance.
The research presented in this paper expands this prior work into a more realistic setting to seek validity for those previous findings and to ascertain if they hold true when students can determine their own prototyping strategy. This is completed by exploring the tendency of first year engineering students to adopt a parallel prototyping strategy after gaining exposure to content on its application, advantages, and disadvantages. This contrasts with the prior work where participants were assigned to using either a parallel process or an iterative process as part of the project; the current work presents a more naturalistic design setting where participants have the freedom to determine and implement their own design process, potentially even changing their processes throughout the project. This ability to self-determine one’s design process is designed to be more like the situation engineers experience in the real world. In addition, this work aims to compare design success between groups that take different approaches to the prototyping process. This study contributes to the larger goal of understanding how different approaches to the physical prototyping process impact design success. This work aims to address the following research questions:
1. To what extent does education on parallel prototyping promote its use and adoption during the design process for a physical design challenge?
2. What differences exist between the outcomes of an iterative or parallel prototyping approach when applied to a physical design challenge?
To address these research questions, the authors propose the following hypotheses:
1. Introducing participants to parallel prototyping and its potential benefits and drawbacks will lead to an increase in its application in a design competition when compared to those who are not exposed to parallel prototyping.
2. Participants in the parallel prototyping condition will be more likely to score and will achieve higher scores in a physical design competition than participants in the iterative prototyping condition.
3. Those who take a parallel approach to the prototyping process will have more favorable design outcomes than those who take an iterative approach to the prototyping process in a physical design competition.
The following sections describe the experiment design, and the tools used to address these research questions, followed by a presentation of results from this study with discussion.
2. Methods
The experiment sought to determine the intended and actual prototyping strategy of students in two first-year CAD and design course sections. Students in one of these sections were taught both parallel and iterative prototyping and while students in the other section were only taught iterative prototyping. Differences between these two sections are used to determine if the employed design method has an impact on the performance of final designs during a ball launcher competition.
2.1. Participants
This study was conducted in a 3-credit hour, first-year undergraduate design course at a research-oriented university in the southeastern United States where students learn the basics of CAD, 3D printing design principles, and freehand sketching. Two sections of this class were provided with the opportunity to participate in this study. Students who chose to participate were compensated with extra credit not exceeding 2% of their overall grade in the course. 85 students consented to and participated in the IRB approved study. Student majors included aerospace, mechanical, electrical, industrial, and materials science engineering. Aerospace and mechanical engineering students are required to take the course while the other students took it as an elective.
2.2. Experimental design
There were two conditions for students’ final design projects, with one condition being the iterative condition and the other condition being the parallel condition, having 42 and 43 participants respectively. For their final design project, students were tasked with designing and producing a 3D-printed ball-launching device to shoot a ball 10 feet into a target. After the students had been introduced to the design challenge, they received a lecture on prototyping strategies where one course section was taught about both parallel prototyping and iterative prototyping, while the other course section was only taught about iterative prototyping. Both lectures emphasized the importance of implementing a prototyping strategy that would account for design changes and 3D-printing issues. As the experiment took place in a first-year engineering course, it was likely that they had not been previously exposed to prototyping strategies and may have never previously produced a design using CAD tools. The lecture on parallel prototyping presented research discussed in Section 1 and explicitly stated that students following that strategy had seen more success in a similar design problem without a perceived increase in workload. The goal of the parallel prototyping lecture was to convince the students to adopt this strategy for their final project. However, they were not required to change to a parallel approach. The prototyping lecture also introduced both sections to the use of test parts in design for 3D printing. No other course lectures were altered while the students developed their designs.
A Ph.D. student in mechanical engineering delivered both lectures so that the course instructor would be blind to which course section was under which experimental condition. This was done so that the instructor would not unintentionally bias the experiment by altering the teaching of the course material to benefit one condition over the other. However, due to interactions with the students, the instructor eventually unintentionally determined the experimental section. The two conditions were not explicitly made aware of each other, though there was no restriction in communication made between the sections.
Students were allowed to create as many prototypes as they wished prior to the competition so long as they were documented, where documentation included photos, written analysis, and saved CAD models. The project took place over 8 weeks during which students had to complete other assignments. Their final reports included details and reflection of their design and manufacturing process as well as discussion of their competition performance.
2.3. Competition details
As a part of their course, students were tasked with creating a 3D-printed ball-launching device. The purpose of the devices was to propel a small foam or wooden ball 10 feet into a set of targets, shown in Figure 1. Students were free to choose between the foam and wooden balls. The target was made using hexagonally rimmed cups so that it would be easy to determine where the target landed. Students each had five attempts to land a ball in the targets and scored points for each successful attempt. Balls that hit the rim between cups of two different scores received the average of those scores, so hitting the rim between a 4-point cup and a 7-point cup would award 5.5 points. A hit in the 0-point cups would only grant points for distance. The closer to 10 feet that the students launched from and successfully scored points, the greater their distance score up to a maximum of 10 points Table 1. The competition score was the total of scored points from hitting the target and the lowest distance score earned, for a maximum of 60 points.
Figure 1. Representation of target used for ball launching competition
Table 1. Point breakdown for distance launched
The devices were to be designed using SolidWorks®, the software used in the course to teach 3D modelling. Student designs had to meet the following criteria:
-
Have maximum dimensions of 4” x 5” x 4”
-
Have at least one interlocking part
-
Have at least four of the following operations: extrusion, loft, revolve, spline, sweep, mirrored portions, fillets or chamfers, or text
-
Have two stable states
Students were encouraged to make as many prototypes as they saw fit, taking screenshots in SolidWorks and pictures of the print for every iteration. Prototypes were to be printed at an on-campus maker space or using a personal printer so long as the fused filament fabrication technique was used.
2.4. Report analysis
The students’ reports were analyzed to determine the prototyping strategy the student intended to use, the prototyping strategy that they actually used, the prototyping strategy they were interested in using in future projects, and whether they used test parts in the creation of the ball launching device. The inclusion of test parts in the analysis is to see if the students followed some part of the prototyping lecture even if it did not affect their overall prototyping strategy. A coding guide was created to define the possible strategies. The intended and actual strategies used by the students were each coded to one of four options: One-shot, Pure Iterative, Pure Parallel, and Mixed Iterative and Parallel. The four codes are shown in Figure 2. The drawings provide a visual representation of each prototyping strategy, where a box indicates a single prototype.
Figure 2. Prototyping strategies
The guide defined a prototype as either physical or digital. Digital prototypes were SolidWorks® CAD models while physical prototypes were 3D printed devices. Since all physical prototypes were first designed in SolidWorks® before being produced, the corresponding digital and physical iterations of a single device were considered to be the same prototype.
Iterative prototyping was defined as “the sequential testing and refinement of a prototype” (Reference Christie, Jensen, Buckley, Menefee, Ziegler, Wood and CrawfordChristie et al., 2012). This meant that a prototype iteration occurs after some form of testing is completed on a prototype and that the development of a digital model and its realization through 3D printing did not count as an iteration. However, a digital prototype that was explicitly redesigned after the designer learned something about the prototype counted as an iteration and not just further development of a single prototype.
Parallel prototyping was defined as cases where “multiple concepts are embodied and compared concurrently” (Reference Camburn, Viswanathan, Linsey, Anderson, Jensen, Crawford, Otto and WoodB. Camburn et al., 2017). According to the definition, parallel prototyping involved physical prototypes that all underwent testing at the same time without further design refinement between tests. Parallel prototyping must involve at least 2 prototypes, where there is some variation in the form or function of the prototypes. As shown in the bottom image for Pure Parallel in Figure 2, iterating on prototypes that are in parallel was still counted as Pure Parallel prototyping and not as Mixed Iterative and Parallel.
As the definitions of parallel prototyping and iterative prototyping were not mutually exclusive, it was possible for both strategies to be used at once in a mixed condition. Therefore, if a student’s prototyping approach independently met the criteria of both parallel and iterative prototyping, it counted as a Mixed Iterative and Parallel strategy.
The One-Shot strategy was described as when “the design team builds and tests only one prototype” (Reference Dahan and MendelsonDahan & Mendelson, 1998). For this study, the One-Shot strategy occurred when the students’ report indicated they created a single design, made no changes to it and used it in the final competition. While this strategy is less prevalent in literature, its inclusion was necessary as, during data immersion as part of the creation of the code guide, it was seen that multiple students used a One-Shot approach.
For the purposes of this experiment, test parts are similar physical components that come in groups of 2 or more that are used to determine ideal measurements and to ensure functionality before developing or manufacturing a full prototype. The measurements typically relate towards determining tolerances needed for interfacing components. Test parts are often simple objects that contain very few features and vary in dimension. If a student discussed usage of something that matched the definition of a test part, that usage was recorded. Test parts, or test artifacts, are traditionally standardization tools and are used to measure the accuracy of 3D printers (Reference Moylan, Slotwinski, Cooke, Jurrens and DonmezMoylan et al., 2014). While our application of test parts diverges from this definition, both forms are used to provide understanding the printability.
The coding analysis was completed manually through the application of the coding guide. A researcher would read the report and determine which categories (more commonly referred to as themes in qualitative literature) the prototyping process belonged to according to the definitions in the guide. As this process is affected by the subjectivity of the researcher, multiple researchers independently reviewed each report then met to discuss any differences in their categorization process.
3. Results
3.1. Competition scores
The parallel condition scored significantly higher on average, with 17 participants scoring and a condition average of 5.4 points (SD = 7.3), compared to the iterative condition, which had only 9 participants scoring and a condition average of 1.6 points (SD = 3.7). This difference was found to be statistically significant through Welch’s Two-sample t-test (t (df = 60.542) = 3.023, p =.004) which accounts for the unequal variances observed (F = 3.870, p <= .001) (Reference Myers, Well and JrMyers et al., 2013). For students who did score, the parallel condition had an average of 13.4 (SD = 4.9) and the iterative condition had an average of 7.6 (SD = 4.6).
Since the number of students that scored was so low, a chi-squared test was used to determine if there was a statistical difference between the number of students that scored . A significant difference was observed (χ2 (1, n = 85) = 3.823, p = .051, calculated using R ver4.4.2). A post-hoc power calculation determined that the chi-squared test for independence had a power of 0.5 and that a power of at least 0.8 could be achieved with a sample size of 86 subjects per condition.
These results indicate that there was some difference between the two conditions that enabled the parallel condition to find more success than the iterative condition, measured by number of scoring participants and average points of those who scored.
3.2. Intended and actual prototyping strategy
The student-submitted reports were coded to determine the strategy the students planned to use and the strategy they actually ended up implementing. Due to students’ time constraints and inadequate planning skills, it was expected that a number of the students in the parallel condition would plan for a parallel prototyping strategy but fail to actually implement it. Initial results of the coding include analysis of 35 of the total 85 reports. Coding of student reports was completed by a Mechanical Engineering PhD student and an undergraduate mechanical engineering student. To ensure consistency, the two coders independently reviewed and coded 15 reports. They then compared their coding to find any differently coded responses and resolved those by discussion. A second round was completed with an additional 10 reports which then led to a refinement of the coding guide. A third round of 10 additional reports was then completed. Those 35 total reports were used for the analysis. Analyzed reports were selected at random from the collection of all reports from both conditions. This was done to hide the condition of each report from the raters and prevent biased coding results. This process resulted in the evaluation of 20 reports from the iterative condition and 15 reports from the parallel condition. Evaluation of the remaining 50 reports is ongoing. There was 94% agreement between the two raters for the 35 reports analyzed in this work.
The frequencies of intended and actual prototyping strategies used by the participants are shown in Table 2 and the frequencies of the desired prototyping strategies for future projects are shown in Table 3, where both tables are divided by experimental condition. The iterative condition behaved as expected where the vast majority intended on using an iterative strategy. One student failed to indicate their intended strategy so was coded as “not reported”. Their actual prototyping strategies followed a similar ratio, with a few more students using a one-shot approach. Within the parallel condition, more than half of the students planned to use a parallel or hybrid approach. However, none of the students executed a prototyping strategy that fit the definition for parallel or hybrid. One student claimed to have used parallel prototyping but there was no evidence that they actually took a parallel approach. Despite the lack of usage of parallel prototyping, both groups expressed interest in using the method in the future. Lastly, six students in the iterative condition and four students in the parallel condition created and used test parts while prototyping their designs.
Table 2. Frequency of prototyping strategies that students intended to use and actually used
Table 3. Frequency of prototyping strategies students intend to use in the future
4. Discussion
There was a statistically significant difference in the number of participants who scored in the parallel condition than in the iterative condition. This hints that the cause of the improved performance was not due to the prototyping methods used but instead due to some other factor. While the authors have some theories on what the cause may be, none of the analyzed data indicates a cause and therefore further research must be conducted.
The parallel prototyping condition performed significantly better, but the reasons for this remain unclear. The initial results from coding the reports indicated that while a majority of students that were introduced to parallel prototyping were interested in using it, none of them actually implemented parallel prototyping. Some of the reports presented reasons for why this was the case. In general, students seemed wary of the time that it would take to complete a parallel prototyping process while working individually, even if its usage could improve their final designs. The parallel prototyping lecture referenced previous literature of a similar undergraduate design task involving parallel prototyping where students reported that they did not feel the parallel approach took any longer (Reference Murphy, Floresca, Fu and LinseyMurphy et al., 2022), but it seemed that the students either did not believe this finding or that it was forgotten. There was also widespread concern over the availability of 3D printers that students could use to create their designs. Students primarily used two large campus makerspaces to make their devices. However, since the makerspaces were accessible to students outside of the course and as many students waited till the last two weeks to create their devices, printer availability became a genuine problem. Lastly, many students expressed difficulty translating their design concepts into CAD models, then into physical devices, and that their device iterations focused on making functional devices instead of improving device performance.
Unlike the usage of parallel prototyping, more than a quarter of students in each condition made use of test parts. This is important since it shows that students did listen to and follow the advice of the prototyping presentations. Since one of the suggested benefits of using test parts was time savings and students discussed their concerns about the time-crunch that they experienced with the project, that may be why the use of test parts was more prevalent. Once the complete results of the coding are produced, it will be insightful to see how many of the students that planned to use parallel prototyping also planned to use test parts.
The prevalence of students that used a one-shot prototyping strategy was an unexpected finding among the research team. Since both conditions were introduced to the benefits of iterating on design and as iteration is generally good practice in design, it was expected that all students would iterate at least once. Students in this category either self-identified as experienced users of 3D printers, started the assignment too late to have time to iterate, or designed highly complex devices that took too long to print and restricted their ability to iterate.
While initial results showed parallel prototyping usage was nearly non-existent, many students indicated a desire to use it on future design projects. It may be that many students who did not score in the competition feel that their approach was lacking and that a parallel approach could have improved their performance. Once the coding is completed, tests can be conducted to see if there is correlation between a score of zero at the competition and a desire to change prototyping strategy. It may also be that after the design challenge had completed and students had gained experience designing and producing 3D printed devices, they felt more comfortable trying a different approach since they would not be attempting so many new tasks at once. Many students also included that they would hope to have more time for future projects as they believed that a longer timeline would be more conducive to parallel prototyping.
5. Conclusion
This work presented the initial results of a study on undergraduate usage of parallel prototyping in a course-based design challenge. Students in two sections of a first-year engineering course focusing on design and CAD modelling were tasked with designing and producing 3D-printed ball launching devices, where one section was taught about both parallel and iterative prototyping while the other section was only taught about iterative prototyping. Students in the parallel condition scored significantly higher on average and had significantly more scoring students than the iterative condition. Initial results of coding reports showed that over half of the students in the parallel condition wanted to use parallel prototyping but that none did use parallel prototyping. However, more than half of the currently coded reports indicated a desire to use parallel prototyping in the future. Lastly, there was prevalent usage of test parts in the design and development of ball launching devices.
The initial results presented in this work continue the effort to understand the benefits of parallel prototyping for undergraduates by preserving the students’ agency in the prototyping process and allowing them to determine their own approach. Interestingly, but maybe expectedly, students had difficulty implementing a parallel prototyping strategy due to other challenges they faced such as lack of printing resources or available design time. As the results of the report coding are completed, further analysis will be conducted to determine if certain prototyping behaviors produced statistically significant differences in competition score. Particular interest will be taken to see if students who implemented parallel prototyping, or at least desired to implement it, scored higher than those who had a purely iterative approach. We will investigate other potential reasons why the parallel condition performed better than the iterative condition.
The experimental conditions caused a change in performance, but it is not yet clear specifically what it was about the conditions which caused that difference. It is hypothesized that presenting the parallel prototyping strategy as effective for difficult design tasks such as the ball launching competition better communicated the difficulty of the task and led to better design outcomes. From prior work, it is clear that parallel prototyping strategies provide advantages but are very difficult for students to implement on their own. This likely indicates that if parallel prototyping strategies are desired within organizations, then management must drive a parallel process as it is likely difficult for individual engineers to implement such a process on their own due to time pressures and access to prototyping.
In addition to completing the report analysis, engineering design self-efficacy surveys that students completed as a part of the experiment will provide further feedback regarding first year engineering student perceptions of parallel prototyping. Future work can also implement easier design problems that provide more measures of success and therefore better metrics for comparing experimental groups. The design challenge could also be presented to teams of students instead of individuals as this could make a parallel approach more appealing and practitioners easier to implement. Furthermore, there was no analysis of the way in which the parallel prototyping lecture tried to convince students to use parallel prototyping. It may be that the arguments presented could be structured differently to make it more appealing to novices, or that another professor or member from industry could be more convincing than a PhD student.
Open access
