Hostname: page-component-cb9f654ff-d5ftd Total loading time: 0 Render date: 2025-08-30T09:37:21.010Z Has data issue: false hasContentIssue false

Understanding ChatGPT’s impact on student-team ideation outcomes for new product development

Published online by Cambridge University Press:  27 August 2025

Benjamin Justin Bunn*
Affiliation:
Brigham Young University USA
Bryan F. Howell
Affiliation:
Brigham Young University USA
Geoff Wright
Affiliation:
Brigham Young University USA

Abstract:

Previous studies found ChatGPT-assisted ideation produced lower fluency, flexibility, and originality with shorter ideation sessions. This research hypothesized that a longer 24-minute session would improve ideation outcomes for the ChatGPT-assisted approach and enhance team engagement. Undergraduate students participated in two design workshops: one using a ChatGPT-assisted approach (n=22), the other using only analogue methods (n=17). Results showed that while the analogue group slightly outperformed the ChatGPT group in flexibility and originality, the fluency difference was larger, with the analogue group producing over twice the number of ideas. Evidence suggests team-based ideation behavior has more impact on ideation outcomes. Future research will explore a hybrid individual-to-team approach that combines individual contributions with team collaboration.

Information

Type
Article
Creative Commons
Creative Common License - CCCreative Common License - BYCreative Common License - NCCreative Common License - ND
This is an Open Access article, distributed under the terms of the Creative Commons Attribution-NonCommercial-NoDerivatives licence (http://creativecommons.org/licenses/by-nc-nd/4.0/), which permits non-commercial re-use, distribution, and reproduction in any medium, provided the original work is unaltered and is properly cited. The written permission of Cambridge University Press must be obtained for commercial re-use or in order to create a derivative work.
Copyright
© The Author(s) 2025

1. Introduction

New product development employs various ideation methods in generating ideas for design proposals. These are taught in a design workshop at Brigham Young University to introduce students to divergent thinking and innovation in the product development process. Associative thinking is one of these methods taught in the workshop and was used as the focus of this ideation study. Associative thinking is used to connect ideas, products, or experiences across different areas of knowledge, industries, or geographies (Furr & Dyer, Reference Furr and Dyer2014). This is done with random idea association, also known as “free association,” by connecting unrelated ideas together to make a new solution to a design problem. It is also done with related idea association, also known as “goal-directed association,” by connecting related ideas with a specific goal in mind (Beaty & Kenett, Reference Beaty and Kenett2023). As associative thinking benefits from having a wide range of experiences to draw from to make new and unexpected connections, it can be constrained by a designer’s lack of experience and knowledge in specific problem domains compared to design experts.

ChatGPT has been used in previous design workshops as a tool for students to answer unknown knowledge questions, explore analogous design concepts, and elaborate on design ideas for associative thinking (Bunn et al., Reference Bunn, Wright, Novoa and Howell2024). ChatGPT’s simple conversational interface excels at quickly generating idea lists of user-specified topics much faster than humans. This technology can aid learning and idea exploration by providing access to a wide variety of gathered online data and insights (Schuetzler et al., Reference Schuetzler2024), (Urban et al., 2023). This allows design students to focus more on tasks such as: generating idea content, judging quality, and appropriateness (Gruner & Csikszentmihalyi, Reference Gruner and Csikszentmihalyi2019), (Haase, Reference Haase and Hanel2023) —tasks humans are more capable of compared to artificial intelligence (Cropley et al., 2022).

This research builds on a prior study that evaluated the effects of ChatGPT use on ideation activities in design education, compared to traditional analogue methods. Previous findings suggested that shorter duration ideation sessions and higher team engagement may have contributed to lower ideation outcomes for ChatGPT-assisted associative thinking. Differences in performance were also attributed to the possibility that a shorter 16-minute ideation session may not have fully exhausted a student’s design ideas when using only analogue methods compared to using ChatGPT to assist with ideation. This study hypothesizes that using a longer ideation session of approximately 24 minutes will (1) improve ideation outcomes for the ChatGPT-assisted approach and (2) validate increased team interaction and engagement among students compared to traditional methods.

Ideation outcomes were assessed following the same methods from the previous study to allow comparisons between the two. These methods relied on Divergent Thinking tests to measure the following three ideation outcomes: (1) fluency, the number of ideas generated by each class; (2) flexibility, the diversity of idea categories proposed; and (3) originality, the number of novel, unexpected, or unique ideas within the relevant context of the idea (Clapham, Reference Clapham, Pritzker and Runco2020). The Consensual Assessment Technique (CAT) was used to evaluate these ideation outcomes (Clapham, Reference Clapham, Pritzker and Runco2020). The same voluntary post-workshop survey to gauge participant perception of the ideation activity was also used for quantitative and qualitative evaluation of results. In-class video recordings of the associative thinking activity were used to evaluate team engagement between the two groups.

2. Method

This study was conducted with two separate sections of a product development class, taught at Brigham Young University during May and November of 2024. The class occurs several times each school semester and takes place during a single seven-hour day. Students work together throughout the day, learning and practicing design exercises and techniques, such as SCAMPER, associative thinking, 5-whys, and mind mapping, to design new product proposals for self-identified product problems (Wright & Jones, Reference Wright and Jones2018).

Free and goal-directed associative thinking were taught to both classes to investigate the impact of analogue ideation versus technology-assisted ideation. Classroom instruction is typically performed in-person by an instructor; however, for the associative thinking segment of the class, students viewed pre-recorded videos in class that explained associative thinking techniques, ensuring consistency across both classes. The class that utilized ChatGPT (version 3.5) had additional video instruction demonstrating how ChatGPT could assist with free and goal-directed association techniques and a laptop computer to facilitate ChatGPT conversations. Each team’s ChatGPT session was displayed on a wall-mounted television nearby, allowing students to contribute questions or prompts and see live ChatGPT interactions. Each team worked at a table with three input keyboards connected to the laptop computer giving the opportunity for all team members to contribute to the ChatGPT input.

2.1. Participants

Thirty-nine students participated in the study. Seventeen students (3 teams of 4 students and 1 team of 5 students) participated in the class focused on analogue ideation. Twenty-two students (2 teams of 6 students and 2 teams of 5 students) participated in the ChatGPT-assisted class. Approximately 70 percent of participants were manufacturing, mechanical, or technology engineering students while 30 percent of participants were students outside of the engineering college (e.g., art history, physics, business, computer science, economics, or geology). All participants were undergraduate students ranging from freshmen (first year) to seniors (fourth year). Participants received no extra credit or compensation.

2.2. Study procedure

2.2.1. Analogue-ideation group

Participants in the analogue-ideation class were taught free-associative thinking with an instructional 2.5-minute video. Immediately following, a second 2.5-minute video teaching goal-directed association was presented. Next, students were instructed to collaborate with their team to generate as many ideas as possible for their design problem using techniques described in the videos. Participants used markers and Post-it notes to document ideas using sketches and text descriptions with enough detail so that someone unfamiliar with their problem could understand the concept. Participants had approximately 24 minutes to ideate and document their ideas; at the end of the ideation session, each team’s Post-it notes were collected and the class proceeded to the next course topic. At the workshop’s conclusion, students were asked to complete an 8-question survey about their associative thinking experience from the class. The workshop setting is shown below (Figure 1).

Figure 1. Analogue-ideation group classroom setting

2.2.2. ChatGPT-assisted ideation group

Participants in the ChatGPT-assisted class followed the same procedures as the analogue-ideation class; however, two additional 1.5-minute videos of ChatGPT instruction and demonstration for free and goal-directed associative thinking were presented. The video demonstration for free association showed how ChatGPT could generate a list of random words and explore associations and connections about ideas or concepts related to a hypothetical design problem.

Figure 2. ChatGPT-assisted ideation group classroom setting

The goal-directed associative-thinking video demonstrated how to use ChatGPT to explore new associations and unfamiliar ideas through question-and-answer chat prompts similar to a conversation with a subject-matter expert for the same hypothetical problem. Participants in the ChatGPT session also generated as many ideas as possible in their teams, documented them on Post-it notes, and collected them after the ideation session. Students were asked to complete the same post-workshop survey at the end of the day. The workshop classroom setting with ChatGPT workstations for each team can be seen below (Figure 2).The previous study introduced free-association first followed by 8 minutes of free-associative thinking. Immediately after, goal-directed association was introduced and a second 8-minute ideation session was completed for a total ideation time of 16 minutes. This study introduced both associative thinking techniques with the same video recordings, but viewed back-to-back. This was immediately followed by a single 24-minute ideation session where students were allowed to use both associative thinking techniques as they desired.

2.3. Synthesis

2.3.1. Ideation outcomes

Both classes’ ideas were categorized and evaluated by two instructors who teach new product development classes. Fluency was measured by quantifying the number of total ideas generated by each team. Ambiguous or incomplete ideas were eliminated from the study results. Identical idea concepts were combined into a single idea to eliminate redundancy. Flexibility was measured by categorizing each team’s ideas through affinity mapping and quantifying the resulting idea groups (Lazar & Feng, Reference Lazar, Feng and Hochheiser2017). Originality was measured by quantifying the number of novel, unexpected, or unique ideas not commonly used in the relevant design domain.

Each team had different colour Post-it notes, to easily distinguish separate team outcomes. An example of organised and quantified ideation outcomes is shown below (Figure 3).

Figure 3. Affinity mapping of ideation outcomes: fluency, flexibility, and originality

2.3.2. Team-ideation behavior

Video recording devices were placed at each team table where participants consented to be recorded during the ideation activity. This was to capture team engagement and interactions between students that relied only on their own ability and fellow team members to generate ideas compared to teams with ChatGPT as an ideation tool. Behaviours were coded and classified thematically to understand student-team ideation behaviour for the different groups.

2.3.3. Post-workshop survey

An internet-based survey was administered to participants at the conclusion of the class. Questions gathered insights into participants’ self-perception of the effectiveness of analogue or ChatGPT-assisted associative thinking using a Likert rating scale from 1-5. Question 1 asked about the perceived effectiveness (i.e., 1 = not effective, 5 = very effective) of associative thinking. Question 2 explored fluency by asking how often the student felt “stuck” when generating ideas for their design (i.e., 1 = often stuck and 5 = rarely stuck). Question 3 uncovered how original they thought their team’s ideas were (i.e., 1 = not original to 5 = very original). Question 4 asked how diverse (flexibility) their ideas were (i.e., 1 = not diverse to 5 = very diverse). Questions 7 and 8 asked how enjoyable and empowering associative thinking was for their teams. In the ChatGPT group, the wording asked how enjoyable and empowering using ChatGPT was to aid associative thinking (i.e., 1 = not enjoyable/empowering to 5 = very enjoyable/empowering). The participants’ major of study and team designation were also gathered. The survey took, on average, less than five minutes to complete.

3. Data analysis and results

3.1. Ideation outcomes

The ideation outcome data from the two classes, team-ideation behaviour, and post-workshop experience survey results are outlined below.

Table 1 summarizes the total fluency, flexibility, and originality results for all teams in each class. Ideation outcomes have been calculated per student because of differing class sizes between the analogue-ideation class (n = 17) and the ChatGPT-assisted class (n = 22).

Table 1. Ideation outcomes for associative thinking idea generation

3.2. Team-ideation behaviour

Classroom video recordings were assessed per student and coded into categories based on proportion of time spent on specific behaviours during the 24-minute ideation activity. Behaviour categories were assigned for time spent on: (1) Documentation (i.e., sketching or writing on Post-it notes), (2) Discussion (i.e., talking with team members), and (3) Passive Activities (i.e., listening to team members, reflecting, reading ChatGPT output). Classifying passive ideation activities (category 3) is more subjective in terms of its direct connection to ideation outcomes; however, these activities were important ideation behaviours observed and noted to help understand performance differences between the analogue and ChatGPT-assisted groups. Passive activities, such as listening to team members or reflecting, were grouped together in the same category due to the challenge of distinguishing between them from outside observation. Other activities such as organizing Post-it notes or standing up to stretch were also observed and included in the passive activity behaviour. Table 2 presents the summary of team-ideation behaviours calculated as the average percentage of participants observed in the video recording of the ideation activity. The number of participants in video-recorded observations is smaller than the class sizes due to technological limitations and the number of students who consented to be recorded.

Table 2. Team-ideation behaviours for associative-thinking idea generation

3.3. Post-workshop survey

The post-workshop survey was completed by 34 of 39 students attending as it was not a mandatory requirement of the course. Results for self-perception of fluency, flexibility, originality, and overall effectiveness for associative thinking from both classes are shown in Table 3. Results are reported on a scale from 1 (lowest) to 5 (highest).

Table 3. Self-perceived ideation outcomes and associative thinking effectiveness

The post-workshop survey results for self-perception of enjoyment and empowerment of associative-thinking activities are shown in Table 4 on a scale from 1 (lowest) to 5 (highest).

Table 4. Self-perceived enjoyment and empowerment rating in associative thinking activities

4. Discussion

4.1. Ideation outcomes

Based on results from Table 1, longer ideation time appears to not improve ideation outcomes for a ChatGPT-assisted approach for associative thinking. It was predicted that students’ idea generation might decline over time as their ideas were exhausted, while a ChatGPT-assisted approach could spark more novel ideas, leading to higher ideation outcomes. However, this is not supported in the results summarized in Table 1. This study continues to show supporting evidence that ChatGPT-assisted ideation yields lower fluency, flexibility, and originality outcomes compared to analogue ideation. Comparatively, flexibility and originality outcomes were only slightly lower in the ChatGPT-assisted class, however, the fluency outcome was larger. Students in the analogue-ideation class produced, on average, over twice the number of total ideas compared to students in the ChatGPT-assisted class.

Compared to the previous study, all ideation outcomes from the 24-minute ideation activity were lower than the 16-minute activity. Students with 1.5 times more time to generate ideas came up with less quantity, diversity, and originality of ideas generated regardless of an analogue or ChatGPT-assisted approach. Other factors, such as the perception of longer ideation times, may be contributing to more team discussion and passive ideation activities compared to classes with shorter ideation time. The perception of longer ideation time may also have contributed to a lack of urgency leading to more team discussion rather than recording interim ideas on Post-it notes. It is important to note that passive activities are not to be interpreted as a distraction or detriment to producing higher ideation outcome results, even though that was the performance metric in these studies. Taking the time to process ideas is important; however, in a classroom setting, it appears that the perception of available time with increased group discussion may have a greater impact on results than either an analogue or ChatGPT ideation approach. Therefore, shorter ideation times are still recommended for a classroom setting to maximize ideation outcomes, maintain student focus, and engagement.

4.2. Team-ideation behaviour

The results from Table 2 offer a clearer understanding of why the ChatGPT-assisted approach yields lower ideation outcomes for student teams, while promoting higher levels of team engagement. It was observed in the previous study that teams were more engaged in discussion, design exploration, and new knowledge investigation compared to the analogue group. It was predicted in this study that ChatGPT interaction (i.e., lists of related ideas, topics, and expert knowledge surrounding a design problem) would stimulate more verbal idea explorations and conversations in a team environment. There is evidence that supports this prediction, as shown by the small percentage increase in time spent discussing ideas in the ChatGPT-assisted group compared to the analogue group. The more notable difference in behaviour is between the proportion of time spent documenting ideas and engaging in passive activities between the two groups. The analogue group spent over four times more time documenting ideas than the ChatGPT group, which better explains the correlation with higher fluency outcomes. With less time spent documenting ideas, the ChatGPT group engaged more in team discussions and passive activities, such as listening to team members, reflecting on new ideas, and evaluating ChatGPT’s output. Since ChatGPT generates lengthy responses to input prompts almost instantly, students spent additional time processing and discussing the information together as a group. Although multiple team members were able to use ChatGPT with multiple keyboard inputs and a wall-mounted television, one person facilitated the ChatGPT interface. This resulted in little to no contribution in generating ideas on Post-it notes for that team member. This may contribute to lower fluency; however, the increased time spent discussing ideas and engaging in passive activities seems to be a more significant factor for the ChatGPT group.

4.3. Post-workshop survey

Results summarising self-perceived ideation outcomes in Table 3 indicate that originality and flexibility were perceived as higher, while fluency was perceived as lower in the ChatGPT group compared to the analogue group. These results are different from the previous study and may be an indication of how longer ideation times affect perceptions of the quantity and originality of ideas produced. Enjoyment and empowerment ratings, as shown in Table 3, indicate that ideation activities in a team environment without ChatGPT still have higher enjoyment levels compared to using ChatGPT with longer ideation times.

4.4. Limitations

The shorter ideation time was perceived as a limitation in the previous study as more ideation time was thought to yield higher ideation outcomes. The correlation between lower ideation outcomes with longer ideation time was surprising and shows shorter ideation time does not limit fluency, flexibility, and originality in a classroom setting. This result merits further investigation beyond this study into team dynamics and how other human factors and perceptions influence ideation outcomes.

4.5. Future work

The majority of participants actively documented ideas on Post-it notes and participated in team discussion. However, it is clear from closely observing team behaviours that each team member is unique and varies in inclinations towards active or passive ideation activities. This was confirmed in video observations where some participants did not contribute to team discussions and chose only to contribute with Post-it note ideas. Other participants chose to not document any ideas on Post-it notes and only contributed to team discussion. This study also highlights the relationship between time needed to document new ideas and discuss those ideas a team environment. Exploring ways to empower all team members as active contributors could empower the diversity of team dynamics as well as improve ideation outcomes. This will be the focus of future research by providing each team member an individual ideation session prior to collaborating together as a team, thereby enabling more opportunities for contribution of all team members.

5. Conclusion

Conducting a longer ideation session for associative thinking was predicted to enhance ChatGPT’s effectiveness in idea generation because shorter duration sessions may not fully exhaust a student’s design ideas using a traditional analogue approach; however, results of this study do not support this prediction. Results show that fluency, flexibility, and originality outcomes were lower using a longer ideation session for the ChatGPT-assisted group compared to the analogue group. Additionally, the longer ideation session resulted in lower fluency, flexibility, and originality for both analogue and ChatGPT-assisted groups compared to the previous study using only a 16-minute ideation session. This striking result of 1.5 times the ideation time resulting in lower ideation outcomes suggests shorter ideation time is preferred for product development workshops and merits further investigation into team dynamics. For longer-duration ideation sessions and limited experience with generative AI, ChatGPT may not be as useful to incorporate in the ideation process if maximising ideation outcomes is the desired result. Team behaviour observations confirmed predictions of higher team engagement with the ChatGPT-assisted approach to ideation. However, this approach resulted in lower ideation outcomes as teams spent less time on documenting new ideas and significantly more time engaging in team discussion compared to the analogue group. Future work will explore combining individual ideation with team collaboration to maximize ideation outcomes while also investigating ChatGPT as an ideation tool in this hybrid approach.

References

Beaty, R. E., & Kenett, Y. N. (2023). Associative thinking at the core of creativity. Trends in Cognitive Sciences, 27(7), 671683. https://doi.org/10.1016/j.tics.2023.04.004 CrossRefGoogle Scholar
Bunn, B. J., Wright, G., Novoa, Munoz M., & Howell, B. F. (2024). EXPLORING CHATGPT’S IMPACT ON STUDENT-TEAM IDEATION OUTCOMES FOR NEW PRODUCT DEVELOPMENT: A PILOT STUDY. In DS 131: Proceedings of the International Conference on Engineering and Product Design Education (E&PDE 2024) (pp. 533538). https://doi.org/10.35199/EPDE.2024 CrossRefGoogle Scholar
Clapham, M. M. (2020). Testing/measurement/assessment? In Pritzker, S., & Runco, M. (Eds.), Encyclopedia of creativity (third edition) (pp. 624631). Academic Press. https://doi.org/10.1016/B978-0-12-809324-5.23825-X CrossRefGoogle Scholar
Cropley, D. H., Medeiros, K. E., & Damadzic, A. (2023). The intersection of human and artificial creativity. In Creative provocations: Speculations on the future of creativity, technology & learning (pp. 1934). Cham: Springer International Publishing. https://doi.org/10.1007/978-3-031-14549-0_2 CrossRefGoogle Scholar
Furr, N., & Dyer, J. (2014). The innovator’s method: bringing the lean start-up into your organization. Harvard Business Review Press. https://hbr.org/product/the-innovator-s-method-bringing-the-lean-start-up-into-your-organization/16914 Google Scholar
Gruner, D. T., & Csikszentmihalyi, M. (2019). Engineering creativity in an age of artificial intelligence. The Palgrave handbook of social creativity research, 447462. https://doi.org/10.1007/978-3-319-95498-1_27 CrossRefGoogle Scholar
Haase, J., & Hanel, P. H. (2023). Artificial muses: Generative artificial intelligence chatbots have risen to human-level creativity. Journal of Creativity, 33(3), 100066. https://doi.org/10.1016/j.yjoc.2023.100066 CrossRefGoogle Scholar
Lazar, J., Feng, J. H., & Hochheiser, H. (2017). Chapter 8-Interviews and focus groups. Research methods in human computer interaction, 187228. https://doi.org/10.1016/b978-0-12-805390-4.00008-x CrossRefGoogle Scholar
Schuetzler, R. (2024). Student Interaction with Generative AI: An Exploration of an Emergent Information-Search Process In Proceedings of the 57th Hawaii International Conference on System Sciences (pp.75007509). https://doi.org/10.24251/hicss.2023.901 CrossRefGoogle Scholar
Urban, M., (2024). ChatGPT improves creative problem-solving performance in university students: An experimental study. Computers & Education, 215, 105031. https://doi.org/10.1016/j.compedu.2024.105031 CrossRefGoogle Scholar
Wright, G. A., & Jones, M. D. (2018). Innovation in the elementary classroom. Technology and Engineering teacher, 77(5), 813.Google Scholar
Figure 0

Figure 1. Analogue-ideation group classroom setting

Figure 1

Figure 2. ChatGPT-assisted ideation group classroom setting

Figure 2

Figure 3. Affinity mapping of ideation outcomes: fluency, flexibility, and originality

Figure 3

Table 1. Ideation outcomes for associative thinking idea generation

Figure 4

Table 2. Team-ideation behaviours for associative-thinking idea generation

Figure 5

Table 3. Self-perceived ideation outcomes and associative thinking effectiveness

Figure 6

Table 4. Self-perceived enjoyment and empowerment rating in associative thinking activities