Hostname: page-component-65f69f4695-2qqrh Total loading time: 0 Render date: 2025-06-26T14:06:49.726Z Has data issue: false hasContentIssue false

Applying user-centered design to enhance the usability and acceptability of an mHealth supervision tool for community health workers delivering an evidence-based intervention in rural Sierra Leone

Published online by Cambridge University Press:  11 April 2025

Cara M. Antonaccio
Affiliation:
Alpert Medical School, Brown University, Providence, RI, USA
Justin Preston
Affiliation:
Alpert Medical School, Brown University, Providence, RI, USA
Chokdee Rutirasiri
Affiliation:
School of Social Work, Boston College, Chestnut Hill, MA, USA
Sunand Bhattacharya
Affiliation:
School of Social Work, Boston College, Chestnut Hill, MA, USA
Musu Moigua
Affiliation:
Caritas Freetown, Freetown, Sierra Leone
Mahmoud Feika
Affiliation:
Caritas Freetown, Freetown, Sierra Leone
Alethea Desrosiers*
Affiliation:
Alpert Medical School, Brown University, Providence, RI, USA
*
Corresponding author: A. Desrosiers; Email: alethea_desrosiers@brown.edu
Rights & Permissions [Opens in a new window]

Abstract

Mobile health (mHealth) platforms have the potential to increase access to evidence-based interventions in low-resource settings. This study applied a user-centered design (UCD) approach to develop and evaluate an mHealth supervision tool for community health workers (CHWs) delivering an early childhood development intervention in rural Sierra Leone. We engaged CHWs (N=8) and supervisors (N=4) in focus group discussions, user testing sessions and exit interviews to gather feedback on the mHealth supervision tool’s usability and acceptability. Mixed methods findings indicated that the tool was generally well-received and perceived as easy to use, but there were also challenges related to connectivity, phone charging and the need for more comprehensive training and support. Overall, this study suggests that a UCD approach can promote the usability of mHealth tools to support CHWs in delivering evidence-based interventions in low-resource settings, highlighting the importance of addressing contextual challenges and providing adequate training and support to ensure the effectiveness and sustainability of such tools.

Information

Type
Research Article
Creative Commons
Creative Common License - CCCreative Common License - BY
This is an Open Access article, distributed under the terms of the Creative Commons Attribution licence (http://creativecommons.org/licenses/by/4.0), which permits unrestricted re-use, distribution and reproduction, provided the original article is properly cited.
Copyright
© The Author(s), 2025. Published by Cambridge University Press

Impact statement

This study illustrates the user-centered design process and evaluation of a mobile health (mHealth) supervision tool to enhance the delivery of an evidence-based early childhood development intervention in rural Sierra Leone. By actively involving community health workers and supervisors in the design and development of the mHealth supervision tool, our goal was to tailor the mHealth tool to their specific needs and challenges. We found that the mHealth tool met supervision needs and empowered users with new technological skills. The user-centered design process in this study has the potential to be replicated and scaled in similar low-resource settings. By bridging the gap between technology, workforce capacity and community-based care, this study contributes to the growing body of evidence supporting the use of mHealth strategies to address key global health inequities.

Introduction

Intergenerational cycles of trauma and violence, coupled with limited societal infrastructure and economic opportunities, pose significant risks for millions of children and families in conflict-affected communities. In Sierra Leone, two decades after an 11-year civil war and nearly a decade after the 2014–2016 Ebola outbreak, child mortality rates remain among the highest globally, and child physical abuse and maltreatment are prevalent (UNICEF, 2024). Resource constraints have limited the government’s ability to address these challenges. However, several programs, such as the Interaction Competencies for Teachers (Masath et al., Reference Masath, Hermenau, Nkuba and Hecker2020) and the Family Strengthening Intervention for Early Childhood Development plus Violence Prevention (FSI ECD+VP/Sugira Muryango; Betancourt et al., Reference Betancourt, Franchett, Kirk, Brennan, Rawlings, Wilson, Yousafzai, Wilder, Mukunzi, Mukandanga, Ukundineza, Godfrey and Sezibera2020a), have effectively increased access to evidence-based child development interventions by training nonspecialists to deliver the intervention and by using alternative delivery approaches (i.e., home delivery) in resource limited settings within sub-Saharan Africa.

While evidence-based interventions (EBIs) like the FSI-ECD+VP demonstrate promise for supporting children and families affected by adversity, access and maintaining fidelity to intervention delivery remain key challenges in settings like the rural regions of Sierra Leone, largely due to implementation barriers (i.e., limited health care workforce, transportation difficulties, poor infrastructure; Lyon and Koerner, Reference Lyon and Koerner2016). Innovative approaches are needed to overcome these barriers to accessing EBIs that promote positive parenting practices and early childhood development outcomes for families with young children. The use of trained nonspecialists to deliver EBIs, combined with mobile health (mHealth) strategies, may help increase the reach and efficiency of service delivery by optimizing resource utilization and improving outcomes (Bunn et al., Reference Bunn, Gonzalez, Falek, Weine and Acri2021; Desrosiers et al., Reference Desrosiers, Schafer, Esliker, Jambai and Betancourt2021; Mudiyanselage et al., Reference Mudiyanselage, De Santis, Jörg, Saleem, Stewart, Zeeb and Busse2024; Winters et al., Reference Winters, Langer, Nduku, Robson, O’Donovan, Maulik, Paton, Geniets, Peiris and Nagraj2019). For example, mHealth strategies can support the ongoing supervision and quality improvement of CHWs providing in-home services (Triplett et al., Reference Triplett, Johnson, Kiche, Dastrup, Nguyen, Daniels, Mbwayo, Amanya, Munson, Collins, Weiner and Dorsey2023). In rural areas of Sierra Leone, where infrastructure challenges are common (i.e., transportation costs, poor roads, low internet connectivity,frequent power outages), mHealth tools can address these challenges during the design and development process by incorporating features like offline functionality, access to cloud storage and using battery-powered tablets. Using mHealth tools with offline functionality can reduce the need to travel for supervision, thus reducing the burden of costs (i.e., fuel) and time related to transportation for in-person meetings or Wi-Fi access.

Integrating user-centered design (UCD) principles in the development and tailoring of mHealth strategies is a critical advancement for effective implementation (Ettinger et al., Reference Ettinger, Pharaoh, Buckman, Conradie and Karlen2016; Poulsen et al., Reference Poulsen, Hickie, Alam and LaMonica2023; Stephan et al., Reference Stephan, Dytz Almeida, Guimaraes, Ley, Mathias, Assis and Leiria2017). This iterative process centers the perspectives, knowledge and needs of people with lived experience, fostering the co-creation of accessible, feasible and sustainable interventions (Lyon and Koerner, Reference Lyon and Koerner2016). This approach has been used previously to design and develop a digital tool for training nonspecialists to deliver an evidence-based psychological treatment for depression in primary care in India (Khan et al., Reference Khan, Shrivastava, Tugnawat, Singh, Dimidjian, Patel, Bhan and Naslund2020).

Building on the potential of mHealth interventions and UCD, the current study applied UCD processes to develop an mHealth-based supervision tool to support delivery of a culturally adapted version of the FSI-ECD+VP in Sierra Leone. The FSI-ECD+VP has demontrated effectiveness in promoting early child development, caregiver mental health and positive parenting practices among families living in extreme poverty in Rwanda (Barnhart et al., Reference Barnhart, Farrar, Murray, Brennan, Antonaccio, Sezibera and Betancourt2020; Betancourt et al., Reference Betancourt, Franchett, Kirk, Brennan, Rawlings, Wilson, Yousafzai, Wilder, Mukunzi, Mukandanga, Ukundineza, Godfrey and Sezibera2020a; Jensen et al., Reference Jensen, Placencio-Castro, Murray, Brennan, Goshev, Farrar and Betancourt2021). The FSI-ECD+VP consists of 12 modules focused on topics such as the importance of early stimulation for child development, positive parenting practices, conflict resolution, stress management, and father engagement. The FSI-ECD+VP has also shown preliminary benefits for promoting caregiver mental health and preventing household violence among vulnerable families with young children in Sierra Leone (Desrosiers et al., Reference Desrosiers, Sarani, Albanese, Antonaccio, Neville, Esliker and Betancourt2024). In this study, we used the analyze, design, develop, implement and evaluate framework (ADDIE; Dick, Reference Dick and Carey1996), a five-phase UCD process, to iteratively design and implement mHealth supervision and fidelity monitoring tools for CHWs and supervisors. By centering the needs and preferences of CHWs and supervisors throughout the development process, we aimed to create a supervision tool that was not only user-friendly and acceptable but also responsive to the practical demands of the context.

Methods

Sampling and recruitment

The study protocol was approved by the Boston College Internal Review Board (Protocol #21.006.01) and the Sierra Leone Ethics and Scientific Review Committee. All participants provided oral informed consent before participating in the study. We recruited CHWs (N = 8; four males, four females) from two Peripheral Health Units (PHUs) in rural areas of the Makeni region in Sierra Leone, along with their direct supervisors (N=4; two males, two females). CHWs were eligible if they were 18 years or older, able to commit to attending three 90-min sessions and currently employed in delivering maternal and child health services within the Makeni region. Supervisors were required to be 18 years or older and actively working as supervisors of CHWs providing services in the same region. The PHU Focal Person recommended CHWs and supervisors who were in good standing and expressed interest in the project. The study Project Coordinator contacted potential participants in the order in which referrals were made by the PHU Focal Person. Those who were eligible and provided informed consent were enrolled in the study until the target sample size was reached.

CHWs and supervisors completed a three-week training on the FSI-ECD+VP, which included a combination of didactic instruction on intervention content, role plays and group discussions. Focusing on a specific intervention may have limited the generalizability of our findings to other interventions; however, this focus also allowed us to develop a more tailored and usable mHealth supervision tool, which could support future scale-out of the FSI-ECD+VP and/or be adapted for use with other family home-visiting services to monitor delivery quality and improve feedback cycles during supervision. In addition to training on the FSI-ECD+VP, participants also completed a 1-day technology training on the mHealth supervision tool.

ADDIE process framework

CHWs and supervisors served as experts in the mHealth tool design and development process. Before beginning the UCD process, we defined the end goal as the creation of a mobile app to enhance the delivery quality of the FSI-ECD+VP as well as the supervision process between CHWs and supervisors. We also explored CHWs’ technical literacy and familiarity with mobile tools (i.e., tablets, mobile phones) in a brief survey to help inform the UCD process and development of the mHealth supervision tool. The survey asked about CHWs’ experience using mobile devices, their ability to use basic features such as texting and browsing the internet, and their comfort level with learning new technologies. We then launched the UCD process. Activities during each phase are described below.

Analyze

The program manager, a member of the in-country research team, facilitated two hybrid problem analysis focus group discussions (FGDs), blending in-person meetings in Makeni with remote teleconference sessions led by the design team. This approach allowed for direct engagement with participants while leveraging the expertise of the design team. The analysis phase explored the current challenges that CHWs and supervisors encountered in their day-to-day practice and pinpointed specific problem indicators, such as the need for improved documentation, data collection and communication between CHWs and supervisors. The insights from this in-depth problem analysis – including user needs, preferences and contextual factors – directly informed the design and development of the mHealth tool to help ensure its relevance and fit within the local health system. The initial problem analysis FGDs, facilitated by the in-country program manager, focused on gaining a deep understanding of the current service monitoring and supervision processes from the perspectives of CHWs and supervisors. We explored their experiences, challenges and perceived needs, and thoughts on how the integration of mHealth tools could potentially streamline and enhance these processes. Participants were encouraged to share specific recommendations for resources that would be most beneficial in supporting CHWs during the use of mHealth technology. We created a “mind map” (a visual representation of important factors and processes) based on the qualitative findings from FGDs to illustrate the supervision process and how it relates to CHWs and supervisors’ specific needs. The mind map revealed key challenges faced by CHWs and supervisors, such as the need for improved documentation, data collection and communication tools. These findings directly informed the design and development of the mHealth tool.

Design and develop

Leveraging findings from the problem analysis phase, we developed an initial prototype of the mHealth app that incorporated the contextual findings from the analysis phase. The design team included two faculty members at Boston College with extensive prior experience in UCD processes as well as one postdoctoral fellow who provided support. The primary design team leader (one faculty member) facilitated hybrid teleconference sessions remotely, while the in-country program manager convened the CHWs and supervisors in-person. We then conducted two rounds of iterative user interface/user experience testing (UI/UX) with both CHWs and supervisors via this hybrid format.

User testing sessions were guided by the Think-Aloud Testing protocol method (Charters, Reference Charters2003), which encouraged participants to articulate their thoughts and actions in real-time as they interacted with the mHealth tool. During the think-aloud testing session, we asked a series of questions to understand CHWs’ experiences with the mHealth supervision tool. Think-aloud questions focused on the clarity and ease of mHealth tool navigation, the visual appeal and structure of the tool, the readability of the text and understandability of icons or images and any aspects that were confusing or challenging. We also asked participants for feedback on potential features to add or remove, what features they thought were the strongest, and whether they believed the tool would be helpful for supervision, performance monitoring and useful for other CHWs and supervisors. Each user-testing session was audio-recorded, translated and transcribed. Real-time observations and feedback from user testing sessions directly informed iterative refinements to the prototype to enhance its user-friendliness.

Implement

Following the design and development phases, the final mHealth supervision tool prototype was implemented by CHWs and supervisors who delivered the FSI-ECD+VP to families with young children in rural areas of the Makeni region. While the mHealth tool prototype was finalized before implementation, we collected feedback through interviews and surveys to inform potential updates and improvements to consider in the future. The phones used in the study were supplied by the research team to the CHWs and supervisors for the duration of the project. CHWs used the mHealth tool to help guide the delivery of session content, track progress with different families and monitor their delivery quality (i.e., fidelity to session content and competency). Sessions were delivered during home visits and recorded by CHWs, with supervisors using the tool to remotely track CHW session delivery progress, assess session delivery quality via digitized fidelity checklists and identify areas for improvement. Supervisors had access to the same information as CHWs on the mHealth tool, which included the session recordings, tracking tools to monitor CHW progress, and fidelity checklists.

Evaluate

CHWs and supervisors completed the System Usability Scale (SUS; Brooke, Reference Brooke1996), a validated Likert-style questionnaire measuring perceived usability, before the FSI-ECD+VP was delivered and during the evaluation phase after implementation of all FSI-ECD+VP sessions had concluded. CHWs and supervisors also completed qualitative exit interviews exploring them Health tool’s feasibility, acceptability and usability. All interviews were audio-recorded, translated and transcribed. Data from the evaluation phase provided insights into the specific design elements and app functions that resonated with users, as well as potential areas for improvement in future iterations of the mHealth tool.

Convergent parallel mixed methods design

We used a convergent parallel mixed methods design (Creswell & Plano Clark, Reference Creswell and Plano Clark2018) to evaluate the usability, feasibility and acceptability of the codesigned mHealth supervision tool among CHWs and supervisors. This design leveraged the strengths of both qualitative and quantitative data collection and analysis techniques integrating the findings for a deeper exploration of user experiences and perceptions (qualitative) while also providing a structured assessment of the tool (quantitative). The mixed methods analysis process involved the first and second authors comparing qualitative themes with quantitative usability data to examine how, if at all, the mHealth tool aligned with user needs and preferences.

Data and procedures

Qualitative data were collected from two problem analysis FGDs with CHWs and supervisors, the think-aloud protocol during user testing, and from qualitative exit interviews with CHWs (N=4) and supervisors (N=2) during the evaluation phase. Quantitative data were collected using the SUS, which was administered before and after the implementation of the mHealth tool, to assess changes in user perceptions on mHealth tool usability.

Analytical approach

Qualitative and quantitative data were analyzed separately and then integrated in a joint display figure. Paired t tests were used to examine mHealth system usability perceptions pre- and post-implementation. Qualitative themes were compared with quantitative usability perceptions to assess whether the tool’s design and implementation aligned with user needs and preferences. This triangulation process helped to promote the comprehensiveness of the findings. The first and second authors (CA, JP) independently coded qualitative transcripts from FGDs and exit interviews with CHWs and supervisors, applying a combination of inductive and deductive coding techniques. Inductive codes emerged organically from the data, while deductive codes were derived from the study’s aims and existing literature.

We then used reflexive thematic analysis (RTA; Braun and Clarke, Reference Braun and Clarke2019) to systematically identify, analyze and interpret patterns (themes) in the qualitative data. We sought to gain a comprehensive understanding of the challenges faced by CHWs and supervisors in their current roles, their specific needs and preferences regarding the mHealth supervision tool and the potential impact of such tools on their work experiences and service delivery. The iterative nature of RTA allowed for the refinement of codes and themes throughout the analysis process and facilitated ongoing awareness of the potential influence of subjective biases related to the interpretation of findings. This reflexive approach ensured that the final thematic framework accurately reflected the complexities and nuances of the data.

Results

The demographic characteristics of the participants are presented in Table 1. The sample (N=12; eight CHWs and four supervisors) was equal in terms of representation from male and female CHWs and supervisors. CHW participants were 36.1 years old, on average (SD=9.1) and of the eight CHWs, one had 1 year of experience in their role, four had between 2 and 5 years of experience and three had between 6 and 10 years. Supervisors ranged in age from 37 to 50 years.

Table 1. Demographic characteristics of CHWs and supervisors

ADDIE process

Findings from FGDs during the analysis phase indicated that CHWs and supervisors’ existing service delivery model primarily relied on in-person interactions, manual note-taking and occasional audio recordings. CHWs and supervisors expressed a desire for tools that could streamline documentation, improve communication and feedback and facilitate data collection and progress tracking. The problem analysis FGD findings and the resulting mind map informed the organization of the components of the mHealth tool’s information infrastructure and UI/UX. For example, FGDs identified the need for streamlined documentation and improved communication in the supervision process; this feedback informed the design and development of the tool’s digital supervision checklist features. Similarly, the UI/UX was influenced by the need to accommodate varying levels of technological literacy, which necessitated a simple and intuitive interface with clear navigation and prominent icons.

The results of the design and development phases are demonstrated in Figures 1 and 2. Figure 1 demonstrates the information architecture of the mHealth supervision tool, illustrating how the tool’s components work together. Figure 2 exhibits the first prototype of the mHealth supervision tool including the UI, core features, and how the various functionalities are used in the supervision process. CHWs and supervisors received a brief training on the functions and features of the tool. CHWs and supervisors completed a 1-day, in-person technology training on the use of the mHealth tools. The training plan was guided by feedback gathered throughout the design process and included a walkthrough of the key functions and descriptions of each (see Figure 3). The training involved hands-on practice and technical assistance to troubleshoot questions. It sought to guide CHWs and supervisors through the features and functions of the mHealth tool and the procedures for using it (in accordance with the study protocol) during their home-visiting sessions with each family. After the training and before implementation, CHWs generally expressed positive perceptions of the tool’s usability. They indicated a willingness to use the system frequently (mean = 4.5 out of 5) and felt confident in their ability to do so (mean = 4.5). The system was perceived as fairly easy to use (mean = 4.14) and not overly complex (mean = 2.38), with well-integrated functions and components (mean = 4.25). However, there was a moderate perceived need for technical assistance (mean = 3.13) and some indication of inconsistency within the system (mean = 2.13).

Figure 1. mHealth app information architecture (version 1).

Figure 2. mHealth supervision tool prototype.

Figure 3. Final mHealth tool training for CHWs and supervisors.

Pre- to post-implementation system usability findings

A mixed methods joint display matrix is presented in Table 2 to demonstrate mHealth system usability findings via the integration qualitative and quantitative user perceptions. The triangulation of quantitative usability data with qualitative evidence from each stage of the UCD process revealed that the mHealth supervision tool met many of the needs and preferences of CHWs and supervisors and it helped to improve their ability to deliver the FSI-ECD+VP sessions with quality. Post-intervention usability findings similarly indicated that CHWs generally found the mHealth supervision tool easy to use (mean = 4.13) and felt confident using it (mean = 4.5). CHWs also reported a high likelihood after using the mHealth tool that they would use the system frequently (mean = 4.75) and perceived the system’s functions and components as well-integrated (mean = 4.5).

Table 2. Joint display of mHealth supervision tool system usability perceptions (N=8)

However, CHWs also found the system somewhat complex (mean = 2.88) and difficult to use (mean = 2.75) and expressed a need for technical assistance (mean = 4.75). The results of the paired t test suggest that, compared to before implementation, CHWs had mixed perceptions of the tool’s usability after using it during FSI-ECD+VP implementation (See Table 2). Small, non-significant increases were observed in the CHWs’ desire to use the system (i.e., the mHealth tool) frequently and in their confidence in using the system. However, there were also small, non-significant increases in the perceived complexity and difficulty of using the system. The only statistically significant change in usability was an increase in the perception that they would need help from a technical person to use the system.

Strengths of the mHealth system

Both the quantitative usability metrics (high scores on ease of use and usefulness) and qualitative feedback indicated that the mHealth tool was generally well-received and perceived as easy to use and helpful in CHWs’ work. For example, one CHW stated, “The experience was good…all the equipment was ok.” (CHW 4) Another CHW simply stated, “It’s easy to use, I use it well.” (CHW 2) Quantitative findings on system usability were consistent with qualitative feedback that highlighted the tool’s helpfulness in providing practical guidance and decision support for home-visiting sessions. One CHW noted, “The tools direct us what to say and during the training we were taught a lot.” (CHW 1), while another shared, “It helps me a lot. I can now advise mothers better.” (CHW 2) Qualitative evidence suggests that the mHealth tool positively impacted service delivery by enhancing CHWs’ ability to conduct home visits, improving communication with supervisors and increasing their knowledge and confidence.

Areas to improve the mHealth system

In addition to perceived strengths of the tool, CHWs identified ways that the mHealth supervision tool and the way that it was implemented could be improved. For example, quantitative analysis revealed a statistically significant increase in the perceived need for technical support after tool implementation. A slight, though not statistically significant, increase was observed in perceived inconsistency within the system along with a slight decrease in the perception that most people would learn to use the system quickly. This might suggest that the tool’s interface and learning curve could be further refined to optimize user experience; and aligns with feedback from a CHW during user testing who stated, “The only concern that I have, the only concern for me, we have not yet learned a lot about it. It’s the only concern that I have. They have just introduced it to us, we have not yet learned about it.” The qualitative feedback also highlighted challenges with connectivity, inconsistent electricity access for phone charging and requests for more comprehensive training and troubleshooting assistance. One supervisor mentioned, “The only challenge for the cell phone is maybe if we don’t have light [electricity],” while a CHW added, “They should add [to] the training.” (CHW 3)

Discussion

This study highlights the potential of mHealth tools to support the delivery and supervision of EBIs in low-resource settings, contributing to a growing body of evidence on the usability of such tools. By centering the needs and preferences of end-users throughout the design process, we developed a tool that was both acceptable and usable for CHWs and supervisors in rural areas of Sierra Leone. The hybrid UCD approach we used can serve as a model for developing and implementing mHealth tools in other resource-constrained settings or for different evidence-based behavioral health interventions, with the primary goal of increasing engagement and adoption of mHealth tools that are user-friendly.

Our findings demonstrate that the mHealth supervision tool was generally well-received by CHWs and supervisors. Both quantitative usability metrics (high scores on ease of use) and qualitative feedback indicated that the mHealth tool was generally well-received and perceived as easy to use and helpful in CHWs’ work. Qualitative feedback suggests that the mHealth tool positively impacted service delivery by enhancing CHWs’ ability to conduct home visits, improving communication with supervisors and increasing their knowledge and confidence. However, we also identified several areas for improvement. For example, quantitative analysis revealed a statistically significant increase in the perceived need for technical support during tool implementation. Qualitative feedback also highlighted challenges with connectivity, inconsistent electricity access for phone charging and requests for more comprehensive training and troubleshooting assistance.

To address these challenges, future efforts should develop more detailed training materials that cover all aspects of the mHealth tool’s functionality. These materials could include digital content from the FSI-ECD+VP manual, “cheat sheets” on key session topics and goals and video tutorials that are accessible both online and offline. Provide ongoing technical assistance to CHWs and supervisors, either through in-person visits or remote support, could also improve usability in the future. The hybrid UCD methodology, while offering flexibility and reducing costs of international travel, presented challenges related to participant engagement and real-time interaction between participants and the research team. In-person engagement with the participants, coupled with remote teleconference sessions, allowed for direct interaction and leverage of the design team’s expertise. However, the remote aspect of the approach might have limited the quality of interaction and rapport-building between the research team and participants, possibly affecting the depth and richness of the feedback obtained. Additionally, technical difficulties and inconsistent internet connectivity presented minor challenges for real-time communication and collaboration during the UCD process.

Despite these contextual challenges, the findings from this study underscore the potential for mobile tools to reduce barriers to EBI implementation and access in low-resource settings. The tool’s flexible design and user-friendly interface make it adaptable to a variety of intervention contexts and content. Additionally, the UCD approach can be applied to the development of mHealth tools for other EBIs to help ensure that the tools are tailored to the needs and preferences of end-users. Findings also highlight the importance of understanding and then designing and developing mobile tools that incorporate features and functions to address contextual challenges (i.e., transportation issues, infrastructural and financial constraints), which may limit the usability and ultimate scalability of mHealth tools in rural, resource-constrained settings. Future research could also consider strategies such as providing transportation allowances, integrating literacy support within the training or exploring alternative solutions for areas with limited connectivity to mitigate some of these challenges.

In conclusion, this study provides preliminary support for applying UCD methods to improve the acceptability and usability of mHealth tools to improve the supervision process and delivery quality of evidence-based behavioral interventions in low-resource settings. By centering the needs and preferences of end-users throughout the design process, it is possible to develop tools that are not only feasible and acceptable, but also highly usable and effective in LMICs and other resource-constrained settings. Findings highlight the importance of addressing contextual challenges, providing adequate training and support, and understanding the local technology infrastructure to maximize the benefits of mHealth tools in rural, resource-constrained contexts.

Open peer review

To view the open peer review materials for this article, please visit http://doi.org/10.1017/gmh.2025.38.

Supplementary material

The supplementary material for this article can be found at http://doi.org/10.1017/gmh.2025.38.

Data availability statement

Data used in this study are publicly available from ClinicalTrials.gov/NCT04481399.

Author contribution

CA and JP conducted mixed methods analyses for the study and conceptualized the presentation of key findings. CR and SB led UCD processes by facilitating codesign and development, leading user-testing workshops, developing training materials and documenting app development. MM and MF facilitated codesign processes in Sierra Leone, leading workshops in-country on behalf of the US-based research team. AD conceptualized the study, oversaw the analysis and provided a critical review of the manuscript. All authors approved the submitted version of the manuscript and agreed to be personally responsible for their own contributions.

Competing interests

The authors declare none.

Funding information

This work was supported by the National Institute of Mental Health (grant number R21 MH124071).

Ethics statement

The study protocol was approved by the Boston College Internal Review Board (Protocol #21.006.01) and the Sierra Leone Ethics and Scientific Review Committee. All participants provided oral informed consent before participating in the study.

References

Betancourt, TS, Franchett, E, Kirk, CM, Brennan, RT, Rawlings, L, Wilson, B, Yousafzai, A, Wilder, R, Mukunzi, S, Mukandanga, J, Ukundineza, C, Godfrey, K and Sezibera, V (2020a) Integrating social protection and early childhood development: Open trial of a family home-visiting intervention, Sugira Muryango, Early Child Development and Care, 190(2), 219235. https://doi.org/10.1080/03004430.2018.1464002Google Scholar
Braun, V and Clarke, V (2019) Reflecting on reflexive thematic analysis. Qualitative Research in Sport, Exercise and Health 11(4), 589597. https://doi.org/10.1080/2159676X.2019.1628806Google Scholar
Brooke, J (1996) SUS—A Quick and Dirty Usability Scale. Usability Evaluation in Industry, 189, 47.Google Scholar
Bunn, M, Gonzalez, N, Falek, I, Weine, S and Acri, M (2021) Supporting and sustaining nonspecialists to deliver mental health interventions in Low- and Middle-Income Countries: An umbrella review. Intervention 19(2), 155179. https://doi.org/10.4103/INTV.INTV_47_20.Google Scholar
Barnhart, DA, Farrar, J, Murray, SM, Brennan, RT, Antonaccio, CM, Sezibera, V and Betancourt, TS (2020) Lay-worker delivered home visiting promotes early childhood development and reduces violence in Rwanda: A randomized pilot. Journal of Child and Family Studies 29, 18041817.Google Scholar
Charters, E (2003) The use of think-aloud methods in qualitative research an introduction to think-aloud methods. Brock Education Journal 12(2). https://doi.org/10.26522/brocked.v12i2.38Google Scholar
Creswell, JW and Plano Clark, VL (2018) Designing and Conducting Mixed Methods Research (3rd ed.). Thousand Oaks, CA: SAGE.Google Scholar
Desrosiers, A, Sarani, I, Albanese, AM, Antonaccio, CM, Neville, SE, Esliker, R and Betancourt, TS (2024) Task-sharing to promote caregiver mental health, positive parenting practices, and violence prevention in vulnerable families in Sierra Leone: A pilot feasibility study. BMC Psychiatry 24(1), 787.Google Scholar
Desrosiers, A, Schafer, C, Esliker, R, Jambai, M and Betancourt, T (2021) mHealth-supported delivery of an evidence-based family home-visiting intervention in Sierra Leone: Protocol for a pilot randomized controlled trial. JMIR Research Protocols 10(2), e25443. https://doi.org/10.2196/25443Google Scholar
Dick, W and Carey, L (1996) The Systematic Design of Instruction (4th ed.). New York: Harper Collins.Google Scholar
Ettinger, KM, Pharaoh, H, Buckman, RY, Conradie, H and Karlen, W (2016) Building quality mHealth for low resource settings. Journal of Medical Engineering & Technology 40(7–8), 431443. https://doi.org/10.1080/03091902.2016.1213906Google Scholar
Jensen, SK, Placencio-Castro, M, Murray, SM, Brennan, RT, Goshev, S, Farrar, J and Betancourt, TS (2021) Effect of a home-visiting parenting program to promote early childhood development and prevent violence: A cluster-randomized trial in Rwanda. BMJ Global Health 6(1), e003508.Google Scholar
Khan, A, Shrivastava, R, Tugnawat, D, Singh, A, Dimidjian, S, Patel, V, Bhan, A and Naslund, JA (2020) Design and development of a digital program for training non-specialist health workers to deliver an evidence-based psychological treatment for depression in primary care in India. Journal of Technology in Behavioral Science 5(4), 402415. https://doi.org/10.1007/s41347-020-00154-7Google Scholar
Lyon, AR and Koerner, K (2016) UCD for psychosocial intervention development and implementation. Clinical Psychology: A Publication of the Division of Clinical Psychology of the American Psychological Association 23(2), 180200. https://doi.org/10.1111/cpsp.12154Google Scholar
Masath, FB, Hermenau, K, Nkuba, M and Hecker, T (2020) Reducing violent discipline by teachers using Interaction Competencies with Children for Teachers (ICC-T): Study protocol for a matched cluster randomized controlled trial in Tanzanian public primary schools. Trials 21(1), 4. https://doi.org/10.1186/s13063-019-3828-zGoogle Scholar
Mudiyanselage, KWW, De Santis, KK, Jörg, F, Saleem, M, Stewart, R, Zeeb, H and Busse, H (2024) The effectiveness of mental health interventions involving non-specialists and digital technology in low-and middle-income countries – a systematic review. BMC Public Health 24(1), 77. https://doi.org/10.1186/s12889-023-17417-6Google Scholar
Poulsen, A, Hickie, IB, Alam, M and LaMonica, HM (2023) User experience co-design of a mobile application to support childrearing in low- and middle-income countries. Studies in Health Technology and Informatics 304, 8690. https://doi.org/10.3233/SHTI230377Google Scholar
Stephan, LS, Dytz Almeida, E, Guimaraes, RB, Ley, AG, Mathias, RG, Assis, MV and Leiria, TL (2017) Processes and recommendations for creating mHealth apps for low-income populations. JMIR mHealth and uHealth 5(4), e41. https://doi.org/10.2196/mhealth.6510Google Scholar
Triplett, NS, Johnson, C, Kiche, S, Dastrup, K, Nguyen, J, Daniels, A, Mbwayo, A, Amanya, C, Munson, S, Collins, PY, Weiner, BJ and Dorsey, S (2023) Understanding lay counselor perspectives on mobile phone supervision in Kenya: Qualitative study. JMIR Formative Research 7, e38822. https://doi.org/10.2196/38822Google Scholar
United Nations Interagency Group for Child Mortality Estimation, Report 2024. www.data.unicef.org, Retrieved April 1, 2024.Google Scholar
Winters, N, Langer, L, Nduku, P, Robson, J, O’Donovan, J, Maulik, P, Paton, C, Geniets, A, Peiris, D and Nagraj, S (2019) Using mobile technologies to support the training of community health workers in low-income and middle-income countries: Mapping the evidence. BMJ Global Health 4(4), e001421. https://doi.org/10.1136/bmjgh-2019-001421Google Scholar
Figure 0

Table 1. Demographic characteristics of CHWs and supervisors

Figure 1

Figure 1. mHealth app information architecture (version 1).

Figure 2

Figure 2. mHealth supervision tool prototype.

Figure 3

Figure 3. Final mHealth tool training for CHWs and supervisors.

Figure 4

Table 2. Joint display of mHealth supervision tool system usability perceptions (N=8)

Supplementary material: File

Antonaccio et al. supplementary material

Antonaccio et al. supplementary material
Download Antonaccio et al. supplementary material(File)
File 444.9 KB

Author comment: Applying user-centered design to enhance the usability and acceptability of an mHealth supervision tool for community health workers delivering an evidence-based intervention in rural Sierra Leone — R0/PR1

Comments

Dear GMH editorial team:

This study details the development and evaluation of a mobile health (mHealth) supervision tool designed for community health workers (CHWs) delivering an early childhood development intervention in rural Sierra Leone. We believe this study is highly relevant to Global Mental Health’s focus on mental health in low- and middle-income countries. Our findings demonstrate the potential of mHealth technology to bridge gaps in mental healthcare access and support evidence-based interventions in resource-constrained settings.  

Specifically, our study highlights:

- The feasibility and acceptability of mHealth supervision tools for CHWs in low-resource contects.  

- The importance of user-centered design in ensuring the cultural relevance and practical usability of mHealth tools.  

- Key considerations for implementation, including training, technical support, and addressing contextual challenges.  

We are confident that this study will be of great interest to the readership of Global Mental Health, particularly researchers, practitioners, and policymakers engaged in mHealth initiatives, early childhood development programs, and global mental health efforts.

Thank you for considering our work.

Sincerely,

Dr. Antonaccio and colleagues

Review: Applying user-centered design to enhance the usability and acceptability of an mHealth supervision tool for community health workers delivering an evidence-based intervention in rural Sierra Leone — R0/PR2

Conflict of interest statement

Reviewer declares none.

Comments

Thank you for sharing this very interesting paper with me, I really enjoyed reading it, it is quite well-written and describes the development process in detail which makes it as easy and compelling read. There are a few minor points for consideration:

1. The tiile is User-centered design of an mHealth supervision tool to enhance the delivery of an evidence-based home visiting intervention (FSI-ECD+VP) for families in rural Sierra Leone

Considering the methodology and aim of the study, the title should reflect feasibility and acceptability in the title instead of simply saying enhancing

2. In introduction, lines 39-54 could include some details about how the innovative solutions would help with infrastructure and transportation related barriers

3. It would be good to know out of how many CHWs and Supervisors was this sample of N=8 and N= 4 selected and what was the sampling method?

4. Though there is some information given, would like to know more details about the kind of training and experience of the CHWs and Supervisors specific to the intervention being implemented

5. How was technical literacy of CHWs explored?

6. What was the mindmap and findings from the problem analysis?

7. Were the supervisors able to access the exact same information that was available to the CHWs on the app?

8. In analyze it is mentioned that there were 2 problem analysis FGDs while in data and procedures, 3 problem analysis FGDs are mentioned. Was there another FGD?

9. In Table 1, CHWs N is mentioned as 8 but in age columns the N adds up to 12, are the supervisors also included in it?

10. How will the discrepancy between qualitative and quantitative data be addressed as it doesn’t give any information about the inconsistencies in the system?

Review: Applying user-centered design to enhance the usability and acceptability of an mHealth supervision tool for community health workers delivering an evidence-based intervention in rural Sierra Leone — R0/PR3

Conflict of interest statement

Reviewer declares none.

Comments

This useful article helps to bring insight into how UCD is being leveraged in the field. A very useful read.

A few comments and suggestions for improving the manuscript are as follows:

1. The inclusion criteria do not state that the CHWs or supervisors had to have any experience with the Family Strengthening Intervention for Early Childhood Development plus Violence

Prevention; yet, throughout that paper, it seems that knowledge or experience in this intervention is critical to the testing, as it is part of the initial UCD goal. Would you clarify this in the article, either via the methods or throughout, whether knowledge in this area was needed, as well as whether focusing on this specific intervention may or may not limit the outcomes, such as translating it more broadly to supervision practices in this area?

2. Please add a general description of the proposed delivery and supervision process of FSI-ECD+VP, either in the intro or as part of the setting description in the methods.

3. The last paragraph of the intro includes a result/conclusion statement (line 40-47). This should be rephrased or removed.

4. The methods need more description from the design team - who they are, where they are, and how many they are. Please also be clearer on the schedule of the sessions that were remote vs in person, such as in a table. Also, itd be ideal to recognize if there was a single facilitator in the design team (the program manager?) of these sessions or multiple facilitators (which could be included when describing who this team is).

5. Who defined the end goal, was it the design team or the design team and the CHWs and supervisors?

6. Who is the program manager?

7. Curious why or whether current examples of similar platforms were used in any of the discovery processes (the first 2 stages) to leverage brainstorming or facilitate a platform that could be useful beyond this setting or intervention, as stated in the goal of the study?

8. Please include the list of questions used in the think aloud session, or at the least examples of the questions with an overall number or range, and detail whether these were asked in structure (all in one order) or flexible (skip around and not need to ask all) during the process.

9. Were any notes taken during observation? Also, was the observation remote or in person, was it a visual observation of how they used the tool by hand or via an app to capture swipes and clicks? Similarly, during the initial stage of discovery with FGDs, did any of the design team sit or observe the actual process of delivery or supervision (if given client permission) and/or visit the site to get sense of comms between CHWs and supervisors? Its not clear whether this type of observation happened.

10. line 28 in “Implement” says the product was finalised, though typically iterative changes should continue during implementation. Please clarify - is this a final product that will only change if more funding comes, or does the UI/UX allow iterative changes during implementation? I noticed further data collected via interviews and surveys for potential changes in the future, but is that really the case or are these all “nice to haves” if you can get to it?

11. Were the same CHWs and supervisors that designed it the ones that tested/implemented it / were there any blind CHWs or supervisors during testing?

12. authors write “brief training”. please elaborate via hours or days, remote or in-person. Also, the figure isn’t exactly clear, it seems like the training was only via an e-course-- was there a trainer that could give hands-on training with feedback? Finally, it describes the training as guiding “proper” use of the tool, but what does “proper” mean?

13. could you clarify if the phones belong to the CHWs and supervisors, or if they are supplied?

14. there is a lot of room to add to the discussion about the results including feedback on wanting to know more about “how to use it”. can the authors say more about how it was used, for instance: how much time does it take to fill out, do they have it out when they’re with clients, do they need permission, how does the permission go for the recording; how does this tool change or compare to the way supervision and fidelity was done before among CHWs; where are the recordings stored; do they need to listen to the whole recording to score and how does a score impact over multiple sessions; do CHWs use it independently as feedback for how to improve, or is that up to the supervisors discretion (what to improve, when), how often do CHWs and supervisors discuss results and is this the issue with the transportation? etc.

15. As a reader, the transportation issues are confusing. Is it that CHWs didn’t visit the home at first? Its hard to understand how that issue interferes with using the application, unless these results are also alongside testing the implementation of the home visits, the intervention itself, and consistent supervision between sessions?

16. Similar to point 15, incentives for who to use what? No incentive was described earlier about testing this app, so what is it, do they mean incentive to do recordings with the client or incentive to deliver the intervention?

17. It seems striking the language barrier wasn’t sorted prior, could the authors say more about this? for instance, was it because these are new users and not those included in the design, or is it related to the clients? Also, what part did the language interfere, useability of the app entirely (knowing where to click), the fidelity check list, all?

18. The discussion could be strengthened, for instance, why not discuss more on exactly what could be improved immediately, such as the training that was asked for by the participants? Similarly, it’s quite common that tech like this needs a helping hand, so why not discuss more on how the authors might answer to this in the future, or whether they will? Especially in terms of applicability beyond FSI.

Review: Applying user-centered design to enhance the usability and acceptability of an mHealth supervision tool for community health workers delivering an evidence-based intervention in rural Sierra Leone — R0/PR4

Conflict of interest statement

I declare no competing interests.

Comments

Thank you for this excellent paper demonstrating the important role of mHealth in reaching difficult to reach populations and the important role of CHW in such areas.

A follow up study with a larger N is recommended.

Recommendation: Applying user-centered design to enhance the usability and acceptability of an mHealth supervision tool for community health workers delivering an evidence-based intervention in rural Sierra Leone — R0/PR5

Comments

There are quite a few concerns raised by one of the reviewers and the other reviewer as well. Hope you will be able to address all these concerns.

Decision: Applying user-centered design to enhance the usability and acceptability of an mHealth supervision tool for community health workers delivering an evidence-based intervention in rural Sierra Leone — R0/PR6

Comments

No accompanying comment.

Author comment: Applying user-centered design to enhance the usability and acceptability of an mHealth supervision tool for community health workers delivering an evidence-based intervention in rural Sierra Leone — R1/PR7

Comments

Cara M. Antonaccio, PhD, MSPH

Postdoctoral Research Associate in Clinical Psychology

Department of Psychiatry and Human Behavior

Brown University Alpert Medical School

2 March 2025

Dear Prof. Dixon Chibanda and Dr. Shidhaye,

Thank you for the opportunity to revise and resubmit our manuscript, “Applying user-centered design to enhance the usability and acceptability of an mHealth supervision tool for CHWs delivering an evidence-based family home visiting intervention in rural Sierra Leone”.

We appreciate the feedback from you and the reviewers, and we have revised the manuscript to address all the concerns raised. In response to the reviewers’ feedback, we have made the changes to the manuscript outlined below.

We believe that these changes have strengthened the manuscript and addressed all of the concerns raised by the reviewers. We hope that you will now find it suitable for publication in Cambridge Prisms Global Mental Health.

Sincerely,

Dr. Antonaccio and colleagues

Review: Applying user-centered design to enhance the usability and acceptability of an mHealth supervision tool for community health workers delivering an evidence-based intervention in rural Sierra Leone — R1/PR8

Conflict of interest statement

Reviewer declares none.

Comments

All queries have been addressed by the authors

Recommendation: Applying user-centered design to enhance the usability and acceptability of an mHealth supervision tool for community health workers delivering an evidence-based intervention in rural Sierra Leone — R1/PR9

Comments

No accompanying comment.

Decision: Applying user-centered design to enhance the usability and acceptability of an mHealth supervision tool for community health workers delivering an evidence-based intervention in rural Sierra Leone — R1/PR10

Comments

No accompanying comment.