Policy Significance Statement

Our analysis reveals key policy trends in the implementation of the UN Convention on the Rights of Persons with Disabilities (CRPD) across all existing State Reports (N = 170). We provide policymakers with a systematic assessment of global disability rights, including a strong focus on service access, legal protection, and accessibility improvements, alongside growing emphasis on social justice and economic inclusion. We identify critical policy gaps in international engagement mechanisms. These findings provide actionable insights for policymakers to strengthen disability rights implementation and inclusive development. This study also introduces an innovative methodology to evaluate policy combining text mining, Natural Language Processing (NLP), and GenAI tools. Even with the ongoing challenges and problems with using GenAI tools, our methodology demonstrates significant potential for governments and civil society organizations to monitor multilateral treaties, offering them a tool for evidence-based policy evaluation and accountability, particularly in human rights treaty monitoring.

2. Purpose

Nearly 18 years after its adoption, assessing the implementation of the CRPD is vital. This study aimed to: (1) establish a replicable process for identifying CRPD implementation priorities at national, regional, and global levels; (2) evaluate the focus on social justice and economic inclusion for persons with disabilities; and (3) assess our hybrid approach of text mining and Natural Language Processing (NLP) with GenerativeAI tools using large language models (LLMs).

3. Literature review

Our literature review examines six areas. The first area provides a brief overview of relevant literature on national, regional, and global monitoring of CRPD implementation. The second reviews methodological approaches comparing traditional text mining and NLP to LLMs for large-scale text analysis. Third, we focus on knowledge translation articles that contextualize our research in terms of broader policy and social implications. The fourth area investigates the shift away from the medical to rights-based model incited by the CRPD. The fifth expands upon the structural and institutional factors that affect CRPD implementation around the world. Finally, the sixth area explores regional variation in disability rights and inclusion data quality and availability for monitoring of CRPD implementation progress.

4. Conceptual framework

CRPD implementation is driven by a combination of state and nonstate actors, including regional and subregional organizations. Country State Reports signal national priorities. Through a detailed analysis of State Reports, we can identify CRPD progress and areas requiring more attention as illustrated by Figure 1.

Figure 1. Conceptualizing CRPD implementation.

5. Research questions

We ask three broad research questions, each operationalized with a series of inductive (exploratory) and deductive (confirmatory) italicized subquestions:

6. Methodology

Our hybrid methodology applies inductive and deductive text mining and NLP techniques to analyze the complete corpus of CRPD State Reports (n = 179) and uses GenAI tools to further analyze a subset of the corpus (n = 20). We use RStudio (http://posit.co/) as our Integrated Development Environment (IDE) and R and Python programming languages. We leverage a suite of R and Python packages including: tm, tidytext, quanteda, Gensim, spaCy, and TextBlob. For the GenAI methods, we used Google AI Studio and the Gemini 1.5 Flash LLM (http://aistudio.google.com) along with their NotebookLM (http://notebooklm.google.com) and the PaLM 2 LLM, and OpenAI’s ChatGPT and the GPT-4o LLM (http://chatgpt.com).

6.5. Generative AI and LLM analysis

To answer RQ4, we utilized Generative AI tools. GenAI tools allow the researcher to harness the capabilities of various LLMs. Using an LLM requires the user to develop a prompt engineering strategy which may include supplying the LLM with “examples.” These examples, called “shots,” may be zero-shot (no examples), one-shot (one example), or few-shot (two or more examples). A related approach is to use retrieval-augmented generation (RAG)-AI approaches. With RAG-AI, the user supports their analysis by supplying specific documents that can be used as “ground truth.” The user then submits the prompt(s) to the LLM using either a GUI or an API. Most current GenAI models use the transformer architecture, such as GPT (generative pretrained transformer), or Google’s BERT (Bidirectional Encoder Representations from Transformers). A Transformer is a deep learning model, first introduced by Google (Vaswani et al., Reference Vaswani, Shazeer, Parmar, Uszkoreit, Jones, Gomez, Kaiser and Polosukhin2017). Once the prompt is submitted, the transformer works to understand your prompt, and then sends it through its multilevel deep learning model to refine the predicted outcome. Figure 2 illustrates this process.

Figure 2. Transformer architecture, adapted from Vaswani et al., Reference Vaswani, Shazeer, Parmar, Uszkoreit, Jones, Gomez, Kaiser and Polosukhin2017.

To address RQ4, we developed a multishot prompt and RAG-AI strategy encompassing all questions from our traditional NLP analysis. We used a sample of 20 State Reports as the ground truth in our RAG-AI approach using the Gemini 1.5 Flash LLM. We set the temperature to 1, leaving other settings unchanged. We used a similar RAG-AI approach using Google’s NotebookLM, which uses PaLM (Pathways Language Model) and a sample of 50 State Reports. NotebookLM has a more user-friendly interface and offers many and growing features but is more limited in terms of customizations. We also used OpenAI’s ChatGPT, selecting the GPT-4 LLM. Our initial prompt for each of these GenAI tools was as follows:

“The twenty documents I provided are reports from a sample of countries that have signed and ratified the United Nations Convention on the Rights of Persons with Disabilities (CRPD), making them a state party to the convention. CRPD States Parties are required to produce these state reports. Please review these documents and respond with the following information, organized by the section numbers below: 1. What are the most important ideas contained in these documents, as represented by keywords and phrases? Also, please identify the top ten ‘topics’ represented in the documents. 2. What ‘named entities’ are found in these documents, please organize the response by ‘Organizations’, ‘Organizations of Persons with Disabilities’, ‘People’, ‘Government Entities’, and ‘Legal Documents’. 3. Which of the 50 articles in the CRPD are most represented in these documents?”

Through iterative prompt development, we asked the GenAI tools research questions that mirrored those we asked using the traditional text mining and NLP techniques.

7. Findings

We now organize and present our findings aligned with our research questions and subquestions. Our first main question, RQ1, asks: What national priorities for CRPD implementation are reflected in the global corpus of State Reports? The goal with this question is to develop a basic understanding of the areas on which State Parties report focusing while implementing the convention. To answer RQ1.1 and RQ1.2 about important keywords and phrases, we used term frequency only. TF*IDF, normally a useful heuristic, yielded country names and is not reported. The most frequent word is “regional,” followed by “required” and “professional,” suggesting prevalent regional approaches. Other frequent words such as “independent,” and “comprehensive” indicate the report structure. “Visual” and “impairment” highlight that community. Well-being-related words such as “abuse” and “risk” are present. Figure 3 illustrates frequently occurring words.

Figure 3. Frequently occurring words.

Figure 4 shows the 25 most frequent bigrams, echoing themes from Figure 3. The blind and low vision community represented by “visually impaired.” Technology featured in “assistive devices,” “assistive technology,” and “universal design.” Well-being represented by “cruel inhuman,” and “sexual abuse,” “recreation leisure,” and “fundamental freedoms.”

Figure 4. Frequently occurring phrases (bigrams).

RQ1.3 asks: What topics are present in the corpus? To answer this subquestion, we used LDA topic modeling. We fit a 10-topic LDA model to the data and identified the 30 most salient terms associated with those topics. The Python Gensim package uses a library called pyLDAvis to produce an interactive visualization of topic models, along with salient words. A static version is presented in Figure 5.

Figure 5. Static representation of the most salient terms in the topic model.

Table 1 summarizes the topics, identifying the terms most salient within that specific topic. Please note, the terms “disability” and “person” are ubiquitous across all the topics (which is to be expected, given the dataset) and are removed to aid topic interpretation.

Table 1. LDA salient terms and topics

Three other topics were multilingual documents and should have been pulled.

Our LDA topic model yields a coherence score of 0.461, which is considered moderate. This score ranges from 0 to 1, with higher scores reflecting better topic quality and semantic relatedness among the top words in each topic. In this case, some topics are cohesive, and others lack definition or contain unrelated words.

To better analyze the variety and kinds of people, places, and organizations referenced in the State Reports, RQ2 asks: What named entities are present in the corpus? We ask two subquestions: RQ2.1: What types of named entities appear most frequently in the corpus?; and RQ2.2: Are prominent organizations of persons with disabilities mentioned in higher/lower frequency than governmental entities? We use the spaCy model to identify the common entity types in Table 2.

Table 2. Most common SpaCy entity types

The spaCy package uses transformer-based deep learning models for NER. These models create contextualized token representations that capture semantic and syntactic patterns, utilizing either integrated pre-trained transformers or spaCy’s own word vectors as foundational features (Honnibal et al., Reference Honnibal, Montani, Van Landeghem and Boyd2020). Figure 6 illustrates the frequently occurring named entities in our corpus.

Figure 6. Frequently occurring entities.

The most frequently occurring named entities in the CRPD State Reports are CARDINAL numbers, organizations, dates, legal references, geopolitical entities, and persons. These categories suggest a strong emphasis on quantitative reporting, institutional actors, and references to legal frameworks and jurisdictions throughout the corpus. The high frequency of CARDINAL numbers suggests that State Reports often include quantitative information, such as statistics, counts of services, or budget figures, to demonstrate compliance or progress. The prominence of organizations (ORG) highlights the central role of institutional actors, including government bodies, international agencies, and civil society organizations, in implementing the CRPD. Frequent mentions of dates indicate temporal framing, such as referencing legislation enactment, program start dates, or reporting periods. The presence of legal references (LAW) reflects a strong reliance on rights-based language and formal frameworks, while the frequent occurrence of geopolitical entities (GPEs) shows that national and regional contexts are a key dimension of how States frame disability inclusion. Finally, the identification of persons (PERSON), although less frequent, may reflect individual case examples, testimonies, or leadership references used to humanize or exemplify progress. Overall, the distribution of named entities reveals how reporting is grounded in quantifiable evidence, institutional structures, and legal commitments, with some narrative elements woven in through references to individuals. These findings are certainly supported by our expert close-reading sanity check as well.

Next, we explored the top 10 named entities within selected high-frequency entity types, including ORG, PERSON, GPE, NORP, and LAW. Figure 7 displays the most frequently mentioned organizational entities (ORG) in the CRPD State Reports.

Figure 7. Frequently occurring ORG entities.

The most common ORG entity is “State,” which likely reflects frequent references to the “State Party” or “the State” in the context of reporting obligations under the CRPD. This aligns with the treaty’s structure, which centers State responsibility for implementation. Other commonly referenced organizational entities include “Convention” and “Government,” underscoring the role of formal governance structures and international legal instruments. Mentions of “EU” and “UN” point to the influence of regional and international organizations in shaping disability policy and monitoring. Terms such as “PWD,” “CRPD,” and “Reply” likely stem from standardized reporting language, including responses to the Committee on the Rights of Persons with Disabilities. The appearance of “the Ministry of Health” indicates the involvement of national ministries, particularly in areas like health service provision and disability assessment. This finding may indicate the continued persistence of “medical model” approaches. These results suggest that organizational references in the reports are heavily institutional and formal in nature, centered on governance, accountability, and international oversight. They also reflect the administrative structure through which States articulate their implementation of the CRPD—primarily through national ministries and through interactions with global bodies.

To better understand what specific portions of the CRPD are receiving the most attention in State Party reporting, RQ3 asks: To what extent is the breadth of the CRPD covered in the corpus? We ask eight operationalized subquestions—to answer RQ3.1 to RQ3.7, we applied the CRPD_dictionaryFinal; to answer RQ3.8, we applied the MedvsRights_dictionaryFinal.

RQ3.1 asks: What CRPD articles are most represented in the corpus? Overall, in the 25 most represented articles shown in Figure 8, Article 8 (Awareness-raising) is the most represented and deals with measures to raise awareness and foster respect for persons with disabilities (UN, 2006). The second is Article 23 (Respect for home and the family) and deals with measures to ensure marital, childbearing, familial, and parental rights for persons with disabilities (UN, 2006).

Figure 8. Most represented CRPD articles.

RQ3.2 asks: What CRPD article paragraphs are most represented in the corpus? Overall, the 25 most represented paragraphs are shown in Figure 9. The most represented CRPD paragraph is Article 1 (Purpose)—Paragraph 2 which outlines who a person with disability is and how their disability in interaction with other barriers may “hinder their full and effective participation in society on an equal basis with others” (UN, 2006). Article 8 (Awareness-raising)—Paragraph 1(b) is the second most represented. It specifies that State Parties should adopt effective measures to “combat stereotypes, prejudices and harmful practices relating to persons with disabilities, including those based on sex and age, in all areas of life” (UN, 2006).

Figure 9. Most represented CRPD paragraphs.

RQ3.3 asks: What categories of the CRPD are most and least represented in the Corpus? The dictionary captures the following article categories: introductory (2, 100%), substantive (13, 61%), general applications (5, 71%), implementation and monitoring (4, 40%), and operational rules (1, 10%). The most frequently occurring article paragraphs come from the following article categories: introductory 2 (100%), substantive 9 (43%), general applications 3 (43%), and implementation and monitoring 3 (30%). Figure 10 illustrates the least represented CRPD articles; they fall into the following article categories: 0 introductory, 8 substantive (39%), 2 general applications (29%), 6 implementation and monitoring (60%), and 9 operational rules (90%).

Figure 10. Least represented CRPD articles.

RQ3.4 asks: How does the distribution of CRPD articles differ by region? The top 10 most represented articles by region are shown in Figure 11. All five regions share two substantive articles: Article 13 (Access to justice), which deals with ensuring effective access to justice for persons with disabilities by facilitating their role as participants (witness, defendants, and so on) and providing training for working with persons with disabilities in the field of administration of justice such as police and prison staff (UN, 2006), and Article 23 (Respect for home and the family). Europe was the only region that had the substantive Article 26 (Habilitation and Rehabilitation), which deals with measures to enable persons with disabilities to attain and maintain maximum independence, full physical, mental, social and vocational ability by organizing, strengthening, and extending comprehensive habilitation and rehabilitation services (UN, 2006).

Figure 11. Most represented CRPD articles by region.

The top 10 least represented articles by region are shown in Figure 12. All five regions only share one substantive article, Article 10 (Right to life), which recognizes the inherent right to life for all human beings and deals with measures to ensure the effective enjoyment of that right by persons with disabilities (UN, 2006). All regions except Oceania share the substantive Article 20 (Personal mobility) that deals with measures “to ensure personal mobility with the greatest independence” (UN, 2006). Europe and Oceania share the substantive Article 17 (Protecting the integrity of the person), which recognizes the right to respect for physical and mental integrity (UN, 2006).

Figure 12. Least represented CRPD articles by region.

RQ3.5 asks: How does the distribution of CRPD articles differ by subregion? The top 10 most represented articles by subregion are shown in Figures 13 and 14. Some subregions show 11 instead of the 10 most represented articles because the percentages of the 10th and 11th are tied. Analysis was done for all subregions but, to save space, only the analysis for the European and Asian regions are shown.

Figure 13. Most represented CRPD Articles by Asia.

Figure 14. Most represented CRPD articles by Europe.

The five Asian regions, shown in Figure 13, share three substantive articles: Article 12 (Equal recognition before the law), which deals with measures to ensure persons with disabilities can enjoy and exercise their legal capacity as well as have equal rights to own or inherit property and have equal access to financial systems (UN, 2006), Article 23 (Respect for home and the family), and Article 27 (Work and employment), which deals with the rights of persons with disabilities to: (1) work on an equal basis with others; (2) gain a living by work freely chosen or accepted in a labor market; and (3) a work environment that is open, inclusive, and accessible (UN, 2006).

South-Eastern, Southern, and Western Asia each have no unique substantive article. Central Asia has two unique substantive articles: Article 26 (Habilitation and Rehabilitation) and Article 30 (Participation in cultural life, recreation, leisure and sport) which deals with recognition of persons with disabilities’ specific cultural and linguistic identity (sign languages, deaf culture, and so on), and the rights of persons with disabilities to participate on an equal basis in cultural life (television shows, museums, and so on), recreational, leisure, and sporting activities (UN, 2006). The only unique substantive article for Eastern Asia is Article 16 (Freedom from exploitation, violence, and abuse) that deals with all legislative, administrative, social, and educational measures to prevent (and assist in recovery from) all forms of exploitation, violence, and abuse (UN, 2006).

The four European regions, shown in Figure 14, share three substantive articles: Article 12 (Equal recognition before the law), Article 23 (Respect for home and the family), and Article 24 (Education) that states that persons with disabilities are able to fully and effectively participate in all aspects of education and educational systems (UN, 2006).

Northern Europe has no unique substantive article. Eastern Europe has two substantive articles: Article 11 (Situations of risk and humanitarian emergencies), which deals with measures to ensure the protection of persons with disabilities in situations of risk such as armed conflict, humanitarian emergencies, and natural disasters (UN, 2006), and Article 30 (Participation in cultural life, recreation, leisure, and sport). Southern Europe’s only unique substantive article is Article 16 (Freedom from exploitation, violence, and abuse). Western Europe’s only unique substantive article is Article 25 (Health) that recognizes and deals with measures to ensure that persons with disabilities have the right to enjoy the highest attainable standard of health without discrimination on the basis of disability (UN, 2006).

RQ3.6 asks: How does the distribution of CRPD articles differ by economic income group? The top 10 most represented articles by income group are shown in Figure 15 below. All four income groups share two substantive articles: Article 23 (Respect for home and the family) and Article 27 (Work and employment). Two substantive articles are shared by all the income groups except the high-income group: Article 11 (Situations of risk and humanitarian emergencies) and Article 24 (Education). The low-income group is the only income group that does not share the substantive Article 12 (Equal recognition before the law). In terms of substantive articles unique to only one income group: Article 16 (Freedom from exploitation, violence and abuse) is unique to the upper-middle-income group and Article 26 (Habilitation and rehabilitation) is unique to the high-income group.

Figure 15. Most represented CRPD articles by income group.

The top 10 least represented articles by income group are shown in Figure 16. All four income groups share two substantive articles: Article 10 (Right to life) and Article 20 (Personal mobility). The high- and upper-middle-income groups are the only ones that share the substantive Article 17 (Protecting the integrity of the person).

Figure 16. Least represented CRPD articles by income group.

RQ3.7 asks: What are the differences in the corpus between states that have and have not ratified the Optional Protocol? The top 10 most represented articles by whether or not the Optional Protocol was ratified are shown in Figure 17. Only the group of countries that had ratified the Optional Protocol had Article 13 (Access to justice)—this was the only substantive article not shared between the two groups.

Figure 17. Most represented CRPD articles by Optional Protocol ratification.

RQ3.8 asks: To what extent is the transition from the medical model to the rights-based model represented in the corpus? Overall, there is indeed a shift from the medical model to the rights-based model as the rights-based model is the most represented globally at 64%. Figure 18 shows that this shift held across all five regions.

Figure 18. Most represented disability model by region.

In order to understand the potential of Generative AI tools to conduct this type of analysis and treaty compliance, RQ4. asks: To what extent can GenAI tools answer the preceding research questions? Our reporting of this analysis is limited by space constraints. In response to our “mega-prompt,” both models returned surprisingly coherent responses. A summarized example of the various LLM responses is presented as follows:

Gemini 1.5 output: Top three “important ideas”: Shifting paradigms: these documents repeatedly emphasize the move away from the “medical model” of disability, which views disability as a personal defect, to the “social model,” which recognizes that societal barriers create disadvantages for people with disabilities. Focus on inclusion: the reports highlight the importance of social inclusion and the need to ensure that people with disabilities have full and equal participation in all aspects of society, including education, employment, and civic life. Accessibility as a fundamental right: the reports stress that accessibility is not just a matter of convenience, but a human right. They address physical, informational, and communication accessibility to ensure equal access.

The top ten topics (using topic modeling): (1) equality and nondiscrimination, (2) awareness-raising and information dissemination, (3) accessibility, (4) education, (5) employment, (6) legal capacity and decision-making, (7) health and medical treatment, (8) rehabilitation and habilitation, (9) participation in public and political life, and (10) social protection.

NER structured by the requested entities (e.g., top five Organizations):

• • National Council for Persons with Disabilities (CONAIPD) (El Salvador, Venezuela)

• • Ministry of Education (multiple countries)

• • Australian Human Rights Commission (Australia)

• • Supreme Council for Family Affairs (Qatar)

• • Productivity Commission (Australia)

GPT-4o output: Top three “important ideas”: Inclusion and accessibility: the documents emphasize the importance of making education, employment, and public facilities accessible to persons with disabilities. This includes physical access, educational materials, and technology adaptations. Legal and policy frameworks: the reports highlight the necessity of robust legal and policy frameworks to protect the rights of persons with disabilities, aligning national laws with the CRPD. Awareness and education: awareness raising campaigns and educational programs are crucial for changing societal attitudes toward persons with disabilities and promoting their rights.

The top ten topics (using topic modeling): 1) education access, 2) employment and labor rights, 3) healthcare and rehabilitation, 4) legislative frameworks, 5) accessibility in public spaces, 6) awareness and advocacy, 7) intersectionality and vulnerable groups, 8) independent living and community support, 9) technology and innovation, and 10) international collaboration.

NER structured by the requested entities (e.g., top five Organizations):

• • National Council for Persons with Disabilities (CONAIPD) (El Salvador, Venezuela)

• • Ministry of Education (multiple countries)

• • Australian Human Rights Commission (Australia)

• • Supreme Council for Family Affairs (Qatar)

• • Productivity Commission (Australia)

On subsequent iterative prompts, the LLMs were able to identify content related to all questions. Space constraints limit our ability to present these responses.

8. Discussion and recommendations

This study highlights varying global CRPD priorities and assesses computational text mining with LLM-based analyses. Word frequency analyses reveal shifts from medical- to rights-based models, confirmed by our custom lexicon showing the rights-based model’s predominance globally and regionally. However, limited emphasis on stigma and barriers indicates reporting gaps. The LDA topic model’s low coherence suggests that future research should improve model parameters and text processing.

Regional differences and their relationship with major regional disability networks designed to support CRPD implementation may contribute to the variation found in our empirical data. In the findings section, we observed specific variation of CRPD articles by regions. These variations reflect differing regional priorities. Europe uniquely emphasizes Article 26 (Habilitation and rehabilitation), highlighting its own regional focus on maximizing the independence of persons with disabilities. The prioritization of Article 26 in Europe could be reflective of the independence-focused influence of major disability networks such as The European Network on Independent Living and the European Disability Forum (EDF, 2021) (ENIL, 2022).

Income group disparities reveal that foundational rights like Articles 23 and 27 (Work) span all groups, while Article 24 (Education) is prominent in low-/middle-income countries and Article 26 in high-income countries. Concerning is Article 12’s (Equal recognition) absence in low-income countries. Countries ratifying the Optional Protocol uniquely emphasize Article 13, suggesting deeper commitment to justice through external accountability.

Our hybrid methodology leverages both GenAI and traditional NLP. Traditional NLP analysis serves as a verification mechanism for GenAI results. While Gemini 1.5 captures the medical-to-rights model shift, neither Gemini nor GPT-4o identify nuances in disability paradigms, well-being, and interventions from our term-frequency analysis. Both LLMs capture aspects of the most represented Articles except Article 23, likely due to over-indexing on early report sections. Additionally, we found periodic reports respond to issues from previous reports rather than providing comprehensive coverage.

Comparing topic models, traditional NLP focuses on children (related to Article 23), while LLMs’ topics reflect their overindexing on Articles 1–5, though GPT-4o less so. Both traditional and LLM-based NER identify similar organization entities, with LLMs adding country context.

Our approach demonstrates traditional NLP’s current advantages over GenAI, though GenAI is improving and will likely address whole-document analysis to mitigate overindexing. This hybrid framework provides valuable CRPD implementation insights while making text analytics accessible to nontechnical researchers. Broadening these methods will help civil society monitor implementation gaps, facilitate parallel reporting, enhance accountability, and improve implementation of complex policies such as the CRPD and SDGs.

9. Limitations

Our analysis, while comprehensive, has limitations. We rely on self-reported data, which may not fully reflect actual policy priorities or impact. Our custom dictionary might miss nuances in CRPD articles, and we did not fine-tune our NER model. Due to token count limits, the LLM analysis covers only a subset of reports, unlike the full-corpus NLP analysis.

10. Conclusions and future research

The study confirms the widespread adoption of the CRPD’s rights-based approach and its role in promoting social justice and economic inclusion. The region with the most complete transition to the rights-based model is the Americas. The paper contributes to both an understanding of the global implementation of the CRPD but also provides a methodology for ongoing monitoring at national, regional, and global levels. We see the paper making a practical contribution to national public policy debates, especially in terms of what are national priorities and where are the gaps in CRPD implementation. Future research will add additional data, including the alternative reports, state action plans, and CRPD Committee reports, and the full corpus for the LLMs.