Intervention development

The intervention was developed based on the CDC outpatient antibiotic stewardship core elements framework. ^{Reference Sanchez, Fleming-Dutra, Roberts and Hicks13} Select members of the VHA Antimicrobial Stewardship Taskforce (ASTF) developed materials for the intervention. ^{Reference Kelly, Jones and Echevarria14} Preparation included development of campaign key messages (Table 1) and educational materials utilized for academic detailing on UTI management. (Supplement 1a & 1b). ^{Reference Wells, Popish, Kay, Torrise and Christopher15} Academic detailing consists of noncommercial, peer-to-peer communication using reinforcement techniques to facilitate change in prescribing practices. ^{Reference Rome, Dancel and Chaitoff16} Additional campaign materials included a step-by-step implementation guide, kick-off slide set, and example UTI-specific computerized electronic order menus.

Table 1. Key messages of pilot UTI campaign

UTI, urinary tract infection.

Metric development and validation

Metrics were developed (Table 2) to align with key messages in support of UTI stewardship in primary care (PC) and emergency department (ED) settings. ^{Reference Rovelsky, Vu and Barrett5,Reference Braunfeld, Samore, Crook, Nevers, Rovelsky and Madaras-Kelly17} Diverse metrics were designed to capture UTI overdiagnosis, antibiotic selection, treatment duration, and appropriate utilization of diagnostic tests. Because limiting the analysis to UTI diagnosis codes did not fully capture use of antibiotics for suspected UTI, a broad set of genitourinary (GU)-related International Classification of Diseases, 10^th revision (ICD-10) codes were utilized. GU codes were classified into tiers: tier 1 (antibiotics always indicated), tier 2 (sometimes indicated), and tier 3 (rarely indicated). ^{Reference Fleming-Dutra, Hersh and Shapiro1} Altogether, 207 GU-related codes were grouped into tiers (Supplement 2). Visits with multiple codes were assigned to the lowest tier for the visit.

Table 2. Metrics utilized in the pilot UTI campaign

Abbreviations: GU, genitourinary; W, with; BPH, benign prostatitic hyperplasia; ASC, acute simple (uncomplicated) cystitis; UTI-NOS, urinary tract infection, not otherwise specified; F, Fahrenheit; TMP/SMX, trimethoprim/sulfamethoxazole; FQ, Fluoroquinolone; UC, urine culture; UTI antibiotic resistance (resistance to FQ, nitrofurantoin, TMP/SMX or cephalosporin), NH, nursing home; UA, urinalysis; UC, urine culture; ASB, asymptomatic bacteriuria; w/, with.

^a A temperature of ≥99.9F (± 1 day) was used to exclude potentially more complicated infection.

Metric A consisted of the proportion of GU visits, including all tiers, which had an antibiotic prescription filled within −1/ + 5 days surrounding the index visit. Metric B was comprised of tier 3 codes for GU conditions from Metric A. Metrics C and D were limited to visits classified as uUTI based on presence of an ICD-10 code for acute cystitis or UTI-NOS and absence of fever (temperature ≥99.9 F). ^{Reference Rovelsky, Vu and Barrett5} The Metric C numerator included trimethoprim/sulfamethoxazole (TMP/SMX), nitrofurantoin, or an oral cephalosporin prescribed, while the denominator included all uUTI visits with antibiotics prescribed. Metric D assessed the proportion of uUTI visits with an antibiotic duration prescribed of ≤7 days. ^{Reference Rovelsky, Vu and Barrett5} In Metric E, the denominator included all UTI visits with an antibiotic resistance risk factor present (≤90-day history of positive UC for antibiotic-resistant organisms, antibiotic prescription, or hospitalization) ^{Reference Rovelsky, Vu and Barrett5,Reference Hooton10–Reference Hooton11,Reference Braunfeld, Samore, Crook, Nevers, Rovelsky and Madaras-Kelly17–Reference Madaras-Kelly, Boyd and Bond20} ; the numerator included all visits in the denominator with a UC collected. Metric F measured the number of urinalyses (UA) performed with an antibiotic prescribed within the (−1/ + 5 d) time window around UA collection, per 1,000 patient visits per month.

A dashboard was developed to display metrics, and a chart-level validation was conducted.^{Reference Braunfeld, Samore, Crook, Nevers, Rovelsky and Madaras-Kelly17} Records for review were randomly selected based on strata [tier, setting (ED vs PC), and antibiotic prescription versus not]. Two reviewers (JB, KMK) independently evaluated records to determine if all inclusion/exclusion criteria were met. Reviewers were blinded to visit diagnostic codes and antibiotic prescription and used clinical judgment to assign a diagnosis and record their confidence in this diagnosis using a Likert scale (1 = highly uncertain, 5 = highly certain). Reviewers also rendered judgments regarding the appropriateness of antibiotic treatment using a Likert scale (1 = highly disapprove, 5 = highly recommend). A Likert score of ≥ 4 was considered a positive endorsement for analysis. Quantitative analysis was used to determine visit placement into tiers, relationships between visit attributes and prescription of antibiotics, and agreement in antibiotic prescription between the reviewer and the visit clinician.

Intervention implementation

The campaign was launched in fall 2022, utilizing a staggered rollout based on facility preference. Stewards were encouraged to use the implementation guide. Stewards were instructed to gain commitment through discussion with facility administration, recruit ED/PC clinical champions, and identify personnel to deliver the intervention. Next, deliver a “Kick-Off” presentation to clinicians on the purpose, key messages, and planned intervention. Facilities were instructed to update UTI electronic order pathways and reflex UC policies if not already aligned with key messages. Stewards were encouraged to track and report campaign metrics to stakeholders through a review of aggregate baseline performance and identification of “clinicians of concern” through metric review. Actions included targeting clinicians of concern with education through academic detailing on key messages and delivery of audit-feedback reports at the clinic and optionally clinician-level for individuals with sufficient visits. However, the initial dashboard was not configured to generate clinician-level reports directly. Based on feedback, the dashboard was updated to allow end users to generate individual clinician-level feedback reports (Figure 1). Stewards were encouraged to prioritize the ED where UTI density was highest. Periodic virtual learning collaboratives for stewards were implemented in January 2023, where barriers and strategies could be shared.

Figure 1. The Figure depicts a two-sided UTI campaign audit-feedback report for an ED clinician. The clinician’s performance is compared to the mean of other clinicians practicing in their ED or PC setting. Reports were emailed or printed and shared directly with clinicians. The acute uncomplicated cystitis definition utilized in the report is intended to reflect the definition for uncomplicated UTI.

Intervention sustainment

The intervention continued through June 2023, when an assessment was developed to ascertain the extent of alignment with the implementation guide, gauge intervention intensity, barriers and enablers, and suggestions for pilot refinement. The assessment, consisting of item selection, short answer, and free text response questions, was distributed to steward leads at each facility. Based on feedback, VISN 21 adopted Metrics C and D as quality improvement measures (minimum performance target of 80%) for fiscal year 2024. Stewards were encouraged to utilize the campaign materials as needed; however, the pilot was concluded.

Cohort development and outcomes

A retrospective cohort of outpatients who met criteria for inclusion was developed for visits between 3/15/2020 and 6/30/2024 from data within the VHA Corporate Data Warehouse (CDW). ^{Reference Fihn, Francis and Clancy21} Inclusion required a visit with a GU ICD-10 diagnosis code in ED or PC settings. Visits with a UTI diagnosis within <30 days, competing tier 1 or 2 coded diagnoses, ^{Reference Fleming-Dutra, Hersh and Shapiro1} hospital discharge within ≤7 days, urological procedure within ± 7 days, pregnancy, or spinal cord injury, marrow or organ transplant in the prior 2 years were excluded (Supplement 2). ^{Reference Rovelsky, Vu and Barrett5}

Data extracted included demographics, visit date, maximal temperature (± 1 d) of visit, UA/UC results, ICD-10 diagnosis, procedures, treatment date, antibiotic, treatment duration, clinician type, clinic setting, facility, antibiotic resistance risk factors, and outcomes.

The primary outcome was a change in the performance rate for each metric pre /and postintervention implementation. A secondary outcome included GU-related return visits between days 7–21 postindex visit date for uUTI pre/post implementation. Additional analyses evaluated the rate of performance on Metrics C and D during the maintenance phase and whether clinicians altered their diagnostic coding practices for uUTI relative to other select GU-diagnoses pre /post implementation.

Two intervention time-periods were evaluated: preimplementation 3/15/2020–9/30/2022 and post-implementation 01/01/2023–9/30/2023. As the intervention was initiated at slightly different time-points in facilities, data during a 3-month transition period (10/1/2022–12/31–22) was excluded. A secondary intervention analysis and time-point included preUTI campaign maintenance data from 01/01/2023–09/30/23 and postintervention maintenance data from 10/01/23–07/01/24.

Statistical analysis

Interrupted time series analyses compared outcomes for intervention facilities with a control VISN. The control was chosen based on the average monthly Euclidean distance of metric rates before the first intervention between each VISN and VISN 21. ^{Reference Madaras-Kelly, Hostler and Townsend22} Regression models were fit using generalized estimating equations (GEE) on each outcome with clustering by facility, assuming an exchangeable covariance matrix, or equal correlation between all subjects within a facility. ^{Reference Hardin and Hilbe23} All models assumed a log link to directly estimate the relative risk of the intervention by exponentiating model covariates. ^{Reference Pedroza and Truong24} Each model included a fixed indicator for the intervention facilities, an indicator for the intervention period, and their interaction. All models additionally adjusted the following as fixed effects: patient age, max temperature, setting, and secular trend designated by an increasing continuous variable incremented by quarter of the year. The main result from each model was the exponentiated covariate representing the interaction between the intervention facilities and intervention period [that is the difference-in-differences (DID) relative risk]. Post hoc contrasts were made to estimate the intervention facilities’ pre to postintervention relative risk and the control VISN’s pre to postintervention relative risk. All analyses were conducted using R version 4.4.1 using the geepack and contrast packages.

The campaign was conducted as an operations activity; however, analysis activities constitute research based on VHA Policy Handbook guideline 1,058.05. This research complies with all federal guidelines and VA policies related to human subjects’ research.