From Frameworks to F1 Scores: Designing a Scalable Communication Assessment System

Overview

Physicians are rigorously tested on medical knowledge, but the communication skills that shape patient outcomes? Almost never measured at scale. At ABIM, I spent six months closing that gap: translating two validated clinical communication frameworks into 11 operationalized rubrics, then building and validating a scalable NLP pipeline that applies them reliably across 985 physician conversations.

Background

Medicine is changing fast. Ambient listening tools transcribe encounters in real time. AI scribes write clinical notes from spoken conversation. Telemedicine has moved care into video calls and async messaging. More physician-patient conversations are being captured than ever before, but the ability to measure what happens in them has not kept pace.

Of all the competencies that determine patient outcomes, communication has the broadest reach: whether a patient understands their diagnosis, follows through on treatment, or feels safe enough to return. It almost always comes down to how the physician communicated. ACGME names Interpersonal and Communication Skills as a core competency alongside Medical Knowledge. Yet medical knowledge gets board exams and communication gets assumptions.

The existing approach is such where trained human raters reviewing transcripts tops out at ~50 conversations per week, with documented reliability problems: rater bias, rater drift, order effects. As transcript volume grows, the gap between what exists and what can be evaluated widens.

ABIM's Innovation team brought the question: can AI detect communication behaviors at scale without losing the measurement quality that makes the output meaningful? We reframed it immediately. This wasn't a model problem: it was a measurement design problem. The rubric, not the classifier, was the answer.

1.1 The Verification Gap in Medicine

ABIM certifies physicians across six core competency domains. While medical knowledge is rigorously tested via formal board examinations, patient interaction is rarely measured at scale:

The result is a massive verification gap: physicians are certified via examination but assumed capable of proper human interaction. The only existing measure, patient satisfaction surveys, is rare, self-selected, and indirect.

Key phrase to use: "We certify doctors via exam but assume they are capable of proper human interaction. This project tried to make that assumption testable."

1.2 Why Communication is Hard to Measure

Communication quality is difficult to quantify because:

• Construct subjectivity: Concepts like empathy, rapport, and shared decision-making are expressed in many different linguistic forms and interpreted differently by different raters.
• Context dependency: What counts as good communication in a routine encounter differs from what is required in serious illness discussions or cross-cultural consultations.
• Invisibility of non-verbal behavior: Body language, eye contact, and tone are not present in transcripts, creating an inherent measurement gap.
• Scale bottleneck: Manual expert review cannot scale beyond ~50 conversations/week without unacceptable quality degradation.

The existing approach (human raters using validated frameworks) has documented reliability problems: rater bias, rater drift, and order effects. These are psychometric problems, reliability and validity issues, not just a technology gap.

1.3 The Research Question

"Can AI-driven pattern recognition achieve the same level of nuance and accuracy as human expert evaluation of clinical communication?"

Three subsidiary questions shaped the study design:

1. Can rubric-grounded annotation reduce rater subjectivity to a reliable, replicable standard?
2. Can LLM-assisted labeling scale annotation without sacrificing construct validity?
3. Can a downstream classifier achieve sufficient per-construct precision/recall to support formative (not deterministic) feedback?

2.1 Study Type: Mixed Methods, Quantitative Primary

The study used a two-strand structure (an enabling qualitative strand feeding into a primary quantitative strand):

Why qual enables quant here: The rubric design phase is the equivalent of instrument design in survey research: it is qual work, but its purpose is to enable the quant phase.

Qualitative Methods Map

Quantitative Methods Map

3.1 Framework Selection

Five frameworks were evaluated. Two were selected as primary:

3.2 From 16 Skills to 12 AI-Detectable Constructs

Framework synthesis identified 16 nuanced medical communication skills. 4 were excluded as non-text-detectable:

Why CCG + NURSE specifically: Together they cover all 12 AI-detectable skills with no gaps, have strong structural integration (NURSE empathy moves sit naturally inside CCG Phase IV), and are both validated, peer-reviewed, and widely used in clinical communication training.

4.1 The Five-Phase Pipeline

Each phase acts as a quality gate, preventing error propagation and construct drift:

4.2 The Pivot to Real Clinical Transcripts

Initial work used synthetic data. We made a deliberate pivot toward real clinical encounters as the primary corpus anchor:

Why this was a construct validity decision: Synthetic conversations contain overly explicit behavioral markers that real encounters lack. A model trained on synthetic data learns a distribution that does not generalize to actual clinical settings: a measurement validity problem, not a model problem.

4.3 LLM-Assisted Labeling Strategy

• 8–10 exemplar conversations manually curated per construct for few-shot grounding.
• Rubric-grounded prompts encoded the full rubric: inclusion criteria, exclusion criteria, structured reasoning output.
• One construct at a time: prevents cross-construct contamination.
• Per-construct model selection: Qwen vs. Llama evaluated separately per construct.

5.1 Inter-Rater Reliability (IRR)

Cohen's κ computed per construct against Landis & Koch (1977) benchmark bands:

The pattern is itself a finding: Constructs with explicit linguistic markers (such as NURSE: Naming: directly labeling a patient's emotion) showed higher agreement. Constructs requiring inference from implicit behavior (such as NURSE: Explore: detecting open-ended emotional inquiry) showed lower agreement. That tells us which behaviors are text-detectable and which require additional signal.

5.2 Classifier Performance (BERT: Per Construct)

Macro-F1 is the headline metric: it averages F1 across all constructs without weighting by class frequency, the right choice for an imbalanced label space.

5.3 Calibration Metrics

A model that says "75% confident" should be right 75% of the time.

Why calibration enables formative feedback: Calibration scores power a tiered routing system: high confidence labels are auto-accepted, mid confidence are flagged for human review, low confidence are routed to expert review. The model knows when it is uncertain. That is the human-in-the-loop design.

A qualitative and quantitative breakdown of Type I (False Positive) and Type II (False Negative) errors across the communication constructs.

Named Entity Recognition detects Protected Health Information spans, classifies them by type, and masks them before any transcript enters the labeling pipeline.

Standard: i2b2/UTHealth 2014 de-identification corpus (HIPAA-compliant)

De-identification was built into the pipeline architecture from the start, not added as an afterthought. That reflects a governance-first design philosophy that shaped everything from output format to what claims the team would and would not make.

7.1 Types of Validity Addressed

7.2 Known Threats to Validity

7.3 Governance: Built In From the Start

ACGME explicitly states that Milestones are formative educational tools, not designed for high-stakes external decisions. Every output format, framing choice, and constraint was shaped by this:

What I Owned

As Innovation Analyst (Data & Research Focused), I owned the data pipeline and labeling architecture:

• Dataset architecture: Evaluating 4 transcript sources, structuring the corpus around real vs. simulated data, making the case for VHA 4C as the realism anchor.
• Label schema design: The 9-prefix labeling ontology with full ACGME competency alignment.
• LLM labeling strategy: The one-construct-at-a-time protocol, per-construct model selection (Qwen vs. Llama), exemplar conversation selection criteria.
• CSV schema architecture: 5-section schema covering keys, outcomes, features, competency alignment, and weak-label covariates.
• Simulated to real pivot documentation: Capturing the rationale with specific failure modes identified (distribution shift, model collapse).

The Decision That Shows Research Judgment

The simulated to real pivot:

"The original plan was to use synthetic ChatNote-style transcripts. I pushed for pivoting toward real clinical encounters (VHA 4C) as the primary data anchor, because if the model learns to detect communication behaviors from synthetic conversations that are cleaner and more explicit than real encounters, the scores it produces won't generalize to actual clinical settings. That is not a model problem. It is a measurement validity problem. The pivot was a research quality decision."

Acknowledged Gaps

Recommended Next Steps

1. Expand gold standard dataset (100+ double-coded conversations for formal ICC).
2. Reintegrate SHARE framework for shared decision-making constructs.
3. Conduct the feedback UX study: how physicians interpret probability-based formative feedback.
4. Run bias audit across demographics, cultural backgrounds, and data sources.