With medical schools growing emphasis on competency-based education, there is a need for standardized, objective skills workplace-based assessments (WPBA). Tools to assess procedural skills, like suturing (Morris 2012), and interviewing (Swick 2006, Piot 2020, Novais 2022, Siemerkus 2023; Pathak 2024) exist and are used for standardized-evaluations of these workplace-based skills. Documentation of patient encounter, including initial history and physical exams (H&Ps) is another workplace-based skill required across medical disciplines. While other disciplines have such tools currently in use (such as the P-HAPEE for Pediatrics), there is not a psychiatry specific assessment tool (King 2017).

In the Legacy and Phoenix Curriculum for the Clinical Clerkship in Psychiatry, one of the WPBA is the graded Psychiatry H&P (Psy-H&P). After becoming the Director of Medical Student Education in Psychiatry, I was tasked in grading Psy-H&Ps. Given there is no available standardized tool for assessment of Psy-H&Ps, I created an assessment tool (Psy-HAPEE) to standardize my grading. This tool has been used and updated since 2022 and the current iteration of the tool has been used since the start of the 2023-24 academic year. I am the process of completing a pilot study with clinical educators from the Department to evaluate Psy-H&Ps with and without the tool to provide feedback on the tool. With permission from the medical school, twelve de-identified Psy-H&Ps written by medical students from AY23-24 were selected for this pilot (Protocol IRB24-0734)

The goal of this study is to finalize the Psy-HAPEE assessment tool, establish reliability, and disseminate the final product to the academic psychiatry community. To do this, we will obtain additional feedback on the current assessment tool using focus groups of expert psychiatrists involved in medical student education across different institutions and would obtain reliability data from a national online survey of the assessment tool.

Current Psy-HAPEE assessment tool:

Psychiatric evaluations include gathering information about a patient's symptoms, history of illnesses, social circumstances, exam findings, formulation of the case, diagnosis and treatment planning. There are some generally shared standards for organization of this information in documentation of Psy-H&Ps and the Psy-HAPEE maps onto these areas. The Psy-HAPEE includes 11 components of differently weighted points. Ten of these graded areas content related and include the history of present illness (HPI), past psychiatric history (PPH), substance use history (SubH), family psychiatric history (FPH), social history (SH), mental status examination (MSE), assessment, differential diagnosis (DfDx), diagnosis (Dx), and plan. The final graded area incorporates an evaluation of the clarity, completeness, and coherence (CCC) of the documentation. The Psy-HAPEE is designed for a total of 100 possible points across all scored components. A PDF of the current tool is attached.

Evaluation of the Assessment Tool using Focus Groups:

The assessment tool will be applied to a standardized set of Psy-H&Ps We selected six of the twelve approved Psy-H&Ps (IRB24-0734) to use for this currently study, representing a range of total scores corresponding to medical student competency levels of Honors (H), High Pass (HP), and Pass (P). These Psy-H&Ps are labeled A through F to assist with study design and statistical analysis.

We will have two focus groups composed of academic psychiatrists involved in medical student education. Each group will six participants. The first focus group will be made up of medical student educators in psychiatry from local institutions through peer contact. The second focus group will have psychiatry educators from national institution selected via a listserv of Medical Student Education Directors in Psychiatry.

Prior to meeting, each focus group participant will be asked to apply the assessment tool to 3 Psy-H&Ps. They will be asked to give feedback on how important each criterion is, the clarity of the criteria and general feedback on the criteria. Each criterion has a pre-assigned points level; each participant will be asked to indicate if the number of points is too many, too few, or just right, and to state the exact number of points they think is appropriate for each criterion. We approximate that using the PSY-HAPEE to score the Psy-H&Ps will take two hours. The focus group members will then meet via zoom for two hours to review and comment on this the Psy-HAPEE until consensus is reached, using a modified Delphi process (Boulkedid 2011). This step provides content validity and consensus about the importance for each of the evaluation criteria.

Online Study of Interrater Reliability:

After finalization of the assessment tool, the next step will be to recruit up to 30 academic psychiatrists to participate in an online study to establish reliability of the final tool. Participants would include psychiatry educators. The online study link will be distributed through listserv to national organizations and to contacts from the focus group participants.

We will distribute the study link through emails. Online study participants would receive written instructions on use of the finalized assessment tool which would be applied to two of the twelve approved Psy-H&Ps (IRB24-0734). We approximate that this will take one hour to complete.

Participation in the study is voluntary and no more than two reminder emails will be sent. Subjects can participate in the study completely anonymously. We are asking participants to enter in their email addresses so they can receive compensation, but they do not have to do so if they do not wish to be paid. We will delete email addresses collected for payment purposes after payment has been disbursed. We are also asking subjects for contact information if they wish to be contacted for test-retest study. This information is optional and voluntary. This contact information will not be destroyed. All data are collected via RedCap survey methods set up at the University of Chicago. At the beginning of the survey, there is an implicit consent form that describes the study.

Test-Retest Study

A subset of 10 online study participants will re-apply the assessment tool to the same subset of Psy-H&Ps one to two month after their initial assessment. We approximate that this will take one hour to complete.