The University of Central Florida invention is a method of pretraining a deep learning transformer model with medical data that can be used to evaluate and automatically score Objective Structured Clinical Examinations (OSCEs). OSCE is a clinical skills assessment tool that is used worldwide for evaluating and teaching health care professionals.

Current transformer models, such as Bidirectional Encoder Representations from Transformers (BERT), are limited to a total of 30,000 common English words found in textual sources. This approach causes the unwanted shattering of key medical terms that define a symptom (for example, hyperthyroidism). Moreover, the text source quality is weak, limited to non-peer-reviewed articles and public medical websites that are not accepted as a source of education for medical students. Nor do they contain domain knowledge about the OSCE.

In contrast, the UCF model’s pretraining objective and computation on key medical textbooks allow the performance to be more precise. The pretraining objective also enables the model to better encode the information in its embedding space, thus increasing interpretability as well as performance when scoring OSCEs.

Technical Details

The UCF invention comprises a method for training a deep learning language model to recognize complex medical terminology for evaluating and scoring examinations, such as the OSCE. The method includes steps of text mining a plurality of medical textbooks with a script prioritizing headings and paragraphs. It also filters or de-prioritizes ancillary content such as figures, tables, captions, footnotes, title, authorship, affiliations, dates, abstracts, references and appendices. Lastly, the method fine-tunes the model with multiple graded OSCE examination reports.

One example application of the invention includes receiving a medical examination dataset, executing a data processing procedure, and providing an automatic short answer grading mechanism. It also includes determining a final decision of the grade and reporting the results' uncertainty. Embodiments may include steps of pre-training with flashcards intentionally populated with incorrect headers to thus increase the sample size. Another embodiment may include applying perturbation to the flashcards, causing the model to perform more computations to adjust a transformer's weights and reduce error.

Adaptable to various platforms, the invention may be implemented in hardware, firmware, software, or any combination. It may also be implemented as instructions stored on a machine-readable medium that is read and executed by one or more processors.

Partnering Opportunity

The research team is seeking partners for licensing and/or research collaboration.