Since studying for USMLE Step 1 with D-2 for about one year we have used a smorgasbord of resources: Sketchy, videos by experienced physicians, videos by other near-age doctors, residents, and students, First Aid, the bible of USMLE study books, Anki flashcards, other flashcards, and Google searches (me), a once-a-week tutor for Ana, the Uworld question bank. Those are what I can recall.
Although Ana's tutor is superb and has given us countless time-management and macro tips she is not a tutor as the term is commonly understood, a personalized instructor. She is more a weekly consultnt.
Although Uworld is the sine qua non study tool )they have the actual USMLE questions), and everybody in Dr. Ana's study support group uses it, everybody else I know of uses it with other tools in their tool box. USMLE takes a village, and a tool shed.
Although fun, the animated Sketchy's were not helpful (so I concluded and eventually convinced Dr. Ana.)
Although contemporary (I LOVED when one whiz kid said "fuck"), the vids by near-age teachers were distracting from the material for the on-camera facial expressions and mannerisms. (We watched one by an Indian teacher and to get to the point Ana put the vid on 1.5 speed. The Indian accent on words ending in "ine", high-pitched by the playback speed, caused us first to smile, then chuckle, then convulse with laughter till our stomachs ached and our eyes teared.).
Although learned, the videos by the experienced physicians who was our favorite (no face) were dense and about broke my (and eventually) Ana's brain. A 20-minute video became much longer as we could only absorb about three minutes before we had to hit pause to take notes. (I could actually feel the brain matter leaking from my ears.)
Although the gold standard of USMLE study books, First Aid, while comprehensive, is condensed into blurbs and abbreviations and signs and totems, anagrams, symbols, formulas; it is difficult to navigate and time-consuming, often to little or no end. As a layman I got frustrated eventually with First Aid and started Googling the key terms in a question.
Although Googling (AI) was a major advance for us, giving us both answer and explanation so that we could actually LEARN, there were a few drawbacks. The first time I Googled as one would who the coach of Oklahoma City was, I got the answer with explanation (ergo, we could learn) before Ana did in First Aid. Her shoulders slumped. I still use it though, with her agreement. On a couple of occasions Google AI failed me spectacularly, giving precisely the WRONG answer. (That is my personal nightmare. I don't know anything about medicine and I'm sitting with my already-doctor daughter-in-law trying to help her and I steer her 180 degrees wrong. I was crestfallen. Those nightmares were rare, thankfully.) But there were other problems with Google AI. Sometimes I couldn't get a direct answer. Sometimes the answer was conditional gobbledygook. I would refine the search and make it more specific, using the exact phraseology and syntax as in the actual question. I was startled the first time that I got the actual USMLE question with the answer box already checked off, without the explanation. That's a cheat sheet, not a study method. You don't learn and then are able to answer the question correctly. You just get the correct answer. The more specific one gets in searching Google, the greater the likelihood that you're going to find the exact question and answer.
Although Anki flashcards are essential to setting the answer in your mind, they are FLASHcards. There is no to little context, little time (12 secs) to even think. The flashcards are meant to be memorized. Not knowing anything I would sometimes get the answer right for having memorized the sentence structure and where the blank was. That's okay IF you have already gone to medical school and know the context. (Today, I got three flashcards in a row correct instantaneously and Ana gave me her highest accolade each time: "Look at YOU!")
Which brings me to like the specific topic of this post. A couple of weeks ago Ana had a get-together with some other USMLE students. Although a social gathering, the subject (as always with those in the field) of medicine was prominent in the discussions. One of D-2's friends, who has already passed Step One, said that she had used ChatGPT extensively and successfully to answer the USMLE study questions. Since then we have used a triple-check on our answers, First Aid, my own Google searches, and ChatGPT. After a very productive day today, I Googled "chatgpt usmle". I got a National Institutes of Health evaluation from Feb. 9, 2023. Now get this: ChatGPT had just come out:
In the past three weeks, a new AI model called ChatGPT garnered significant attention due to its ability to perform a diverse array of natural language tasks [9].
Get this: NIH was not evaluating ChatGPT as a study aid for USMLE, they were testing how smart ChatGPT was by HAVING IT TAKE THE ACTUAL FUCKING USMLE.
...we evaluated the performance of ChatGPT, a language-based AI, on the United States Medical Licensing Exam (USMLE).
Like the AI chess programs, let's see if they can beat a Grand Master.
In this study, we evaluate the performance of ChatGPT, a non-domain specific LLM, on its ability to perform clinical reasoning by testing its performance on questions from the United States Medical Licensing Examination (USMLE). The USMLE is a high-stakes [You're telling me.], comprehensive three-step standardized testing program covering all topics in physicians’ fund of knowledge, spanning basic science, clinical reasoning, medical management, and bioethics. The difficulty and complexity of questions [ditto] is highly standardized and regulated, making it an ideal input substrate for AI testing. The examination is well-established, showing remarkably stable raw scores and psychometric properties over the previous ten years [10]. The Step 1 exam is typically taken by medical students who have completed two years...
USMLE questions are textually and conceptually dense [🖕]; text vignettes contain multimodal clinical data (i.e., history, physical examination, laboratory values, and study results) often used to generate ambiguous scenarios [🖕🖕]with closely-related differential diagnoses.[🖕🖕🖕] Due to its linguistic and conceptual richness, we reasoned that the USMLE would serve as an excellent challenge for ChatGPT.
Could it? Could THREE-WEEK OLD ChatGPT actually PASS USMLE, one of the hardest tests on planet Earth?
We found that ChatGPT performed at or near the passing threshold of 60% accuracy.
:o
It was the first AI program to ever accomplish this:
Being the first to achieve this benchmark, this marks a notable milestone in AI maturation.
It was mature enough at three-weeks. Now it's a two-year toddler.
Impressively, ChatGPT was able to achieve this result without specialized input from human trainers. Furthermore, ChatGPT displayed comprehensible reasoning [not always the case with other tools] and valid clinical insights, lending increased confidence to trust and explainability. Our study suggests that large language models such as ChatGPT may potentially assist human learners in a medical education setting...
ChatGPT performed at or near the passing threshold for all three exams without any specialized training or reinforcement. Additionally, ChatGPT demonstrated a high level of concordance and insight in its explanations. These results suggest that large language models may have the potential to assist with medical education...
We hit the mother lode.
NIH, of course, cannot endorse. I, of course, CAN. As a retired criminal law attorney and medical ignoramus, I endorse ChatGPT as a premier USMLE study tool for your toolbox.
