Wednesday, 4 June 2008

Training a Bunch of Monkeys

Because a) I've got nothing better to do, b) I quite enjoy it and c) the £40 will pay for a takeaway, today I volunteered yet again as a patient for the 3rd year OSCE exam. It is the 5th or sixth time I've done so but each time I learn something new about the whole process. Today I had aortic regurgitation and 39 candidates did their best to impress the examiner as I laid there watching the ducks out of the window.

Now I should say that I think OSCEs are a very important assessment of a medical student's abilities. However, they aren't, certainly at my institution assessed effectively in my opinion. I recognise the difficulties in running a fair exam but when you're trying to do so simultaneously at 4 sites, with different equipment and different examiners in the morning and afternoon but supposedly the same stations it must be impossible! You might think that it all averages out in the end, and perhaps it does eventually but there are huge inconsistencies. I would have thought some methods are employed in an effort to standardise the marks but it would be almost impossible to do this for every conceivable variable. Blah, blah, blah, I could moan about this all day long.

Lets take today for example, admittedly I only saw a snap-shot of what went on, just 1 out of 13 or 14 stations. We started in the morning with one examiner, who's marking was rather erratic. There was one student who was a country mile above the rest who scored average to below average whereas others who were average at best were scoring higher. It doesn't help that the mark scheme leaves things very open to interpretation by the examiner but what can you do? An open mark scheme is too open to subjectivity but a more rigid mark scheme makes it more difficult for students to score consistently. Overall that examiner was fairly generous. In the afternoon, same station, different examiner. This time, a much more consistent marker but at the same time, much harsher with the marks (perhaps rightly so). Next we should consider the students, frankly, overall they weren't that great but I'll get to that in a minute. For no particular reason that I could see there was a significant variation between the morning lot and afternoon lot. The morning lot were on the whole, ok (remember they also had a generous marker) whereas the afternoon lot were, apart from one, not as good (and also had a harsher marker). I might not have explained that too well but what I'm getting at is that there was more than one variable affecting each group skewing the distribution of marks further between morning and afternoon.

As for the students themselves, let me explain what I thought was wrong. This might sound a little harsh and hypocritical (I was exactly the same when I was a 3rd year). Neither is it really the students fault. Every student was able to make a decent effort at completing a cardiovascular examination. But, you could teach a monkey to do a textbook cardiovascular examination. Very few students looked like they were actually looking to elicit signs. Perhaps the main issue here is that it is such a fake situation and the students are expecting a normal volunteer to examine therefore there won't be any signs so it doesn't matter. I spoke to the examiner about this and he agreed, he had previously examined a thyroid station where students had all done a textbook examination but 5cm above where the thyroid actually is. They are only 3rd years, perhaps it is unfair to expect them to elicit signs at such an early stage. You can clearly tell the students who've examined real patients and who are actually trying to elicit the signs from those who are just going through the motions, the former group generally score higher, I believe rightly so.

During finals for instance we had 6 stations of real patients, these were not only the most difficult but were the most interesting and dare I say it, enjoyable stations from my point of view. Why not have a few real patients for earlier years? Yes, it would take more organisation but patients are usually quite willing to spend a morning being examined. We're not trying to train monkeys, we're trying to train competent, dynamic doctors who're able to think on their feet so why aren't we assessing this properly?

My experience of OSCEs from both sides has led me to the following conclusion:
Performance = 30% luck in each and every station + 30% examiner subjectivity + 30% confidence (or false confidence) + 10% knowledge

Clearly, luck makes up a rather large proportion and this is what needs to be eradicated if possible. How we do this I'm not quite sure. Perhaps having more stations 30, 40 (as some other medical schools do I believe) to average things out, or improving the consistency of marking by reducing the subjectivity.

I love to get my hands on a chunk of results data to run some of my own statistical analysis on. Although the chances of that happening are about the same as winning the lottery.... 4 times. If I asked for it, I'd just be ignored.... again.

One other thing from today; what is with all the all black stethoscopes? At least half the students had them and yes they might look quite cool but they sure as hell don't make you any better at using them!

If you've not already done so check out my latest, and probably final post here:


Michael Anderson said...

Don't you think you're being a little harsh on the 3rd years TLM?

The question I'd ask is what are the students being tested on? Is it their ability to conduct a competent clinical examination or is it their ability to elicit clinical signs?

The two are obviously closely linked but, as you point out, are not the same thing.

Doctors need to be able to pick up on and interpret signs, but if the students are being asked to examine someone with no signs, then it's a little harsh to mark them down for "not looking like they could find them" don't you think?

If you really want to test their ability to elicit signs, then they should have a patient with signs to elicit, I reckon. It gives the student a fighting chance.

the little medic said...

I did say that it wasn't really the students fault as they are doing what they think they're expected to do. It is their first clinical year so they're doing well to just have a system.

My point was really what you picked up on. I think they should have patients with real signs. I would never devote all, or even the majority of the marks to getting the right diagnosis at such an early stage but I think it would be really useful to start finding things early.

They have clearly shown they can learn something and recite it by just doing the examination. I don't think it would take much, if anything extra to apply that to real situations, but as you say, real patients would be required for that.

skinnyminny8 said...

I agree with your point, but I also think its not fair to give a student 5 minutes only... well, 3 1/2 when it comes down to it to examine a patient with real pathology. Make each station longer, and then, by all means get patients with real pathology.

I also think the first thing we must learn as medical students is what to do, it takes more time to learn to pick up the signs. If you cannot do the examination process, you cannot pick up the signs! So I understand why just examining normal people is important early on in training ( as well as learning what is normal!!)

I personally think Osces are very stupid, its a whole act and not particularly useful or fair.

My 3rd year exam had patients with pathology and normal people so my med school does what you suggest! But then it is even less fair. For instance each of my medical sites had different neurology patients with tremor. Eg. Parkinsons, Supranuclear Palsy et.c

Congrats on finishing your exams, that will be me next year. I'm already worrying!

The Manchester Medic said...

I would probably have to agree with a lot of what you are saying. While I was doing that station, I was mentally working through a protocol that was deeply ingrained into my mind - giving very little thought to what I was actually doing. And yes, this is wrong. But it has served me well through 4 years of OSCEs so far and I am yet to fail an examination station by doing so. So, unless they force us to change the way that we tackle such stations, I'm not changing my approach.

To be fair, they do try and discourage us from reeling off a long list of signs etc. by focusing the examination on a particular condition. But it's not really enough. I have been told by more than one examiner that, if you don't know how to focus the examination, then try and cover everything and you won't fail. But if, like you said, they included some real patients for us to examine then that would make the whole situation a lot less artificial. As I left the examination room, I thought to myself about how hard finals were going to be,as we would be seeing real patients for the first time in an examination. It would definitely make sense to start introducing proper pathology into the OSCEs at this stage. Maybe not anything too subtle but a patient with a loud murmur, for example. This would force students to actually perform the examination rather than recite the steps whilst pretending to carry them out.

I'm not really sure why I am in favour of making examinations harder for myself. Probably because I realise that I need to make a lot of progress if I am going to be a competent doctor in just over two years time.

As for MR station in particular, I imagine that I was one of the students who was "average at best". I definitely didn't perform as well as I could have done. This was partly due to nerves as it was an early station for me and I hadn't yet "settled" into the OSCE routine. I would be very interested in hearing how the station was marked, although I imagine that you wouldn't be allowed to divulge that information to a student who took the exam?

And yes, I do have a black stethoscope. And whilst they may not function any better than any other colours, they most definitely are the coolest ones! (you must have been really bored by the way, to be noticing the colours of people's stethoscopes!)

Mr-Mobius said...

If you're going to do it, do it looking good. I've a black stethoscope and so do quite a few of the 1ts years at my uni. I guess a stethoscope has a slight status symbol (though people in halls I lived in during this year almost pissed themselves laughing when I appeared in with white coat and stethoscope before the OSCEs for 1st year).

OSCEs are quite fake and it's hard to look sincere when you have 5 minutes which feels too short to do the stuff, so you just end up going through the motions for the sake of the motions.

Only found your blog a few days ago but nice blogging. It's quite relevant to me and along with the other medical blogs I've found since finishing 1st year, they've all been great insights into further along medical school. My medical school uses mostly didactic teaching with pbl tutorials every week. I guess PBL works if you prepare before but lectures are so less stressful and better for exams as know the topics are covered in sufficient detail.

Dragonfly said...

We are told about how theatrical we have to be during OSCEs (or were in 3rd year anyway). It really annoyed me, because it all seemed so fake. Now I just do it the way I would on the ward and ignore that crap. I have yet to fail anything (though this may be famous last words).

the little medic said...

skinnyminny - That is a good point. 5 minutes, or more likely 3 and bit is not long enough to do something properly. This is another factor which encourages students to just go through the motions.

As you say, OSCEs can be a bit stupid due to their fakeness. Perhaps a move towards F1 like placement based assessements like DOPS and CEX would take away some of the fakeness. I think my school is going to do some of that from next year.

manchester medic - Why should you change unless you're forced to. Afterall, as you say, that is what you've been taught to do and told by examiners that you'll pass if you do it. You will, but at least in that station, in the afternoon, you had to look like you knew what you were doing to score well.

You might think real patients would be harder but in fact, they're actually easier in my opinion as it is somewhat less fake and you can do things properly.

As for the stethoscopes, I did get rather bored but so many people had black ones I couldn't fail to notice and I'd never really seen them before.

mr mobius - If you went to our clinical OSCEs these days in a white coat you'd be asking for trouble (unless it has short sleeves) but I do like all the pockets they offer.

I agree that time may well be an important factor which makes the stations even faker than they are.

Glad you enjoy the blog, keep reading.

dragonfly - good for you, I think that is definitely the best way to go.

Mr-Mobius said...

My uni is still insisting on wearing white coats to OSCEs and hospitals unless the doctor at the hospital says to not, and to everything have shirt and tie even though there are now regulations (or at least advice) that says that should be short sleeved and no ties. I'm sure they'll change this as the years go on but maybe the legislation hasn't quite came in to N.Ireland or NIBMA hasn't mentioned any need yet. Honestly I've no idea.

Fiona said...

To be honest, your experience sounds just like every interview. In everything subjectivity plays a huge role. A friend recently had an interview for a post where there were 500 interviewees over a month. There were at least two different interview panels. As you state, no matter how closely scrutinised the process would be, there is no way it can be fair.
I hate saying it as I love your blog, but I have come to the very sad conclusion over the last few years that what you describe in yout hospital/university is normal and repeated wholesale everywhere else. In which case, it sometimes best to leave the gripes at work and just live your life, remembering that you work to live and not live to work. (Although don't quit the blog!)

Calavera said...

Hmm, I tend to agree with you there. On wards I tried to examine as many patients as I could. I was also lucky in that the house officers I was teamed with were amazing and found patients with signs that I could elicit, so after a while I got really good at eliciting reflexes, listening to heart sounds and suggesting the underlying pathology etc.

However I do remember that a lot of students who hadn't examined very many patients were still fairly good at going through the motions and overall got the same marks.

the OSCEs at my medschool just basically have a ticklist so you get the mark for attempting something, regardless if it's the correct way or not. I suppose that's where the examiner's discretion comes in.

Noel said...

Having actual patients is only part of the answer, being able to elicit findings has to matter too. Our OSCE's use actual patients but they're graded the same way yours are. I got to see one of the marking sheets that the examiners carry and there are no points for eliciting a sign or making the diagnosis. Instead you can get 2 points for auscultating in the correct positions or mentioning that you would like to do a blood pressure.

Also, at the end of our examination they usually ask us what our differentials are but that's just a springboard to discussing management. If you can't give them differentials they just tell you the diagnosis anyway so they can move on to management. I actually proved that it wasn't in our benefit to actually examine the patients: Last year (my 4th) during my elective my consultant asked me if I wanted to do the 3rd year OSCE for the "educational benefit." I said yes but instead of doing a full exam I just stopped at the first sign I found i.e. splinter haemorrhages or the obviously enlarged thyroid. I presented my differentials, skipped to the discussion and somehow I passed. I got told off by the consultant later but the examiners kept telling me how well read I was (they didn't know I wasn't 3rd year) and only one of them noticed that I had taken their OSCE and made it an Oral exam.

PS: This only goes for our internal exams though. Final MBBS exams have external examiners and they tend to mark differently.

Noel said...

BTW, can I get an invitation to your other blog? Thought I might check it out before you close it.

Name: Noel Cowell

Ms Medic said...

I'll get back to on this after long cases on Tuesday, where we have real patients, but...

I think for long cases it's an advantage to have been on the wards and seen real patients, so you can do a 10 minute CVS examination, (hope to) find the murmur, and interact with the patient like you really would. For short cases though I think it's a disadvantage. In real life, you tailor your approach, your questioning, everything, to the individual and the circumstance. In short OSCEs you have to tick all the boxes, so most people just learn the relevant mark sheets and adapt a few words to the situation. Not really identifying the potentially great clinicians is it?

Also, funny I read this after my own mini-OSCE rant...

anna gregory said...

I have tried to open the blog link you mentioned, but it says that I need to be invited (Can I apply or is it VIP stylee!?)

Please don't let this be your final post - I woudl love to see how you get on in F1 year


Anna said...

My medical school has real patients in all OSCEs - during my final year GI station, my patient was bright yellow (ah ha!) but was too sick for me to examine properly... However, I'm of the opinion pretend OSCEs on normal volunteers are a waste of time and effort. The same value as practicing on your friends - useful for getting the sequence right, but crap for actually learning anything about medicine. Some of the most interesting patients I've ever seen have been in my OSCEs.

And how come no-one's picked up on the most shocking point in this post - £40 on a single takeaway?!

Medical Aesthetics Jobs said...

It is inevitable for things of this sort to happen, especially in today's time. Whether you have a large company or not, I think product managers should really take a look into their Product Opportunity Gap (POG) and really see if they can make a difference or not. Many calculations have to be taken into consideration when looking at the company at an All-Around perspective. Judgment's cannot necessarily be made upon feelings rather logic. Being a business man and reading this, I sincerely think a better alternative could be formed. But, if not, then instead of worrying about losing money, unravel some sort of 'secret plan' to your organization to help boost confidence and productivity rate. That's my personal opinion.

Anonymous said...

It interesting on how you touched on the difference between the groups of students though the day. I'm currently revising for my Year 3 OSCEs, and I've been allocated to go in the first group on ther first day. The stations run for 4 days, so by the 4th day its realistic to think that those students will know every station thats coming up. The school argues that according to their data there is no disadvantage sitting it first. However I cannot help but think that the 1st quater of the year who are allocated on the first day are at a serious disadvantage. Passing the year depends on how many stations are failed, there is no set mark for a pass, it is dependent on the mean mark students score, so those who are likely to score less will be on the first day, therefore likely to fail more stations. With medical school becoming increasingly competitive, and FY1 applications too (however this might change), this needs to change. Test the whole year in one day, yes this might not be so cost effective, but its fair. Even easier, release the data they so call analyse...

Anonymous said...

do med students get this practical exam every year or just in the 3rd year? do med students hoping to pursue a career in surgery get one of these and if so, does it come after their 2 years of dissecting a cadaver? really grateful for any replies. thanks.

Anonymous said...

This has been on my mind for some time..... It does lead to other issues...