Building study plans for students to succeed

How we do user research to help medical students find the best way to study.

Studying for exams is difficult. But for medical students, studying for exams includes another dimension of difficulty. What’s not as obvious is that medical students spend a significant amount of time figuring out the best way to study. For example, what learning style works best for them, which resources they should use, how much time they should spend on going through questions, reviewing questions, deep diving into topics they are weak at, and so on — it’s a huge time management challenge.

Helping students study efficiently was one of the reasons AMBOSS was created, and why the Custom Study Plans (CSP) feature is one of the key ways the product helps students navigate the challenges of learning how to study for their upcoming exams.

In this article, I’ll share how our users helped shape our new CSP feature, and how research and product testing gave us insights into what our medical students need when preparing for their exams.

Jumping on board

When I joined AMBOSS and the team working on the new CSP feature, I was excited to support figuring out how to best help our students ace their exams. Prototypes were already being developed, internal interviews with our team of doctors (former medical students themselves!) were underway and we already had a solid idea of how students generally go about studying.

Our first challenge was to find out if our CSP prototype would meet our users’ needs and expectations, and what other issues we would have to tackle along the way.

As this was my first AMBOSS research project, and my first time working on an e-learning product, I wanted to understand what we already knew about our users’ mental models when creating their study plans:

• what did they do first,
• who did they talk to,
• how did they rationalize their decisions on what resources to use,
• at what point and in which order.

In order to find all this out, I first spoke to the UX designer, PM, stakeholders, and other colleagues who knew the student market.

One of my most insightful moments of these conversations was when I was speaking with our Editor-in-Chief for the student market about how users organize their studying.

To convey this in the best way possible, he drew me a detailed graph showcasing the timetables of M1 and M2 students, and the various points of when they start to study, how often, and how this changes when they start going from pre-dedicated to dedicated learning periods. The steep and complex learning curve of studying for all these exams really put the challenge of creating a study plan feature that would aid students into perspective.

After having these conversations with the team I felt I’d put in the necessary groundwork and was ready to talk to our users.

Meeting our users for the first time

In order to figure out if our CSP prototype would work for our test group of users, I first interviewed each user to find out more about the study plan they had for their most recent exam (USMLE Step 1 or Step 2 CK). I tried to schedule the interviews just after users had finished their exams so they wouldn’t be stressed and were able to reflect on their experience as it would still be fresh in their heads.

Specifically, I wanted to find out what worked well during their studying, what didn’t, and if not what would they do differently, and why. I also asked questions around what resources, learning methods, advice and techniques they found most helpful, and how they stayed motivated.

One of the biggest things we wanted to achieve with our CSP prototype was to keep our users on track for their upcoming exam, keep them focused, and encourage them to stay motivated to continue studying even if they had a bad day and didn’t reach their daily goals. So finding out if aspects of the CSP could do this was key.

The interviewees were so open and honest about the way they studied, particularly their study plan pitfalls, and how they realized the limits to how long and how much they could realistically study everyday. It also was a good refresher for them to reflect on when they created their study plan, how they figured out what worked and what didn’t, and what support they got from their peers, teachers and online resources.

It was also a-not-so-typical situation for this particular group of medical students, as COVID-19 completely disrupted their exams, and many of them had to reschedule their exam, often multiple times. They also had to rethink how to keep this knowledge fresh in their minds and stay prepared for the exam, while remaining calm and not feeling overwhelmed by the pandemic.

Testing our prototype

After the interview part was done, I tested the CSP prototype with them. When we test prototypes with users we encourage them to think aloud as they make their way through the different scenarios, for example by creating a CSP, answering and reviewing questions, exploring the different progress metrics, and so on.

The golden rules of user testing are to observe the user because the best insights come from what we see and hear. By prompting users to explain further what they’re thinking, we’re able to find out the ‘why’ behind the problems in our prototypes, and use these findings to come up with an alternative design.

Because the users had already reflected on their own study plans during the interview, they were able to draw better comparisons between their own experiences and the prototype. This was helpful from a UX standpoint as it helped validate the assumptions we had made when creating the design and add additional clarity as to why they commented on particular aspects of the prototype.

For example, one user mentioned that the gamification elements we’ve included in our question blocks encouraged and motivated them. This is exactly what we wanted the gamification elements in our CSP to do — keep users motivated so that when they didn’t finish everything that they’d planned for that day, they would come back, pick up from where they left off, and feel encouraged to keep going.

The interviews also helped confirm our assumptions about variation in our users’ daily routines. Our users tend to organize studying for their USMLE Step exams in similar ways — they aim to get through hundreds of questions per day, review, and then prioritize the ones they missed or were incorrect. But the order in which this is done varies with each user.

Some preferred to do new questions in the morning before they break for lunch, then review in the afternoon and catch up on missed or incorrect questions in the evening. Others preferred to first redo the questions they got wrong the day before, start a new set of questions in the afternoon, and then review for the rest of the day. Our users also tend to adjust their routines as time goes on as they start to discover what works well for retaining all the information.

The CSP design gives users the flexibility to do their daily questions at a time that suits them, and organize their work by noting the questions they got correct, incorrect, any questions they missed, and catch-up on questions at a later date.

Overall the results from the interviews and usability tests gave us direction on what worked well and what we could still optimize to meet the needs of our users. We validated a lot of our assumptions, identified minor fixes, and gained some additional considerations to take into account. Ultimately, we had a much better understanding of our users’ overall goals, daily tasks, and pain points.

Testing the beta version

Once we developed the CSP, the next step was to assess how it performed with our current users. We asked approximately 1000 users to try out the CSP over a period of at least 2 weeks, and surveyed them at different points in their journey.

The first survey was sent after users had completed the setup and started using the CSP, the second after they had used their CSP for 2 weeks, and a third and final survey was completed by users who had abandoned their CSP, or set it up but hadn’t started using it.

Once all the results from the surveys were in, it was clear that our users were invested in the CSP and we’d made quite an impression with our new design. The CSP setup worked well and our users gave us insightful feedback on how to improve the usability of the current version.

The questions in the surveys prompted a lot of our users to send detailed, well-thought out feedback that made it easy to understand why a certain aspect of the CSP didn’t work as well as it could have and what could be done to make it better.

My favorite feedback of all time was one user comparing their experience of studying for the USMLE Step 1 to an Indiana Jones crusade. As well as amusing, it helped the team put themselves in the user’s shoes and that these exams can feel like the be-all and end-all of their med student journey.

What’s next for Custom Study Plans?

I learnt a lot about our users and our product from the project. With the insights I gathered from the interviews, prototype tests and surveys I was able to provide the team with guidance on how we could tweak the CSP to meet our users’ needs even better.

Some of the insights included how to help users set up their ideal number of daily question blocks, what additional customizations we could add while keeping the cognitive load low, how and why studying questions in exam mode improves their test taking endurance, and what keeps users motivated during their studying.

These findings will be incorporated into the CSP 2.0, so stay tuned!