How might we help students be efficient by giving them the fastest and easiest path to achieving test outcomes?
Students were looking for personalized recommendations where Kaplan will identify their weaknesses and based on that would recommend what they need to do in order to improve their weakest areas. We also heard from PPI & MCAT students that it would be interested to compare their previous practice exams/tests with their recent one and see if Kaplan could predict future exam scores based on how they scored previously.
We want to make sure that we fully understand our end goal in terms of performance based recommendations, so that we can make scoping and release decisions that move us in the direction of that goal. We also want to make sure that our decisions are based on Learning Experience best practices and input and that what we build will meet the needs of the majority of our business use cases.
Research Goals
Understand how students and internal business stakeholders define performance recommendations & what they expect it to be based on.
Understand where students & internal business stakeholders would want to see the recommendations.
Understand when students would use recommended resources vs non-recommended.
Understand if business stakeholders would want to enable the recommendations in all product enrollments or just selected.
Participants
Students (n = 10)
MCAT: 5
CFA: 2
PPI: 3
Age: 18-55
Country: USA
Education: High-school graduate (minimum)
Methodology
Moderated one-on-one semi-structured interviews
Ideation workshops with internal stakeholders
Internal Stakeholders (n = 18)
LX
LSAT
Bar
MCAT
CFA
Things we discovered
After the research with students and internal stakeholders, I identified the following insights.
Recommendation based on performance
To students recommendations should be made based on their performance (weak areas). Internal business stakeholders have also identified that recommendations should be based on students' performance.
Optimal vs priority recommendations
Some students think of recommendations as optional resources whereas others think of them as prioritized resources.
Internal business stakeholders want the recommendations to be optional for students and not the priority allowing to turn on & off them as needed.
Negative messages notifications
Students don't like messaging saying that they're behind & they want to know why they're being prompted to push test date back.
Different types of recommended resources
Students & internal stakeholders want to see variety of recommended resources suggested e.g. live classes + workshops, videos, flashcards, Qbank, readings.
Recommendations on the calendar
Majority of students want to see recommendations on their Calendar, some want to see them in a form of notifications or a banner. Some business stakeholders expect the recommendations to show on the Calendar but highlighted in a different color to distinguish recommended materials from non-recommended. Other stakeholders expect to see recommendations on a separate page.
Study Timeline
Students don't expect recommendations to impact their study timeline. In fact, they would prefer if it didn't.
Reasoning behind recommendations
Some students want to see a short explanation of why certain resources are recommended for them, others don't expect seeing reasoning behind recommendations because they trust Kaplan.
Recommendations for full set of study products
Business stakeholders expect the recommendations to be enabled to all the products which provide full set of study products instead of just the Qbank as an example with the ability to turn on/off.
Dynamic Calendar
Students would be open to recommendations assigned to them by default. In our prior research with LSAT & GRE students we also found out that they preferred dynamic study plan. However, during the interviews students did mentioned that they want to see recommendations on their Calendar but distinguish them by highlighting them or in a form of notification or a banner.
On the other hand, business stakeholders don’t want to prioritize recommendations but rather keep them optional.
UR Recommendations
What recommendations should be based on?
Recommendations should be based on student’s performance on the content quizzes, exams, tests, homework quizzes, Qbank, outcome best practices and time spent answering the questions.
What type of recommendations to suggest?
We should suggest variety of recommended resources to students including live classes and workshops. If students are early on in their study program, they should spend more time with all resources where they have knowledge deficiencies. If they are later on in their course, recommended resources would be more targeted intervention e.g. specific readings or videos.
Recommendations as optional
Business stakeholders want the recommendations to be option for students and not the priority. Some students also think of recommendations as option and would consider reviewing recommended materials if they’ll have time. Stakeholders also want the ability to turn on and off the recommended feature.
Hence, it is recommended to have the ability to toggle recommendations on and off.
Where recommendations should be located?
Some Business stakeholders and the majority of students want the recommendations to be placed on the Calendar but show them in different color to distinguish recommended vs non-recommended resources. Students and business stakeholders also mentioned the idea of having recommendation in a form of notifications or as a banner.
Before jumping to a solution of where we would place them, it is recommended to discuss this as a team internally via an ideation workshop to identify the feasibility and limitation of these ideas both from the tech and design sides.
Recommendations only for full set of study products
Business stakeholders expect the recommendations to be enabled to all the products which provide full set of study products instead of just the Qbank as an example.
Hence, it is recommended to enable recommendations only for full set of study products to give that extra value to students.