The PBL assessment
As you may have noticed, throughout a PBL activity there are numerous opportunities for each student to demonstrate cognitive and socio-emotional skills. Based on such demonstrations, the student's tutor and peers can assess the student's competencies as well as give feedback regarding their perceptions. The evaluation can be done in a weighted way, so that the student's evaluation is based on the grade given by the teacher, on the average of the grades given by colleagues, and also on the grade given by the student about himself, as shown below.
The self-assessment is also an important component of the PBL methodology. Because it is interesting for the students to compare how they sees themselves individually with how they are seen by their colleagues and by the tutor. And this comparison cannot necessarily be based on thoughtfully calculated numbers, as in the figure above. They can also be accompanied by comments that, if done appropriately, can give a valuable information to students on how to improve their skills, especially the socio-emotional ones.
The concept of evaluation in PBL is very broad. More than classifying students according to the grades they obtain, the PBL assessment aims to identify aspects that need to be improved. Each medical program uses its own evaluation policy, determines the competencies that need to be evaluated, determines the grade weighting to be used. Moreover, some programs include the PBL process itself and its elements: the educational resources, the tutor, the deadlines, the content of the case book, etc.
What is assessed
PBL means what stands for its acronym, implies learning by solving problems in the real world or similar to those that arise in the real world. It means learning by doing, dealing with a practical approach based on science, in a controlled and safe way. In this sense, two main classes of skills to be developed among medical students were identified:
Cognitive skills (Hard skills)
Health problems that need to be treated by a doctor are often highly complex, depending on many factors that influence the patient’s health: diet, habits, social and family relationships, psychological issues, etc. Thus, the real world works in an holistic way, which implies that knowledge from different disciplines of medicine needs to be used for correct diagnosis and treatment.
In this way, the students need to have their cognitive learning evaluated insofar as they demonstrates, to the tutor and to their colleagues, their ability to overcome the difficulties in arriving at an adequate diagnosis and prognosis, justifying all their arguments.
Socio-emotional skills (Soft skills)
The professional practice of a doctor in a hospital environment can hardly occur without him being integrated into a team (nurses, physiotherapists, psychologists, social workers, assistants in general, etc.). This implies developing skills in self-knowledge, communication, collaboration, conflict negotiation, leadership, among others. If the physician has such developed socio-emotional skills, his dealings with the work team tend to have better results, and in a hospital environment this can save lives.
In a curricular PBL context, medical students intensely exercise the analysis and problem solving, guided by a protocol that promotes team collaboration. The result is newly graduated physicians who are already well prepared to face the hospital work routine, in a team work scenario. Furthermore, they are able to search and acquire new knowledge that they may eventually need in their professional practice. This is why the socio-emotional skills need to be assessed.
How it is assessed
Considering only aspects related to student learning, the figure below shows a possibility of composing notes to consolidate the evaluation of the PBL activity. In this case, 70% of the grade may be linked to the assessment of cognitive learning, while 30% of the activity grade may be related to the assessment of the socio-emotional aspects listed for the activity.
Different strategies
The need to calculate grade weighting does not stop at the division shown above. After all, both cognitive and socio-emotional learning can be divided into distinct competencies, as shown in the figure below.
Note that cognitive learning is assessed on two criteria, while social-emotional learning is assessed on three criteria. I divided each of the evaluations into different percentages to explain how much work this can be for the teacher to deal with in the implementation of a PBL activity using rubrics.
Once you understand how the evaluation composition of a PBL activity occurs, it is important to realize two difficulties intrinsic to this process in which a student is evaluated by his peers:
- Standardization of the way in which the competences involved in the assessment are evaluated, since different colleagues can evaluate the same student differently with regard to the same competence.
- Processing of data generated by peer review, which grows geometrically in relation to the number of reviews and peer reviewers.
It is no wonder that many PBL implementations simplify the assessment of PBL activities due to the high workload that peer assessment and based on multiple competencies ends up generating. I visited a medical program where social-emotional skills assessment was done once every 10 PBL activities, and each student was assessed by only one peer. This compromises some of the fundamentals of the PBL methodology: frequent feedback (in each activity), and the student being able to have feedback/evaluation from all colleagues.
IMPORTANT
Due to the operational difficulty to implement the assessment, mentioned above, it is important for you to understand the concept of rubric, presented below. Because using rubrics is a way to improve the standardization of the assessment carried out by multiple students, and once the OpenPBL platform is used, the collection and processing of data generated by such an assessment is automated.
Rubrics
The PBL assessment can be implemented through different tools. The student can only receive feedback from the tutor and colleagues, with comments about their performance. However, in order to register performance, it is necessary to collect evaluation data. This collection may include a simple numerical assessment, a simple rubric, or even a complete rubric. All of these are addressed below.
Numeric scale
The most commonly numeric scales are 0-10 and 0-100, but it can be any scale (1-3, 0-5, etc). The limitation of using a simple scale is because the student receives a single note on what was observed in terms of global learning. The evaluation is done in a unidimensional way, Therefore, there is no way to assess more than one aspect of the learning process, and then it is not possible to separate the assessment of cognitive learning from socio-emotional learning.
Of course the assessment is not normally done with plaques that are held up by each assessor, as shown in the figure above, used for illustration only. The assessments are collected through web forms (Google Forms or similar), or even on paper, to feed a spreadsheet that processes the grades. There are also platforms, such as OpenPBL, which collect and process the scores given by the evaluators.
The simple scale assessment, therefore, is presented here for the record only. After all, to enable the evaluation of different criteria, it is necessary to use something more sophisticated: rubric.
Simple rubric
The simple rubric is defined as having different criteria to be evaluated. In this way, it is possible to evaluate criteria associated with cognitive learning and others associated with socio-emotional learning. However, each criterion is evaluated using a simple numerical scale, such as the example shown below.
We have a simple rubric implemented on paper, where each criterion is specified on one line. The first two (analysis capacity and knowledge acquisition) are related to cognitive skills, as they demonstrate the perception of how much the student has learned. The last three lines (communication, leadership and conflict management) refer to socio-emotional skills. For each line or competence, the evaluation is based on a star scale, which goes from 1 to 3. Thus, we have 5 evaluation criteria, each with three levels of score.
Even in the case of a form with a simple numeric scale, note that the form above needs to be completed by each student with a rating above the others (one form for each colleague). Afterwards, the forms are given to the tutor, who needs to enter the data in a spreadsheet, which converts the number of stars into a score, generating a resulting final grade.
Below you can see an alternative form for the above example, grading the competencies in a 0-100 scale. In both examples, colecting data represents a workload for the tutor, except if an app / platform is used to do this task.
Note that even though using a scale from 0 to 100 to rate each criterion included in the assessment, the scale used for each criterion is the same. Therefore, the rubric is defined here as simple. Notwithstanding, it is enough to the majority of the PBL assessment scenarios.
Complete rubric
Note that in the example shown above, it is a scale from 0 to 100 to evaluate each criterion included in the evaluation. There is no guidance on how each criterion should be evaluated. A student can give a grade of 95 to evaluate a colleague in a certain criterion, while another evaluates the same colleague, in the same criterion, with a grade of 75. There is a particularization of the gradation applied in the evaluation of each criterion within the numerical scale used.
To reduce this variation in the assessment of each criterion, it is possible to specify different levels, with a small text indicating the grade levels for each criterion, constituting a rubric that I define here as complete. The figure below illustrates an example of such a rubric type.
Note that competencies, whether cognitive or socio-emotional, are spread across the rows of the table. The first two lines correspond to cognitive skills (blue lines), accounting for 70% of the activity assessment (30% + 40%). Socio-emotional skills are in the last two lines (in light red), weighing 30% of the activity assessment (20% + 10%).
Each line (criterion) has different levels, with a small sentence to help different evaluators to grade their evaluation following the same logic, specified by the rubric author. Each square has a different score for each criterion, so choosing a squarte for a criterion implies the contribution of a number of points according to the level chosen for that criterion.
All the evaluators have to do is just choose the level that is closest to what is perceived by the evaluator according to the corresponding text written inside the blue and pink squares. Thus, it is easier to standardize the evaluation carried out by different evaluators, on different criteria, considering subjective aspects of evaluation.
