Buddycheck | Staff guide

1. This BuddyCheck evaluation video guide will help you get started.
2. Go to Canvas
3. Click on a module
4. Add group configurations to your Canvas course.  
5. Click on the BuddyCheck menu item in the Canvas course menu. If BuddyCheck is not visible in the Canvas course menu in your Canvas course, go to settings and the navigation tab and add it to the active menu list and save.  
6. Click 'Create evaluation'. You will perhaps see a verification screen asking you to confirm connecting your account temporarily to BuddyCheck. This way, BuddyCheck can pull the group and student data from your course on behalf of you. 
7. This will load the BuddyCheck page. From here you can: Create, modify, monitor or delete evaluations  
8. When creating an evaluation you can:  
   • Choose a name for the evaluation  
   • Choose the Start and end date of the evaluation  
   • Choose (on / off) which questions should be used in the peer evaluation.  
   • Select (on / off) team questions: Additional questions about the functioning of the group as a whole.  
   • Switching (on / off) private comments towards the instructor  
   • Preview the evaluation as a student  
   • Feedback: The instructor can ask students to write personal feedback towards each group member. This feedback is only visible to the receiving student and not anonymous.  
   • Click 'manage questions' in the middle of the form and a dialogue opens where you can create new questions, edit those questions and even import custom questions from an old course you were involved in. Our advice: Use default questions. The default evaluation method is designed by the people behind CATME .They have years of experience with this evaluation method and are convinced that using this method consistently, offers the best results. In addition, allowing instructors to create their own questions requires more time. Because creating good questions requires time and testing. In order to offer an easy-to-use peer assessment solution, we thought it would be best to keep it simple and consistent with a proven method. 
   • Once you have created your first evaluation, click the Results button of this evaluation to see if all groups are added correctly. 
9. When an assessment is created, students receive an automated (customisable) email to complete the survey. 

1. When the start date has passed, all students will receive an email from BuddyCheck requesting them to evaluate their group. Each email has a unique link that enables the student to get started without having to go to Canvas first. However, it is also possible for students to open their evaluation from within the Canvas course and clicking the BuddyCheck tab. As an instructor, you will see the progress and on the Results page of the evaluation, you can monitor whether your students are filling in the evaluation correctly. If they do not complete in the time you can ask students to redo their evaluation.
2. The student fills in a number of topics for himself and his peers.

Actions you might need to take during the evaluation 

In the results tab, you can select one or multiple students in the results table using the checkboxes on the left. By doing so, action buttons will appear on the top of the results page:

Email to students who haven’t completed

The tutor can send a reminder to all students that have not yet submitted the evaluation. Click on the evaluation title and then you see a button 'More'. Click the button and click 'Send a reminder'. An e-mail with a link to the survey will be sent to each student who hasn't completed the survey yet.

Undo or redo an evaluation

Use the undo or redo button if you want students to fill out the survey again

Remove a Student

Use the remove button if students have left the course in the meantime. (Remember to also remove them from the groups in the Canvas course itself, otherwise, they will be added again when you create a new evaluation). 

View Results 

You can view all individual results in a table. Clicking on a student name shows more detailed data. Labels such as 'Underconfident' or 'High performer' quickly provide useful information for you. The final adjustment factor can be used to adjust the grade of a student in the Canvas grades area.  

Release Results

Go to the results page and click 'More '> 'Release results'. Once the results are released to the students, it not only becomes locked but it also sends an e-mail to all students with their results. You can see a preview of each student result page. Go to the results page and expand a student row in the table and click 'Preview result page'.

If you want to use the marks for a summative assignment please use the following method:

Assessing In Canvas 

1. Set up a group assignment in Canvas. 
2. Set the assignment up so that you are marking the students as a group 
3. Mark the students as a group 
4. Change the setting on the assignment to mark the students individually 

Downloading Grades In Buddycheck 

1. Go to the BuddyCheck assignment you created and the students have completed 
2. Remove Students who are no longer taking the assessment (see below)
3. Click the More button 
4. Select Publish grades 
5. Select the assignment  
6. Select Continue 
7. Adjust the weighting factor and review the effects of the grades 
8. reduce the decimals if needed
9. Select Accept and Continue 
10. Select Download backup which will keep a record of your previous marks and help you if you want to restore the grades 
11. Select Update existing grades 
12. In the Canvas grades area, note that the students will receive a comment telling the student that their grade was adjusted and by how much. 
13. Consider changing any grades and feedback relating to the change in grades made by Buddycheck

Remove Students

The student groups may contain students who have either left the course during the group work, or who have not taken part in the group activity because of illness, or other reasons. Leaving these in the BuddyCheck process could distort the adjustment factor calculations in such groups. These students can be identified by the fact that they received an average score of 1-2 and probably have not submitted themselves. We recommend that you remove these students from BuddyCheck. This can be done by clicking on the checkbox next to their names and using the remove button at the top of the column. Please note once removed they cannot be readded to this BuddyCheck evaluation. It is recommended to remove them from the group set in Canvas as well.

After the evaluation

When the evaluation deadline has passed, the evaluation is formally closed. However, students can still submit their survey. They will be marked red to indicate they were late. As soon as you are happy with the results, you can release the results to the students. After release, all students will receive an email with a link to look at their individual result. Students who haven't submitted the survey will also get the results. Students will not see the individual rating from group members. Just a total average of all group members. 

A ‘weighting factor’ is generated for each individual group member which is derived from each student’s input against defined criteria. Based on the total mark given to the group task, assessed and allocated by the academic tutor in the usual way, the weighting factor is then used to moderate marks, providing an individual mark for each student. 

This value is meant to be used as a way to evaluate how well students can work in teams. This factor can be used to adjust the grade of a student in the Canvas grades area. The adjustment factor is a value calculated by averaging the students’ scores across all question categories and dividing this by the average of the overall rating each team member received. It can be used as a percentage if multiplied by a hundred. 

How is the Adjustment Factor calculated? 

An Adjustment Factor value is a calculated number that attempts to show the contribution of a given student relative to the other members of their team. Specifically, this Adjustment Factor is the average rating of the student divided by the overall average rating for all members of the team.  

This video is from a previous version of Buddycheck but describes in detail how the adjustment factor is calculated.

With or without self 

With Self means that the student's self-ratings factored in to the adjustment factor. In some cases, exceptional conditions such as 'Cliques' may have an impact on the validity of the Adjustment Factor values, and in these cases, the Adjustment Factor columns will be highlighted.  

Original or capped 

You have the ability to choose between the original or capped adjustment factor. By default, the capped adjustment Factor value is capped at a maximum value of 1.05. Also, values above 0.95 are rounded up to 1.00 (our findings are that values in this range are just 'noise'). You can get an explanation of the capped adjustment factor by hovering over the switch on top of the results table. 

What if a student hasn’t completed an evaluation? 

If a student does not complete the evaluation, there is less peer evaluation data in the group of this student. This means that the results will be less reliable especially if it is a small group or if there are more students in the same group that have not completed the evaluation. That being said, an average and an adjustment factor of a student is simply calculated using completed evaluation data from the peers in the group. So, you can just use all results even if 100% of students have not completed the evaluation. Students that have not submitted do get an average and an adjustment factor because peers probably did complete an evaluation with this person included. 

Working with the results 

In the results tab, you might see Notes appear such as 'High' or 'Low' or 'Under'. On the top of the page, you can find for information on each label. As an instructor, you can use the results to give credits to high performers and talk to low performers or teams with problems. You can use the Adjustment factors to update the course grades too. However, be careful not to directly copy all Adjustment factors, since it often requires your human interpretation to correctly value the trustworthiness of the results and the assessments. The Adjustment Factors are visible on the right side of the Results tab. You have two types: Without Self and With Self. Without, Self is the factor without his or her own (self) ratings. With, Self is with the self-ratings included. For example the Adjustment factor is 1.05 and the student has a 7 (out of 10) as a course grade. You multiply 7 with 1.05, which means the final grade would be 7.35

You can amplify the effect of the adjustment on the final score by using weighting.

Weighting is a multiplier of the difference made by the adjustment factor.

The first part of the calculation works out the adjustment factor of the individual student grade.

So, for example, a student is doing well in the team and has 1.2 adjustment factor and a current grade is 70 out of 100 this equals a difference of 14.

70 x 1.2 = 84

84 - 70 = 14 (the difference between the old and new score)

If you set the weighting of this difference to 0.5 it becomes 70 + (14 x 0.5) =  77

if a student is not doing so well in the team and has 0.8 adjustment factor and a current grade is 70 out of 100 this equals a difference of 14.

70 x 0.8 = 56

70 - 56 = -14 (the difference between the old and new score)

If you set the weighting of this difference to 0.5 it becomes 70 + (-14 x 0.5) =  63

Yes, this is possible. The functionality is called 'Peer messages'. Peer messages allow you to ask your students to send optional feedback to each peer directly. Only the student that received the feedback will see the message if you release the results and it will not be anonymous. You can also see these messages. 

Yes, you can! Assuming you haven't locked or released the results yet. If you locked the results, you have to unlock the evaluation first. 

You go to your evaluation results where you see all students in a table. On the left of each student is a checkbox. Select each student evaluation you want to reopen and on the top of the page buttons appear, one of them being 'Reopen'. Click on it and optionally tune the email text and send it.  

Students will directly receive the email. With the link in it, they can double-check or adjust their evaluation and submit it again.  

If you already released the results but really need to reopen the evaluation for some or all students, please contact support using the chat inside the BuddyCheck interface. 

There are two options to make sure students cannot submit any more: Lock evaluation or release results. 

Lock evaluation 

Go to the results page and click 'More '> 'Lock evaluation'. Students cannot submit any more, even if you reminded them or reopened the evaluation for some students. 

Release Results 

Go to the results page and click 'More '> 'Release results'. Once the results are released to the students, it not only becomes locked but you also email all students with their results. You can see a preview of each student result page. Go to the results page and expand a student row in the table and click 'Preview result page'. 

Simply change the groups in Canvas groups area. You can synchronize groups and students after the start of the evaluation without losing any results. In case of a change, students are only asked to update their evaluation if the change applies to them.  

Go to the evaluation you created and click 'More' > 'Sync with course'. BuddyCheck will then ask for permission and get all students and groups and make changes accordingly. Only in scenarios where a student has to do the evaluation again, an email will be sent.  

If a student does not complete the evaluation, there is less peer evaluation data in the group of this student. This means that the results will be less reliable especially if it is a small group or if there are more students in the same group that have not completed the evaluation. That being said, an average and an adjustment factor of a student is simply calculated using completed evaluation data from the peers in the group. So, you can just use all results even if 100% of students have not completed the evaluation. Students that have not submitted do get an average and an adjustment factor because peers probably did complete an evaluation with this person included. 

Yes, students can still submit after the deadline. Their submissions will be marked red to indicate they were late. 

Previously the only option was a scale from 1-5 when students assessed their group members. Many asked for more flexibility, so now tutors can choose between a variety of scales for students to select from:

• 1-5

• 1-7

• 1-10

• Custom numeric scale

The custom rating scales functionality enables setting different point ranges for different evaluation questions, permitting tutors to make some aspects of the group work as more significant than others.

Tutors now have more refined control to manage outlier scores students may give their teammates. You can “mute” certain scores on a student’s given or received reviews. Muting excludes it from the grade adjustment calculations, preventing skewed results from unfair ratings.

Access the score management tools while viewing the evaluation results. You can choose specific ratings to mute rather than removing the entire review.

With easier scale configuration and more precise score handling, Buddycheck provides robust options for fair group work assessment. Take advantage of these new features for your next peer review project!