For the second calibration workshop, inter-rater reliability of scoring was determined for each rubric criterion via intra-class correlations (ICCs) in a two-way mixed model.
After the second rubric calibration workshop, inter-rater reliability of the rubric was determined through a follow-up exercise in which raters independently applied the rubric to assess performance on a third sample of work: a different assignment for a different course.
During the early part of the study--spring 2012 through fall 2013--instructors could choose between two versions of the community rubric when assigning peer review: the numeric rubric, which requires students to score rubric criteria on a five-point scale, and the discuss rubric, which requires students to write textual comments regarding these criteria rather than scores.
Beyond the rubric scores, students and instructors provided on intermediate drafts reported on in this study, reviewers provided written comments, including .pdf sticky notes and drawings and text notes on top of students' papers, rubric-based comments, Community Comments, summary notes, and revision plans.
Rubric 2: Planning to Support Varied Learning Needs
For rubric 2, the teacher candidates were rated on how they used knowledge of the children to support their varied learning needs.
Before this action research project was conducted in the first semester of 2016, in terms of assessment, students were asked to carry out an individual presentation on different topics and they were assessed through a rubric that used seven criteria (introduction of the topic; knowledge of the topic; ability to engage and involve the audience; suitability of presentation for purpose and audience; voice: clarity, pace, and fluency; vocabulary: sentence structure and grammar; and pronunciation) with a maximum score of 70 points.
The first rubric (see Appendix A) was used during the first two months of the semester to assess individual presentations; in this activity, each student was required to prepare a 15-minute presentation on a topic related to the contents of the BS1 course.
* To explore the effects of rubric
on the reliability of the raters while assessment of a written essay.
The first specification implies that every participant that uses the rubric
to score a particular task should be able to select the desired level of detail.
This linguistic strategy represents a departure from the language of other ESL and EFL writing rubrics
, like the TOEFL rubrics
, as well as rubrics
that assume native-speaker test takers, like the SAT writing rubric
It does not take too long before we, as teachers, seem to develop an innate ability to score remaining projects more quickly--a feat accomplished largely based on prior experience and expertise with the content area, assignment and assessment rubric