viewpoints viewpoints

Building A Better Mousetrap: The Rubric Debate

by Madeline Marcotte

The origin of the term “rubric” (roo´ brik) gives us our first clue as to why educators often argue over the use of this assessment tool. Because the headings, titles, and sections of early books were often printed in red and, in prayer books, the directions for conducting religious services were also printed in red, “rubric,” which comes from the Latin for “red” or “reddish,” has evolved to mean “an established custom or rule of procedure.” (Online dictionary) The term was adopted by educators in the 1980s to refer to a set of standards and/or directions for assessing student outcomes and guiding student learning. So, when we discuss scoring or grading rubrics in the Teaching Center, we are talking about a system designed to measure the key qualities (also referred to as “traits” or “dimensions”) vital to the process and/or product of a given assignment, a system which some educators see as stultifying and others see as empowering.

Well known in elementary and secondary schools as crucial to state-mandated student performance assessment, rubrics are now used similarly by post-secondary educators in all disciplines to assess outcomes in learning situations that require critical thinking and are multidimensional. Making up an assessment team called the “Synthesis Coalition,” Flora McMartin and others have found, after careful study, the rubrics their institution developed can be used to reliably score the performance-based and problem-solving assignments that now form a significant part of the undergraduate engineering curriculum at the University of California at Berkley. Similarly, in revamping their undergraduate engineering program to met new accreditation standards based “on what students have actually learned rather than what they have been taught,” the South Dakota School of Mines and Technology in Rapid City is using rubrics to establish “performance benchmarks” for the “behavioral objectives” appropriate to each year in the program. Engineering programs, of course, are not the only ones using rubrics. Joan B. Garfield of the University of Minnesota points out in “Beyond Testing and Grading: Using Assessment to Improve Student Learning,”

as goals for statistics education change to broader and more ambitious objectives, such as developing statistical thinkers who can apply their knowledge to solving real problems […] it is no longer appropriate to assess student knowledge by having students compute answers and apply formulas, because their methods do not reveal the current goals of solving real problems and using statistical reasoning.

Rubrics, Garfield continues, have become an important assessment tool in “achiev[ing a] new vision of statistics education.” Closer to home, our own successful Allied Health programs depend on rubrics to both assess and encourage student learning.

In short, well-designed rubrics help instructors in all disciplines meaningfully assess the outcomes of the more complicated assignments that are the basis of the problem-solving, inquiry-based, student-centered pedagogy replacing the traditional lecture-based, teacher-centered approach in tertiary education. “Meaningfully” here means both consistently and accurately—accurately measuring the specific entity the instructor intends to measure consistently student after student. However, given their association with standardized assessment and the dangers of those that are poorly designed, rubrics are not without their critics. Yet, a clear understanding of how rubrics operate can help educators of all levels design rubrics that facilitate, rather than obviate, student learning and teacher improvement.

The argument for using rubrics and sharing them with students

Rubrics can be used either for “filtering”—as they are used in placement testing—or for “latticing,” or “scaffolding”—if they are shared with students prior to the completion of any given assignment. When instructors plan on grading student thinking and not just student knowledge, they should articulate the vital features that they are looking for and make these features known to the student. Doing so, many educators argue, increases the likelihood of a quality product. Kathleen Montgomery, in her article “Authentic Tasks And Rubrics: Going Beyond Traditional Assessments In College Teaching,” contends that traditional assessment practices used to grade papers, for example, are not helpful to the students struggling to write the paper: “The instructor’s comments on papers and tests are done after rather than before the writing, so they cannot serve as guidelines, compromising the value of writing comments at all.” Moreover, rubrics can help the student with self-assessment; what is most important here is not the final product the students produce, but the habits of mind practiced in the act of self-assessment. However, for the student to successfully use a rubric this way, the criteria must be made clear to them and the jargon used must not only be understandable to the student but also be linked specifically to classroom instruction.

Some educators advocate going beyond merely sharing rubrics with students. Rick Stiggins, of the Assessment Training Institute, contends that we ought to illicit student input when constructing rubrics: “Perhaps the greatest potential value of classroom assessment is realized when we open the assessment process up and welcome students into that process as full partners” (qtd. in Skillings and Ferrell). When students are full partners in the assessment process, as Mary Jo Skillings and Robin Ferrel illustrate in their study on student-generated rubrics, they tend to “think more deeply about their learning.”

In any case, withholding assessment tools (whether they are rubrics or more nebulous modes of evaluation) from students is not only unfair and makes self-assessment more difficult, it maintains the traditional gap between what the teacher knows and what the student knows. When instructors do not explicitly delineate the qualities of thought that they are looking for while grading, they reduce learning to a hit or miss endeavor, where “assessment remains an isolated […] activity and the success of the learner is mostly incidental” (Montgomery). The result is many students struggle blindly, especially non-traditional, unsuccessful, or under-prepared students, who tend to miss many of the implied expectations of a college instructor, expectations that better prepared, traditional students readily internalize. As both institutional enrollment needs and social pressures for access raise the need of remediation (Soliday), rubrics become increasingly important to student success. Gisselle O. Martin-Kneep, in “Standards, Feedback, and Diversified Assessment: Addressing Equity Issues at the Classroom Level,” reports that extensive use of rubrics can help minimize students’ educational disparities and bring fairness into assessment on numerous levels: “In short, explicit performance criteria, along with supporting models of work, make it possible for students to use the attributes of exemplary work to monitor their own performance.”

The argument against using rubrics

While many educators make a compelling argument for sharing rubrics with students, others worry that doing so will encourage formulaic writing. That “rubric” is listed in most thesauruses as a synonym for “formula” does nothing to dismantle such fears. Well-designed rubrics, though, should not do this; unfortunately, most state issued rubrics used in secondary school standardized testing are poorly designed rubrics that list specific static elements encouraging students to simply make sure their essays have those features. A rubric that tells students, as a typical example, that they will get an A for writing a 1000 word essay that “cites x number of sources and supports its thesis with at least three arguments” will lead students to perceive writing as a kind of “paint-by-number” endeavor (Mathews).

The rubric debate has even made the popular news—not the prime time news, of course, but an article in the Washington Post, “Writing By the Rules No Easy Task,” the publication of which reveals both how widely rubrics are used in standardized testing and how concerned educators are about their potential to harm students learning. In the Washington Post article, Kenneth Bernstein, a high school social studies teacher “acknowledges that ‘some rubrics are dumb.’” He recounts,

I once gave extra credit to a student who realized that without providing a shred of meaningful content she could meet all the requirements of a state writing rubric he posted in his classroom. As required she used the word “persuade” and two synonyms, composed a clear topic sentence and closing sentence, and made no spelling or grammatical errors. But she did it without saying anything coherent.

Moreover, some teachers have noticed how students who were good writers become wooden when writing under the influence of a rubric. Dona Patrick, an elementary school teacher noticed that while her sixth grade students did well on their state writing test, those students who had been natural writers, those students who had “stylistic voices full of humor and surprises, produced less interesting essays when they followed the rules [as outlined in a rubric]” (Mathews). Similarly, Heidi Andrade, in her study, “The Effects of Rubrics on Learning to Write,” has found that, while rubrics increased her students’ knowledge of the grading criteria and helped most of her students (especially the young male students) do well on the state writing test, many of the young female students, who had been more expressive in previous writing assignments, wrote poorly when writing, as we might say, to the rubric.

More conceptually, critics claim that rubrics, in effect, dehumanize the act of writing. According to Thomas Newkirk, an English professor at the University of New Hampshire, “rubrics promote ‘mechanical instruction in writing’ that bypasses ‘the human act of composing and the human gesture of response’” (Mathews). This overly scientific view of writing, Newkirk and others argue, stunts the learning process. Moreover, Judith Halden-Sullivan sees a disconnect between the learning goals of Writing Across the Curriculum programs and the rubrics often designed to assess that learning. Assessment of this sort seems at odds with such concepts as “deep learning,” which implies a kind of learning that is beyond measurement, an elusive hard to describe enlightenment, but identifiable in the same way good art is: teachers know deep learning when they see it. Rubrics, Halden-Sullivan contends, reduce “deep learning” to “checksheets.”

It’s the design

However, these critics of rubrics, while their critiques should be considered, mistake the design of specific rubrics with the concept of rubrics in general. Rubrics that are prescriptive rather than descriptive will promote thoughtless and perfunctory writing; such rubrics are as limiting to the development of rhetorical mastery as the five-paragraph essay. And, rubrics cannot be the sole response to a student’s paper; sound pedagogy would dictate that rubrics should be used in conjunction with other strategies, such as the letter writing/dialogic approach to assessment that Halden-Sullivan describes as preferable to the rubric. Her complaint that rubrics simply assess the written product (thereby placing the emphasis on presentation of ideas and not encouraging a deep, intense and messy grappling with ideas) is, again, a complaint about rubric design. Rubrics can be designed to measure either product or process or both; and, they can be designed with dimensions describing the different levels of that “deep learning” so valued in WAC programs.

In spite of the problems associated with many of the rubrics designed for use in state mandated testing, for the last fifteen years or more, advocates of rubrics at all educational levels have argued that rubrics provide students with clear and specific qualities to strive for in those assignments that “are open-ended, aligned more closely to real-life learning situations and the nature of learning” (Skillings and Ferrell) and mitagate both teacher bias and the perception of teacher bias (Mathews). Indeed, since rubrics allow for widespread assessment of higher-level thinking skills, performance-based assessment is replacing or complementing more traditional modes of testing; this in turn means that teachers are changing their instructional modes to prepare their students for these tests. Kenneth Volger, in his study, “The Impact of High-Stakes, State-Mandated Student Performance Assessment on Teacher’s Instructional Practices,” analyzes the survey responses of 257 10th grade English, math, and science teachers and concludes that, since the implementation of such tests in public schools, there has been “notable increases in the use of open-response questions, creative/critical thinking questions, problem-solving activities, […] writing assignments, and inquiry/investigation.”

Build a better mousetrap

By keeping the arguments against rubrics in mind, college instructors can devise better rubrics avoiding the pitfalls often associated with state-issued rubrics imposed on public primary and secondary schools and capitalizing on the benefits cited by so many educators. Indeed, as Gregory Schafer and Miles McCrimmon both argue in their respective papers in the March 2005 issue of Teaching English at the Two Year College (TETYC), college instructors—particularly those at community colleges—will need to deal with the ill effects of state designed rubrics imposed on their students during high school. To have the necessary important conversations about rubrics—to build better ones, fix the problems casued by poor ones, and pre-empt encroaching state interference, college faculty need a shared vocabulary and a basic understanding of how rubrics operate.

Rubrics 101

Most of this section is a paraphrased synthesis of Chicago Public Schools' excellent overview of rubrics and Barbara Moskal's article, "Scoring Rubrics: What, When and How?

General or specific

To begin with, rubrics can be either “general” or “specific.” General rubrics can be applied to various assignments; for example, one rubric can be used to assess all of the different papers assigned in a freshman composition course. Specific rubrics, on the other hand, are particular to a given assignment—one rubric for a narrative essay, another one for an argumentative essay. Both types of rubrics benefit the teacher and the student in varying degrees: the teacher who relies on a general rubric does not have to develop a new one for each assignment and the student grows to understand fundamental standards in writing—like form and coherence—exist across the board; meanwhile, the teacher that uses specific rubrics is always composing new descriptions of quality work, but their students have clearer directions for each assignment. Of course, a teacher could have the best of both worlds here, by designing a rubric on a PC that allows for the easy insertion of assignment specific traits.

Dimensions and scales

There are two basic elements to a scoring rubric, whether general or specific: 1. the vital “traits,” key qualities, or “dimensions,” to be rated, and 2. the “rating scale.” The traits, or dimensions, will serve as the basis for judging the student response and should reflect the vital aspects of the assignment; that is, if a writing assignment is designed to improve critical thinking, then the rubric should describe traits or dimension of writing that signify critical thinking has occurred. If a teacher wishes to assess how well her students have learned the basic elements of composition, she would use a rubric that described those traits. For example, Illinois State Board of Education’s (ISBE) scoring guidelines for writing measures four separate attributes of composition: Focus, Support, Organization, Conventions.

An instructor can measure student learning by referring to detailed, specific descriptions of the trait as it manifests itself at different levels. For example, a trait like “support” might be described at the higher end of the rating scale as “extensive, reliable, and well-documented support” while at the lower end it might be described as “unconnected or irrelevant support.” Consequentially, when rubrics are published in the classroom, students striving to achieve the descriptions at the higher end of the scale in effect guide their own learning. We must keep in mind, however, that other aspects of good pedagogical practice play into student success: rubrics that are outside of the students “zone of proximal development” are useless to the students.

Rubrics can have any number of points along a scale—the ISBE’s rubric rates each trait on separate six-point scales—as long as each point on the scale is well-defined. This may be difficult to do for longer scales. While longer scales make it harder to get agreement among scorers (inter-rater reliability), extremely short scales make it difficult to identify small differences between students.

Analytical or holistic

Rubrics are also described as either “analytic” or “holistic.” A rubric with two or more separate scales is called an analytical rubric, as it takes apart or breaks up the rating system for each trait; a rubric that uses only a single scale is called a holistic rubric. A holistic rubric is more efficient and the best choice when criteria overlap and cannot be adequately separated; an analytical rubric, however, will yield more detailed information about student performance and, therefore, will provide the student with more specific feedback.

Weighting

Usually a numerical value is assigned to each point on a scale. You can weight dimensions differently if you feel that one dimension is more important than another. There are two ways in which you can express this value judgment: 1. You may give a dimension more weight by multiplying the point by a number greater than one. For example, if you have four dimensions (content, organization, support, conventions) each rated on a six-point scale, and you wish to emphasis the importance of adequate support, you could multiply the support score by two. 2. You may devise scales of unequal length, which would mean that the shorter scales would count less than the longer ones. For example, organization, support, and content could each be rated on separate 6 point scales, while punctuation and / or spelling could be rated on separate 3 point scales. A paper that was well organized and punctuated would yield 6 for organization and 3 for punctuation. A paper that was perfectly punctuated but poorly organized might yield a 3-3 score.

The issue of weighting may be another area in which you can enlist the help of students. At the beginning of the process, you could ask a student to select to select which aspect she values the most in her writing and weight that aspect when you assess her paper.

Adopt a rubric

You can adopt a rubric—“use an existing one ‘as is’” (Chicago Public Schools) A search on Google will list hundreds (of thousands) of sites, but here is a list of sites cut and pasted from the Middle States document, “Best Practices in Student Outcomes Assessment.”

Adapt

You can adapt a rubric—

modify or combine existing rubrics; re-word parts of the rubric; drop or change one or more scales of an analytical rubric; omit criteria that are not relevant to the outcome you are measuring; mix and match scales from different rubrics; change the rubric of use at a different grade; add a “no-response” category at the bottom of the scale; divide a holistic rubric into several scales. (Chicago Public Schools)

And, you could add references to your specific handbook or style guide or change an analytical rubric into a holistic one.

Start from scratch

Or you can build your own rubric from scratch—convert existing revision or discovery heuristics into rubrics; convert comments that used to show up on A, B, C, D, and F papers into descriptive phrases, or start completely anew. The Chicago Public Schools web-site offers simple guidelines to follow when designing your own rubric. If you visit the web page I cut and pasted this from, you will find that each item is hyperlinked to a full explanation of the step.

Steps in developing a scoring rubric

In addition to these basic directions, you should consider your purpose and audience.

Purpose and audience

First, you must decide whether you need a rubric. Consider what the purpose of assessment is. While the fundamental focus of assessment is always to promote learning, there are other reasons why we engage in assessment: curriculum reform, placement, promotion, diagnosis, accountability, and so on (Critical Issue). Then, consider what you are assessing. If the outcomes you wish to measure are multi-dimensional, chances are you need a rubric whatever the purpose of assessment is. Clearly defining the purpose of assessment and what you want to assess is the first step in developing a quality rubric. The second step is deciding who your audience is going to be. If the rubric is primarily used for instruction and will be shared with your students, then it should be non-judgemental, free of educational jargon, and reflect the critical vocabulary that you use in your classroom.

Build a metarubric

A constructivist approach to teaching would dictate that, if we have assigned ourselves the task of getting a good rubric to use, we need a rubric to judge our performance—that is, we need a meta-rubric to assess our rubric. Make a list of the vital qualities, or dimensions, that you feel would make up a quality rubric for your audience and your purpose. Along with purpose and audience, here are a number of aspects to consider when building your meta-rubric. Chicago Public Schools web-site lists the following criteria for evaluating scoring rubrics, criteria they have adapted from Herman, Aschbacher and Winters (1992), Arter (1990), and ISBE (1994):

Does the rubric relate to the outcome(s) being measured? Does it address anything extraneous? […] Does it cover important dimensions of student performance? Do the criteria reflect current conceptions of excellence in the field? […] Are the dimensions and scales well defined? […] Is there a clear basis for assigning scores at each scale point? […] Can different scorers consistently apply the rubric? […] Can students and parents understand the rubric? […] Is the rubric developmentally appropriate? […] Can the rubric be applied to a variety of tasks? […] Is the rubric fair and free from bias? Does it reflect teachable skills or does it address variables over which students and educators have no control, such as the student’s culture, gender or home resources? […] Is the rubric useful, feasible, manageable and practical? […] Will it provide the kind of information you need and can use effectively?

Barbara Moskal, in her article “Scoring Rubrics: What, When, and How?” insists that rubrics should be non-judgmental: “Each score category should be defined using description of the work rather than judgments about the work.” For example, “sentence structure follows current conventions” would be better than “sentence structure is good.” So, here is another question we may ask, if we agree with Moskal: Is the description of criteria judgemental?

Giselle Martin-Kniep, in her article, “Standards, Feedback, And Diversified Assessment: Addressing Equity Issues At the Classroom Level,” lists a number of criteria that she values. Among them are these two: “Is the assessment responsive to what we know about how [students] learn?” and “Does the assessment help students become the kinds of [citizens] we want them to be?”

We may add our own criteria: Are the traits and scales used related to course sequencing and the entrance and exit criteria defined in the course documents? How well is the rubric tied to instruction? That is, does the rubric use the same critical vocabulary used in our instruction? Does the rubric encourage risk taking? Creativity? Self-expression? Do the students find the rubric helpful? Does the rubric encourage students to be independent writers? Does the rubric exaggerate or minimize past educational inequities? Does the rubric evaluate the product or the process?

What else?

Feedback

Be prepared to evaluate your rubric, using your meta-rubric and feedback—direct feedback from the students and indirect feedback from the quality of their work. Modify accordingly.

Works Cited

“Critical Issue: Integrating Assessment and Instruction in Ways That Support Learning.” Pathways To School Improvement. North Central Regional Education Laboratory. 3 Jan 2003.

Andrade, Heidi Goodrich. “The Effects of Instructional Rubrics on Learning to Write.” Current Issues in Education [On-line] 4.4 (2001). 15 Aug 2005.

Chicago Public Schools. Chicago Board of Education. “Ideas and Rubrics.” Instructional Intranet 15 Aug 2005 .

Garfield, Joan B. “Beyond Testing and Grading: Using Assessment to Improve Student Learning.” Journal of Statistics Education, 2.1 (1994). 15 Aug 2005.

Halden-Sullivan, Judith. “Writing to Learn, Assessing to Learn.” Language and Learning Across the Disciplines, 3.1 (1993): 25-40. 15 Aug 2005

Martin-Kniep, Giselle O. “Standards, Feedback, and Diversified Assessment: Addressing Equity Issues at the Classroom Level.” Reading and Writing Quarterly, 16.3 (2000). 6 Jan 2003.

Mathews, Jay. “Writing by the Rules No Easy Task; ‘Rubrics’ Can Help Students Focus on Basics, But Some Teachers and Parents Say They Squelch Creativity.” The Washington Post, 24 Oct 2000. 3 Jan 2003.

McMartin, Ann McKenna and Ken Youssefi. “Establishing the Trustworthiness of Scenario Assignments as Assessment Tools for Undergraduate Engineering Education.” 29th ASEE/IEEE Frontiers in Education Conference, 10-13 Nov 1999. San Juan, Puerto Rico. 15 Aug 2005.

Montgomery, Kathleen. “Authentic Tasks and Rubrics: Going Beyond Traditional Assessments in College Teaching.” College Teaching, 50.1 (2002). 3 Jan 2003.

Moskal, Barbara M. “Scoring Rubrics: What, When and How?Practical Assessment, Research and Evaluation, 7.3 (2000). 12 Dec 2002.

Schafer, William D. “Effects of Teacher Knowledge of Rubrics on Student Achievement in Four Content Areas.” Applied Measurement in Education, 14.2 (2001). 6 Jan 2003.

Skillings, Mary Jo and Robin Ferrell. “Student-Generated Rubrics: Bringing Students Into The Assessment Process.” Reading Teacher, 53.6 (2000). 6 Jan 2003.

Volger, Kenneth E. “The Impact of High-Stakes, State-Mandated Student Performance Assessment on Teacher’s Instructional Practices.” Education, 123.1 (2002). 6 Jan 2003.

index button

©Copyright 2006. Contact author for permission

Maintained by Jay Howard,January 2006