Proficiency Illusion

The proficiency illusion

Are schools in Kansas as good as you think they are? The record on state assessments looks positive.

Kansas on state assessments, percent proficient or above
Source: Kansas Report Card (KSDE)
Year Reading Math
2003-04 70.5% 65.3%
2004-05 73.7% 68.6%
2005-06 80.3% 74.7%
2006-07 81.0% 78.3%
2007-08 84.3% 82.0%
2008-09 85.8% 83.5%

But let’s look at some other metrics.

When you’ve got a phenomenon you’d like to evaluate–in this case, student performance–it’s useful to use several tools to look at it. State assessments are one tool. Another is the National Assessment of Educational Progress (NAEP), a national test that is given across the country. As you can see from our page on the NAEP, the performance of Kansas students on that test is not so impressive. As least as far as reading scores go, the performance of Kansas students has been stagnant.

So when you compare the NAEP with the state assessments, you find some very interesting results.

Kansas assessments (“meet standards” or better) versus NAEP (“proficient” or better) 

2009

Reading (4th) Reading (8th) Math (4th) Math (8th)
Meet standards+ (KS) 87 85 87 77
Proficient+ (NAEP) 35 33 46 39
Difference 52 52 41 38

Proficiency Games: The incentives

This discrepancy between state tests and the NAEP is not unique to Kansas. A 2008 paper from the Flint Hills Center for Public Policy (available here) gives an extended treatment. Under No Child Left Behind, states must achieve 100% proficiency, but they are free to define proficient as they see fit. This gives states an incentive to redefine proficiency in a way that makes it easier for schools to comply with NCLB, with or without actually improving student performance.

Education Week

The average graduation rate in Kansas schools is, according to Education Week, 74%, rather than the 88% reported by the Kansas Building Report Card.

Education Next

In an article entitled, Few States Set World-Class Standards, Paul E. Peterson (Harvard) and Frederick Hess gave Kansas a C-, citing the discrepancy between the stat’es NAEP scores and reported assessment scores.

U.S. Department of Education

A report published by the U.S. Department of Education, “Mapping 2005 State Proficiency Standards onto the NAEP Scores,” translated (or mapped) state tests, which vary from state to state, onto NAEP, a common standard. Here are some of its findings:

  1. What counts as “proficient” in most states does not even meet “basic” (the lowest grade) on the NAEP.
  2. What counts as proficient in one state could be as much as 80 NAEP points (out of 500!) different from what counts as proficient elsewhere.
  3. Kansas students could score just above “basic” (barely know what they’re doing) on the NAEP and score as proficient on the state test, at least on some tests.
  4. To get proficient on a Kansas test for 4th-grade math in 2005, you would have to score a 218 on NAEP. The cutoff for scoring “basic” was 214. For proficient, it was 249. By the way, “basic” means “partial mastery of prerequisite knowledge and skills that are fundamental for proficient work at each grade assessed.” The cutoff for “proficient” was (Table 3 and Figure 4)
  5. To get proficient on a Kansas test for 8th-grade reading in 2005, you would have to get a 242 on NAEP. The cutoff for scoring “basic” was 243; for proficient it was 281.  (Table 2 and  Figure 3)
  6. To get proficient on a Kansas test for 4th-grade math in 2003, you would have to get a 226 on NAEP. (Table D-3) The cutoff score for basic was 214; for proficient, 249.
  7. To get proficient on a Kansas test for 8th-grade reading in 2003, you would have to get a 253 on NAEP. (Table D-2) The cutoff score for basic was 243; for proficient, 281.

There’s one very big caution in order: the state gave new names to the scoring categories, starting with the 2005-06 (see this PDF for the 2005-06 accountability report), and changed the labels for the 2006-07 school year. (PDF memo). It is possible that the new tests are more rigorous.

Thomas B. Fordham Institute

In its report, “The Proficiency Illusion,” the institute measures what counts as proficient on the Measures of Academic Progress (MAP) with what counts as proficient on state assessments. The technical name for this is “cut scores.”

Here’s what it said in the Kansas-specific section (PDF) of the report.

  1. When all grades are considered, Kansas is an average state. That is, its tests are average in difficulty.
  2. In 8th-grade reading and 8th-grade math (both subjects and grade levels tested on NAEP), Kansas tests were somewhat easier.
  3. It is more difficult to score as proficient in 5th-grade reading than in any other subject or grade level, if you compare Kansas with other states.
  4. It’s generally easier for a Kansas student to score as proficient in reading than in math.
  5. It’s easier for a Kansas student to score as proficient in math in the earlier grades, even after adjusting to the obvious increase in difficulty in later grades.

Bottom line

While Kansas students do better on state assessments, they aren’t improving on the national assessments. In a national economy, the discrepancy should at least draw some attention.

Trackbacks

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s

Follow

Get every new post delivered to your Inbox.

%d bloggers like this: