No Child Left Behind (NCLB), the country’s most recent version of our Federal education legislation has been called many things, from a tough-love policy that will right educational wrongs for traditionally underserved students to a grossly misguided set of punitive mandates that severely limit education while benefiting the private education testing and text-book sector.

As NCLB is due to be both modified and reauthorized soon (though rather minimally and not this year—see the November 8th and October 18th School Beat columns for details), the debate over NCLB and what federal education legislation should be has been extensive and heated. Many educators have pointed to the ways in which NCLB fails, and will continue to fail, to support a solid, well-rounded, engaging educational system. These same individuals have proposals for implementable legislation with accountability goals and methods that are actually achievable.

Many researchers have also focused on the facets of NCLB that are unfair and/or simply unachievable, the most notorious being the mandate that all school-age children in the United States will be proficient in math and reading by 2014. Unless someone’s hiding a magic wand in Washington D.C., that deadline is never going to be met.

An interesting perspective on the structural weaknesses of NCLB as a piece of legislation has come in the form of two reports over the last two years, one of which was just published this month. Kevin Carey, Research and Policy Manager at Education Sector, just released a new study called The Pangloss Index: How States Game the No Child Left Behind Act (“The Pangloss Index”). This new paper follows up on the report he issued in May of 2006 entitled Hot Air: How States Inflate Their Progress Under NCLB (“Hot Air”) where he first unveiled the aptly named “Pangloss Index” as a measure and reflection of the serious flaws in NCLB.

These reports are quite compelling for many reasons, the most significant of which is that they examine and critique NCLB on its own grounds. Up for question in these studies are not the general goals and methods of NCLB (not for instance, if standardized testing is useful or useless), but the ultimate outcomes that NCLB is supposed to generate. The premise of this body of research is that if NCLB is working as advertised, then states that are doing well or poorly according to the different measurements required by NCLB should appear consistently strong or weak when those same areas are evaluated using other methods. If this is not true, then the scores and assessments of state educational systems that NCLB causes to be produced are unreliable and by definition, NCLB is not effective at achievement its goals. [REDO]

Carey’s approach has been to take the core set of statistics states are required to report on and turn them into a single index called the “Pangloss Index” (named after the proverbial “best of all possible worlds” view of life) against which states can be ranked relative to each other, including ‘...student proficiency rates in elementary, middle, and high schools, the percent of schools and districts making “adequate yearly progress,” high school graduation and dropout rates, school violence ratings, teacher and paraprofessional qualifications and teacher access to high-quality professional development.’ (See “Hot Air” p 11). While other researchers have looked at individual areas, this composite allows for an assessment of the overall effectiveness of NCLB, in some ways giving the policy and it’s supporters a taste of their own medicine by being reduced to a single score.

Since all state plans are approved by the US Department of Education (DOE), according to NCLB logic, the data each state sends should truly be an adequate “apples to apples” comparison. Of course this is not the case at all, because each state’s standards are so vastly different, with DOE heads Spellings and her predecessor Rod Paige granting variances to favored states to make it easier to meet requirements. Further, states have such latitude in defining the standards that define success—and again the DOE seems willing to support that flexibility—that comparisons are fraught and the measures can be quite meaningless. In other words, what looks like educational success in one state is often just a cover-up for continued poor performance and what looks like failure in another state is sometimes a reflection of more rigorous standards.

According to Carey’s Pangloss Index for data filed in March 2006, Wisconsin was doing the best, Hawaii was doing the worst, and California ranked 44th. Massachusetts, which has some of the most rigorous standards, assessments and correspondingly high results according to a multitude of nationally normed tests (such as the National Assessment for Education Progress) got ranked an unbelievable 39th. While the California ranking is unfortunately not that startling, given where we’re placed with funding levels and the challenges of our diverse population, the results for Massachusetts are an indicator that something is off.

Something is indeed off, and it’s the ability of states to legally, overtly and with the DOE’s blessing, mess with the system. Carey’s 2007 study, “The Pangloss Index,” provides an in-depth look at how one state, Alabama, accomplished this. In 2006, Alabama was 22nd on the Pangloss Index; in 2007 it was 5th. A variety of strategies were used to make this jump, most of which were standards manipulation and not educational program changes, since there were be little way for such a dramatic jump to occur otherwise.

Carey documents the specific tactics Alabama employed, from creating inimum subgroup sizes for reporting that were so big they excluded large groups of minority students’ scores to using the statistical tool of confidence intervals that are used in sampling. This second strategy is invalid because NCLB required data are not samples but reports for all students required to be tested and reported on. Nonetheless the DOE approved it and Alabama, was able to adjust test score reporting so that if a school’s score was below the NCLB required target but within the confidence interval, it would still count as passing.

Although Carey’s research is quite critical of NCLB, it should be more generally read as a cautionary tale for both policy makers and supporters. The moral of the story is that standards should be applied equally to all states and tools such as peer review and precision about requirements and definitions are essential. Underpinning everything must be a commitment on the part of agencies such as the DOE to enforce and implement policies fairly and appropriately, which of course is the real lynchpin and the most difficult aspect to achieve, as policy goals are so easily trumped by immediate political and monetary rewards. This is where legislators need real educational practitioners on hand to advise them not only about goals and objectives, but about how progress towards those goals and objects can be fairly and consistently measured.

Lisa Schiff is the parent of two children who attend McKinley Elementary School in the San Francisco Unified School District and is a member of Parents for Public Schools of San Francisco and the PTA.