February 1, 2013
By Steve Shewfelt and Ellen Grantham, NEA Office of Research & Analysis
Art Texture – Colored Canvas 2 by flickr user paurian
One of the most rewarding aspects of working in the Office of Research & Analysis is our frequent collaboration with artists and others in the arts and cultural fields—a passionate, creative bunch. In our time with the Arts Endowment, we have found such opportunities to be endlessly surprising and interesting.
It’s a little-known fact, however, that researchers also like to think of themselves as passionate and creative. True, they tend to get excited about different things than artists, but solving difficult methodological and empirical challenges demands passion and creativity. So when the time came to work on the Post-Grant Review (PGR) project, a new pilot initiative at the NEA, we jumped at the opportunity. The PGR not only promised the opportunity to work more closely with program specialists here at the NEA, but it also presented a host of research challenges we were excited to tackle.
Why Post-Grant Review?
Advancing excellent art is at the core of the NEA’s mission and goals, as outlined in the agency’s Strategic Plan. PGR is part of the NEA’s multi-faceted effort to hold ourselves accountable for achieving a key strategic goal and outcome: “Creation of Art that Meets the Highest Standards of Excellence” so that “the Portfolio of American Art is Expanded.”
There is a larger backdrop to all this. Our focus on outcomes-based measurement responds directly to Congressional and White House requirements for performance accountability. The relevant statutes are designed so that federal agencies will:
- Improve the confidence of the American people in the capability of the federal government, by systematically holding federal agencies accountable for achieving program results;
- Improve program performance by requiring agencies to set goals, measure performance against those goals, and report publicly on progress.
With this heightened sense of purpose, the NEA’s Office of Research & Analysis has embarked on an effort to help us better understand where, when, and how excellence is achieved in NEA-funded projects.
But our goals with this project go beyond fulfilling an external mandate. We are excited about the wealth of information the project promises to yield, information that can aid the NEA in years to come. We want to build a process that will help everyone at the NEA better understand how we’re doing with respect to our goals. We want to build a process that can give our staff useful information about their programs and that, over time, will help them think about what are the best ways to advance artistic excellence in their disciplines and in the United States.
Designing the Pilot Project
To begin the project, we selected three of the agency’s arts program offices to participate in the pilot: dance, media arts, and presenting. Then came the fun part. We in the ORA began to work closely with the pilot disciplines to develop a set of review criteria. The program offices have extensive experience convening independent experts and using well-established review criteria to assess applications for funding. How would they want to approach the idea of developing criteria and processes for reviewing finished work products?
Some of the challenges we encountered in the course of our conversations were almost existential (admittedly, these were also the most exciting questions): What is excellence in the creation of art? What should we ask reviewers to tell us about the nature of and dimensions along which it was achieved?
Other challenges had more of a policy bent: Whom should we ask to participate as reviewers? What kinds of conflict-of-interest rules should apply? What kinds of materials would we be able to provide the reviewers, beyond what we already collect in the form of final reporting? How would we analyze and report our findings?
Still other challenges were logistical. How many grants would it be reasonable to ask our reviewers to assess? How many reviewers would we ask to participate and how would we get materials to the reviewers?
As we worked through these challenges, we frequently reminded ourselves that this was a pilot test whose purpose was to better understand how the process might work. In many ways, it is an unprecedented undertaking for the NEA, and our goal at this stage is more about learning what does and doesn’t work than it is about the findings themselves.
For example, we decided that each program office or artistic discipline would develop and test its own review criteria and tools for collecting information from reviewers. We settled on five reviewers and 20 grants to be reviewed per discipline. And we determined early on that grant-level results should remain private. That is, assessments will not be shared among reviewers and neither scores by individual reviewers nor overall results for a particular grant will be available to the public or considered in future pre-award panels. Results will be made available only after they have been aggregated at the level of the discipline, thus ensuring that the initiative will in no way affect grantees’ future applications for an NEA award.
For all the work we’ve done to get to this point, we are still in the relatively early stages of this pilot project. In the next few weeks, we will be contacting grantees from the three pilot disciplines that were randomly selected to participate in the process to request any available work samples that could be distributed to reviewers. Each of the discipline directors on our pilot team will be reaching out to potential reviewers, looking for experts in the field who are adventurous enough to go through this pilot test with us. We expect the actual reviews to be conducted later this spring.
We in the Office of Research & Analysis are truly excited about the prospects for this project. For one thing, it holds the promise of helping us speak in a more meaningful way to our ultimate bosses, the American people, about what is being accomplished with NEA funding. Moreover, it represents for us a vivid demonstration of our commitment to being thoughtful about pursuit of our mission and about self-improvement. We also hope that this undertaking will teach us some lessons and push the envelope in thinking about how funders can understand the on-the-ground impact of their grant-making activities.
Perhaps above all, we have enjoyed working with the others here at the NEA and having the opportunity to get our creative juices flowing in a new research direction. Stay tuned for more information on the project and the lessons we’ve learned.