1. When was the BI Criterion implemented?

The merit review criteria were changed from the four criteria of the 1981 version to two criteria in 1997. The revised criteria retain the elements of the 1981 criteria, but add three areas: 1) the creativity and originality of concepts in a proposed activity (incorporated into the Intellectual Merit criterion); 2) the specific intention to promote teaching, training, and learning in addition to advancing discovery and understanding; and 3) broadening the participation of underrepresented groups. The last two of these are incorporated into the broader-impacts criterion. The need for research to benefit society and contribute to STEM infrastructure were requirements prior to the implementation of the broader-impacts criterion per se.

2. Why was the BI Criterion implemented?

Some researchers view the broader-impacts criterion as an arbitrary dictum. Existing broader-impacts-related materials do not stress enough the correlation between broader impacts and the health of the STEM research enterprise. For example:

1. Restructuring the criteria makes the broader impact and societal objectives more visible to the research communities and to Congress. We need to provide policy makers with arguments they can use to justify STEM research funding increases.
2. US demographics are undergoing a marked change as racial and ethnic minorities comprise a larger fraction of the total population. The US Census Bureau predicts that, by 2050, Hispanics/Latinos are projected to comprise 24% of the total population and African-Americans will comprise 13%. Continuing the historical under-representation of members of these groups in STEM fields will decrease even further the talent pool from which STEM disciplines and industries can draw. This is critical to the establishment of a qualified STEM workforce at all levels.
3. One rationale for the involvement of STEM researchers with K-12 education is that U.S. superiority in STEM research has rested on the excellence of its higher education institutions; however, global comparisons show that the US lags behind other countries in STEM education at the K-12 level. Continued excellence of STEM research and education will be unsustainable if the scientific research community does not engage in significant partnerships with K-12 schools to improve their capacity to implement consistent, high-quality STEM instruction. Few K-12 school systems have the expertise or experience to do this solely from internal resources, and the STEM research community is unlikely to be pleased with outcomes not developed in collaboration with STEM researchers.
4. A science-literate public is critical to increasing community support for science funding, and for making well-reasoned decisions about STEM issues such as low-level radioactive waste dumps, global climate change, stem-cell research, and a host of other topics that will be of increasing importance. Emphasizing the rationale for the broader-impacts criterion will emphasize its importance to the sustainability of STEM research in the US.

3. How is BI used in reviewing proposals?

The short answer is: Unevenly, although there have been a lot of changes since the BI Criterion was implemented.

NSF has made several changes to encourage equitable usage of both criteria in the review process. In FY 2001, FastLane was modified to include two separate response boxes in which reviewers are required to address each criterion separately. Explicit instructions to reviewers request "detailed comments on the quality of this proposal with respect to each of the two NSF Merit Review Criteria". Effective October 1, 2002, NSF returns without review proposals that do not separately address both merit review criteria in the one-page Project Summary and reiterates that the broader impacts of the proposed project must be addressed in the Project Description and described as an integral part of the narrative.

An internal NSF task force found that, from a sample of FY01 reviews, approximately 69% of reviews provided evaluative comments in response to the broader-impacts criterion. In FY02, 84% of reviews contained information in both the intellectual merit and broader impacts text boxes, with this number rising to 90% in FY03; however, these numbers do not reflect the quality of information contained in the reviewer boxes or in the proposal.

A number of NSF Committee of Visitors (COV) reports find that, in addition to omitting entirely the broader-impacts criterion, some reviewers did not evaluate the broader impacts based on the stated program guidelines and some reviewers identified broader impacts on behalf of applicants who did not explicitly address this criterion in their proposals. They also found inconsistency between panel and mail reviews (with panel reviews providing more consistent evaluation of broader impacts than mail reviews) and an overall lack of understanding of the review criteria by reviewers.

Similar inconsistencies are reported in the use of both merit criteria by program directors in justifying funding decisions. In FY02, approximately 78% of program officer review analyses commented on aspects of both merit review criteria. A statistically determined sample of FY03 review analyses found that approximately 53% of review analyses contained comments on both merit review criteria. (The FY03 review differed from the FY02 review in that it distinguished between substantive reviews that comment on specific material in the proposal and summaries that contain only reviewer and/or panel comments, or contain generic or boilerplate analysis not specific to the proposal).

Why such problems? A few reasons have been identified. The NAPA report (James Colvard, Mary Jane Bostrom, and Sandra Hale, A Study of the National Science Foundation's Criteria for Project Selection, (National Academy of Public Administration, Washington, DC, 2001) points out that the extensive list of activities that qualify for broader impacts suggests a check off list in which each item must be addressed. COV reports also suggest that the breadth of possible activities makes it difficult for reviewers to consistently evaluate and compare the broader-impacts portions of proposals.10 The metrics for performance in the broader-impacts area are 'much less well defined than those for intellectual merit'.10 Some criteria listed under 'intellectual merit' (such as qualifications of the applicant in executing the project and access to resources) apply equally to broader impacts.