This is part of NSERC’s ongoing commitment to evaluate and improve its processes. The information will help NSERC find better ways to compare overall levels of excellence across disciplines, according to international best practices. These and other considerations, such as changing demographics and differences in cost structure across disciplines, will help determine appropriate budget allocations between the various Discovery Grants Evaluation Groups in the future. New methods to allocate funding across disciplines would make the system more objective and better able to respond to the ongoing evolution of the research environment.

The CCA is expected to name the members of the expert panel during the first quarter of 2011. NSERC encourages members of the research community to respond to any CCA requests to participate in the assessment. A report is expected in 2012. At that point, NSERC would develop the methodology in consultation with key stakeholders The earliest implementation of any resulting changes to budget allocations would happen for the 2013 Discovery Grants competition.

The CCA will post information periodically about the  status and process of the assessment.

FAQs

What specific question will this assessment address?

The assessment will address one overarching question, divided into three sub-questions:

What do the scientific evidence and the approaches used by other funding agencies globally have to offer, in terms of performance indicators and related best practices in the context of research in the natural sciences and engineering, carried out at universities, colleges, and polytechnics?

1. What existing qualitative and quantitative indicators and metrics are relevant to budget allocation in the context of support for research in the natural sciences and engineering, and how can they be categorized (e.g., “shelf life”; cross-disciplinary and international comparability; relevance to interdisciplinary vs. focused disciplinary areas; and applicability to emerging vs. established research areas)?
   
   In this context, quantitative indicators include researcher population dynamics and cost-based indices, while qualitative indicators include collective quality/excellence of research and areas of strength indices.
   
   
2. What are international best practices in the construction, methodological review, and use of quantitative and qualitative indicators for research evaluation and budget allocation in support of research in the natural sciences and engineering?
   
   
3. Considering the foregoing, and in light of the Government of Canada’s Science and Technology Strategy and NSERC’s objectives for the support of research, what key considerations (e.g., risks, advantages/disadvantages, behavioural and institutional consequences) and principles emerge in determining defensible use and balance/weighting of performance indicators/metrics for budget allocation?

What is the rationale for this assessment?

• For the Discovery Grants (DG) Program, the budget allocation between disciplines is weighted toward demand-driven historical precedents; it should better account for other factors in the evolution of the research environment, such as the changing demographics of the disciplines, differences in cost structure across disciplines and areas of strength and opportunity, by identifying excellence factors related to areas of research.
• The challenge for NSERC, and other research funding agencies, lies in identifying appropriate and robust indicators, as well as available metrics, to measure and compare population dynamics, cost of research and research performance across disciplines.
• There is a knowledge gap—the scholarly literature and practices elsewhere offer no single methodology applicable to the Canadian context and to the DG Program.

How might this assessment affect the amounts of individual or team grants?

• The process for peer review of grant applications is outside the scope of this assessment.

Why is the assessment needed now?

• The timing for this assessment follows two reviews of the DG Program and the termination of the budget re-allocation mechanism used previously.
• This assessment addresses a concrete and timely need. NSERC requires a new mechanism to systematically review and update the allocation of funds between 12 discipline clusters.

How will this assessment help fix the problem?

• In terms of impact, a CCA assessment of performance indicators for basic research will identify metrics and assess their relevance to enable comparisons across disciplines and areas of research.
• These metrics, once validated, will improve NSERC's allocation of $350 million per year in research support, and make the program more responsive to an evolving environment (a federal S&T Strategy commitment).
• NSERC will use the assessment to inform decisions on the methodology to allocate funds across areas of research in support of science and engineering.
• The assessment is intended to enable policy development, rather than prescribe policy. In other words, the expert panel will provide lists of possible indicators and metrics, not the methodology for their application to budget allocation. This will be developed by NSERC in consultation with key stakeholders.

Why was the CCA asked to conduct this assessment?

• A Canadian assessment is required to inform Canadian policy.
• NSERC could not conduct this analysis for reasons of capacity.
• A CCA assessment is sought because it is critical that NSERC’s methodology is informed through a transparent process (e.g., publicly available assessment report) and based on principles and data that have been validated through scientific input and an arms-length consultation of the research community.
• Confidence that indicators are valid and robust is required in the light of the risks of creating unintended distortions in the allocation process.
• A CCA assessment will enable NSERC to build on available scientific evidence that is validated through scientific input from across natural sciences and engineering disciplines.

How is this assessment science-based?

• It will be informed by the scientific literature; in particular, scholarly research evaluation literature.
• It will require expert opinion from the scientific community with knowledge of practices in the conduct of science and engineering research and the evaluation of science and engineering research across disciplines.
• It will address current knowledge gaps in relation to:
  • applying the existing knowledge to the Canadian context;
  • the validity and appropriate uses of a number of existing S&T metrics.

What is this assessment expected to deliver?

• A catalogue/library of quantitative and qualitative indicators to measure and compare needs/performance across areas of research and an assessment of their relevance, applicability and “shelf-life” for use in budget allocation.
• A contextual piece on best practices in research evaluation and budget allocation with overall guiding principles and criteria for the selection of indicators, as well as an evaluation of a set of practices and indicators.
• In light of current knowledge and experience and of NSERC's objectives for support for basic research, a discussion of the key considerations (risks, advantages/disadvantages, behavioural and institutional consequences) and principles that emerge in determining defensible use and balance/weighting of performance indicators/metrics in the context of a budget allocation process.

How will the research community participate in this assessment?

• The CCA will determine when and how the research community will be consulted for this assessment. Information about the status and process of the assessment will be posted periodically on the CCA Web site.

Why does the question refer to colleges and polytechnics when the Discovery Grants Program is for university researchers only?

• Posing a broader question draws from a potentially wider set of approaches and allows for the possibility of applying the findings to other NSERC programs, including funding that involves colleges and polytechnics.

Links

• Update on the new peer review structure for NSERC Discovery Grants (February 2009)
• NSERC publishes results of the Grant Selection Committee (GSC) Structure Review (June 2008)
• Recommendations to NSERC - Grant Selection Committee Structure Review Advisory Committee
• Next steps - Grant Selection Committee (GSC) Structure Review (May 2008)
• Progress Report (February 2008)
• Goals
• Advisory Committee Terms of Reference
• GSC Structure Review Advisory Committee Membership
• Presentation to Societies

Contact

Isabelle Blain,
Vice-President, Research Grants & Scholarships, NSERC
E-mail: isabelle.blain@nserc-crsng.gc.ca
Tel.: 613-995-5833

Message from the President (May 2008)


Appendices

• A Portrait of Canadian R&D
• A Review of Canadian Publications and Impact in the Natural Sciences and Engineering, 1996 to 2005
• Briefing Notes
• Description of NSERC and its programs. People. Discovery. Innovation.
• Detailed Interviews: International Review of the Discovery Grants Program
• International Review of Discovery Grants Program Researcher Survey
• NSERC’s Discovery Grants Program
• Summary of Feedback provided by non-Academic Organizations and Individuals
• Summary of Interviews conducted with Discovery Grants Selection Committee Members
• Management Response to the International Review of the Discovery Grants Program

Contact

Isabelle Blain,
Vice-President, Research Grants and Scholarships, NSERC
E-mail: isabelle.blain@nserc-crsng.gc.ca
Tel.: 613-995-5833