Demystifying the review process for NSERC Discovery Grants
July 19, 2012
This video describes the various steps of the peer review process for each application for an NSERC Discovery Grant.
As NSERC's largest program, Discovery Grants provide a major source of research funding for Canadian universities. Discovery Grants foster internationally competitive research and contribute to Canada's science and technology needs. They promote and maintain a diversified base of high-quality research capability in the natural sciences and engineering, and support a stimulating environment for research training. And, unlike other funding options, they subsidize ongoing research programs with long-term goals rather than short-term projects.
This video explains what happens during each step of processing and evaluating an application for a Discovery Grant.
Roughly 3,500 applications for Discovery Grants are submitted annually. To make sure all funded research meets the required standards of excellence, and that all applicants are treated fairly, funding decisions are based on an internationally recognized peer review process.
NSERC relies on expert scientists and engineers from academia, industry and government who volunteer their time to review applications. Every February, close to 400 researchers from Canada and around the world meet in Ottawa for Competition Week, where each grant application is reviewed. They form the membership of 12 Evaluation Groups, each covering a specific discipline.
Applications are assessed against the three Discovery Grants selection criteria:
These criteria and the specific information that should be included in Form 100 and Form 101 of your application are discussed in greater detail in another NSERC video, titled "Tips on applying for a Discovery Grant."
As an applicant, you should be aware that how well you prepare your application has a direct bearing on your chances of success.
The first step in applying for a Discovery Grant includes completing Form 180 – "Notification of Intent to Apply for a Discovery Grant."
On Form 180, you are requested to suggest which of the 12 Evaluation Groups is the most appropriate to review your application. Based on your selection, your application is assigned to a team at NSERC and supported by a program officer, who consults with the appropriate Evaluation Group members in order to assign it to the most suitable peer reviewers.
As well, you are asked to select up to five Research Topics – in decreasing order of relevance specific to the research you propose. These topics can be from any of the Evaluation Groups, but the first one should be from the Evaluation Group you selected.
If you choose a Research Topic by searching for key words, you should check its relevance to your selected Evaluation Group. It is important to pick Research Topics directly related to the scope of the work proposed in your upcoming application.
Information about an applicant's Research Topics, as well as a description of the proposed work, is important for several reasons.
First, it determines which Evaluation Group and which specific reviewers will be assigned to your application once NSERC receives it in November.
Every application is normally reviewed by five Evaluation Group members, each of them experts in their field. The selection of these members is based on the best possible match between reviewers' expertise and your proposed research. In making these decisions, NSERC takes into consideration any conflicts of interest and the language in which the application was written.
Second, a member with relevant expertise within the Evaluation Group identifies external referees. The selected referees are normally a combination of those you proposed on your Form 180 and other names suggested by the member. These referees provide additional input to the members that will assess the applications.
The referee reports can be useful in a variety of ways. As a reviewer, what I do is review the grant and come to my own conclusions about it and then I go read the referees' reports. When the referees agree with me, I have a good deal more confidence in my assessment of the grant. But often they'll point out things that I've missed because they are experts in that field and I'm not necessarily an expert in that field. So it's nice to have reviewers' reports to pick up on things that are perhaps significant within the field that aren't obvious to you. I think the external referees are also well placed to judge the significance of the research in the field and if they make comments to that effect, then that's very useful.
Finally, the Research Topics are used to identify applications that would benefit from a joint review involving more than one Evaluation Group. Increasingly, research on significant problems in science and engineering requires the combined knowledge, expertise and contributions of researchers from various fields. Where appropriate, an application may be reviewed by members from more than one Evaluation Group.
Once received at NSERC, applications are grouped together by research area and assigned to the appropriate reviewer.
Members receive copies of their assigned applications in December.
Before coming to Ottawa, all committee members work hard reading the proposal, thinking about the proposed research, also reading your samples of research contributions. That's very, very important.
Assessment of applications is a two-step process that separates the scientific evaluation of the application from the funding recommendation.
In the scientific evaluation step, reviewers discuss the strengths and weaknesses of each application based on their assessments. After discussion, each reviewer rates the application individually for each of the three criteria: scientific or engineering excellence of the researcher, merit of the proposal and contributions to the training of highly qualified personnel. Members might also assess the cost of the proposed research program, relative to the norm in the field.
For each selection criterion, an application is assigned a rating of exceptional, outstanding, very strong, strong, moderate or insufficient.
The second step of the process determines the funding allocated to applications. After all the applications have been evaluated, they are automatically assigned to quality categories, also referred to as "bins," based on the ratings they received. All applications of similar quality are grouped together in the same bins.
For example, applications that receive "exceptional" ratings in each of the three criteria are put into bin A. Bin B includes all applications with two exceptional ratings and one outstanding. Applications in Bin D could be composed of three outstandings or a similar combination. Applications in Bin J are those with three Strong ratings or equivalent. If an application is not meritorious in one criterion, it will not be funded – regardless of the ratings in the remaining two criteria.
The quality bin determines the amount of funding an application will receive. Applications assigned to bin A receive the highest possible funding. The allotted funding is reduced for each successive bin, within the scope of the available budget.
Funding levels also vary across Evaluation Groups, but all applications in the same bin within an Evaluation Group receive a similar grant amount. The final amounts may fluctuate slightly to reflect the reviewers' assessment of the cost of research
We receive more good proposals than there are funds available to fund. So it's not a question about whether a proposal is rejected because the science is insufficient in the proposal. What it is is it's a peer competition. So that means you're competing with other people in the community. And so what happens is we do run out of money before we get to the line of where the proposals would be insufficient to fund anyway.
Following the competition, the results are communicated to each applicant in the form of a notification of decision. In addition, each applicant is mailed a "message to applicant" that indicates the individual ratings assigned for each of the three criteria. The message may include additional comments reflecting the consensus opinion of the reviewing members. Applicants will also receive copies of the external referee reports used to evaluate their application, edited in accordance with privacy legislation.
Anyone looking for additional information about the peer review process should talk with a Research Grants Officer at their institution or contact NSERC. You can find out more about applying for Discovery Grants or other funding opportunities on the NSERC Web site.