The CDMRP Grant Review Process: Maximizing Success

Panel Members: Virgil Simons, MPA, The Prostate Net; Elizabeth Platz, ScD, MPH, Johns Hopkins Bloomberg School of Public Health; COL Paul Taylor (Ret), MPA, ZERO; Ganesh Raj, MD, PhD, UT Southwestern Medical Center at Dallas.

Virgil Simons: I think one of the things we want to look at here and as a guide in some ways to help you become better investigators is to focus on what the Institute of Medicine has promulgated in terms of the definition of patient centricity, where the patient is involved in every aspect of things from research, clinical practice, to�downstream after the�helping develop clinical trials. So we�ve got to think in some ways that the patient is not really just a T2 situ tumor you�re looking at; it can become an active part of your research program, an active part of what you do, and I think, as you see, as represented by all of us up here, patients have been a significant part of the DoD program.

Elizabeth Platz: Research isn�t just for the sake of research; research needs to be done because we�re trying to make a difference. There�s some clinical unmet need or a public health unmet need with respect to prostate cancer, the development of the disease�the care of men with the disease, and their wellbeing that we needed to actually incorporate as we think of our research ideas and we think about the research approach we want to use.

Paul Taylor: It�s extremely important�and�and I think, a very, very well-executed program that achieves results. I mean, I�m a�I�m a patient that has used both Abiraterone and Enzalutamide during the course of my treatment, and�and those two drugs came out of the PCRP pipeline. So, you know, I�m sitting up here today because of the work that has been done by the PCRP.

Ganesh Raj: I have been fortunate also to serve on the DoD Panels since 2008, and it�s been great�it�s been great because you learn what is a grant that works in DoD and what doesn�t.

Stress Impact and Innovation

Simons: Now, part of the thing as investigators and you�re writing a grant, you�re thinking about the Review Panel. What do you need to tell them that�s going to make your grant compelling?

Platz: Typically, when we write a grant application, we write for our�our peers, right, so fellow scientists who are within the same research area that we are in. But when�once you get out of Peer Review, and your application is with the Integration Panel members, you�re no longer with your niche area. And so you have to think carefully about those Panel members who will be judging the innovation that�of the work that you�re describing and, most importantly, the impact. And so what we want to do is encourage you to take the sections of the grant application that deal with impact and innovation seriously. Don�t�don�t write those parts at the last minute. In fact, it might be helpful to think about those upfront. Why do you really want to do this research? Why is it important? Why is it cutting edge? And why is it research that wouldn�t be funded by NIH? So really focus on the�the statement of impact and on the innovation and don�t think of those as just being perfunctory and don�t just be cursory. Don�t say, �Well I�m studying this pathway because maybe someday someone could develop a drug, if we know this pathway.� That�s not enough; you know, really think through what would happen if you have the finding that you think you�re going to have, if your hypothesis is true, and your experiments or your observational research�the findings happen to�to match the hypothesis, what would you do next? How would you actually move along your discovery to the point of making a difference?

Raj: At the Peer Review level, I think that those two things, the impact and innovation, are particularly emphasized. From what I�ve seen every single time, if�if you have something that�that is fascinating about some micro-RNA 37 or something else that is going to do something�that if you knock it out it does something in prostate cancer, well that�s nice and all, but how is that translated? You�ve got to have an impact. For the past couple of years, I�ve been on the NIH Review Panels as well; there, they love incremental change. You�ve done three-quarters of the proposed work you�re�you�re supposed to do, and you�re making�you�re saying I�m going to do this much more. Great. That kind of approach for a DoD grant, I think, fails. I think you have to have a big�a leap forward and, I think, the�even though you have to kind of make it supported by preliminary data, it�s got to be a leap forward, not just an incremental step forward. And I think that�s a critical thing that we look for in the grants.

The Consumer Perspective

Taylor: When your proposal is being reviewed by a panel, there are going to be three, you know, consumer reviewers on that�on that Panel, right? Guys just like me, public school education, 23 years in the Army, maybe not just like me, all right, but not doctors, not scientific folks, and we get an equal vote. And they look to us to provide a different perspective, and there were times when, you know, consumer reviewers turned the tide of a different proposal, and it ended up being weighted much�more heavily at the end because a consumer was very interested in the process and thought that it was very impactful for patients. Consumer reviewers like me only read, evaluate two portions: the lay abstract and the impact statement. So you have to write those things so that we can understand. If I can�t get through it or have a basic understanding of what you�re trying to do, maybe with the help of some of the scientific reviewers on�on the Panel, then I�m not going to be inclined to support it. It�s like anything, right? You know, you have to know how the process works, and you have to be able to put a product in front of the Panel that will be received by everyone, not just the scientists.

The Value of Synergy

Simons: The thing that really bothers us when we look at an application, and we can see two different people [Laughs] �have written this, and it�s been cut and paste. And it�s a case where, you know, one paragraph might be eloquent, and the other one, you know, has been written by the guy who didn�t do well in English class. So it�s a case where you can�t cut and paste. You got to be able to make us believe that there is synergy here; they�ve got to believe that there has been a progression of thought and of application and of work.

Platz: The best applications were the ones where you could see that there was truly interaction, even if they weren�t on the same campus, even if they were in�in different states.

Raj: You have to show that two teams are somewhat cohesive; they have that ability and need for each other. If I can do one thing, and Elizabeth can do something completely different, and the two of us together can�we can demonstrate that the combination of us together is better than each one of us individually, that is�that�s a synergistic win. Very few grants are like that. Most of them are additive at best. Some of them are less than additive�I�m going to do�I can do most of the stuff that Elizabeth does, and Elizabeth can do most of the stuff I can do, but we�re going to come together because we�re both kind of interested in the same thing and want to split�split up the�the pie. That doesn�t work. That�s not synergy.

Can it be done?

Taylor: One thing that, I think, speaks very highly of a�a proposal is its achievability, right? So there�s lots of proposals that come in with great ideas, but once you start actually taking a look at the proposals, you realize the amount of work that�s involved, and a researcher by himself or herself can�t do that. And so it becomes, I think, very important and�and the�the Panels take a hard look at the personnel that have been brought in as part of a team to be able to accomplish this because I can remember situations where we said this is�this is probably impactful work that they want to do, but it�s a 1,000 man-hours, and they have two people, right, one who�s full-time and one who�s part-time. You know, how are you going to be able to get all that work done?

Simons: One of my particular beefs and concerns: when it comes to the area of patient recruitment into the study. You know, we look at�you know, a grant that says we�re going to have a �in� of 100 patients. And, in the course of the discourse, they say, �Well, we will get the 100 patients.� All right; from where? [Laughs] You know, how are you going to get them? You know. Then, they�ll say well, we�ve got, you know, our Urology Department that will give us three patients a month. Okay; that�s how many years before you do it, and the term of the grant is only three years? So it�s a case of saying, at that point, you know, really looking at what you�re saying, what you�re talking about. If I�m going to be recruiting in a given patient population, if we look to say that we know that certain groups present with prostate cancer at a different rate. We know that Ashkenazi Jews present at a higher rate of prostate cancer. We know that African Americans; we know that other groups have a different presentation rate, incidence rate. So where is your resources that is going to help you get into those areas of patient influence? HPCU as a point of information is historically black colleges and universities. So, if you�re going to be doing a health disparities grant that targets African American or Latino populations, hopefully, you�ll know, hopefully you�ll know where to find them, number one, and how to build partnerships and alliances there as well.

Team Research Adds Value

Platz: Team research is now the norm. Team members can bring resources, and so what you want to do is learn how to build a team. So you have your own laboratory. You have your own population science group. But that usually in the modern world is not sufficient to accomplish everything that you may want to accomplish. And, so what you want to do, is find investigators from maybe other disciplines, other settings. So, for example, if you�re at a�at an institution that�s not a, you know, a historically black college or university or a university that has great diversity, maybe you do want to reach out and find a partner at such an institution because you will be exposed to other perspectives and potentially other, and I hate to the use the word resources, but meaning other kinds of participants, other types of patient populations. I think it�s really incredibly important to do. So how do you actually do that networking? Well one, you can reach out to people that you know are part of this DoD program. You can reach out to Virgil. You can reach out to Wes. You can even reach out to people like me, who over time have become networked. So we can help you network along the way.

If you are proposing to do so in one of your applications, make sure you get a letter from, say it�s another investigator who is not a co-PI with you or not an investigator with you, to�to document that they�re willing to share those resources or help you to recruit participants into your study. It�s not enough just to say, �Oh, I made this arrangement.� You have to have documentation, so make sure you get a letter. And, for example, working through the PCBN, you want to get a letter saying that, in theory, they would approve release of tissue to you or other bio-specimens, were you to be funded. So make sure you have documentation of that.

Raj: I think the way you got to think about a team, if you�re going for a big multi-investigator grant, is to kind of think of your team as�you�re�you�re the coach or the quarterback of a�of a major NFL team. You�re asking for a lot of money. You�re asking for $2 million over three years, or you�re asking for a significant amount of money. You can recruit whoever you want; bring the best you can. Don�t be afraid to get the best persons you can onto your team. Get the best wide receiver you can get. Get�get the best offensive line you can get. And I think that�s what makes a compelling story. The compelling part of it is to define the best team you can make and to create that team in a cohesive way that brings it forward.

Folakemi Odedina: I always think that there�s absolutely nobody that I can�t reach and I can�t touch. So you�re only an email away or you�re only a telephone call away. And the only thing they can say is say no to me. I think, all in all, you just have to not give up.

Raj: The first thing you do is look at whether the science makes sense. Are they proposing something that is doable? Are they proposing something that can be done by the team that they have? If the investigator is proposing to do some complicated proteomics experiment that he has no proteomics experience with, but�and doesn�t bring somebody in with that relevant experience, I�m not so convinced that guy can do that. Then the big question comes. What is the impact of this? What is the innovation? Is this something they�re proposing that is really cool? Is this is a new mouse model? Is this something different? Are they just saying they worked on micro-RNA 17 their entire lives and, today, they want to start working and do the exact same experiments with� and looking at micro-RNA 18? Not so interested in that. So it�s more interesting�is it truly innovative? Is it truly groundbreaking?

Have We Mentioned Innovation and Impact?

Platz: What excited me, while I was a member of the Integration Panel, is when research was truly new. It seemed like each year there would be a whole cluster of applications on exactly the same thing. Whatever was hot in the field, everybody would be on it, but then, every now and then, there�d be a different�an application addressing something completely different, and you could actually see the faces of the Integration Panel members all light up when someone would say, you know: �I�ve not actually seen this proposed before anywhere. [Emphasis Added] This is exciting.� So that, for me, that�that was�those were the best moments because, after a while, when you�re on the third, fourth, fifth application that�s on exactly the same thing, it can get a little boring.

Taylor: I�m a prostate cancer patient, right? I mean, the things that I�m excited about when I read a proposal are things that have near-term therapeutic value. You learn to balance that�that bias with, you know, some of the more, like we talked about, innovative research that�s being done that�s going to have a long-term benefit to the whole community. Clearly articulate, you know, what the game plan is for that if there�s not a near-term therapeutic value to the research that you�re doing. Because, from a consumer perspective, you know, we�we, at least I did, I zeroed right in on that. If we talked about that research and, oh by the way, there�s these three drug targets that are immediately available that could potentially be used as a byproduct of this research and the, you know, the�the bench to bedside time is going to be minimal.

Understanding the Two-Tiered Review Process

Platz: I think we should also point out that there�there�s actually a big difference between the Integration Panel and the�and the Peer Review Panel in what their�their goals are. The peer reviewers are responsible for reviewing that the science is doable, it�s feasible; that�s what proposed matches what the research questions are, the hypotheses; also judging the innovation and impact and so on. The Integration Panel, at least in the old days, didn�t even receive the whole application. We now have it, like available somewhere, we can go look at it, but that�s not our goal. Our goal isn�t to re-review the science. It�s to take what the peer reviewers have said about the science and about the impact and about the innovation and then for us to say, �Is it going to make a difference?� Right? So peer reviewers say it�s good stuff. And then we say, �Is it going to make a difference?� And then we say, �Is it going to make a difference within the context of the portfolio of research already being funded by the DoD?� Okay, so if there�s already something on that same area, maybe we don�t need another project in that area. But it�s also to make sure, within a funding year, that there�s a diversity of research types, so not just what�s already in the portfolio, but we wouldn�t want to be recommended for funding research that�s all on imaging in one year. We have to make sure there�s the breadth so that we can move the entire field of prostate cancer research forward.

Fangliang Zhang: As a new PI [inaudible] researcher, if you have one DoD grant and, at that stage, you perform it excellently, I mean a new one, would the performance in the previous one, would it matter in your next one?

Platz: So, unlike the competing renewal applications for NIH, where there�s a progress report, and you would talk about exactly what you did and while�why this next increment is needed, DoD�this program doesn�t work that way. However, in the mechanisms that come out each year, some�actually most of them require preliminary data. And preliminary data could be the research that you performed previously under a DoD award. And so you would talk about that. Now, not at the Integration Panel level or the Peer Review level, but the actual program managers, they know if you have completed your tasks from�from the prior round, and they�ll know if you are a researcher who merits continuing funding. To�I don�t know�do you want to speak to that a little bit or not?

Melissa Cunningham: There is nothing in our program announcements, unless it�s specifically stated, that says that any future funding opportunities are at all dependent on what you�ve done before. Every single proposal is evaluated as a brand new idea, as a brand new proposal; that�s why there�s no ties between things. That doesn�t mean that, if we know that we need to watch you, that we�re not going to watch you like a hawk. [Laughs] But we don�t hold any issues because every�every idea is a new idea. Every day is a new day, and that could be the one idea that makes a difference. So, unless it�s specifically stated that that�s a�a condition, then it�s�it�s not a condition.

Look at the Big Picture

Natasha Kyprianou: So this is particularly important for this forum because we�re dealing with the young investigators participating and being invited to this IMPACT Meeting. So productivity is extremely important to show that it�s not just coming with the idea but so, I guess, when, not only the Integration Panel, but the primary scientific reviews, when they look at the, you know, the background of productivity and the intellectual property that the young investigators generated, that is going to be considered as a positive, you know, forward, because this is how they advance their science. That�s how they advance their research. It could be taking an idea from a pre-clinical point of view to the clinical setting, towards the clinical trial or a new biomarker. So I think, looking at the overall picture, this is taken into consideration when the Integration Panel reviews, you know, in terms of the success of their early you know DoD-funded grants.

Timothy McDonnell: I think it�s important for young investigators, if they�re considering a problem, for example, they�re laboratory mechanism-based types of scientists, and they�re trying to develop a clinical context or a clinical application for something, that that expertise really needs to be represented in a meaningful fashion on the proposal. And, conversely, it�s sometimes seen that clinicians in clinical patient care environments who really don�t know one end of a pipette from another that are proposing detailed mechanistic studies, they�they need to have that expertise represented on the proposal. So I think that�that speaks directly to feasibility and the ability to translate findings.

Joel Nowak: One of the issues that come up at least three or four times during each review session is when aims don�t work together; for instance, if you have four or five aims in your proposal, and they�re each reliant upon the prior one, and aim one doesn�t work, what happens with two, three, four, and five? Now it may have taken you $100,000 to resolve aim one, but you have another $400,000. So we want to know that that money is not being wasted. So I think it�s important to think about what happens when your aims don�t work and how it impacts other aims, because we certainly do. I think this is something that I think trips a lot of proposals. We don�t want to find that we�ve given $100,000 to something that�s worth $10,000.

Closing Thoughts

William Dahut: So it�s�it�s been alluded to, but it�s actually much more important that folks realize that you make sure that the way your grant is written is appropriate for the funding mechanism you�re looking at. Because we often look at ideas that are really good ideas, but they�re in the wrong funding mechanism. So, if you have a highly innovative grant, but your�your mechanism, that�s all about short-term, medium impact, you�re not going to get funded. And often times we�ll say, �I wish they had applied for this other funding mechanism.� So, you can get thrown out very quickly and triaged early in the process if the way your funding mech�your grant�s design is not appropriate for what you�re looking at.

Don Tindall: If you�re going to do something innovative in the very beginning, it will probably not be very impactful because you�re�you�re starting at the very beginning. You�re amassing data that shows that it has potential for impact. But you�re a long ways from getting to the impact. So I would say, within the span of a grant proposal, you may not be able to achieve from A to Z, but the critical thing is to lay out a vision that you have data to support the vision, that you have the resources and the expertise to move forward with that vision, and you�ve got a plan that will eventually make this vision impactful.