Program Committees and Paper Selection

In this post, I’ll discuss paper selection—how a program committee considers a set of papers, each with a collection of reviews, and produces a program for a conference.

Paper Reviewing vs. Paper Selection

It’s first important to note that the paper review process and the paper selection process are related, but paper reviewing and paper selection are actually two separate events.  There is a human process behind paper selection that involves distilling the reviews of a paper and ultimately making a determination that a paper should be published, but the reviewing and selection have many independent aspects.  Sometimes, a paper’s reviews may appear positive, but the paper may still be ultimately rejected (or, the reviews may appear negative but the paper is accepted in spite of the reviews). More commonly, papers tend towards the middle of the review distribution: when reviewers score papers, they tend to regress towards middling scores such as “weak reject” and “weak accept”, rather than taking a strong stance.  A significant fraction of papers thus end up with almost the same average “scores”, with only a handful of papers that should clearly be accepted.  The number of clear accept papers is never enough to fill the entire program.  Suppose, for the sake of example, that a conference can accept 40 papers.  The conference may receive several hundred submissions; in most cases I’ve observed, about ten of these submissions receive uniformly glowing reviews.  The next best 30–40 papers often fall into a rough equivalence class with the next best 30–40 papers, leaving anywhere from 60–80 papers that are “good enough to be published” for about 30 spots in a conference program.  The job of a program committee is to create a smaller program from this larger set of papers.

Whenever a larger number of roughly equivalent candidates competes for a smaller number of positions, some amount of randomness can and will ensue.  Much of this randomness is completely beyond the author’s control.  As an author, the best way to avoid being subjected to the randomness of a program committee is to write your paper so that it is among the ten best submissions to the conference.  Unfortunately, this is far easier said than done, and it’s not really possible to engineer this, although following various research and writing tips can increase the likelihood that your paper ends up in this class. (Speaking from experience, I can say that those tips aren’t fail-proof—even the best researchers I know routinely have papers rejected—but you can at least improve your odds!)

Unfortunately, unless you can guarantee that your paper always falls among the very best papers (a secret that nobody has yet succeeded in unlocking), much of the randomness that results from the paper selection process is out of your control as an author, but other factors can help create a sane selection process.  The program committee chair (or chairs) have tremendous power for creating a sane process.  These roles include setting the right tone for reviewing and mindset for paper selection, ensuring that papers are reviewed by the right program committee members, selecting the papers that will be discussed at the program committee meeting, and reducing the effects of program committee psychology, such as paper discussion order (e.g., the weight of one biased or incorrect review, and so forth).  I will describe more details and tips for program committee chairs below, based on my own experiences and observations.

The Intermediate Dispositions of Papers

Before the program committee meeting itself takes place, the program committee chairs must determine whether each paper should be accepted or rejected without any discussion, or whether the paper should be discussed at the meeting itself.

Before the meeting: Selecting papers for discussion.  Program committee chairs typically select papers for discussion at the program committee meeting based on their assessment of the paper’s overall quality, based on review scores, their interpretation of the paper’s reviews, and their assessment of any subsequent “online discussion” that may take place before the physical program committee meeting (most conference reviewing systems allow reviewers to discuss each other reviews in the system itself via asynchronous, email-like messages).  Program committee chairs will often try to ask reviewers to reach consensus about whether a paper should be discussed at the program committee meeting before the meeting.

Best case: Acceptance without discussion.  If you are very lucky, your paper will not be discussed at the program committee meeting and will be accepted without discussion based on high review scores.  I think this is the best possible outcome, because a paper always has flaws, and the more a paper is discussed, the more likely that someone at the meeting will hear something they don’t like about the paper and raise an issue with it.  I have seen instances where a paper that receives uniformly high scores is rejected because it is brought up for discussion and someone (often who hasn’t yet read the paper but might read it during the meeting) hears something they don’t like.  Such a discussion doesn’t mean that your paper will be rejected—a good PC chair will ensure that these dynamics don’t wrongly result in the unjust rejection of an otherwise good paper—but it does mean that your paper may ultimately be subjected to some of the dynamics I discuss below.  Unfortunately, as I mentioned, you can’t guarantee that your paper falls into this category, but you can sometimes be lucky, and you can certainly take many steps to increase your odds.

Next best case: Discussion.  Otherwise, if you are still reasonably lucky, your paper will be discussed at the program committee meeting.  Discussion represents hope for acceptance, and typically because there are a large number of papers that are of roughly equal score or rank, if your paper is being discussed, it generally has as good of a chance or being accepted as most of the other papers being discussed.  The program committee chair will set the order in which these papers are discussed, which can sometimes play a surprisingly significant role in whether a paper is accepted or rejected.

The Dynamics of the Program Committee Meeting

The program committee meeting itself involves the winnowing of a larger set of papers (say, 60–80 papers) down to a final program of, say, 30–40 papers.  Typical math I’ve seen in systems and networking conferences is that while overall acceptance rates hover between 10–20%, about 50% of the papers that are discussed will be accepted.

Attendance.  An in-person meeting is more effective than other alternatives (e.g., conference calls), if and when it is feasible.  Most top-ranked conferences—and even some competitive workshops—have in-person program committee meetings.  When program committee chairs select the members of the program committee, they typically confirm that the program committee member can attend the program committee meeting; and, if the meeting is an in-person meeting, they confirm that the PC member can attend in person.  If all PC members are not present at the meeting, it is nearly impossible to have anything close to a fair paper selection process.  Consider that a paper under discussion might have anywhere from 3–6 reviews, and that most of those reviews might hover around average scores, but one or two reviews may be outliers.  The fate of a paper can shift dramatically if the PC member who feels strongly (either positively or negatively) about a paper is not in attendance.  Without a champion present, the paper may simply fade into the mix; if a reviewer who raises major concerns about a paper is not present, the rest of the committee may be more likely to discount those concerns, since they cannot hear about them first-hand.  Some of these attendance-related quirks can also materialize when program committee members depart early (e.g., to catch a plane flight).

Calibration. Every reviewer has a different opinion for what constitutes a reasonable threshold for acceptance.  These differences in calibration are exacerbated by the fact that every program committee member has a unique set of papers to review.  That is, while each paper has many common reviewers, no two PC members review exactly the same set of papers.  Therefore, calibrating the PC is incredibly important for controlling the meetings dynamics.  A poorly calibrated PC may pre-emptively reject good papers (as I mentioned above), only to accept weaker papers later on in the meeting.  A good PC chair will spend considerable effort to calibrate the PC to reduce randomness.  There are various approaches to calibration, which I will discuss below in my advice for PC chairs.  One of them is setting an appropriate order for paper discussion, since the order in which papers are discussed will affect how the PC calibrates itself at various points in the meeting.

Discussion order.  There is no accepted standard for setting the order in which papers are discussed, and every program committee chair seems to have a slightly different approach.  That said, discussion order has a significant effect on the ultimate disposition of a large number of papers that sit on the borderline between acceptance and rejection.  Perhaps the most striking example of ordering effects is evident when a PC discusses papers in decreasing order of score (i.e., from highest ranked to lowest ranked), which is, in my opinion, the worst possible ordering.  People arrive at a meeting highly energized and eager for discussion.  What often happens is that papers that are discussed in the morning are subjected to a fine-toothed comb and vigorous discussion—sometimes the result of these discussions can result in pre-mature rejection of a good paper.  Later in the day, PC members become more tired and are less likely to pick apart a paper’s flaws or have a paper accepted only “over his or her dead body”.  Similarly, excessive aggression in the early parts of the meeting can create a situation where, by the end of the day, there are not enough papers to fill the conference program (quite an irony, given the starting point of twice as many acceptable papers as slots!)  This seems to create a dynamic where papers that are discussed later in the day sometimes have a better chance of being accepted.  If papers are being discussed strictly in order from best to worst, there is the potential for “inversion”, where better papers are rejected early on in the meeting, making room for weaker papers to be accepted later on, as PC members acquiesce and PC chairs are frantically trying to fill the program.  PC chairs can take various steps to reduce this randomness.  One tweak I have seen (and used) is to simply treat all papers under discussion as roughly equivalent, banning all discussion or consideration “scores”.  In these cases, the discussion order can be set such that papers on related topics are discussed together.  Another that I have used is to employ a two-pass approach to discussion, where every paper that is not accepted immediately or upon initial discussion is discussed a second time—in other words, a paper is never rejected on the first pass.  A third tweak that is sometimes used is to discuss a few “highly ranked” papers first (sometimes even the papers that are supposed to be “accepted without discussion” are used as quick calibration examples), followed by a few “low ranked” papers, and so forth.

Timing of discussion.  A PC meeting does not leave time for extended consideration of any single paper.  If there are 80 papers to discuss, and a PC meeting has eight hours of real work (discounting breaks, lunch, and so forth), then each paper sees, on average, 6 minutes of attention. Thus, discussion must be crisp and focused.  A “discussion lead” for a paper (typically one of the reviewers) will typically summarize the paper and the strengths and weaknesses, effectively summarizing all of the reviews.  This summary should last no more than 90 seconds.  This is not a lot of time.  Therefore, a good PC member who is leading discussion on the paper will prepare that summary ahead of time, so that it is as efficient as possible.  The paper’s ultimate fate is typically decided in the remaining time (less than five minutes).  Many disputes and disagreements about certain papers cannot be resolved in five minutes.  This truth is the most common downfall of a PC meeting—a single contentious paper runs the risk of monopolizing discussion time, leaving less time for the remaining papers, thereby increasing randomness, acquiescence, etc. for papers that are discussed towards the end of the day.  I find that using the “two-pass approach” for paper discussion helps mitigate this problem.  I’ll discuss this approach in more detail below.

Personalities.  A paper that has a single champion tends to fare much better than a paper with middling reviews, even if both papers have the same average score.  In some sense, that is exactly the right outcome—a paper that someone really likes is likely to be appreciated by others, whereas an average paper might not excite anyone in particular.  However, strong personalities do hold tremendous sway over a paper’s outcome, as well.  A paper stands a very good chance if it has a champion on the PC who is articulate, confident, and strong-willed enough to persistently argue for a paper’s acceptance in spite of the paper’s flaws and detractors.  Every paper has flaws; a strong-willed PC member can convince the rest of the PC to overlook those flaws in favor of other bright spots.  Likewise, a single similarly strong-willed PC member can amplify the weaknesses of a paper and send the paper to the reject pile.  These effects are exacerbated if the strong-willed PC member asserts expertise over the topic area.  Sometimes, a PC member’s expertise is as important (if not more) important as the volume or persistence of the PC member’s argument.  Simple statements from an domain expert such as “I learned a lot by reading this paper” can be enough to tilt a paper towards acceptance.

Psychology.  The paper selection process is a human process that involves a lot of psychology.  Rather than recount the various psychological factors that can play a role in program committee meetings, I refer you to Matt Welsh’s blog post on the topic.

Tips for Program Committee Sanity

Everything I learned about how to run a program committee, I learned from Jeff Mogul, who co-chaired NSDI this past year with me.  Jeff is a veteran program committee chair who has chaired pretty much every major systems and networking conference at some point in his career.  He is meticulous, thorough, ethical, and fair, which are indispensable qualities for any program committee chair.  Working through the process with him, I picked up several tactics and strategies, which I can recommend to others and will undoubtedly use if and when I chair another major conference program committee.  Being a program committee chair has countless tasks; some of these tasks are incredibly important, and the ones that are paramount weren’t immediately obvious to me until after I’d gone through the whole process.  Below, I highlight tips for what I think are the most important steps of running a program committee.

  • Pick your program committee carefully.  Your program committee members need to be thoughtful, reliable, and conscientious.  Ultimately, the program committee will write the reviews that authors of submitted papers see, and they will also determine the fate of each submitted paper.  Don’t simply pick your friends or people you know well—they may not always make the best reviewers. Take extra time to do homework on program committee members’ performance on past committees.  Did they write thorough reviews?  Were they active participants in discussion at the PC meeting?  Did they turn in reviews on time (or, if not, were they communicative enough to help the chairs plan around timing hiccups?).  Are they considered an expert in a particular area that will see a lot of submitted papers for the conference?  Beyond selecting individuals, the chairs also need to ensure that every area that may see submitted papers has significant coverage.  For example, if the conference will see many submissions on (say) wireless networking, it is incumbent on the chairs to ensure that there are enough reviewers for that particular paper topic.
  • Make sure that every program committee member “bids” on papers, and assign reviews manually based on preferences and expertise.  Most conference reviewing systems allow program committee members to review all submitted papers and express preferences for which papers they would like to review.  Some conference systems also enable chairs to automatically assign reviews. Do not rely exclusively on auto-assignment.  When Jeff and I chaired NSDI, we used the auto-assignment feature in HotCRP (after ensuring every PC member filled in review preferences for each submitted paper) and then manually reviewed each paper to ensure that no reviewer was assigned to review a paper for which they had expressed a negative preference.  This process is painstaking, but it is perhaps the most important step in the entire paper selection process, because it ensures that reviewers are assigned papers that they are capable of and willing to review.  A reviewer who knows a paper topic can typically write a thoughtful review (and often do so quickly).  In contrast, a PC member who reviews a paper for which they lack expertise or enthusiasm will typically invest only minimal effort and write a superficial review (thereby increasing randomness and likely reducing overall conference quality).
  • Insist on (and monitor) online discussion in advance of the meeting.  Selecting discussion leads and asking the lead to type a summary in the online discussion helps organization; the discussion lead can essentially read this typed summary (or notes) at the meeting itself, ensuring that the summary discussion concludes quickly and on time.  Online discussion can also encourage reviewers to identify contentious issues before the meeting (where there is limited time to resolve disputes or disagreements), and potentially resolve them ahead of time, thereby averting protracted and unfocused arguments at the PC meeting itself.  Sometimes, consensus on a paper can be reached in online discussion before the meeting occurs, saving precious time at the meeting itself.
  • Use a two-pass approach to paper discussion.  Every other program committee I’ve served on tries to reach a conclusion about a paper’s disposition after a (short) discussion.  In the best case, this works OK; in the average case, hasty and incorrect decisions can result; in the worst case, the discussion monopolizes meeting time, and the meeting is derailed and ends in a frantic rush to accept papers at the end of the meeting.  The two pass approach we used worked as follows: (1) in the first pass, a discussion lead would summarize the paper and its reviews; in the remaining 3–4 minutes, the reviewers would try to agree on an outcome for the paper, but none of the outcomes were reject.  Rather, possible outcomes were: accept, incremental/boring (indicating that the paper was technically correct, but not particularly interesting or groundbreaking), risky (indicating that the paper could be groundbreaking but that reviewers had concerns about correctness or something else), and discuss (indicating that there was absolutely no agreement on the paper, and that more time would be needed to reach an outcome).  (2) In the second pass, our original intent was to re-discuss everything, but effectively what happened was that all papers labeled as risky were accepted by default unless someone wanted to argue against one of them.  Similarly, all papers that were labeled as incremental were rejected by default unless someone wanted to advocate for one of them.  (We had papers in each of these categories.) Pulling reject off the table on the first pass kept the meeting going (we could reasonably end discussion after a fixed amount of time because people knew we could come back to discussion later), and it also reduced the “ordering effects” that I described above, since every paper that wasn’t quickly accepted received two passes, with the second pass occurring after the PC had seen the complete set of papers.
  • Read every paper that is being discussed.  Although it is not possible to carefully read 80 papers in advance of a meeting, two PC chairs can split the workload and at least perform a quick read of about 40 papers (and take notes on them) in advance of the meeting.  Jeff and I did this, and it proved to be incredibly useful, for several reasons.  Having your own opinion about a paper as the chair can help a chair to moderate the PC meeting discussion, by ensuring that a strong personality (or, sometimes, a PC member who has not even read the paper!) from unfairly swaying the discussion or perception of the paper.  In the end game, sometimes PC members will never come to agreement about whether a paper should be accepted or rejected.  In these cases, the chair becomes the “tie breaker” and can accept or reject a paper by fiat.  This typically happens at least once in every PC meeting.  A PC chair who has not read a paper cannot exercise fiat effectively, so having read every paper is critical in these situations.

The Worst Process, Except for Every Other Process

The program committee process for paper selection is far from perfect; it is an inherently human process, and in the final analysis, a small handful of gatekeepers (indeed, sometimes even a single gatekeeper) can determine whether a paper is accepted to a top-tier conference and read by many others, or shelved.  I personally would like to see my own community have a long discussion about (and experiment with) ways to improve this process.  In the meanwhile, I trust that this post can help authors (particularly students) understand some of the dynamics of the process and also help chairs and PC members ensure that the process is as fair as possible.  Although the process is far from perfect—and it is unlikely that a perfect process exists—understanding the mechanics that go on “behind the closed doors” of a program committee meeting hopefully also indicate that the outcome of the process (either acceptance or rejection) should not be interpreted as universal praise or condemnation, but rather the result of the opinions of a small number of people and the outcome of a human process.  Ultimately, the proof of a research idea lies not in the outcome of a single program committee, but in an ideas ultimate acceptance, adoption, and impact (which is, in itself, a topic for a future post).

Cultivating Your Research Taste

Just as each of us develops taste in books, music, art, and food, every researcher ultimately develops a taste for research problems.  Every researcher should spend some time developing “good taste” in research problems.   The world has many challenging problems to work on, and as researchers, we have limited time and bandwidth.  It’s therefore important to develop (good) taste in selecting problems, so that we end up working on the problems that are worth a significant investment of time and energy.    Many research problems will take years to run their course, so it is worth spending some time developing taste in problems.

Cultivating taste requires aggressive sampling, and both good and bad experiences.

Many professions, ranging from designers to architects to programmers to managers, need to develop good taste.  In this post, we’ll focus mainly on developing taste in research problems, although some of these tips for cultivating taste may apply more generally (in fact, some of the pointers below were inspired from an article I recently read about developing taste in design).

Cultivating your own research taste.  We are not born with good (or bad) taste; rather, we develop taste by way of education and exposure to many different opportunities and experiences.  Just as one might cultivate taste in other areas of life, one must cultivate taste in research.  In this post, I’ll offer some tips for cultivating research taste that I’ve found work for me.  Some of these tips I have discovered (and applied) by way of analogy for developing taste in other realms (e.g., music, food).  I’ve spent a fair bit of effort developing taste in music; where applicable, I’ll draw some analogies below.

  • Seek out others with good taste.  Perhaps the most important step for developing good taste is to associate and learn from others who have good taste.  These people are generally acknowledged as “having good taste” and are often easy to identify.  Just as you may have friends who you know are well-read about music, wine, or food (you know these people because you’re always asking them for recommendations), the research community has “thought leaders” who are widely acknowledged as having good taste in research problems.  It’s not too hard to figure out who these people are with a little research.  Ask several colleagues who these people are and see whether trends begin to emerge.  Poke around on Google Scholar and see which researchers in your area have highly cited articles on a particular topic.  Intersect these people with those who share your interests, as well.  Once you have identified others with common interests who appear to have good taste in problems, try very hard to associate, exchange ideas with, and work with those people.  Become an apprentice.  As a Ph.D. student, seek these researchers out as possible advisors.  The Ph.D. years are perhaps the most formative for developing research taste, and your taste will likely be shaped heavily by your advisor, so taking the time to find an advisor who will develop your research taste is perhaps one of the most important decisions you will make as a Ph.D. student.
  • Read trend-setting conference proceedings, and develop opinions about research problems and trends in your area. The “top tier” conferences in your area are essentially the Architectural Digest, Wine Enthusiast, or Pitchfork for your discipline.  Track your conference proceedings to determine the research areas that the best researchers in your area are working on.  You don’t necessarily have to “jump on the bandwagon” and start working on any of the research areas that are the current hot topics at this year’s conference—just as you might not spend several hundred dollars on the latest wine that’s reviewed or go right out and buy every new music release—but it certainly doesn’t hurt to learn about the latest trends, even if you don’t always resonate with them.  Exposing yourself to the latest trends and developing opinions about them (positive or negative) is an important step in cultivating research taste.  It’s typically not necessary to read the entire conference proceedings to get a feel for what’s going on in an area; simply looking at the names of sessions and groupings of papers can help you quickly identify areas that are receiving a fair amount of attention from the community.
  • Sample and experiment with abandon.  Developing taste in any genre involves gaining exposure to many different examples, good and bad.  Just as it’s much easier to appreciate a truly fine wine, dish, or performance after having seen mediocre offerings, since every experience allows us to better articulate what we do or don’t like, sampling a wide variety of research problems (and solutions) is a necessary step for developing taste in research.  In developing taste in music, I find myself reading continually about new artists and albums and listening to new material that pushes boundaries in new ways, and sometimes subjecting myself to music that in the end I might decide I don’t like all that well.  Similarly, in research, we must continually experiment and sample to develop and cultivate our taste.  It can be tempting (and certainly easier) to “turn the crank” on problems that we know how to solve, but ultimately this will result in research that becomes stale and boring.  As researchers, we should be continually learning about new techniques, tools, problems, approaches, and so forth.  We will likely find that some topics, areas, and solutions we encounter are incredibly boring, but those “bad samples” ultimately help us determine what aspects of a research problem or solution we do or don’t like.
  • Keep a list of ideas that you like and exchange your favorite ideas with colleagues.  Keeping lists of things you like is good; exchanging them is even better.  Every year, I tabulate a list of my “top 10” music albums for the year; at the end of the year, my friends and I exchange lists.  The list is fine, but making the list is actually more important.  Knowing that I am going to be exchanging a list of music with friends at the end of the year keeps me accountable and ensures that I am always “on the lookout” for new gems.  Similarly, making lists of neat research ideas and regularly exchanging them with colleagues is another way to ensure that you always have a healthy appetite for new, creative ideas.  I don’t exchange lists of research ideas, although perhaps I should; I do, however, regularly exchange papers, articles, and ideas with a small, trusted group of colleagues, often multiple times a week.
  • Attend conferences.  Just as a music enthusiast attends live concerts or a book enthusiast might attend a reading, as a researcher you should attend research conferences.  I think it is worthwhile to attend at least one major research conference in your area every year.  Attending a major research conference ensures that you are staying current with research trends (in case you’d like to “watch the movie”—that is, attend the talks—instead of simply poring over the conference proceedings).  Perhaps more important than attending the talks, however, is meeting other researchers.  Conferences are excellent places to seek out others with good taste, and to have conversations with other researchers that can help you develop opinions about various research problems and areas.  These conversations and interactions are all part of the process of developing taste as a researcher.
  • Consider the principles and theories that underpin a specific research paper or project.  No research paper (or project) is perfect.  In fact, most papers are flawed, and many are badly flawed.  Yet, papers are often published not for the particular artifact that they produce, but for some underlying concept or idea that they embody or espouse.   When reading research papers, it is far too easy to be dismissive of a paper because of a flaky implementation, a bogus evaluation, or poor exposition.  Try to dig a bit deeper and understand the value that reviewers might have seen in a particular paper.  For example, a paper may develop a new theory or changes our way of thinking.  It might open new avenues for research.  It might be applicable across many disciplines.  It might frame an old problem more clearly.   Look beyond the flaws of a particular instantiation of an idea and consider whether the underlying theories or concepts of a particular approach or solution have value.

Evaluating your experiences and encounters.  Simply having exposure to many research problems and areas is not sufficient to cultivate taste; you need a way to evaluate each new research problem, paper, or area that you encounter.  Just as a connoisseur develops metrics for evaluating a new piece of music, a new book, a bottle of wine, or a restaurant, we need to have yardsticks for evaluating research papers and problems.  When evaluating research that I encounter, I consider the following questions to help further develop my taste.  The answers to these questions are subjective (i.e., reasonable people may disagree about the value of the same piece of work), but they can nonetheless help you articulate why you like (or dislike) a particular research problem or solution.

  • Is the problem important?  Of course, “important” is subjective and defined by your own set of values.  I personally like problems of practical importance.  For example, if the solution to a problem would ultimately improve a network’s performance, security, or availability and could affect the lives of a large number of people in a meaningful way (e.g., developing better spam filters, circumventing Internet censorship), then I deem the problem as important.
  • What is the “intellectual nugget”?  I like research problems (and solutions) that have a simple, elegant solution that’s intellectually profound and easy to articulate.  That might sound trite, but there are plenty of research papers (and researchers who write them) that involve a hodge-podge of solutions with no crisp intellectual contribution (e.g., “We encountered problem A, so we applied X.  Then, we encountered problem B, so we applied Y.” Repeat ad nauseam.)  In my opinion, many of the best research problems (and solutions) can be succinctly summarized in a single sentence.  I value simplicity.  You may not, just as some designers like ornate designs and others prefer minimalist ones.
  • What is the solution or main conclusion?  Is it important?  Although I believe that the size of the problem is at least as important as the goodness of the solution, it is worth considering whether the solution is important, usable, or worthwhile, as well.  Just as “important” is subjective when evaluating problems, the importance of a solution is also subjective—perhaps even moreso than the importance of a problem (which at least might have the benefit of some community consensus if the problem has been studied for long enough).   Determining the importance of a solution is really difficult, even for people with developed taste.  In fact, sometimes developed taste might allow us to overlook a particularly new or innovative solution, just as we can become comfortable with a particular genre of music and fail to recognize a true gem when something disruptive appears.  The establishment is particularly bad at recognizing disruptive breakthroughs because they are used to thinking according to established paradigms.  Our failure to reliably recognize good solutions is perhaps one of the biggest flaws of the peer review system.  (More on reviewing in a later post.)
  • Does the content support the conclusion?  Does the content of the work actually support the solution?  Are the methods sound and state-of-the-art, or have they since been obsoleted (an example of this in networked systems is the now-questionable use of simulation, which has been obsoleted in favor of prototyping and, more recently, operational deployment).  What assumptions does the paper make, and if those assumptions change over time, do the conclusions still hold, and are they important?  (For example, one might ask if a system that assumes the deployment of a particular protocol is still important if that protocol is never deployed, or is currently deployed but likely to be replaced.)

You might have a different value structure for what you think is an important research problem or solution and, hence, you might have different taste in problems.  It’s important to articulate what you think is important, and taking some of the steps outlined above will help you refine your answer to these questions.

You and Your Research Proposal

It’s that time of year again, when researchers young and old gear up to write research proposals. Graduate school hopefuls are preparing research statements, Ph.D. students are writing fellowship applications, students who are trying to graduate are writing thesis proposals, and professors are writing grant applications to funding agencies.  At the core of each of these activities is a single kernel: a research proposal.  Since research proposals show up in many forms throughout one’s research career, figuring out how to write a good one is one of the most important skills that a researcher can learn (and hone).  I think it’s also important to embrace the process—as a researcher you’ll be writing a lot of proposals, so learning to enjoy the process (and becoming good at it) is an important part of one’s happiness (and, hence, ultimate success) as a researcher.

I was initially motivated to write this post as advice for Ph.D. students applying for fellowships, as part of the Intro to the Ph.D. class I’m teaching—we had a long discussion about fellowship applications last week, and some of the content in this post crystallizes that discussion.  As I thought about it, however, I realized that much of the advice that applies to writing a fellowship application can be applied to writing a good research proposal in general, so I’ve decided to generalize the advice that we discussed.

What is a research proposal?

A research proposal is exactly what it sounds like: it is a proposal to perform a certain research project.  It is a plan that is typically non-binding.  In the words of Eisenhower: “Plans are worthless, but planning is everything.”  Your research proposal will thus outline a vision and a plan of work for a research agenda that spans generally three to five years.  This timeframe appears to be common across almost every form of research proposal.  Shorter timeframes generally do not account for the fact that research is an inherently uncertain process and ideas take time to germinate (i.e., if you knew you could complete the work in one or two years, it likely wouldn’t be a research project).  Timeframes that are longer than five years are impossible to plan—generally, there is far too much uncertainty in life, in the world around us (technologies, other discoveries and advances, etc.) to say anything credible about reasonable problems to work on ten years from now.

In all likelihood, nobody will hold you to the plan that you outline in a research proposal.  It is well understood (at least by your colleagues who will review your proposal) that research can take unintended twists and turns, and you may find an exciting thread to pursue mid-course that you could not have imagined or foreseen at the beginning of your journey.  Therefore, as long as you do not stray too far from your original mission, you generally have the flexibility to adapt your research as you proceed.   There is definite value in specificity (I always try to outline specific paper-sized tasks in a proposal), but the broader vision is equally important: The specifics of your proposal (and how or whether certain aspects of your proposal are ultimately executed) are likely to evolve, but the vision is likely to stay roughly the same over the course of the project.

For these reasons, I think the process of writing a research proposal can be tremendously fun.  It is also a crucial part of the research process.    Writing a research proposal is an opportunity to think more broadly about a research agenda, and to be introspective about what problems you think are really important.  Because it is an opportunity to think as far as five years in advance, you can think about the bigger problems that you really want to solve and the best ways to go about solving them.  Because you have a longer time period to solve a problem, you can think about the best methods to solve the problems with and the best people to work on those problems with—even if you don’t know everything about those methods now or aren’t working with those people yet.  Thinking in this unconstrained fashion about bigger problems on a three-to-five year arc allows us as researchers to think beyond the next paper and consider how the work we do fits together into a larger picture. It is a lot of fun.

A recipe for a winning proposal

Of course, we should be clear that success is never guaranteed.  Many factors will affect your proposal’s chances of success, including some factors that are completely out of your control.  A research proposal is often evaluated by a committee.  In the case of a fellowship or a grant proposal, that committee may be made up of people of varying levels of experience and familiarity with your area of research.  They also bring their own biases and assumptions, some of which may be incorrect but be nonetheless held with strong conviction.  You cannot control who reviews your proposal or whether their pre-conceived biases will clash with your vision of a better world.  Fortunately, a few simple steps can dramatically increase your chances of success.

Get to the point.  I remember when I asked a colleague to read my NSF CAREER proposal.  He said, “Here are some comments. Sorry, these are based on only about 30 minutes of reading, but that’s probably the most any panelist will spend reading the proposal, anyway.”  You may find it disheartening that you spend hours, days, or weeks putting together a research proposal, only to have its fate turn on the whims of a reviewer who spends 30 minutes or less on your proposal before rushing off to teach a class or returning to tomorrow’s paper deadline.  That’s life.  Remember that there is more to the process of proposal writing than what one person ultimately thinks of the proposed work (see above on the benefits of the proposal-writing process).  What’s more, if you can’t capture someone’s attention in a few minutes of reading, chances are you need to work on distilling your message more.  You should be able to draw the reader in with just a few sentences or a paragraph at most.  A reviewer is obligated to start reading your proposal, but they are not obligated to finish reading it—get to the point as fast as possible (more below on what your point should be), and entice the reader to continue.  Make it easy for the reader to digest the key points; use bullets and bold headings as necessary (this post is an example of such a writing style).

Tell a story with vivid contrasts.  Everyone loves a good story.  If you want people to enjoy reading your research proposal, then the proposal should tell a good story.  The story, of course, needs to be of a certain type, and written in a certain style (I would not recommend the “mystery novel” approach, for example).  One of my favorite recipes for telling a story is to set up the problem context, explain why the problem is important and hard to solve, and then draw a succinct, stark contrast between your approach and every other previous approach.  For example, many research proposals I wrote about email spam filtering followed this recipe: (1) spam filtering is an important problem; (2) everyone else has been trying to filter spam by looking at the content of email messages; (3) in contrast, I will develop spam filters that discriminate good from bad based on the network traffic, without looking at the content of the messages at all.  I’d then proceed to explain why this was a promising approach and likely to result in new breakthroughs (which it ultimately did).  Your research proposal can and should also follow this recipe.  Tell the reader what sets your work apart, and why it’s likely to succeed where others have failed or otherwise come up short.  Effectively, you are painting your work as so promising and so different from everyone else’s approach that it would be foolish not to fund the proposed work, because nobody else is going to do the work, and not doing it could result in a missed opportunity for a breakthrough.

Answer the Four “Whys”: Why Important, Why Hard, Why Now, Why You?  I think every proposal should answer the following four questions.  Every proposal I write aims to answer these questions, and when I review a proposal, I also look for the answers to these questions:

  • Why is the problem important?  I heard a professor at MIT once tell a fellow Ph.D. student, “There are an infinite number of hard problems.  You might as well work on one that’s worth solving.”  In your proposal, it is important to convince the reader that there is a problem that needs to be solved, and, if your research is successful, it will result in solutions that will make the world a better place for some people.  You might be ridding the world of a certain kind of cyberattack, designing better user interfaces for some class of technology, making it easier to debug software programs, or something else.  There are many ways to advance knowledge and make the world a better place in doing so.  You should first convince the reader that there is a problem out there that needs to be solved—in fact, you should convince the reader that the problem is too important to be left unsolved.
  • Why is the problem hard?  I’ll follow up on this more in a later post, but you should beware of industry “bulldozers”.  The problem that you are working on should require deep thinking and insights, and possibly the application of tools and techniques from multiple disciplines.  It should require a level of thinking that goes beyond the next couple of months (quarterly deadlines, etc.).  If your problem does not pass this test, it’s likely that industry could solve your problem, and they could likely solve it better.  Industry has the capability to hire armies of software engineers to rapidly churn out code.  If the solution to the problem that you propose is a “simple matter of engineering”, and the problem is worth solving, then there is a strong risk that industry will solve the problem better and more quickly.  Convince the reader that the problem that you are working on cannot (or will not) be solved by industry, and that investing money in research on the problem is the best (or only) way to solve the problem.
  • Why now? Most research problems are not entirely new. You may think that you are the first to propose a particular problem.  In many cases, however, problems are longstanding, and many people have proposed variants of the same problem before.  For example, to return to the example of spam filtering: people had been working on the problem for at least ten years; why now is there a possibility to make headway on an old problem, where many others had attempted the same problem?   The answer to this question may be a recent technological advance (e.g., the ability to monitor traffic at high speeds); it might also be the emergence of new technologies in other areas that bring new “hammers” to an old nail (e.g., a new machine learning algorithm that makes an old approach more tractable, efficient, or accurate).  Regardless of what creates the “perfect storm” for doing the research at this time, you should aim to convince the reader that “things are different this time” because of recent advances, changes, etc., and that you’re equipped to take advantage of these new opportunities.
  • Why you?  I think this is perhaps one of the most important elements of a proposal, and one that is commonly forgotten.  Why are you the right person to carry out this research?  You may have convinced the reader that you have identified a hard problem that is worth solving, but if you are a networking researcher who has identified a hard and worthwhile problem in complexity theory (or vice versa), you will have a very hard time convincing the reader that your proposal should be funded.  You must establish credibility, and convince the reader that you are qualified (and, ideally, uniquely qualified) to carry out the work that you have proposed.  Establish your “secret weapon” that you will use to solve the problem that other people don’t have (e.g., domain expertise, a certain body of knowledge, collaborations with people in the appropriate discipline).  Tie back to successes from your own previous work, where possible, and establish bridges between your old (successful) work and the new work that you are proposing to do.  This aspect is where some delicate balancing comes in: You should lean on your past record to establish credibility for the proposed work, yet the proposed work should be visionary enough to encompass three-to-five years of future work.  One way to do this is to include some preliminary work in the proposal to demonstrate that your vision is feasible and that you are qualified to carry it out.  It’s worth noting that this is not the time to be modest.  You’re not talking with friends at a cocktail party; you are selling yourself and your research.  If you don’t sell your work, someone else is going to sell their work, and their sales job may edge your proposal out.  We can be cynical about the need to give a sales pitch and promote ourselves, but the fact of the matter is that if you don’t do it, the other researchers who are competing for the same fellowships, grants, etc. will anyway, so you might as well put your best foot forward and so that your proposed work can be judged on a level playing field.

Be meticulous.  Make sure your proposal is accepted or rejected for the appropriate reasons. Absolutely do not forget to include all mandatory sections of the proposal (e.g., the National Science Foundation takes education, diversity, and outreach extremely seriously; leaving out discussion of these aspects is almost certainly a showstopper for your NSF proposal).  Don’t forget to read the fine print about certain things that reviewers expect to see; if a call for proposal explicitly asks questions, be sure to answer them.  Perhaps most importantly, spell check your proposal, and have it read by a native English speaker before you submit it.  Sure, we all have the occasional typo, but more than one or two typos suggests extreme carelessness and sloppiness.  How can someone trust you to conduct your experiments carefully if you can’t even be meticulous with a short research proposal?  Can someone trust your code or research results if he or she can’t trust your ability to proofread a short, simple document?  Do not convey carelessness, ever.  It is a surefire way to put off reviewers and set you back significantly.  Running a spell-check is super easy, so there is absolutely no excuse for spelling errors.

Have fun, and enjoy the process. Hopefully these pointers will help you in the proposal-writing process.  Not every proposal will win the fellowship or get funding. Remember that there are always factors that you cannot control.   Like many things in life, the process is often as important as the outcome, and with these tips, hopefully the process of proposal writing can be both enlightening and fun.

[Update (October 23, 2013): From Craig Partridge]

I had a great chat with Craig Partridge, one of the Internet’s luminaries, chief scientist at BBN, and an all-around great researcher.  Craig has spent a large part of his career at BBN, which submits proposals for government contracts regularly.  He rightly pointed out that much of what I wrote above is accurate for academics who are applying to funding agencies like the National Science Foundation, but also that some funding agencies are extremely regimented about the format and content of a proposal.  He offered the following tips for the proposal writing process, which researchers at BBN regularly practice when replying to government funding solicitations.  Thanks, Craig!

“Two thoughts to the recipe for a winning proposal.

  1.  A best practice for being meticulous is to create a checklist.  Scour the solicitation for words like “must” and “should” and  “required” and make those sentences into a checklist.   Before you  submit your proposal, confirm that every item on your checklist is in the proposal.
  2.  Get outside reviewers.  The common practice among corporate research   proposers is to use a “pink team” and a “red team”.  A pink team  reads the solicitation and an outline of your proposal about 6 weeks  before it is due and tells you where it sees intellectual or practical  gaps in the outline.  It is an early chance to find problems.  A red team reads the proposal (preferably with your checklist and the solicitation  as supporting material) about a week before it is due and gives you a list  of problems that need to be fixed before submission.”


Welcome!  We (Professors Nick Feamster and Alexander Gray) have created this site as a resource for advice on research and creativity methods and techniques for Ph.D. students.  Our intended audience is Ph.D. students in computer science programs, but many of the concepts that we present on this site may also apply to other disciplines.

The material we have provided will prepare you to perform great research in computer science, regardless of the area you ultimately choose to pursue for your Ph.D. The material should:

  • Teach you many skills that you will keep in your “research toolbox” for the rest of your career:
    • time management
    • productivity and (selective) procrastination
    • how to read a research paper
    • how to review a research paper
    • how to write a research paper (technical writing)
    • how to generate ideas, creativity, sources of problems
    • information management (research notebooks, etc.)
    • how to give a good talk
    • how to write a proposal
    • how to be a good TA
  • Find some inspiration regarding open problems and big ideas
  • Offer general tips for life in graduate school and beyond

The material that we have provided on this site is based on a class that was designed by Professors Nick Feamster and Alex Gray from Fall 2006 through Fall 2010 at Georgia Tech and is now being revamped in Fall 2013 by Professor Nick Feamster.

This project started in Fall 2006, when the two of us were asked to prepare a course for incoming Ph.D. students at Georgia Tech to help them become exposed to research methods early in their career.  After agreeing to take on the preparation of this new course, we quickly discovered that, while there is a wealth of knowledge about research techniques and methods, and many thoughts on skills for creative and critical thinking, this material had not been aggregated or distilled into a single document or course.  We spent the next five years developing a course at Georgia Tech, “CS 7001: Introduction to Graduate Studies”, refining the concepts, methods, and assignments each year.

We have learned a lot from these course offerings.  We have distilled many of our insights and lessons from teaching this course in a an ACM SIGCSE paper.  On this site, we will codify the modules from the course.  We have also made our course notes available on this site, for the benefit of both other Ph.D. students and for faculty at other universities who may choose to use this course as a model for similar courses at their institutions.

Various aspects of this course have been replicated at other universities.  We have made the material from the course available to others for the benefit of both computer science Ph.D. students and others who might wish to teach a similar course.

With Fall 2013 upon us, we are now planning to add material to the site on a weekly basis, as students take the course at Georgia Tech this fall.  By the end of fall term 2013, we will have amassed a full set of resources for graduate students as they embark on their graduate careers.

We are working on adding all of the content to this site.  Meanwhile, please also feel free to peruse some of the previous course offerings (e.g., 20082009).

We will welcome feedback on the material as we post it.


Nick FeamsterPrinceton University
Alexander GraySkyTree, Inc.