In this post, I’ll discuss paper selection—how a program committee considers a set of papers, each with a collection of reviews, and produces a program for a conference.
Paper Reviewing vs. Paper Selection
It’s first important to note that the paper review process and the paper selection process are related, but paper reviewing and paper selection are actually two separate events. There is a human process behind paper selection that involves distilling the reviews of a paper and ultimately making a determination that a paper should be published, but the reviewing and selection have many independent aspects. Sometimes, a paper’s reviews may appear positive, but the paper may still be ultimately rejected (or, the reviews may appear negative but the paper is accepted in spite of the reviews). More commonly, papers tend towards the middle of the review distribution: when reviewers score papers, they tend to regress towards middling scores such as “weak reject” and “weak accept”, rather than taking a strong stance. A significant fraction of papers thus end up with almost the same average “scores”, with only a handful of papers that should clearly be accepted. The number of clear accept papers is never enough to fill the entire program. Suppose, for the sake of example, that a conference can accept 40 papers. The conference may receive several hundred submissions; in most cases I’ve observed, about ten of these submissions receive uniformly glowing reviews. The next best 30–40 papers often fall into a rough equivalence class with the next best 30–40 papers, leaving anywhere from 60–80 papers that are “good enough to be published” for about 30 spots in a conference program. The job of a program committee is to create a smaller program from this larger set of papers.
Whenever a larger number of roughly equivalent candidates competes for a smaller number of positions, some amount of randomness can and will ensue. Much of this randomness is completely beyond the author’s control. As an author, the best way to avoid being subjected to the randomness of a program committee is to write your paper so that it is among the ten best submissions to the conference. Unfortunately, this is far easier said than done, and it’s not really possible to engineer this, although following various research and writing tips can increase the likelihood that your paper ends up in this class. (Speaking from experience, I can say that those tips aren’t fail-proof—even the best researchers I know routinely have papers rejected—but you can at least improve your odds!)
Unfortunately, unless you can guarantee that your paper always falls among the very best papers (a secret that nobody has yet succeeded in unlocking), much of the randomness that results from the paper selection process is out of your control as an author, but other factors can help create a sane selection process. The program committee chair (or chairs) have tremendous power for creating a sane process. These roles include setting the right tone for reviewing and mindset for paper selection, ensuring that papers are reviewed by the right program committee members, selecting the papers that will be discussed at the program committee meeting, and reducing the effects of program committee psychology, such as paper discussion order (e.g., the weight of one biased or incorrect review, and so forth). I will describe more details and tips for program committee chairs below, based on my own experiences and observations.
The Intermediate Dispositions of Papers
Before the program committee meeting itself takes place, the program committee chairs must determine whether each paper should be accepted or rejected without any discussion, or whether the paper should be discussed at the meeting itself.
Before the meeting: Selecting papers for discussion. Program committee chairs typically select papers for discussion at the program committee meeting based on their assessment of the paper’s overall quality, based on review scores, their interpretation of the paper’s reviews, and their assessment of any subsequent “online discussion” that may take place before the physical program committee meeting (most conference reviewing systems allow reviewers to discuss each other reviews in the system itself via asynchronous, email-like messages). Program committee chairs will often try to ask reviewers to reach consensus about whether a paper should be discussed at the program committee meeting before the meeting.
Best case: Acceptance without discussion. If you are very lucky, your paper will not be discussed at the program committee meeting and will be accepted without discussion based on high review scores. I think this is the best possible outcome, because a paper always has flaws, and the more a paper is discussed, the more likely that someone at the meeting will hear something they don’t like about the paper and raise an issue with it. I have seen instances where a paper that receives uniformly high scores is rejected because it is brought up for discussion and someone (often who hasn’t yet read the paper but might read it during the meeting) hears something they don’t like. Such a discussion doesn’t mean that your paper will be rejected—a good PC chair will ensure that these dynamics don’t wrongly result in the unjust rejection of an otherwise good paper—but it does mean that your paper may ultimately be subjected to some of the dynamics I discuss below. Unfortunately, as I mentioned, you can’t guarantee that your paper falls into this category, but you can sometimes be lucky, and you can certainly take many steps to increase your odds.
Next best case: Discussion. Otherwise, if you are still reasonably lucky, your paper will be discussed at the program committee meeting. Discussion represents hope for acceptance, and typically because there are a large number of papers that are of roughly equal score or rank, if your paper is being discussed, it generally has as good of a chance or being accepted as most of the other papers being discussed. The program committee chair will set the order in which these papers are discussed, which can sometimes play a surprisingly significant role in whether a paper is accepted or rejected.
The Dynamics of the Program Committee Meeting
The program committee meeting itself involves the winnowing of a larger set of papers (say, 60–80 papers) down to a final program of, say, 30–40 papers. Typical math I’ve seen in systems and networking conferences is that while overall acceptance rates hover between 10–20%, about 50% of the papers that are discussed will be accepted.
Attendance. An in-person meeting is more effective than other alternatives (e.g., conference calls), if and when it is feasible. Most top-ranked conferences—and even some competitive workshops—have in-person program committee meetings. When program committee chairs select the members of the program committee, they typically confirm that the program committee member can attend the program committee meeting; and, if the meeting is an in-person meeting, they confirm that the PC member can attend in person. If all PC members are not present at the meeting, it is nearly impossible to have anything close to a fair paper selection process. Consider that a paper under discussion might have anywhere from 3–6 reviews, and that most of those reviews might hover around average scores, but one or two reviews may be outliers. The fate of a paper can shift dramatically if the PC member who feels strongly (either positively or negatively) about a paper is not in attendance. Without a champion present, the paper may simply fade into the mix; if a reviewer who raises major concerns about a paper is not present, the rest of the committee may be more likely to discount those concerns, since they cannot hear about them first-hand. Some of these attendance-related quirks can also materialize when program committee members depart early (e.g., to catch a plane flight).
Calibration. Every reviewer has a different opinion for what constitutes a reasonable threshold for acceptance. These differences in calibration are exacerbated by the fact that every program committee member has a unique set of papers to review. That is, while each paper has many common reviewers, no two PC members review exactly the same set of papers. Therefore, calibrating the PC is incredibly important for controlling the meetings dynamics. A poorly calibrated PC may pre-emptively reject good papers (as I mentioned above), only to accept weaker papers later on in the meeting. A good PC chair will spend considerable effort to calibrate the PC to reduce randomness. There are various approaches to calibration, which I will discuss below in my advice for PC chairs. One of them is setting an appropriate order for paper discussion, since the order in which papers are discussed will affect how the PC calibrates itself at various points in the meeting.
Discussion order. There is no accepted standard for setting the order in which papers are discussed, and every program committee chair seems to have a slightly different approach. That said, discussion order has a significant effect on the ultimate disposition of a large number of papers that sit on the borderline between acceptance and rejection. Perhaps the most striking example of ordering effects is evident when a PC discusses papers in decreasing order of score (i.e., from highest ranked to lowest ranked), which is, in my opinion, the worst possible ordering. People arrive at a meeting highly energized and eager for discussion. What often happens is that papers that are discussed in the morning are subjected to a fine-toothed comb and vigorous discussion—sometimes the result of these discussions can result in pre-mature rejection of a good paper. Later in the day, PC members become more tired and are less likely to pick apart a paper’s flaws or have a paper accepted only “over his or her dead body”. Similarly, excessive aggression in the early parts of the meeting can create a situation where, by the end of the day, there are not enough papers to fill the conference program (quite an irony, given the starting point of twice as many acceptable papers as slots!) This seems to create a dynamic where papers that are discussed later in the day sometimes have a better chance of being accepted. If papers are being discussed strictly in order from best to worst, there is the potential for “inversion”, where better papers are rejected early on in the meeting, making room for weaker papers to be accepted later on, as PC members acquiesce and PC chairs are frantically trying to fill the program. PC chairs can take various steps to reduce this randomness. One tweak I have seen (and used) is to simply treat all papers under discussion as roughly equivalent, banning all discussion or consideration “scores”. In these cases, the discussion order can be set such that papers on related topics are discussed together. Another that I have used is to employ a two-pass approach to discussion, where every paper that is not accepted immediately or upon initial discussion is discussed a second time—in other words, a paper is never rejected on the first pass. A third tweak that is sometimes used is to discuss a few “highly ranked” papers first (sometimes even the papers that are supposed to be “accepted without discussion” are used as quick calibration examples), followed by a few “low ranked” papers, and so forth.
Timing of discussion. A PC meeting does not leave time for extended consideration of any single paper. If there are 80 papers to discuss, and a PC meeting has eight hours of real work (discounting breaks, lunch, and so forth), then each paper sees, on average, 6 minutes of attention. Thus, discussion must be crisp and focused. A “discussion lead” for a paper (typically one of the reviewers) will typically summarize the paper and the strengths and weaknesses, effectively summarizing all of the reviews. This summary should last no more than 90 seconds. This is not a lot of time. Therefore, a good PC member who is leading discussion on the paper will prepare that summary ahead of time, so that it is as efficient as possible. The paper’s ultimate fate is typically decided in the remaining time (less than five minutes). Many disputes and disagreements about certain papers cannot be resolved in five minutes. This truth is the most common downfall of a PC meeting—a single contentious paper runs the risk of monopolizing discussion time, leaving less time for the remaining papers, thereby increasing randomness, acquiescence, etc. for papers that are discussed towards the end of the day. I find that using the “two-pass approach” for paper discussion helps mitigate this problem. I’ll discuss this approach in more detail below.
Personalities. A paper that has a single champion tends to fare much better than a paper with middling reviews, even if both papers have the same average score. In some sense, that is exactly the right outcome—a paper that someone really likes is likely to be appreciated by others, whereas an average paper might not excite anyone in particular. However, strong personalities do hold tremendous sway over a paper’s outcome, as well. A paper stands a very good chance if it has a champion on the PC who is articulate, confident, and strong-willed enough to persistently argue for a paper’s acceptance in spite of the paper’s flaws and detractors. Every paper has flaws; a strong-willed PC member can convince the rest of the PC to overlook those flaws in favor of other bright spots. Likewise, a single similarly strong-willed PC member can amplify the weaknesses of a paper and send the paper to the reject pile. These effects are exacerbated if the strong-willed PC member asserts expertise over the topic area. Sometimes, a PC member’s expertise is as important (if not more) important as the volume or persistence of the PC member’s argument. Simple statements from an domain expert such as “I learned a lot by reading this paper” can be enough to tilt a paper towards acceptance.
Psychology. The paper selection process is a human process that involves a lot of psychology. Rather than recount the various psychological factors that can play a role in program committee meetings, I refer you to Matt Welsh’s blog post on the topic.
Tips for Program Committee Sanity
Everything I learned about how to run a program committee, I learned from Jeff Mogul, who co-chaired NSDI this past year with me. Jeff is a veteran program committee chair who has chaired pretty much every major systems and networking conference at some point in his career. He is meticulous, thorough, ethical, and fair, which are indispensable qualities for any program committee chair. Working through the process with him, I picked up several tactics and strategies, which I can recommend to others and will undoubtedly use if and when I chair another major conference program committee. Being a program committee chair has countless tasks; some of these tasks are incredibly important, and the ones that are paramount weren’t immediately obvious to me until after I’d gone through the whole process. Below, I highlight tips for what I think are the most important steps of running a program committee.
- Pick your program committee carefully. Your program committee members need to be thoughtful, reliable, and conscientious. Ultimately, the program committee will write the reviews that authors of submitted papers see, and they will also determine the fate of each submitted paper. Don’t simply pick your friends or people you know well—they may not always make the best reviewers. Take extra time to do homework on program committee members’ performance on past committees. Did they write thorough reviews? Were they active participants in discussion at the PC meeting? Did they turn in reviews on time (or, if not, were they communicative enough to help the chairs plan around timing hiccups?). Are they considered an expert in a particular area that will see a lot of submitted papers for the conference? Beyond selecting individuals, the chairs also need to ensure that every area that may see submitted papers has significant coverage. For example, if the conference will see many submissions on (say) wireless networking, it is incumbent on the chairs to ensure that there are enough reviewers for that particular paper topic.
- Make sure that every program committee member “bids” on papers, and assign reviews manually based on preferences and expertise. Most conference reviewing systems allow program committee members to review all submitted papers and express preferences for which papers they would like to review. Some conference systems also enable chairs to automatically assign reviews. Do not rely exclusively on auto-assignment. When Jeff and I chaired NSDI, we used the auto-assignment feature in HotCRP (after ensuring every PC member filled in review preferences for each submitted paper) and then manually reviewed each paper to ensure that no reviewer was assigned to review a paper for which they had expressed a negative preference. This process is painstaking, but it is perhaps the most important step in the entire paper selection process, because it ensures that reviewers are assigned papers that they are capable of and willing to review. A reviewer who knows a paper topic can typically write a thoughtful review (and often do so quickly). In contrast, a PC member who reviews a paper for which they lack expertise or enthusiasm will typically invest only minimal effort and write a superficial review (thereby increasing randomness and likely reducing overall conference quality).
- Insist on (and monitor) online discussion in advance of the meeting. Selecting discussion leads and asking the lead to type a summary in the online discussion helps organization; the discussion lead can essentially read this typed summary (or notes) at the meeting itself, ensuring that the summary discussion concludes quickly and on time. Online discussion can also encourage reviewers to identify contentious issues before the meeting (where there is limited time to resolve disputes or disagreements), and potentially resolve them ahead of time, thereby averting protracted and unfocused arguments at the PC meeting itself. Sometimes, consensus on a paper can be reached in online discussion before the meeting occurs, saving precious time at the meeting itself.
- Use a two-pass approach to paper discussion. Every other program committee I’ve served on tries to reach a conclusion about a paper’s disposition after a (short) discussion. In the best case, this works OK; in the average case, hasty and incorrect decisions can result; in the worst case, the discussion monopolizes meeting time, and the meeting is derailed and ends in a frantic rush to accept papers at the end of the meeting. The two pass approach we used worked as follows: (1) in the first pass, a discussion lead would summarize the paper and its reviews; in the remaining 3–4 minutes, the reviewers would try to agree on an outcome for the paper, but none of the outcomes were reject. Rather, possible outcomes were: accept, incremental/boring (indicating that the paper was technically correct, but not particularly interesting or groundbreaking), risky (indicating that the paper could be groundbreaking but that reviewers had concerns about correctness or something else), and discuss (indicating that there was absolutely no agreement on the paper, and that more time would be needed to reach an outcome). (2) In the second pass, our original intent was to re-discuss everything, but effectively what happened was that all papers labeled as risky were accepted by default unless someone wanted to argue against one of them. Similarly, all papers that were labeled as incremental were rejected by default unless someone wanted to advocate for one of them. (We had papers in each of these categories.) Pulling reject off the table on the first pass kept the meeting going (we could reasonably end discussion after a fixed amount of time because people knew we could come back to discussion later), and it also reduced the “ordering effects” that I described above, since every paper that wasn’t quickly accepted received two passes, with the second pass occurring after the PC had seen the complete set of papers.
- Read every paper that is being discussed. Although it is not possible to carefully read 80 papers in advance of a meeting, two PC chairs can split the workload and at least perform a quick read of about 40 papers (and take notes on them) in advance of the meeting. Jeff and I did this, and it proved to be incredibly useful, for several reasons. Having your own opinion about a paper as the chair can help a chair to moderate the PC meeting discussion, by ensuring that a strong personality (or, sometimes, a PC member who has not even read the paper!) from unfairly swaying the discussion or perception of the paper. In the end game, sometimes PC members will never come to agreement about whether a paper should be accepted or rejected. In these cases, the chair becomes the “tie breaker” and can accept or reject a paper by fiat. This typically happens at least once in every PC meeting. A PC chair who has not read a paper cannot exercise fiat effectively, so having read every paper is critical in these situations.
The Worst Process, Except for Every Other Process
The program committee process for paper selection is far from perfect; it is an inherently human process, and in the final analysis, a small handful of gatekeepers (indeed, sometimes even a single gatekeeper) can determine whether a paper is accepted to a top-tier conference and read by many others, or shelved. I personally would like to see my own community have a long discussion about (and experiment with) ways to improve this process. In the meanwhile, I trust that this post can help authors (particularly students) understand some of the dynamics of the process and also help chairs and PC members ensure that the process is as fair as possible. Although the process is far from perfect—and it is unlikely that a perfect process exists—understanding the mechanics that go on “behind the closed doors” of a program committee meeting hopefully also indicate that the outcome of the process (either acceptance or rejection) should not be interpreted as universal praise or condemnation, but rather the result of the opinions of a small number of people and the outcome of a human process. Ultimately, the proof of a research idea lies not in the outcome of a single program committee, but in an ideas ultimate acceptance, adoption, and impact (which is, in itself, a topic for a future post).
Learning how to review papers not only (obviously) makes you a better reviewer, but it can also help you as an author, since an understanding of the process can help you write your paper submissions for an audience of reviewers. If you know the criteria that a reviewer will use to judge your paper, you are in a much better position to tailor your paper so that it has a higher chance of being accepted.
There are many good resources that describe the paper reviewing process already, including those that explain the process (and its imperfections) and those that provide instructions for writing a good review (as well as techniques to avoid). There are also a few nice summaries of the review process for conferences in different areas of computer science that lend visibility into the process (e.g., here and here). Program committee chairs sometimes provide guidelines for writing reviews, such as these. I will not reiterate or summarize those previous articles here, but they are all definitely worth a read. Instead, I will discuss the importance of the review process and how it differs from simply reading a paper; I’ll also talk about how to prepare (and ultimately write) a review.
I will not talk about the paper selection process (i.e., what determines whether a paper is ultimately accepted or rejected), but will instead focus on the creation of a paper review. Program committee meetings are an important part of the paper selection process—at least in computer science—and I will be devoting a complete post to this topic next week. Meanwhile, I recommend reading Matt Welsh’s post on the psychology of program committees.
The Review Process
Why understanding the review process is important. Whether you end up reviewing a lot of papers as a Ph.D. student, your research will definitely be subject to the paper review process. It is imperative as a researcher to understand this process. Knowing the process can help you better write your paper for an audience of reviewers (and a program committee), and it can also help you maintain perspective when your paper is accepted or rejected. The process is far from perfect, and the outcome of the process is neither validation nor condemnation of your work. How you react—and how you adapt your research or follow through on it after the acceptance (or rejection)—is far more important to long-term success.
In the “Introduction to the Ph.D.” class at Georgia Tech, I ask students to create a research idea and write it up; a subsequent set of assignments asks the students to review and evaluate the ideas as part of a “mock” program committee. The process isn’t exactly the same as the review process for a full paper, but it is a lightweight way to have students experience the process first-hand in a low-stakes setting, and see both sides of the process (submission and review) at the same time. In next week’s blog post, I will discuss program committee meetings in general, as well as some observations from this year’s (and previous years’) in-class experiences with the mock PC.
Reviewing vs. reading. There are some significant distinctions between reading papers vs. reviewing them. When reading a paper for your own enrichment, your goal is to gather information as quickly as possible. In this case, you are a scientist who seeks to understand the context and content of existing work, to (for example) better understand how your own research might fit into the bigger picture or learn about techniques that might apply to your own work. The goal of reviewing is different. A reviewer’s goal is to first and foremost determine the suitability of a paper for some conference and second, to provide feedback to the authors to help them improve the paper in subsequent revisions. Remember that the reviewer’s primary goal trumps all other objectives: A reviewer often has a large number of papers to process and is typically not deeply devoted to improving the content of any particular paper. If you are lucky, you will get a diligent, thoughtful reviewer who provides thorough feedback, but do not be surprised if a review is not as thorough as you would have liked, or if the review “misses” some point you were trying to make. We would all like reviewers to make three passes through your paper submission—and, these are the instructions I would give, too, in an ideal world. Unfortunately, however, you will be lucky in many cases to get two thorough reads. The reviewer’s main goal is to determine the paper’s suitability for publication. As an author, you shouldn’t be surprised if some of the comments seem trivial: there may be underlying issues of taste that drove the reviewer’s opinion on your paper that a reviewer may not explicitly state. Whenever I read reviews I receive for a rejected paper, I try to look past specific detailed quibbles (or “excuses” for rejecting the paper) and figure out the big picture: the reviewer couldn’t find a reason to accept the paper.
Calibration: Reviewing one paper vs. reviewing many papers. The paper review process can differ depending on who, exactly, is reviewing the paper. For example, as a Ph.D. student, you may review one or two papers at a time, as an “external reviewer” for a conference or journal. Journal editors and program committee chairs often seek the help of external reviewers if they need a particular subject-matter expert to review a paper. Later in your Ph.D. career, you may have established yourself as an expert on a particular topic and find yourself reviewing a paper here and there on a handful of topics. Sometimes a member of the program committee (e.g., your advisor) might ask you to help review a particular paper. As you progress in your career, you will be asked to serve on program committees yourself, whereupon you’ll find yourself with tens of papers to review over the course of a couple of months. Ironically, it is sometimes easier to review a group of papers than a single (or a few) papers, because seeing a group of papers helps you “calibrate” your scores and rankings of papers according to the general quality of papers that have been submitted to the conference. If you have been asked to review a single paper for a conference, you should either figure out how to calibrate your assessment with respect to other papers that might have been submitted, or simply review the paper on its merits while reserving judgement as to the paper’s ultimate disposition.
Does the Paper Realize a Great Idea?
Look for a reason to accept the paper. Does it realize a great contribution or idea? Every paper is imperfect. The paper may have made an incorrect or imperfect assumption. The experiments may not have been as thorough as you liked. The graphs may be difficult to read. Parts of the paper may be difficult to understand. These types of issues certainly reflect problems with a paper, but they do not necessarily constitute a reason to reject a paper if they do not affect the correctness or significance of the main underlying conclusion or contribution of the paper. Therefore, the first two questions I ask myself when reviewing a paper are: (1) Does the paper have a great idea?; and (2) Does it realize the great idea? (or, alternatively, to what extent does it realize that great idea, since typically no paper is water-tight).
What makes an idea “great”? Judging a paper’s contribution turns out to be highly subjective, which is why the review process remains so uncertain. A paper isn’t judged on a set of fixed checkboxes, a grading “key”, or any notion of absolute correctness. Reviewers often reserve considerable judgment based on “taste“, and reasonable people will disagree as to the merits of the main contribution or idea in a paper. In fact, there has been a fair amount of documentation that, as reviewers, we are often quite terrible at predicting the merits of a particular piece of submitted work: There’s a great article on this topic, as well as some parodies to illustrate the subjective nature of the process. Many fields have also introduced a “test of time” award to papers from past decades, to recognize accepted papers that have truly had long-term positive impact (implicitly acknowledging that this is almost impossible to assess when a paper is first published). Due to the subjective nature of this judgment, it is all the more important that your writing is clear, and well-matched to what a reviewer is looking for (i.e., the contributions and ideas).
Invariant questions. Different conferences may have different value structures, and the chairs of any given conference may ask the reviewers to focus on different criteria when judging a paper. Regardless, there are some invariant questions that most reviewers would (or at least should) always consider, including:
- Is the problem important? What problem is the paper trying to solve, and is it important? Seek to summarize the paper’s contribution in one sentence. Make this short summary the beginning of your review, as well. Try to convince yourself (by reading the paper or otherwise) that a solution to the problem that the paper is proposing would advance knowledge or significantly improve the state of affairs for some group of people. Note that you may not care about the problem, but also ask yourself whether you can imagine some group of readers who will be interested in the solution to the problem. When asking yourself this question about a paper, try to divorce your own taste about the problem’s importance from the more general question concerning whether there is some group of people who would be interested in the problem the paper is addressing and solving.
- To what extent does the paper solve the problem it describes? A single paper very rarely closes the book on a single problem, but it may take an important step towards solving the problem. It might solve the problem for an important set of operating conditions or under a new set of assumptions. Or, if the problem area is completely new, perhaps the paper doesn’t really solve the problem at all, but simply articulating a new problem area for follow-on work is a significant contribution.
- What is the “intellectual nugget”? As a reviewer, I try to identify whether a paper has a particular intellectual kernel that lies at the heart of the solution. This kernel is often what separates an important research contribution from a simple matter of engineering. This intellectual nugget might be the application (or invention) of a particular technique, a proof of correctness (where one previously did not exist), or an attempt to put the solution into a broader intellectual context. In other words, the intellectual contribution might be to take a general problem and tackle a specific sub-problem (e.g., under certain assumptions or conditions), or to take a specific problem and generalize it (e.g., develop a general theory, proof of correctness, or taxonomy). Looking through the paper for applications of specific research patterns can help identify an intellectual nugget, if one exists.
- What is the main contribution or conclusion? Is it important? As a reviewer, I try to concisely articulate the paper’s main contribution (or small number of contributions). Often, a paper will helpfully summarize those contributions somewhere in the introduction (Jim Kurose’s advice on writing paper introductions advises the writer to explicitly do so). The reviewer’s job is then to assess whether those contributions are significant or important enough to warrant a publication. The significance of those contributions often depends on the perceived increment over previous work. All work is incremental to some degree, as everything builds on past work. The author’s job is to convince the reviewer that the increment is important, and the reviewer’s job is to assess the author’s claims of significance.
- Does the content support the conclusion? An introduction may make broad (or wild) claims, and it is important to dig into the paper to determine whether the content of the paper supports the conclusion. Are the experiments run correctly? Are they based on the correct set of assumptions? If the conclusion involves comparison to previous work, is the comparison performed in a controlled manner, using an equivalent (or at least fair) experimental setup? If applicable, have the authors released their code and data so that you (or others) can check the claims yourself?
Preparing Your Review
Consider the audience. Not every publication venue is the same. Some venues are explicitly geared towards acceptance of early, incomplete work that is likely to generate discussion (many workshops use this criterion for acceptance). Other venues favor contributions that constitute well-executed, smaller increments. When reviewing a paper, either externally or as a member of a committee, your first question should be to consider the audience for the conference, workshop, or journal, and whether the likely audience for the venue would benefit from reading the paper. The question of audience involves that of both the “bar” for acceptance (Does the paper meet the audience’s standards for something that is worth reading?) and the “scope” of the venue (is the paper on-topic for the venue?). Often, scope can be (and is) broadly construed, so the key question really boils down to whether the likely audience for the paper will benefit from reading it.
Consider the standards. Your standards will (and should) vary depending on the venue for which you are reviewing a paper submission. Workshops are typically more permissive as far as accepting “vision” papers that outline a new problem or problem area or papers that “foster discussion” than conferences, which typically aim to accept more complete pieces of work. Nevertheless, even the standards for a conference review process will vary depending on both the conference itself, the program committee chair’s instructions about how permissive to be, and the relative quality of the group of papers that you are reviewing. A good way to get a sense for the standards of a conference for which you are reviewing is to read through the complete set of papers that you have been asked to review and rank them, before writing a single review. This will ensure some level of calibration, although it is still biased based on the set of papers that you are reviewing. Reading past proceedings of the particular journal or conference can also help you determine the appropriate standard to set for acceptance.
Consider the purpose. Different papers serve different purposes. Multiple paper submissions to the same venue might in fact have quite different purposes, and it is important to establish what the paper is contributing (or attempting to contribute) before passing judgement. For example, a paper might be a complete piece of work, but it might also be a survey, a tutorial, or simply a proposal. If the paper is one of the latter types, your first questions as a reviewer should concern whether the audience would benefit from the survey, tutorial, or proposal, and whether such a paper meets the standards for the conference. If the answers to those questions are “yes”, then your evaluation should be tailored to the paper’s purpose. If the paper is a survey, your assessment should be based on the completeness of the survey, with respect to the area that the paper is claiming to summarize. If the paper is a tutorial, is the description correct and clearly described? If the paper is a proposal, does the proposed research agenda make sense, and is the outcome (if the proposal is successful) worthwhile?
Consider the big picture. Every paper can be rejected. It is always easy to find reasons to reject a paper. The reviewer’s goal should not be to identify the reasons to reject a paper, but rather to determine whether there are any reasons to accept the paper. If the answer to that question is negative, then it is always easy to find “excuses” to reject a paper (recall the discussion above). You should be aiming to figure out whether the paper has important contributions that the audience will benefit from knowing about, and whether the paper supports those contributions and conclusions to the level of standard that is commensurate with the standard of the audience and the venue. One litmus test I use to ensure that a negative aspect of a paper does not condemn it is to ask myself whether the problem (1) affects the main conclusion or contribution of the paper; and (2) can be fixed easily in a revision. If the problem doesn’t affect the main contribution or conclusion, and if it can be easily fixed, then it should not negatively affect a paper’s review.
Writing Your Review
Start with a summary of the paper and its contributions. A short, one-paragraph summary describing the paper’s main contribution(s) demonstrates to the authors (and to you!) that you understand the main point of the paper. This helps you as a reviewer articulate the main contributions and conclusions of the paper for the purposes of your own evaluation. Try to address the type of paper it is (is it a survey paper, for example?), the context for the paper (i.e., how it builds on or relates to previous work), its overall correctness, and its contributions. If you cannot concisely summarize the paper, then the paper is not in good shape, and you can reflect this assessment in the review, as well. These summaries are very helpful to authors, since they may not match the authors’ views of the main contribution! For example, as an author, you can easily figure out if you’ve “missed the mark” or whether the reviewer fundamentally misunderstood the paper by reading a reviewer’s summary of your own work. If the summary of the contribution does not match your own view of the paper’s contribution, then you know that you have some work to do in writing and presentation.
Assess whether the paper delivers on the main claims and contributions. You should provide an assessment, for each of the paper’s main claims and contributions, whether it delivers on that claim. If the main contribution of the paper is flawed, you should indicate whether you think a flaw is “fatal”, or whether the authors could simply fix the flaw in a revision if the paper is accepted. Sometimes flaws (e.g., inconsistent terminology) are fixable. Other flaws (e.g., a questionable experimental setup) may or may not be fixable. While it might seem that a broken experimental setup is “fatal”, ask yourself as a reviewer whether the conclusions from the paper’s experiments as is are still meaningful, even if the authors have not interpreted the results correctly. If the conclusions from the experiments can be restated and still turn out to be meaningful contributions—or, if the flaw in an experiment doesn’t affect the main contribution or conclusion—then even a flaw in experiments can likely be fixed in revision. Occasionally, however, experiments may need to be completely redesigned because they don’t support any meaningful conclusion. Or, the content of the paper may simply be incorrect; sometimes correctness issues are difficult for a reviewer to spot, so a paper isn’t necessarily “correct” simply because a reviewer has validated the paper. Regardless, if there are correctness issues that affect the main contribution of the paper that call into question whether the main result or contribution is correct in the first place, the paper’s review should reflect these concerns and likely cannot be accepted.
Discuss positive aspects of the paper; always try to find something positive, even in “bad” papers. It is easy to identify problems with a paper. It can be much trickier (especially with “average” papers) to identify the positive aspects and contributions, but most papers typically have at least some small kernel of goodness. Even for particularly bad papers, there might be one sentence in the introduction, discussion, or future work section that makes an interesting point or highlights a possibility for interesting contributions. In a pinch, if you can’t find anything positive, those are good places to look. As a reviewer, you can remark that those observations are interesting, and that you would really like to see those parts of the work further developed. These positive comments aren’t just for author morale (although that’s important, too): They give the author a direction to move forward. The worst reviews are those that reject a paper but don’t provide any specific action for moving forward. The best reviews are those that highlight the positive aspects of the work, while identifying weaknesses and areas where the work could be further developed to address weaknesses or build on the paper’s existing strengths.
Criticize the paper, not the authors. When writing your review, consider the type of review that you would like to receive. Always be polite, respectful, and positive. Don’t be personal. Choose your language carefully, as it will help convey your message. For example, if you say “the authors don’t consider the related work”, that is a much more personal statement than “the paper doesn’t consider the related work”. (In fact, you don’t know if the authors considered a particular piece of related work anyway; they may have simply chosen not to include it in the writeup!) Talking about “the authors” gets personal, and it will put the authors themselves on the defensive when reading your review. Instead, focus on “the paper” and frame your critique around “suggestions for improvement”. Never, ever insult the authors; don’t accuse the authors of being sloppy or unethical researchers. As a reviewer, you don’t always know the full context, so limit your judgement to what you can directly conclude by reading the paper.
Consider the type of feedback you would like to receive. Receiving reviews for rejected papers is a part of the research process, but it is never fun for the authors (particularly new Ph.D. students). Do your part to contribute positively to the process by suggesting changes that you’d like to see if you had to review the paper again. In all likelihood, you may see the paper again in the form of a revision!
Little did I know that a Ph.D. in computer science would teach me more about how to write than any other previous experience I’d had. Indeed, research isn’t only about making new discoveries, but also about expressing those discoveries so that other people can appreciate them and their significance.
Below, I’ll present some thoughts on writing that I’ve picked up over the years and continually aim (and encourage students) to put into practice. I’ll focus on story-telling, presentation, and efficiency: how to write your paper so that it clearly, cleanly, and efficiently tells a good story.
What this post is not about. At least in this post, I won’t write about grammar—there is too much English grammar to cover in a single post, and there are plenty of good courses one can take to learn grammar. I also won’t talk about style (common phrases) or usage (e.g., commonly misused words). For now, my only advice on the topic of learning English grammar and style—both for native speakers and foreigners—is to read daily. Reading the New York Times, or The Economist are great places to start: the writing is generally of high quality, and you’ll also have something interesting to talk about outside of the lab. I have the impression that foreign students can tend towards reading their local news from home; that’s understandable, but I would encourage those students to also read periodicals and publications that are written in the style of English that people are used to reading in the Western world (indeed, English in different parts of the world has different tones, styles, idioms, and turns of phrase).
Why Does Writing Well Matter?
Writing is part and parcel of the research process. Writing communicates your research results, and—perhaps contrary to intuition—writing well is even more important when the results that one wants to present are “strong” (after all, if the results are that great, you should want as many people as possible to benefit from understanding them). But writing isn’t just something that happens at when the research is “done”; rather, writing should occur throughout the course of a research project. Writing and thinking are tightly coupled—writing is a reflection of our own thoughts, and writing well can actually sometimes clarify our own thinking about a problem and lead us to a solution.
Communicating your results. A research finding or discovery that others cannot understand or appreciate is as good as no discovery in the first place. A well-written paper with a half-baked idea that people can understand and appreciate will often be accepted to a conference over a paper with a (possibly) stronger result that nobody can understand or appreciate. Cynics may dismiss this as “unfair”, as “laziness” on the reviewer’s part for not reading the details thoroughly, or as “accepting a paper because it was well-written” (insinuating that the results themselves might not be that strong). The fact of the matter is that none of this matters: Independently of the quality of the research, the results must be written well, or people will not be able to appreciate the value of your work. Writing well is perhaps one of the most important skills that budding researchers must attain; it is equally important as learning how to conduct the research.
Clarifying your own thinking. Writing is a reflection of your thinking. You should never be content with vague language or imprecise descriptions. Muddy writing often reflects muddy thinking. An attempt to write down a concept, idea, or solution can actually help you realize that you don’t know what you are talking about! (And, help you crystallize your thinking.) You may have had the experience of realizing you don’t understand something as well as you thought you did when someone asks you to explain something and you have trouble. Writing achieves the same effect: as you attempt to write something down, you may realize that you don’t know why you are working on something, why a particular phenomenon appears, and so forth. If you don’t force yourself to try to write down your explanations and observations, you may never discover that you don’t know what you’re doing until it’s too late. Therefore, start writing early in the course of a research project. Most of the early text you write may not find its way into the final version of the paper, but taking the time to express your ideas on paper can help you crystallize your thoughts and guide your research.
Tell Me a Story
Every paper should tell a story. For a given research result, there are many stories that a paper could tell, and it’s important to think about what the story should be. This decision should be made early on in the writing process. The story can be changed later on, to match the strongest results in the paper, or if the context for the research changes (for example, sometimes when working on a research problem, people can find it necessary to re-formulate the problem that they are solving). But, it’s always good to figure out a story before you go too far down the path.
Determining the Big Picture: Composition and “Flow”. Composition is the structure and flow of your paper writeup, and how individual paragraphs fit together. When people talk about the “flow” of a piece of writing, they’re talking about composition: how sentences flow together to form paragraphs, how paragraphs flow together to form sections, and how sections flow together to form a paper. Composition (or “flow”) is the most important aspect of technical writing. I’ve found that reviewers (and readers) are often forgiving of the occasional grammatical error, especially if a paper was written by non-native English speakers. The human brain is pretty good at applying “error correction” on grammar. Correcting “flow”, on the other hand, is extremely hard for a reader to do. Often, the reader will complain that they “can’t understand where [the paper] is going” and simply give up. Grammar is also easier to teach (and learn) than composition because there are fixed rules for grammar; composition is more of an art form, and it does involve a bit of intuition. It involves thinking about the big picture, the story that you want to tell, and how the pieces fit together. It takes practice to develop intuition for composition, and I’ve found that native English speakers are not necessarily any better at composition than foreigners are (that’s good news for Ph.D. students coming from abroad: the playing field is actually more level than it appears). Even though composition is a bit of an art form, I’ve found that the following tips can be incredibly helpful for improving flow:
- Build the scaffolding before filling in the details. Write sections and topic sentences first. There is always the temptation to simply jump in and start writing full paragraphs. This rarely ends well. Nobody would build a house before first drawing plans. After the plans comes a foundation, and scaffolding. Only after the structure is in place can many of the details be filled in. You should think about writing a paper in the same way: before you can fill in the details, you need to figure out how you want the paper to read when its done; this is your outline. Start at a high level, by outlining sections. Once you have an outline of sections, you can begin to slowly fill in more details: outline each paragraph within a section by writing only the topic sentence for the paragraph. See whether your story makes sense if you read each topic sentence in sequence. Your “paper” should roughly make sense if you read these topic sentences in sequence (in fact, some reviewers or readers may try to read your paper in exactly this fashion, by skimming topic sentences!). If reading your topic sentences in sequence doesn’t make sense, something is wrong with your flow. Determining which topic sentences to write takes a bit of practice, but I find that making a bulleted list of points that you want to include in the section and then clustering those points into related points and topics can help identify the topics you want to cover (and, hence, the topic sentences you want to write). If you find that the topics you want to cover are not all related to one another, you may need to create additional subsections. Once you have topic sentences, you can fill in paragraphs, which are groups of closely (and logically) related sentences. If your sentences aren’t closely related, they don’t belong in the same paragraph and you may need to consider adding a new topic (and topic sentence).
- Assess balance and budget pages accordingly. Once you have your outline, it’s important to budget pages, to figure out how much you plan to write about each topic. Topics of equal importance or equal relevance should receive “balanced” coverage in the paper. If your system has two components of relatively equal importance, then the sections covering those components should be of approximately equal length in the paper. Once you have your outline (from the previous step), you can assess how much space you want to budget for each point. If some points in a section are more important than others (and hence require more page budget), you can go back and think about how to re-balance your writeup (e.g., by creating a new devoted section to the topic that’s more important, shortening or eliminating the less important point, and so forth).
- Follow convention. Continuing the analogy of a house or building, many designs follow convention: a restroom in a restaurant is often in the back. A powder room in a house is often near the front door. These common “design patterns” make life easier for us when we visit a new building or house. Similarly, your reader is a visitor of your thought process. A reader who has read (and written) many research papers will similarly expect the structure of your paper to follow certain conventions. These conventions differ by field: some fields write papers completely in the passive voice, whereas others use active voice. Some fields prefer related work sections at the beginning of the paper, and others prefer related work at the end. Some fields expect an introduction that spends time framing the problem, and others expect an introduction that starts with a clear statement of the main result. There may be an expectation of (or aversion) to mathematics in the main line of the paper. Within a section, people may expect the introduction to flow in a particular way (in computer science, Jim Kurose’s description of how to write an intro is the best example I’ve seen for writing a conventional paper introduction).
- Signpost your paper. Even if your paper plan and outline make perfect sense to you, the reader doesn’t start out “on the same page” as you. Even if the builders have built a house completely according to the blueprints, even if the design of the house follows convention, visitors still typically appreciate an explanation of where the bathroom is. The reader needs instructions for how to read the paper and a clear view for how the paper and story will proceed. Signposts throughout the paper help the reader understand how the paper is organized. Examples of signposts include: an outline of the paper at the end of the introduction (“The rest of this paper proceeds as follows.”), a preamble to each section (“In this section, we discuss…”), declarative subsection titles, and (within reason) bold paragraph headings (such as those in this blog post).
Make a Good Impression
A reader forms an impression about your research based on your presentation of the work. Just as people quickly form first impressions about other people that may be difficult to subsequently correct, your reader will quickly form a first impression of your work based on a quick read of the title, abstract, and introduction and perhaps a quick skim through other parts of the paper. Therefore, it is important to make a good impression by writing a clear abstract and introduction (and also ensuring that the rest of the paper also is commensurate with the impression that you create).
Spend a lot of time on your introduction; start early. The introduction summarizes the story of your paper (now, it should be doubly clear why you need a story!) and is the most important part of your paper. Many people won’t read past the introduction, even if they find the work interesting. Yet, if people don’t understand your “story” (the problem you’re solving, its importance, and your approach to solving it), they will never make it past the introduction into the details of the paper. There are two schools of thought for writing the introduction: if you write it first, you ensure that you have a clear story. If you write it last, then your introduction will clearly reflect the ultimate story and be bolstered by your results. My advice is to do both. Writing the introduction is probably the most difficult part of writing the paper, which may come as a surprise to new students. After all, the introduction is high-level and often doesn’t reflect the nitty gritty details of how difficult that experiment was to run, and how much toil went into writing the system. To the contrary, distilling your work from those details so that the rest of the world can digest the takeaways—stepping out of the weeds and trying to figure out why your work matters—can take more effort than anything. The same results can be presented in many different ways, so you will likely need to perform many iterations on your story. Typically, you can “test” your story out on other colleagues and readers with a draft introduction in parallel with completing the rest of your work.
Write the introduction first—and last. I typically try (and encourage students) to write a draft of the paper’s introduction before writing other parts of the paper. You will most likely eventually discard this draft completely, since it’s very hard to predict what one should be emphasizing in the introduction before the “meat” of the paper is done. Yet, writing an early introduction draft allows a researcher to put the work in context, and to argue (1) what the problem is, (2) why it is hard, and (3) why the solution (if it is achieved) will be interesting to readers. If an introduction draft can’t articulate answers to these questions, even at an early stage in the research, then the work likely needs some focus, as well. When you write the second version of your introduction (after the rest of the paper is written), you are in a good position to refine your story, making sure that any claims you make are clearly supported by your results and data. Your results should be stated as clearly and specifically as possible. You should neither understate your results (which might cause the reader to lose interest and stop reading) or overstate them (which might cause the reader to be annoyed or “let down” when they dig in and find that you don’t deliver on what you’ve promised).
Perform some “landscaping” on your paper. Your paper should be pleasing to the eye. Care in preparation of the paper suggests that you also cared about the research itself. Conversely, if the preparation of the paper is sloppy, how can the reader trust that you were not similarly sloppy in writing code, setting up and running your experiments, and analyzing your results? Therefore, take some time to make your paper look nice. Here are a few quick tips on “landscaping” your paper.
- Eliminate widows. Section headings at the bottoms of columns or pages look sloppy. Depending on your word-processing tool, there are good ways to get rid of widows. In Latex, you can always use a “\vfill” to kick the section heading to the top of the next column. Similarly, widows on paragraphs (paragraphs that end with a single word on the last line) look sloppy. Typically you can eliminate such widows by wordsmithing (see below on omitting needless words). Latex also has macros such as “\widowpenalty” that help ensure that the text layout avoids widows whenever possible.
- Place (and create) signposts, figures, and graphs to create whitespace and avoid “walls of text”. Reading long installments of technical writing is tiring, and seeing a page of uninterrupted technical writing can psychologically tire the reader before they even begin to read. Create whitespace and take other steps to break up large chunks of text. Use paragraph headings to break up long strings of text (ensuring that your headings match the topics you’ve outlined, of course). Try to space figures, tables, and graphs throughout the paper so that most pages have a mix of figures and text. If you don’t have a figure to break up the text, consider making one. Often, a table summarizing results can be useful. Or, you might find that a figure (or block of pseudocode) describes a concept better than a paragraph in your writeup. Take steps to ensure that your writeup is not a long stream of text.
- Check your spelling. Run a spell checker. Every word-processing tool has one, and a writeup that is rife with spelling errors reflects carelessness. Again, if you can’t be bothered to check your spelling, why should a reader trust that you spent time debugging your experiment or code? While a reader can forgive grammatical errors, especially when the writeup comes from a non-native English speaker, anyone can run a spell checker, so spelling errors simply reflect carelessness. That said, not all spell-checkers are perfect, and sometimes typographical errors actually manifest as real words; spell checkers won’t catch those, so they should be used only as a first pass.
- Make the last page look decent. If you’ve edited your text down to remove redundancy, you may find that you have some extra space at the end of your paper. Try to adjust your formatting so that the last page of the paper looks decent. If your paper is double-column and you end with references, balance the references across the two columns so that both columns are both equally (partially) filled. Latex has a “\balance” command that you can use to balance the columns on the last page of your writeup. If you have a bit more time, you can also make small adjustments to margins, space between columns, space around figures, sizes of figures, and so forth to ensure that the last page doesn’t look like you just suddenly decided to stop writing.
“I’m sorry I have had to write you such a long letter, but I did not have time to write you a short one.” –Blaise Pascal
After you’ve made a good impression, you still need to engage the reader at least long enough to convey your main message. You should assume that the reader is busy, has a short attention span, and (unlike the situation when they are, say, reading a news article or novel) simply wants to learn what you’ve done as quickly as possible and then move on. In contrast to writing a novel, your goal is not to hold the reader in suspense or to illustrate your command of vocabulary or literary construction. Rather, the goal is to transfer information as efficiently as possible.
Tailor the length of your paper to the information content. People sometimes feel compelled to stuff a paper to the conference’s or journal’s page limit (after all, if the paper is too short, isn’t there “more work to do”?). I’ve had a few papers that do fall short of the page limit, although I’ve found that this scenario is the exception rather than the rule (see Blaise Pascal: it takes time to shorten a paper!). Remember that your goal is to efficiently transfer information. Therefore, economize on words. Do not add unnecessary sentences, paragraphs, or other content that do not support a topic or point that you’ve explicitly outlined.
Keep words simple and sentences short. The easier it is to read your paper, the more likely a reader will keep reading. Therefore, don’t make the reader work harder than they have to. On a sentence level, follow Strunk and White’s most indispensable advice: Omit needless words (e.g., “to” as opposed to “in order to”, “optimizing” instead of “the problem of optimizing”, “This module…” instead of “This is a module that”, “more quickly” instead of “in a shorter time”, “whether” instead of “the question as to whether”). Avoid padding phrases such as “the fact that” and unnecessary adverbs (e.g., “eliminated” instead of “totally eliminated”, “fast” instead of “very fast”). Keep words simple (e.g., “use” instead of “utilize”, “new” instead of “novel”), and keep sentences short.
Eliminate redundancy. Always make at least one (and probably several) editing passes over your paper to cut out redundancy. You likely (and hopefully) did not write your paper in a linear fashion, and different co-authors may have written different sections or pieces of the paper. It’s likely that you’ve stated the same point multiple times throughout the paper. Cutting redundancy takes practice, and it sometimes requires divorcing the editing process from your ego (after all, isn’t that sentence you wrote beautiful? It’d be a shame to cut it, right?). If you need help cutting, ask someone else (a co-author or colleague) to help you—they’re not as married to your text as you are.
Be as specific and precise as you can. Precision is especially important when describing results, and when writing up an experiment or evaluation section. For example, a good rule of thumb for writing up an evaluation section is: Would someone be able to reproduce your experiment from your description? If not, you need to be more clear. In an evaluation section, clearly state your assumptions; in what context(s) do your results hold? How general are they? Clearly describe your evaluation setup, as it sets the context for the results themselves. What machines did you use to run the evaluation? What data did you gather to support your conclusions? What algorithms did you use to analyze the data? The evaluation setup should read like a recipe. Someone reading it should be able to reproduce what you’ve done, at least well enough to understand the context of the results. Similarly, the results should be clearly and precisely described. Avoid writing sentences such as “This optimization significantly reduced the system’s running time.” Such vague statements frustrate the reader. Instead, substitute precise and specific information: “This optimization reduced the system’s running time by 20%”.
Use clear and consistent terminology. Unclear, vague, and inconsistent terminology is incredibly frustrating and also typically reflects muddy thinking. If you call something a “node” in one part of your paper, a “host” in another part of the paper, a “client” in another part, and a “machine” in yet another part, people will have absolutely no clue that all of these terms are in fact referring to the same thing. Before you get too far along in the writing process, clearly define your lexicon. Defining your terms up front can help clarify your own thinking, by helping you to identify and articulate the important concepts that you will present. Ultimately, this clarity will be reflected in your writeup.
Reduction to Practice: Write Every Day
Writing a technical paper involves clearly presenting a good story, as efficiently as possible. The above tips should help you compose your story and ensure that it “flows” and ultimately produce a final result that leaves a good impression and communicates your results as efficiently as possible. But, simply reading this post won’t make you a good writer: putting these tips into practice takes tireless repetition. Try writing something every day to practice these tips. Keep a “running draft” of your research paper. Start a blog and practice composing posts that present shorter and simpler ideas and thoughts than a full research paper. Keep a diary or notebook. Ultimately, becoming a good writer requires relentless, daily practice, with continual attention to the above tips.
Just as each of us develops taste in books, music, art, and food, every researcher ultimately develops a taste for research problems. Every researcher should spend some time developing “good taste” in research problems. The world has many challenging problems to work on, and as researchers, we have limited time and bandwidth. It’s therefore important to develop (good) taste in selecting problems, so that we end up working on the problems that are worth a significant investment of time and energy. Many research problems will take years to run their course, so it is worth spending some time developing taste in problems.
Many professions, ranging from designers to architects to programmers to managers, need to develop good taste. In this post, we’ll focus mainly on developing taste in research problems, although some of these tips for cultivating taste may apply more generally (in fact, some of the pointers below were inspired from an article I recently read about developing taste in design).
Cultivating your own research taste. We are not born with good (or bad) taste; rather, we develop taste by way of education and exposure to many different opportunities and experiences. Just as one might cultivate taste in other areas of life, one must cultivate taste in research. In this post, I’ll offer some tips for cultivating research taste that I’ve found work for me. Some of these tips I have discovered (and applied) by way of analogy for developing taste in other realms (e.g., music, food). I’ve spent a fair bit of effort developing taste in music; where applicable, I’ll draw some analogies below.
- Seek out others with good taste. Perhaps the most important step for developing good taste is to associate and learn from others who have good taste. These people are generally acknowledged as “having good taste” and are often easy to identify. Just as you may have friends who you know are well-read about music, wine, or food (you know these people because you’re always asking them for recommendations), the research community has “thought leaders” who are widely acknowledged as having good taste in research problems. It’s not too hard to figure out who these people are with a little research. Ask several colleagues who these people are and see whether trends begin to emerge. Poke around on Google Scholar and see which researchers in your area have highly cited articles on a particular topic. Intersect these people with those who share your interests, as well. Once you have identified others with common interests who appear to have good taste in problems, try very hard to associate, exchange ideas with, and work with those people. Become an apprentice. As a Ph.D. student, seek these researchers out as possible advisors. The Ph.D. years are perhaps the most formative for developing research taste, and your taste will likely be shaped heavily by your advisor, so taking the time to find an advisor who will develop your research taste is perhaps one of the most important decisions you will make as a Ph.D. student.
- Read trend-setting conference proceedings, and develop opinions about research problems and trends in your area. The “top tier” conferences in your area are essentially the Architectural Digest, Wine Enthusiast, or Pitchfork for your discipline. Track your conference proceedings to determine the research areas that the best researchers in your area are working on. You don’t necessarily have to “jump on the bandwagon” and start working on any of the research areas that are the current hot topics at this year’s conference—just as you might not spend several hundred dollars on the latest wine that’s reviewed or go right out and buy every new music release—but it certainly doesn’t hurt to learn about the latest trends, even if you don’t always resonate with them. Exposing yourself to the latest trends and developing opinions about them (positive or negative) is an important step in cultivating research taste. It’s typically not necessary to read the entire conference proceedings to get a feel for what’s going on in an area; simply looking at the names of sessions and groupings of papers can help you quickly identify areas that are receiving a fair amount of attention from the community.
- Sample and experiment with abandon. Developing taste in any genre involves gaining exposure to many different examples, good and bad. Just as it’s much easier to appreciate a truly fine wine, dish, or performance after having seen mediocre offerings, since every experience allows us to better articulate what we do or don’t like, sampling a wide variety of research problems (and solutions) is a necessary step for developing taste in research. In developing taste in music, I find myself reading continually about new artists and albums and listening to new material that pushes boundaries in new ways, and sometimes subjecting myself to music that in the end I might decide I don’t like all that well. Similarly, in research, we must continually experiment and sample to develop and cultivate our taste. It can be tempting (and certainly easier) to “turn the crank” on problems that we know how to solve, but ultimately this will result in research that becomes stale and boring. As researchers, we should be continually learning about new techniques, tools, problems, approaches, and so forth. We will likely find that some topics, areas, and solutions we encounter are incredibly boring, but those “bad samples” ultimately help us determine what aspects of a research problem or solution we do or don’t like.
- Keep a list of ideas that you like and exchange your favorite ideas with colleagues. Keeping lists of things you like is good; exchanging them is even better. Every year, I tabulate a list of my “top 10” music albums for the year; at the end of the year, my friends and I exchange lists. The list is fine, but making the list is actually more important. Knowing that I am going to be exchanging a list of music with friends at the end of the year keeps me accountable and ensures that I am always “on the lookout” for new gems. Similarly, making lists of neat research ideas and regularly exchanging them with colleagues is another way to ensure that you always have a healthy appetite for new, creative ideas. I don’t exchange lists of research ideas, although perhaps I should; I do, however, regularly exchange papers, articles, and ideas with a small, trusted group of colleagues, often multiple times a week.
- Attend conferences. Just as a music enthusiast attends live concerts or a book enthusiast might attend a reading, as a researcher you should attend research conferences. I think it is worthwhile to attend at least one major research conference in your area every year. Attending a major research conference ensures that you are staying current with research trends (in case you’d like to “watch the movie”—that is, attend the talks—instead of simply poring over the conference proceedings). Perhaps more important than attending the talks, however, is meeting other researchers. Conferences are excellent places to seek out others with good taste, and to have conversations with other researchers that can help you develop opinions about various research problems and areas. These conversations and interactions are all part of the process of developing taste as a researcher.
- Consider the principles and theories that underpin a specific research paper or project. No research paper (or project) is perfect. In fact, most papers are flawed, and many are badly flawed. Yet, papers are often published not for the particular artifact that they produce, but for some underlying concept or idea that they embody or espouse. When reading research papers, it is far too easy to be dismissive of a paper because of a flaky implementation, a bogus evaluation, or poor exposition. Try to dig a bit deeper and understand the value that reviewers might have seen in a particular paper. For example, a paper may develop a new theory or changes our way of thinking. It might open new avenues for research. It might be applicable across many disciplines. It might frame an old problem more clearly. Look beyond the flaws of a particular instantiation of an idea and consider whether the underlying theories or concepts of a particular approach or solution have value.
Evaluating your experiences and encounters. Simply having exposure to many research problems and areas is not sufficient to cultivate taste; you need a way to evaluate each new research problem, paper, or area that you encounter. Just as a connoisseur develops metrics for evaluating a new piece of music, a new book, a bottle of wine, or a restaurant, we need to have yardsticks for evaluating research papers and problems. When evaluating research that I encounter, I consider the following questions to help further develop my taste. The answers to these questions are subjective (i.e., reasonable people may disagree about the value of the same piece of work), but they can nonetheless help you articulate why you like (or dislike) a particular research problem or solution.
- Is the problem important? Of course, “important” is subjective and defined by your own set of values. I personally like problems of practical importance. For example, if the solution to a problem would ultimately improve a network’s performance, security, or availability and could affect the lives of a large number of people in a meaningful way (e.g., developing better spam filters, circumventing Internet censorship), then I deem the problem as important.
- What is the “intellectual nugget”? I like research problems (and solutions) that have a simple, elegant solution that’s intellectually profound and easy to articulate. That might sound trite, but there are plenty of research papers (and researchers who write them) that involve a hodge-podge of solutions with no crisp intellectual contribution (e.g., “We encountered problem A, so we applied X. Then, we encountered problem B, so we applied Y.” Repeat ad nauseam.) In my opinion, many of the best research problems (and solutions) can be succinctly summarized in a single sentence. I value simplicity. You may not, just as some designers like ornate designs and others prefer minimalist ones.
- What is the solution or main conclusion? Is it important? Although I believe that the size of the problem is at least as important as the goodness of the solution, it is worth considering whether the solution is important, usable, or worthwhile, as well. Just as “important” is subjective when evaluating problems, the importance of a solution is also subjective—perhaps even moreso than the importance of a problem (which at least might have the benefit of some community consensus if the problem has been studied for long enough). Determining the importance of a solution is really difficult, even for people with developed taste. In fact, sometimes developed taste might allow us to overlook a particularly new or innovative solution, just as we can become comfortable with a particular genre of music and fail to recognize a true gem when something disruptive appears. The establishment is particularly bad at recognizing disruptive breakthroughs because they are used to thinking according to established paradigms. Our failure to reliably recognize good solutions is perhaps one of the biggest flaws of the peer review system. (More on reviewing in a later post.)
- Does the content support the conclusion? Does the content of the work actually support the solution? Are the methods sound and state-of-the-art, or have they since been obsoleted (an example of this in networked systems is the now-questionable use of simulation, which has been obsoleted in favor of prototyping and, more recently, operational deployment). What assumptions does the paper make, and if those assumptions change over time, do the conclusions still hold, and are they important? (For example, one might ask if a system that assumes the deployment of a particular protocol is still important if that protocol is never deployed, or is currently deployed but likely to be replaced.)
You might have a different value structure for what you think is an important research problem or solution and, hence, you might have different taste in problems. It’s important to articulate what you think is important, and taking some of the steps outlined above will help you refine your answer to these questions.
It’s that time of year again, when researchers young and old gear up to write research proposals. Graduate school hopefuls are preparing research statements, Ph.D. students are writing fellowship applications, students who are trying to graduate are writing thesis proposals, and professors are writing grant applications to funding agencies. At the core of each of these activities is a single kernel: a research proposal. Since research proposals show up in many forms throughout one’s research career, figuring out how to write a good one is one of the most important skills that a researcher can learn (and hone). I think it’s also important to embrace the process—as a researcher you’ll be writing a lot of proposals, so learning to enjoy the process (and becoming good at it) is an important part of one’s happiness (and, hence, ultimate success) as a researcher.
I was initially motivated to write this post as advice for Ph.D. students applying for fellowships, as part of the Intro to the Ph.D. class I’m teaching—we had a long discussion about fellowship applications last week, and some of the content in this post crystallizes that discussion. As I thought about it, however, I realized that much of the advice that applies to writing a fellowship application can be applied to writing a good research proposal in general, so I’ve decided to generalize the advice that we discussed.
What is a research proposal?
A research proposal is exactly what it sounds like: it is a proposal to perform a certain research project. It is a plan that is typically non-binding. In the words of Eisenhower: “Plans are worthless, but planning is everything.” Your research proposal will thus outline a vision and a plan of work for a research agenda that spans generally three to five years. This timeframe appears to be common across almost every form of research proposal. Shorter timeframes generally do not account for the fact that research is an inherently uncertain process and ideas take time to germinate (i.e., if you knew you could complete the work in one or two years, it likely wouldn’t be a research project). Timeframes that are longer than five years are impossible to plan—generally, there is far too much uncertainty in life, in the world around us (technologies, other discoveries and advances, etc.) to say anything credible about reasonable problems to work on ten years from now.
In all likelihood, nobody will hold you to the plan that you outline in a research proposal. It is well understood (at least by your colleagues who will review your proposal) that research can take unintended twists and turns, and you may find an exciting thread to pursue mid-course that you could not have imagined or foreseen at the beginning of your journey. Therefore, as long as you do not stray too far from your original mission, you generally have the flexibility to adapt your research as you proceed. There is definite value in specificity (I always try to outline specific paper-sized tasks in a proposal), but the broader vision is equally important: The specifics of your proposal (and how or whether certain aspects of your proposal are ultimately executed) are likely to evolve, but the vision is likely to stay roughly the same over the course of the project.
For these reasons, I think the process of writing a research proposal can be tremendously fun. It is also a crucial part of the research process. Writing a research proposal is an opportunity to think more broadly about a research agenda, and to be introspective about what problems you think are really important. Because it is an opportunity to think as far as five years in advance, you can think about the bigger problems that you really want to solve and the best ways to go about solving them. Because you have a longer time period to solve a problem, you can think about the best methods to solve the problems with and the best people to work on those problems with—even if you don’t know everything about those methods now or aren’t working with those people yet. Thinking in this unconstrained fashion about bigger problems on a three-to-five year arc allows us as researchers to think beyond the next paper and consider how the work we do fits together into a larger picture. It is a lot of fun.
A recipe for a winning proposal
Of course, we should be clear that success is never guaranteed. Many factors will affect your proposal’s chances of success, including some factors that are completely out of your control. A research proposal is often evaluated by a committee. In the case of a fellowship or a grant proposal, that committee may be made up of people of varying levels of experience and familiarity with your area of research. They also bring their own biases and assumptions, some of which may be incorrect but be nonetheless held with strong conviction. You cannot control who reviews your proposal or whether their pre-conceived biases will clash with your vision of a better world. Fortunately, a few simple steps can dramatically increase your chances of success.
Get to the point. I remember when I asked a colleague to read my NSF CAREER proposal. He said, “Here are some comments. Sorry, these are based on only about 30 minutes of reading, but that’s probably the most any panelist will spend reading the proposal, anyway.” You may find it disheartening that you spend hours, days, or weeks putting together a research proposal, only to have its fate turn on the whims of a reviewer who spends 30 minutes or less on your proposal before rushing off to teach a class or returning to tomorrow’s paper deadline. That’s life. Remember that there is more to the process of proposal writing than what one person ultimately thinks of the proposed work (see above on the benefits of the proposal-writing process). What’s more, if you can’t capture someone’s attention in a few minutes of reading, chances are you need to work on distilling your message more. You should be able to draw the reader in with just a few sentences or a paragraph at most. A reviewer is obligated to start reading your proposal, but they are not obligated to finish reading it—get to the point as fast as possible (more below on what your point should be), and entice the reader to continue. Make it easy for the reader to digest the key points; use bullets and bold headings as necessary (this post is an example of such a writing style).
Tell a story with vivid contrasts. Everyone loves a good story. If you want people to enjoy reading your research proposal, then the proposal should tell a good story. The story, of course, needs to be of a certain type, and written in a certain style (I would not recommend the “mystery novel” approach, for example). One of my favorite recipes for telling a story is to set up the problem context, explain why the problem is important and hard to solve, and then draw a succinct, stark contrast between your approach and every other previous approach. For example, many research proposals I wrote about email spam filtering followed this recipe: (1) spam filtering is an important problem; (2) everyone else has been trying to filter spam by looking at the content of email messages; (3) in contrast, I will develop spam filters that discriminate good from bad based on the network traffic, without looking at the content of the messages at all. I’d then proceed to explain why this was a promising approach and likely to result in new breakthroughs (which it ultimately did). Your research proposal can and should also follow this recipe. Tell the reader what sets your work apart, and why it’s likely to succeed where others have failed or otherwise come up short. Effectively, you are painting your work as so promising and so different from everyone else’s approach that it would be foolish not to fund the proposed work, because nobody else is going to do the work, and not doing it could result in a missed opportunity for a breakthrough.
Answer the Four “Whys”: Why Important, Why Hard, Why Now, Why You? I think every proposal should answer the following four questions. Every proposal I write aims to answer these questions, and when I review a proposal, I also look for the answers to these questions:
- Why is the problem important? I heard a professor at MIT once tell a fellow Ph.D. student, “There are an infinite number of hard problems. You might as well work on one that’s worth solving.” In your proposal, it is important to convince the reader that there is a problem that needs to be solved, and, if your research is successful, it will result in solutions that will make the world a better place for some people. You might be ridding the world of a certain kind of cyberattack, designing better user interfaces for some class of technology, making it easier to debug software programs, or something else. There are many ways to advance knowledge and make the world a better place in doing so. You should first convince the reader that there is a problem out there that needs to be solved—in fact, you should convince the reader that the problem is too important to be left unsolved.
- Why is the problem hard? I’ll follow up on this more in a later post, but you should beware of industry “bulldozers”. The problem that you are working on should require deep thinking and insights, and possibly the application of tools and techniques from multiple disciplines. It should require a level of thinking that goes beyond the next couple of months (quarterly deadlines, etc.). If your problem does not pass this test, it’s likely that industry could solve your problem, and they could likely solve it better. Industry has the capability to hire armies of software engineers to rapidly churn out code. If the solution to the problem that you propose is a “simple matter of engineering”, and the problem is worth solving, then there is a strong risk that industry will solve the problem better and more quickly. Convince the reader that the problem that you are working on cannot (or will not) be solved by industry, and that investing money in research on the problem is the best (or only) way to solve the problem.
- Why now? Most research problems are not entirely new. You may think that you are the first to propose a particular problem. In many cases, however, problems are longstanding, and many people have proposed variants of the same problem before. For example, to return to the example of spam filtering: people had been working on the problem for at least ten years; why now is there a possibility to make headway on an old problem, where many others had attempted the same problem? The answer to this question may be a recent technological advance (e.g., the ability to monitor traffic at high speeds); it might also be the emergence of new technologies in other areas that bring new “hammers” to an old nail (e.g., a new machine learning algorithm that makes an old approach more tractable, efficient, or accurate). Regardless of what creates the “perfect storm” for doing the research at this time, you should aim to convince the reader that “things are different this time” because of recent advances, changes, etc., and that you’re equipped to take advantage of these new opportunities.
- Why you? I think this is perhaps one of the most important elements of a proposal, and one that is commonly forgotten. Why are you the right person to carry out this research? You may have convinced the reader that you have identified a hard problem that is worth solving, but if you are a networking researcher who has identified a hard and worthwhile problem in complexity theory (or vice versa), you will have a very hard time convincing the reader that your proposal should be funded. You must establish credibility, and convince the reader that you are qualified (and, ideally, uniquely qualified) to carry out the work that you have proposed. Establish your “secret weapon” that you will use to solve the problem that other people don’t have (e.g., domain expertise, a certain body of knowledge, collaborations with people in the appropriate discipline). Tie back to successes from your own previous work, where possible, and establish bridges between your old (successful) work and the new work that you are proposing to do. This aspect is where some delicate balancing comes in: You should lean on your past record to establish credibility for the proposed work, yet the proposed work should be visionary enough to encompass three-to-five years of future work. One way to do this is to include some preliminary work in the proposal to demonstrate that your vision is feasible and that you are qualified to carry it out. It’s worth noting that this is not the time to be modest. You’re not talking with friends at a cocktail party; you are selling yourself and your research. If you don’t sell your work, someone else is going to sell their work, and their sales job may edge your proposal out. We can be cynical about the need to give a sales pitch and promote ourselves, but the fact of the matter is that if you don’t do it, the other researchers who are competing for the same fellowships, grants, etc. will anyway, so you might as well put your best foot forward and so that your proposed work can be judged on a level playing field.
Be meticulous. Make sure your proposal is accepted or rejected for the appropriate reasons. Absolutely do not forget to include all mandatory sections of the proposal (e.g., the National Science Foundation takes education, diversity, and outreach extremely seriously; leaving out discussion of these aspects is almost certainly a showstopper for your NSF proposal). Don’t forget to read the fine print about certain things that reviewers expect to see; if a call for proposal explicitly asks questions, be sure to answer them. Perhaps most importantly, spell check your proposal, and have it read by a native English speaker before you submit it. Sure, we all have the occasional typo, but more than one or two typos suggests extreme carelessness and sloppiness. How can someone trust you to conduct your experiments carefully if you can’t even be meticulous with a short research proposal? Can someone trust your code or research results if he or she can’t trust your ability to proofread a short, simple document? Do not convey carelessness, ever. It is a surefire way to put off reviewers and set you back significantly. Running a spell-check is super easy, so there is absolutely no excuse for spelling errors.
Have fun, and enjoy the process. Hopefully these pointers will help you in the proposal-writing process. Not every proposal will win the fellowship or get funding. Remember that there are always factors that you cannot control. Like many things in life, the process is often as important as the outcome, and with these tips, hopefully the process of proposal writing can be both enlightening and fun.
[Update (October 23, 2013): From Craig Partridge]
I had a great chat with Craig Partridge, one of the Internet’s luminaries, chief scientist at BBN, and an all-around great researcher. Craig has spent a large part of his career at BBN, which submits proposals for government contracts regularly. He rightly pointed out that much of what I wrote above is accurate for academics who are applying to funding agencies like the National Science Foundation, but also that some funding agencies are extremely regimented about the format and content of a proposal. He offered the following tips for the proposal writing process, which researchers at BBN regularly practice when replying to government funding solicitations. Thanks, Craig!
“Two thoughts to the recipe for a winning proposal.
- A best practice for being meticulous is to create a checklist. Scour the solicitation for words like “must” and “should” and “required” and make those sentences into a checklist. Before you submit your proposal, confirm that every item on your checklist is in the proposal.
- Get outside reviewers. The common practice among corporate research proposers is to use a “pink team” and a “red team”. A pink team reads the solicitation and an outline of your proposal about 6 weeks before it is due and tells you where it sees intellectual or practical gaps in the outline. It is an early chance to find problems. A red team reads the proposal (preferably with your checklist and the solicitation as supporting material) about a week before it is due and gives you a list of problems that need to be fixed before submission.”
A distinguishing feature of a research career—particularly in academia—is the unstructured nature of the job. Graduate students, research scientists, professors, and postdocs are generally masters of their own time. Although this autonomy can be liberating, it can also result in tremendous inefficiency if one does not develop effective time-management tactics. There are countless books on time management, and it is impossible to provide a comprehensive compendium of time-management tactics in a single post. Hence, what I aim to do in this post is identify specific time management tactics that may be useful for academics (or anyone who works in an unstructured environment). The tactics I have compiled below are the result of much reading on this topic over many years, as well as empirically determining what works for me. Some of these tips are adapted from other readings, but most are simply tactics I’ve devised that seem to work well for me and I think might be useful for other academics and people in jobs where time is unstructured (and potentially unbounded).
Perhaps the most important characteristic of time that underscores the need for effective time management is that time is an asset that you are always spending, and it can never be replenished or replaced. You have plenty of other assets: possessions, money, and so forth, but other assets can be replaced and replenished. It is possible to save money or reduce spending; unfortunately, we have not yet figured out how to stop time, so we are always spending it,regardless of whether we want to or not. It is always possible to make more money or acquire more possessions—but time that is spent can never be regained. Thus, given that you are always spending time, the best you can hope to do is to always be making the best use of your time. A question to repeatedly ask yourself is: “Is this the best use of my time right now?” By continually asking yourself this question, you can often correct course and spend your time in the best possible way.
Strategy: Make a plan and prioritize
The cliched metaphor that “time is money” turns out to be extremely useful for thinking about time management. In planning how we spend our time, we can draw an analogy to financial goals. In the same way that you might have a financial goal that requires financial strategic action (e.g., buying a house requires a savings plan), you might have a personal or career goal that requires strategic planning (e.g., obtaining a faculty job requires achieving various subgoals, each of which require a particular investment of time). You may have multiple goals: in the same way that you may want to save enough money to buy a house and go on vacation, you might also want to have enough time to write the conference paper and spend time with your family. Achieving (and balancing) these goals requires careful, meticulous prioritization and planning. Given a plan, it is much easier to budget time and make strategic decisions. Continuing the financial analogy: without a budget, it can be difficult to know whether you can afford the expensive dinner and a weekend away, but a budget makes these questions easy to answer. Similarly, without a time budget, it may be difficult to determine whether accepting an invitation to review a paper is the best use of your time (and whether you have the time to do it in the first place). With a plan in place, these decisions become much easier.
Priorities are an important part of any time management plan. Decide what’s important in your life. What will be your top priorities? Some academics may make regular publication in the top conference the utmost priority. Others may place higher emphasis on technology transfer and entrepreneurship. Still others may place more value on teaching. Likewise, you may have career goals, such as landing a faculty position, or attaining a coveted fellowship. What is important to you? How important is it to you? Make some decisions and use those priorities to help you formulate a time plan. Although priorities naturally will differ from person to person, I might suggest one thing: Personal health, well-being, family, and friends should come above all other goals. Without these things in life, you will not be successful, nor will you be able to enjoy your successes. There’s no point to becoming wildly successful if your health deteriorates and you become unhappy or ineffective mid-career. Likewise, your family and friends count on you for support, and you will need their support when the going gets tough, as well. Without this support network, you will never be successful. Thus, above all else, take care of yourself and make personal interrupts a top priority. With that said, I’ll offer some tactics that I’ve found to be really effective for implementing my time management plans.
Tactics: Apply the “Five Bs”
Having a plan in place is helpful, but you also need tactics to help you execute your plan. This is where I find that there are certain time-management tactics that are particularly helpful in the academic environment. I call my tactics the “five Bs”: bits, budgets, buffers, bounds, and barriers.
Bits. Time is fluid and continuous and should be treated as such. Unfortunately, most of us have a tendency to schedule tasks into fixed, discrete time blocks that are generally too rigid and too large. For example, a common practice might be to schedule a meeting, lunch, or some other activity for an hour. Most of these activities don’t take an hour, so time is wasted “ramping up” and “winding down”, since activities tend to fill the time allotted for them, regardless of how much time they actually require. If something happens to end early or take less time, we end up wasting the extra allotted time. For example, if a meeting ends “early”, we might use the extra found time to get a coffee, browse Slashdot, or otherwise spend time in a way that is not the best use of our time (thus violating the main criteria). To solve this problem, I recommend viewing time as a much more fluid resource, or at least one that can be spent in smaller bits.
- Maintain a list of tasks that can be accomplished in short time “bits”. Most tasks do not take an hour, and often significant progress can be made in a very small amount of time. When you find that you have a bit of time (e.g., ten minutes), use the time to knock something off your list. Note that bit tasks need not be insignificant: You can take a larger task and divide it into bits. For example, instead of having a task on your to-do list such as “write the conference paper” or even “write the introduction to the conference paper”, you might have a task like “write a paragraph for the introduction of the conference paper”. These bit tasks are less daunting (thus, easier to get started on), and you can do them in the little bits of found time throughout the day. You will be surprised how many “time bits” you have during the day and how much you can accomplish by breaking tasks into these bits.
- Before taking a break, use a time bit to start a new task. One of the most difficult aspects of getting things done is getting started. I find it can be incredibly difficult to get back “into the zone” after taking a break, or when shifting from one task to another. Therefore, I try not to align my tasks on discrete boundaries like hours. Rather, I use my time bits to start a new significant task. Consider the following: You have ten minutes before your next meeting. You could take a break, talk to your colleague, get a coffee, etc.—or you could use the ten minutes to get started on your next task. If you have a smaller task on your “time bits list” that involves getting started on a bigger task (e.g., writing the first paragraph of the paper intro), then you can use the time to get started on a task, which is often the hardest part. When you come back later (e.g., at the top of the hour, after your meeting ends), you’re already started! I find that using time bits to make a concrete (however small) start on a larger task almost eliminates the cost of context switching later.
Budgets. Spend your time well. This does not mean that you have to work nonstop (see below on “barriers”, for example). What it does mean is that you should be purposeful in how you are spending your time. Your purpose may be to make progress on a conference paper, a piece of code, or some other task; or, your purpose might actually be leisure or relaxation. The point isn’t to work yourself into the ground, but rather to always have a purpose with how you are spending your time. With this in mind, I apply two specific tactics:
- Have a purpose. Always have a goal, and spend your time with that goal in mind. You need long-term goals and short-term goals. For example, a long-term goal might be to finish a conference paper, achieve a promotion, or train for a marathon. A short-term goal (e.g., for an afternoon) might be to finish an assignment, piece of code, or section of a paper; or, it might be to drop work entirely and recharge. Have a goal and be purposeful about how you spend time in pursuit of that goal.
- When it comes to meetings, demand an agenda. This tactic is almost the same as having a purpose, but it is particularly useful for graduate students and faculty members. I now refuse to attend meetings for which there is no set agenda in advance. An agenda gives clarity of purpose to a meeting, and it is also a plan for how the time will be spent. It makes the need for the meeting clear, it allows for immediate and efficient use of meeting time, and—most importantly—it makes it clear when the meeting is over. Once the agenda is complete, the meeting is over. If it takes an hour, that’s fine. If it takes ten minutes, that’s also fine. Without an agenda, a meeting can drag on to fill the time allotted; this steals time away from your opportunity to accomplish tasks with time bits. Use agendas to make efficient use of meeting time.
Buffers. It can be tempting to pack meetings back-to-back, one after another on hourly boundaries. This is, in my opinion, an awful way to schedule time. Instead, I try to use time buffers to make better use of my time:
- Create time buffers in between scheduled activities. I recommend scheduling a 50% time buffer for any activity. If you think a meeting will take 20 minutes, schedule 30. If you think an activity will require 60 minutes, schedule 90. This rule applies to pretty much everything: meetings, dentist appointments, dinner engagements, etc. We can sometimes have a tendency to pack engagements tightly, but this often results in stress, lateness, and frantic thought and action. There may be some apprehension about scheduling time buffers: one might think, for example, that if a 90-minute slot is scheduled for a meeting that only goes 60 minutes, then the extra time is “wasted”. To the contrary! Applying the time bits strategy above can make these extra found times in the time buffers incredibly useful for accomplishing important tasks.
- Show up early. If you have time buffers in between activities, you can actually be early to your next activity, rather than frantically running from one thing to the next. This ensures that you are composed and focused for your next activity, rather than frazzled and behind. Showing up early also need not involve “wasting” time if you have useful items on your time bits list that accomplish if you happen to be five or ten minutes early.
From a personal perspective, having a time buffer is also a good idea: it can take a bit of time to “change modes” at the end of the work day (or work week); your family will appreciate it much more if you buffer some time to change modes.
Bounds. Creating time bounds is perhaps one of the most important time management tasks that an academic can learn. The academic lifestyle can go unbounded. It is always possible to write another paper, perfect the lecture notes further, write another proposal, and so forth. The sky is the limit, and the sky is boundless. It can be tempting to continue to say yes, to keep working after diminishing returns have set in, and so forth. It is critical to set bounds. Here are some tactics that I use for setting bounds:
- Do not let the perfect be the enemy of the timely. I encounter this problem regularly and I am guilty of violating this principle myself. An example where this creeps up in academic life is paper reviewing. It’s possible to write ultra-thorough, thoughtful reviews and spend hours fine-tuning and perfecting your feedback. But, at some point you hit diminishing returns. And, if you don’t manage your time wisely, you will be late with your reviews and annoy the people who are counting on you. I have been guilty of this myself. Try to keep in mind that sometimes it’s more important to be timely than it is to be perfect. Plus, being perfect is not attainable, but being timely is.
- Use a deadline as a bound for declaring victory; create a deadline if one doesn’t exist. Sometimes deadlines are explicit (e.g., review deadlines, paper deadlines). If you are working on a task that does not have a bound (e.g., a journal paper), make one. Declare a date or time by which you will be done and stick to it. This will ensure that you do not over-optimize. There is a definitely 80-20 phenomenon in research. Setting deadlines will help you ensure that you are not stuck in the realm of diminishing returns. Set deadlines and bounds on the lengths of meetings, how much time you will spend on an email thread, etc. Do not be afraid to declare victory once the time is up.
- Beware of time thieves, particularly those that masquerade as “productive activities”. Many interrupt-driven activities steal our time in fits and spurts. Email is perhaps the most notorious of these thieves. Of course, sometimes an email thread is useful: correspondence with a colleague can result in refinement of an idea, clarity of thought, and so forth. But, this is the exception. Most of the time, replying to email is a Sisyphean task that simply generates more email. By the end of the day, we’ve done nothing but reply to email, sometimes to limited effect. Be particularly wary of email threads without a clear purpose, and look out for email threads that will resolve themselves without your involvement. I make it a point to identify whether an email might resolve itself. If there’s even a small chance the issue will resolve itself, I will wait to reply. (We’ve all seen this phenomenon: Someone sends you a frantic email about some issue that has to be solved now, only to send you an email 30 minutes later letting you know that it was all a mistake or that they’ve figured out the problem themselves. I clear these threads from my deck with one stroke of the delete key!) I also try to identify whether email is the most efficient mode for resolving the problem. Just because someone contacts you by email does not mean you have to reply, and just because someone contacts you by email does not mean you need to reply by email. For example, sometimes a phone call, IM, or in-person chat is faster and more effective. Also, I recommend only replying to email at certain times (e.g., morning, lunchtime, and end-of-day). You might think that people “expect you to reply right away”. Don’t worry—when you stop replying immediately, people stop expecting an immediate reply. If something is truly urgent (i.e., can’t wait a couple of hours), people will find another way to contact you. Most things are not that urgent anyway and can wait a few hours for a reply.
- Be purposeful about online (and other passive) “leisure” activities. Sure, we all like to browse the web from time to time. But, even for these types of “leisure”, it helps to be purposeful. Leisurely is not the same as aimless. Social media sites like Facebook are particularly amenable to aimless time-wasting. Track your time on these sites using tools like StayFocusd, TimeStats, and so forth. Upon more careful accounting, you might end up finding that the cost of the time wasted far outstrips any “benefit” that these sites might provide. If there are sites where you find yourself spending lots of aimless time, consider blocking them or canceling your account entirely. Similarly, be wary of other passive leisure activities such as watching television. This is not to say that you should never watch television, although that might be a safe bet; rather, have a purpose (watch a particular show, sporting event, etc.). Shut the television off when you’ve achieved your goal (by the way, even “vegging out” might be a reasonable short-term goal, but you should still be purposeful about how long you want to do that for!).
Barriers. Establish times for specific activities, and be ruthless about enforcing the barriers between those activities.
- Unless there is an emergency, ruthlessly protect your scheduled time. Students or colleagues will sometimes ask me if I “have a minute”. The first thing to recognize is that nothing ever takes a minute. Even if something actually only takes a minute (which it almost never does), it creates a context switch and interrupt that can disrupt your flow and actually cost you 10-15 minutes (an entire time bit, during which you could have knocked off a task). I used to think that saying “no” was rude; however, it is a perfectly reasonable answer. On the contrary, if you let someone encroach on a planned meeting or activity, it is unfair to whatever or whomever you had previously scheduled for that time block. Do not let people encroach on activities that you’ve planned unless there is a real emergency.
- Ruthlessly protect your personal time. This is effectively the same rule as above, but re-stated and re-emphasized for your personal time. During non-work time, devote your complete attention and energy to not doing work. I make it a point to avoid looking at my phone when I am spending time with my family. Nothing ruins a family picnic in the park and raises one’s blood pressure like an email from your colleague on a Saturday afternoon telling you that the introduction you wrote for the paper is “all wrong” and, by the way, your text has been chucked out and can we meet “ASAP” to discuss it. Do yourself a favor and compartmentalize. Not much needs to happen on a Saturday, in particular, and if you don’t get back to your colleague, department chair or whomever before Saturday night or Sunday or even Monday morning (presuming there’s no immediate deadline, of course), life will go on just fine. On the contrary, you can’t have the family picnic on Sunday night or Monday morning.
- Ruthlessly protect your personal space. I now establish spaces in the house for working, and spaces where work is off limits. This also helps me establish time barriers. I make it a point to leave electronics out of the bedroom, for example, to reduce the temptation to “work” (i.e., which is never real work anyhow but rather checking email…see above) when the purposeful use of my time should be getting rest or sleep. Creating these physical barriers helps ensure that there are barriers to protect your personal time, as well. Another place I enforce barriers on personal space is when I go running: I go for long runs as part of marathon training (sometimes for hours at a time) with absolutely no electronics. Nobody can bother me for a work-related request during these times. I am simply not reachable; that time is mine. The personal space I have created helps me enforce the barrier on my use of that time.
- Learn how to say “no”. I admit, this is incredibly hard for me. It’s also incredibly hard for many assistant professors or junior faculty seeking promotion. With muddy bars like “tenure”, it can be incredibly difficult to determine how much one needs to do to clear the bar, and what tasks are enriching versus superfluous. Additionally, overachievers may tend to have difficulty saying “no” and quickly find themselves overcommitting. Overcommitment is a potential catastrophe, because no matter how well you manage your time, there will be no way to fit all of the tasks into a fixed number of hours (using the financial metaphor, you may reach a point of bankruptcy if you over-spend). I am not quite sure of the best way to learn how to say no, but I am currently trying two different strategies. The first (which I’ve been using for awhile) is to find several colleagues whom I trust and ask them if it’s OK to say no to something. Getting multiple opinions is useful here—certain people may have biases or ulterior motives, even the people you trust. Calibrated, trusted opinions from senior colleagues and mentors are invaluable. The second approach I have started to use is an accountability and reward system for saying “no”. Every week, I catalog the opportunities that I have declined. I make sure that this list has several items every week. If I’m not saying “no” enough, I re-evaluate. For the ambitious overachiever, I find that this tactic is useful—actively documenting activities to decline can become a game or a challenge. How long is your list?
As I mentioned, there are countless books on time management and in some ways, this post represents “yet another list”. However, I find that many of the tactics above are specifically useful for academics or those who work in environments where time is unstructured and working hours can potentially go unbounded. Many of these techniques have worked well for me, and they’ve taken me years to learn and refine. Hopefully some of the tactics in this point will save you some time in the future!
The past several years have seen a few blog posts concerning careers in academia and industry; much of the debate has been colored by high-profile departures of university professors for a home in industry. Notably, various former professors have bemoaned the difficulty of obtaining funding, the difficulty of doing large-scale systems research, quixotic panels and program committees, the lower salaries, and so forth.
Although there is always some truth to the complaints you may have read about, I also feel that these previous posts paint a somewhat one-sided view of the nature of academic life. These posts are often enough to send a graduate student running for industry, thinking that the academic life is not for them. I believe that the picture between industry and academia is a bit more nuanced.
I should provide some context for my comments, which will hopefully also lend some credibility. I’ve now spent about seven years as a university faculty member. I have also worked in at a small startup, large industrial research labs, and at several universities (as an undergraduate, grad student, postdoc, and now as a professor). One of my first jobs was as a programmer for LookSmart, a directory-based search company that was eventually bought by AltaVista; I wrote LookSmart’s first web crawler, back when it was a lot easier to crawl the web. I’ve also spent time at both Hewlett-Packard Labs (where I wrote my first research paper), and I spent a summer at Bell Labs, as well.
Ph.D. students and mid-career researchers alike may wonder about the choice between academia or industry. Here are some of my thoughts on the tradeoffs, based on having spent a reasonable amount of time in both places. First, let’s have a look at some of the things that academia has to offer:
- Freedom. Perhaps one of the most appealing aspects of a job in academia is the freedom that the position offers. A professor sets out the day’s agenda and generally has complete control over how to spend the day. For example, I can determine exactly how much time I want to spend meeting students, writing papers, preparing classes, sleeping, and so forth. I don’t typically have to answer to anyone for how I’m spending my time. This is an extraordinary advantage, because it means that I can allocate my time based on the tasks where I believe that I can derive the most value (and happiness). Of course, this freedom also means that a professor needs to become an expert at time management (more on that in a later post). Another advantage of this flexibility that some professors take advantage of, too, is that they can determine when they want to work; it’s not necessary to work 9-to-5, so if it’s necessary to run an errand during the day, that type of thing is generally completely possible.
- Working with students. One of the biggest differences between industry and academia is the opportunity to work with students, of all stripes. I very much enjoy teaching—one of the main reasons I wanted to become a professor was that I deeply admired my own university professors and wanted to emulate them. Working with students is probably one of the most awesome things about being a professor that is hard to replicate in other jobs. As a classroom instructor, you have the opportunity to interact with a large number of incredibly bright people, who are continually asking questions that shed new light on problems you’ve been teaching for years. Recently, with the opportunity to teach online classes, the opportunity to reach a large number of students has presented itself, which is even more amazing—we now have the opportunity to teach (and influence) an entirely new demographic on a massive scale. As an advisor of Ph.D. students, a professor can shape the professional (and research) development of many students over multiple years. Perhaps even more rewarding is that a professor can learn regularly from students; I have learned many things about a variety of different areas (e.g., statistics, machine learning, software engineering, networking, operating systems) directly from my students; learning from one’s students turns out to be a really exciting (and efficient) way to learn. Working with students and helping them develop research taste, presentation skills, and life skills is simply one of the most rewarding aspects of the job. An industrial researcher may offer that they have the opportunity to work with summer interns—sorry, but it’s not the same. Industrial researchers typically work with a summer intern for a short number of months, not a span of many years (and certainly not continuously over many years). The relationship with the student is not as deep, and (as I have observed many times when my own students take internships) an industrial researcher is typically not as invested in a student’s success or overall career path. There are exceptions—my mentor at one summer internship became a second advisor to me and ultimately became my postdoc advisor, but it stands to reason that the relationship that a professor can have with students cannot be replicated in any other profession, and it is one of the most awesome aspects of the job.
- Easy interaction with experts from other fields. The university makes it incredibly easy to approach experts in other areas. Most universities will not hire an army of experts in a single area, which means that the faculty member down the hall from you (or even right next door) might be an expert in a completely different subject matter. This environment creates an extraordinary opportunity for learning, cross-disciplinary thinking and research, and general enrichment. While it’s always fun to talk to people in your same research area, often the best ideas come from applying ideas across disciplines. The university environment makes this type of interaction much more natural than other environments do.
- Opportunity for long-term impact. As a professor, you do not have quarterly deadlines to meet, monthly reports to file, or a boss that you are regularly accountable for. (In truth, a few funding contracts do have monthly reports, but those are few and far between and generally they are pretty lightweight.) Professors generally have to report to their funding agencies about progress on research projects on an annual basis, but even these reports are typically fairly lightweight and involve describing the research that was carried out over the course of the year—there is very little pressure to meet a specific target or make a particular deadline. Faculty drive themselves to excel (the flip side of this, of course, is that to be a good professor, you really have to have initiative and be a self-starter, because nobody is going to crack the whip on you). Freedom from the constraints of short-term deadlines and having to answer to others who are setting the agenda really presents the possibility of thinking about the “right” solution to a problem (or the right problem to work on), rather than simply hacking together something that just works well enough to get the job done. In my opinion, this aspect of being a professor presents the greatest opportunity for adding value: people working in industry often do not have the time (or interest) in stepping back and thinking about completely new ways of solving a problem, and they may also not be able to bring in completely new perspectives to problem solving. They are often way too busy “fighting fires”. I have found many of my research problems by visiting people who work in industry, asking them about “pain points”, and thinking about solutions to the problems that they face that they simply cannot implement because they don’t have the time to step back and think about a completely new approach.
- Relative stability. It almost goes without saying that tenure is unique to academia; that level of stability offers both peace of mind and the freedom to take risks (something I have taken advantage of; for example, post-tenure, I have taken on much bigger systems-building projects that have taken several years to produce papers). Industry labs are typically not around forever. Many of the industrial research lab “giants” of the past are shadows of their former selves. Even if a research lab does not disappear completely, it can be realigned with a business unit to the point where the “research” becomes advanced development and the employees of that unit lose their relative autonomy. Even in my relatively short research career, I have seen several research labs disappear, often with very little warning. Although one of the oft-stated benefits of working in a research lab is that there is no need to raise funding, the flip side of this feature is that when the funding disappears, you lose the perks that you once enjoyed. Regardless of whether you are in industry or academia, you are always at the mercy of your funders. Personally, I would at least rather hold the pursestrings myself, so that I can determine my own research agenda, the scale of the research program that I would like to pursue, and so forth.
- Bridge-building. A professor is not beholden to any particular person, dogma, ideology, company, or other alliance. This stands in contrast to industrial researchers, who have a mission to contribute to the company’s overall value and also have some amount of loyalty (and ties) to that company. As a professor, I have no such ties. I’ve had meetings with Microsoft in the morning, Google in the afternoon, and Amazon in the evening in a single day. In these meetings, a professor can learn amazing things about industry’s view on problems. It may be difficult for an industrial researcher to get the same perspective: it’s unlikely, for example, that an employee from one company will get to hear about its competitor’s approach to a problem. In contrast, if they seize the opportunities, professors can learn an incredible amount from a diverse body of knowledge and have the opportunity to synthesize that knowledge in ways that others can’t.
- Outreach. Activities that transfer research results to the broader society are part of a professor’s mission. A professor has the opportunity to give tutorials, develop online courses, write blog posts, teach at local high schools, organize and run workshops, and so forth. This is not to say that the same opportunities wouldn’t exist in other research positions, but outreach is often not as central to the mission of other jobs as it is to being a professor. For example, every National Science Foundation grant that I write explicitly asks how I will transfer the results of the research to society; this type of outreach is thus very explicitly part of my mission, and I truly enjoy the opportunity for outreach.
- Summers. Someone once told me that the top four reasons to become a professor are May, June, July, and August. Summer offers huge chunks of unstructured time to pursue larger projects and independent interests. For example, over the past summer, I escaped Georgia Tech for several months, ensconced myself in Cape Town, and used the focused (and relaxed) time away to prepare a massive, open online course on Software Defined Networking, a topic that I had long been yearning to dive into the details on.
I cannot speak as much to the benefits of working in an industrial research lab, as I have less experience, but from my limited experience, I have observed that an industrial research lab has the following benefits:
- Structure. In contrast to a professor’s day, which is completely self-determined, time in an industrial research lab is more structured, typically because interaction with peers (as opposed to students) is more common, and as a researcher you are less of an independent actor. You may have specific team meetings that are set up without your control that occur at the same time every day, for example, or regular meetings with your manager. For some people, this type of structure can be extremely helpful for their productivity.
- Money (when it exists). When times are good, there is no place like a research lab—money flows like honey, and equipment, travel, interns and so forth seem to spring from an infinite well. Unlike professors, industrial researchers do not have to interrupt their rhythm to write grants, visit funding agencies, attend “principal investigator (PI)” meetings, and so forth. Although I personally enjoy the process of writing grants, as it gives me the opportunity to think about the big picture of my research agenda, it certainly does interrupt the flow of work, and it can be distracting, since the scheduling of funding deadlines is beyond my control, and they can certainly come at convenient times. Of course, when money dries up, there is probably no more frustrating place to work than a research lab; when funding disappears to the point where your company won’t send you to a conference to present your paper and you have to travel on your own dime, the once-good life can start to look pretty dire.
- Direct and tangible impact. Professors can often struggle to have their ideas adopted in practice; in fact, I find that I spend a lot of time “evangelizing” my ideas to industry to try to transfer them into practice, where they can have a more lasting impact. Industrial researchers do not face this problem: the problems that that industrial researchers work on are often directly motivated by business units. Although this does remove some autonomy, the positive aspect of the lost autonomy is that the research results are often immediately and directly useful. Researchers at industrial labs such as Microsoft and AT&T can regularly cite examples of the algorithms or methods they invented finding their way into products that thousands (or millions) of people use everyday. It is much more difficult to have that kind of impact as a professor; it is not impossible, but it requires a lot more will, and a lot more initiative.
- Problem focus. Professors often work on many different problems at once. I find this breadth refreshing and invigorating, but it certainly comes at the cost of depth. A researcher in an industrial lab typically has much more time to think deeply about a problem, hack on code, and directly contribute to the solutions that find their way into published papers. Typically, as a professor, the great insights, code, and so forth that find their way into papers are the work of students, not the work of the professor. A professor often simply does not have the time to be as deeply involved in the details of solutions. This, of course, can vary depending on the style of the professor: some professors, for example, advise only one or two students at a time and work very deeply on problems with their students. But, typically, this is not the norm, and even in these cases the student still has more time for focused work. In contrast, an industrial research does have the opportunity to spend many focused hours on a single problem, working with much the same level of focus as a graduate student might.
- Time to do technical work. Professors are distracted with many different tasks—teaching, committees, fundraising, advising, and so forth. Although these activities are often extremely rewarding, they are certainly often non-technical. They are a far cry from hacking on code, for example. Some people appreciate the opportunity to spend countless focused hours on technical work. For a professor, that kind of time is an incredible luxury: it presents itself from time-to-time, but it is certainly not the norm.
Ultimately, the choice between academia and an industrial research lab involves many tradeoffs, and the best “fit” often depends on individual preferences and working style. I personally think that being a professor is perhaps the best job one can have. There are certainly drawbacks, and other jobs certainly offer different benefits and perks that jobs in academia do not offer, but the job is far better than one might otherwise be led to believe from recent posts on academic departures. One can observe that, although there have been several loud departures for industry, there still remains a very large cadre of extremely happy university professors.