I remember applying for NSF’s Graduate Research Fellowship many years ago and being asked to answer a question describing my experiences “integrating research and education”. At the time, I was baffled by the question, as I hadn’t yet done much teaching. I thought: Aren’t teaching and research orthogonal? I’m told by current students that the question no longer exists in the fellowship application, which I think is unfortunate. That question has stayed with me throughout my career: I regularly re-ask myself questions about integrating research and education.
At least in the United States (and presumably elsewhere, too), university researchers are regularly asked to tie our research back to education: for example, faculty members are regularly asked to describe the “broader impact” of their research, which includes how the results of the research will be incorporated into the curriculum. I’ve learned that this is no accident; to the contrary, I think it is one of the most important (and under-appreciated) things that researchers should be thinking about.
Although researchers are sometimes asked to think about how research can be integrated in the classroom, I’ve also found that efforts in the classroom can also ultimately result in better research. In fact, although many educators are not necessarily researchers, the converse is undeniable: It is no accident that some of the best researchers are also excellent teachers. And, while some strong researchers who are not good teachers do exist, I believe that purposeful teaching effort does in fact result in much better research.
In this post, I’ll describe my views on the relationships between research and teaching, in both directions. I’ll begin with the more “obvious” notions of how our research ultimately affects education and the curriculum and continue to what I think is the less apparent (and more interesting) direction of how our work on education can also make us better researchers. Of course, teaching also helps us develop many “general purpose” skills that are also useful in research, including mentoring and supervisory skills, learning to analyze others’ understanding, learning to give feedback, and so forth. Below, I’ll eschew these practicalities and instead focus on how the relationship between research and education ultimately result in better research ideas.
How Research Affects Teaching
Research results instill fresh material in the classroom. Although some subjects we learn in the classroom are fairly well-established, many areas of computer science (and I would assume certain other fields, too) are rapidly evolving. With the rise of large content and service providers such as Google, Amazon, and Facebook; the proliferation of mobile devices; and the spread of connectivity to developing regions (to name a few developments), computer networking looks almost nothing like twenty years ago, and, while certain principles persist, the constraints of the domain and the applications of the technologies are continually evolving. Students strive for concrete examples and applications of concepts to the world that they know which is, incidentally, different from the world we knew when we were students. New research results represent prevailing theories, the outcome of our cumulative understanding, and the application of concepts to the most relevant problem domains or our time. I find that there is no better way to keep my course material current than to peruse the latest research and update the material so that it reflects current understanding.
Industry tracks research; students should, too. Our understanding continues to evolve as new research results emerge. In many areas, industry aggressively tracks new technologies and research results, and students aim will be more poised to make important contributions in industry if they are well-versed in current technologies. Students periodically thank me for covering a certain topic or concept in the classroom because “someone asked me about it in a job interview”. Certainly, there is a balance between educating our students on the big picture and “timeless” concepts (something I discuss more below), but I find that students are often quite grateful for having some exposure to the concepts and problems that industry is thinking about today. Instilling course material with fresh research results is one important way that instructors can help this process.
How Teaching Affects Research
I think the more surprising notion is that investing effort in teaching well can actually make us better researchers. I sometimes find that certain faculty members are too eager to minimize teaching responsibilities in favor of “leaving more time to get research done”. Now, it is worth acknowledging the source of this angst: many of the administrative aspects of teaching (e.g., grading, responding to student emails, organizational logistics) are incredibly time consuming and do not necessarily offer inherent benefits to research. Nevertheless, I find that the intellectual aspects of teaching are an indispensable aspect of my own efforts to become a better researcher. Below, I’ll explain more abstractly why I think teaching makes us better researchers, and, where appropriate, I’ll describe some of my own concrete experiences in this regard.
To create new knowledge, we must first master the existing body of knowledge. Research is the process of creating new knowledge. Making progress in creating knowledge requires a significant amount of background knowledge, before one can reach the “frontier” of a topic, where the interesting questions are. Herb Simon once attested that it takes about ten years of experience to get to the point of great accomplishment in any one area, simply because it takes a significant amount of time to accumulate knowledge in an area. This necessarily implies that we can’t become great researchers in a subject area merely by taking a class (or even a few classes); we must embed ourselves in that topic area. I find that teaching a subject is perhaps one of the most efficient ways to become embedded in a subject matter, since the process of explaining concepts to students leaves no room for “cutting corners” in my own understanding. The process of building understanding in a particular area allows us to develop a deep understanding the paradigms and theories that currently exist, and how those paradigms and the existing knowledge base might be extended (or amended). Teaching Ph.D. students about a particular subject matter is also a way to bootstrap research, by helping our students get to the frontier of knowledge more quickly than they otherwise would; I sometimes teach seminars on cutting-edge topics (above and beyond my teaching “requirements”) simply because I find the process to be an efficient way of helping students quickly ramp up on a topic where I would like to see more research happening.
On a personal note, I found the process of preparing a Massive Open Online Course (MOOC) on Software Defined Networking over the past summer tremendously helpful in solidifying my own knowledge in this budding topic area. This particular sub-field has seen rapid developments over the past five years, and I had found it difficult to take the time to deeply understand many of the latest developments. I found that teaching the course was a wonderful “forcing function” to familiarize myself with new technologies and ways of thinking, and to gain hands-on experience with tools that had been recently developed. My hands-on experience with development tools helped me in two ways: First, I was able to suggest better tools for my students to use in their own research; in several cases, students who had been “stuck” using older technologies quickly familiarized themselves with technologies I learned well enough to appreciate. By investing time to deeply understand how new techniques and technologies might be applied, I was able to make connections between problems we had been trying to solve in the research lab and tools that could be useful for solving them. Second, I was able to make connections between concepts that had recently been developed to help solve some problems that we had been working on that hadn’t yet been solved. In one case, for example, as I taught concepts about composition techniques for network policies, I realized that the techniques could be applied to help some of our own technologies scale to much larger networks, which provided a breakthrough on a problem that we had been thinking about for years.
In the process of explaining an existing phenomenon, you might discover that existing explanations, technologies, or theories don’t actually suffice. According to Thomas Kuhn, research breakthroughs often occur when old paradigms are discarded (or at least amended), thus changing our way of thinking about problems completely. New paradigms begin with the need to explain or treat facts or situations that existing paradigms don’t handle well. As instructors, when we attempt to explain various facts or situations to students, we sometimes find that we can’t explain why things are a certain way—our attempts to explain may reveal instances that are not handled or explained well by current paradigms, thus exposing glaring needs to develop new technologies, theories, and paradigms.
I remember my experiences as a teaching assistant for computer networking, as my advisor and I planned lessons to teach Internet routing. My advisor had long worked on problems where correctness properties and bound were well-defined (e.g., Internet congestion control). When we came to the topic of Internet routing, however (a topic on which I had some mastery as a result of a summer internship), I found him continually asking me how (or whether) Internet routing offered any guarantees of correct behavior. How could we be certain that Internet routing algorithms would actually send traffic where it was supposed to go, for example? We realized in our attempts to codify this in lecture material that no such guarantees existed! Frustrated by our inability to explain Internet routing correctness, we spent the next several years formally defining correctness properties for Internet routing and developing tools that checked Internet routing configuration for correctness. The work eventually resulted in tools that were used by hundreds of network operators and a best paper award at a top networking conference. When I think about that work, I regularly trace its success to my teaching experience with my advisor, and our initial frustrated attempt to explain some seemingly basic concepts about networking to students. If it weren’t for that teaching experience, I think that research probably would never have happened.
Teaching encourages us to think about the long road, the big picture, and what “really matters” about a particular research contribution. I aim to explain why something is the way it is, beyond simply explaining a concept. As I explained above, efforts to explain why something is the way it is might sometimes fail to produce a good explanation, opening new possibilities for research. In other cases, research may offer solutions to a problem du jour, but sometimes research projects or papers are fairly self-contained, and it takes additional thought to really establish why (or whether) a particular result has broader implications that a student might care about. As an instructor, I strive to think about the big picture, and why a student should care about a particular research result, theory, or concept five or ten years down the road, long after they have left our classroom and received their degree. This exercise of thinking about broader implications can make classroom material more palatable to students, most of whom won’t specialize in the particular field you happen to be teaching. But, it also forces us as researchers to step back and think about why the problems we are working on have broad impact and why they matter to society at large. Explaining to a classroom of students why a particular result matters is perhaps one of the most useful exercises for distilling a research contribution to its essence.
Motivated Students + Inspiring Teachers = Great Research
I admired my university professors and wanted to emulate them; they are one of the main reasons I wanted to become a university professor in the first place. Teachers can influence and affect a large number of students in tremendously positive ways. Indeed, giving students the thirst for knowledge to the point that they want to not just consume existing knowledge but make discoveries themselves is a unique opportunity that we have as educators. And, certainly, developing smart young students into the researchers of current and future generations is yet another way that our efforts in the classroom can pay long-term dividends for research.
In this post, I’ll discuss paper selection—how a program committee considers a set of papers, each with a collection of reviews, and produces a program for a conference.
Paper Reviewing vs. Paper Selection
It’s first important to note that the paper review process and the paper selection process are related, but paper reviewing and paper selection are actually two separate events. There is a human process behind paper selection that involves distilling the reviews of a paper and ultimately making a determination that a paper should be published, but the reviewing and selection have many independent aspects. Sometimes, a paper’s reviews may appear positive, but the paper may still be ultimately rejected (or, the reviews may appear negative but the paper is accepted in spite of the reviews). More commonly, papers tend towards the middle of the review distribution: when reviewers score papers, they tend to regress towards middling scores such as “weak reject” and “weak accept”, rather than taking a strong stance. A significant fraction of papers thus end up with almost the same average “scores”, with only a handful of papers that should clearly be accepted. The number of clear accept papers is never enough to fill the entire program. Suppose, for the sake of example, that a conference can accept 40 papers. The conference may receive several hundred submissions; in most cases I’ve observed, about ten of these submissions receive uniformly glowing reviews. The next best 30–40 papers often fall into a rough equivalence class with the next best 30–40 papers, leaving anywhere from 60–80 papers that are “good enough to be published” for about 30 spots in a conference program. The job of a program committee is to create a smaller program from this larger set of papers.
Whenever a larger number of roughly equivalent candidates competes for a smaller number of positions, some amount of randomness can and will ensue. Much of this randomness is completely beyond the author’s control. As an author, the best way to avoid being subjected to the randomness of a program committee is to write your paper so that it is among the ten best submissions to the conference. Unfortunately, this is far easier said than done, and it’s not really possible to engineer this, although following various research and writing tips can increase the likelihood that your paper ends up in this class. (Speaking from experience, I can say that those tips aren’t fail-proof—even the best researchers I know routinely have papers rejected—but you can at least improve your odds!)
Unfortunately, unless you can guarantee that your paper always falls among the very best papers (a secret that nobody has yet succeeded in unlocking), much of the randomness that results from the paper selection process is out of your control as an author, but other factors can help create a sane selection process. The program committee chair (or chairs) have tremendous power for creating a sane process. These roles include setting the right tone for reviewing and mindset for paper selection, ensuring that papers are reviewed by the right program committee members, selecting the papers that will be discussed at the program committee meeting, and reducing the effects of program committee psychology, such as paper discussion order (e.g., the weight of one biased or incorrect review, and so forth). I will describe more details and tips for program committee chairs below, based on my own experiences and observations.
The Intermediate Dispositions of Papers
Before the program committee meeting itself takes place, the program committee chairs must determine whether each paper should be accepted or rejected without any discussion, or whether the paper should be discussed at the meeting itself.
Before the meeting: Selecting papers for discussion. Program committee chairs typically select papers for discussion at the program committee meeting based on their assessment of the paper’s overall quality, based on review scores, their interpretation of the paper’s reviews, and their assessment of any subsequent “online discussion” that may take place before the physical program committee meeting (most conference reviewing systems allow reviewers to discuss each other reviews in the system itself via asynchronous, email-like messages). Program committee chairs will often try to ask reviewers to reach consensus about whether a paper should be discussed at the program committee meeting before the meeting.
Best case: Acceptance without discussion. If you are very lucky, your paper will not be discussed at the program committee meeting and will be accepted without discussion based on high review scores. I think this is the best possible outcome, because a paper always has flaws, and the more a paper is discussed, the more likely that someone at the meeting will hear something they don’t like about the paper and raise an issue with it. I have seen instances where a paper that receives uniformly high scores is rejected because it is brought up for discussion and someone (often who hasn’t yet read the paper but might read it during the meeting) hears something they don’t like. Such a discussion doesn’t mean that your paper will be rejected—a good PC chair will ensure that these dynamics don’t wrongly result in the unjust rejection of an otherwise good paper—but it does mean that your paper may ultimately be subjected to some of the dynamics I discuss below. Unfortunately, as I mentioned, you can’t guarantee that your paper falls into this category, but you can sometimes be lucky, and you can certainly take many steps to increase your odds.
Next best case: Discussion. Otherwise, if you are still reasonably lucky, your paper will be discussed at the program committee meeting. Discussion represents hope for acceptance, and typically because there are a large number of papers that are of roughly equal score or rank, if your paper is being discussed, it generally has as good of a chance or being accepted as most of the other papers being discussed. The program committee chair will set the order in which these papers are discussed, which can sometimes play a surprisingly significant role in whether a paper is accepted or rejected.
The Dynamics of the Program Committee Meeting
The program committee meeting itself involves the winnowing of a larger set of papers (say, 60–80 papers) down to a final program of, say, 30–40 papers. Typical math I’ve seen in systems and networking conferences is that while overall acceptance rates hover between 10–20%, about 50% of the papers that are discussed will be accepted.
Attendance. An in-person meeting is more effective than other alternatives (e.g., conference calls), if and when it is feasible. Most top-ranked conferences—and even some competitive workshops—have in-person program committee meetings. When program committee chairs select the members of the program committee, they typically confirm that the program committee member can attend the program committee meeting; and, if the meeting is an in-person meeting, they confirm that the PC member can attend in person. If all PC members are not present at the meeting, it is nearly impossible to have anything close to a fair paper selection process. Consider that a paper under discussion might have anywhere from 3–6 reviews, and that most of those reviews might hover around average scores, but one or two reviews may be outliers. The fate of a paper can shift dramatically if the PC member who feels strongly (either positively or negatively) about a paper is not in attendance. Without a champion present, the paper may simply fade into the mix; if a reviewer who raises major concerns about a paper is not present, the rest of the committee may be more likely to discount those concerns, since they cannot hear about them first-hand. Some of these attendance-related quirks can also materialize when program committee members depart early (e.g., to catch a plane flight).
Calibration. Every reviewer has a different opinion for what constitutes a reasonable threshold for acceptance. These differences in calibration are exacerbated by the fact that every program committee member has a unique set of papers to review. That is, while each paper has many common reviewers, no two PC members review exactly the same set of papers. Therefore, calibrating the PC is incredibly important for controlling the meetings dynamics. A poorly calibrated PC may pre-emptively reject good papers (as I mentioned above), only to accept weaker papers later on in the meeting. A good PC chair will spend considerable effort to calibrate the PC to reduce randomness. There are various approaches to calibration, which I will discuss below in my advice for PC chairs. One of them is setting an appropriate order for paper discussion, since the order in which papers are discussed will affect how the PC calibrates itself at various points in the meeting.
Discussion order. There is no accepted standard for setting the order in which papers are discussed, and every program committee chair seems to have a slightly different approach. That said, discussion order has a significant effect on the ultimate disposition of a large number of papers that sit on the borderline between acceptance and rejection. Perhaps the most striking example of ordering effects is evident when a PC discusses papers in decreasing order of score (i.e., from highest ranked to lowest ranked), which is, in my opinion, the worst possible ordering. People arrive at a meeting highly energized and eager for discussion. What often happens is that papers that are discussed in the morning are subjected to a fine-toothed comb and vigorous discussion—sometimes the result of these discussions can result in pre-mature rejection of a good paper. Later in the day, PC members become more tired and are less likely to pick apart a paper’s flaws or have a paper accepted only “over his or her dead body”. Similarly, excessive aggression in the early parts of the meeting can create a situation where, by the end of the day, there are not enough papers to fill the conference program (quite an irony, given the starting point of twice as many acceptable papers as slots!) This seems to create a dynamic where papers that are discussed later in the day sometimes have a better chance of being accepted. If papers are being discussed strictly in order from best to worst, there is the potential for “inversion”, where better papers are rejected early on in the meeting, making room for weaker papers to be accepted later on, as PC members acquiesce and PC chairs are frantically trying to fill the program. PC chairs can take various steps to reduce this randomness. One tweak I have seen (and used) is to simply treat all papers under discussion as roughly equivalent, banning all discussion or consideration “scores”. In these cases, the discussion order can be set such that papers on related topics are discussed together. Another that I have used is to employ a two-pass approach to discussion, where every paper that is not accepted immediately or upon initial discussion is discussed a second time—in other words, a paper is never rejected on the first pass. A third tweak that is sometimes used is to discuss a few “highly ranked” papers first (sometimes even the papers that are supposed to be “accepted without discussion” are used as quick calibration examples), followed by a few “low ranked” papers, and so forth.
Timing of discussion. A PC meeting does not leave time for extended consideration of any single paper. If there are 80 papers to discuss, and a PC meeting has eight hours of real work (discounting breaks, lunch, and so forth), then each paper sees, on average, 6 minutes of attention. Thus, discussion must be crisp and focused. A “discussion lead” for a paper (typically one of the reviewers) will typically summarize the paper and the strengths and weaknesses, effectively summarizing all of the reviews. This summary should last no more than 90 seconds. This is not a lot of time. Therefore, a good PC member who is leading discussion on the paper will prepare that summary ahead of time, so that it is as efficient as possible. The paper’s ultimate fate is typically decided in the remaining time (less than five minutes). Many disputes and disagreements about certain papers cannot be resolved in five minutes. This truth is the most common downfall of a PC meeting—a single contentious paper runs the risk of monopolizing discussion time, leaving less time for the remaining papers, thereby increasing randomness, acquiescence, etc. for papers that are discussed towards the end of the day. I find that using the “two-pass approach” for paper discussion helps mitigate this problem. I’ll discuss this approach in more detail below.
Personalities. A paper that has a single champion tends to fare much better than a paper with middling reviews, even if both papers have the same average score. In some sense, that is exactly the right outcome—a paper that someone really likes is likely to be appreciated by others, whereas an average paper might not excite anyone in particular. However, strong personalities do hold tremendous sway over a paper’s outcome, as well. A paper stands a very good chance if it has a champion on the PC who is articulate, confident, and strong-willed enough to persistently argue for a paper’s acceptance in spite of the paper’s flaws and detractors. Every paper has flaws; a strong-willed PC member can convince the rest of the PC to overlook those flaws in favor of other bright spots. Likewise, a single similarly strong-willed PC member can amplify the weaknesses of a paper and send the paper to the reject pile. These effects are exacerbated if the strong-willed PC member asserts expertise over the topic area. Sometimes, a PC member’s expertise is as important (if not more) important as the volume or persistence of the PC member’s argument. Simple statements from an domain expert such as “I learned a lot by reading this paper” can be enough to tilt a paper towards acceptance.
Psychology. The paper selection process is a human process that involves a lot of psychology. Rather than recount the various psychological factors that can play a role in program committee meetings, I refer you to Matt Welsh’s blog post on the topic.
Tips for Program Committee Sanity
Everything I learned about how to run a program committee, I learned from Jeff Mogul, who co-chaired NSDI this past year with me. Jeff is a veteran program committee chair who has chaired pretty much every major systems and networking conference at some point in his career. He is meticulous, thorough, ethical, and fair, which are indispensable qualities for any program committee chair. Working through the process with him, I picked up several tactics and strategies, which I can recommend to others and will undoubtedly use if and when I chair another major conference program committee. Being a program committee chair has countless tasks; some of these tasks are incredibly important, and the ones that are paramount weren’t immediately obvious to me until after I’d gone through the whole process. Below, I highlight tips for what I think are the most important steps of running a program committee.
- Pick your program committee carefully. Your program committee members need to be thoughtful, reliable, and conscientious. Ultimately, the program committee will write the reviews that authors of submitted papers see, and they will also determine the fate of each submitted paper. Don’t simply pick your friends or people you know well—they may not always make the best reviewers. Take extra time to do homework on program committee members’ performance on past committees. Did they write thorough reviews? Were they active participants in discussion at the PC meeting? Did they turn in reviews on time (or, if not, were they communicative enough to help the chairs plan around timing hiccups?). Are they considered an expert in a particular area that will see a lot of submitted papers for the conference? Beyond selecting individuals, the chairs also need to ensure that every area that may see submitted papers has significant coverage. For example, if the conference will see many submissions on (say) wireless networking, it is incumbent on the chairs to ensure that there are enough reviewers for that particular paper topic.
- Make sure that every program committee member “bids” on papers, and assign reviews manually based on preferences and expertise. Most conference reviewing systems allow program committee members to review all submitted papers and express preferences for which papers they would like to review. Some conference systems also enable chairs to automatically assign reviews. Do not rely exclusively on auto-assignment. When Jeff and I chaired NSDI, we used the auto-assignment feature in HotCRP (after ensuring every PC member filled in review preferences for each submitted paper) and then manually reviewed each paper to ensure that no reviewer was assigned to review a paper for which they had expressed a negative preference. This process is painstaking, but it is perhaps the most important step in the entire paper selection process, because it ensures that reviewers are assigned papers that they are capable of and willing to review. A reviewer who knows a paper topic can typically write a thoughtful review (and often do so quickly). In contrast, a PC member who reviews a paper for which they lack expertise or enthusiasm will typically invest only minimal effort and write a superficial review (thereby increasing randomness and likely reducing overall conference quality).
- Insist on (and monitor) online discussion in advance of the meeting. Selecting discussion leads and asking the lead to type a summary in the online discussion helps organization; the discussion lead can essentially read this typed summary (or notes) at the meeting itself, ensuring that the summary discussion concludes quickly and on time. Online discussion can also encourage reviewers to identify contentious issues before the meeting (where there is limited time to resolve disputes or disagreements), and potentially resolve them ahead of time, thereby averting protracted and unfocused arguments at the PC meeting itself. Sometimes, consensus on a paper can be reached in online discussion before the meeting occurs, saving precious time at the meeting itself.
- Use a two-pass approach to paper discussion. Every other program committee I’ve served on tries to reach a conclusion about a paper’s disposition after a (short) discussion. In the best case, this works OK; in the average case, hasty and incorrect decisions can result; in the worst case, the discussion monopolizes meeting time, and the meeting is derailed and ends in a frantic rush to accept papers at the end of the meeting. The two pass approach we used worked as follows: (1) in the first pass, a discussion lead would summarize the paper and its reviews; in the remaining 3–4 minutes, the reviewers would try to agree on an outcome for the paper, but none of the outcomes were reject. Rather, possible outcomes were: accept, incremental/boring (indicating that the paper was technically correct, but not particularly interesting or groundbreaking), risky (indicating that the paper could be groundbreaking but that reviewers had concerns about correctness or something else), and discuss (indicating that there was absolutely no agreement on the paper, and that more time would be needed to reach an outcome). (2) In the second pass, our original intent was to re-discuss everything, but effectively what happened was that all papers labeled as risky were accepted by default unless someone wanted to argue against one of them. Similarly, all papers that were labeled as incremental were rejected by default unless someone wanted to advocate for one of them. (We had papers in each of these categories.) Pulling reject off the table on the first pass kept the meeting going (we could reasonably end discussion after a fixed amount of time because people knew we could come back to discussion later), and it also reduced the “ordering effects” that I described above, since every paper that wasn’t quickly accepted received two passes, with the second pass occurring after the PC had seen the complete set of papers.
- Read every paper that is being discussed. Although it is not possible to carefully read 80 papers in advance of a meeting, two PC chairs can split the workload and at least perform a quick read of about 40 papers (and take notes on them) in advance of the meeting. Jeff and I did this, and it proved to be incredibly useful, for several reasons. Having your own opinion about a paper as the chair can help a chair to moderate the PC meeting discussion, by ensuring that a strong personality (or, sometimes, a PC member who has not even read the paper!) from unfairly swaying the discussion or perception of the paper. In the end game, sometimes PC members will never come to agreement about whether a paper should be accepted or rejected. In these cases, the chair becomes the “tie breaker” and can accept or reject a paper by fiat. This typically happens at least once in every PC meeting. A PC chair who has not read a paper cannot exercise fiat effectively, so having read every paper is critical in these situations.
The Worst Process, Except for Every Other Process
The program committee process for paper selection is far from perfect; it is an inherently human process, and in the final analysis, a small handful of gatekeepers (indeed, sometimes even a single gatekeeper) can determine whether a paper is accepted to a top-tier conference and read by many others, or shelved. I personally would like to see my own community have a long discussion about (and experiment with) ways to improve this process. In the meanwhile, I trust that this post can help authors (particularly students) understand some of the dynamics of the process and also help chairs and PC members ensure that the process is as fair as possible. Although the process is far from perfect—and it is unlikely that a perfect process exists—understanding the mechanics that go on “behind the closed doors” of a program committee meeting hopefully also indicate that the outcome of the process (either acceptance or rejection) should not be interpreted as universal praise or condemnation, but rather the result of the opinions of a small number of people and the outcome of a human process. Ultimately, the proof of a research idea lies not in the outcome of a single program committee, but in an ideas ultimate acceptance, adoption, and impact (which is, in itself, a topic for a future post).
Learning how to review papers not only (obviously) makes you a better reviewer, but it can also help you as an author, since an understanding of the process can help you write your paper submissions for an audience of reviewers. If you know the criteria that a reviewer will use to judge your paper, you are in a much better position to tailor your paper so that it has a higher chance of being accepted.
There are many good resources that describe the paper reviewing process already, including those that explain the process (and its imperfections) and those that provide instructions for writing a good review (as well as techniques to avoid). There are also a few nice summaries of the review process for conferences in different areas of computer science that lend visibility into the process (e.g., here and here). Program committee chairs sometimes provide guidelines for writing reviews, such as these. I will not reiterate or summarize those previous articles here, but they are all definitely worth a read. Instead, I will discuss the importance of the review process and how it differs from simply reading a paper; I’ll also talk about how to prepare (and ultimately write) a review.
I will not talk about the paper selection process (i.e., what determines whether a paper is ultimately accepted or rejected), but will instead focus on the creation of a paper review. Program committee meetings are an important part of the paper selection process—at least in computer science—and I will be devoting a complete post to this topic next week. Meanwhile, I recommend reading Matt Welsh’s post on the psychology of program committees.
The Review Process
Why understanding the review process is important. Whether you end up reviewing a lot of papers as a Ph.D. student, your research will definitely be subject to the paper review process. It is imperative as a researcher to understand this process. Knowing the process can help you better write your paper for an audience of reviewers (and a program committee), and it can also help you maintain perspective when your paper is accepted or rejected. The process is far from perfect, and the outcome of the process is neither validation nor condemnation of your work. How you react—and how you adapt your research or follow through on it after the acceptance (or rejection)—is far more important to long-term success.
In the “Introduction to the Ph.D.” class at Georgia Tech, I ask students to create a research idea and write it up; a subsequent set of assignments asks the students to review and evaluate the ideas as part of a “mock” program committee. The process isn’t exactly the same as the review process for a full paper, but it is a lightweight way to have students experience the process first-hand in a low-stakes setting, and see both sides of the process (submission and review) at the same time. In next week’s blog post, I will discuss program committee meetings in general, as well as some observations from this year’s (and previous years’) in-class experiences with the mock PC.
Reviewing vs. reading. There are some significant distinctions between reading papers vs. reviewing them. When reading a paper for your own enrichment, your goal is to gather information as quickly as possible. In this case, you are a scientist who seeks to understand the context and content of existing work, to (for example) better understand how your own research might fit into the bigger picture or learn about techniques that might apply to your own work. The goal of reviewing is different. A reviewer’s goal is to first and foremost determine the suitability of a paper for some conference and second, to provide feedback to the authors to help them improve the paper in subsequent revisions. Remember that the reviewer’s primary goal trumps all other objectives: A reviewer often has a large number of papers to process and is typically not deeply devoted to improving the content of any particular paper. If you are lucky, you will get a diligent, thoughtful reviewer who provides thorough feedback, but do not be surprised if a review is not as thorough as you would have liked, or if the review “misses” some point you were trying to make. We would all like reviewers to make three passes through your paper submission—and, these are the instructions I would give, too, in an ideal world. Unfortunately, however, you will be lucky in many cases to get two thorough reads. The reviewer’s main goal is to determine the paper’s suitability for publication. As an author, you shouldn’t be surprised if some of the comments seem trivial: there may be underlying issues of taste that drove the reviewer’s opinion on your paper that a reviewer may not explicitly state. Whenever I read reviews I receive for a rejected paper, I try to look past specific detailed quibbles (or “excuses” for rejecting the paper) and figure out the big picture: the reviewer couldn’t find a reason to accept the paper.
Calibration: Reviewing one paper vs. reviewing many papers. The paper review process can differ depending on who, exactly, is reviewing the paper. For example, as a Ph.D. student, you may review one or two papers at a time, as an “external reviewer” for a conference or journal. Journal editors and program committee chairs often seek the help of external reviewers if they need a particular subject-matter expert to review a paper. Later in your Ph.D. career, you may have established yourself as an expert on a particular topic and find yourself reviewing a paper here and there on a handful of topics. Sometimes a member of the program committee (e.g., your advisor) might ask you to help review a particular paper. As you progress in your career, you will be asked to serve on program committees yourself, whereupon you’ll find yourself with tens of papers to review over the course of a couple of months. Ironically, it is sometimes easier to review a group of papers than a single (or a few) papers, because seeing a group of papers helps you “calibrate” your scores and rankings of papers according to the general quality of papers that have been submitted to the conference. If you have been asked to review a single paper for a conference, you should either figure out how to calibrate your assessment with respect to other papers that might have been submitted, or simply review the paper on its merits while reserving judgement as to the paper’s ultimate disposition.
Does the Paper Realize a Great Idea?
Look for a reason to accept the paper. Does it realize a great contribution or idea? Every paper is imperfect. The paper may have made an incorrect or imperfect assumption. The experiments may not have been as thorough as you liked. The graphs may be difficult to read. Parts of the paper may be difficult to understand. These types of issues certainly reflect problems with a paper, but they do not necessarily constitute a reason to reject a paper if they do not affect the correctness or significance of the main underlying conclusion or contribution of the paper. Therefore, the first two questions I ask myself when reviewing a paper are: (1) Does the paper have a great idea?; and (2) Does it realize the great idea? (or, alternatively, to what extent does it realize that great idea, since typically no paper is water-tight).
What makes an idea “great”? Judging a paper’s contribution turns out to be highly subjective, which is why the review process remains so uncertain. A paper isn’t judged on a set of fixed checkboxes, a grading “key”, or any notion of absolute correctness. Reviewers often reserve considerable judgment based on “taste“, and reasonable people will disagree as to the merits of the main contribution or idea in a paper. In fact, there has been a fair amount of documentation that, as reviewers, we are often quite terrible at predicting the merits of a particular piece of submitted work: There’s a great article on this topic, as well as some parodies to illustrate the subjective nature of the process. Many fields have also introduced a “test of time” award to papers from past decades, to recognize accepted papers that have truly had long-term positive impact (implicitly acknowledging that this is almost impossible to assess when a paper is first published). Due to the subjective nature of this judgment, it is all the more important that your writing is clear, and well-matched to what a reviewer is looking for (i.e., the contributions and ideas).
Invariant questions. Different conferences may have different value structures, and the chairs of any given conference may ask the reviewers to focus on different criteria when judging a paper. Regardless, there are some invariant questions that most reviewers would (or at least should) always consider, including:
- Is the problem important? What problem is the paper trying to solve, and is it important? Seek to summarize the paper’s contribution in one sentence. Make this short summary the beginning of your review, as well. Try to convince yourself (by reading the paper or otherwise) that a solution to the problem that the paper is proposing would advance knowledge or significantly improve the state of affairs for some group of people. Note that you may not care about the problem, but also ask yourself whether you can imagine some group of readers who will be interested in the solution to the problem. When asking yourself this question about a paper, try to divorce your own taste about the problem’s importance from the more general question concerning whether there is some group of people who would be interested in the problem the paper is addressing and solving.
- To what extent does the paper solve the problem it describes? A single paper very rarely closes the book on a single problem, but it may take an important step towards solving the problem. It might solve the problem for an important set of operating conditions or under a new set of assumptions. Or, if the problem area is completely new, perhaps the paper doesn’t really solve the problem at all, but simply articulating a new problem area for follow-on work is a significant contribution.
- What is the “intellectual nugget”? As a reviewer, I try to identify whether a paper has a particular intellectual kernel that lies at the heart of the solution. This kernel is often what separates an important research contribution from a simple matter of engineering. This intellectual nugget might be the application (or invention) of a particular technique, a proof of correctness (where one previously did not exist), or an attempt to put the solution into a broader intellectual context. In other words, the intellectual contribution might be to take a general problem and tackle a specific sub-problem (e.g., under certain assumptions or conditions), or to take a specific problem and generalize it (e.g., develop a general theory, proof of correctness, or taxonomy). Looking through the paper for applications of specific research patterns can help identify an intellectual nugget, if one exists.
- What is the main contribution or conclusion? Is it important? As a reviewer, I try to concisely articulate the paper’s main contribution (or small number of contributions). Often, a paper will helpfully summarize those contributions somewhere in the introduction (Jim Kurose’s advice on writing paper introductions advises the writer to explicitly do so). The reviewer’s job is then to assess whether those contributions are significant or important enough to warrant a publication. The significance of those contributions often depends on the perceived increment over previous work. All work is incremental to some degree, as everything builds on past work. The author’s job is to convince the reviewer that the increment is important, and the reviewer’s job is to assess the author’s claims of significance.
- Does the content support the conclusion? An introduction may make broad (or wild) claims, and it is important to dig into the paper to determine whether the content of the paper supports the conclusion. Are the experiments run correctly? Are they based on the correct set of assumptions? If the conclusion involves comparison to previous work, is the comparison performed in a controlled manner, using an equivalent (or at least fair) experimental setup? If applicable, have the authors released their code and data so that you (or others) can check the claims yourself?
Preparing Your Review
Consider the audience. Not every publication venue is the same. Some venues are explicitly geared towards acceptance of early, incomplete work that is likely to generate discussion (many workshops use this criterion for acceptance). Other venues favor contributions that constitute well-executed, smaller increments. When reviewing a paper, either externally or as a member of a committee, your first question should be to consider the audience for the conference, workshop, or journal, and whether the likely audience for the venue would benefit from reading the paper. The question of audience involves that of both the “bar” for acceptance (Does the paper meet the audience’s standards for something that is worth reading?) and the “scope” of the venue (is the paper on-topic for the venue?). Often, scope can be (and is) broadly construed, so the key question really boils down to whether the likely audience for the paper will benefit from reading it.
Consider the standards. Your standards will (and should) vary depending on the venue for which you are reviewing a paper submission. Workshops are typically more permissive as far as accepting “vision” papers that outline a new problem or problem area or papers that “foster discussion” than conferences, which typically aim to accept more complete pieces of work. Nevertheless, even the standards for a conference review process will vary depending on both the conference itself, the program committee chair’s instructions about how permissive to be, and the relative quality of the group of papers that you are reviewing. A good way to get a sense for the standards of a conference for which you are reviewing is to read through the complete set of papers that you have been asked to review and rank them, before writing a single review. This will ensure some level of calibration, although it is still biased based on the set of papers that you are reviewing. Reading past proceedings of the particular journal or conference can also help you determine the appropriate standard to set for acceptance.
Consider the purpose. Different papers serve different purposes. Multiple paper submissions to the same venue might in fact have quite different purposes, and it is important to establish what the paper is contributing (or attempting to contribute) before passing judgement. For example, a paper might be a complete piece of work, but it might also be a survey, a tutorial, or simply a proposal. If the paper is one of the latter types, your first questions as a reviewer should concern whether the audience would benefit from the survey, tutorial, or proposal, and whether such a paper meets the standards for the conference. If the answers to those questions are “yes”, then your evaluation should be tailored to the paper’s purpose. If the paper is a survey, your assessment should be based on the completeness of the survey, with respect to the area that the paper is claiming to summarize. If the paper is a tutorial, is the description correct and clearly described? If the paper is a proposal, does the proposed research agenda make sense, and is the outcome (if the proposal is successful) worthwhile?
Consider the big picture. Every paper can be rejected. It is always easy to find reasons to reject a paper. The reviewer’s goal should not be to identify the reasons to reject a paper, but rather to determine whether there are any reasons to accept the paper. If the answer to that question is negative, then it is always easy to find “excuses” to reject a paper (recall the discussion above). You should be aiming to figure out whether the paper has important contributions that the audience will benefit from knowing about, and whether the paper supports those contributions and conclusions to the level of standard that is commensurate with the standard of the audience and the venue. One litmus test I use to ensure that a negative aspect of a paper does not condemn it is to ask myself whether the problem (1) affects the main conclusion or contribution of the paper; and (2) can be fixed easily in a revision. If the problem doesn’t affect the main contribution or conclusion, and if it can be easily fixed, then it should not negatively affect a paper’s review.
Writing Your Review
Start with a summary of the paper and its contributions. A short, one-paragraph summary describing the paper’s main contribution(s) demonstrates to the authors (and to you!) that you understand the main point of the paper. This helps you as a reviewer articulate the main contributions and conclusions of the paper for the purposes of your own evaluation. Try to address the type of paper it is (is it a survey paper, for example?), the context for the paper (i.e., how it builds on or relates to previous work), its overall correctness, and its contributions. If you cannot concisely summarize the paper, then the paper is not in good shape, and you can reflect this assessment in the review, as well. These summaries are very helpful to authors, since they may not match the authors’ views of the main contribution! For example, as an author, you can easily figure out if you’ve “missed the mark” or whether the reviewer fundamentally misunderstood the paper by reading a reviewer’s summary of your own work. If the summary of the contribution does not match your own view of the paper’s contribution, then you know that you have some work to do in writing and presentation.
Assess whether the paper delivers on the main claims and contributions. You should provide an assessment, for each of the paper’s main claims and contributions, whether it delivers on that claim. If the main contribution of the paper is flawed, you should indicate whether you think a flaw is “fatal”, or whether the authors could simply fix the flaw in a revision if the paper is accepted. Sometimes flaws (e.g., inconsistent terminology) are fixable. Other flaws (e.g., a questionable experimental setup) may or may not be fixable. While it might seem that a broken experimental setup is “fatal”, ask yourself as a reviewer whether the conclusions from the paper’s experiments as is are still meaningful, even if the authors have not interpreted the results correctly. If the conclusions from the experiments can be restated and still turn out to be meaningful contributions—or, if the flaw in an experiment doesn’t affect the main contribution or conclusion—then even a flaw in experiments can likely be fixed in revision. Occasionally, however, experiments may need to be completely redesigned because they don’t support any meaningful conclusion. Or, the content of the paper may simply be incorrect; sometimes correctness issues are difficult for a reviewer to spot, so a paper isn’t necessarily “correct” simply because a reviewer has validated the paper. Regardless, if there are correctness issues that affect the main contribution of the paper that call into question whether the main result or contribution is correct in the first place, the paper’s review should reflect these concerns and likely cannot be accepted.
Discuss positive aspects of the paper; always try to find something positive, even in “bad” papers. It is easy to identify problems with a paper. It can be much trickier (especially with “average” papers) to identify the positive aspects and contributions, but most papers typically have at least some small kernel of goodness. Even for particularly bad papers, there might be one sentence in the introduction, discussion, or future work section that makes an interesting point or highlights a possibility for interesting contributions. In a pinch, if you can’t find anything positive, those are good places to look. As a reviewer, you can remark that those observations are interesting, and that you would really like to see those parts of the work further developed. These positive comments aren’t just for author morale (although that’s important, too): They give the author a direction to move forward. The worst reviews are those that reject a paper but don’t provide any specific action for moving forward. The best reviews are those that highlight the positive aspects of the work, while identifying weaknesses and areas where the work could be further developed to address weaknesses or build on the paper’s existing strengths.
Criticize the paper, not the authors. When writing your review, consider the type of review that you would like to receive. Always be polite, respectful, and positive. Don’t be personal. Choose your language carefully, as it will help convey your message. For example, if you say “the authors don’t consider the related work”, that is a much more personal statement than “the paper doesn’t consider the related work”. (In fact, you don’t know if the authors considered a particular piece of related work anyway; they may have simply chosen not to include it in the writeup!) Talking about “the authors” gets personal, and it will put the authors themselves on the defensive when reading your review. Instead, focus on “the paper” and frame your critique around “suggestions for improvement”. Never, ever insult the authors; don’t accuse the authors of being sloppy or unethical researchers. As a reviewer, you don’t always know the full context, so limit your judgement to what you can directly conclude by reading the paper.
Consider the type of feedback you would like to receive. Receiving reviews for rejected papers is a part of the research process, but it is never fun for the authors (particularly new Ph.D. students). Do your part to contribute positively to the process by suggesting changes that you’d like to see if you had to review the paper again. In all likelihood, you may see the paper again in the form of a revision!
Little did I know that a Ph.D. in computer science would teach me more about how to write than any other previous experience I’d had. Indeed, research isn’t only about making new discoveries, but also about expressing those discoveries so that other people can appreciate them and their significance.
Below, I’ll present some thoughts on writing that I’ve picked up over the years and continually aim (and encourage students) to put into practice. I’ll focus on story-telling, presentation, and efficiency: how to write your paper so that it clearly, cleanly, and efficiently tells a good story.
What this post is not about. At least in this post, I won’t write about grammar—there is too much English grammar to cover in a single post, and there are plenty of good courses one can take to learn grammar. I also won’t talk about style (common phrases) or usage (e.g., commonly misused words). For now, my only advice on the topic of learning English grammar and style—both for native speakers and foreigners—is to read daily. Reading the New York Times, or The Economist are great places to start: the writing is generally of high quality, and you’ll also have something interesting to talk about outside of the lab. I have the impression that foreign students can tend towards reading their local news from home; that’s understandable, but I would encourage those students to also read periodicals and publications that are written in the style of English that people are used to reading in the Western world (indeed, English in different parts of the world has different tones, styles, idioms, and turns of phrase).
Why Does Writing Well Matter?
Writing is part and parcel of the research process. Writing communicates your research results, and—perhaps contrary to intuition—writing well is even more important when the results that one wants to present are “strong” (after all, if the results are that great, you should want as many people as possible to benefit from understanding them). But writing isn’t just something that happens at when the research is “done”; rather, writing should occur throughout the course of a research project. Writing and thinking are tightly coupled—writing is a reflection of our own thoughts, and writing well can actually sometimes clarify our own thinking about a problem and lead us to a solution.
Communicating your results. A research finding or discovery that others cannot understand or appreciate is as good as no discovery in the first place. A well-written paper with a half-baked idea that people can understand and appreciate will often be accepted to a conference over a paper with a (possibly) stronger result that nobody can understand or appreciate. Cynics may dismiss this as “unfair”, as “laziness” on the reviewer’s part for not reading the details thoroughly, or as “accepting a paper because it was well-written” (insinuating that the results themselves might not be that strong). The fact of the matter is that none of this matters: Independently of the quality of the research, the results must be written well, or people will not be able to appreciate the value of your work. Writing well is perhaps one of the most important skills that budding researchers must attain; it is equally important as learning how to conduct the research.
Clarifying your own thinking. Writing is a reflection of your thinking. You should never be content with vague language or imprecise descriptions. Muddy writing often reflects muddy thinking. An attempt to write down a concept, idea, or solution can actually help you realize that you don’t know what you are talking about! (And, help you crystallize your thinking.) You may have had the experience of realizing you don’t understand something as well as you thought you did when someone asks you to explain something and you have trouble. Writing achieves the same effect: as you attempt to write something down, you may realize that you don’t know why you are working on something, why a particular phenomenon appears, and so forth. If you don’t force yourself to try to write down your explanations and observations, you may never discover that you don’t know what you’re doing until it’s too late. Therefore, start writing early in the course of a research project. Most of the early text you write may not find its way into the final version of the paper, but taking the time to express your ideas on paper can help you crystallize your thoughts and guide your research.
Tell Me a Story
Every paper should tell a story. For a given research result, there are many stories that a paper could tell, and it’s important to think about what the story should be. This decision should be made early on in the writing process. The story can be changed later on, to match the strongest results in the paper, or if the context for the research changes (for example, sometimes when working on a research problem, people can find it necessary to re-formulate the problem that they are solving). But, it’s always good to figure out a story before you go too far down the path.
Determining the Big Picture: Composition and “Flow”. Composition is the structure and flow of your paper writeup, and how individual paragraphs fit together. When people talk about the “flow” of a piece of writing, they’re talking about composition: how sentences flow together to form paragraphs, how paragraphs flow together to form sections, and how sections flow together to form a paper. Composition (or “flow”) is the most important aspect of technical writing. I’ve found that reviewers (and readers) are often forgiving of the occasional grammatical error, especially if a paper was written by non-native English speakers. The human brain is pretty good at applying “error correction” on grammar. Correcting “flow”, on the other hand, is extremely hard for a reader to do. Often, the reader will complain that they “can’t understand where [the paper] is going” and simply give up. Grammar is also easier to teach (and learn) than composition because there are fixed rules for grammar; composition is more of an art form, and it does involve a bit of intuition. It involves thinking about the big picture, the story that you want to tell, and how the pieces fit together. It takes practice to develop intuition for composition, and I’ve found that native English speakers are not necessarily any better at composition than foreigners are (that’s good news for Ph.D. students coming from abroad: the playing field is actually more level than it appears). Even though composition is a bit of an art form, I’ve found that the following tips can be incredibly helpful for improving flow:
- Build the scaffolding before filling in the details. Write sections and topic sentences first. There is always the temptation to simply jump in and start writing full paragraphs. This rarely ends well. Nobody would build a house before first drawing plans. After the plans comes a foundation, and scaffolding. Only after the structure is in place can many of the details be filled in. You should think about writing a paper in the same way: before you can fill in the details, you need to figure out how you want the paper to read when its done; this is your outline. Start at a high level, by outlining sections. Once you have an outline of sections, you can begin to slowly fill in more details: outline each paragraph within a section by writing only the topic sentence for the paragraph. See whether your story makes sense if you read each topic sentence in sequence. Your “paper” should roughly make sense if you read these topic sentences in sequence (in fact, some reviewers or readers may try to read your paper in exactly this fashion, by skimming topic sentences!). If reading your topic sentences in sequence doesn’t make sense, something is wrong with your flow. Determining which topic sentences to write takes a bit of practice, but I find that making a bulleted list of points that you want to include in the section and then clustering those points into related points and topics can help identify the topics you want to cover (and, hence, the topic sentences you want to write). If you find that the topics you want to cover are not all related to one another, you may need to create additional subsections. Once you have topic sentences, you can fill in paragraphs, which are groups of closely (and logically) related sentences. If your sentences aren’t closely related, they don’t belong in the same paragraph and you may need to consider adding a new topic (and topic sentence).
- Assess balance and budget pages accordingly. Once you have your outline, it’s important to budget pages, to figure out how much you plan to write about each topic. Topics of equal importance or equal relevance should receive “balanced” coverage in the paper. If your system has two components of relatively equal importance, then the sections covering those components should be of approximately equal length in the paper. Once you have your outline (from the previous step), you can assess how much space you want to budget for each point. If some points in a section are more important than others (and hence require more page budget), you can go back and think about how to re-balance your writeup (e.g., by creating a new devoted section to the topic that’s more important, shortening or eliminating the less important point, and so forth).
- Follow convention. Continuing the analogy of a house or building, many designs follow convention: a restroom in a restaurant is often in the back. A powder room in a house is often near the front door. These common “design patterns” make life easier for us when we visit a new building or house. Similarly, your reader is a visitor of your thought process. A reader who has read (and written) many research papers will similarly expect the structure of your paper to follow certain conventions. These conventions differ by field: some fields write papers completely in the passive voice, whereas others use active voice. Some fields prefer related work sections at the beginning of the paper, and others prefer related work at the end. Some fields expect an introduction that spends time framing the problem, and others expect an introduction that starts with a clear statement of the main result. There may be an expectation of (or aversion) to mathematics in the main line of the paper. Within a section, people may expect the introduction to flow in a particular way (in computer science, Jim Kurose’s description of how to write an intro is the best example I’ve seen for writing a conventional paper introduction).
- Signpost your paper. Even if your paper plan and outline make perfect sense to you, the reader doesn’t start out “on the same page” as you. Even if the builders have built a house completely according to the blueprints, even if the design of the house follows convention, visitors still typically appreciate an explanation of where the bathroom is. The reader needs instructions for how to read the paper and a clear view for how the paper and story will proceed. Signposts throughout the paper help the reader understand how the paper is organized. Examples of signposts include: an outline of the paper at the end of the introduction (“The rest of this paper proceeds as follows.”), a preamble to each section (“In this section, we discuss…”), declarative subsection titles, and (within reason) bold paragraph headings (such as those in this blog post).
Make a Good Impression
A reader forms an impression about your research based on your presentation of the work. Just as people quickly form first impressions about other people that may be difficult to subsequently correct, your reader will quickly form a first impression of your work based on a quick read of the title, abstract, and introduction and perhaps a quick skim through other parts of the paper. Therefore, it is important to make a good impression by writing a clear abstract and introduction (and also ensuring that the rest of the paper also is commensurate with the impression that you create).
Spend a lot of time on your introduction; start early. The introduction summarizes the story of your paper (now, it should be doubly clear why you need a story!) and is the most important part of your paper. Many people won’t read past the introduction, even if they find the work interesting. Yet, if people don’t understand your “story” (the problem you’re solving, its importance, and your approach to solving it), they will never make it past the introduction into the details of the paper. There are two schools of thought for writing the introduction: if you write it first, you ensure that you have a clear story. If you write it last, then your introduction will clearly reflect the ultimate story and be bolstered by your results. My advice is to do both. Writing the introduction is probably the most difficult part of writing the paper, which may come as a surprise to new students. After all, the introduction is high-level and often doesn’t reflect the nitty gritty details of how difficult that experiment was to run, and how much toil went into writing the system. To the contrary, distilling your work from those details so that the rest of the world can digest the takeaways—stepping out of the weeds and trying to figure out why your work matters—can take more effort than anything. The same results can be presented in many different ways, so you will likely need to perform many iterations on your story. Typically, you can “test” your story out on other colleagues and readers with a draft introduction in parallel with completing the rest of your work.
Write the introduction first—and last. I typically try (and encourage students) to write a draft of the paper’s introduction before writing other parts of the paper. You will most likely eventually discard this draft completely, since it’s very hard to predict what one should be emphasizing in the introduction before the “meat” of the paper is done. Yet, writing an early introduction draft allows a researcher to put the work in context, and to argue (1) what the problem is, (2) why it is hard, and (3) why the solution (if it is achieved) will be interesting to readers. If an introduction draft can’t articulate answers to these questions, even at an early stage in the research, then the work likely needs some focus, as well. When you write the second version of your introduction (after the rest of the paper is written), you are in a good position to refine your story, making sure that any claims you make are clearly supported by your results and data. Your results should be stated as clearly and specifically as possible. You should neither understate your results (which might cause the reader to lose interest and stop reading) or overstate them (which might cause the reader to be annoyed or “let down” when they dig in and find that you don’t deliver on what you’ve promised).
Perform some “landscaping” on your paper. Your paper should be pleasing to the eye. Care in preparation of the paper suggests that you also cared about the research itself. Conversely, if the preparation of the paper is sloppy, how can the reader trust that you were not similarly sloppy in writing code, setting up and running your experiments, and analyzing your results? Therefore, take some time to make your paper look nice. Here are a few quick tips on “landscaping” your paper.
- Eliminate widows. Section headings at the bottoms of columns or pages look sloppy. Depending on your word-processing tool, there are good ways to get rid of widows. In Latex, you can always use a “\vfill” to kick the section heading to the top of the next column. Similarly, widows on paragraphs (paragraphs that end with a single word on the last line) look sloppy. Typically you can eliminate such widows by wordsmithing (see below on omitting needless words). Latex also has macros such as “\widowpenalty” that help ensure that the text layout avoids widows whenever possible.
- Place (and create) signposts, figures, and graphs to create whitespace and avoid “walls of text”. Reading long installments of technical writing is tiring, and seeing a page of uninterrupted technical writing can psychologically tire the reader before they even begin to read. Create whitespace and take other steps to break up large chunks of text. Use paragraph headings to break up long strings of text (ensuring that your headings match the topics you’ve outlined, of course). Try to space figures, tables, and graphs throughout the paper so that most pages have a mix of figures and text. If you don’t have a figure to break up the text, consider making one. Often, a table summarizing results can be useful. Or, you might find that a figure (or block of pseudocode) describes a concept better than a paragraph in your writeup. Take steps to ensure that your writeup is not a long stream of text.
- Check your spelling. Run a spell checker. Every word-processing tool has one, and a writeup that is rife with spelling errors reflects carelessness. Again, if you can’t be bothered to check your spelling, why should a reader trust that you spent time debugging your experiment or code? While a reader can forgive grammatical errors, especially when the writeup comes from a non-native English speaker, anyone can run a spell checker, so spelling errors simply reflect carelessness. That said, not all spell-checkers are perfect, and sometimes typographical errors actually manifest as real words; spell checkers won’t catch those, so they should be used only as a first pass.
- Make the last page look decent. If you’ve edited your text down to remove redundancy, you may find that you have some extra space at the end of your paper. Try to adjust your formatting so that the last page of the paper looks decent. If your paper is double-column and you end with references, balance the references across the two columns so that both columns are both equally (partially) filled. Latex has a “\balance” command that you can use to balance the columns on the last page of your writeup. If you have a bit more time, you can also make small adjustments to margins, space between columns, space around figures, sizes of figures, and so forth to ensure that the last page doesn’t look like you just suddenly decided to stop writing.
“I’m sorry I have had to write you such a long letter, but I did not have time to write you a short one.” –Blaise Pascal
After you’ve made a good impression, you still need to engage the reader at least long enough to convey your main message. You should assume that the reader is busy, has a short attention span, and (unlike the situation when they are, say, reading a news article or novel) simply wants to learn what you’ve done as quickly as possible and then move on. In contrast to writing a novel, your goal is not to hold the reader in suspense or to illustrate your command of vocabulary or literary construction. Rather, the goal is to transfer information as efficiently as possible.
Tailor the length of your paper to the information content. People sometimes feel compelled to stuff a paper to the conference’s or journal’s page limit (after all, if the paper is too short, isn’t there “more work to do”?). I’ve had a few papers that do fall short of the page limit, although I’ve found that this scenario is the exception rather than the rule (see Blaise Pascal: it takes time to shorten a paper!). Remember that your goal is to efficiently transfer information. Therefore, economize on words. Do not add unnecessary sentences, paragraphs, or other content that do not support a topic or point that you’ve explicitly outlined.
Keep words simple and sentences short. The easier it is to read your paper, the more likely a reader will keep reading. Therefore, don’t make the reader work harder than they have to. On a sentence level, follow Strunk and White’s most indispensable advice: Omit needless words (e.g., “to” as opposed to “in order to”, “optimizing” instead of “the problem of optimizing”, “This module…” instead of “This is a module that”, “more quickly” instead of “in a shorter time”, “whether” instead of “the question as to whether”). Avoid padding phrases such as “the fact that” and unnecessary adverbs (e.g., “eliminated” instead of “totally eliminated”, “fast” instead of “very fast”). Keep words simple (e.g., “use” instead of “utilize”, “new” instead of “novel”), and keep sentences short.
Eliminate redundancy. Always make at least one (and probably several) editing passes over your paper to cut out redundancy. You likely (and hopefully) did not write your paper in a linear fashion, and different co-authors may have written different sections or pieces of the paper. It’s likely that you’ve stated the same point multiple times throughout the paper. Cutting redundancy takes practice, and it sometimes requires divorcing the editing process from your ego (after all, isn’t that sentence you wrote beautiful? It’d be a shame to cut it, right?). If you need help cutting, ask someone else (a co-author or colleague) to help you—they’re not as married to your text as you are.
Be as specific and precise as you can. Precision is especially important when describing results, and when writing up an experiment or evaluation section. For example, a good rule of thumb for writing up an evaluation section is: Would someone be able to reproduce your experiment from your description? If not, you need to be more clear. In an evaluation section, clearly state your assumptions; in what context(s) do your results hold? How general are they? Clearly describe your evaluation setup, as it sets the context for the results themselves. What machines did you use to run the evaluation? What data did you gather to support your conclusions? What algorithms did you use to analyze the data? The evaluation setup should read like a recipe. Someone reading it should be able to reproduce what you’ve done, at least well enough to understand the context of the results. Similarly, the results should be clearly and precisely described. Avoid writing sentences such as “This optimization significantly reduced the system’s running time.” Such vague statements frustrate the reader. Instead, substitute precise and specific information: “This optimization reduced the system’s running time by 20%”.
Use clear and consistent terminology. Unclear, vague, and inconsistent terminology is incredibly frustrating and also typically reflects muddy thinking. If you call something a “node” in one part of your paper, a “host” in another part of the paper, a “client” in another part, and a “machine” in yet another part, people will have absolutely no clue that all of these terms are in fact referring to the same thing. Before you get too far along in the writing process, clearly define your lexicon. Defining your terms up front can help clarify your own thinking, by helping you to identify and articulate the important concepts that you will present. Ultimately, this clarity will be reflected in your writeup.
Reduction to Practice: Write Every Day
Writing a technical paper involves clearly presenting a good story, as efficiently as possible. The above tips should help you compose your story and ensure that it “flows” and ultimately produce a final result that leaves a good impression and communicates your results as efficiently as possible. But, simply reading this post won’t make you a good writer: putting these tips into practice takes tireless repetition. Try writing something every day to practice these tips. Keep a “running draft” of your research paper. Start a blog and practice composing posts that present shorter and simpler ideas and thoughts than a full research paper. Keep a diary or notebook. Ultimately, becoming a good writer requires relentless, daily practice, with continual attention to the above tips.
There is a lot of literature on public speaking, but most of it is general public speaking advice and is not geared towards presenting technical topics. Although researchers need to master general presentation skills as well, I found that Joey Asher’s book, Even a Geek Can Speak, has a lot of good tips on how to present complex technical topics to a general audience. These skills are useful for academics, who often have to present their research to people outside of their immediate area. For example, although your audience at a conference talk will be domain experts, almost every other situation where you will deliver a talk (department-wide presentations, job talks, presentations to industry) entails presenting to a significant fraction of people who are not well-versed in your topic.
Presenting a good technical talk involves several different aspects. Preparation requires (1) careful determination of your audience and your purpose for delivering a talk and (2) ample practice. The talk itself typically involves both (1) presentation of visual aids and (2) delivery of the talk itself. We’ll explore each of these aspects in depth below.
Determine Your Audience and Purpose
Academia presents many opportunities to present your research in a variety of settings, and these presentations can have many purposes. Your purpose and audience determines the talk’s content—including how you motivate your work, the level of detail that you choose to include or omit, whether you show graphs or simply summarize results, and so forth. Therefore, your first goal should be to determine your purpose and your audience. You must connect with your audience to deliver an effective talk, and the ways that you connect to your audience will differ depending on your purpose and your audience.
Consider the following purposes for giving a talk, and how you might tailor your approach depending on differing purposes (and audiences):
- Encourage other researchers to read your paper(s), or at least appreciate your work. Many published papers ultimately entail a presentation of the paper at a conference. In these cases, your purpose may be to encourage other researchers in your field to read your paper. The talk is, in essence, an advertisement for the paper. Your goal and purpose is to convince other researchers in your area that you have done something intellectually deep and interesting, and possibly to get them to build on your results. In a conference talk, your foremost aim is to help other researchers understand and appreciate your work, and to encourage them to read the paper for more details. Distilling complex research contributions into a short conference talk can be extremely challenging; for this reason, these types of talks require significant iteration.
- Find collaborators. You may wish to present a piece of work to researchers across disciplines to encourage other researchers to work with you. For example, you may be working on a particular problem and find that you need to enlist the help of a domain expert in another area. Sometimes, presenting your work to the experts in that area is a way to “cast a net” to find a collaborator who appreciates your problem and shares your passion. In this type of talk, your goal is to make your content as accessible as possible to researchers in that area. If you are presenting to researchers in another specific area (e.g., if you are a networking researcher presenting to theoreticians), you will need to tailor your vocabulary to the audience (avoiding domain-specific jargon at all costs), and it can also help to draw analogies to concepts or techniques in the area that your audience is more familiar with (e.g., a networking researcher may be able to abstract the details of underlying protocol implementations and present a problem in terms of a simple graph).
- Obtain feedback. In the early stages of working on a particular research problem, you may want to give a talk to obtain feedback on your problem formulation, your approach or solution, or some other aspect of your work. Designing a talk to obtain the feedback that you want requires engaging the audience on your problem so that they understand what you are working on and care enough about the problem to give you feedback on it. In this type of talk, your audience may typically be a smaller group of trusted researchers, such as other students and faculty members in your research group, or a small community of researchers who may convene for a workshop that is focused on a niche topic. You may be presenting to other colleagues with whom you may have pre-existing relationships or trust. You may also deliver these talks in more intimate settings (e.g., research group meetings, small workshops). These talks need not be completely polished, but they still need to be coherent and clear—if your audience cannot understand your thought process or cannot appreciate the importance of your problem, they may not be motivated to help, or the feedback that they provide may be completely off-target.
The “Message Objective”: Help the talk listeners achieve their goals. While each of these scenarios has a different audience and requires a different approach to the talk, all of them require “connecting” with the audience. Connecting with an audience first requires determining why someone is listening to your talk in the first place. Typically every audience member has a reason or goal for attending the talk. A listener’s goal may be to learn more about the topic area (at a conference); to assess you, your research, or your teaching capability (at a job talk); to find new collaborators (at a workshop or working meeting); or to give you feedback (at a practice talk). In each case, you want to help the listener achieve their goals. Otherwise, they will feel like they are “not getting anything out of your talk” and quickly tune out. If you can connect your talk with the listener’s goals, you will have an easier time keeping your audience engaged.
Every talk should have a message objective, which has two components: (1) persuading the audience to do something (read your paper, work with you, give you feedback); and (2) explaining what’s in it for them. For example, your message objective might be “By working with me on this problem, you’ll get to work on a practical problem in an under-explored area.” or “By downloading my software, you’ll be able to solve the software troubleshooting problems that you’ve been having so much frustration with.” or “By reading my paper, you’ll learn techniques that you could apply to your own research.” or “By giving me a job, you will gain a great colleague.” There are a variety of message objectives, but every talk should have one.
Practice, Practice, Practice
Never give any talk without some form of practice. Typically, you should deliver your practice talk to some of your colleagues. For “lower stakes” presentations (e.g., working meetings), you may be able to get away with a quick run-through by yourself, but as a general rule, you should always practice. Any talk represents you, your work, your advisor, and your institution, so it is generally a good idea to incorporate as much practice as you can practically achieve.
For “high-stakes” talks (e.g., job talks, conference talks), where you must communicate the value of your work to an audience with a variety of backgrounds, biases, or both, it is critical to hone your message and delivery. Talks at conferences are, in some sense, “previews” of what your job talk will eventually become, and they contribute to your reputation as a researcher. People remember clearly delivered talks; on the other hand, if your talk is not clear, the best-case scenario is that they forget the talk (and that you delivered it).
My own rule of thumb is that if I can mentally go through the talk in my head—in terms of knowing slide order, and roughly how I will start each slide—then my delivery will be smooth. If I can’t mentally “rehearse” my slide order, I know I’m not that well-prepared.
Practice makes more. Practicing is absolutely essential, but practicing alone is not enough—feedback and iteration is essential. The old adage “practice makes perfect” isn’t quite right. If you practice the wrong habits repeatedly, you’ll merely solidify bad habits. Therefore, it’s best to practice with other listeners in the room, to determine if your talk needs revisions and course corrections. If possible, try to have people listen to your practice talk who share the background and biases of the audience that you’ll eventually present to. If you are preparing a department-wide colloquium or job talk, packing the room with only researchers in your area is not likely to yield the best feedback; rather, you want people from outside your area to also listen to the talk, to simulate the environment, questions, and reactions that you will receive at the talk itself.
Practice. Then, practice some more. Depending on your purpose and audience, you may need to practice a lot. You will not get this right on the first “take”, and, just as you would revise a paper, revising a talk to match the audience and your purpose takes iteration. Just as you would obtain feedback on a paper draft, rewrite it based on feedback, and then read it again, you must re-deliver your talk after implementing feedback. In doing so, you may find that some suggestions didn’t work that well, or that you need to refine your implementation of certain suggestions. Also, every revision disrupts your level of comfort with the talk, because content and flow keeps changing. Practice talks help restore comfort and confidence with revised slides and material.
Don’t let the lack of an audience prevent you from getting feedback. Although it is always better to practice delivering a talk in person, in front of a live audience, sometimes circumstances don’t allow it. Scheduling practice talks can be difficult sometimes, particularly with busy, overcommitted faculty members. You may happen to be busy or traveling around the time of the conference, and schedules may not line up. In these trying circumstances, though, you can still do better than simply practicing by yourself (which does not give you the same type of “performance” practice) or mailing around copies of slides (which can often be unintelligible—or hard to interpret or react to—without the appropriate “audio track”). In these circumstances, I recommend Josh Reich’s “Delay Tolerant Talk Preparation“.
A significant portion of your talk preparation will include preparing the appropriate visuals. The point of visuals should be to assist you in your delivery of material, but the focus should remain on you, your speaking, and the work. Slides should only serve as visuals to help your delivery; they are there to illustrate, clarify, solidify, or corroborate what you are saying. Anything that distracts from you or what you are saying is not productive.
Keep it simple and spartan. Some presentations have flashy graphics, animations, colors, and design elements. Generally speaking, none of these “frills” matter, and they can actually backfire. I remember attending one particular conference talk that had particularly flashy animations (zooms, flyovers of network graphs, etc.). After the talk, nobody was talking about the work; rather, people were discussing how flashy the talk was. I find that black, sans serif text on a white background with minimal animation works well. Similarly, keep figures and plots as simple as possible while still conveying the point. People are bad at multitasking; if they are reading your slide (or otherwise trying to digest it), they are almost certainly not listening to what you have to say. Keep your visuals simple, so that people can return to paying attention to what have to say as quickly as possible.
Don’t use text when an appropriate figure will do. If a picture is worth a thousand words, then the limited real estate of a slide suggests that you should use figures whenever possible. That said, the keyword is “appropriate” figure. Figures should be as simple as possible. They should omit details of the concept that you are not going to describe, or are not central to the idea you are presenting. The goal of a figure is illustration, not necessarily accuracy. For example, when presenting a network architecture, it may be reasonable to leave out certain nodes, edges, or system components that aren’t central to the idea.
Don’t use a graph when an appropriate phrase will do. Sometimes a graph is a great way to illustrate a point, but presenting graphs is not “free”—they require significant audience effort and bandwidth to digest, so you should always ask yourself if a simple summary of the result will do (especially if the audience can find the details and graphs in the paper). Writing “our system reduced median latency by 20%” might convey a point much more clearly and efficiently than a complicated graph like a cumulative distribution function, particularly if your audience is not used to reading that type of graph. Remember that for every graph you include, you need to explain what the graph is, what data or experiment was used to present it, what the axes mean, and how to read it. If I must include a plot, I always put a box on the slide with an arrow pointing to the part of the graph that I want people to look at that conveys the main result. If the result is a reduction in median latency by 20%, for example, draw an arrow on the slide with a text box that states the main result. This way, even if people don’t understand how to read the graph, they might still be able to digest the main result.
Beware small fonts and strange colors. Fonts should always be at least 24 point font. As far as colors are concerned, beware the projector. Every projector is different, and certain colors do not show up well. Also, some audience members may be color blind. Therefore, take care in choosing colors that project well, and try to design your slides so that understanding them does not depend on the ability to decipher color. For example, lines on a graph should not only have different colors, but also different markers for the data points on the lines. White text on black backgrounds is also generally a bad idea: Black is the absence of light, and projectors are designed to project light. Therefore, black is never truly black, and on some projectors, poor contrast can make white text very tough to read. Also, people are generally used to reading black text on white backgrounds (books, magazines, etc.), and inverting this mode is distracting at best.
Practice. See above. It’s worth mentioning again. A smooth delivery depends on lots of practice. As you gain more comfort with your talk, filler words (“um”, “like”, “you see”, and so forth) will disappear.
Tell a story. The human mind is very good at processing and remembering stories. People love stories and are much more likely to stay engaged if you tell a good one. Stories always have a beginning, middle, and and end. A good story also typically also has a “dramatic arc”, which may build anticipation, contain surprises in the plot, and otherwise manipulate the reader’s emotions.
- The beginning of your talk is straightforward: it has a short (and easily repeatable) summary of the topic of the talk and your message objective. (“This talk is about [topic]… By the end of the talk, [message objective]…”).
- The middle of the talk should start with a “hook” that draws the audience in and helps them identify with your topic or problem. Tell an anecdote. Relate your work to a current event. Present a surprising statistic. Ask a question. Do something concrete to engage your listeners within the first minute, or you may never get them back.
- At the end of the talk, restate your message objective and leave them with no more than three things that you want them to remember from the talk. If you have one thing you particularly want the audience to remember, tell them. Also, tell them what you want them to do as a result of listening to the talk. Do you want them to contact you? Read the paper? Try out your software? Give you feedback? If you don’t tell people what you want them to do, they won’t do it. Also, make it easy for them to do it. When answering questions at the end of the talk, leave your conclusion slide up with the 1–3 takeaway points, and “instructions” for what you want them to do (e.g., if you want them to download code, provide a URL; if you want them to contact you, provide an email address).
Slow down. Nobody knows your topic as well as you do. In a technical talk, people are not only listening to what you are saying, they are processing the information as you present it, typically learning it for the first time. Talk slowly. The temptation to race is pretty significant with slides because you aren’t bound to discuss everything that’s on a slide, and you may even forget to explain something on a slide if you are going too fast (in fact, I find that using slides in classroom lectures is much less effective than writing on a whiteboard, which forces me to slow down my delivery).
In addition to cognitive reasons for slowing down, acoustics also play a role. In a large room with your voice being piped through a microphone, echoes and delays are at play, and speaking too quickly can obscure your delivery. My general rule of thumb is that each slide should take about 1-2 minutes. If you are progressing through slides more quickly than that, you’re going too fast. If you feel as though you are speaking too slowly, you are probably going at about the right pace.
Breathe. Remember to regularly take deep breaths. This will slow you down, but it will also help you gain some comfort with silence. It can be terrifying to stand in front of a group without sound coming out of your mouth, so a second of silence can feel like an eternity. However, people appreciate a slower pace, and unless your silence is erratic or unexpected, it will likely improve your delivery. Breathing is also a useful tool for eliminating filler words such as “um”. The next time you are not sure what to say next in a talk, try: (1) pausing; (2) breathing in deeply. It takes some practice to breathe in rather than using a filler word, but its effects are magical: with no change to actual content (and no additional practice), your delivery sounds pensive and thoughtful, rather than bumbling and unrehearsed.
Use vocal energy. Volume adds emphasis. Whispering (assuming you can still be heard) can act like a magnet, drawing people in. Speed can energize. A slower pace can inspire wonder or cause people to pay closer attention. Pauses are useful for emphasis. Avoid a monotone presentation. Some parts of a presentation are clearly more important than others—the beginning where you get people interested, the main result, the “sync points” where you draw in audience members whose attention may have drifted—use vocal energy to keep the audience on track.
Get the little stuff out of the way. Getting comfortable with the room in advance can help you relax, as well, so that you don’t have to worry about the “little things”. If possible, go stand on the podium before you give your talk so you can get comfortable with “the view” that you will have when giving the talk. Make sure that your laptop works with the display to avoid last-minute panics. Always have a last resort: Put a backup copy of your slides on the Web in a commonly readable format (e.g., PDF), so that, if all else fails, you can quickly load them up on someone else’s laptop.
Relax and have fun. Remember that, if you are presenting your own research, you know your topic better than anyone else in the room. Relax and stay confident. Most importantly, have fun. With the appropriate preparation, giving talks can be quite a lot of fun. It’s a way to advertise your work, meet new people, educate, and increase the impact of your work. It can also be a celebration of an excellent piece of work. Make the most of every opportunity!
Research creates new ideas, paradigms, and discoveries. The process of creating new ideas can seem daunting, and somewhat of a mystery. From some apparently unstructured environment, the mythical process of research will produce new discoveries and knowledge. Picking a good problem is one important piece of the puzzle; as I have previously discussed, developing taste in research problems and learning how to identify good problems can take many years of experience (indeed, part of the Ph.D. process itself is gaining experience and research taste).
Before we begin our research careers, we experience structured training: we attend classes, read textbooks, and grind through problem sets in areas that might be called “established science”. In contrast, the research process appears completely unstructured—researchers create new ideas and discoveries, seemingly from nowhere. Research papers convey the outcomes of research, but they do not shed light on the process of how the problem was identified or how the solution was devised. (In particular, most papers don’t discuss the countless problem formulations or solution attempts that failed before the breakthrough occurred.) A paper typically only tells a story about the broader context and contributions of the results after the breakthrough has occurred, with the benefit of hindsight and perspective. Reading a research paper without insight into the process can make research seem even more mystical. “How did they think of that?”, you might wonder? In this post, I’ll try to at least partially demystify the process of finding research problems and solving them.
Research patterns. In fact, the research process is a lot more formulaic than it might appear. I find that applying particular formulaic approaches tend to work fairly well for identifying (and solving) important problems. In fact, I would posit that there are research patterns, similar to the software engineering concept of design patterns. A design pattern is a reusable solution to a commonly occurring problem. Example design patterns include iterators (for lists) and locks and thread pools (for concurrency). In research, we also face common problems: selecting a problem, understanding the problem, and planning and executing a solution to a problem. Fortunately, there are research patterns—reusable solutions to each of these challenges—that can help solve each of these problems. Below, I will propose some research patterns for each of these challenges that we commonly face in research.
Finding a Problem
How does one find a problem to work on? There are certainly mainly difficult problems to work on, some of which are worth solving as a researcher and others which are best left to others. Developing research taste can help us determine which problems are worth solving, but where does one find research problems in the first place? Below are a few research patterns that I have applied and found useful.
Hop on a trend; look “upstream” for the trends. Researchers tend to coalesce around certain paradigms as they emerge. Now, you could of course try to create your own paradigm, but doing so involves effectively staging a revolution that changes the way an entire community thinks (including established researchers who may be slow to adopt radical thinking, yet hold considerable clout; the topic of revolutionary science deserves a post of its own, and I’ll write more about scientific revolutions in a subsequent post). Short of setting your own trend, you can very easily look around and identify (and even predict!) the community zeitgeist. Reading conference proceedings to identify the zeitgeist is reasonable, but this might place you as an “also ran” in an area where most of the initial breakthroughs have occurred. A better place to identify trends, I think, is “upstream” of conferences. Funding agencies often set research agendas, for example, through “calls for proposals”, and the research community chases those opportunities (“following the money”). So, for example, if you see a funding call for (say) digital health, it’s reasonable to expect that three or four years down the road, a trend or sub-community may emerge around this topic. Funding agencies even hold workshops to help shape these trends, before calls for proposals are released. Those workshops often provide significant hints about topics that might appear in a call for proposals—and, hence, what might eventually become a “trend”. In other words, trends don’t come from thin air—people define and set trends. If you can’t be a trend-setter yourself (something that’s very tough to do as a Ph.D. student), it’s best to be on the early side of a trend, and looking upstream can be incredibly helpful for doing so.
Develop a secret weapon and look for nails to hit with your hammer. Become a domain expert at something, or develop knowledge or resources that others don’t have. By developing some expertise, you become attractive and valuable to many people, and people will begin approaching you with ideas and want to collaborate with you. People will actually start coming to you with their problems. The expertise you develop can come in many forms, and it effectively becomes your secret weapon. You might, for example, develop a software toolkit or system that other people can build on. By developing a system that others want to use, you place yourself at the epicenter of new ideas that might build on that system. Or, perhaps you might become an expert on a particular topic or area—a particular set of network protocols, a particular piece of hardware, a particular programming language, or a particular set of statistical methods. Once you have developed that secret weapon, you can use it to solve problems that others are not able to solve, or to develop solutions that others might not be able to think of. Once you have developed your secret weapon, you can begin to look for problems where you might be able to apply your unique knowledge, system, dataset, or expertise.
Revisit old problems where assumptions may have changed. Old problems can be a great source of new problems. Previous solutions to old problems may have assumed certain constraints about processing power, storage, the cost of memory, the set of prevailing applications or protocols, and so forth. Yet, underlying technologies are continually advancing. A problem that was difficult to formulate or otherwise intractable five years ago might suddenly become solvable because old assumptions are now invalid, or because the emergence of a new technology (or algorithm) makes previously hard problems suddenly easier to manage. It is worthwhile to periodically revisit problems that have remained difficult and unsolvable for many years and to consider whether recent developments have made the problems any more tractable. Lots of areas of research have unsolved problems (not only theory!). For example, network management problems have remained particularly vexing in computer networking for a long time, but the emergence of recent new technologies has suddenly provided ways to make traction on problems that were previously hard to even formulate. Your area may have similar developments, and you should continually be on the lookout for those underlying phase shifts.
Look for pain points; eliminate them. Automate existing solutions. It is worth looking to industry (and even other researchers) to identify problems that continually recur. As programmers, if we have to do something more than a few times, we eliminate that pain point by writing a script. In research, if the same problem continually recurs and is being solved in the same silly, inefficient, or suboptimal way, certainly there must be a better way. People in industry often face real, important problems, but are may be too busy to step back and take a fundamentally new approach to solving an old, recurring problem. Fortunately, as a researcher, you have time to consider whether the problem is being solved in the right way, or whether a “pain point” exists that could be eliminated with a fundamentally new approach. Along these lines, many existing problems remain manual and painful, and automation of an existing solution can often significantly improve the state of the art. Yet, automation can be incredibly difficult because it often requires complex reasoning, which may be the basis of an important research problem.
Dream, and maintain wish lists. Many research solutions are ultimately about making everyone’s lives better. Think about the “grand” research problems—curing cancer, putting a man on the moon, and so forth. These are dreams that seem like science fiction, yet they are achievable. Each of these dreams involves the many sub-problems, some of which require the creation of new knowledge. (If they didn’t require new discoveries, the problems would be solved already, because we’d just “turn the crank”.) A good way to identify research problems is to maintain a wish list of your own, as well as a list of questions that you’d like to know the answer to. If you don’t know the answers and can’t easily find them, then the answers may not be known. Answering them is then, by definition, research. These might be large dreams or questions, or they might just be simpler things that bother you and you wish you had a solution for. Maybe you’re getting too much email spam and want to come up with a better way to solve it. Maybe you want to know how authoritarian governments block access to certain information or materials. A good place to start for many research problems is to identify which problems you would like solved that could make your world a better place.
Specialize the general. Applying constraints to a general problem in different ways can sometimes yield a new class of problems. For example, in networking, “routing” (the process of computing a path between two points) is a classic general problem. One can create new classes of problems by applying a constraint to the general problem: routing in wireless networks, routing in sensor networks, routing in delay-tolerant networks, and so forth. Of course, it’s best to find constraints that reflect realistic and important real-world scenarios.
Generalize the specific. Some problem areas have been subject to solutions that “chip away” at the problem, offering point solutions, rather than trying to tackle the larger problem area. When reading a collection of papers on a particular problem, it can be helpful to determine whether any particular paper proposes a general approach or algorithm, or whether each merely offers a point solution to some aspect of the problem. If particular papers only offer point solutions, there may be room for generalization.
Look for linchpins. When looking for problem areas that are in need of generalization, I think it is helpful to identify if a problem area has a linchpin, whereby a collection of papers each tackle a problem in a specific way, based on the assumption that some underlying problem is already solved. That keystone might represent a significant research problem, since so many specific solutions rely on a solution existing. For example, one (open) problem in censorship circumvention is “bootstrapping”: every paper on censorship circumvention that I have read assumes that there is some way of distributing the software and the initial information about how to configure that software to reach an initial set of trusted parties. All of the existing systems assume a solution to this problem, but no such solution exists. A solution to the problem could crack open an entire area. That’s a good problem to work on. The theoretical computer science and cryptography communities have some famous examples of this (e.g., P vs. NP, discrete log problem, factorization), but I believe most communities have these linchpin, although they may not be quite as commonly accepted or well-known.
Teach. I have found that there is an extremely tight coupling between teaching and research. When teaching concepts, I aim not only to teach the concept, but also explain the rationale behind the concept. For example: How was a particular network protocol designed? What is the rationale behind a particular design choice in a system? In seeking to explain certain concepts or theories, sometimes, we can find that things are difficult to explain. Sometimes, this reflects a gap in our own knowledge or cognition, but sometimes what we are attempting to explain might actually not make sense—the concept, approach, result, or system you are attempting to explain might in fact reflect an incorrect result, a bad design choice, or (perhaps more likely) one that no longer makes sense given today’s constraints, circumstances, or common assumptions. When you find yourself having trouble explaining the rationale behind a concept, you may have identified a new opportunity or problem area.
Solving the Problem
If solving a problem were easy, someone else would have likely already solved it. Therefore, identifying a solution to a problem (much like finding the problem itself) often requires a new perspective or way of thinking. Fortunately, problem solving also has several research patterns that seem to repeatedly arise. I have commonly used and observed the following approaches to solving research problems.
Consider related problems. Try to restate the problem you are trying to solve in a different way. Consider different terminologies and representations of the problem that you are trying to solve. By changing the form of the problem and trying to describe and represent it in different ways, you might find that your problem matches a general problem that is already formalized. For example, the problem of identifying whether an Internet service provider is intentionally degrading performance might be referred to as a “treatment” of class of customers. That terminology might make you think about “random treatment”, a process by which biologists can determine whether a particular drug or chemical has any positive (or negative) effect on a group of humans. Trying to recreate the conditions of random treatment in a network environment might lead you to a statistical approach to solving the problem of identifying ISP service degradation. This thought process was exactly that of one of my former Ph.D. students, which led to this solution.
Make analogies. Analogies are incredibly powerful. We use analogies to learn all the time, because we learn a new concept best by relating it to a concept that we already understand. Similarly, you can solve a hard problem by relating it to a problem that you already know how to solve. Analogies often create the biggest breakthroughs when they come from outside of your immediate discipline (these are the conceptual blockbusters that other people often aren’t thinking of because they’re typically not looking for solutions outside their immediate discipline). Computational thinking was one highly publicized example of applying analogies to problem solving—the notion that concepts that we learn in computer science (sorting, queuing, etc.) can help us solve problems outside of the discipline. But, these analogies can also be applied in reverse. For example, researchers have applied concepts from epidemiology to understand how computer viruses spread. These analogies—when applied well—can also often point exactly to a solution, since the solution that applies to the analogous problem can sometimes be translated to the problem you are studying.
Change the problem to one you can solve. Make simplifying assumptions that violate some of the problem constraints, or define some approximation of the ideal solution. In some sense, this is the dual to the “finding the linchpin” approach to finding a research problem. Many of the censorship circumvention papers would not have been written if they did not first assume some ability to securely and covertly distribute software and set up an initial configuration. By making some simplifying assumptions, these papers have made some progress. And now, these papers have created a linchpin that’s an important area for new unsolved problems.
Just get started, with anything. Just like writing anything down on the page is one way to overcome writer’s block, starting with any solution to a problem (however seemingly bad) can get you started. The process of iterative refinement is powerful. In algorithms, propose a simple (sub-optimal) algorithm, and check its correctness. Refine until you have something that works. In data analysis, measurement, or modeling, look at simple statistics of a dataset—timeseries, averages, histograms, etc., and begin to look at anomalies. (I like to say that an anomaly in data analysis is either a bug or a paper!) In systems, start with a simple, if imperfect, design. Try to implement it and see where stumbling blocks arise. Those stumbling blocks may represent the hard research problems, the solutions to which might result in new discoveries.
Consider nature. When thinking about solutions, ask yourself the questions: How does a human naturally solve this problem? How does nature solve this problem? Certain problems in computer security have been solved by considering, for example, human immunology. Other advances in miniature robotics have come about by studying the behavior of bees. We estimate our distnace from a lightning strike by watching the lightning and then counting the time it takes for us to hear the thunder; the time elapsed multiplied by the speed of sound gives us the distance. This same approach has been applied in location systems. Considering nature’s approach to solving problems is a specific way of applying analogies.
Work backwards from the goal. In the words of Yogi Berra, “If you don’t know where you’re going, you might not get there.” I find it helps to sometimes have a desired end result in mind, and then to figure out what is needed to get there by breaking the problem of reaching the end state into smaller sub-problems. I sometimes use a specific version of this approach by asking my students to draw the graphs that they would like to see in their final paper—complete with axis labels and trends. Given that graph, table, or result, what data do you need to gather to produce the result? If you can’t get exactly that data, can you approximate it? Do you need to develop (or apply) any special analysis techniques to produce the result? Or, suppose you’re aiming to have a working system that performs some task. What do you want that system to achieve? Now, what are the building blocks that need to be in place for the system to achieve those goals? Those building blocks are either solved problems, problems that you need to solve, or problems that you need to dispatch with some simplifying assumptions (see above on simplifying problems). Working backwards from your goal in this way can often provide a useful roadmap towards the solution. It can also allow you to solve smaller parts of a larger problem separately (starting with the parts that are most tractable); sometimes, finding a solution to a good sub-problem can turn out to be a significant research contribution by itself.
Think in speech or pictures. We have all had the experience of trying to verbalize a problem to a friend or colleague, only to find ourselves saying “nevermind, I’ve figured it out” before we even finish describing the problem. Sometimes, the process of thinking in speech can help us arrive at a solution, because verbalizing the problem can help us make the problem concrete and structured in a way that it wasn’t when it existing solely in our heads. Similarly, drawing a picture can help us think about how a problem is structured and how various sub-problems and components relate to one another. While the process of communicating a problem in words or pictures can help us arrive at a solution, it also sometimes helps a lot to have someone who is listening and approaches a problem in a different way. Describing a problem to a colleague from a completely different area might trigger a seemingly naive question that causes you to break out of your way of thinking or discard your current set of operating assumptions—breaking down mental barriers and leading you to the breakthrough. On that note, it is also useful to relentlessly come at a problem from a variety of different angles. When applying one set of algorithms doesn’t work, put it aside and try something else.
Teach (and learn). Just as teaching is useful for identifying problems, it can also help us identify solutions. If we are teaching “well”, we are constantly trying to keep abreast of new technologies and building blocks for solutions. By keeping abreast of the latest technologies, we can stay aware of tools that we can apply to problems where we might otherwise remain stuck. For example, I had been working on a certain network management problem for some time. In the process of teaching new approaches to network control, I realized that new programming languages that were being developed allowed us to solve some longstanding scalability problems with the solutions that we had come up with. Technology and knowledge is never in a steady-state; it is continually advancing around us. Someone is probably working on a technology, concept, or theorem that could help you solve your problem. Teaching is a good forcing function to continue learning about these advances and increasing the likelihood that you become aware of them.
Relax, and let your subconscious work. Above all, immerse yourself in the problem, but stay relaxed. Take breaks. Generally, creative insights come when we give our minds a break and let them think without structure. We sometimes create a process for this unstructured thinking, which we call “brainstorming”. But, probably better than brainstorming is “brain-resting”, where you take a break, go for a walk, go for a run, take a nap, or just go do something else. Often, the solutions will come when you least expect it.
Just as each of us develops taste in books, music, art, and food, every researcher ultimately develops a taste for research problems. Every researcher should spend some time developing “good taste” in research problems. The world has many challenging problems to work on, and as researchers, we have limited time and bandwidth. It’s therefore important to develop (good) taste in selecting problems, so that we end up working on the problems that are worth a significant investment of time and energy. Many research problems will take years to run their course, so it is worth spending some time developing taste in problems.
Many professions, ranging from designers to architects to programmers to managers, need to develop good taste. In this post, we’ll focus mainly on developing taste in research problems, although some of these tips for cultivating taste may apply more generally (in fact, some of the pointers below were inspired from an article I recently read about developing taste in design).
Cultivating your own research taste. We are not born with good (or bad) taste; rather, we develop taste by way of education and exposure to many different opportunities and experiences. Just as one might cultivate taste in other areas of life, one must cultivate taste in research. In this post, I’ll offer some tips for cultivating research taste that I’ve found work for me. Some of these tips I have discovered (and applied) by way of analogy for developing taste in other realms (e.g., music, food). I’ve spent a fair bit of effort developing taste in music; where applicable, I’ll draw some analogies below.
- Seek out others with good taste. Perhaps the most important step for developing good taste is to associate and learn from others who have good taste. These people are generally acknowledged as “having good taste” and are often easy to identify. Just as you may have friends who you know are well-read about music, wine, or food (you know these people because you’re always asking them for recommendations), the research community has “thought leaders” who are widely acknowledged as having good taste in research problems. It’s not too hard to figure out who these people are with a little research. Ask several colleagues who these people are and see whether trends begin to emerge. Poke around on Google Scholar and see which researchers in your area have highly cited articles on a particular topic. Intersect these people with those who share your interests, as well. Once you have identified others with common interests who appear to have good taste in problems, try very hard to associate, exchange ideas with, and work with those people. Become an apprentice. As a Ph.D. student, seek these researchers out as possible advisors. The Ph.D. years are perhaps the most formative for developing research taste, and your taste will likely be shaped heavily by your advisor, so taking the time to find an advisor who will develop your research taste is perhaps one of the most important decisions you will make as a Ph.D. student.
- Read trend-setting conference proceedings, and develop opinions about research problems and trends in your area. The “top tier” conferences in your area are essentially the Architectural Digest, Wine Enthusiast, or Pitchfork for your discipline. Track your conference proceedings to determine the research areas that the best researchers in your area are working on. You don’t necessarily have to “jump on the bandwagon” and start working on any of the research areas that are the current hot topics at this year’s conference—just as you might not spend several hundred dollars on the latest wine that’s reviewed or go right out and buy every new music release—but it certainly doesn’t hurt to learn about the latest trends, even if you don’t always resonate with them. Exposing yourself to the latest trends and developing opinions about them (positive or negative) is an important step in cultivating research taste. It’s typically not necessary to read the entire conference proceedings to get a feel for what’s going on in an area; simply looking at the names of sessions and groupings of papers can help you quickly identify areas that are receiving a fair amount of attention from the community.
- Sample and experiment with abandon. Developing taste in any genre involves gaining exposure to many different examples, good and bad. Just as it’s much easier to appreciate a truly fine wine, dish, or performance after having seen mediocre offerings, since every experience allows us to better articulate what we do or don’t like, sampling a wide variety of research problems (and solutions) is a necessary step for developing taste in research. In developing taste in music, I find myself reading continually about new artists and albums and listening to new material that pushes boundaries in new ways, and sometimes subjecting myself to music that in the end I might decide I don’t like all that well. Similarly, in research, we must continually experiment and sample to develop and cultivate our taste. It can be tempting (and certainly easier) to “turn the crank” on problems that we know how to solve, but ultimately this will result in research that becomes stale and boring. As researchers, we should be continually learning about new techniques, tools, problems, approaches, and so forth. We will likely find that some topics, areas, and solutions we encounter are incredibly boring, but those “bad samples” ultimately help us determine what aspects of a research problem or solution we do or don’t like.
- Keep a list of ideas that you like and exchange your favorite ideas with colleagues. Keeping lists of things you like is good; exchanging them is even better. Every year, I tabulate a list of my “top 10″ music albums for the year; at the end of the year, my friends and I exchange lists. The list is fine, but making the list is actually more important. Knowing that I am going to be exchanging a list of music with friends at the end of the year keeps me accountable and ensures that I am always “on the lookout” for new gems. Similarly, making lists of neat research ideas and regularly exchanging them with colleagues is another way to ensure that you always have a healthy appetite for new, creative ideas. I don’t exchange lists of research ideas, although perhaps I should; I do, however, regularly exchange papers, articles, and ideas with a small, trusted group of colleagues, often multiple times a week.
- Attend conferences. Just as a music enthusiast attends live concerts or a book enthusiast might attend a reading, as a researcher you should attend research conferences. I think it is worthwhile to attend at least one major research conference in your area every year. Attending a major research conference ensures that you are staying current with research trends (in case you’d like to “watch the movie”—that is, attend the talks—instead of simply poring over the conference proceedings). Perhaps more important than attending the talks, however, is meeting other researchers. Conferences are excellent places to seek out others with good taste, and to have conversations with other researchers that can help you develop opinions about various research problems and areas. These conversations and interactions are all part of the process of developing taste as a researcher.
- Consider the principles and theories that underpin a specific research paper or project. No research paper (or project) is perfect. In fact, most papers are flawed, and many are badly flawed. Yet, papers are often published not for the particular artifact that they produce, but for some underlying concept or idea that they embody or espouse. When reading research papers, it is far too easy to be dismissive of a paper because of a flaky implementation, a bogus evaluation, or poor exposition. Try to dig a bit deeper and understand the value that reviewers might have seen in a particular paper. For example, a paper may develop a new theory or changes our way of thinking. It might open new avenues for research. It might be applicable across many disciplines. It might frame an old problem more clearly. Look beyond the flaws of a particular instantiation of an idea and consider whether the underlying theories or concepts of a particular approach or solution have value.
Evaluating your experiences and encounters. Simply having exposure to many research problems and areas is not sufficient to cultivate taste; you need a way to evaluate each new research problem, paper, or area that you encounter. Just as a connoisseur develops metrics for evaluating a new piece of music, a new book, a bottle of wine, or a restaurant, we need to have yardsticks for evaluating research papers and problems. When evaluating research that I encounter, I consider the following questions to help further develop my taste. The answers to these questions are subjective (i.e., reasonable people may disagree about the value of the same piece of work), but they can nonetheless help you articulate why you like (or dislike) a particular research problem or solution.
- Is the problem important? Of course, “important” is subjective and defined by your own set of values. I personally like problems of practical importance. For example, if the solution to a problem would ultimately improve a network’s performance, security, or availability and could affect the lives of a large number of people in a meaningful way (e.g., developing better spam filters, circumventing Internet censorship), then I deem the problem as important.
- What is the “intellectual nugget”? I like research problems (and solutions) that have a simple, elegant solution that’s intellectually profound and easy to articulate. That might sound trite, but there are plenty of research papers (and researchers who write them) that involve a hodge-podge of solutions with no crisp intellectual contribution (e.g., “We encountered problem A, so we applied X. Then, we encountered problem B, so we applied Y.” Repeat ad nauseam.) In my opinion, many of the best research problems (and solutions) can be succinctly summarized in a single sentence. I value simplicity. You may not, just as some designers like ornate designs and others prefer minimalist ones.
- What is the solution or main conclusion? Is it important? Although I believe that the size of the problem is at least as important as the goodness of the solution, it is worth considering whether the solution is important, usable, or worthwhile, as well. Just as “important” is subjective when evaluating problems, the importance of a solution is also subjective—perhaps even moreso than the importance of a problem (which at least might have the benefit of some community consensus if the problem has been studied for long enough). Determining the importance of a solution is really difficult, even for people with developed taste. In fact, sometimes developed taste might allow us to overlook a particularly new or innovative solution, just as we can become comfortable with a particular genre of music and fail to recognize a true gem when something disruptive appears. The establishment is particularly bad at recognizing disruptive breakthroughs because they are used to thinking according to established paradigms. Our failure to reliably recognize good solutions is perhaps one of the biggest flaws of the peer review system. (More on reviewing in a later post.)
- Does the content support the conclusion? Does the content of the work actually support the solution? Are the methods sound and state-of-the-art, or have they since been obsoleted (an example of this in networked systems is the now-questionable use of simulation, which has been obsoleted in favor of prototyping and, more recently, operational deployment). What assumptions does the paper make, and if those assumptions change over time, do the conclusions still hold, and are they important? (For example, one might ask if a system that assumes the deployment of a particular protocol is still important if that protocol is never deployed, or is currently deployed but likely to be replaced.)
You might have a different value structure for what you think is an important research problem or solution and, hence, you might have different taste in problems. It’s important to articulate what you think is important, and taking some of the steps outlined above will help you refine your answer to these questions.