How to write good systems papers?

First and foremost, I want to thank Yizheng Chen for inspiring and encouraging me to start my journey of academic blogging. Most tips in this post come from my group meeting with Hank Hoffmann.

As systems researchers, we build computer systems and then spread the idea to the world. This post discusses several tips for writing good systems papers from why to how.

Why do we care about writing?

After we build a new system, it is no use if only a few people know about it because we want people to use this system. Therefore, being able to argue why this idea really matters and why it is a good system is extremely important. It is what you have to do if you stay in academia.

Preparation

Before writing, keep a big picture in mind. From the expectations of reviewers, building systems is to provide new capabilities and better solutions of existing problems. With these goals in mind, ask yourself the following questions before writing:

1. What kind of problems do you want to solve? It is also the first question to ask before building a system. For example, it is not uncommon to spend a month benchmarking a system, but is it the problem you want to solve? Before you do anything, ask yourself if it will be a good use of your time. If you are not sure what problems to solve or just lose your track, talk to your advisor.
2. How to demonstrate your idea? How to convince the world that your idea and system are good? These are questions to ask for the presentation.
3. How to design experiments? What performance metrics to use? How to evaluate results? How will the experiments will look like? To answer these questions, it is helpful to draw some bar charts— computer architecture papers’ favorite — to formulate the whole picture in mind. For example, in the bar charts, x-axis represents different workloads, y-axis represents the performance metric, each bar with different colors represent different baseline methods. You will have multiple bar charts because you have data on different systems. If you are not sure of these, pause and think, and talk to your advisor.

Six questions

When you have data and start writing, answer the following six questions:

1. What problem is this paper solving?
2. What have other people done for this problem?
3. What are the limitations of other people’s work?
4. What are the key insights of this work to address these limitations?
5. How to implement systems and evaluate results?
6. What is the impact of the solution?

A good paper will be easy to answer the six questions. These questions are also interrelated to each other; i.e., 1 is related to 5. Ideally, you want to answer each question with one sentence. Take 2 as an example, it is not a good idea to just list a bunch of papers from prior work. Instead, the trick is to extract the consistent truth from these papers.

So, try to answer these questions before writing. During writing, you will also continuously refine the answers to these questions. Writing papers is not like writing an essay applying college or graduate schools: write, edit, and done. It is like constructing a building: prepare foundation first, then complete rough framing, and then keep installing and adding.

Writing anatomy

To write a systems paper, it is always helpful to draw a picture first. This picture is a clear illustration of each module and workflow for your system.

• Abstract: Most people prefer writing abstract last, while Hank suggested writing abstract first. The reasoning is that you can use this process to refine thinking and use six sentences to answer the aforementioned six questions that can be naturally constructed as an abstract. Of course, you will get back to it later and keep polishing it.
• Introduction: The beginning of the introduction discusses why the problem you solve is important (with many reference citations, of course). Then you discuss prior work and their limitations. Then, you discuss briefly how this paper address these limitations, usually with the help of the workflow picture you draw. The most important part is to list contributions for your work, which also tells reviewers how you will evaluate your implementations and system.
• Related work: This part just expands what you write for prior work in the introduction from one sentence to a paragraph. Usually, you can talk about an example from prior work with limitations and then it will be natural to bring up the insights of this paper to address them.
• Motivational example: Most systems paper have a motivation section to illustrate the limitations of prior work and explain the key insights with a simple example. The goal is to use a real but simple example to take a stand that you formulate the problem in the “correct” way.
• Technical: This part just describes how each module work and complement each other in the workflow picture. In the end, it is always good to add implementation details and limitation discussion. It is tricky to write good limitations because although all systems in the world have limitations, you do not want to write the rejection letter for the reviewers.
• Results: There are two types of results to show in the paper. One is called table stakes results. This is a term from poker saying that you can only play when you have money on the table. In the paper, you have to show results that you can do better than prior work. The other is called why results. You do not want to only show better results because every paper has them. You need to explain why your system works, and it is not because of your better implementation or other engineering tricks. You need to show that there are some properties, or key insights you are claiming are the reasons for the improvement.

General tips: from hating to tolerable

Here are some general tips to make writing less challenging and more productive:

• Force yourself to sit down. It is rare that someone is born to enjoy writing. Not to mention we are distracted by meetings, emails, social media, etc. all the time. Therefore, the first step is that you have to force yourself to sit down and focus. Some tools such as Pomodoro technique may help. With more practice, you can sit longer.
• Set separate goals at different stages. Beginners are usually slow to write. A familiar situation is that you start writing the first sentence, and then feel it is not good, so rewrite it. When you finish writing the second sentence, you feel the first still terrible, so you go back to change it again. Then, you spend three hours finishing only one paragraph, and still feel it not good. To address this issue, Hank introduced a game changer from a book called “How to write a lot”. The secret is to set separate goals at different stages. Here are some concrete steps: (1) You should write every day; (2) Allow yourself to only generate texts in the beginning; (3) At some point, switch focus and start editing. You can only edit texts at this point. Generally, the key is to set different goals and strictly stick to them. Another goal can be length. Systems paper submission has a strict page limit and it is not uncommon to exceed that. So set the length goal at some point and only shrink the paper rather than generating or editing.
• Put an outline. When you write a section, it is helpful to put an outline first by listing what each paragraph is going to do. For each paragraph, write the topic sentence and then start expanding it. It is like constructing a building again: roughly frame and then keep adding.

Practice makes perfect

Writing is not taking classes. You cannot just sit in a class and then get a high score, and done. Writing is a constant process. Keep calm and enjoy writing!