Writing Good Software
Engineering Research Papers
Based on the paper
Mary Shaw, Writing Good Software Engineering
Research Papers
Proceedings of the 25th International Conference on
Software Engineering, IEEE Computer Society, 2003,
pp. 726-736.
1
Research Papers

The basic and most important activities
of the research
• Visible results, quality stamp
• Means for communications with other
researchers
2
A good research paper should answer a number of
questions
 What, precisely, was your contribution?
•
•
•


What question did you answer?
Why should the reader care?
What larger question does this address?
What is your new result?
•
•
•
•
What new knowledge have you contributed that the reader can use
elsewhere?
What previous work (yours or someone else’s) do you build on? What
do you provide a superior alternative to?
How is your result different from and better than this prior work?
What, precisely and in detail, is your new result?
Why should the reader believe your result?
•
•
What standard should be used to evaluate your claim?
What concrete evidence shows that your result satisfies your claim?
If you answer these questions clearly, you’ll probably
communicate your result well.
3
Maturity of software
engineering discipline


Other fields of science and engineering
(physics, medicine…) – well known
methods
Software engineering – still not well
developed and understood
research/presentation guidance
4
1.
What, precisely, was your
contribution?

To precisely answer this, proper
(research) questions should be stated

What kinds of questions do software
engineers investigate?
5
,,,
6
Which type of questions
dominate?


Human-Computer Interaction: - many
new trends break through
Software Engineering
• mostly incremental (improved model,
improved technique)
7
8
What do program committees
look for?

The program committee looks for
•
•
•
a clear statement of the specific problem you solved
the question about software development you
answered
an explanation of how the answer will help solve an
important software engineering problem.
You'll devote most of your paper to describing
your result, but you should begin by explaining
what question you're answering and why the
answer matters.
9
2.

What is your new result?
Explain precisely
• what you have contributed to the store of
•
software engineering knowledge
how this is useful beyond your own project.
10
11
12
What do program committees
look for?

The program committee looks for
•
interesting, novel, exciting results that significantly
enhance our ability
• to develop and maintain software
• to know the quality of the software we develop
• to recognize general principles about software
• or to analyze properties of software.

You should explain your result in such a way
that someone else could use your ideas.
13
What do program committees
look for? What’s new here?
Use verbs that shows
Results Not only efforts
Try not. DO, or DO NOT.
There is no Try / YoDA
14
What do program committees
look for? What’s new here?
15
What do program committees look
for? What has been done before?
How is your work different
or better?

What existing technology does your research build on?

What existing technology or prior research does your
research provide a superior alternative to?

What’s new here compared to your own previous work?

What alternatives have other researchers pursued?

How is your work different or better?
16
Explain the relation to other
work clearly …
17
What do program committees look
for? What, precisely, is the result?





Explain what your result is and how it works. Be concrete and
specific. Use examples.
Example: system implementation
If the implementation demonstrates an implementation
technique, how does it help the reader use the technique
in another setting?
If the implementation demonstrates a capability or
performance improvement, what concrete evidence does
it offer to support the claim?
If the system is itself the result, in what way is it a
contribution to knowledge? Does it, for example, show
you can do something that no one has done before
18
3.
Why should the reader believe your
result?

Show evidence that your result is valid—
that it actually helps to solve the problem
you set out to solve.
What kinds of validation do software
engineers do?
19
20
21
What do program committees look
for? Why should the reader believe
your result?





If you claim to improve on prior art, compare your result
objectively to the prior art.
If you used an analysis technique, follow the rules of that
analysis technique.
If you offer practical experience as evidence for your
result, establish the effect your research has. If at all
possible, compare similar situations with and without your
result.
If you performed a controlled experiment, explain the
experimental design. What is the hypothesis? What is the
treatment? What is being controlled?
If you performed an empirical study, explain what you
measured, how you analyzed it, and what you concluded.
22
4.
How do you combine the elements
into a research strategy?

Not all combinations of a research
question, a result, and a validation
strategy lead to good research.
Question
result
validation
23
Combination question - research - validation
24
5.
Does the abstract matter? (YES)

people judge papers by their abstracts and
read the abstract in order to decide whether to
read the whole paper.

It's important for the abstract to tell the story.

Don't assume, though, that simply adding a
sentence about analysis or experience to your
abstract is sufficient; the paper must deliver
what the abstract promises
25
5.
Example of an abstract structure:

Two or three sentences about the current state of the art,
identifying a particular problem

One or two sentences about what this paper contributes
to improving the situation

One or two sentences about the specific result of the
paper and the main idea behind it

A sentence about how the result is demonstrated or
defended
26
Is this presentation a receipt
how to succeed?

Hm?

Several other conferences offer "how to write a paper"
advice:


In 1993, several OOPSLA program committee veterans gave a panel on "How to Get a Paper
Accepted at OOPSLA"

Partridge offers advice on "How to Increase the Chances Your Paper is Accepted at ACM
SIGCOMM" [15].

SIGCHI offers a "Guide to Successful Papers Submission" that includes criteria for evaluation and
discussion of common types of CHI results, together with how different evaluation criteria apply
for different types of results [13].

The SIGGRAPH conference program chair wrote a discussion of the selection process, "How to
Get Your SIGGRAPH Paper Rejected" [10].


The 2003 SIGGRAPH call for papers [21] has a description of the review process and a
frequently-asked questions section with an extensive set of questions on "Getting a Paper
Accepted".
27
Example




Challenges of component-based development
Ivica Crnkovic, Magnus Larsson
The paper presented at ICSE 200, as the first paper on the
conference
Selected as one between three papers published in JSS
28
Challenges of component-based development
Abstract

It is generally understood that building software systems with components has
many advantages but the difficulties of this approach should not be ignored.
System evolution, maintenance, migration and compatibilities are some of the
challenges met with when developing a component-based software system.

Since most systems evolve over time, components must be maintained or
replaced. The evolution of requirements affects not only specific system
functions and particular components but also component-based architecture
on all levels. Increased complexity is a consequence of different components
and systems having different life cycles.

In component-based systems it is easier to replace part of system with a
commercial component. This process is however not straightforward and
different factors such as requirements management, marketing issues, etc.,
must be taken into consideration.

In this paper we discuss the issues and challenges encountered when
developing and using an evolving component-based software system. An
industrial control system has been used as a case study.
Motivation
Problem description
Paper Overview:
(Implicit question)
what is the result
validation
29
Paper outline
1.
2.
3.
4.
5.
Introduction
The Case Study
Different Aspects of Reuse
Integrating Standard Components
Conclusion
30
Introduction



Reuse and an open component-based architecture are the keys to
the success of systems with a long lifecycles. Designing a system
that supports this approach, requires more effort in the design
phase and the time to market might be longer, but in the long run,
the reusable architecture will prove profitable.
…..
On each level of reuse there are specific demands on the reusable
components, on the component management and on the integration
process.

This paper describes important issues related to the development
and maintenance of reusable components and as an example uses
the ABB Advant industrial process control system.

In section 2 we give an overview of the Advant system design and
the main characteristics of Advant reusable components. Section 3
outlines all the development and maintenance aspects of a
component based system which must comply with customer
requirements. During evolution of the system new technologies
were developed which resulted in the appearance on the market of
many components with the same functionality as the proprietary
ones. The fact that new components must be incorporated into the
existing systems introduces new demands on the system
development process. These new issues are discussed in section 4.
What is missing?
Motivation
Problem description
Paper Overview:
- result
Detailed overview
31
Story/concept:
the pattern
Case:
Problem
Relevance of problem
Observation
Analysis/
Generalisation
Solution/
Analysis/
Example/case
Weak side of the paper:
Success factors:
Related work missing
Realistic situations
question(s) not explicitly stated
relevant problem
up/to date problem
Holistic approach
Technically sound
systematical validation trough the case
32