IEEE Transactions on Software Engineering Review Form
General Information
Log Number: _________________
Title: __Temporal Logic Query Checking : A Tool for Model Exploration_
Please Enter Your Full Name: _____Yunja Choi and Mats P.E. Heimdahl_
Section I. Overview
A. Reader Interest
1. Which category describes this manuscript?
___Practice/Application/Case Study/Experience Report
_X Research/Technology
___Survey/Tutorial/How-To
2. How relevant is this manuscript to the readers of this periodical? Please explain your rating.
___Very Relevant
_X_Relevant
___Interesting - but not very relevant
___Irrelevant
The paper is about a technique that can be used for model understanding that has a potentially large
application domain. The content of this manuscript is largely about the technical soundness of the approach
and does not cover applicability or methodology. Strictly speaking, the theoretical extension from previous
techniques is not all that significant. If the authors’ made a strong case for the feasibility of applications or
methodological support, it would have been a lot more interesting.
B. Content
1. Please explain how this manuscript advances this field of research and/or contributes something new
to the literature.
This manuscript extends the existing query checking technique to queries with multiple placeholders, shows
that query checking is an instance of a multi-valued model checking, and argues that query checking has
various application areas in model understanding including test case generation.
2. Is the manuscript technically sound? Please explain your answer.
_X_Yes
___Appears to be - but didn't check completely
___Partially
___No
This technique is an extension of existing approaches that have been accepted as technically sound.
For the extensions, the authors provided rigorous proofs. We have not found any problems with the proofs
after careful review.
C. Presentation
1. Are the title, abstract, and keywords appropriate? Please comment.
___Yes
_X_No
The title is somewhat misleading. It sounds like this manuscript is about a Tool (as in piece of
software) implementation of the Temporal Logic Query Checking, or something practical. In fact, most of
the content is about the soundness of the extended approach and translation from the query checking to
multi-valued model checking. In the abstract, the authors put an emphasis on the application of the
technique, but in our opinion, the major contribution of this paper is in its extension of the query checking
technique and the use of multi-valued model checking for the purpose—this should be clearly indicated in
the paper.
2. Does the manuscript contain sufficient and appropriate references? Please comment.
___References are sufficient and appropriate
___Important references are missing; more references are needed
_X__Number of references are excessive
Are all of 6, 7, 9, 10, 18, 19 necessary ?
3. Does the introduction state the objectives of the manuscript in terms that encourage the reader to
read on? Please explain your answer.
_X_Yes
___Could be improved
___No
The topic itself is quite interesting to us and the problem statement and background history are clear
and understandable. It also summarizes the major part of the work very clearly.
4. How would you rate the organization of the manuscript? Is it focused? Is the length appropriate for
the topic? Please comment.
_X__Satisfactory
___Could be improved
___Poor
It was reasonably well organized except that we do not understand why Section 5 is put ahead of the
other theory parts.
5. Please rate and comment on the readability of this manuscript.
___Easy to read
_X__Readable - but requires some effort to understand
___Difficult to read and understand
___Unreadable
Section II. Summary and Recommendation
A. Evaluation
Please rate the manuscript. Explain your choice.
___Award Quality
___Excellent
_X_Good
___Fair
___Poor
The theoretical results and the application of multi-valued model checking for query checking are quite
interesting and we enjoyed reading this manuscript. The approach has many potential application areas. We
can think of using this approach for severer problems we are currently investigating with theorem provers.
It certainly has theoretical and technical value. However, the idea itself is not entirely new and the
feasibility of this approach in real application is questionable. The Application and Experience section is
quite weak and the complexity section is somewhat puzzling.
B. Recommendation
Please make your recommendation and explain your decision.
___Accept with no changes
_X_Author should prepare a minor revision
___Author should prepare a major revision for a second review
___Revise as a short paper
___Revise as a regular paper
___Reject
Please address issues specified in Section III.
Section III. Detailed Comments
A. Public Comments (these will be made available to the author)
*** In an effort to adhere to our strict page budget and maintain a healthy yet short publication queue, we
are trying to better enforce our long standing page limitations and formatting guidelines with our authors. In
order to help them adhere to these guidelines, we need your support. One way you can help us meet these
objectives is by suggesting ways to maintain the lengths of their manuscripts should you decide to ask the
author to add new content.***
______________________________________________________________________
We have a couple of minor comments as follows.
First, on page 17, it is said a simplified version of Cruise Control System is used for experiments while the
original one is available. We wonder why. We cannot see the significance of the performance data over a
system with 14 variables. If the tool is implemented and the full version of specifications exists, we assume
using the full version would lend somewhat more credibility without much extra effort. If there is a reason
why the full version cannot be used, please clarify this. Also, we do not agree with the last statement of
section 8.2 and the last sentence of the first paragraph on page 34. The example, and associated
performance data, is woefully inadequate to say anything about how this will perform in practice. To us, it
seems like the example used in this work is an exceptional case, not at all representative of what any
practical system will actually look like.
Second, the reference list is somewhat excessive.
Third, more performance data on queries with multiple placeholders, queries with placeholders depending
on multiple variables, and queries with several temporal operators and mixed placeholders, and so on, will
be more interesting and more relevant to this paper. We assume it would not take much time to experiment
more with such queries and include some more performance data.
Reviewer one writes:
I also have some issues regarding the theoretical complexity and experimental results.
On page 33, theorem 13 says the worst-case complexity of query solving with one placeholder is linear in
the number of temporal operators in a query and the number of states, and exponential in the number of
atomic propositions in the structure, which is independent from the number of variables a placeholder
depends on. However, on page 35, it is discussed that increasing the number of variables that a placeholder
depends on slows down the analysis significantly and table 6 shows the slow down. The result makes
intuitive sense, but I don’t see how it is related to the theorem. The authors do not give a clear reason why
the performance should slow down for that case. Also, the complexity argument about queries with multiple
placeholders is in terms of | var(?) |. How do you relate this with theorem 13 and theorem 14?
I also do not comprehend the relationships between the complexity of naïve approach, which is expressed in
terms of the number of variables a placeholder depends on, and the other, which is totally independent from
it. Can we compare them?
Intuitively, the complexity should be related to the number of variables a placeholder depends on.
Shouldn’t it be somewhat proportional to the number of variables a placeholder depends on? (at least,
Table 6 backs up this argument). It seems like the complexity theory is too coarse to reveal this relation.
_______________________________________________________________________
B. Confidential Comments (authors will not see these comments)
_______________________________________________________________________
_______________________________________________________________________