Summary:
The authors describe a framework for thoroughly testing IGS software (specifically IGSTK) to ensure that the software will behave as expected. In particular, IGSTK uses an explicit state machine within the software classes to make certain the the application is limited in its scope to only "safe" actions. The authors spec. out some goals of the testing framework and then describe their approach to testing.


Hypothesis:
Although there is no specific hypothesis, the authors' purpose is to (1) state that testing is important to IGS software and (2) they have an approach to test such software.

Evidence:
Although there is an implied completion of this work, the paper is mostly theoretical in nature. The authors make a clear case for why a person should test, what a person should test for, and how a person can achieve the testing framework.

Open Science:
The methods are completely open and clear. The IGSTK software is readily available; however, I can't find the code which does the testing.

Reproducibility:
Without the code, reproducibility is a bit more tough. The description seems to be clear enough that I could recreate the work and test it, although it would require a significant effort.


Use of Open Source Software:
Yep, everything here is open source, but again where is the testing code?

Open Source Contributions:
Can't say

Code Quality:
Can't say, but it appears to adhere to well-defined computer science principles. It is a benefit that the approach is compatible with other software (such as graphviz) so the authors are conscience of the open science effort.


Applicability to other problems:
Yes, this will be applicable to other forms of code testing. When the code becomes available, I hope that the approach can be better formalized to allow it to be used consistently between different applications and toolkits.

Suggestions for future work:
Got to get the code out there.

Requests for additional information from authors:
Ditto

Additional Comments:
[This is a free-form field]

Summary:
An architecture validation toolset is presented for a Image Guide Surgical Software (IGStk). The toolset is focused on validating the component state machines in IGStk.
Hypothesis:
The paper is based in state machine definitions that the authors assume in the IGStk context.
Open Science:
The authors develop the toolset as part of an open source project that is provided in a web site. Although, they don't provide experiments to reproduce, I haven't browse
so much in the source code to look for some testing examples.
Reproducibility:
After reading the opinion of the experts reviewers before, I haven't tried the code. Moreover, the authors don't provide experiments to reproduce. Perhaps,
the paper is too theoretician.
Use of Open Source Software:
All software is open source.
Open Source Contributions:
They develop their contribution for an open source project.
Code Quality:
I haven't checked the source code of this toolset.
Applicability to other problems:
This contribution could be used to adapt to other architecture validation.
Additional Comments:
The manuscript doesn't follow the IJ template, and it loses their advantages (http://insightsoftwareconsortium.org/wiki/index.php/IJ-Article-Template#Why_we_don.27t_use_the_Springer_LCNS_Style_anymore)
Some references in the text are missed.

Summary:
The authors present a set of tools to validate the state machine responsible for user interaction in the open source toolkit IGSTK. These tools, implemented mainly in Java (but some also in C++ and Ant scripts), analyze the XML definition of the state machine as exported by IGSTK and allow visualization, replay of recorded actions and a specialized coverage testing. Results can be displayed on the dashboard.

Hypothesis:
Invalid state machine definitions (leading to potentially fatal consequences in medical software) can be detected by a white box testing approach as described in this paper.

Evidence:
No specific claims are made in the paper, so I didn\'t expect any evidence. It rather is a description of the test system that was implemented.

Open Science:
IGSTK being an open source project and the paper a description of a certain part of that project, I would rate it as open science though there are not many results in the paper that could be reproduced.

Reproducibility:
I downloaded the IGSTK code and tried to find the test system described in this paper, but without success. There are more than 300 files in the source tree and without a detailled description of where to look it was too hard for me to find (if it is part of the sources I downloaded at all).
So all in all, I could not reproduce the work. The authors should point out where exactly the described toolchain can be found.

Use of Open Source Software:
All used software is open source.

Open Source Contributions:
As stated above, apparently the authors contributed the code of the testing system to IGSTK, but I could not find it in the latest release version.

Code Quality:
Not applicable.

Applicability to other problems:
In theory, the proposed test system can be applied anywhere where the interaction of a progam is described by state machines and a suitable export function to XML exists. In practice though, I think it will be limited to IGSTK - which is not too bad, since IGSTK being a toolkit can support many different applications.

Suggestions for future work:
Validate the validation tool: Feed invalid state machines to the system and check if the errors are detected properly.

Requests for additional information from authors:
It would be interesting to know if the proposed system has already helped to spot errors in the state machine and if yes, how many.

Additional Comments:
In general, I\'m missing the presentation of results of this work. It is not clear whether the system has been tested at all, I have got the impression that it currently is more a plan than an implemented system.

Summary:
The paper describes an in-development effort to create a set of tools for validating the state-machines by which the components of the open-source Image-guided Surgery Toolkit (IGSTK) are controlled. The tools (will) validate the conformance of state-machines to the IGSTK rules w.r.t. structure and behavior, support high coverage of possible state-machine execution paths by means of generated tests, and provide the possibility to replay state-machine executions. The paper focuses on the principles of the validation approach rather than on the usage of the (so far) implemented tools.

Evidence:
The paper focuses on the principles of the tools. There is only a single example which could be seen as a result (figure 4).

Open Science:
The code seems to be part of the IGSTK distribution, but it is not clearly described where to find it there. IGSTK contains the state-machine definitions used for the (few) examples presented in the paper, but not enough details (or a link to where to find this information) are provided to be able to replicate these examples or the other features described in the text (e.g., replaying of state-machine executions).

Reproducibility:
I downloaded IGSTK, but could not find the described toolset, except a few methods in some classes (e.g., ExportDescriptionToLTS) which seem related to the toolset.

Use of Open Source Software:
The toolset relies on another open-source project (“Apache Commons SCXML”), and on the “Labelled Transition System Analyzer” (LTSA, Java), which does not seem to be open-source (but is available for download).

Open Source Contributions:
IGSTK is open-source and easily available on a well-structured web-site. The described validation toolset seems to be part of the IGSTK distribution, but it is not clearly described where to find it there.

Code Quality:
IGSTK is generally well documented, coded in a modern style and portable. But I could only find very little of the code belonging to the validation toolset described in the paper.

Suggestions for future work:
Section 6 (“Future Work”) of the paper already describes the probably most important next steps.

Requests for additional information from authors:
Most importantly it would be desirable to have a link where to find further information how to find and apply the (already available parts of the) toolset. Some more results would be interesting. A reference where to find LTSA would be helpful (possibly http://www-dse.doc.ic.ac.uk/concurrency).

Additional Comments:
The aims and the approach seem to be very reasonable and I probably would have rated the paper much higher, if there had been more results and more information on how to replicate the results.