Summary:

The paper describes an open source software framework for building a multi-modality image guided therapy application.

Hypothesis:
The designed software framework is useful for building image guided therapy applications.

Evidence:
Five applications developed using SIGN have been presented. However, source code was not provided for these applications.

Open Science:
The software framework design and distribution scheme follows open science principles. The software framework uses heavily other open source toolkits such as OpenTracker, GIHI, VTK, ITK, KWWidgets, Xerces and Aceto. The source code for these toolkits are readily availbale. And, the source code for SIGN is scheduled to be released by Oct.19.

Reproducibility:
N/A

Use of Open Source Software:
The developers of SIGN have used open source toollkits such as OpenTracker, GIHI, VTK, ITK.

Open Source Contributions:
Authors stated that the source code will be released by Oct 19th.

Code Quality:
N/A
Applicability to other problems:
SIGN is a useful software framework is useful to develop other image guide therapy applications.

Suggestions for future work:

Requests for additional information from authors:
Safe software design principles and practicses employed in the SIGN software framework have not been discussed in the paper.

Additional Comments:
[This is a free-form field]

Summary:
This paper describes a framework for developing various IGT applications.

Hypothesis:
Non Applicable

Evidence:
The author claimed that, “…with the purpose of providing a safe, dynamic and extensible software framework for image-guided therapy in a novel operation facility called AMIGO”, however he didn’t provide further evidence on how this framework achieved these design goals.

Open Science:
The seamless integration with Slicer gives this toolkit a good support for developing variety of applications for surgical guidance and therapy. Its intention to go open source will surely benefit the research community

Reproducibility:
There is no source code published with this paper at this moment.

Use of Open Source Software:
This work has taken full advantage of existing open source software toolkits, such as Slicer, OpenTracker, GIHI, and ACE etc.

Open Source Contributions:
The source code will be release under open source license.
Will the example applications also be available for the public?

Code Quality:
Not applicable

Applicability to other problems:
Some interesting clinical applications have been shown in this paper developed using the framework. It shows that this framework is quite flexible and has a wide range of clinical applications.

Suggestions for future work:
Have you considered the synchronization issue between hardware devices and application? Difference hardware devices in the surgical guidance system have different response frequency. There is also latency issue when communicating through the network. When integrating the information from difference sources together in the surgical scene, you need to be very careful about the accuracy of the information you are presenting to the surgeon.

Requests for additional information from authors:
None.

Additional Comments:
None.

Summary:
This paper describes the Slicer Image Guided Navigator (SIGN) software being developed at BWH. It adds support for imaging and tracking systems to 3D Slicer (it seems), using the GIHI and OpenTracker open source libraries, respectively.

Hypothesis:
N/A

Evidence:
N/A

Open Science:
The paper promises to adhere to the principles of open science, although the primary source code will not be available until October 19.

Reproducibility:
The SIGN source code is not available for download. I downloaded the GIHI source code (SVN repository) but did not compile it. I tried to find OpenTracker V1.3 but it is not available from the specified web site, which lists V1.1.1 as the latest version. There is a note to contact the project administrator to get a developers version from SVN. I did not do this. Some of the documentation refers to OpenTracker V1.2 and later, so clearly the repository must contain more recent versions.

Use of Open Source Software:
The SIGN projects uses a lot of open source software, including VTK, KWWIDGETS, ITK, OpenTracker, GIHI, XERCES, and ACE. It also uses the SPLOT package (extensions to OpenTracker), which I assume will be available as open source software. The relationship to 3D Slicer is not entirely clear. The paper states that the SIGN is designed to be “highly interoperable” with 3D Slicer. Can the SIGN run without Slicer? If so, it seems it would duplicate some of the functionality of Slicer. If not, Slicer should be added to the list of open source software.

Open Source Contributions:
The paper states that The SIGN will be released as open source on Oct 19. It also says that information will be given at www.ncigt.org. I only found a brief reference to the SIGN under the “3D Slicer Engineering” project.

Code Quality:
I looked at the GIHI source code, which is written in C for portability to older platforms. It is supported on Linux and Solaris at this time. The code quality seems fine, though one cannot call it "modern". Note, however, that The SIGN source code, which is the primary subject of this paper, may not have any similarities to GIHI.

Applicability to other problems:
The SIGN is an application framework and is therefore applicable to a large number of problems. 3D Slicer has been successfully applied to a number of applications, and it is reasonable to expect the SIGN (which seems to add intraoperative imaging and tracking support to 3D Slicer) to also be widely applicable.

Suggestions for future work:
Obviously, finish the software and make it available as open source! Also, improve the operating system support (e.g., some components, such as GIHI, are limited at this time -- just Linux and Solaris).

Requests for additional information from authors:
This paper presents a clear overview, but is short on technical details. Section 2.3 is especially brief and refers to (11) for comprehensive details. There actually is no reference 11 (maybe this should be 10?). I would have liked to see some explanation of the dynamic data structures.

Some block diagrams, flowcharts, etc. describing the software would be a welcome addition.


Additional Comments:

Following are some minor typos:

• Section 2.3 (noted above) and 2.4 refer to (11), but there is no reference with this number.


• The Medscan description says “in order to encoding all services”, which should be “in order to encode all services”.


• The section numbering seems to be off. I believe Sections 3.1, 4, 4.1, 4.2, 4.3 should have been 3.1, 3.2, 3.3, 3.4, 3.5.


• In Section 4.1 (Cardiac navigation), “course orientation” should be “coarse orientation”.


• In Conclusions, “My” should be “By”.

Summary:
The paper demonstrated a new software framework (named “Slicer Image-Guided Navigator” or SIGN) for Image-Guided Therapy that aims to provide real-time intraoperative surgical navigation with dynamic visualization and interactions with multiple imaging and tracking devices, through a seamless integration of dynamic data structures (medical reality modeling language, or MRML). The software package was developed based on various open-source frameworks and will itself be open-source soon to be available to the public.

Hypothesis:
The authors argue that newly emerging image-guided procedures and therapies have seen increasing use of multiple imaging modalities and tracking devices at the same time in the OR. I agree that new surgical navigation applications are therefore desired to seamlessly interact with different imaging and tracking systems through a virtually transparent interface (to the users) with significant portability and expendability. In addition, more and stricter regulatory requirements pose new challenges on academic software development for research tools in clinical treatments.

Evidence:
Several clinical-relevant applications were presented using the current development of SIGN. The software framework was itself very well established and clearly presented through detailed designs of various system components that handle different hardware and software interfacings (i.e. GIHI for the imager interfacing, OpenTracker for the tracker interfacing, and Hybrid for event and data handling). References to the open-source resources used were also adequately provided.

Open Science:
I think the proposed software framework is clearly designed with open-source development and distribution in mind. According to the paper, SIGN was developed primarily based on the open-source 3D Slicer framework and has extensively employed many other open-source frameworks (e.g., OpenTracker, GIHI, VTK, ITK, etc.) to develop its major system components (e.g., MedScan). The authors have also promised the release of SIGN’s original source codes on October 19th, 2006 to the open-source community. So it is evident to me that the proposed software framework will eventually benefit the public by contributing to the open-source community.

Reproducibility:
Since the source codes of SIGN will not be made available to general public until October 19th, 2006, I haven’t been able to reproduce the authors work as described in this paper. However, I was able to download and install the OpenTracker package from the link provided by the paper. I should point out that there is one small mistake in the paper regarding the latest version of OpenTracker (the paper mentioned that the latest version is v1.3, while on OpenTracker’s website, it states that v1.1.1 is its latest release). For GIHI, I kept getting a “connection timeout” to the link suggested by the paper, so I have not been able to test it during the writing of this review.

Use of Open Source Software:
An extensive use of the existing open-source software is evident in the development of SIGN. The framework was developed primarily based on the open-source 3D Slicer framework and has employed many other open-source frameworks (including OpenTracker, GIHI, VTK, ITK, KWWidgets, Xerces and Aceto) to develop its major system components.

Open Source Contributions:
The authors have promised the release of SIGN’s original source codes on October 19th, 2006 to the open-source community. No test is possible until then.

Code Quality:
The authors have promised the release of SIGN’s original source codes on October 19th, 2006 to the open-source community. No examination of the code quality is possible until then.

Applicability to other problems:
My understanding of the major strength of SIGN is in its ability to interface multiple imaging and tracking devices with seamless dynamic data integration and visualization. This framework could be extended to various existing image-guided surgical navigation systems that employ a broad range of imaging (e.g., CT, MRI, and US) and tracking (e.g., optical or magnetic) systems. In addition, the dynamic data encapsulation of SIGN (using MRML) may be applicable to many data management applications (not limited to medical applications) that wish to easily share, convert, and transfer data between different formats.

Suggestions for future work:
Was there a high-level design approach employed to outline the interrelationship between all the interfacing components in SIGN? Event handling for hybrid devices (devices of all kinds) in a generic interface is very attractive to many software engineering applications, but in general remains extremely complicated (essentially each device has its own protocol for communications). So in the future design of SIGN (and its extensions), a high-level design approach (e.g., Rational Rose or generic UML) is very helpful here to sort out all relationships between multiple system components at different interacting levels.

Requests for additional information from authors:
Please see the following additional comments.

Additional Comments:
1. In imaging hardware (Section 1.1), ultrasound (US) was not mentioned here as newly emerging imaging modalities. However, later on in the following sections, US was indicated apparently as one of the imagers in SIGN.

2. In tracking devices (Section 1.2), the authors illustrate many types of tracking systems including their advantages and disadvantages. However, no single reference was given here. More references are necessary here.

3. AMIGO (Section 1.4) was described as the context of the proposed framework (the ultimate application), however, there was no a word or two later in the paper briefing how, in a larger picture, SIGN is (or will be) adopted in AMIGO. Please provide a brief explanation of this link somewhere after SIGN is introduced.

4. GIHI was written in C. Shouldn't it consider using C++? Or at least a combination of C (for portability) and C++ (expandability)? Seems to me that GIHI requires many service/component extensions, either current or in the future. C is indeed portable and efficient but fairly poor at expandability, especially for a large system like SIGN that has multiple software components.

5. Was there a specific design software used to design the overall structure of SIGN and its interactions with other system components (like OpenTracker and GIHI)? SIGN seems to be a very large, complicated, and distributed (multiple components acting in an asynchronous fashion) system employing a great amount of events and their handlers. It seems (at least to me) virtually impossible to design such a dynamic system efficiently and completely without the assistance of some professional design approaches (e.g, Rational Rose).

6. In hybrid interfacing (Section 2.3), from a system-design point of view, why Terason is designed as a separate module that is parallel to MedScan? At a high level in the overall system, shouldn't Terason be part of the GIHI library whose primary duty is to handle the interfacing with various imaging devices (which include US)? Please clarify.

7. In the illustration of SIGN (Section 2.4 the last paragraph), I was a bit confused here: the tracker is presumably a monitoring device that traces the motion of the objects in real time. It is supposed to be a passive observer. So what does it mean that once connected to the data object in MRML tree, the tracker could move an object around the scene? Or maybe it was my incorrect interpretation? Please clarify a bit here.

8. In the introduction of SIGN (Section 2.4), the authors claimed that “The framework was designed to be highly interoperable with the 3D Slicer …”. However, the exact relationship between SIGN and 3D Slicer still remains unclear to me even after reading through the entire paper. Was SIGN designed and developed primarily based on the 3D Slicer framework? Or they simply just share the same data structure protocol (using MRML)? Please clarify the term “interoperable” in more details here.

Summary:
The abundance of imaging devices, tracking devices and other electronic devices which image guided therapy relies on has resulted in many efforts to develop sofware frameworks which abstracts the inherent complexity of integrating such devices in image guided therapy software applications. Important requirements of such frameworks are portability, usability, synchronization, flexibility and speed. The authors describe a new framework which aims at supporting all of these requirements. Three main software components constitute the software framework. OpenTracker is a general data streaming components which can interface to such datasources as tracking devices, ECG devices and imaging sources. Another component handles interfacing to imaging hardware, GIHI, which can be used to extract tracking data from imaging devices where that is relevant, extract images in addition to provide a interface for controlling imaging parameters. The last component is VTK which is used for visualization purposes. XML is used to configure the framework and can be set up such that data can flow between the components in an almost seamless manner.

Hypothesis:
Not applicable.

Evidence:
A number of clinical examples are presented in the paper. However, the only example which is backed up by a reference to show its clinical relevance is the endoscopic navigation.
I don\'t really see how ITK is integrated in this framework.

Open Science:
Most of the source code is currently available such as OpenTracker and GIHI. The rest of the source code is set to be released in October. However, most of the source code which is due to be released are the parts which provides the interfaces between the various components. However, I didn\'t quite understand whether the source code for the examples also will be released?

Reproducibility:
Not applicable since the source code is not publicly available.

Use of Open Source Software:
I don\'t know. There is no section which discusses the licenses of the software components.

Open Source Contributions:

Code Quality:

Applicability to other problems:
The framework should be applicable to many image guided applications due to its flexible nature.

Suggestions for future work:
I would definitely like to see how ITK is integrated in the framework. A seamless integration of ITK, or other image processing libraries, into image guided therapy frameworks is something I still haven\'t witnessed.

The paper doesn\'t discuss the advantages/disadvantages of The SIGN compared to other image guided therapy frameworks such as IGSTK, Julius etc.

Requests for additional information from authors:

Additional Comments:
I\'m looking to following the further development of the framework as it has many good things going for it. In particular, the interface to the popular slicer, VTK and ITK in addition to the novel hybrid data flow component.

Summary:
Authors presents a software framework whose purpose is to develop specific image-guided applications using multiple intra-operative imaging. The framework is based upon an extended open-source interfacing of the open-source OPenTracker library and an imager hardware interface GIHI, using 4 separate new modules. The framework was implemented to support multiple system interactions and visualization tasks, with a dynamic execution model.

Hypothesis:
There is a need in image-guided therapy for software framework satisfying the following requirements: using multiple imaging instruments with proper functionality, portability, abstraction and speed. The fact of tracking multiple tools or getting multiple images at the same time in real-time is of course of great interest for image-guided surgery, in particular for augmented reality or augmented virtuality.

Evidence:
The demonstration of the interest was based on the presentation of 5 clinical applications. However, the code is not yet available, it will be on the 19th of October. I was then not able to reproduce the presented results. The need of a framework allowing aplication as separate silos was justified by the regulatory requirements.

Open Science:
The presented framework is is the spirit of open science, for 2 reasons: 1) it is based upon existing open-source libraries, and extended or interfaced them 2) The source code of the programs used in their experiments was not provided yet but it will be open source. It consists on an add-on to an existing open source library Open tracker and to a global framework library, the SIGN, interfacing other opens ource librairies.

Reproducibility:
Not available since no code was not provided

Use of Open Source Software:
The framework The SIGN is based upon (instead of duplicating) existing open-source efforts, as OpenTracker, VTK, KWWidgets, ITK, SPLOT , GIHI, Xerces and Ace.

Open Source Contributions:
No code available. I was not able to compile the GIHI part, and the openTracker v1.3 is not available directly on the indicated link.

Code Quality:
The developments was made in C, said to respect portability rules. But the code is not available. Nothing said about dart test or others.

Applicability to other problems:
The authors have shown the applicability to a large range of problem, by showin 5 different applications. The replay posibilities could be really interesting for pedagogical purposes, or to make some e-books for surgeon.

Suggestions for future work:
It will be of interest in the future first release to have the demonstrations application code, and an image simulator, or at least the replay. Moreover you could include to your software an aplication validation strategy, which could lead to both performance evaluation and validation of each developped application using the software. Most of time when developping an application using intraoperative imaging, you have not a wide access to the intraoperative imaging. Including simulators of different kind (for tools, images, etc...) is really important for rapid prototyping.

Requests for additional information from authors:
The relation with the AMIGO operating room was not clear, since it was no more mentionned after its presentation. Figures are of interest for demonstrations of the workflow interest, I would like to have them bigger and with a detailed description. What is the black window in figure 4? It would be of interest to underline the differences with other available Image-guided surgery toolkits, as IGSTK. What is reference (11)? Reference (9) and (10) are not cited in the text.
Could you add a reference for MRML?

Additional Comments:
The note was given following the reviewer's guideline: detail of note: 1/2 Code not available but built upon open-source libraries+1/1 adress a real software-based need within the community+ 0.5/1 No material provided but concepts are well described + 1/1 all research projects in image-guided surgery would benefit of this work . Total: 3.5/5=> 4