F-T4: Best Practices in Software Development
MICCAI 2010 - September 24, 2010
Best Practices in Software Development:
Users and Developers Unite!
Bennett Landman, Ph.D.
David N. Kennedy, Ph.D.
Xiaodong Tao, Ph.D.
Paul A. Yushkevich, Ph.D.
Gary Hui Zhang, Ph.D.
www.nitrc.org
nitrc.nih.gov
F-T4: Best Practices in Software Development
MICCAI 2010 - September 24, 2010
Schedule
13:30 - 15:00 Session 1: Presentations
13:30 - 13:40 Introduction to the Tutorial
13:40 - 14:00 Neuroimaging Informatics Tools and Resources Clearinghouse: NITRC
David N. Kennedy
14:00 - 14:20 3DSlicer3 Extension Modules: Concept and Implementation
Xiaodong Tao
14:20 - 14:40 Challenges of Collaborative Open-Source Development: Lessons learned
from the early evolution of the Java Image Science Toolkit
Bennett Landman
14:40 - 15:00 Making others use your software: Lessons learned from ITK-SNAP
Paul A. Yushkevich
15:00 - 15:30 Tea/Coffee Break
15:30 - 17:00 Session 2: Panel Discussion & Demonstrations
15:30 - 16:15 Panel Discussion
16:15 - 17:00 Demos
www.nitrc.org
nitrc.nih.gov
F-T4: Best Practices in Software Development
MICCAI 2010 - September 24, 2010
Academic Objectives
All participants should gain knowledge regarding:
• Methods for open source software development
• Methods for software distribution, including licensing
issues
• Techniques for software validation
• Techniques for user interactions (documentation,
tutorials, feedback/communication, etc.)
• Resources for finding existing relevant software
projects.
• See http://www.nitrc.org/projects/best_practices
www.nitrc.org
nitrc.nih.gov
F-T4: Best Practices in Software Development
MICCAI 2010 - September 24, 2010
Faculty
Bennett Landman, Ph.D.
Assistant Professor
Department of Electrical Engineering
Vanderbilt University Institute of Image Science
bennett.landman@vanderbilt.edu
David N. Kennedy, Ph.D.
Professor, Department of Psychiatry
University of Massachusetts Medical School
Worcester, MA 01605
David.Kennedy@umassmed.edu
Xiaodong Tao, Ph.D.
Computer Scientist
GE Global Research Center
Imaging Technologies
taox@research.ge.com
Gary Hui Zhang, Ph.D.
PICSL, Department of Radiology
University of Pennsylvania School of Medicine
3600 Market Street, Suite 320
Philadelphia, PA 19104
huiz@seas.upenn.edu
Paul A. Yushkevich, Ph.D.
Assistant Professor
Penn Image Computing and Science Laboratory
Department of Radiology
University of Pennsylvania
pauly2@mail.med.upenn.edu
www.nitrc.org
nitrc.nih.gov
F-T4: Best Practices in Software Development
MICCAI 2010 - September 24, 2010
Neuroimaging Informatics Tools and Resources Clearinghouse - NITRC
Software Tools - A Users Perspective
David N. Kennedy, Ph.D.
Director, Division of Neuroinformatics
Department of Psychiatry, UMass Medical School
Child and Adolescent NeuroDevelopment Initiative
www.nitrc.org
nitrc.nih.gov
F-T4: Best Practices in Software Development
MICCAI 2010 - September 24, 2010
With apologies
and credit to
Gary Larson The Far Side
www.nitrc.org
nitrc.nih.gov
F-T4: Best Practices in Software Development
MICCAI 2010 - September 24, 2010
users
With apologies
and credit to
Gary Larson The Far Side
developers
How do I change the
contrast? How do I add
layers? How do I invert the
transform?
blah blah blah
blah blah blah
RTFM blah blah
www.nitrc.org
nitrc.nih.gov
F-T4: Best Practices in Software Development
MICCAI 2010 - September 24, 2010
developers
v
With apologies
and credit to
Gary Larson The Far Side
users
Well, this is really a
very complex piece of code.
Have you read my paper from
MICCAI? It’s probably too
complex for you…
www.nitrc.org
nitrc.nih.gov
F-T4: Best Practices in Software Development
MICCAI 2010 - September 24, 2010
User Scope:
• Data Acquisition & Analysis Normal Development and Studies in Child Psychiatry (Autism, Bipolar Disorder, etc.)
– Structure
– Diffusion
– Resting State
www.nitrc.org
nitrc.nih.gov
F-T4: Best Practices in Software Development
MICCAI 2010 - September 24, 2010
Finding the Right Tool
“MRI segmentation software brain”
• Google?
– About 157,000 results
– Top Hit: Rescue
• Source Forge?
• NITRC?
www.nitrc.org
nitrc.nih.gov
F-T4: Best Practices in Software Development
MICCAI 2010 - September 24, 2010
Google
Top
Hit
www.nitrc.org
nitrc.nih.gov
F-T4: Best Practices in Software Development
MICCAI 2010 - September 24, 2010
Finding the Right Tool
“MRI segmentation software brain”
• Google?
– About 157,000 results
– Top Hit: Rescue
– Results include: papers, pdfs, ppts, programs,
pictures, videos, etc.
• Source Forge?
• NITRC?
www.nitrc.org
nitrc.nih.gov
F-T4: Best Practices in Software Development
MICCAI 2010 - September 24, 2010
Finding the Right Tool
“MRI segmentation software brain”
• Google?
– About 157,000 results
• Source Forge?
– 15,539 results
– Top Hit: PortableApps.com: Portable Software/USB
– Results include: you name it…
• NITRC?
www.nitrc.org
nitrc.nih.gov
F-T4: Best Practices in Software Development
MICCAI 2010 - September 24, 2010
Source
Forge
Results
www.nitrc.org
nitrc.nih.gov
F-T4: Best Practices in Software Development
MICCAI 2010 - September 24, 2010
Finding the Right Tool
“MRI segmentation software brain”
• Google?
– About 157,000 results
• Source Forge?
– 15,539 results
• NITRC?
– 39 Software Packages
– Top Hit: ITK-SNAP
– Results include: All the ‘usual suspect’ tools, and
more!
www.nitrc.org
nitrc.nih.gov
F-T4: Best Practices in Software Development
MICCAI 2010 - September 24, 2010
NITRC
Result
www.nitrc.org
nitrc.nih.gov
F-T4: Best Practices in Software Development
MICCAI 2010 - September 24, 2010
Refining the Search
• Operating System: MacOS
• Supported Image Type: DICOM
• Results in: 16 Resources
www.nitrc.org
nitrc.nih.gov
F-T4: Best Practices in Software Development
MICCAI 2010 - September 24, 2010
NITRC
Result
Refined
www.nitrc.org
nitrc.nih.gov
F-T4: Best Practices in Software Development
MICCAI 2010 - September 24, 2010
Best Practices Take-Home Points:
User Requirements
Developer Requirements
1. Findability
Description/Promotion
2.
3.
4.
5.
6.
7.
8.
www.nitrc.org
nitrc.nih.gov
F-T4: Best Practices in Software Development
MICCAI 2010 - September 24, 2010
Installation
• Findability is just first step. Can I install it,
easily?
• Examples:
– FSL
– Mango
www.nitrc.org
nitrc.nih.gov
F-T4: Best Practices in Software Development
MICCAI 2010 - September 24, 2010
FSL Installation Steps
1. Click download link
2. Agree to license
3. Provide some
personal information
4. Download
appropriate release
5. Run FSL installer
script (from
command line)
6. Test the installation
www.nitrc.org
nitrc.nih.gov
F-T4: Best Practices in Software Development
MICCAI 2010 - September 24, 2010
Mango Installation Steps
1. Click download link
2. Open the download w/ ‘Archive Utility’
3. Run Installer (Double click from UI) Answer usual questions
4. Run Program (Double click from
‘Applications’ directory)
www.nitrc.org
nitrc.nih.gov
F-T4: Best Practices in Software Development
MICCAI 2010 - September 24, 2010
Best Practices Take-Home Points:
User Requirements
Developer Requirements
1. Findability
Description/Promotion
2. Easy to Install
Easy to Install/Verify
3.
4.
5.
6.
7.
8.
www.nitrc.org
nitrc.nih.gov
F-T4: Best Practices in Software Development
MICCAI 2010 - September 24, 2010
Ease of Use
• Now what? You’ve found and installed
your program, how do you use it?
• From the UI, is it clear what to do?
www.nitrc.org
nitrc.nih.gov
F-T4: Best Practices in Software Development
MICCAI 2010 - September 24, 2010
User Interface
www.nitrc.org
nitrc.nih.gov
F-T4: Best Practices in Software Development
MICCAI 2010 - September 24, 2010
Documentation
• Of course the software is very powerful, and
it’s impossible for everything to be intuitively
obvious from the UI, who do you think we are,
Microsoft?
• Users are obligated to RTM (read the
manual) prior to bugging the developers with
questions.
• But, where is the manual? Is it easy to use?
Are there examples? Tutorials? FAQ?
www.nitrc.org
nitrc.nih.gov
F-T4: Best Practices in Software Development
MICCAI 2010 - September 24, 2010
FSL Documentation
www.nitrc.org
nitrc.nih.gov
F-T4: Best Practices in Software Development
MICCAI 2010 - September 24, 2010
Best Practices Take-Home Points:
User Requirements
Developer Requirements
1. Findability
Description/Promotion
2. Easy to Install
Easy to Install/Verify
3. Easy to Use
Intuitive UI/Good Documentation
4.
5.
6.
7.
8.
www.nitrc.org
nitrc.nih.gov
F-T4: Best Practices in Software Development
MICCAI 2010 - September 24, 2010
Help!
• OK, I REALLY did read the manual, and I just
can’t figure out how to [insert problem here]…
• I checked the FAQ, and searched on the mail list
archive…
• Oh, wait a minute, is there a FAQ? Is there a mail
list? Is there a users ‘forum’? Where are they?
• OK, I checked those too, how do I get more help?
– User Support Forums, Mailing Lists
– Email an administrator?
– What’s the preferred method for this project?
www.nitrc.org
nitrc.nih.gov
F-T4: Best Practices in Software Development
MICCAI 2010 - September 24, 2010
Best Practices Take-Home Points:
User Requirements
Developer Requirements
1. Findability
Description/Promotion
2. Easy to Install
Easy to Install/Verify
3. Easy to Use
Intuitive/Good Documentation
4. Get Help
Bug Tracker/Forums/Mailing List
5.
6.
7.
8.
www.nitrc.org
nitrc.nih.gov
F-T4: Best Practices in Software Development
MICCAI 2010 - September 24, 2010
Latest Version?
• Once you succeed at contacting the
developers, how often does the following
happen to you?
– Oh, you can’t do [blah]? ‘That’s been added in
the ‘most recent’ version’ Or worse yet, ‘that’s
only available in the development version’. Or,
‘that’s planned for release 3.X’.
• Users: Know your versions!
• Developers: Keep your public releases up
to date! (yes, ALL of them…)
www.nitrc.org
nitrc.nih.gov
F-T4: Best Practices in Software Development
MICCAI 2010 - September 24, 2010
Best Practices Take-Home Points:
User Requirements
Developer Requirements
1. Findability
Description/Promotion
2. Easy to Install
Easy to Install/Verify
3. Easy to Use
Intuitive/Good Documentation
4. Get Help
Bug Tracker/Forums/Mailing List
5. Up-to-Date
Versioning/Synchronization/Dashboard
6.
7.
8.
www.nitrc.org
nitrc.nih.gov
F-T4: Best Practices in Software Development
MICCAI 2010 - September 24, 2010
I want the best!
• Of course, I only want to use the best software know to
human kind for my research.
• Unfortunately, ‘best’ is exceedingly domain dependent, and
many problems (and software packages) cross many
domains in a specific problem (skull striping, registration,
pre-filtering, statistical test, etc.).
• Also, ‘best’ is data dependent. The ‘best’ surface extraction
in post mortem tissue may not be the best in patients w/ AD.
• Users: Know what best means in your context!
• Developers: Provide quality metrics (on standard cases)
when possible!
www.nitrc.org
nitrc.nih.gov
F-T4: Best Practices in Software Development
MICCAI 2010 - September 24, 2010
Best Practices Take-Home Points:
User Requirements
Developer Requirements
1. Findability
Description/Promotion
2. Easy to Install
Easy to Install/Verify
3. Easy to Use
Intuitive/Good Documentation
4. Get Help
Bug Tracker/Forums/Mailing List
5. Up-to-Date
Versioning/Synchronization/Dashboard
6. High Quality
Validation/Comparison Metrics
7.
8.
www.nitrc.org
nitrc.nih.gov
F-T4: Best Practices in Software Development
MICCAI 2010 - September 24, 2010
Say I’m not the only one…
• Even if using the ‘best’ tool, the user will run into
publications problems if no one else has ever used
that tool before.
• Have you gotten reviewers who say: “You should
have used the Talairach coordinate system”, or
“You should have used SPM…”?
• Even if it’s not fair or optimal, users need to know
how popular a resource is. # of downloads  # of
users  # of publications, etc.
www.nitrc.org
nitrc.nih.gov
F-T4: Best Practices in Software Development
MICCAI 2010 - September 24, 2010
Best Practices Take-Home Points:
User Requirements
Developer Requirements
1. Findability
Description/Promotion
2. Easy to Install
Easy to Install/Verify
3. Easy to Use
Intuitive/Good Documentation
4. Get Help
Bug Tracker/Forums/Mailing List
5. Up-to-Date
Versioning/Synchronization/Dashboard
6. High Quality
Validation/Comparison Metrics
7. Use Metrics
Downloads/Users/Publication Counts
8.
www.nitrc.org
nitrc.nih.gov
F-T4: Best Practices in Software Development
MICCAI 2010 - September 24, 2010
I can make it better…
• I’ve found a great tool, it works really well for me,
and does everything I need… almost.
• I submitted a ‘Feature Request’, and I completely
understand that your funding and time is limited,
and you will not get to my request until 2021.
• But hey, I’m a wicked good [insert programming
language here] programmer myself. Let me at your
source code…
• Open Source is not the solution for everything, but
with proper management and policy, it’s an
important tool for extensibility, transparency, and
public relations. Besides, it ought to be the law…
www.nitrc.org
nitrc.nih.gov
F-T4: Best Practices in Software Development
MICCAI 2010 - September 24, 2010
Best Practices Take-Home Points:
User Requirements
Developer Requirements
1. Findability
Description/Promotion
2. Easy to Install
Easy to Install/Verify
3. Easy to Use
Intuitive/Good Documentation
4. Get Help
Bug Tracker/Forums/Mailing List
5. Up-to-Date
Versioning/Synchronization/Dashboard
6. High Quality
Validation/Comparison Metrics
7. Use Metrics
Downloads/Users/Publication Counts
8. Contribute/Extend
Source Code Availability/Policy
www.nitrc.org
nitrc.nih.gov
F-T4: Best Practices in Software Development
MICCAI 2010 - September 24, 2010
Summary
• Top 10 reasons users use a specific software
package:
6. I wrote it… when I was a graduate student… 20 years
ago…
7. Is part of the LONI pipeline…
8. Is easiest to use…
9. Has the most publications…
10. It’s what my lab uses…
www.nitrc.org
nitrc.nih.gov
F-T4: Best Practices in Software Development
MICCAI 2010 - September 24, 2010
Summary
• Top 10 reasons users use a specific software
package:
1. At least it’s not SPM…
2. Everyone uses SPM…
3. It’s written in ITK…
4. It works with 3D Slicer…
5. My graduate student wrote it… who is a botany major…
with no formal programming training…
www.nitrc.org
nitrc.nih.gov
F-T4: Best Practices in Software Development
MICCAI 2010 - September 24, 2010
Summary, continued
• Top 1 reason NOT listed as a reason to use
a specific software package:
1. It is the best…
This is where developers and users have to continue to work together to:
• Insist on the best
• Define the parameters of best
• Develop standards for performance
• Document the performance
• And use the guidelines of Best Practices for software development!
www.nitrc.org
nitrc.nih.gov
F-T4: Best Practices in Software Development
MICCAI 2010 - September 24, 2010
Acknowledgements
•
•
•
•
All NITRC Users and Developers
Today’s Presenters & Audience
MICCAI Organizers
NITRC Development Team
(Neuromorphometrics Inc, TCG Inc, UCSD)
• NITRC Sponsors (NIH Blueprint, NIBIB)
• The ‘tutorial’ lives on at
http://www.nitrc.org/projects/best_practices
www.nitrc.org
nitrc.nih.gov
F-T4: Best Practices in Software Development
MICCAI 2010 - September 24, 2010
Best Practices Take-Home Points:
User Requirements
Developer Requirements
1. Findability
Description/Promotion
2. Easy to Install
Easy to Install/Verify
3. Easy to Use
Intuitive/Good Documentation
4. Get Help
Bug Tracker/Forums/Mailing List
5. Up-to-Date
Versioning/Synchronization/Dashboard
6. High Quality
Validation/Comparison Metrics
7. Use Metrics
Downloads/Users/Publication Counts
8. Contribute/Extend
Source Code Availability/Policy
www.nitrc.org
nitrc.nih.gov