ATML Pad Beta – Beta Evaluation
Form
October 2013
Thank you for evaluating the ATML Pad Beta. Please type your answers inline in the document
below.
1. User Interface
1. Is the general layout of the user interface intuitive? If not, what would you improve?
2. Is the nomenclature intuitive and easy to understand? If not, what would you change?
3. Is user interaction intuitive? Please rate the usability of the following interactivity
features:
a. Menus
b. Toolbars
c. Keyboard navigation
4. Would you prefer a ribbon instead of the current toolbar?
5. Is information feedback easy to use and understand? Please rate the following features:
a. Descriptions provided in tooltips
b. Descriptions provided in user selection forms (ex. when a validation error is
encountered)
c. Visual feedback for online validation
d. Display of errors from on-demand validation
6. Did you evaluate the editing of scalar data types? If yes, did you find it intuitive? Was
on-line data validation useful?
7. Did you evaluate the editing of array data types? If yes, did you find it intuitive? Is the
current input mechanism (as separated strings) acceptable in the short term? If not, what
type of visual array editor would you like to see added to ATML Pad?
8. Did you evaluate on-demand validation? If yes, were you able to understand the error
information displayed by ATML Pad? Were you able to use the automatic navigation to
the location of the error? What would you improve?
9. Did you evaluate the XML Editor of ATML Pad? If yes, did you find it intuitive?
10. Do you have other suggestions for improving the user experience?
Copyright © Reston Software, LLC. All rights reserved.
1
2. Validation
1. Did you evaluate the various validation features of ATML Pad (on-line, on demand,
XML schema validation)? Did you find the distinction between these features easy to
understand? If not, how would you improve the presentation of these features to the use
(user interface, documentation, etc.)?
2. Were you able to use XML validation in the XML Editor? Did you find it sufficient?
3. ATML Document Import
1. Do you plan to import in ATML Pad ATML documents created manually, or by other
“producer” applications? If not, skip this section.
2. Did you experiment with the import of such ATML documents? If yes, did you
encounter errors during import?
3. Are there ATML features that the imported documents support and ATML does not? If
yes, how important are those for your application?
4. Microsoft Word Import
1. Do you plan to import in ATML Pad ATML Test Description data from Microsoft Office
files or other external data files? If not, skip this section.
2. Did you evaluate the “proof of concept” Word Import plugin? If yes, do you find it
potentially useful? Which features would you like to see added?
3. Would you use import from other document formats (ex. wire lists from Excel), if
supported in ATML Pad? What type of information and which file formats would be
useful to support?
4. Do you have a need to create custom import plugins for ATML Pad?
5. TestStand Export
1. Do you currently use or plan to use the TD Translator from the “NI TestStand ATML
Toolkit”. If yes, do you use custom code generators?
2. Do you plan to export ATML Test Description data directly to NI TestStand? If not, skip
this section.
3. Did you evaluate the TestStand Export plugin? If yes, do you find it useful? Did you
encounter any errors?
4. Did you experiment with the post-processing functionality implemented in the plugin?
If yes, do you find it potentially useful? Which features would you like to see added?
Would you rather have post-processing implemented in a custom code generator?
5. Do you have a need to create custom export plugins for ATML Pad?
Copyright © Reston Software, LLC. All rights reserved.
2
6. ATML Feature Set
Note: The information requested in this section will help us focus new development, addressing
features that are most important to users. Your feedback in this area is greatly appreciated.
6.1.
Test Group Types
1. Did you evaluate the “Sequence” and “Serial” test groups? Which of these test group
types are you planning to use?
2. Did you evaluate Sequence Post Actions? If yes, did you find their use intuitive? If not,
how would you improve usability?
3. Are you planning to use the test group types “Parallel”, “Diagnostic Model” or
“Unspecified Order” (currently not supported by ATML Pad)? If yes, how important are
these types for your applications (must have, generally useful, nice to have)?
6.2.
Data Types
1. Currently ATML Pad supports array data types with some limitations (ex. maximum
two dimensions, only rectangular and jagged arrays, etc.). Did you find the current level
of support sufficient? If not, which new array capabilities would you like to see
supported?
2. Currently ATML Pad does not support Collection data types. Is this feature important
for your application?
6.3.
IEEE 1641 XML
1. Do use IEEE 1641 XML to define signals and test behavior (with different applications)?
Which applications are you currently using or plan to use for producing / consuming
IEEE 1641 XML?
2. Are you planning to use IEEE 1641 XML for describing test behavior in ATML Test
Description documents? If not, skip this section.
3. Did you evaluate the use of IEEE 1641 XML in ATML Pad? If yes, is the current level of
support (direct XML editing) adequate for your use?
4. Would you use a graphical editor for IEEE 1641 XML content, if one would be available
in ATML Pad?
6.4.
Operations
1. Are you using Operations for describing test behavior in ATML Test Description
documents? If not, skip this section.
2. Did you evaluate Operations in ATML Pad? If yes, is the current level of support (partial
visual editing, partial direct XML editing) adequate for your use?
Copyright © Reston Software, LLC. All rights reserved.
3
3. Currently ATML Pad requires direct XML editing for several Operation types. For
which of these would you like to see visual editing support in ATML Pad? How
important are these for your applications?
4. Currently ATML Pad requires direct XML editing for signal definitions in Setup
operations. Would you use a visual editor /wizard for signal definitions, if one would be
available in ATML Pad?
5. Currently ATML Pad requires direct XML editing for connection definitions in Connect
operations. Would you use a visual editor / wizard for connection definitions, if one
would be available in ATML Pad?
6. Are you planning to use the synchronization features of Operations (currently not
supported by ATML Pad)? If yes, how important are these features for your
applications?
7. Are you planning to use custom (user-defined) Operation types? If yes, would you
create plug-in visual editors for these Operations, if ATML Pad would provide a plug-in
mechanism?
6.5.
Extensions
1. Do you use the extensibility features of ATML? If yes, which extension mechanisms do
you use (“Extension” elements, type derivation)?
2. Are you planning to use extensions in ATML Test Description documents that can be
edited with ATML Pad? If not, skip this section.
3. Which type of extensions are you planning to use with ATML Pad?
a. “Extension” elements
b. Custom data types
c. Custom operations
d. TSF Libraries
e. Other
4. Did you evaluate the Extension Schemas feature of ATML Pad? If yes, did you
encounter problems? Do you have any suggestions for improvement?
5. Did you evaluate the TSF Libraries feature of ATML Pad? If yes, did you encounter
problems? Do you have any suggestions for improvement?
6.6.
Other ATML Features
1. Currently ATML Pad requires direct XML editing for several children of the root
element (Interface Requirements, Signal Requirements, Performance Characteristics,
Failure and Fault Data, Automatic Generation). For which of these would you like to see
visual editing support in ATML Pad? How important are these for your applications?
Copyright © Reston Software, LLC. All rights reserved.
4
2. Please list other ATML Test Description features that you would like to see supported in
ATML Pad. For each feature, identify the level of criticality for your application (must
have, generally useful, nice to have).
3. Please list other ATML schemas that you would like to see supported in ATML Pad (ex.
UUT Description, Interface Adapter, Instrument Description, etc.).
7. Documentation
1. Did you use the electronic documentation provided with ATML Pad? If yes, did you
find it useful? How would you improve it?
2. Did you find it sufficient? If not, what type of information is missing?
3. Does ATML Pad need a “traditional” Help file that describes all the elements of the user
interface and all user commands?
8. Operating Environment
Note: The information requested in this section will help us target the future ATML Pad
releases.
1. Which Operating Systems are you using, or will be using in the near future?
2. What type of Windows account are you using (ex. Administrator, Standard, etc.)?
3. Does your organization have special requirements for allowing applications to be
installed? Is Microsoft Windows 7 Certification (a.k.a. Windows 7 Client Software Logo
Program) a plus?
4. What type of licensing activation do you prefer (please grade all options):
a. Online validation (requires an Internet connection from the computer running
ATML Pad)
b. License key downloaded from the Internet on a different computer
c. License file provided by email
d. Hardware key (dongle)
e. Floating user licenses (requires licensing server)
9. Bugs
Please describe below any bugs you discovered while evaluating ATML Pad (unless already
reported online or via email).
10.
Other Feedback
Please provide below any feedback on topics not addressed in this document.
Copyright © Reston Software, LLC. All rights reserved.
5