STATE OF MAINE
DEPARTMENT OF PROFESSIONAL AND FINANCIAL REGULATION
OFFICE OF SECURITIES
121 STATE HOUSE STATION
AUGUSTA, ME
JUDITH M. SHAW
SECURITIES ADMINISTRATOR
TO:
NASAA Board of Directors
Matt Kitzi, Chair, Enforcement Section
FROM:
Judith Shaw, Chair
Standardized Training and Technology Committee
RE:
Evaluation of January, 2012 Enforcement Training
DATE:
April 11, 2012
The purpose of this memorandum is to provide the Board and the Chair of the
Enforcement Section with feedback regarding the 2012 Enforcement Training held in
New Orleans this past January.
One of the responsibilities of the Standardized Training and Technology Committee is to
attend and evaluate each NASAA training in order to provide feedback to the Section
Chair and the Board. The Enforcement Training is the first NASAA training evaluated
by the STT Committee.
In furtherance of the consistency and standardization goals of the Committee, this
training will be measured against other trainings in order to develop best practices that
can be used by all NASAA Sections. A final report on best practices will be provided to
the Board and all Section Chairs at the end of the year once all NASAA trainings have
been evaluated.
I attended the Enforcement Training and evaluated most, but not all, sessions. Given the
extensive training and the fact there were three tracks, I was unable to attend every
session. However, I attended a portion of most of the sessions and observed enough to be
able to provide some general observations.
Scope of Evaluation
The STT Committee developed evaluation forms to be used in assessing NASAA
trainings. The forms cover review of the following: 1) facility and location; 2) general
overall format; 3) individual speakers; and 4) individual moderators.
Facility and location
In evaluating the facility and location, the STT Committee considers the comfort of the
meeting space and guest room facilities, acoustics, audio-visual, food and beverage,
program location, and the ease of travel. Unless the Board feels differently, in the future
the Committee can provide feedback on facility and location directly to Lonnie Martin
rather than to the Board and Section Chair.
Meeting space and guest room facilities: The meeting space and guest rooms were
comfortable and appropriate for training purposes. Several attendees noted that the
meeting rooms tended to be hot. At times there was significant background noise largely
due to an open concept lounge in the hotel and the fact the training was taking place
during the NFL playoffs. However, Lonnie Martin quickly recognized and addressed
these concerns.
Acoustics and audio-visual: Generally, the audio was inconsistent from speaker to
speaker. Maintaining an appropriate level can be difficult since some speakers project
their voices more than others. The training sessions were being videotaped in order to
post the training on the NASAA Distance Education website. The placement and
intensity of the lighting needed for the videotaping resulted in some of the speakers being
blinded. Additionally, I was told by some speakers that they would have preferred to
have been told in advance that the session was to be videotaped.
Food and beverage: All meals, beverages, and snack items were enjoyable and provided
a good variety of choices.
Location and ease of travel: The program location was a relatively short ride from the
airport. The shuttle from the airport was comfortable and easily accessed.
General Overall Format
In evaluating the overall format for the program, the STT Committee considers the length
of the program, the pace of the program, the scope of the program, and the balance of the
format. In addition, the Committee evaluates whether the topics are current and timely,
whether the learning objections are clear and appropriate, and whether the learning
objectives were met.
The program ran January 22nd and 23rd. The length seemed appropriate for the training as
did the pace.
a. Enforcement Zones - Some participants felt holding the enforcement zone
meetings simultaneously with some of the training sessions created a dilemma
particularly for attorneys who were seeking Continuing Legal Education credits.
Although the Zone meetings were intended for Track Three participants, a
number of participants in the other two tracks were in attendance. The
importance of collaboration and cooperation among the states cannot be
2
overemphasized and one way to accomplish it is to hold robust Enforcement Zone
meetings. The Committee recommends that in the future Zone meetings not be
scheduled at a time that overlaps with training components of the event.
b. Tracks - The format seemed well balanced in terms of presentation versus case
study. Topics were current and timely especially in Track Three. Feedback from
participants was favorable. One observation is that participants did not seem clear
as to which track was most appropriate for them. This resulted in a significant
number of participants either changing tracks part way through the training or
bouncing from track to track depending upon the topic being covered.
I did observe that a considerable number of participants seemed to view the
program as more of a conference in that they attended some sessions in their
tracks and not others; jumped from track to track; left the program before the end
of the day; and conducted personal work during presentations. It is unclear what,
if anything, NASAA can do to improve on this since the goals and objectives for
individual participants and their respective administrators may vary.
c. Learning objectives - The learning objectives for the program could be clearer. In
speaking with various participants there was a desire for a more distinct
expression of learning objectives and expectations. For instance, one participant
noted it would be useful to be told at the beginning of the case study what one
could expect to gain in terms of tools and skills that could be put to use upon
returning to their office. The learning objectives were more evident with some
presentations than others.
d. Case studies - Multiple participants expressed a desire to have the program
presented as more of an immersion in the case study from the first day. That is to
say, the participants hoped to start the training with an introduction of the case
study, the establishment of teams, and assignments for the teams to work on at the
end of the day. This would instill in the participants the idea that the program is
an educational opportunity as opposed to a conference.
Case studies can be an excellent training tool particularly when they provide the
attendee an opportunity to practice and work with new skills. If case studies are
to be used, they should be used not as a way to impart substantive information
but, rather, to apply knowledge and develop new investigative and enforcement
tools that will readily translate to everyday experiences. It is recommended that
any use of case studies be done in a way that attendees actually engage in
application of skills and tools such as actually developing an initial investigative
plan, writing a case summary report, drafting interview questions, reviewing and
scheduling records, and the like.
The Committee believes that best practices regarding the use of case studies and multiple
tracks will become evident as we evaluate trainings for the other sections.
3
Speakers
In evaluating the speakers, the STT Committee considers the knowledge of the subject
matter, speaking style, openness to questions, ability to handle questions, quality of
handouts, quality of any audio-visual presentation, ability to make the materials
comprehensible, and level of preparation.
Generally, the speakers were knowledgeable and able to convey the material in a way
that was engaging. Most presentations were understandable and tied back to the
discussion. Speakers were well prepared.
It should come as no surprise that the most effective speakers were the educators and
consultants who make presentations for a living. These individuals were engaging
interjecting humor when appropriate. They also projected their voices and changed their
tone which helps to keep attendees interested. Experienced speakers who were a part of a
panel knew to involve other members of the panel in the dialogue. They also engaged the
audience by asking questions of the audience and walking among attendees rather than
sitting behind a table. These speakers were more animated and held the attention of the
crowd.
To the extent that speakers used PowerPoint presentations, the quality varied. This again
is likely to be reflective of the experience of the speaker. Some PowerPoint presentations
were flat in terms of color contrast as well as content. Some presentations were mostly
multiple slides of long text in which case the information became overwhelming
particularly for Track One participants. The most effective presentations were those that
contained slides with pictures as well as text and that were somewhat interactive and
dynamic.
As to the inclusion of handouts, less than 50% of the sessions included written materials.
Of those materials that were provided by NASAA corporate and NASAA member
jurisdictions, few contained any NASAA branding. A number of attendees expressed
disappointment regarding the availability of written materials. One participant suggested
that all of the written material be provided to attendees on a flash drive rather than in a
notebook. The individual thought a flash drive would allow Lonnie Martin to easily
download materials received during the training to the flash drives rather than making
photocopies at the hotel. Additionally, participants would have the materials for all three
tracks available to share with colleagues upon their return.
Moderators
In evaluating breakout group moderators for the case study, the STT Committee
considers knowledge of the subject matter, the ability to manage the group, openness to
questions, the ability to answer questions, the ability to obtain participation from the
group, use of time, and level of preparation. The case study was used for Tracks One and
Two. As noted previously, several participants mentioned to me that they would prefer a
4
clearer description of learning objectives and defined tools and skills which can be
readily applied upon return to their office.
All moderators were familiar with the subject matter and prepared. Time was used
appropriately with each person facilitating the case study properly balancing analysis
time within the individual groups with time for discussion within the larger group. The
success of individual moderators depended upon the moderator’s experience with small
group dynamics as well as the composition of the group. Some groups were comprised
mostly of very new investigators/attorneys while others were more balanced.
The most effective moderators were able to engage all members of the group and did not
allow one person take over the discussion. Additionally, most moderators used time
within the smaller group wisely affording adequate time to analyze the problem, engage
in discussion, and reach a general consensus on how to approach the problem.
A couple of challenges presented themselves during the case study. One was the issue of
participants finding themselves in a Track that did not suit their needs. As a result,
participants who had not received the case study in advance joined the group late and had
to try to catch up with the other attendees. The other challenge was a slight lack of
cohesiveness relative to how the case study was approached in the individual groups.
Several participants mentioned to me that they would have preferred a more methodical,
hands on approach whereby they would walk through each step from the point that the
case entered the door. This comment goes back to the desire to leave with a set of tools
that can be put into practice.
Recommendations
Based upon my observations and informal conversations with speakers, moderators and
participants, the STT Committee makes the following recommendations for
consideration:
1. The existence of the Distance Learning site and purpose of the videotaping should
be made clear to speakers and attendees. Program materials and introductory
comments can include information on the Distance Learning site and encourage
attendees to use the site to complement their training experience.
2. Audio-visual equipment, particularly lighting, should be positioned in a way that
is the least disruptive to speakers and audience members.
3. More detailed information regarding the content and learning objectives for each
track might be made available in advance so that participants can make better
decisions in selecting a track. Participants should be discouraged from moving
between tracks since it hampers their ability to gain the most from their learning
experience. In addition, participants should be encouraged to attend all sessions
for their particular track and not leave the training before it is completed.
5
In order to assist with this particular item, the STT Committee is going to develop
a catalog of in-person and online training programs available to NASAA
members. The catalog will provide a more detailed description of each program
and the individual tracks so that participants can make better decisions about
which track is appropriate.
4. Speakers and moderators that are either NASAA Corporate or NASAA members
and their staff should be provided with information on effective presentations and
small group dynamics. To that end, the STT Committee would like to tape a short
training on effective speaking and presentation skills as well as facilitation. This
video can be posted to the NASAA website to be viewed by anyone who may be
making a presentation, participating on a speakers’ panel, making an individual
speech, or facilitating or moderating small group discussions. Presenters, once
selected, should be encouraged to view the video.
5.
Presenters might be encouraged to review the NASAA Training Handbook in
developing any written or electronic presentation. Written or electronic
presentation materials to be used as part of a NASAA training program, if
prepared by NASAA Corporate or NASAA members, should include NASAA
branding and should be provided to the Project Group far enough in advance to be
included with the training materials notebooks
6. NASAA might consider providing training materials on a flash drive rather than
individual notebooks. Recognizing that a cost benefit analysis would need to be
considered, a flash drive would afford attendees the opportunity to more readily
share materials with colleagues and download additional materials provided
electronically during or after the training. On the other hand, use of flash drives
may create a problem other logistical problems. This recommendation would
need to be explored closely before deciding whether to implement it.
7. Any zone meetings could either be held before the training program commences
or after it ends in order to increase participation by all states present. Holding the
zone meetings while training sessions are continuing creates an unnecessary
challenge for those wishing to attend. Another alternative is to clearly indicate
only participants in Track Three should attend the zone meetings but that is a less
favored approach.
8. Use of case studies should be for the purpose of practical application with training
participants actually developing the types of tools and documents one could
expect to utilize in their day-to-day work including proper case summaries,
interview questions, investigative timelines, and the like. If desired, the
Committee can develop a resource tool on how to develop and use case studies in
training for use by the Sections.
6
7