New Mount Controller Software
(the 30,000 foot view)
"The reasonable man adapts himself to the world; the unreasonable one persists in trying to adapt the
world to himself. Therefore, all progress depends upon the unreasonable man." -- G. B. Shaw
D. Clark
10/21/14
Project Goals
Replace ancient computer hardware and
software with a solution that is:
–
–
–
–
–
–
Modular
Maintainable
Robust
Leverages new technologies
Uses appropriate “best practices”
Replaces special software and drivers with an
open-source solution
Modular Software
• Mount control software is broken down into key
components that are assigned to specific people.
• Key components tend to have one or only a few
functions.
• Components communicate with each other and
the outside world using a common framework.
• Functions are decoupled using messages across
the framework to enable distributed operations.
Maintainable Software
• Avoid use of special hardware and software libraries.
• Use standard Linux libraries, particularly thread-safe POSIX
routines in time-critical code.
• Use “off-the-shelf” software components, e.g. redis,
AstroPy, SOFA, Aladin, among others as examples.
• Enforce language compatibility among the development
team (C, Python, Node.js, Javascript, others?).
• All software is maintained in a git repository, and the build
environment is kept generic and not specific to one
developer’s particular machine and toolchain setup.
• Regular meetings are held among the development team to
share progress and information about how their software
works.
Robust Software
• The use of a stable CentOS kernel and building code only
against that kernel library provides general insurance
against kernel and security updates breaking functionality.
• A policy of using 3rd-party libraries and software packages
offloads bugfixes and maintenance from the MMTO staff
(so long as packages are updated as necessary).
• General software maintenance given commonality in the
core languages makes it possible for more than one person
to work on the code.
• An effort to document the software as the project
progresses will pay dividends in the future as the
Observatory’s needs evolve.
Leveraging New Technologies
• Computer languages and software packages evolve
rapidly and MMTO should not depend on a particular
set of tools and algorithms necessarily being forever
static – within reason!
• HTML5, Node.js, redis, and other software that may
become available in the future will position us well for
future improvements and upgrades as needed.
• The mount control hardware, therefore, needs to be
designed around the concept of flexibility (e.g. NO
special hardware and a generic PC). Hence, the choice
of EtherCAT and ethernet for i/o to decouple the
computer hardware from the drive system electronics.
Use “Best Practices”
• A git repository for archiving the software.
• Modern browser-based interfaces as the GUI front-end to the
system.
• Use an up-to-date computer and Linux kernel.
• Do avoid “bleeding edge” Linux distros: go with a stable one and
plan to upgrade only with major releases (e.g. CentOS 7  CentOS
8) on roughly 5-year increments.
• Use standard string-messaging protocols (e.g. JSON) and avoid
strange libraries and non-standard message dictionaries.
• Leverage high-performance NoSQL software as a fast data
framework, so that parts of the code could be distributed to
machine(s) other than the mount PC.
• Modularized code sections can and should be individually tested
before deployment.
Mount Control I/O
MMTO Ethernet
Dedicated Ethernet Port
EtherCAT Master
Heidenhain EIB741 Library
Axis Encoders
Drive amplifiers, etc.
Software Components
Not all software shown here necessarily runs on the
mount computer; network sockets and the message
backbone make it possible to distribute the work.
D:\Shuttle Documents\mount_software_breakdown.vsd
Software Component Details
Time Service
Uses the standard Linux NTP service along with the MMT GPS time server and outside network services
(IERS) to get the current DUT (+ polar motion), and outputs UT1 and TT values accurate to milli-seconds,
adjusting the mount computer clock on a regular basis. This proper date/time is consumed by
astronomy computations and other “customers”.
Astronomy Computations
Uses the standard libraries (SOFA) to compute target positions to point the mount to the observed star
position. No distinction should be made between sidereal and non-sidereal targets for uniformity of the
code paths (e.g. use SOFA’s UT1 and TT time objects).
TPOINT Corrections
The necessary pointing corrections from the astronomy computation should be applied with this code
to separate out the introduction of pointing-fit corrections to the computed target position.
Encoder and Telescope Corrections
Here we add whatever offsets or other effects from tube flexure and hysteresis that aren’t covered by
TPOINT corrections. We may choose to implement a LUT or polynomial correction factor to the encoder
position data, for example.
Focal Plane Adjustments
Provides adjustments to the demanded target position to align the telescope to a given point in the
image plane (e.g. feed an off-axis imager).
Apply Limits
In this portion of the code, the computed mount position should be checked against the motion limits
and only if they can be met will the position be sent to the mount servo loops. The “time to limit”
values for each axis can be computed here as well.
Catalog Interface
Provide users with a way to browse and pick a target star from a standard catalog, or use a catalog provided
by the observer(s) and send the chosen coordinates to the astronomy code.
User Interface
Much like the existing GUI, provide a way to turn drives on and off, enter fixed positions for the mount, and
other necessary functions for operation.
Safety Interlocks
Communicates with the interlock hardware and responds to either an interlock event (killing a drive axis), or
a user interaction (turning on the drives). The interlock VME computer will eventually be replaced with an
EtherCAT interface currently in the hardware-design pipeline.
Servo Loops
Code that actually uses the EtherCAT and EIB741 interfaces to the drive hardware and controls the telescope
position.
Pointing Offsets
Provide a socket-based mechanism for pointing offsets to be introduced from other “customers” of the
telescope (example: instrument offsets sent by a separate computer).
Autoguiding Inputs
Another socket-based mechanism for accepting tracking correction demands from autoguiders.
Paddle Inputs
Support manual guiding corrections over a different network socket.
Event Logging
Service to provide notice of important changes to the mount system, such as drives being enabled, etc.
Publish/Subscribe Interface
Service to expose variables within the mount control system to other consumers of the data via the standard
publish/subscribe methods within redis (or others – SAO messages?).
Telemetry Service
Code that continuously collects system performance data, and writes collections of the data to a database.
The servo performance data collection will be very similar to the existing system, only with much higher
granularity since all the loop data from all axes will be continuously available.
Existing Software Inventory
• Of 56 source C files in the mount git repository, about 5 can be
ported to the newer software infrastructure almost as-is. These are
the interlock computer interfaces, which cannot be changed until
the interlock computer itself is replaced.
• Of the remaining 51 files, 25 are completely removed.
• Now only about half the existing code base needs to be rewritten,
and large portions of that (e.g. servo code) are auto-generated.
Many can be greatly simplified in the process.
• Parts of the code (e.g. EtherCAT and EIB741 interfaces) are already
available, or can quickly be produced (e.g. time service mostly uses
existing Linux NTP service plus the IERS query server we already
have – with minor modifications).
• All of the Xephem software in the /libastro source directory can be
replaced with SOFA libraries; SOFA can compute sidereal and solarsystem target coordinates through use of UT1 and TT time objects.
Mount Source File List
File List
protos.h
Remove
Here is a list of all files with brief descriptions:
reboot.c
Remove
[detail level 123]
host
ren.c
Replace with publish/subscribe interface
rmount.c
Replace with publish/subscribe interface
src
libastro
Replace with SOFA
rtw
Removed
axis.c
Replace with servo *.h C header files
cell_udp_telem.c Replace with publish/subscribe interface
config.h
Removed
encoder.c
Replace with EIB741 interface
filters.c
Removed
filters.h
Removed
hardware.c
Replaced by EtherCAT hardware i/o
hires.c
Removed
http_mount.c
Removed
interlock.c
Port to new interlock C application
interlock.h
Port to new interlock C application
main.c
Replaced by new servo loop and IPC backbone
mount.h
Needed information ported to C header
net.h
Replace with publish/subscribe interface
net0.c
Replace with publish/subscribe interface
net1.c
Removed
ocs_mount.c
Removed
offsets.c
Replace with user interface and redis backbone
param.h
Move to text file or C header
point.c
Replace with scripted interface – “pointing” service
preces.c
Replaced by SOFA
rem_mount.c Replaced by user interface + pub/sub + others
rotator.c
Replace with new servo loop
servmath.c
Remove
servolib.c
Remove
servolib.h
Remove
settings.c
Replaced by pub/sub + others
site.h
Moved to new site header/text file
switch.c
Remove
t628.c
Remove
taj.c
Remove
tcs.h
Replaced with SOFA
tcs_debug.c
Remove
tcs_loop.c
Remove
tcs_main.c
Remove
tcs_move.c
Remove
tcs_tune.c
Remove
telem.c
Replaced by servo telemetry services
telem.h
Replaced by servo telemetry services
telem_rd.c
Replaced by servo telemetry services
telem_sv.c
Replaced by servo telemetry services
telem_ts.c
Replaced by servo telemetry services
telrel.c
Replaced with SOFA and pointing service
timer.c
Remove
xephem.c
Remove
tune
dodft.c
Remove
rdata.c
Remove
rtune.c
Remove
First Steps
We will build up a complete single-axis controller for the rotator axis for use in day-time testing and
software validation. This computer and hardware:
• Uses a reclaimed Shuttle box running CentOS 7 patched to run an rtpreempt kernel.
• Has the etherlab.org EtherCAT master stack installed.
• Will be attached to the interlock system and the rotator axis via the appropriate EtherCAT
terminals, already present in the interlock cabinet.
• Still need to either adapt the existing 11uA output heads to 1Vpp signals, or buy new encoder
heads to attach the rotator tape encoder to the Heidenhain EIB741. Alternatively, we have in hand
incremental encoder input terminals for EtherCAT and can use the existing interpolator box
outputs.
• Using the new computer and EtherCAT, we will be able to easily switch over the system cables and
run the new equipment in parallel with the existing mount control system to perform testing as
needed.
• The computer will need a minimal set of the code base necessary to run as an alternate axis
controller:
 Time Service
 Astronomy Code to compute needed rotator positions
 Servo code and logic
 Interlock interface
 User interface
 Telemetry
In this way, the needed system software can be “test flown” without disturbing the existing mount
control system.
Next Stage
Partially in parallel with the first stage, application development can be
start with moving the mount software over to the new system:
1. Port the interlock computer interface to the new computer –
prepare for the eventual replacement of the interlock VME crate.
2. Develop and test EtherCAT-based encoder readout, and/or work
on moving to the proposed new Heidenhain RCN8500 units
(which cannot be installed in the existing mount system as-is).
3. Get the telemetry and publish/subscribe interfaces ready to run.
4. Complete development of new servo loops for the main telescope
axes.
5. Add in the autoguiding and paddle network support services.
6. Prepare hardware for day-time testing of new main-axis controls.
7. Finally, complete new user GUIs and catalog interfaces.
Final Stage
Once all the hardware and software is ready for fullup testing, we should be ready to switch over to the
final form of the hardware and software:
A. New Heidenhain RCN8500 encoders can be
installed.
B. Final version of mount computer with all
necessary network hardware configured, tested
and installed.
C. All-new telescope axis servo loops, interlock
computer ready to replace, and can go to
regular night-time use.
Team Roles
Given the availability of the various people involved in the new mount system
development, the team breaks down as follows* –
Person
Area of Responsibility
Dusty Clark
Servo loops, system integration, hardware interfaces, telemetry services (mainly for servos),
and interlocks.
Keith Powell
Servo loop design and analysis assistance and upport for active vibration suppression
development.
Duane Gibson
User interfaces, catalog interface, message backbone, event logging and telemetry service
back-end.
Dallan Porter
User interface, catalogs, and network services (e.g. autoguiding and paddle inputs).
Skip Schaller
System OS administration, integration of EtherCAT library, astronomy code support,
VxWorks code porting support.
Richard Cool
Joanna Hinz
Validation of user and catalog interfaces, astronomy code testing, and support for
instrument interfaces to the mount computer system.
Tom Trebisky
Documentation, general support, and advice on porting code from the existing VxWorks
system.
(emeritus member)
*Considerable overlap among the various developers is expected, not all roles/responsibilities will necessarily be fixed.
Schedule
This is a guess, based on where we are now, and given the bump in the
schedule known as “aluminizing”:
• 4Q 2014: finish getting rotator-axis test computer and necessary EtherCAT
interfaces and hardware ready to run servo loops.
• 1Q 2015: prepare new time service and astronomy code to bring the
rotator-axis control operational equivalent to existing axis. Daytime
operation only (night-time during M&E).
• 2Q 2015: Development suspended due to aluminization activities.
• 3Q-4Q 2015: Begin development of the other needed software
infrastructure to begin main-axis controller operation. Develop main-axis
control loops with existing encoders.
• 4Q 2015 – 2Q 2016: Prepare for arrival of new encoders, complete mainaxis servo development, finish development of new user interfaces,
publish/subscribe, and telemetry back-end.
• 3Q 2016: Final development and deployment of new encoders. Begin
regular night-time operations on the new system.