WHITE PAPER
© 2011 Brian M. Kelly
Open Source Enterprise Automation
With Net::FullAuto Software:






1
extreme ease
extreme power
extreme control
extreme security
extreme flexibility
extreme cost savings
Volume One: Introducing Net::FullAuto
©2011 brian.kelly@fullautosoftware.net
Contents:
01
08
13
14
15
16
17
18
19
20
21
2
Introduction (The Net::FullAuto Vision)
Something Un-Expect-ed (Really!)
Follow the Money (Use Cases and Cost Effectiveness)
Skip the Scheduler (Reduce Job Scheduling Dependency)
Getting Started is EASY (Simple to Setup)
Back to Basics (Net::FullAuto Use Cases and Examples)
Need to Know Basis (Role Based Security)
No Blind Alleys (Robust Error Checking/Handling/Reporting)
One Button Everything (Full Enterprise Automation)
The Network is the Computer (At Last!)
The Future of Net::FullAuto (UNLIMITED!)
Volume One: Introducing Net::FullAuto
©2011 brian.kelly@fullautosoftware.net
INTRODUCTION
In 2000, Brian Kelly, the author of Net::FullAuto, was working as an independent
consultant for a large insurance company. As a Software Configuration Manager,
it was his job to manage source control and code movement for a large multimillion dollar customer portal project. What continually frustrated him was how
much of his job was manual, and how tools that promised to automate tasks,
often failed – often spectacularly. Tools that did exist were outrageously
expensive, required dedicated personnel, long lead times, huge learning curves,
and even then it never took long to find a nuance or use case that even the pricey
“software giant” itself failed to handle adequately. The reason more often than
not, that otherwise simple manual tasks resist full automation, has to do with the
reason most software in this world is written in the first place. Proprietary
software is not written to solve problems, it is written to MAKE MONEY. But no
one is going to purchase expensive software that does “nothing”, so instead some
problem needing a solution is selected, and that becomes the new money magnet
around which whole organizations rally. In some cases, more time is spent on
insuring the software is not copied, or used beyond its licensing period, than on
the actual problem it was originally conceived to solve. In fact, one multi-billion
dollar software company charges a license fee for each job run on its software.
Now you know just one of the reasons it’s a multi-billion dollar a year
organization! Many of course simply call that “success”, and any criticism of that
success as sour grapes – and that is hard to argue. But from the perspective of
someone just trying to solve technical problems as part of their daily occupational
duties, such “success” is often the “problem” to get away from – as fast as
possible.
Such was the Net::FullAuto author’s experience. Organizations he consulted with
always wanted to avoid the expensive solutions whenever possible, and
constantly looked to him for creative alternatives. But the simple fact was, no
viable open source solutions to robust enterprise automation existed. And so,
with a mountain of optimism, and an equal amount of naiveté, the author of
Net::FullAuto set out to fill the void. At the time, he figured it would take six
months to ready Version 1.0. That was well over ELEVEN years ago. It took more
than eleven years and thousands and thousands of hours of development, but
Version 1.0 is now a reality.
3
Volume One: Introducing Net::FullAuto
©2011 brian.kelly@fullautosoftware.net
What took so long? Stubbornness! The author simply refused to compromise his
vision. What the author wanted, was a programmable tool that worked just like a
user working in a command shell environment – the environment most commonly
used by IT personnel. At the time, secure shell was in its infancy, and telnet was
the principle means through which activities on remote hosts were performed.
What he wanted was a tool that would work against an unknown host, without
any knowledge of the host beyond an IP address and login ID and password much the same way a human user works with telnet (and today – secure shell or
SSH). A human user logging into a remote host via SSH for the first time does not
need to know what operating system she is accessing, what the command prompt
is, or what shell language is set as the default. However, it turned out that
software programs aspiring to work like humans needed that information (until
Net::FullAuto that is).
So what? How difficult is it for an administrator or programmer to find out what
the default shell is and simply hard code it or create a configuration file for the
automation code to use?
Answer: it’s not difficult at all – for one host. But what about a THOUSAND
hosts? Now suddenly, the problem is not so trivial. This is even truer of
something like “command prompts” and “login banners”. In fact, it is now more
often the case that command prompts are dynamically generated with such items
as the hostname, username and directory location. Here is an example of the
default prompt with a fresh installation of Cygwin:
4
Volume One: Introducing Net::FullAuto
©2011 brian.kelly@fullautosoftware.net
Note the presence of the username: KB06606; the hostname: CHG211134 and
the current directory: ~ (tilde or the home directory). Now change to a new
directory:
Now consider the Cywin login banner:
5
Volume One: Introducing Net::FullAuto
©2011 brian.kelly@fullautosoftware.net
Obviously, these mnemonics where created for HUMAN users, not programs. Yet
programs that aspire to work just like humans have to be intelligent enough to
cope with such dynamic and unexpected data.
Many computer scientists have considered this an impossible task. Consider the
opinion of Randall L. Schwartz, expert programmer and author of numerous
books and manuals, on the perlmonks.org collaboration site:
I don't believe there's any way that a program that sits on a pty1 (like Expect.pm) can ever
distinguish where the data is coming from. This is the blessing and curse of Unix I/O.
So, no, I think you're just gonna have to consider the various things to wait for, some of
which will be error-like messages coming at unexpected times.
Randall L. Schwartz, http://www.perlmonks.org/?node_id=28096
To date, and to the author’s knowledge, no program has successfully gotten past
this problem – that is, until Net::FullAuto.
Again: So what? The author has created a solution in search of a problem, the
critics will say. WRONG – if that was truly the case, then no one would have to
shell out a hundred dollars just to add a new job for nightly execution from
software that they’ve already paid thousands of dollars to acquire in the first
place! Yet, all over the world, money flows for these fees every single day. If there
was a viable open source alternative, this would surely not be the case. (More on
schedule based automation in a later chapter.)
Consider that every computer job that was ever automated was in its genesis run
and tested manually. What that indicates, assuming there was some means by
which those manual steps could be translated into automated steps (with no
more effort or expertise than was needed to create the original job, task, process,
etc.), is that such automation would be the preferred way to proceed.
In fact, what the author truly envisioned was a tool where the automation would
essentially be “automatically” created at the same time and in the same scripting
1
A pty is also known as a pseudo-terminal and is the principle channel through which software can attempt to
emulate human interaction with command shells. Pseudo-terminals are natively available on all Unix variants, but is
available on Microsoft Windows only with additional software – like the Open Source Cygwin Linux emulation layer
available for free from Redhat. http://www.cygwin.com*
*
Cygwin is automatically installed and configured by the Net::FullAuto Windows installer (so no worries!)
6
Volume One: Introducing Net::FullAuto
©2011 brian.kelly@fullautosoftware.net
space as the business logic itself. The result would be code that could be run
either manually or automated – depending on the needs of the organization –
without any additional expense or development time or resources needed.
How cool would that be?
That tool NOW exists – and its name is Net::FullAuto.
7
Volume One: Introducing Net::FullAuto
©2011 brian.kelly@fullautosoftware.net
SOMTHING UN-Expect-ED …
Critics would say that an Open Source command shell automation tool already
exists, and its name is Expect.
True enough.
The homepage of Expect makes a pretty bold claim:
Expect can make easy all sorts of tasks that are prohibitively difficult with anything else.
You will find that Expect is an absolutely invaluable tool - using it, you will be able to
automate tasks that you've never even thought of before - and you'll be able to do this
automation quickly and easily.
http://www.nist.gov/el/msid/expect.cfm, as modified April, 2010
Expect has been around for years, and indeed, is in use all over the globe. Most
IT professionals have heard of Expect. But, most IT professionals don’t actually
use Expect. An article published in the Linux Gazette gives us some insight into
why this might be:
In the design of automated systems in Expect, one of the more difficult hurdles many
programmers encounter is ensuring communication with ill-behaved connections and
remote terminals.
David Fisher, Advanced Programming in Expect: A Bulletproof Interface, Linux Gazette, Issue 48, 1999
That is a bit of an understatement. The fulcrum of automation in the enterprise is
precisely communication with remote terminals (or systems/hosts). It is precisely
communication with remote systems, and the desire to automate processes
involving them that prompts IT professionals to experiment with Expect in the
first place! When such an individual (or team) discovers that Expect is anything
but a silver bullet, they more often than not fall back on what they know best –
manual processes.
Expect expert David Fisher goes on to provide a little more detail about the
specific problems Expect has with remote communications:
If you're reading this article, though, I suspect that you've encountered some of the
problems I have when programming in Expect, and you're looking for the solution here.
If you're just reading out of interest, the problems arise when automating a session on a
machine off in a lab, or on the other side of the world. Strange characters pop up over the
connection, and the terminal you're connected to does weird things with its echo, but
8
Volume One: Introducing Net::FullAuto
©2011 brian.kelly@fullautosoftware.net
everything is working. It becomes very difficult to determine if what was sent was
properly received when you have noise on the connection, terminal control codes inserted
in the echo, and even server timeouts between the automation program and the remote
session.
David Fisher, Advanced Programming in Expect: A Bulletproof Interface, Linux Gazette, Issue 48, 1999
Fisher goes on to provide brilliant workarounds for many of these issues. Indeed
Expect is used successfully by large firms all over the globe for almost countless
mission critical tasks. But he also cautions:
The code provided in this article is executable, but needs to be incorporated into any
system in which it is to be used.
David Fisher, Advanced Programming in Expect: A Bulletproof Interface, Linux Gazette, Issue 48, 1999
So in other words, even today, twelve years after he wrote that article, if you play
with Expect and don’t have Mr. Fisher’s articles and code, you’re simply going to
run into the same problems thousands have before. In fact, he even suggests as
much in another article he wrote just this past July regarding spawned ids with
Expect:
Given the number of times I’ve had to fix this issue in existing code, and the number of
times I’ve had to explain this simple solution in interviews and in courses I’ve taught in
automation, it seems beneficial to share it with the entire community.
David Fisher, Advanced Programming in Expect: sHellspawn – Demons in the Daemons, Linux Gazette, July, 2011
It’s a bit disturbing to think that thousands of individuals have wasted thousands
and thousands of hours (or days) running into the same stumbling blocks over
and over again – especially when solutions exist, and have for quite some time.
But this phenomenon doesn’t just exist between organizations, but even more
frightening to managers everywhere, it happens within them just as much.
The Wikipedia article on Expect, points out a couple additional “limitations” with
Expect:
Expect inherits the same syntax convention as Tcl, which may seem unfamiliar if
accustomed to other script languages. Compared to languages such as bash, csh, and Perl,
Expect has a different twist. It is sometimes challenging to remember when a variable must
be prefixed with a "$", and when it must not. …
Another limitation is the difficulty in porting Expect scripts between platforms. For
example, an Expect script that was written to use several Unix-based tools, might not be
9
Volume One: Introducing Net::FullAuto
©2011 brian.kelly@fullautosoftware.net
suitable if migrated to a Windows platform. If possible, the programmer must find
counterpart command-line applications that provide the same information, and this will
probably require changing the send/expects, which can be a major part of the script.
(Emphasis added.)
http://en.wikipedia.org/wiki/Expect as modified on 12 November 2011 at 13:02
It’s tempting to think, that the words “Expect has a different twist” really mean
“The Expect syntax is difficult, and non-intuitive. Creating automation solutions
with Expect is HARD - a bit contrary to the claims on the Expect home page.”
Indeed – harder than it should be. This is precisely why most IT professionals
have heard of Expect, but don’t actually use it. But in fairness to the creators of
Expect, automation itself is HARD, and it is not unreasonable to expect (play on
words) that a program created to accomplish it, would also demand a lot of skill
from its users.
The second limitation, which has added emphasis to highlight it, underscores a
very important point: That being - that with Expect, each command, each element
of business logic, or code instruction, would have to be accompanied or “wrapped”
with its own individual set of “send/expects, which can be a major part of the
script”. This works against any real desire to combine business logic with
automation code in any truly manageable sense – that is, until Net::FullAuto.
Expect utility’s very name communicates it’s methodology for parsing pseudoterminal data; it expects, what it is told to “expect”. A fresh default installation of
Expect is not capable of dynamic discovery of the host it is working against. It
needs to be told what to “expect”. Here is a typical, simple example of Expect
using SSH to connect to a remote host, and running the ‘uname –a’ command:
#!/usr/bin/expect
set timeout 1
set cmd {uname -a}
spawn ssh bkelly@localhost
expect_after eof { exit 0 }
## interact with SSH
expect "yes/no" { send "yes\r" }
expect "password:" { send "password\r" }
10
Volume One: Introducing Net::FullAuto
©2011 brian.kelly@fullautosoftware.net
expect "# "
send "$cmd\r"
expect "$cmd\r"
expect "(.*)\r"
send "exit\r
Note the existence of no less than FIVE “expect” statements, just to run the
‘uname –a’ command on the localhost (in this case a laptop) via SSH.
Now let’s see the same thing with Net::FullAuto:
$localhost->cmd( ‘uname –a’ );
Believe it or not, that’s it!! Everything else is handled automatically by
Net::FullAuto.
If I want to connect to a different host:
( $remotehost ) = connect_ssh ( ‘Remote_Host’ );
$remotehost->cmd( ‘uname –a’ );
Net::FullAuto handles ALL the other details for you, so you can focus on the
business task – not the automation one.
Lest ye thinks the author doth protest too much, let it be said that that Expect is
an important tool, and that until now, was indeed about the only truly practical
way to automate almost countless tasks. In fact, there is an Expect module for
Perl that can be used right inside a Net::FullAuto script. For some tasks, it will
still be the best means for providing an automation solution. But for most tasks,
a default installation of Net::FullAuto will provide more ease, more power, more
control, more security, more flexibility and more cost savings than any ambitious
Enterprise Automation solution using Expect alone.
Net::FullAuto truly is un-Expect-ed: in more ways than one. Not only does it
function differently, and in the process eliminate most of Expect’s limitations, but
indeed, most of the IT industry does not see it coming. In fact, it appears that
many think automation utilities are about as advanced as they are ever going to
get. Consider the opinion of a commentator in response to a wiki.tcl.tk topic page
entitled It seems like Expect is not actively developed:
11
Volume One: Introducing Net::FullAuto
©2011 brian.kelly@fullautosoftware.net
I think people are just having a hard time accepting that Expect in its current form is
complete. Does what it is supposed to. No need for a new version.
Tykling, as posted at http://wiki.tcl.tk/19877 22nd Nov 2007
All that can intelligently be said in response - is that it is a good thing “Tykling” is
an anonymous poster! Obviously, such an individual would find Net::FullAuto to
be not just a surprising innovation, but indeed, perhaps a shocking one.
And finally, a quote used in this White Paper’s introduction is so “spot on”, that it
bears repeating here:
I don't believe there's any way that a program that sits on a pty (like Expect.pm) can ever
distinguish where the data is coming from. This is the blessing and curse of Unix I/O.
So, no, I think you're just gonna have to consider the various things to wait for, some of
which will be error-like messages coming at unexpected times.
Randall L. Schwartz, http://www.perlmonks.org/?node_id=28096
Randall? Tykling?
Prepare to be amazed …
12
Volume One: Introducing Net::FullAuto
©2011 brian.kelly@fullautosoftware.net
FOLLOW THE MONEY
Because Enterprise Automation is HARD (until Net::FullAuto), it follows logically
that it would also be VERY expensive; and it is. Enterprise computing typically
encompasses a number of “concerns”, but by far the most important is the
quality of the end user experience. Whether that user is accessing a public
internet site, or an internal intranet site, organizations focus overwhelmingly on
optimizing functionality for the end user. In service to that goal, IT departments
around the world employ armies of specialists dedicated to activities from
database development, to website image design, to server administration.
Depending on the size of the organization, the number of hosts used to service all
these activities can number from dozens to thousands.
All of these hosts utilize modern software to perform tasks that require some form
of communication with other hosts and devices: data transfers, remote software
installation and patching, system monitoring, backups, configuration changes,
user administration, job scheduling and vulnerability testing – to name just a
few. Within those categories are high frequency, moderate frequency, low
frequency and one-time activities.
High frequency tasks are usually handled via some form of customized, clientserver application – usually written in Java or C++, often tightly coupled to a
robust database such as Oracle or even a mainframe. Net::FullAuto is not
designed to compete with these solutions. There are numerous competing
technologies for this business. It’s all client-server in architecture and very
expensive, in terms of hardware, software and/or skilled personnel.
High frequency tasks are measured in micro-seconds. Again, Net::FullAuto is not
designed for this niche. But for every other niche, from moderate frequency tasks
(separated in time by at least seconds) to one-time only activities, Net::FullAuto
can either support or supplement or even replace costly solutions currently being
utilized the world over – including and especially, manual processes.
There-in lies the one of the biggest misapplications of technology in the world
today – software designed for high-frequency activities, being marketed and sold
for less frequent tasks. It is of course easy to understand why. Software
companies are always looking for new revenue streams, and since high frequency
applications were developed in response to an almost endless river of cash being
thrown at them, it is no surprise that organizations would look for other “uses”
13
Volume One: Introducing Net::FullAuto
©2011 brian.kelly@fullautosoftware.net
for the code base and skill sets they spent so much money developing. The result
is client-server architecture for practically “everything” that involves
communication between hosts.
This is yet another reason that Expect never got the usage its power and
versatility suggested (in spite of its limitations) – there was always a competing
client-server approach to the same problem with LOTS of marketing personnel
behind them. In contrast, practically no one was marketing solutions created
with Expect. Expect could have saved lots of organizations, LOTS of money over
the last two decades. Definitely it’s a missed opportunity, and one the
Net::FullAuto author would have pursued himself, had he known then what he
knows now.
But that was yesterday, and today is today. Today we have Net::FullAuto without
Expect’s limitations to “finally” take on the client-server behemoth. The biggest
problem with client-server architecture is that at the very least, it requires a
“client” or “agent” on each and every host in the process group. Each client or
agent is often a separate license fee, and also represents just one more piece of
software on a host that has to installed, inventoried, tracked, synchronized,
configured and eventually upgraded. One or two is not too bad, but get into the
thousands and suddenly it’s not such a trivial undertaking!
Net::FullAuto eliminates ALL those headaches, for it works with the host’s own
native installation of SSH and SFTP. (If it’s a Windows host, a free installation of
Cygwin2 will supply these.)
But leveraging a high frequency activity code base is not the only reason for the
proliferation of client-server architecture for non high frequency tasks. The other
reason recalls the earlier Expect complaint explored by David Fisher:
“communication with ill-behaved connections”. This is an issue with all remote
communications – from computer hosts to cell phones to CB radios (remember
those?). Client-server works well for coping with this issue because the same
application controls the software on both sides of the connection. There are no
compatibility issues, protocol disagreements, or timing issues. Furthermore, prior
to the advent of SSH and SFTP, client-server architecture was about the only
reliable way to provide encrypted communications between hosts – a critical
consideration for Enterprise computing.
2
Cygwin can be automatically installed and configured on remote Windows hosts with the Net::FullAuto Windows
installer. Unlike Unix remote hosts, Windows remote hosts require this additional step. (Blame Bill Gates!)
14
Volume One: Introducing Net::FullAuto
©2011 brian.kelly@fullautosoftware.net
But again, that was yesterday. Today SSH and SFTP are used billions of times a
day all over the world, and their reliability and security are as trusted as any
other software in use on the planet today. At one time, proprietary client-server
solutions were the only game in town. Not anymore.
There is now an option. Organizations can continue shelling out mountains of
cash for licensed proprietary, or custom home-grown client-server solutions to do
data transfers, remote software installation and patching, system monitoring,
backups, configuration changes, user administration, job scheduling, application
testing, vulnerability testing, etc.; or even worse, continue to pay personnel to do
these tasks manually (as most, if not all organizations do for at least some tasks –
tasks that can easily and cost-effectively be automated away with Net::FullAuto.)
15
Volume One: Introducing Net::FullAuto
©2011 brian.kelly@fullautosoftware.net
SKIP THE SCHEDULER
Without a doubt, the most widely used tool for Enterprise Automation, is actually
not a “true” automation tool. It’s actually a “clock”. True automation, is output
based automation. A task is initiated in response to the output (on both the
STDOUT and STDERR channels) of the preceding task. This is how humans
interact with computers. A human operator wanting to move a file, will run a
command to get a directory listing, and look for a particular file she was told
would be there. If it’s not there, she sends an email to the person who was
supposed to deliver the file. If the file is there she sends it to a destination host. If
the destination host is unavailable, she opens a ticket with the network team.
A job scheduler however, (like the ubiquitous “cron” utility on Unix hosts) is able
to ask one and only one question: “what time is it?” And if it happens to match
the time scheduled for a job to kick off, then it runs that job (or command). It has
no other “coupling” with the process it kicked off.
This is unfortunate, because distributed processes often have dependency
chains, or even require some form of transactional handling (i.e., rollback) in the
event that some task in the event chain fails or produces an error. Yet, because
tools like Expect have been underutilized for the last two decades (for reasons
outlined earlier), the vast majority of distributed processes are handled by
launching their itemized tasks in distinct time-separated slots. This is a rather
crude, almost brute-force way to achieve distributed process automation.
The costs are many. The most obvious one being, what happens when there is a
failure or error? “A chain is only as strong as its weakest link,” is a truism that all
of us have experienced and pondered many times in our lives. This is no less true
of a “process chain”. If just one link, one task, breaks, the whole process is
broken (or at least compromised). To cope with this reality, IT professionals often
equip each link in the process chain, with additional code to test the integrity of
the output of the previous time-and-order dependant task. For instance, if the
scheduler launches a script to move a file that was transferred there by an
otherwise non-coupled task launched earlier by ‘a’ scheduler (it doesn’t even have
to be the same scheduler), and there is a problem with that transfer, the
scheduler- launched script must contain code to not only do the task it was
written to do, but to also check the integrity of the file, and perhaps send an
email to alert monitoring staff if it finds a problem.
16
Volume One: Introducing Net::FullAuto
©2011 brian.kelly@fullautosoftware.net
With such architecture, transactional rollback is available only through direct
human intervention. This is yet another reason why there is such a heavy
dependency and emphasis on the client-server distributed application approach
to distributed processing – it can provide true transactional error handling. But
writing a custom client-server application from scratch is expensive, time
consuming, and prone to error. It’s a lot required just to make sure the ones and
zeros show up in the right order.
It seems like a lot of overkill, to create custom client-server architecture, just to
insure that a rather small amount of data, in an infrequent but important
activity, gets transferred from one host to another accurately – and provide an
automated means of rolling back in the event of an error. But I assure you, it’s
more the norm than otherwise – especially in large organizations that can afford
the costs of a development staff.
Job sequencing cannot provide the kind of “quality assurance” that transactional
processing can. Yet, where true transactional processing is not required (or not
affordable) time spaced job execution is the primary means of process
automation. While less expensive and less effort than a client-server application;
a lot of effort and supplemental code still goes into the integrity checking of
artifacts produced by earlier jobs in the process chain. All that is extra time,
effort and expense that is not needed with a utility like Net::FullAuto.
Because Net::FullAuto is an output based utility, and not a time-dependant one,
true transactional capability is possible without having to resort to a customized
client-server application. Net::FullAuto provides a more robust and cost effective
alternative to the crude time dependant (i.e., job scheduled) approach. And while
not “more” robust than a custom client-server application, it can be just as
robust, and MUCH more cost effective (i.e. cheaper) than a custom application
(assuming of course that the activities being automated are less than “high”
frequency).
Using a job scheduler to connect tasks into a distributed process chains has
other downsides as well. Aside from no transactional capability, IT staff has to
manage the relationships of all these “jobs”. This is one reason why job
scheduling software can be some of the most expensive third party software that
organizations utilize. Jobs have to be scheduled accurately, and the relationships
of jobs to each other have to be tracked and managed. Process chains have to
tested, and launch times adjusted to create optimal results. Too much time
between jobs and process completion is needlessly delayed. Too little time and the
17
Volume One: Introducing Net::FullAuto
©2011 brian.kelly@fullautosoftware.net
robustness of the process can be jeopardized. Jobs with relationships to other
jobs have to be rescheduled, terminated, changed, etc, in coordination with those
other jobs. Miss a job, and the dependant process FAILS – often with little or no
error information to assist IT staff in troubleshooting the failure. The result can
be minor or catastrophic – depending on the importance of the affected process.
Tasks connected via Net::FullAuto, are not time dependant, but output based. The
output of the previous task determines what the next task will be. If the output
indicates an error, error handling can be executed, up to and including full
transactional rollback. Beyond rollback, error handling could include task retries,
or other process avenues that might involve using a different host if the one being
initially targeted is unavailable. The options are almost limitless. Unlike Expect,
and even most job scheduling software packages (including some of the very
expensive big-name ones), Net::FullAuto has access to ALL channels of
information on the remote host – both STDOUT and STDERR. Furthermore,
Net::FullAuto can even monitor log files of remote processes executed, and take
actions on conditions uncovered as a result – including terminating and rerunning the errant task.3
We’ve discussed client-server approaches to Enterprise Automation, and we’ve
discussed job scheduler approaches as well. But the titan is the client-server
equipped job scheduler. This is by far the most widespread means of enabling
distributed processing – Enterprise Automation – in use today. It’s also by far the
most expensive. The difference between these giant applications, and a simple
scheduling utility like “cron”, is that the client-server based job scheduler’s are
designed to manage thousands of jobs across thousands of hosts. Large
organizations in particular, have critical need for a “big picture” vantage point.
Such software goes a long way to providing such a vista. Not only can such
software run jobs on remote hosts, and retrieve the output for examination and
archival purposes, but they are also equipped with full featured dashboards that
are indispensable for monitoring staff. Currently, Net::FullAuto cannot provide
this level of functionality – but that is not to say it won’t in the future.
3
This would require running two remote processes to the same remote host in different “threads”, and enabling the
log monitoring thread to terminate the primary task thread when an error condition is discovered. It’s a much more
complex and ambitious utilization of Net::FullAuto than most users would likely attempt; and much more resource
intensive. But to borrow a line from Perl (which Net::FullAuto is written in), Net::FullAuto makes easy tasks easy, and
difficult tasks possible.
18
Volume One: Introducing Net::FullAuto
©2011 brian.kelly@fullautosoftware.net
Net::FullAuto can however, reduce the volume of jobs currently running on such
platforms by replacing time-dependant process chains with output based process
chains – necessitating that only the Net::FullAuto parent process be scheduled.
(This is nothing more than calling the Net::FullAuto executable with a “plan”
number or “code” argument, and other optional arguments. There is more on
Net::FullAuto “plans” and “custom code blocks”, coming up in a later chapter.)
Net::FullAuto scripts can be executed from any scheduler – as simple as cron, or
as gargantuan as some of the huge proprietary packages. This can save
considerable expense when things like license fees are charged for each new job
definition added to the scheduler. Ponder that for a moment – not only have
organizations paid for the software installation, they have to pay to use it as well!
If you’re a stockholder of a company selling this kind of software, you gotta LOVE
it! As asserted earlier – if there was a viable Open Source alternative, it’s doubtful
that organizations would be willing to go along with this kind of “commercial
partnership”. (And that’s being kind.)
But there’s more! Net::FullAuto has its own integrated scheduling! So it’s not even
necessary to use separate third-party scheduling software at all. Net::FullAuto
uses the systems own “cron” utility4 – but expands its capabilities. Net::FullAuto
uses its own menu facility to make setting up jobs fast and easy. Additional
metadata about jobs are stored in Net::FullAuto’s internal database5.
Net::FullAuto also extends beyond cron’s default capabilities by providing an
expiration option for jobs.
4
On Microsoft Windows systems, cron is available to Net::FullAuto via the required Cygwin installation (Cygwin is
installed automatically with the Net::FullAuto Windows installer).
5
Net::FullAuto uses Open Source Oracle BerkeleyDB internally. Oracle BerkeleyDB is automatically downloaded,
installed and configured by the Net::FullAuto installer. (So again, no worries!)
19
Volume One: Introducing Net::FullAuto
©2011 brian.kelly@fullautosoftware.net
Earlier in the discussion, cron was characterized as being limited because it
lacked client-server capability. That is still true – but when partnered with
Net::FullAuto, it doesn’t matter. Net::FullAuto handles all the remote connectivity
and job execution, all cron has to do is start Net::FullAuto (with a “plan” or “code”
argument) on only one host!
It doesn’t get any easier!
So skip the scheduler! Aside from an Enterprise scoped dashboard (which is
planned for future development), Net::FullAuto can do everything any scheduler
in the world can do – and more.
20
Volume One: Introducing Net::FullAuto
©2011 brian.kelly@fullautosoftware.net
GETTING STARTED I S EASY
“Easy” of course, is a very relevant word. Brain surgery is easy – for brain
surgeons. So when it is said that Net::FullAuto makes Enterprise Automation
easy, it is not being asserted that it is easy for kindergartners, sous chefs, and
most grandmothers. Net::FullAuto was created for IT professionals (though being
Open Source, it can be utilized by anyone), and for IT professionals, it is easy.
Consider that the software is named “FullAuto” – so it would be quite
disingenuous if the author didn’t make installation as “automatic” as possible. It
was shared earlier that the author spent nearly 12 years developing
Net::FullAuto. But what might be even more surprising, he spent more than a
year just writing the installer.
It’s as simple as it can get. If it can’t be installed, it can’t used. The easier it is to
install, the more folks will try it. Since the author wanted LOTS of folks to try it,
(and ultimately use it) he figured he better make it pretty darn easy to install.
You be the judge.
This chapter contains a detailed exploration of the installation process. The
reader is highly encouraged to follow along, even if you do not anticipate actually
performing any installations. (i.e., you are perhaps a Director?) The reason being
is that many key features of Net::FullAuto itself will be demonstrated and
discussed in the course of covering the installation procedure. These are very
important features, and they will contribute heavily to any decision you make
regarding the potential use of Net::FullAuto in your organization.
Net::FullAuto is written entirely in Perl. In some ways, it is unfortunate to have to
reveal the language that Net::FullAuto was actually written in. That is because of
the very human proclivity of “jumping to conclusions”. However, that is not to
give “jumping to conclusions” a bad name. We ALL do it – a lot – everyday. If we
didn’t, we’d all go insane in just a few hours. There are just too many people and
too much information competing for our attention. Successful, intelligent people
filter – there is no shame in that. Some readers of this White Paper, and one of
those readers might be you, will have a less than favorable opinion of Perl. Others
will be excited to learn that Net::FullAuto was written in Perl. (Your mileage may
vary). If you happen to be one of those not “sold” on Perl, The author humbly
asks that you allow him a few paragraphs to “make his case”, before deciding that
Net::FullAuto is not suitable for your organization.
21
Volume One: Introducing Net::FullAuto
©2011 brian.kelly@fullautosoftware.net
Perl is not widely recognized as an application platform, but rather as a
“scripting” language. Frankly, it is both. Majordomo is one of the most widely
used internet mailing list automation tools in the world today. A great many web
hosting services provide Majordomo to their customers. Majordomo is written
entirely in Perl. Yet one does not need to know how to code Perl itself to
successfully use Majordomo to automate mailing lists. One does not need to
know how to program C++ to use Microsoft Windows – which is written in C++.
Granted, the list of Perl applications (not requiring Perl knowledge to use) with
widespread use like Majordomo is not a long one, but that is more a result of
Perl’s reputation rather than its capability.
Perl was chosen for Net::FullAuto, not because of Perl’s reputation, but for its
capabilities. Bear in mind, that Net::FullAuto development began twelve years
ago, and at that time, Perl had fewer competitors than it has today.
John Napiorkowski is a software engineer in New York, who teaches Perl
programming:
As someone that spends a lot of time talking about Perl to people in management and people
from outside the Perl community, I've heard a lot of things and thought I could share what
I've heard, be it anecdotal or whatever.
Broadly there’s two groups I speak to Perl about: Management Types (Director level and
above) and Out Community Peers (programmers and systems folks that have heard about
Perl but are using Python, Ruby, etc.)
When I speak with management types, the thing I hear in terms of negative branding for
Perl is that 1) Perl is hard to hire for, 2) Perl is not really suitable for big application
development.
John Napiorkowski, Perl 5?, Perl 6?, Perl X?, http://jjnapiorkowski.typepad.com/modern-perl/2011/06/perl-5-perl-6-perlx.html#tp, 06/27/2011
Perl knowledge is not needed to successfully install Net::FullAuto on any platform
it supports – both Unix/Linux operating systems, as well as Microsoft Windows
systems (both desktop and server versions). Nor is Perl knowledge necessary to
use Net::FullAuto, though some familiarity with shell scripting and batch
processing is assumed, as indeed that is primarily the environment that is being
automated.
That said however, someone who knows Perl or is willing to learn it will be able to
make Net::FullAuto configuration even more powerful, and more capable of
22
Volume One: Introducing Net::FullAuto
©2011 brian.kelly@fullautosoftware.net
solving more complex automation problems. But again, the author implores the
reader to categorize Net::FullAuto properly – it is not a “Perl script”, but rather an
application that happens to be written in Perl. To underscore that distinction,
Net::FullAuto has its own executable: fullauto (or fullauto.exe on Windows). Perl
programmers are not, strictly speaking, needed, to successfully utilize
Net::FullAuto for Enterprise Automation.
Perl may not be the most suitable language for your organization’s large
application development for things like customer portals, statistical analysis,
inventory control, etc., but it was (and still is) suitable for solving the unique
problems encountered with Enterprise Automation goals. As mentioned earlier,
Net::FullAuto automation execution decisions are output based, not time
dependant. As such, there is a critical need for advanced output parsing
capability. Perl’s regular expression implementation is widely recognized as the
most powerful in the world. In fact, nearly all other regular expression
implementations in any language you care to examine, borrow heavily from the
Perl implementation. No other language in the world is better at text parsing.
Even Perl’s most vocal critics, are quick to acknowledge this dominance.
Another reason Perl was selected is the fact that it is an interpreted language. As
a result, Net::FullAuto configurations do not require compilation. This saves
development time, and allows for custom Perl code and CPAN modules to be
mixed with Net::FullAuto constructs – allowing for almost infinite capacity to
solve automation problems.
Finally, Perl was selected for its CPAN repository. CPAN stands for
“Comprehensive Perl Archive Network”. Essentially, it is a HUGE central
repository of third-party authored modules or libraries. Net::FullAuto itself
resides in the CPAN, and is freely obtained from it.
Experienced Perl programmers often comment that half of Perl's power is in the CPAN. It
has been called Perl's killer app.[5] Though the TeX typesetting language has an equivalent,
the CTAN (and in fact the CPAN's name is based on the CTAN), few languages have an
exhaustive central repository for libraries. (Emphasis added). The PHP language has PECL
and PEAR, Python has a PyPI (Python Package Index) repository, Ruby has RubyGems,
Lua has LuaRocks, … but none of these are as large as the CPAN. (Emphasis Added).
Other major languages, such as Java and C++, have nothing similar to the CPAN (though
for Java there is central Maven, and C++ has the Boost C++ Libraries).
23
Volume One: Introducing Net::FullAuto
©2011 brian.kelly@fullautosoftware.net
The CPAN has grown so large and comprehensive over the years that many people learning
Perl seem to elevate it to a sort of mythical status, and express surprise when they begin to
encounter topics for which a CPAN module doesn't exist already.
http://en.wikipedia.org/wiki/CPAN as modified on 29 October 2011 at 14:01
Enterprise Automation is predominantly an endeavor of connecting otherwise
disparate, incompatible, and isolated tasks that differ in location, data types,
operating systems, and syntax. Because of the rather daunting reality of the
“mission”, those determined to succeed in this difficult undertaking need the
most powerful tools available for the task. The CPAN provides these tools more
than any other language, or community existing today. End of discussion – no
one (seriously or credibly) disputes that. There are over 100,000 modules
available (and growing) that provide functionality for just about any computer
science problem that one is ever likely to encounter. Because Net::FullAuto was
written in Perl, it is fully compatible with the vast majority of these modules.
As asserted earlier, considering that the software is named “FullAuto” – it would
be quite disingenuous if the author didn’t make installation as “automatic” as
possible.
On Microsoft Windows, the Net::FullAuto installer installs everything. It is
actually easier to set up Net::FullAuto on Windows than it is on Unix/Linux6.
Simply download the self-extracting executable to the host you wish to equip:
http://www.fullautosoftware.net/Setup%20FullAuto%20v.90%20MSWin32-x86.exe
and double-click. You will see a ‘cmd’ window pop up:
6
See Appendix A for complete instructions on installing Net::FullAuto on Unix/Linux systems.
24
Volume One: Introducing Net::FullAuto
©2011 brian.kelly@fullautosoftware.net
Depending on many factors, it may take up to two or more minutes for you to see
activity. The first thing you will see is the Welcome Banner:
This will be followed in a just a few seconds with the license agreement screen.
25
Volume One: Introducing Net::FullAuto
©2011 brian.kelly@fullautosoftware.net
Net::FullAuto is licensed under the GNU Public License Version 3.
The GPL is the first copyleft license for general use, which means that derived works can
only be distributed under the same license terms. Under this philosophy, the GPL grants the
recipients of a computer program the rights of the free software definition and uses copyleft
to ensure the freedoms are preserved, even when the work is changed or added to.
http://en.wikipedia.org/wiki/GNU_General_Public_License as modified on 25 November 2011 at 21:19
You must agree to the license provisions in order to use Net::FullAuto. Type ‘1’
and hit ENTER.
The next screen will display Cygwin installation options. Recall that on Windows
systems, Cygwin is a requirement. (Cygwin is also licensed under GPLv3):
26
Volume One: Introducing Net::FullAuto
©2011 brian.kelly@fullautosoftware.net
This is an important screen. Choice one will automatically install Cygwin, Perl,
OpenSSH, BerkeleyDB and Net::FullAuto (along with other required components
too numerous to list here). Choice two will install only base Cygwin and
OpenSSH. Choice two is for equipping remote Windows hosts that Net::FullAuto
will interact with, but which will not contain a complete Net::FullAuto
installation. Remember, no Windows host is accessible to Net::FullAuto without
an OpenSSH SSH service/daemon running on it. (Other Windows SSH services,
are currently not supported – but may be in the future. This restriction does not
apply to Unix/Linux hosts. On them, any variety of SSH daemon will suffice.)
Type ‘1’ and hit ENTER.
27
Volume One: Introducing Net::FullAuto
©2011 brian.kelly@fullautosoftware.net
This screen asks you to select a text editor. Net::FullAuto has integrated editing
and is coded and configured using the editor you choose in this menu. The editor
itself is launched via the Net::FullAuto executable (Note – the following two
screens are not part of the installation, but seen after the installation during
normal user interaction with Net::FullAuto):
28
Volume One: Introducing Net::FullAuto
©2011 brian.kelly@fullautosoftware.net
The selected file will then open in the editor you are selecting now (below is the
same editor selection screen shown earlier):
Microsoft Wordpad is the default choice – indicated by the ‘star’ symbol (*).
‘Vim’ is a version of ‘vi’ which is the most ubiquitous editor in the Unix/Linux
world. Most IT professionals are familiar with it.
‘Joe’ is a good selection in that it can restrict the user to just the file opened. The
internal features of accessing the external OS environment can be easily disabled
29
Volume One: Introducing Net::FullAuto
©2011 brian.kelly@fullautosoftware.net
with this editor. This is a potentially important security consideration. (There is
more on Net::FullAuto security in a following chapter.)
If ‘TextPad’ is already installed on the Windows host, it will be displayed in the
choice screen – otherwise it will not be available (the other editors will be
automatically installed, and Wordpad is available on all Windows systems).
Other editors can be utilized, but will need to be manually added to the
configuration after Net::FullAuto is fully installed. So select one of the above in
the meantime, and continue.
If you are installing on Windows XP, you will see this screen:
30
Volume One: Introducing Net::FullAuto
©2011 brian.kelly@fullautosoftware.net
Are you beginning to see a pattern? What you are witnessing is how Net::FullAuto
can be coded and configured to provide wizard-like workflows that confine users
to pre-defined criteria sets. With Net::FullAuto, non-technical users can navigate
these workflows themselves, without technical assistance, and with FULL
security because the menu-driven workflow only gives them options they are
authorized to select from. Yes, Net::FullAuto’s primary mission is to encapsulate
and run process workflows in an unattended fashion. However, such workflows
need to be created, configured, altered, customized and modified – either
frequently or infrequently. Net::FullAuto’s integrated and fully customizable menu
system makes ALL these tasks, EASY, PRECISE and SECURE.
Continuing with our installation ‘task’ – this will be the next screen:
31
Volume One: Introducing Net::FullAuto
©2011 brian.kelly@fullautosoftware.net
Make a choice and continue:
Note how the whole workflow is essentially self-documenting. All the instructions
needed are included right in the workflow. No need for external documentation!
32
Volume One: Introducing Net::FullAuto
©2011 brian.kelly@fullautosoftware.net
And we’re off! In most cases the install will complete without any additional user
input needed:
33
Volume One: Introducing Net::FullAuto
©2011 brian.kelly@fullautosoftware.net
Please be patient, the Net::FullAuto installation may run for up to an hour (or
even more).
In the majority of cases, that is all there is to it! See how easy that was? With
Net::FullAuto, any distributed process in your organization can be just as easily
contained, precisely routed, and powerfully controlled the way the Net::FullAuto
installation was.
Now that we have Net::FullAuto installed and ready to go, let’s use it!
34
Volume One: Introducing Net::FullAuto
©2011 brian.kelly@fullautosoftware.net
BACK TO BASICS
Once upon a time, on a Dumb Terminal far, far away … all user-system
interaction was text based. Ironically, in the mainframe world, this paradigm
never really changed. Today, using 3270 emulation software, mainframe users
still interact with mainframe functionality the same way they always have – for
more than 40 years! If such an approach to user-machine communication was
really so archaic – even barbaric – then why has it persisted for so long?
The reason is simple – efficiency. GUIs – Graphical User Interfaces – are not a
mainframe or even Unix innovation, but personal computer innovation. The GUI
most are familiar with emerged with the first Apple Macintosh computer, and
Microsoft Windows was essentially a copy of that innovation for PCs. GUI’s are
important for particular user communities, and the Apple Macintosh became the
low cost entry portal to the digital world for graphic artists. Today – nearly 30
years later, graphic artists still prefer Apple workstations. But ironically,
mainframe users and Unix users have continued to cling tenaciously to their text
based command windows. Ask them why, and the answer is always the same –
efficiency.
Enterprise Automation is really all about increased efficiency. The point is not to
linger with configuration tasks, the way one might with an online game or news
article (or how a graphic artist might with a creative task) – but to get them done
quickly and precisely. It’s not about the “user experience.” In fact, it is the
contrary: Enterprise Automation is all about extracting the “user” from the
“experience” – and letting the systems do it ALL; to the greatest extent possible.
With that approach in mind, Net::FullAuto was created to maximize efficiency –
which means it emphasizes direct text based interaction over GUI based
interaction. Does any manager really want their systems personnel “browsing”
during working hours? Then why is there a perception among the Management
community that a serious application has to have a fancy, glitzy, browser-based
GUI interface???
Mainframe computers are not personal computers. In fact “personal” use would
be a firing violation in most organizations. Mainframe users are not really “users”,
but process automation “architects” – for mainframe computing is really about
PROCESSING – and nothing more. The goal of the mainframe user is to get the
systems to do more and more and MORE, the user – less and less and LESS.
35
Volume One: Introducing Net::FullAuto
©2011 brian.kelly@fullautosoftware.net
There is a desire – almost a wistful kind of longing – for the same kind of
paradigm to be practical in the distributed system world (the world of PC’s and
Unix systems). To date, that goal has remained largely a dream, for reasons it
would take more space than is available here to explore. One of the biggest
reasons was discussed earlier – ill-behaved connections, and the inability of
automation software to cope with “un-expected” data. But another reason this
goal may have been unobtainable, may in some ways be the very fault the of
Management community that simultaneously wishes for ever greater efficiency
and ever greater automation: too much insistence on the availability of a GUI
interface. This “demand” results in software companies spending more time (and
money) focusing on the “user experience” than on the actual automation problem
that is supposed to be raison d’être of their efforts! Net::FullAuto solves that
problem by being FREE. Unencumbered by over-the-shoulder directives from
those whose primary goal is making money, the author was free to pursue his
vision, and take as much time as was needed to solve the technical hurdles
encountered – which were so many in number, that there simply was no time to
develop a GUI. (That is not to say that one won’t be developed later.) No software
company in the world is going to wait twelve years for a deliverable (the time it
took the author to complete Version 1.0 of Net::FullAuto, and that was without
the “non-negotiable requirement” for an accompanying GUI interface!).
Net::FullAuto configuration does not use a GUI – it uses the command window.
Why the command window? Because it is the one interface that both Unix and
Windows systems have in common. It is also the one interface that enables direct,
unencumbered access to remote hosts. If one wishes to obtain mainframe
efficiency in the distributed world, maybe it’s time to get back to basics. Maybe
it’s time to get back to the command window.
But getting back to the command window, does not mean making things
“difficult”, and requiring the user to know more than they need to if “only” a GUI
was available. You saw in the previous chapter how using Net::FullAuto menus
could take ALL the guesswork out of a workflow. And that is the goal. Confused
users are of course, not efficient users. Net::FullAuto is designed from the ground
up (so to speak) to funnel user interaction to the precise information and decision
sets they need – no less, and just as important: NO MORE (which is discussed
more in the next chapter on Net::FullAuto security). A GUI is just not needed in
this time-tested approach to efficient user-system interaction. In fact, it more
36
Volume One: Introducing Net::FullAuto
©2011 brian.kelly@fullautosoftware.net
often than not gets in the way, and produces even greater bottlenecks, confusion
and inefficiency7.
Now – on with the SHOW!
Let’s open the venerable command window (We’ll use the Cygwin Terminal
supplied with Cygwin, started via the following shortcut on your Windows
desktop or Start Menu):
You should see something similar to the screen below (though username and
hostname will of course differ):
We’ll begin with a short demo. This will demonstrate if Net::FullAuto was setup
successfully:
7
Again, lest ye complain that the author doth protest too much, it is quite true that a good DASHBOARD is extremely
desirable for Enterprise system monitoring. Indeed, full featured GUI Dashboards are in the plans for future
development efforts. Don’t cancel your monitoring software license renewal just yet …
37
Volume One: Introducing Net::FullAuto
©2011 brian.kelly@fullautosoftware.net
Type fullauto and hit ENTER.
You will see the following screen:
Type in a new username here -OR- press ENTER to keep the current one
(indicated between the two arrow characters). You will then see a password
prompt:
38
Volume One: Introducing Net::FullAuto
©2011 brian.kelly@fullautosoftware.net
Type in your login password (the same one you log in to the system with), and
press ENTER. (The password will not be displayed on the screen).
You will then see the following message: --> Logging into localhost via ssh . . .
39
Volume One: Introducing Net::FullAuto
©2011 brian.kelly@fullautosoftware.net
Assuming the installation went as planned you should next see the following
menu:
Type ‘1’ to select the ‘HELLO WORLD TEST’:
If you see the screen above – CONGRATULATIONS! You have successfully run
your first Net::FullAuto custom code block - your first job! (Wasn’t that easy!)
40
Volume One: Introducing Net::FullAuto
©2011 brian.kelly@fullautosoftware.net
Of course, the point of Net::FullAuto is to run the job unattended. To enable this
to occur in a way that any scheduler can use, we need to automate the login,
password and menu steps – and go straight to “HELLO WORLD”.
The name of the custom code block in the fa_demo set fa_code.pm file is hello_world
and the code block looks like this:
sub hello_world {
print “\n
}
Net::FullAuto says \”HELLO WORLD\”\n”;
8
To set up this job to run in a scheduler, we simply need to feed the fullauto
executable the proper arguments:
Hey! Where is the password?
That’s right – no clear text passwords on the command line! Passwords on the
command line are a VERY bad security problem. Yet, this goes on everyday all
over the world – in thousands of organizations. Instead, with Net::FullAuto, the
8
Isn’t that Perl? Wasn’t it said the Perl programming knowledge was not necessary? Well – yes, it is Perl. However, if
the user follows the Net::FullAuto documentation carefully, then that user will have no need to consult outside Perl
resources. No previous Perl experience is necessary (but of course, it is helpful).
41
Volume One: Introducing Net::FullAuto
©2011 brian.kelly@fullautosoftware.net
user setting up the job must always do a ‘dry run’ on the job before adding it to a
scheduler.
So go ahead and press ENTER:
Note how you are no longer prompted for a login id, but you are prompted for a
password. Go ahead and type in the password and see what happens next:
42
Volume One: Introducing Net::FullAuto
©2011 brian.kelly@fullautosoftware.net
Surprised? Net::FullAuto is built to be SECURE. Known security vulnerabilities do
not plague Net::FullAuto processes. They truly are secure. Let’s set the expiration
for one week. So type ‘5’ and press ENTER:
Note how there are 53 Total Choices available – there are 53 weeks in a year, of
course. To view the next set of 10 choices, just press the DOWN ARROW to scroll:
43
Volume One: Introducing Net::FullAuto
©2011 brian.kelly@fullautosoftware.net
Of course one can always go backward by pressing the UP ARROW:
Also, we can simply change our mind, and go back to the previous menu. Press
the LEFT ARROW key:
Note the dash ‘-‘ next to the number ‘5’. That’s tells us we’ve returned from a
child menu screen. Let’s choose ‘FULL CALENDAR’ – type ‘1’:
44
Volume One: Introducing Net::FullAuto
©2011 brian.kelly@fullautosoftware.net
Type ‘1’ to select 2011. Press ENTER:
Type ‘1’ to choose November. Press ENTER:
Notice how the menu choices build off of the previous selections. With this
capability, organizations can dynamically build PRECISE data selection sets, so
that users see only the data that is relevant to the decision they are expected to
45
Volume One: Introducing Net::FullAuto
©2011 brian.kelly@fullautosoftware.net
make. In this example, notice how only two months were available for selection –
derived from the chosen year (2011) and date of execution (November 26th in our
example). The next menu should show only the dates left in November – and it
does:
Choose ‘1’ again to select today’s date:
46
Volume One: Introducing Net::FullAuto
©2011 brian.kelly@fullautosoftware.net
Note how the current time is displayed in the banner. All items of the menu are
dynamically configurable.
Choose ‘1’ to Show Minutes:
Hit the DOWN ARROW key a couple of times:
47
Volume One: Introducing Net::FullAuto
©2011 brian.kelly@fullautosoftware.net
And we will choose 10:50pm – Number 29, for our expiration time. Type ‘29’ and
press ENTER:
We now see our ‘HELLO WORLD’ at the end of the output. So now, we have a
saved password, enabling our process to be added to a scheduler. But consider –
and this is important: because the password is saved in Net::FullAuto internal
encryption, a process command string can be passed on to other scheduling
personal without a cleartext password! Fewer people are privy to it. How great is
that!
Let’s run the process one more time. In fact, now is a good time to point out that
the fullauto executable has a shortcut – fa (so there can be even less typing):
48
Volume One: Introducing Net::FullAuto
©2011 brian.kelly@fullautosoftware.net
So let’s review. With just a simple ‘HELLO WORLD’ example, the reader got a
glimpse of just how much information can go into setting up an automated
scheduled job – and how Net::FullAuto makes that simple, PRECISE, and
SECURE. Net::FullAuto integrated menus enable complex decisions to be broken
down into very small decision units that dynamically build with each selection.
Now granted, a simple GUI calendar app is arguably “better” than the granular
selection process used here. However, consider the first menu gave other
selection options:
49
Volume One: Introducing Net::FullAuto
©2011 brian.kelly@fullautosoftware.net
If the user were to select a different expiration choice method – like Number of
WEEKS, the whole task would have been completed in just two lightning fast
selections: arguably MUCH faster and efficient than a mouse picking a date from
a GUI calendar app.
But consider further that choosing a date is just one of infinite possible uses for
this kind of dynamic menu. The data can be anything – and the number of
selections is limited only by the amount of memory in the operating system. And
unlike GUI interfaces, this menu, and custom code blocks, can be set up and
configured almost as fast as processes can be defined. The author remembers
numerous times working with business customers as they attempted to
communicate the details of the process they wanted automated. In the time it
took them to describe and outline the steps, the author was able to code the
entire process. Imagine their considerable SHOCK when the moment they said
“that’s it” – the author replied “me too”. The entire process was AUTOMATED.
Finally, note that we logged into the localhost via SSH in the example:
50
Volume One: Introducing Net::FullAuto
©2011 brian.kelly@fullautosoftware.net
Just as easily we can log into remote systems. We simply need login credentials
and an IP address or hostname. A more detailed example and walk-through is
available in the appendix.
The author hopes that you are beginning to grasp the POWER and capability of
Net::FullAuto. It truly is a unique tool. And its potential is just beginning to be
tapped. It’s amazing what can be accomplished when you get back to basics.
51
Volume One: Introducing Net::FullAuto
©2011 brian.kelly@fullautosoftware.net
NEED TO KNOW BASIS
There is a joke the author likes to tell – that he is the owner of a multi-billion
dollar hedge fund. The only problem is, it is so ‘secure’, that even he doesn’t know
where it is!
Here’s another: “What is the best way to secure the internet?”
Answer: SHUT IT DOWN.
Those jokes are good for a laugh, but they also convey very real truth. There is a
tradeoff between strong security, and digital flexibility. The more secure a system
is, the more difficult it is to do anything on it.
Organizational security doesn’t end at just stopping hackers and unauthorized
intruders. There is also a real obsession with limiting the access of all staff
members to just the information they need to do their job. The problem is the
information changes. The result being that access privileges are almost
continuously being reevaluated; granted and revoked. This is a huge productivity
barrier, not to mention a security concern in its own right, as deadline demands
constantly push IT personnel to compromise security policies. The author has
experienced this “soap opera” too many times to recall. He asks for access to a
system, and is quickly denied as it is “against policy”. He informs his manager,
who calls a meeting with the security team to “explore options”. A week later
(when everybody has room on their calendar), the meeting commences, and an
hour of “brain-storming” takes place. When the allotted time is nearly up, and
folks are gathering in the hallway for the next meeting, and no viable, realistic
options have been presented, the author’s manager aims for the target: “just give
my guy temporary access” – everybody nods, and 20 minutes later the author has
the access he needs. TOO MUCH access in fact. He’s given access to the ENTIRE
SYSTEM when in fact he only needed access to a particular file system directory.
On top of that, “temporary” was never defined, and the reality was, he in fact had
indefinite (often permanent – i.e., for the entire length of his employment) access.
Question: “WHERE IS THE SECURITY IN THAT STORY?” Consider all the time
and money wasted with as many as five or more people in a meeting, a week after
it was called, wringing their hands over whether or not to give one individual
“access” to a resource he needed – and after all that wasted time and effort, end
up giving him TOO MUCH.
52
Volume One: Introducing Net::FullAuto
©2011 brian.kelly@fullautosoftware.net
Sounds like a Dilbert comic strip, but be assured, this happens all the time.
Could Net::FullAuto have changed that outcome?
ABSOLUTELY!
Let’s define the problem a little more concisely. There is an ongoing need in most
organizations to discover at any given time, what the differences are between a
file in the development domain, and one in the production domain. What is
needed is a utility that will allow the development staff to discover these
differences – without access to the production server, and just as importantly,
with no ability to replace or alter the file in production. In most organizations,
what has to happen is the developer contacts the “server team” and asks them to
retrieve a copy of the file and send it them via email. This is such a bad practice
from a security standpoint, that it would take too much space to break it all
down. Yet, this happens every day all over the world. And we haven’t even
mentioned the time needed to contact the server team (usually by ticket), and the
time used by the server team member to fulfill the request and close the ticket –
which may even result in a charge from the server team to the development team!
Also consider the delay cost of having to wait – perhaps hours – for the server
team to ultimately deliver the file.
Now, let’s see how Net::FullAuto could change this. Ultimately, what we would
like is a desktop icon that when double-clicked, gives us a menu where we can
choose the file we want compared, and then have a diff file “arrive” in a directory
(also on the desktop). For this exercise we will use an intermediate system that
fulfills the role of Net::FullAuto server. Use of a Net::FullAuto server will make
setting up automated processes easier and more secure. For this exercise, we will
use a Windows XP laptop running Cygwin (of course) and two virtual Unix servers
running inside of Oracle’s Virtual Box software running on the same laptop. One
an Ubuntu Linux host, and the other an Open Solaris (SunOS) host:
53
Volume One: Introducing Net::FullAuto
©2011 brian.kelly@fullautosoftware.net
We will designate the Ubuntu host as the Net::FullAuto server (FA host), and the
Open Solaris host as our “Production” host (PROD host) and our Cygwin
equipped laptop as the “Development host” (DEV host). Let’s say we wish to
compare the fa_code.pm file on the DEV host with the file of the same name on the
PROD host.
54
Volume One: Introducing Net::FullAuto
©2011 brian.kelly@fullautosoftware.net
The Unix diff utility, requires that files being compared co-exist on the same host.
Since this requirement cannot be altered, the approach will be to bring the two
files together on a secure, intermediate host – i.e., the Net::FullAuto server.
The process begins with double-clicking an icon on the laptop (Windows/Cygwin)
host:
The icon actually starts the Cygwin Terminal program (C:\cygwin\bin\mintty.exe)
configured to run the Net::FullAuto custom diff code on the Net::FullAuto server.
Right click on icon to get Properties window.
55
Volume One: Introducing Net::FullAuto
©2011 brian.kelly@fullautosoftware.net
The actual text in the Target: box of the Compare fa_code.pm Properties Window is:
C:\cygwin\bin\mintty.exe -i /Cygwin-Terminal.ico -e /cygdrive/c/cygwin/bin/bash -l -c "/bin/ssh -p 2222
ubuntu@localhost /usr/local/bin/fa -l --login ubuntu --password --code compare_fa_code"
It looks like a lot (and it is) but it’s not difficult to decipher. From left to right:
C:\cygwin\bin\mintty.exe
->
The Cygwin Terminal Program
-i /Cygwin-Terminal.ico
->
Icon argument (icon that appears in top left of
window that appears when shortcut is clicked)
-e /cygdrive/c/cygwin/bin/bash ->
Argument for minty.exe that invokes the local
Cygwin bash shell program (bash.exe) [ which
in turn calls the local ssh program (ssh.exe) ]. Ultimately,
four separate programs – three on the local host
and one on the remote – are working in a left to
right chained fashion:
mintty.exe -> bash -> ssh  fa (FullAuto on remote host)
-l
->
Argument for bash (.exe extension is optional) that tells it
to act as if it had been invoked by a login shell
-c "/bin/ssh
->
Another argument for bash that instructs it to
launch the local (included with Cygwin) ssh program
-p 2222
->
Argument for ssh that instructs it to connect to the
remote host with port 2222. In most cases this
argument is not needed – it is used here because
this example uses Oracle Virtual Box hosts
ubuntu@localhost
->
Argument for ssh that instructs it to login to
localhost (with port 2222 as indicated with the previous -p argument)
with the login id ubuntu. (Normally localhost would not be the
“remote” target – but in this case the remote host is “Virtual”)
/usr/local/bin/fa
->
Argument for the remote shell (whatever it is) sent via
the ssh connection to invoke Net::FullAuto (fa)
[finally!]
-l
->
Argument for fa instructing it log output to a
default (pre-configured) file on the remote host
56
Volume One: Introducing Net::FullAuto
©2011 brian.kelly@fullautosoftware.net
--login ubuntu
->
Argument for fa instructing it login into the
now localhost (from the perspective of fa that is) using the
login id ubuntu (the host is in fact the virtual Ubuntu host running in
Oracle Virtual Box )
--password
->
Argument for fa instructing it to use the
previously saved password on the Ubuntu host.
Note: there is no password actually following this
argument! (Which would be very insecure!)
--code compare_fa_code"
->
Argument for
instructing it to use the
compare_fa_code custom code block pre-written and
located within the Net::FullAuto installation on the
Ubuntu host. (Note the closing quote – the opening
quote was in front of the ssh argument shown
earlier. The quotes contain one big argument being
fed to the local bash program via the –c bash
option)
fa
OK – that indeed was a lot! However, if you could follow along and grasp what’s
going on, then you will have the understanding to create a truly infinite number
of variations on this theme. Process automation essentially means combining
multiple hosts and programs together into ONE operation – with everything
working together like a symphony; each element playing its part in
synchronization with all the other “pieces”. So far we’re seeing how ONE process
is being set up using (so far) four programs and three hosts. Yes – it’s a lot of
complexity, but then so is a symphony. Aficionados understand – if you want a
sublime experience, there is a price to be paid. Just as every celebrated orchestra
needs a world-class conductor – a powerful and robust automated process needs
one as well. That conductor is Net::FullAuto – but it still needs YOU to fill in the
rest of the “orchestra” (other programs), and provide the scores (custom code) to
be played.
Continuing the “orchestra” analogy – most of the work is in the initial creation of
the symphony. Once created, a symphony can exist almost indefinitely. Individual
pieces (players) come and go, but the “orchestra” itself persists. The same can be
said of a Net::FullAuto server setup for enterprise process automation. Once the
57
Volume One: Introducing Net::FullAuto
©2011 brian.kelly@fullautosoftware.net
initial setup is completed – adding and changing it’s “pieces” is relatively easy.
And just as a world-class orchestra can “play” just about any score on the planet,
a Net::FullAuto setup can automate just about ANYTHING.
Continuing with our example, we’ll examine how this diff process actually works.
It’s as simple as double-clicking the icon and waiting for a window with a
password prompt:
The program asking for the password is the localhost based ssh program we saw
listed above. This is the only input the user needs to worry about – the rest of the
operation runs fully automated.
58
Volume One: Introducing Net::FullAuto
©2011 brian.kelly@fullautosoftware.net
59
Volume One: Introducing Net::FullAuto
©2011 brian.kelly@fullautosoftware.net