Functional Test Plan
Page 1
3/9/2016
0 Reviewers
This document is a rough draft of the functional test plan being written for the NovaNET scaling and
migration project. By default, it displays in the ‘Outline View’. You can change this by going to the
‘View’ menu and selecting any other layout you might be more comfortable with. ‘Outline View’ is
handy because you can minimize portions of the document, easily drag and drop portions of text to
different areas of the document, build a table of contents, etc.
Please insert any review comments using the ‘Review’ style built into this document. You can find it in
the ‘Styles and Formatting’ toolbar in the ‘style’ dropdown menu or in the ‘Styles and Formatting’ task
pane. If you don’t know where the ‘Styles and Formatting’ task pane is, go to the ‘Format’ menu and
select ‘Styles and Formatting’. Pretty simple.
You can also use Word’s built-in commenting system if you’re familiar with that.
This draft of the document includes input from John Hegarty, Ray Thomsen, Frank L’Odense, Mark
Webb, Charlie Gschwind, Steve Litt, Renee Inman, and Ross Bentley.
1 Introduction
Sometime in 2006, an initiative was started to horizontally scale the NovaNET environment. After
realizing the costs and difficulties of such a project, it was decided to horizontally scale the NovaNET
environment. That project evolved into a migration project wherein a new production system would be
built and the former production environment would become the disaster recovery environment. TO DO:
Find out the real background information
1.1 Background
As part of a migration/scaling project for NovaNET, a functional test plan became necessary in order to
document a testing procedure for new and updated configurations of NovaNET.
1.2 Purpose
The purpose of this test plan is to outline the test activities that will be used during the
migration/scaling of NovaNET. As new NovaNET systems are built and configured for migration, scaling,
and disaster recovery, a formalized process documenting the expertise of NovaNET employees becomes
necessary. Furthermore, in efforts to upgrade individual components and/or subsystems within
currently-existing NovaNET systems, a testing procedure should be documented in order to ensure
compatibility of replacement components/subsystems.
1.3 Scope
The testing will be limited to those test cases that can be used to verify system components and
features have been integrated correctly by demonstrating their successful execution.
1.4 References
Expertise in various areas was referenced in the creation of this document. The overall structure of this
document is derived from meetings involving Julie Hoban, Ross Bentley, Renee Inman, Keith
Harrington, Marshall Moritomo, and Anthony Garone. Much of the text has been taken directly from
documentation written by Paul Christensen, Joey Charneski, Tom England, John Hegarty, and Ray
Thomsen. Areas of expertise with respect to this document’s structure and performance of tests are as
follows.
Developing Draft : Anthony Garone
Functional Test Plan





3/9/2016
NovaNET Central Processing Subsystem
o
System Administration: Steve Litt and Frank L’Odense
o
NovaNET Software Operation and Administration: John Hegarty
NovaNET Leased-Line Subsystem
o

Page 2
Ray Thomsen
NovaNET DMZ Customer-Facing Subsystem
o
Hardware and Infrastructure: Ray Thomsen
o
Software and Configuration: John Hegarty and Ray Thomsen
NovaNET NOW Subsystem
o
Oracle Database Administration: Laryssa Kachorowsky
o
MySQL Database Administration: Mark Webb
o
Software Configuration/Administration: John Hegarty
Backup and Recovery
o
NovaNET-specific Scripting: John Hegarty
o
Frank L’Odense
End-User Testing
o
Ross Bentley and Renee Inman
Much of the content was derived from the various forms of documentation written by Paul Christensen,
Tom England, Ray Thomsen, Renee Inman, and others.
1.5 Definitions
Since NovaNET has been in existence for nearly 50 years, there are many acronyms and functions
inherent to the system that may not be immediately understood. The NovaNET environment has been
broken down into the following subsystems: Central Processing Subsystem, Leased-Line Subsystem, DMZ
Customer-Facing Subsystem, and NOW Subsystem.
This section provides a brief, high-level description of each subsystem. For more detailed information,
please read relevant sections of this document.
NovaNET is a terminal-based application. Clients use an application called Portal to connect to the
NovaNET environment. As such, the subsystems operate to perform fundamental tasks:

Central Processing Subsystem
o
All client operations are handled through this subsystem. Without it, there is no
NovaNET.
Developing Draft : Anthony Garone
Functional Test Plan

All clients connecting to NovaNET using serial data connections utilize this subsystem.
DMZ Customer-Facing Subsystem
o

3/9/2016
Leased-Line Subsystem
o

Page 3
All clients connecting to NovaNET using modern network equipment and protocols
connect through the DMZ. Internet connectivity is also provided for users connected via
serial data lines.
NOW Subsystem
o
Handles dynamic user data and Instructional Management System software.
1.5.1 Production Component List
The following is a list of primary NovaNET components that should be able to communicate with one
another in a functional production system [Production machine names in brackets]:




Central Processing Subsystem
o
Central AlphaServer [Mesa (production server), Esker (development server)]
o
Central Server SAN [EMC CX400 SAN]
Leased-Line Subsystem
o
Atlas DS-3 Multiplexer
o
Adtran 1241 Channel Banks (4)
o
Link Units (6)
o
Network Interface Units (2) [NIU1, NIU2]
DMZ Customer-Facing Subsystem
o
Ethernet Link Access Units (2) [Wilmington, Pilot]
o
Internet Network Interface Units (2) [INIU4, INIU5]
o
Stargate Server
o
Lynx Server
NOW Subsystem
o
NOW Servers (7) [Amy, Basil, Clara, Desmond, Ernest, Fanny, George]
o
Curriculum Server [Horus]
o
Catalog Application Server [Osiris]
o
Oracle Database Multiplexers (2) [Chmpgappmplex1, Chmpgappmplex2]
Developing Draft : Anthony Garone
Functional Test Plan
Page 4
o
Oracle Database Servers (2) [Cdc-480-1, Cdc-480-2]
o
Oracle Database SAN [EMC CX400 SAN]
o
MySQL Database Multiplexer [Honey]
o
MySQL Database Server [Chmpgdbsmysql]
3/9/2016
Throughout this document, we refer to some operating systems as “Linux-based.” In our current
production environment, our Linux-based operating systems are Red Hat Enterprise Linux and Fedora
distributions. Although not within the scope of this scaling and migration project, we are making an
effort to standardize our Linux-based systems to run the latest supported versions of Red Hat
Enterprise Linux.
1.5.2 NovaNET Acronyms
The following is a list of acronyms affiliated with NovaNET:

DMZ: De-Militarized Zone

ELAU: Ethernet Link Access Unit

EMP: Embedded Monitor Protocol

IMS: Instructional Management System

INIU: Internet Network Interface Unit

LAU: Link Access Unit

LSM: Logical Storage Management

LU: Link Unit

NIU: Network Interface Unit

NN: NovaNET

SAN: Storage Area Network

SU: Site Unit
1.5.3 CNet: NovaNET’s Proprietary Communications Protocol
Most components in the NovaNET environment communicate through a proprietary communications
protocol called CNet. CNet was developed before the Transmission Control Protocol (TCP) was
invented. CNET uses another proprietary protocol, X-51, to perform error-checking and recovery.
The primary components of CNET are Nodes and Systems. A CNET System is a particular machine that
can communicate over CNET. CNET Nodes are “sockets” via which a process can send or receive CNET
messages. Each CNET system has a process called the Network Processor that manages the creation and
deletion of nodes, and processes CNET traffic being transmitted by, received on, or passed through its
CNET system. CNET systems were once manually loaded but are now self-contained and self-booting.
CNET issues can be diagnosed using the ‘netmon’ lesson within NovaNET, but are usually fixed by
reloading/rebooting a component.
Developing Draft : Anthony Garone
Functional Test Plan
Page 5
3/9/2016
With the introduction of satellite telecommunications, NovaNET was able to broadcast across North
America using asymmetric bandwidth. The X-51 Protocol was developed to optimize this
communication method both for error detection and recovery.
More information on CNET can be found in the Ops Manual
[http://medusa.nn.com/opsmanual/nova/cnetview.html] or in the CNET documentation manual.
1.6 Test Strategy
The strategy for a functional test plan is to verify connectivity and functionality of key NovaNET
components and subsystems (both hardware and software). In order to thoroughly test the NovaNET
application, we must perform hardware tests, basic software tests (using NovaNET’s built-in “author
mode”), and end-user tests (using real NovaNET user data). Using this ground-up approach, we can
begin by eliminating telecommunication, connectivity, and hardware issues from the start.
Since NovaNET is a very complicated application using multiple communication protocols and hardware
(and production dates spanning decades of time), it is best to start with basic, non-NovaNET tests.
Once hardware and connectivity are verified, we verify that the basic functionality of the application is
in order using the functional hardware and communication systems. Running NovaNET in author mode
allows us to test core components of the application before we run into potential data issues. Once
author mode functionality is verified, we test NovaNET using real-world application data to ensure
interoperability between NovaNET and the database systems.
2 Subsystem Readiness
The following hardware and connectivity tests are performed as readiness checks for NovaNET
performance. Before we can approve a subsystem as being ready for NovaNET functionality, we must
verify the functionality of hardware (and operating systems) as well as the connectivity within and
between disparate subsystems. Most of these tests avoid testing actual NovaNET software.
2.1 NovaNET Central Processing Subsystem
The core of the NovaNET system, which provides the central processing/control functions, is an HP
AlphaServer running Tru64 Unix. The AlphaServer has its own disk system (SAN) on which are stored the
text-based lessons of the NovaNET system (among other things). The Alpha and SAN are the two
hardware components we need to test within this subsystem.
The NovaNET Central Processing Subsystem performs the following functions:

Authenticates and keeps track of all active users; since NovaNET is a timesharing system
handling thousands of simultaneous users, this is a very important function.

Connects users to the Instructional Management Systems.

Stores thousands of text-based student lessons.

At the request of users, runs those learning lessons that are text-based, which are stored and
run on the central system and displayed on the user’s workstation.

Manages the storage and retrieval of student data.

Keeps track of what users are using what lessons and builds a set of monthly usage statistics
and automatically transfers this data to control and support systems.
Developing Draft : Anthony Garone
Functional Test Plan
Page 6
3/9/2016

Provides user communication tools that allow for communication both in real-time, by sharing
of user experience with an instructor or support person, and via forms of email (called Personal
Notes) and discussion groups (called Notes). Some lessons take advantage of this model to
allow students to interact within the content across locations.

Connects users seamlessly to the Internet when user activity requires Internet resources
(through the Lynx text-based web browser).

Stores and runs NovaNET system management utilities.
As the central processing/control subsystem, this subsystem communicates with all of the other
NovaNET subsystems.
2.1.1 Hardware and Software Configuration
2.1.1.1 Alpha
As part of the scaling and migration project, there are multiple AlphaServers we can describe. In our
current production environment, the central server is called Mesa and is a Compaq ES40 AlphaServer
with 4 CPUs. This server is called Mesa (mesa.nn.com). We have another developmental AlphaServer
called Esker (esker.nn.com). According to the scaling and migration project timeline, Esker will be
replaced by a new HP AlphaServer (described in further paragraphs in this section) in order to test its
functionality. This new AlphaServer will eventually replace Mesa as the production AlphaServer.
The new HP Alpha server is a model ES80 with 6GB RAM and 6 1.15Ghz processors. The ES80 consists of
3 separate cabinets, called 2P drawers, each of which contains 2 CPUs and 2 GB of RAM. The cabinets
are mounted together in the rack and are connected via the white IP cables. Each cabinet has its own
set of 2 power cables and a CAT 6 cable that connects to a dedicated router (see later in this section
for more information about the router. The figure below shows the cabling for a single 2P drawer.
Figure: 2P Drawer Rear Cabling
Developing Draft : Anthony Garone
Functional Test Plan
Page 7
3/9/2016
Each of the power cables plugs into one of 2 Power Distribution Units (PDU). Each PDU is split in 2 and
has two cables for a total of 4 electrical connections under the floor. Each cable of each 2P drawer
should be plugged into a different PDU to ensure redundancy should a circuit fail.
Below is a view of the front panel of each of the 2P drawers as well as a graphic describing the
meaning of each of the LEDs and their combinations. When the ES80 is functioning properly, the amber
LED should be off and the green LED on.
Figure: Status LEDs on the 2P Drawer Front
Each 2P is connected to a LAN router thereby creating a private interconnection used for management
and intercommunication between the 2Ps.
The benefit (and primary justification for the purchase) of this server is to allow NovaNET to handle
thousands more simultaneous connections. While there will be other benefits for the new Alpha,
running more instances of NovaNET Executor software to handle these connections is the primary
benefit.
2.1.1.2 SAN
The primary data and backup areas for the NovaNET Central server is stored on an EMC CLARiiON CX400
SAN. The connectivity is made via two fiber channel host based adaptors (HBAs) located in the I/O slots
in the rear of the server. Each of the HBAs are connected to one of two fiber channel switches. 2 HBAs
are used for redundancy in the event of an HBA, cable, or fiber switch failure. The green LED on the
HBA should be on and the amber LED should be flashing to indicate proper connectivity to the switch.
There are two storage processors (SP) on the SAN. Each SP is connected to one of two fiber channel
switches. This ensures redundancy back to the host in the event of a failure of an SP, cables, or
switches.
There are 2 logical disks presented to the ES80 by the SAN. Each logical disk is 200GB and is in a RAID5
configuration to provide redundancy in the event of a disk failure.
2.1.2 Tests
Tests for validating the Central Processing Subsystem can be found in section 5.1.1 on page 23.
Developing Draft : Anthony Garone
Functional Test Plan
Page 8
3/9/2016
2.2 NovaNET NOW Subsystem
The NovaNET NOW Instructional Management System (IMS):

Provides a catalog of courseware organized by subject with details about the courses available
on NovaNET

Provides pre-built curricula based on national and state standards

Provides tools for instructors to build their own curricula out of NovaNET courseware

Allows the instructor to assign lessons/activities to a student or groups of students, manage
their progress, and report on the results

Guides students through the lessons/activities that have been assigned to them

Stores the results of the students’ performance
While NovaNET has only one course catalog that spans all of NovaNET, there are two IMSs currently in
use:

NovaNET NOW, which is the default IMS and is used by nearly all customers

C-Router, which is an older IMS that is still in use for some of its customization abilities.
The C-Router IMS executes entirely on the central server. All of its data is stored on the Mesa file
system.
The NOW IMS runs on multiple machines:

Osiris: a catalog application machine. It executes the NovaNET online catalog application that
contains user accessible information about NovaNET lessons, resources, and utilities. The
catalog is built on an instance of MySQL.

Horus: a curriculum delivery machine. It runs the curriculum service application for the NOW
IMS; handles relatively static curriculum data, such as titles and descriptions of all curriculum
elements, as well as the way each curriculum is put together.

Amy, Basil, Clara, Desmond, Ernest, Fanny, and George: NOW instructor machines and NOW
student machines. These machines run the NOW instructor software named IMSGUIDE used by
instructors to manage their students. These machines use the curriculum resources located on
Horus and store student data in the Oracle database. These machines also run the NOW student
software named IMSLEARN used by students to take lessons on NovaNET. These machines use
the curriculum resources located on Horus and store student data in the Oracle database.

Oracle database servers: NOW database machines. They run an instance of an Oracle database
that stores dynamic data, such as student records, bulletins, sections, and assignments of
students managed by the NOW IMS. Some basic information, however, is stored on the Mesa
server file system. NOTE: The curriculum information for students is duplicated in the database
and the curriculum service system running on Horus.

Database multiplexer machines. These keep continuously open connections between NOW DB
and NOW applications IMSLEARN and IMSGUIDE so the applications do not have to open and
close connections for the thousands of transactions that occur.
Developing Draft : Anthony Garone
Functional Test Plan
Page 9
3/9/2016
Software for the NOW IMS:

IMSGUIDE: The NOW instructor system used by instructors to manage their students.

IMSLEARN: The NOW student system used by students to take lessons on NovaNET.

Osiris Library: Provides the graphical user interface, system interfaces, and cross-platform
frameworks for the Osiris applications IMSGUIDE and IMSLEARN.

Nowsupport: A utility used to create NOW groups, associate state alignments, view group
action logs, edit group access, transfer students between NOW groups, post instructor
messages, as well as turn on or off already-existing NOW groups. It is an Osiris app, so it
requires NovaNET to be up and running on the Osiris application machine. To the person using
“nowsupport,” it runs like any other NovaNET application on the NovaNET system.

Osiris Catalog Application: Online catalog application that contains user accessible information
about NovaNET lessons, resources, and utilities.

NOW Curriculum Service: When the NOW servers request curriculum information, the
proprietary curriculum service application executing on Horus accesses the XML and then
delivers the needed curriculum.

Multiplexer: Serves as platform-independent connection pooling software between Osiris
applications and Oracle and MySQL databases.
2.2.1 Osiris Execution Subsystem
Within the Osiris Execution Subsystem are the following components:

NOW Curriculum Server: Horus

NOW Catalog Application Server: Osiris

NOW Servers: Amy, Basil, Clara, Desmond, Ernest, Fanny, George (also known as cdc-1955-1
through -7)
2.2.1.1 Hardware Description
The components in this subsystem run on standard computer hardware. In the current production
environment all of these servers are run on Dell blade servers.
2.2.1.2 Software Description
All of the servers in this subsystem run on Linux-based operating systems.
The NOW applications are executed on the NOW servers by lessons on the Central Processing
Subsystem.
2.2.1.3 Connectivity Description
NOW servers communicate with Horus and Mesa over TCP. They also communicate with the NOW
database multiplexers, which are located in the NOW Database Subsystem, for storing and retrieving
data.
Osiris communicates with Mesa and Honey.
Developing Draft : Anthony Garone
Functional Test Plan
Page 10
3/9/2016
Horus communicates with NOW servers through Honey.
2.2.1.4 Tests
Tests for the Osiris Execution Subsystem portion of the NOW Subsystem are located in section 5.1.2.1
on page 25.
2.2.2 NOW Database Subsystem
Within the NOW Database Subsystem, we have the following components:

Oracle database multiplexers: chmpgappmplex1 and chmpgappmplex2

Oracle database servers: chmpgdbsnow

Oracle database SAN

MySQL database multiplexer: Honey

MySQL database server: chmpgdbsmysql
The multiplexers serve as connection interfaces for NovaNET clients. Rather than have individual user
accounts constantly opening and closing connections to the database servers, the multiplexers do the
connectivity work for them.
2.2.2.1 Oracle Databases
The Oracle databases host curriculum and student data. Nearly all customer-related data is stored in
these databases. In our current Oracle environment, NovaNET is running Oracle 10g with Data Guard. It
is the intention of the Scaling and Migration Project that we replace this current setup with an Oracle
RAC solution during phase VI. The RAC solution gives us increased availability and scalability. This will
be developed alongside phases II through V. Please see section 1.3 on page 1 for more information on
the project schedule.
2.2.2.1.1
Hardware Configuration
The database multiplexers run on standard computer hardware. Currently, the servers are Dell
PowerEdge 1550s. The multiplexers connect the NovaNET NOW application servers with NOW Oracle
database. It is the intention of NovaNET staff to integrate MySQL multiplexing into these servers and
phase the Honey server out of the production environment but this will not happen as an outcome of
the Scaling and Migration project.
The Oracle database servers, however, run on Sun Microsystems machines. The data is stored on a
CX400 SAN. This SAN hosts both the NOW database and the Enterprise Services and Support system. The
backups are copied to an ES45 AlphaServer so that they can be written to tape. This setup
communicates with the NOW Application servers via the NOW Database Multiplexers and connects to
the NOW database cluster machines.
2.2.2.1.2
Software Configuration
The multiplexers that connect to Oracle run two processes: ‘procplex’ and ‘procseb’. Procplex is the
multiplexer app and procseb protects us from bugs in the exe driver. These processes are indirectly
dependent upon ODBC drivers and are installed as services in the operating system.
In the NOW database, there is an autoarchive tool. It runs once a week. Inactive sites have students
removed after two years of non-use. Active sites have students removed after 5 years of non-use.
Autoarchive remotes signons from the groups in central server, also.
Developing Draft : Anthony Garone
Functional Test Plan
2.2.2.1.3
Page 11
3/9/2016
Tests
Tests for the Oracle Databases can be found in section 5.1.2.2 on page 26.
2.2.2.2 MySQL Databases
While the Oracle databases handle the majority of production-related data, the MySQL server handles
some production data, but hosts a wealth of support data. From a production standpoint, the MySQL
database delivers basic catalog data to NovaNET clients. Doing so relieves some bandwidth and
processing cycles from what would otherwise be depending upon the Oracle servers.
From a support standpoint, the MySQL server hosts data for the NovaNET incident database, nonprimetime gaming, usage statistics, engineer on duty, an exchange database, INIU data for a webbased INIU configuration utility, and a backup database. The server also hosts a curriculum conversion
utility.
2.2.2.2.1
Hardware Configuration
Honey connects the Osiris catalog application server with the NovaNET catalog database stored in
MySQL. It employs standard computer hardware.
2.2.2.2.2
Software Configuration
Honey runs Windows 2000.
In the NovaNET production environment, MySQL communicates to the Osiris catalog application through
Honey. For support utilities (primarily web-based), support staff communicates directly with the
database without the use of a multiplexer.
2.2.2.2.3
Tests
Tests for the MySQL Databases can be found in section 5.1.2.3 on page 27.
2.3 NovaNET DMZ Customer-Facing Subsystem
Created before the existence of the Internet, the NovaNET system needed to be modified to take
advantage of Internet resources to provide online instruction. It was important that these modifications
allowed users to stay within NovaNET while accessing Internet resources and not have to leave the
NovaNET interface to do so. Also, as Internet connectivity has improved over the years, it became
important to rely less upon serial leased lines and to rely more on Internet connectivity to customers.
The purpose of this subsystem is to provide that seamless access and to allow customers to connect to
the NovaNET system using the Internet.
2.3.1 Hardware and Software Configuration
Within the DMZ, we have the following hardware:

Ethernet Link Access Unit (2): Aggregates up to 720 NovaNET users onto a fractional T1

Internet Network Interface Unit (2): Delivers network traffic between the Framer (application
on the Alpha) and individual users connecting over the Internet with TCP

Stargate Server (1): Handles telnet, gopher, weather, and Lynx connections to the Internet
acting as an intermediary between NovaNET users and the Internet

Lynx Server (1): Provides text-based web browsing to NovaNET users via the Lynx web browser
Developing Draft : Anthony Garone
Functional Test Plan
Page 12
3/9/2016
2.3.1.1 Ethernet Link Access Units (ELAUs)
The ELAU is a Linux-based workstation that can aggregate up to 720 NovaNET users onto a fractional T1
circuit that can range from 56kbps to 512kbps. The ELAU establishes a TCP socket with users
connecting with Portal. It has a high-speed serial card that connects it back to the NIU. Although the
Linux platform is generic, a custom-written device driver is required for communication with the serial
card.
In the current production environment, there are two ELAUs being hosted in Champaign: Pilot and
Wilmington. The Pilot ELAU (pilot.nn.com) is used by all customers given a free trial period to sample
the NovaNET software. Wilmington is used for internal testing.
Unlike the LU and SU, the ELAU software is built atop other non-proprietary code. The Linux operating
system provides volumes of utilities and built-in functions for system management.
2.3.1.1.1
Hardware Configuration
The ELAU has 2 DB-9 serial connectors, one of which is used to connect an out-of-band management
modem. The Digi 570i card provides a DB-37 serial connector that is used to connect to a DSU/TSU
supplying a digital circuit back to the central NovaNET system. An adapter cable is used between the
V.35 DTE connector on the DSU/TSU and the DB-37 connector on the Digi serial card.
2.3.1.1.2
Software Configuration
Each ELAU needs to be configured to know which LAU addresses it handles. The ELAU can support up to
six LAU addresses. However, if more than one LAU address is assigned, certain rotary configurations are
unavailable. The ELAU must also know its assigned IP address on the customer LAN/WAN, as well as
valid subnet mask describing the scope of the customer network.
After logging into the ELAU via the standard OS login, the message of the day file is shown. This
provides a list of the ELAU commands available. When connecting to the ELAU via EMP, you are not
required to log into the ELAU; therefore you will not see the message at the start of an ELAU EMP
connection. This message can be displayed at any time, via any connection method, by entering the
‘nnhelp’ command.
Much more detailed information is available in the NovaNET ELAU Manual.
2.3.1.2 Internet Network Interface Units (INIUs)
The Internet Network Interface Unit, also known as the INIU (or sometimes called the NovaNET Internet
Concentrator), functions as a combination of an NIU and an ELAU. Like an NIU, it connects directly to
the central system’s Framer app, and, like an ELAU, it provides connections to the end-users of
NovaNET. The INIU can provide connections to NovaNET for over 5000 simultaneous users without any
special backplane chassis, VCOM cards, channelizing adapters, or high-speed digital serial cards.
2.3.1.2.1
Hardware Configuration
The INIU is a non-proprietary computer running Linux.
2.3.1.2.2
Software Configuration
The INIU requires any Linux kernel version 2.6.9 or later. The INIU software requires no additional code
libraries apart from standard libraries that come with Linux: libstdc++, libfam, and libxml2.
All INIUs keep a master copy of their configuration file in the CVS repository on Isis (isis.nn.com).
Nagios also monitors the INIU.
Developing Draft : Anthony Garone
Functional Test Plan
Page 13
3/9/2016
The INIU’s configuration file consists of XML data and resides in the ‘/nova/data/hybrids/’
directory. It specifies numerous parameters, among them:

which ports to listen on: for the connection to each framat (framat listener), the connections
to users of the INIU’s CLI (status listener), and the connection to users of NovaNET (rotary
listeners)

for each “FramatConnection”, which RotaryListeners to create

for each “RotaryListener”, the IP address and port on which to listen as well as which rotaries
to create (almost always just one rotary per RotaryListener)

for each rotary, the parameters that characterize that rotary—such as the name of the rotary,
the maximum number of users allowed, the customer code, the site number, the subsite
number, and the time zone.
The INIU reads its configuration file during runtime and can detect when the file has changed. INIU4
uses ‘/nova/data/hybrids/prod1.xml’ as its config file. INIU5 uses
‘/nova/data/hybrids/prod2.xml’.
Besides providing connectivity through TCP/IP for users of NovaNET, the INIU supports comprehensive
runtime monitoring. In conjunction with the web-based management interface, the INIU provides a
plethora of data on all aspects of its configuration, system performance, and usage patterns, including
a detailed view of the current status of any specified rotary. The web-based management tool is
available at http://chmpgdbsmysql.nn.com/nowadmin/hybrid/.
The INIU’s source code resides in the ‘/root/novanet/hybridniu/’ directory. The makefile also
resides in this directory. The INIU’s installation resides in the ‘/nova/’ directory. This directory
contains several subdirectories. The INIU’s configuration resides in ‘/nova/data/hybrids/’. The
files that run the INIU reside in ‘/nova/bin/’. The INIU keeps its log files in ‘/nova/logs/’ and
rotates its old log files daily to ‘/nova/logs/archive/’.
The INIU delivers network traffic between the FRAMAT and individual users running Portal, thereby
providing an alternative to the user of NIUs, link units, site units, and ELAUs. The INIU runs on standard
hardware with standard TCP and entirely abandons the CERL X-51 protocol, CNET, and nonstandard
hardware. The INIU also provides configuration of users, runtime monitoring, and historical data
collection. The software is in C++.
For detailed documentation and a list of commands available for INIU configuration, please see the INIU
Manual 3.1 document.
2.3.1.2.2.1 Run Files
The INIU software consists of a single C++ program called ‘nc’ and several support scripts written in
tcsh and Perl. ‘nc’ behaves as a server from the standpoint of the framat (which acts as a client) and
as a server from the standpoint of users running Portal (which also acts as a client). In addition, it
provides another server socket for direct communication through a simple command line interface for
the purposes of management and monitoring.

autoupdate: Script run by cron every 5 minutes to update the configuration file.

create_config, disaster recovery site only: script that converts the IP addresses in the
configuration file to their DR equivalents.
Developing Draft : Anthony Garone
Functional Test Plan
Page 14
3/9/2016

get_config, disaster recovery site only: script run by cron every 30 minutes to fetch the
latest configuration file from Champaign.

hybridniu: script that launches the INIU process.

logarchive: script run by cron daily to rotate the log file.

nc: the INIU program

prepend_date: helper script to prepend a timestamp.
2.3.1.2.2.2 Log Files
alarms: notifications of when the INIU cannot open a new session for a user because the INIU has lost
its connection to the framat.
cvs.log: notifications from CVS when the autoupdate cron job updates the configuration file or
experiences a problem.
hybrid.log: main log file; written by the INIU process.
stderr.log: error output from the INIU process; currently used by the XML parser to indicate that it
encountered a syntax or other error while reading the config file.
2.3.1.2.2.3 Support Scripts
Two support Perl scripts, ‘iniu’ and ‘iniutop’, reside in the ‘/opt/customer/psadmin/bin/’
directory. The iniu script provides access to the INIU’s command line interface. The iniutop script
collects CPU usage data for a custom SNMP check run periodically by Nagios. Both Perl scripts require
Perl libraries ‘getopts.pm’ and ‘process.pm’ located in ‘/opt/customer/psadmin/bin/lib/’.
2.3.1.3 Stargate Server
A Linux-based machine that runs FRX, a stripped down FRAMAT substitute for users that are executing
Internet activities within NovaNET. The server’s primary role is to act as an intermediary between
NovaNET users and the Internet for the tools on NovaNET that have Internet capabilities. It handles
telnet, weather, and Lynx connections to the Internet. Mesa passes control of customers to this
machine while accessing the Internet. Stargate passes control of the customer back to Mesa once telnet
or Lynx is exited. It also acts as a backup for DNS and mail between NN and the Internet. It
communicates with Mesa using the proprietary CNET protocol over the User Datagram Protocol (UDP).
Stargate also provides customer-accessible stats via gopher. It used to provide access lists to give
customer port usage via gopher, though this feature is not in use.
2.3.1.3.1
Hardware Configuration
Stargate is a standard Linux-based machine without any special hardware configuration.
2.3.1.3.2
Software Configuration
TO DO: Get more information about the software configuration of Stargate. Stargate is a heavilyconfigured server according to John Hegarty.
2.3.1.4 Lynx Server
Lynx is a Linux-based machine that runs the Lynx text-based web browser for NovaNET lesson usage. As
of this writing, Lynx’s usage is extremely limited as most customers have web browsers with graphical
Developing Draft : Anthony Garone
Functional Test Plan
Page 15
3/9/2016
user interfaces accessible on their local machines. Furthermore, NovaNET lessons requiring access to
Lynx only gather data when such data is not available locally. Nearly every lesson that uses/used Lynx
has the related data stored in the NovaNET environment, leaving the need for this server as dubious.
2.3.1.4.1
Hardware Configuration
Lynx runs on a standard Linux-based machine with no special hardware configuration.
2.3.1.4.2
Software Configuration
The Lynx web browser comes installed by default in many Linux distributions. If it is not available in a
Linux distribution, it is easily available for download and installation.
TO DO: Find out if any custom configurations for Lynx have been developed. It SHOULD be a default
installation, but we need to define that.
2.3.2 Tests
Tests for the DMZ Customer-Facing Subsystem can be found in section 5.1.3 on page 27.
2.4 NovaNET Leased-Line Subsystem
The purpose of the Network Subsystem is to link users running the Portal software with the NovaNET
central system. Students performing activities which are run on the NovaNET central server (called
TUTOR lessons) are bound by the performance of the central server and the communications path
between them and the central system.
There are two communication paths over which the Portal Interface can connect to the central
processing/control subsystem:

Over the Internet

Over Leased Lines
Internet customers connect through Internet Network Interface Units (INIUs). There are also two
Ethernet Link Access Units (ELAUs), one for testing and the other for pilot customers. The INIUs and
ELAUs are located within the DMZ.
Leased lines are used for all other customers to ensure sufficient, dedicated bandwidth to provide a
positive user experience. These customers connect to NovaNET via Network Interface Units (NIUs) using
proprietary protocols. There are two production NIUs and one developmental NIU within NovaNET.
The majority of NovaNET users (99%) connect via TCP on a combination of LANs, WANs, and the
Internet, plus some NovaNET-specific hardware. The NN-specific hardware at the other end of this TCP
connection is either a Site Unit or an Ethernet Link Access Unit.
2.4.1 Hardware and Software Configuration
The Leased-Line Subsystem consists of the following hardware:

Adtran Atlas DS-3 Multiplexer (1): Multiplexes T1 connections onto a DS3 for delivery to the
carrier.

Adtran 1241 Channel Bank (4): Multiplexes our leased-line phone data prior to it going to the
telco cloud. It aggregates serial connections and generates a T1 signal.
Developing Draft : Anthony Garone
Functional Test Plan
Page 16
3/9/2016

NovaNET Link Unit (6): Aggregates up to 240 NovaNET users’ traffic onto 56kbps digital circuits.

NovaNET Network Interface Unit (2): Handles network connectivity and bandwidth for leasedline communications between customers and the Central Processing Subsystem.
2.4.1.1 Hardware Configuration
Because we are migrating hardware from one physical location to another as part of the Scaling and
Migration project, and because the hardware configuration data files are mapped to physical hardware
addresses, it is important to replicate physical hardware connections existing in the production
environment equipment and our testing environment equipment. This is essential for all
telecommunications (“telco”) hardware in our testing environment. Thus, in order to properly
configure the hardware, we need to look at the physical telco equipment in the production
environment and do port-to-port matching on our testing equipment.
2.4.1.1.1
Connection Hints
Link Units and ELAUs connect directly to the NIU from the Atlas. Site Unit circuits hosted in Champaign
get routed to a channel back to an OCU card and then off the back of the channel bank to the CSU/DSU
to the racks in Champaign and then to the LU.
2.4.1.2 Software Configuration
Once the physical hardware configuration is complete, we need to ensure that the software
configuration in our testing telco equipment matches current production telco equipment. This is
accomplished by connecting to each production telco box via telnet, downloading the config file from
the production box, and uploading that config file to the matching test box.
Once each production config file is uploaded to its respective testing environment counterpart, we
need to verify the validity of the config file. This is done through menu-driven options or Unix file
system commands via telnet over the Ethernet interfaces for each piece of hardware.
2.4.2 Tests
Tests for the Leased-Line Subsystem can be found in section 5.1.4 on page 29.
2.5 Backup and Recovery
2.5.1 NovaNET Application Disk Packs
There exist two types of NovaNET disk pack backups: dailies and exacts. Exact backups are bit-for-bit
copies of all NovaNET data stored by the Alpha on the SAN. In the current production environment,
exacts are performed early on Saturday mornings after the system is taken down for weekly
maintenance. The backups are performed by the command center in our Champaign office.
Backups are performed using custom scripts written by John Hegarty. System-level
operations/commands are written into these scripts and nearly all of the backup process is automated.
The scripts catalog the disk packs and create tar files that are sent to tape.
2.5.1.1 Hardware
Application disk packs reside on the Alpha’s SAN. There are two Logical Unit Numbers (LUN) on the SAN
that are configured as a mirrored pair on the Alpha using LSM (Logical System Management for building
volumes, etc). NovaNET binaries are stored on local mirrored disks on the Alpha. This software is run
on local drives with data sources coming from the SAN. The Alpha has Host Bus Adapter cards, so there
are two paths to the SAN switches, which connect to the SAN for redundant pathing back to data. If a
switch fails, cable, HBA, we have a failover thanks to the HBAs.
Developing Draft : Anthony Garone
Functional Test Plan
Page 17
3/9/2016
2.5.1.2 Software
TO DO: Translate. John’s custom software. Records where files are within the disk packs and takes that
catalog and stores it in Mysql db on another server and that data is put into tars and noted in the mysql
db.
System level: John’s shell scripts that initiate the mirror splits, mounting, unmounts, merges back, etc.
and LSM, which maintains mirror.
2.5.1.3 Configuration
TO DO: Turn into paragraphs. Specific login details are coded into the scripts. Tarfiles are stored in
/nova/data/tarfiles, though the scripts are being reprogrammed to go to /nova_restore
2.5.1.4 Tests
Tests for the NovaNET Application Disk Packs Backup and Recovery can be found in section 5.1.5.1 on
page 33.
2.5.2 Databases
TO DO: Translate. Oracle database backups are performed on the local machine using Oracle’s RMAN
backup software and copied to an nfs mounted directory. The files are then swept to tape. Backups are
performed nightly (incremental nightly/full weekly???). All database datafiles and archived transaction
logs are backed up. The backup files are then written to ‘/opt/orabkup/novaprd1’,
‘/opt/orabkup/novaprd1/hot_backups’, and ‘/opt/orabkup/novaprd1/exports’, then copied to nfs
mount ‘/opt orabkup/novaprd1/hot_backups/TO_ALPHA’.
MySQL database backups are performed daily using a Dell-branded DLT4000 tape autoloader that stores
7 DLT type IV tapes. Two sets of 7 tapes are used for this process with tapes being overwritten every
two weeks. Other than manually swapping tape sets, the backup process is automated.
2.6 Subsystem Assembly
At this point, we have verified the hardware and software functionality of disparate subsystems. We
can now assume that all subsystems will interoperate. This is really a matter of attaching each
subsystem to the proper network switches and moving on to ‘author-mode’ testing.
3 Author-Mode Verification
“Author-mode” is a part of the NovaNET environment that provides behind-the-scenes access to
NovaNET software and lessons. We use author-mode tests to verify that NovaNET is functioning from an
operational viewpoint. Whether or not user data is fully-functional is not a concern for this stage of
testing. If you can get to the sign-on page and get into author mode, NovaNET is considered to be
operational.
If you can get to the sign-on page, then sign on through author mode, see if you can run ‘comstat,’ if
you can get into a notes file, see if the system is ‘generally working.’ Nothing specific in mind. It’s
pretty black and white, if you can log on and do anything, there’s no reason to believe that you can’t
do everything. You can either get into the app or you can’t. I would be hard pressed to think of a way
that NN could be up and running, other than NOW. Maybe one thing to do is to obtain a NOW signon
from support and log it on to prove that the NOW system is functioning.
Developing Draft : Anthony Garone
Functional Test Plan
Page 18
3/9/2016
3.1 NovaNET Portal Software
3.1.1 Install Portal
To install the NovaNET Portal software, use the three-disc installation media with the following
configuration:
Host: novanet4.nn.com
Port: 722
Leave all other options set to default. After the installation, change the execution target of your
desktop shortcut to: "C:\path\to\novanet\WPORTAL\wportal.exe" /m/a/c. The flags at the
end of the target call enable certain features that may be necessary for testing.
3.1.2 Tests
Tests for verifying Portal installation are found in section 5.2.1 on page 34.
3.2 NovaNET Central Processing Subsystem
NovaNET was originally developed on a CDC Cyber and later a Zephyr mainframe. To migrate off of this
unsupportable hardware, an emulator was created to run on the Alpha that could run the original Cyber
code. The emulator is a binary called xcp. Because emulation is costly from a performance standpoint,
a translator was also written to translate native Cyber code into native Alpha code.
When running NovaNET, the AlphaServer runs multiple instances of a binary called executor. The
executor keeps track of all active users and runs TUTOR lesson binaries. The executor is essentially the
NovaNET operating system. It is a low-level NovaNET system file. It is written in COMPASS (a language
proprietary to the CDC mainframe), processed by the Unix Port software tools, and run by an instance
of xcp. Multiple copies of the executor can be run to distribute the user load.
The executor is an operating system unto itself. It provides timesharing for thousands of simultaneous
users who run in their own virtual threads of TUTOR code within this program, as well as a file system,
memory facilities, timing, and other facilities. Written in COMPASS and running within the emulator,
certain heavily-executed portions of the COMPASS image have been modified so they could be
translated into Alpha machine language by the translator and are loaded into the emulator as shared
libraries. Other small portions have been implemented as C subroutines. The bulk of the code is
emulated COMPASS machine code.
The emulator is a binary written in C that provides emulation/execution of NOS executables for CDC
mainframes in the Tru64 Unix/Alpha environment. The programs CONDEN (a lesson condenser) and
PLATO (executor) run within the emulator.
Applications that run on the Alpha are Compass, Loader, Translator, PSCM, Ncd (Cnet), C-Router,
Format, Framat, Mastor, OsirisFileD, and OsirisRXD. Most of these are part of the Tutor development
Environment.
TO DO: Move. NCD: CNet Daemon. The CNet Server, /nova/bin/ncd, implements communication
between the executor (including TUTOR lessons) and external machines, such as the NIU, and the
computers that support services such as Telnet and e-mail exchange. These systems also run CNet
servers of their own. The CNet servers exchange short CNet messages encapsulated in UDP datagrams.
The executor initiates CNET connections by contacting the CNet server through MASTOR; once a
connection is established, messages are passed directly between the CNET server and executor. TUTOR
lessons can access CNET features using the –cnet- TUTOR command. Supervisory messages between the
Developing Draft : Anthony Garone
Functional Test Plan
Page 19
3/9/2016
executor and the NIU travel via CNET, but not terminal input and output. NCD, the CNET server, is
written in C, and the same source code is used on all CNET systems. It runs directly under UNIX.
3.2.1 Tests
Tests for the Central Processing Subsystem are found in section 5.2.2 on page 34.
3.3 NovaNET NOW Delivery Subsystem
To test the NOW Delivery Subsystem, we must access a lesson in author-mode that makes use of the
NOW components.
3.3.1 Tests
Tests for the NOW Delivery Subsystem can be found in section 5.2.3 on page 36.
3.4 NovaNET DMZ Customer-Facing Subsystem
3.4.1 Stargate and Lynx
The Lynx web browsing server is only available to customers connecting via an NIU. As such, this test
will only be possible by connecting through a serial data connection to NovaNET.
When a NovaNET user accesses a lesson utilizing the functionality of Lynx, control of the user is passed
from the central AlphaServer to Stargate and Lynx. If Lynx is functional within NovaNET, we can verify
that Stargate is functioning properly.
To access Lynx within NovaNET, use the ‘lynx’ or ‘internet’ lessons. The following is a screenshot of
the ‘lynx’ lesson:
Developing Draft : Anthony Garone
Functional Test Plan
Page 20
3/9/2016
Figure: ‘lynx’ lesson demonstrating Lynx web browser functionality
3.4.2 ELAU
Using the Portal application, we will connect to NovaNET by configuring Portal to use an ELAU for its
connection host. Assuming the Portal configuration as described in section 3.1.1, we now have access
to the administrative menus available in the Portal application. In our current production environment,
if Portal is configured to use ‘staff.nn.com’ on port 722, it will connect through the Wilmington testing
ELAU.

Use the ‘File > Close Connection’ option. Portal will display a white screen with black text
confirming disconnection.

Use the ‘Settings > Communications’ option. Configure the connection according to what you
are testing.

Use the ‘File > Open Connection’ option to connect using the new configuration settings.
If Portal can connect using the new settings, we have verified the functionality of the ELAU.
Developing Draft : Anthony Garone
Functional Test Plan
Page 21
3/9/2016
3.4.3 INIU
Using the Portal application, we will connect to NovaNET by configuring Portal to use an INIU for its
connection host. Assuming the Portal configuration as described in section 3.1.1, we now have access
to the administrative menus available in the Portal application. In our current production environment,
if Portal is configured to use either ‘novanet4.nn.com’ or ‘novanet5.nn.com’ on port 722, it will
connect through an INIU.

Use the ‘File > Close Connection’ option. Portal will display a white screen with black text
confirming disconnection.

Use the ‘Settings > Communications’ option. Configure the connection according to what you
are testing.

Use the ‘File > Open Connection’ option to connect using the new configuration settings.
If Portal can connect using the new settings, we have verified the functionality of the INIUs.
3.5 NovaNET Leased-Line Subsystem
Assuming we have verified the functionality of the Alpha and the NOW delivery mechanism (in author
mode), we can test the serial telecommunications subsystem. The best way to locally test the LeasedLine Subsystem is using the EMP tools built into NovaNET. EMP is a proprietary protocol that allows us
to monitor and manage NovaNET hardware and connectivity throughout the environment.
3.5.1 ‘Comstat’ Tests
Comstat will tell us that the system is able to communicate with the Link Units connected to each NIU.
Because LUs are daisy-chained, comstat will be able to tell us the status of all LUs within our network
in a short amount of time. If the NIUs can see connectivity from an LU, then we can deduce that all
other circuits will follow suit.
To access ‘comstat’ we must log into NovaNET using an account in the group ‘novanet’. This will bring
us to the author mode page prompting us to type in a lesson name. It is here that we will type
‘comstat’ and hit enter.
4 End-User Verification
The purpose of End-User testing is to verify that the system is functional from end to end using the
different roles that interact with the system. These roles include the NovaNET Site Director, Instructor
and Student.
4.1 User Scenarios
The following table defines at a high level the scenarios that will be executed for End-User testing for
the Scaling Project.
ID
Scenario 1
Scenario Name
Scenario Summary
Create Student and Assign
Curriculum
Create a section and a student, add the student
to the section. Add a curriculum and catalog to
the site (via the Curricula button), then add them
to the section, then assign some curricula to the
student within the new section.
Developing Draft : Anthony Garone
Functional Test Plan
Page 22
Complete Lesson and View Report
Modify User data and privileges
Login as a Student (from Scenario 1) and
complete assignments within assigned curricula
to different points. Login as Teacher and Run
Reports on Student to verify Usage
Modify the signon's name and access privileges
Scenario 4
Deactivation/Reactivation
Deactivate and reactivate a signon, Deactivate a
signon and create a new signon with the same
name
Scenario 5
Scenario 6
Delete a section and move
assignments
Reports
Delete a section. Move a signon's assignments
between sections
Run several types of reports with varying options
Scenario 7
Navigation - Go To
Use the Go To feature to switch between
students and flags
Scenario 8
Assignment Templates
Create an assignment template for the section.
Modify the settings for the section.
Scenario 9
Bulletins
Create bulletins at the site, section and student
levels
Scenario 2
Scenario 3
3/9/2016
Specific execution steps are defined and contained in a Test Scenario document (location).
4.2 Test Environment
The testing will take place from a Windows or Macintosh workstation. Since the focus of testing is on
the NovaNET system, the client interface testing will not be a significant factor in the testing. The
client will point to the specific NovaNET system under test.
4.3 Test Reporting and Defect Tracking
Results of these scenarios will be captured and provided upon completion. Issues found will be directed
to the appropriate person. It has not been determined what kind of issue tracking will be done.
4.4 Exit Criteria
Successful completion of the above scenarios will finally validate the various components of the
NovaNET system. Successful completion of all scenarios is required to Exit the Test Phase of this
project.
5 Tests
This chapter houses all of the testing methods to verify all matter discussed in previous chapters. Just
as the document contains chapters for Subsystem Readiness, Author-Mode Verification, and End-User
Verification, this chapter is broken into sections so we can easily find the test we need for each of
these major stages in our testing plan. Beyond each of those sections, the tests are aggregated by
subsystem for consistency and navigability.
Chapter 6 contains checklists relevant to the tests in this chapter. Each checklist should be filled out as
a test is being performed. The checklist provides organization to the flow of the implementation and
accountability as to who performed each test and how long it took. The checklists are, essentially,
high-level overviews of the tests found in this chapter. Thus, each checklist and its respective test
should be printed and accompany the tester during the time of testing.
In Chapter 7, there exists a table (see section 7.1 on page 39) showing which steps are performed in
which phase. The tests are further organized into logical “testing rounds” with each round containing
tests to verify certain aspects of the phase’s implementation. This chapter provides no guidance as to
which tests need to be performed during which phase, but rather serves to allow quick access to all
tests and their methodologies.
Developing Draft : Anthony Garone
Functional Test Plan
Page 23
3/9/2016
5.1 Subsystem Readiness Tests
5.1.1 Central Processing Subsystem
5.1.1.1 Verify Remote Login to Alpha
SSH must be used to verify the ability to remotely access the Alpha server. Options to do this include a
standard SSH client from a desktop system (e.g. putty) or use of the SSH command line utility from
another system. A user other than root must be used to log in to the system; root authentication via ssh
is disabled. Here is a sample SSH connection:
psadmin@cdc-480-1 : ssh 159.182.104.104
psadmin@159.182.104.104's password:
Last login: Wed Dec 19 21:48:38 CST 2007 from 159.182.104.75
HP Tru64 UNIX V5.1B (Rev. 2650); Fri Nov 30 09:56:50 CST 2007
NHD Rev(V7.0) installed
The installation software has successfully installed your system.
There are logfiles that contain a record of your installation. These are:
/var/adm/smlogs/install.cdf
/var/adm/smlogs/install.log
/var/adm/smlogs/install.FS.log
/var/adm/smlogs/setld.log
/var/adm/smlogs/fverify.log
-
configuration description file
general log file
file system creation logs
log for the setld(8) utility
verification log file
No mail.
cdc-es80-1.nn.com>
5.1.1.2 Verify Disk Storage Connectivity Between Alpha and SAN
The ‘/nova/data/disks/’ file system is located on the SAN. To verify that this is mounted on the
system, the following can be executed in a command line:
cdc-es80-1.nn.com> df -k | grep /nova/data/disks
nova_domain3#nova_data_disks 94367744
55369151
/nova/data/disks
38988992
59%
Something similar to the line above should be returned after the ‘df’ command is executed.
To verify read and write capabilities, perform the following as the NovaNET root user without error:
# cd /nova/data/disks
# touch testfile
# ls -l testfile
-rw-r--r-1 root
# rm testfile
novanet
0 Dec 19 22:49 testfile
To ensure both paths to the SAN are active, start by determining which devices are being presented
from the SAN. Perform the following for the CX400 connected Alpha systems:
# hwmgr view devices |grep DGC
275: /dev/disk/dsk5c
DGC
276: /dev/disk/dsk6c
DGC
Developing Draft : Anthony Garone
RAID 5
RAID 5
IDENTIFIER=0
IDENTIFIER=1
Functional Test Plan
Page 24
3/9/2016
This shows that dsk5 and dsk6 are presented from the CX400.
Now, execute the following:
# hwmgr -show scsi –full
SCSI
DEVICE
DEVICE DRIVER NUM DEVICE FIRST
HWID: DEVICEID HOSTNAME
TYPE
SUBTYPE OWNER PATH FILE
VALID PATH
------------------------------------------------------------------------266: 0
cdc-es80-1 disk
none
2
1
dsk0
[1/0/0]
WWID:0c000008:000c-50ff-feb7-f702
BUS
TARGET LUN
PATH STATE
--------------------------------1
0
0
valid
SCSI
DEVICE
DEVICE DRIVER NUM DEVICE FIRST
HWID: DEVICEID HOSTNAME
TYPE
SUBTYPE OWNER PATH FILE
VALID PATH
------------------------------------------------------------------------267: 1
cdc-es80-1 disk
none
2
1
dsk1
[1/1/0]
.
. [deleted]
.
.
SCSI
DEVICE
DEVICE DRIVER NUM DEVICE FIRST
HWID: DEVICEID HOSTNAME
TYPE
SUBTYPE OWNER PATH FILE
VALID PATH
------------------------------------------------------------------------275: 7
cdc-es80-1 disk
none
2
4
dsk5
[0/0/0]
WWID:01000010:6006-0160-6ee4-0e00-8593-e392-e5d6-da11
BUS
TARGET LUN
PATH STATE
--------------------------------0
0
0
valid
0
1
0
valid
3
0
0
valid
3
1
0
valid
SCSI
DEVICE
DEVICE DRIVER NUM DEVICE FIRST
HWID: DEVICEID HOSTNAME
TYPE
SUBTYPE OWNER PATH FILE
VALID PATH
------------------------------------------------------------------------276: 8
cdc-es80-1 disk
none
2
4
dsk6
[0/0/1]
WWID:01000010:6006-0160-6ee4-0e00-8493-e392-e5d6-da11
BUS
TARGET LUN
PATH STATE
--------------------------------0
0
1
valid
0
1
1
valid
3
0
1
valid
3
1
1
valid
.
Developing Draft : Anthony Garone
Functional Test Plan
Page 25
3/9/2016
. [deleted]
.
Find the sections that describe dsk5 and dsk6. Note the ‘PATH STATE’ portion of those disks—all
paths should say ‘valid’. If any say ‘stale’, there is either a problem with the path or with the LUN
itself.
5.1.1.3 Verify Read/Write to SAN
To verify that the CX400 is functioning properly, access the SAN management utility Navisphere via
http://159.182.104.70. After logging in, ensure that there are no faults on the main screen. This is
what the CX400 looks like with no faults (note: there are other applications hosted on the SAN, so
faults on the SAN may not necessarily be relevant to the NovaNET Central Server):
Figure: Navisphere Interface
5.1.1.4 Verify Connectivity to Primary NovaNET Components
Since the AlphaServer is the central component in the NovaNET network, nearly every component
communicates with it.
While remotely logged into the AlphaServer, we should be able to ping each of the servers/components
listed in the Production Component List in section 1.5.1 on page 1.
5.1.2 NOW Subsystem
5.1.2.1 Osiris Execution System
5.1.2.1.1
Verify remote login to all NOW Servers
Just as we performed in section 5.1.1.1 for the AlphaServer, we need to open SSH connections to each
of the NOW servers (including the application and catalog servers). Please refer to the Appendix
section 7.3 for a table of important IP addresses and hostnames.
Developing Draft : Anthony Garone
Functional Test Plan
5.1.2.1.2
Page 26
3/9/2016
Verify existence of NOW apps on NOW Servers
Verify the applications ‘imsguide2_2’ and ‘imslearn2_2’ are in the ‘/nova/Osiris/bin/’
directory. Verify they are owned by ‘Osiris/Osiris’ with read and execute permissions. While these
applications are locally executable, they perform with severely limited functionality outside of a
working NovaNET environment.
5.1.2.1.3
Verify connectivity between NOW servers and Alpha
Because these servers communicate using a proprietary NovaNET communications protocol, it is nearly
impossible to tell if these machines have functional connectivity without a working NovaNET
environment. We can, however, send ping packets to make sure the machines can communicate.
5.1.2.1.4
Verify connectivity between NOW servers and multiplexers
Much like connectivity between NOW servers and the Alpha, it is difficult to determine production
connectivity and functionality outside of a working environment. We can, however, send ping packets
between the machines to ensure basic network connectivity.
5.1.2.2 Oracle Databases
There is a series of tests to be performed to ensure functionality of an Oracle database. Please see
Oracle database migration checklist located in the appendix of this document.
5.1.2.2.1
Verify Connectivity and Functionality of Oracle SAN
TO DO: Translate. This is a sys admin task. They connect to the SAN and present mount points to the
DBA team for use. The admins would need to provide steps on verifying that the SAN works properly
and is properly configured.
All a DBA would do on this task is ensure that database mount points are owned by user oracle group
dba. Also would verify that user oracle can write files to the mount points.
5.1.2.2.2
Verify Functionality of Clean Oracle Installation
TO DO: Provide link to the checklist in Appendix provided by Laryssa.
5.1.2.2.3
Migrate Data from Production to Testing Environment
TO DO: Laryssa will fill in process
5.1.2.2.4
Verify NovaNET Architecture (users, tablespaces) and security are
properly set up in the database
5.1.2.2.5
Verify Connectivity between NOW Application Servers and the
Database Multiplexers
TO DO: translate, get real information. This is not a DBA task as we have nothing to do with the NOW
App servers and the DB multiplexers. Not sure who would handle this.
5.1.2.2.6
Verify Connectivity between Multiplexers and Oracle Database
Server
The Oracle multiplexers execute a process called ‘procseb’ that logs into the Oracle database. With
this subsystem running, check the Oracle database connections to ensure that user ‘procseb’ is
connected to the database.
TO DO: Laryssa will fill in process
Developing Draft : Anthony Garone
Functional Test Plan
Page 27
3/9/2016
5.1.2.3 MySQL Databases
5.1.2.3.1
Verify Remote Login to MySQL Server and Multiplexer
TO DO: Create process
5.1.2.3.2
Verify Functionality of MySQL
While there is not much to test in an installation of MySQL without concerning ourselves with data,
there are still steps we can take to ensure the functionality of the application itself. We can connect to
the MySQL engine using the local command line utilities. We can also connect remotely using the
MySQL command line utilities on another machine.
Once connected, we can run sample queries to ensure true relational database functionality.
5.1.2.3.3
Migrate Data from Production to Testing Environment
While migrating data from one environment to another is fairly trivial, there are still critical steps to
verify that the migration worked properly. We can do so by comparing table sizes (rows, physical data
sizes, etc) between the production environment and the testing environment.
Since the current production environment does not make use of more modern relational database tools
(triggers, stored procedures, etc.), we do not need to worry about anything besides the integrity of the
data.
5.1.3 DMZ Customer-Facing Subsystem
5.1.3.1 Hardware and Software Tests
5.1.3.1.1
All Components
While the DMZ Customer-Facing Subsystem contains an array of differing hardware and software, we
must still test connectivity of all components to one another and to other primary NovaNET
components as listed in Appendix section 7.3.
5.1.3.1.1.1 Verify Remote Login Connectivity
5.1.3.1.2
Ethernet Link Access Units (ELAUs)
While the ELAU is largely non-functional outside of a live NovaNET environment, we can still test to
verify its readiness. The Linux operating system should be in working order and allow remote and local
logins.
5.1.3.1.2.1 Hardware Test
5.1.3.1.2.1.1 Verify Serial Card Functionality
The ELAU runs proprietary software written in C and Perl. Although the Linux platform is generic, a
custom-written device driver is required for communication with the serial card. It connects user PCs
to the NovaNET NIUs across leased lines.
How do we verify this?
5.1.3.1.2.2 Software Test
5.1.3.1.2.2.1 Check Configuration File
Developing Draft : Anthony Garone
Functional Test Plan
Page 28
3/9/2016
There’s a script on the ELAU that checks to make sure the config file is compatible with NN. Just
because you have a good config doesn’t mean that it’ll work as an ELAU. You can’t tell if it’ll work
unless it’s plugged into NN. Where is this script?
The ELAU is configured by editing the text file ‘/elau/adm/config’. The file must adhere to a rigorous
format and includes both mandatory and optional fields. A template config file is included on each
newly-assembled ELAU. Each section of the config file is delimited by a label in brackets, i.e. main,
rotary, alias.
It is highly recommended that you execute the ‘checkconfig’ script after any editing of
/elau/adm/config to ensure that the file still meets the format guidelines. The ELAU process will
restart if /elau/adm/config is not in the proper format, but users will not be able to connect to the
ELAU. You must also run the command ‘nnschedulecycle’. Without this command, you could lose all
changes after the nightly system restart.
5.1.3.1.3
Internet Network Interface Units (INIUs)
5.1.3.1.3.1 Machine Operation
Similar to ELAU. Log on, Linux. See if they’re working, see if the INIU process is configured and
running.
5.1.3.1.3.2 Verify Existence/Locations of Run Files, Support Scripts, and Logs
5.1.3.1.3.3 INIU Software Operation: ‘nc’ Process
The INIU process ‘nc’ must run under user ‘root’, should run at all times, and does not ever need
restarting. It has an entry in ‘/etc/inittab/’ for this purpose: ‘hn:2345:respawn:/nova/bin/hybridniu’.
5.1.3.1.4
Stargate Server
TO DO
Stargate is a custom-configured box designed to work specifically within NovaNET using the cnet
protocol. There is only one Stargate server amongst all NovaNET environments.
‘netmon’ is a lesson in NovaNET that will allow us to see if Stargate is running.
5.1.3.1.5
Lynx Server
Lynx is a Linux-based machine running the Lynx text-based web browser. It handles Stargate’s web
requests. NovaNET has a gateway to text-based web browsing.
5.1.3.1.5.1 Machine Operation
This is a Linux-based server, so we can log in to the system and verify that everything looks normal and
operational. Verify that Lynx is installed on the server.
5.1.3.1.5.2 Verify Lynx installation and connectivity to Internet
This step cannot be completed unless the NN portal client is connected via an NIU.
5.1.3.2 Connectivity Tests
TO DO: Translate. Attempting to connect on all domain names that exist on INIUs and Pilot ELAU. So,
using NN portal software to attempt to connect to existing rotaries. I don’t know if you can do this
when it’s not “live.” The IPs will all be translated so we can say, “I know that this IP is going to be the
one used for customer2.nn.com, so if I can connect to that IP address and one of the available port
numbers, then we’ve proven that once the DNS has changed, then it’ll work.”
Developing Draft : Anthony Garone
Functional Test Plan
5.1.3.2.1
Page 29
3/9/2016
Verify external access to DMZ
The simplest way to verify external access to the DMZ is to run Portal from outside the Pearson
network. This can be as trivial as going to a local coffee shop and connecting to their wireless service
or as complicated as connecting to a home network via VPN and testing Portal on a remote machine.
Either way, the end result of such tests yields the same result: whether the DMZ is functional or not.
5.1.3.2.2
Verify internal access to DMZ
TO DO: Translate. Assuming that NN system we’re testing is up and running, to me the thing to do
would be to use the NN portal software and connect to a rotary and see the sign-on page come up.
Clarification: To prove there is access, you prove there is connectivity. The easy way to prove there is
connectivity to a NovaNET delivery device (no matter what the device is: SU, ELAU, INIU) is to connect
as a customer would. In other words, to a configured rotary on that device using the NovaNET Portal
software.
5.1.4 Leased-Line Subsystem
5.1.4.1 Hardware and Software Tests
The following tests cannot be performed without physical connectivity between components. Such
connectivity is described in the previous section.
5.1.4.1.1
Atlas DS-3 Multiplexer
The Atlas multiplexes T1 connections onto a DS3 for delivery to the carrier.
5.1.4.1.1.1 Hardware Test
The Atlas has 8 card slots with either DS-3 cards or “quad T-1” cards (cards with four jacks for four T1
connections). There are LED indicators on the faces of these modules to show the ready-state of each
card. After the Atlas has been powered on and its software booted, none of the modules in the unit
should be in the “alarm” state.
If the Atlas is not connected to connection signal, the alarm light will turn on. If the Atlas is connected
to a signal and the alarm light persists, turn off the unit and re-seat the appropriate modules. Ensure
that the cable connectors are properly seated on both ends of the connection.
5.1.4.1.1.2 Software Test
After the hardware has been verified as functional and no modules are in their alarm state, we can
connect to the console port of the Atlas to verify that the internal software is functional.
5.1.4.1.2
Adtran 1241 Channel Banks
The Adtran 1241 channel bank multiplexes our leased-line phone data prior to it going to the telco
cloud. It aggregates serial connections and generates a T1 signal.
5.1.4.1.2.1 Hardware Test
The Adtran 1241 is similar to the Atlas Multiplexer in that the alarm LEDs will turn on if the connection
modules have no connectivity. If the alarm lights persist after attaching the appropriate connections,
re-seat the module(s) and check cabling at both ends of the connection.
5.1.4.1.2.2 Software Test
After the hardware has been verified as functional and no modules are in their alarm state, we can
connect to the console port of the Atlas to verify that the internal software is functional.
Developing Draft : Anthony Garone
Functional Test Plan
5.1.4.1.3
Page 30
3/9/2016
Network Interface Units
The NIUs are custom-made computers with a VME multi-bus chassis. On the VME bus is an “Alpha-on-acard” running Digital Unix and SBE 8-port serial cards. The serial cards are the connections from either
ELAUs or LUs and can handle either 8 fractional T1 serial connections or one full T1 connection. There
may be more than one LU on a single line, however only one ELAU may be plugged into a serial port. As
of this writing, there are 12 serial cards in NIU1 and 8 in NIU2.
The NIU’s primary functions are to transmit and receive frames between the Framer and the rest of the
NovaNET leased-line network, to meter all data to downstream communications equipment at their
optimum rates, to prevent any port from exploiting more than its fair share of bandwidth, to respond
to CNet commands, to manage EMP sessions, to continuously gather network statistics, and to send
notifications in real-time to the operators and others about communications equipment that goes
offline or experiences problems.
The NIU utilizes high-speed synchronous serial lines that connect to ELAUs and LUs. It communicates
directly with the Alpha over two “channels”: TCP for lesson-related communications with NovaNET
users and CNet proprietary protocol over UDP for device management/monitoring functions.
5.1.4.1.3.1 Hardware Test
TO DO: Find a hardware test to do on the NIU.
From Ray’s e-mail: The serial cards do have some LED indicators. A run, timeout, and failure and then
8 lights for 0-7. When up and running, the run LED is the only lit LED. When booting, I believe they light
up in sequence over and over again – but I can’t swear to that.
The Alpha card has a single digit LED display which shows a spinning bar when booting. The two NIUs in
the machine room are displaying a “0”.
5.1.4.1.3.2 Software Test
Testing the NIU before connecting it to NovaNET is limited to ensuring that Unix is running properly.
We can test this by connecting to the NIU via telnet over the Ethernet interface. Ensure that the NIU
processes—aniu and ncd—are running. ‘aniu’ is the Alpha NIU process and ‘ncd’ is the cnet daemon.
Source files are located in ‘/nova/src’. Run files are located in ‘/nova/bin’. The main executable is
‘aniu’. ‘niugo’ starts the NIU process. ‘niustop’ stops the NIU process. ‘niulog’ does a ‘tail’ of
the NIU’s log file. The log file is located at ‘/nova/adm/logs/niu.log’. Configuration is
accomplished through a NovaNET lesson called ‘niuconfig’, which updates the config file and notifies
the NIU process to re-read the config file. The config file resides at ‘/nova/etc/niu.config’.
Each NIU has a configuration file that describes the physical layout of its part of the NovaNET network.
It uses this layout information to route and meter NovaNET frames. The connections to the NIU are
called “lines,” and each line handles one or more LAU addresses. The NIU not only routes the LAU
addresses to the proper lines, but it also performs data metering. Metering makes the best possible use
of the bandwidth available on our leased lines. In order to do effective metering, the NIU needs to
know how equipment is physically connected and what the data rates are. NIUs are configured by the
central office with a NovaNET lesson called NIUCONFIG.
On some VCOM54 boards we use a piece of proprietary equipment, a Channelizing Adapter, to directly
generate a T1 signal. The Channelizing Adapter saves us the cost and bother of using channel banks for
this task.
Developing Draft : Anthony Garone
Functional Test Plan
Page 31
3/9/2016
In the downstream direction, the NIU communicates solely by serial connections. In the upstream
direction, the NIU communicates to the Framer and to other central system components using cnet via
raw TCP/IP over 10BaseT Ethernet.
5.1.4.1.4
Link Units
The Link Unit is a proprietary device running custom firmware. It serves as a store-and-forward
multiplexer implementing the X-51 protocol. It aggregates up to 16 Site Units onto a fractional T1
circuit and connects directly to the NIU over a serial RS-232 line. The LU is based on five Motorola
microprocessors and contains 16 synchronous RS-232 connector, called “taps.” Two synchronous V.35
connectors, called “Head” and “Tail,” provide communications to the NIU and daisy-chained LUs.
On the front of the device is an RJ-45 console port. There is also a push-button and a series of status
LEDs (“indicators”). They are explained as follows:

Mode switch: push button. This switch changes the state of the mode indicator light. Since the
state of the mode indicator light can be read over the network, a press of this button can be
used to verify that field personnel are at the proper LU.

Mode indicator: yellow LED. This indicator is turned off and on by the mode switch, or via the
network.

Offline indicator: red LED. This indicator is on when the LU is unable to communicate with the
NovaNET central system for over two seconds. It should normally be off.

Error indicator: red LED. This indicator goes on briefly when the LU detects an internal
software problem. It should normally be off.

Running indicator: green LED. This indicator is on when the LU is running its internal firmware
program. If this indicator is off, it indicates that the LU has failed.

Power indicator: green LED. This indicator should be on when power is applied to the LU. If this
indicator is off, there is no power to the LU, or the LU’s power supply has failed.
On the back of the device are controls, indicators, and connectors:

Fuse. 1A slow-blow.

AC power connector. This connector is for a 120VAC power cord.

Tap connectors (16). These sixteen 25-pin DB-connectors provide access to the LU’s 16
synchronous taps for connecting Site Units. These connectors are used with CSU/DSU.

Local tap 0 connector. This RJ-45 is used to connect a Site Unit without CSU/DSU. The SU must
be less than 50 feet distant. If the Local Tap 0 connector is in use, the other Tap 0 connector
may not be used.

10BaseT Ethernet connector. This RG-45 connector may be used for TCP/IP connectivity.

Console connector. This RJ-45 connector allows connection of an RS-232 diagnostic console.
There is an identical connector on the front of the box.

Head connector. This V.35 M34 is the connection to the NovaNET central system, or to the Tail
connector of another LU. It may be connected to the V.35 DTE port of the head-circuit DSU
Developing Draft : Anthony Garone
Functional Test Plan
Page 32
3/9/2016
using a Remote Link Unit able, or, for short distances, a Local Link Unit Cable may be used
without a CSU/DSU.

Tail connector. This V.35 M34 connector is used to daisy-chain LUs. It allows one LU to connect
to another LU. It may be connected to the V.35 DTE port of the head-circuit DSU using a
Remote Link Unit Cable, or, for short distances, a Local Link Unit Cable may be used without a
CSU/DSU.
Documentation exists going into much greater detail about the LU in the NovaNET Link Unit Manual.
5.1.4.1.4.1 Hardware Test
When the LU is initially powered on, the power LED should come on solid and the mode, offline, error,
and running LEDs will light in sequence. Then, when the LU begins its normal operation, only the power
and running indicators will be on.
Assuming the LU is connected to the appropriate hardware via the Head and Tail ports, we should see
the following indicator LED states:

Mode LED: Off (Yellow)

Offline LED: Off (Red)

Error LED: Off (Red)

Running LED: On (Green)

Power LED: On (Green)
If the Offline LED is on (red), the LU is either not connected properly, the data circuit is not functional,
its configuration is incorrect, the NIU to which it is connected is not configured to connect to it, there
exists a configuration problem with another LU in a daisy chain, or the data rates in the configuration
file are incorrect.
If, after initial power-on, the Mode, Offline, Error, and Running indicators are flashing in unison, the LU
is in self-test mode. This mode is used during manufacturing and initial programming of new LUs. New
(non-programmed) LUs will go into this mode when powered on. Programmed LUs can be put into selftest mode by holding the Mode switch in during power-on.
5.1.4.1.4.2 Software Test
The LU is configured through software using a simple command-line interface called the Link Unit
Monitor. This interface can be accessed by connecting a terminal (typically a laptop with an ASCII
terminal program) to either of the LU consol ports. In addition, the command interface can be
accessed from the NovaNET central system using the Embedded Monitor Protocol (EMP). The console
port is intended to be used by a technician on site during installation and setup. The central system
interface makes it possible to remotely diagnose and correct some LU problems.
To connect a computer to the LU console port, use a standard LAB8 adapter and a straight-through RJ45 cable. Set a standard ASCII terminal program to 8 bits, no parity, with 1 stop bit. You can use any
baud rate between 1200 bps and 115,200 bps (57,600 bps is recommended). After connecting press
RETURN to let the console port auto-baud to the correct speed. This will cause the LU to respond with
a >>> prompt. At this prompt you can type ‘help’. This will list the LU commands and variables.
The LU variables have names with periods in them. The variable values can be inspected by typing the
variable name and hitting return. The variable values can be changed by typing the variable name,
Developing Draft : Anthony Garone
Functional Test Plan
Page 33
3/9/2016
typing an ‘=’ and then typing a new value for the variable. Changes to variable values are permanently
stored using the ‘config save’ command.
Detailed instructions for LU commands can be found in the NovaNET Link Unit Manual document.
5.1.4.2 Connectivity Tests
5.1.4.2.1
Atlas Loopback Testing
In order to test the telco hardware and network, we need to set up loopback test with the channel
service unit (CSU) and data service unit (DSU) at both ends of our serial-line connection. With each
loopback test, we work with the telco vendor (Verizon as of this writing) to verify that they see our
loopback and we verify that we see their loopback. We “see” a loopback by sending a particular data
pattern to Verizon’s loopback and it “loops back” to us. Verizon does the same on their end when we
set up our loopback.
The loopback being performed is software-based. Hardware-based loopback tests depend on wire
swapping in the RJ-45 plug. In a typical 56kbps T1 RJ-45 connector, there are two wires for data
transmission and two wires for data retrieval. By swapping one of the transmit wires for one of the
receive wires, we create the loopback-capable cable. Instead of physical wire-swapping, we depend on
the electronics within the telco equipment to do a software-based wire swap.
The loopback test will be performed using our Atlas DS-3 Multiplexer. Using the Ethernet interface, we
will telnet into the hardware and use the menu-driven interface to send signal patterns.
Make sure all configuration of NIUs, INIUs, LUs are copied over from production in order for the sites to
run properly when production is moved.
TO DO: Step-by-step process
5.1.5 Backup and Recovery
5.1.5.1 NovaNET Application Disk Packs
5.1.5.1.1
Running the backup
An operator at CDC or wherever logs into the alpha as operator and run ‘scripts’ that brings user to
directory that contains backup scripts.
http://medusa.nn.com/opsmanual/nova/exacts.html
5.1.5.1.2
Verifying the backup
Run an exact, have the backup client in IAC (Tivoli) write to tape. Make request to have that tar file
restored from tape. Run John’s software to run specific file and verify that it was successful.
Restoring involves running John’s software to find out which tar file we need to retrieve using the
Mysql db catalog. Make request to have that tar file restored from tape. John is modifying his C++
software to read the restore tar file from /nova_restore so when you make a request for a tar file, you
need to specify that you need it in /nova_restore. Once you know the file exists, then run John’s
software.
May or may not be current: http://medusa.nn.com/opsmanual/nova/backuprequests.html
Located on esker at ‘/nova/proc/scripts’; points to ‘/nova/scripts/backup/’.
Developing Draft : Anthony Garone
Functional Test Plan
Page 34
3/9/2016
Mesa RAID is #29 on Waldo. Mesa’s scripts are at ‘/nova/proc/scripts’, which call stuff at
‘/nova/scripts/Backup/’. These are “expect scripts”—do what you would do on Waldo—so the password
is contained in these scripts. Exacts: Saturday morning; ‘/nova/proc/scripts/diskclone’ runes
‘/nova/scripts/Backup/DoExacts’.
5.1.5.2 Database Backups
5.1.5.2.1
Verify Oracle Backup
TO DO: Get process
5.1.5.2.2
Verify MySQL Backup
TO DO: Get process
5.2 Author-Mode Tests
5.2.1 Portal Application
5.2.1.1 Testing Connectivity

Double-click the NovaNET icon to test connectivity

The login screen will prompt you for the following items. Type these adjoining items:
o
Group Name: nndemo
o
Username: nncheck
o
Password: not required
If you do not see any errors and see the student index screen, your connection to the system is
working. If you receive any errors, please call our Product Support Contact Center for assistance.
TO DO: Make sure flags in shortcut are called
5.2.2 Central Processing Subsystem
As we’ve already established, the Alpha is the most critical piece of the NovaNET environment. If the
Alpha and its software are not fully functional, NovaNET cannot be fully functional. First we will make
sure everything is in place for NovaNET to run and then we will attempt to run NovaNET locally. Once
the NovaNET executables are running, we can determine (somewhat) if NovaNET will operate
successfully when all subsystems are attached. If there are execution errors, we must determine what
they are and fix them. When moving from one AlphaServer to another, using different versions of Tru64
Unix, we can experience interoperability issues between the different executables. Some of these
issues can be attributed to compilation issues while others may be less intricate, such as an improperly
formatted configuration file.
By verifying the functionality of the NovaNET software, we also verify communication between the
AlphaServer and the SAN and ensure its proper functionality.
5.2.2.1 Verify Nova Installation
The software that is executed on the Alpha is called Nova. The terms ‘Nova’ and ‘NovaNET software,’
in the context of the Alpha, may be used interchangeably.
Developing Draft : Anthony Garone
Functional Test Plan
5.2.2.1.1
Page 35
3/9/2016
Ensure Proper Directory Structure Exists
Four directories should exist within ‘/nova/’ for software, data, and configuration:

/nova/data

/nova/adm

/nova/bin

/nova/dumps/current
5.2.2.1.2
Ensure Executables Exist
Nova binaries will exist in /nova/bin.
5.2.2.1.3
Ensure Proper Configuration
Configuration for the new Alpha will be copied directly from the Mesa AlphaServer. Since we will only
be changing IP addresses on the two machines (thereby “replacing” Mesa with the new Alpha), we will
not need to reconfigure the new Alpha in any other way.
Not everything nova-related is based on config files. Some binaries needed recompiling to work on the
new Alpha because the compilers between the old and new Alphas were different and the binaries
exhibited shared library issues upon initial execution.
5.2.2.1.4
Ensure Executable Interoperability
When we execute the Nova application, there are ways of determining whether the binaries are
interoperable and able to run at the same time. We have log files to assist us, data dumps from
execution errors, and other tools at our disposal.
5.2.2.2 Verify NovaNET SAN Data
Because the ssh client in the Tru64 Unix distribution on the new Alpha exhibited performance issues,
we will use the freely-distributed OpenSSH application to create an SSH tunnel between the two Alphas
and perform scp commands to copy all relevant data.
5.2.2.3 Verify NovaNET Operation
With the NovaNET executables verified as functional, we can try logging into NovaNET using the Portal
application.
5.2.2.3.1
Log In Using Portal
In order to log in and enter ‘author-mode’, you must use a login within the ‘novanet’ group.
NOTE: The configuration of Portal may need to be updated to log into the developmental Alpha we are
testing.
5.2.2.3.2
Access Notes File
At the ‘author-mode’ screen, type ‘notes’ and hit enter to access the Notes lesson. We can try writing
a note and sending it, but if we’ve gotten this far, the notes should work fine.
Developing Draft : Anthony Garone
Functional Test Plan
Page 36
3/9/2016
5.2.3 NOW Delivery Subsystem
5.2.3.1 NOW Student Login
At the login screen, try to log in using a student login from the NOW student system. If the login
attempt is successful, we can be assured that basic NOW functionality is available and that the NOW
delivery subsystem is responding.
5.2.3.2 ‘now’ Lesson
At the ‘author-mode’ screen, type ‘now’ and hit enter to access the NOW lesson. A window with a
graphical user interface should show up displaying a scrollable field. If this happens, then the NOW
Delivery Subsystem is working fine.
5.2.3.3 ‘testor’ Lesson
The NovaNET system provides a ‘lesson’ that lets you measure the performance of the network (among
other things). You can either run lesson ‘testor’ from your author mode signon or log in as user ‘testor’
in group ‘m’ to access the testor functionality directly.
Option ‘a’ draws a grid that fills the entire Portal screen then outputs (draws) circles to the grid as fast
as possible. This test exercises forward bandwidth.
Option ‘p’ automatically sends a keystroke to the central system every 5 seconds or so and tracks the
round trip time in milliseconds, minimum, maximum, and standard deviation. This test provides a way
to gauge the performance of the network.
5.3 End-User Tests
6 Checklists
6.1 Subsystem Validation Tests
6.1.1 Central Processing Subsystem
6.1.1.1 Verify Remote Login to Alpha

Remotely connect via SSH to AlphaServer
The hostname of the ES80 in Champaign is cdc-es80-1 and its IP address is 159.182.104.104. If
connecting to a different Alpha, write the IP address of the AlphaServer here:
_______._______._______._______
Test completed by: ________________________________
Test completed on: ___/___/______ at ___:___
6.1.1.2 Verify Disk Storage Connectivity Between Alpha and SAN

Remotely connect via SSH to AlphaServer. The hostname of the ES80 in Champaign is cdc-es801 and its IP address is 159.182.104.104. If connecting to a different Alpha, write the IP address
of the AlphaServer here: _______._______._______._______
Developing Draft : Anthony Garone
Functional Test Plan

Page 37
3/9/2016
Execute ‘df –k | grep /nova/data/disks’. Look for something similar to:
nova_domain3#nova_data_disks 94367744
55369151 38988992
/nova/data/disks
59%

Execute the following as the root user:
cd /nova/data/disks
touch testfile
ls –l testfile
rm testfile
Ensure that the testfile was created and removed.

Execute the following as the root user:
hwmgr –show scsi –full
Find the sections that describe dsk5 and dsk6. Note the ‘PATH STATE’ portion of those
disks—all paths should say ‘valid’. If any say ‘stale’, there is either a problem with the path
or with the LUN itself.

Execute the following as the root user:
mount
Check that the nova_domain1, nova_domain2, and nova_domain3 file domains are
mounted. To do this, look through the output of the mount command and compare it to the
following, looking specifically for (rw) after each file domain:
root_domain#root on / type advfs (rw)
/proc on /proc type procfs (rw)
usr_domain#usr on /usr type advfs (rw)
var_domain#var on /var type advfs (rw)
nova_domain1#nova1 on /nova type advfs (rw)
nova_domain2#nova_var on /nova/var type advfs (rw)
nova_domain2#nova_data_tarfiles on /nova/data/tarfiles type advfs (rw)
nova_domain2#nova_restore on /nova_restore type advfs (rw)
nova_domain3#nova_data_disks on /nova/data/disks type advfs (rw)
Test completed by: ________________________________
Test completed on: ___/___/______ at ___:___
6.1.1.3 Verify Read/Write to SAN

Access Navisphere via http://159.182.104.70.

Ensure there are no faults on the main screen.
Test completed by: ________________________________
Test completed on: ___/___/______ at ___:___
6.1.1.4 Verify Connectivity to Primary NovaNET Components

Ping the NIUs [NIU1, NIU2, NIUX], INIU [INIU4, INIU5], Osiris [cdc-1955-5, -6, -7, -8, -9], and
CNET machines [stargate, mail, medusa, esker]
Test completed by: ________________________________
Test completed on: ___/___/______ at ___:___
Developing Draft : Anthony Garone
Functional Test Plan
Page 38
6.1.1.5 Prepare the Alpha for NovaNET Installation

Install additional software packages: java14 & libraries

Create directories in /nova for software, data, and configuration:
o
/nova/data
o
/nova/adm
o
/nova/bin
o
/nova/dumps/current
6.1.1.6 Install/Verify NovaNET Installation

Copy binaries, data, and configuration from Mesa using scp via OpenSSH

Start up system

Log in to NovaNET

Verify TUTOR execution
6.1.2 Backup and Restore
6.1.2.1 Backup and Restore on Alpha


Daily Backups
o
Run by cron
o
Restore as per current procedure
Exact Backups
o
TO DO: John will come up with run method
o
TO DO: John will come up with restore method
6.2 Author-Mode Tests
6.2.1 Central Processing Subsystem
6.2.1.1 Verify Nova Operation

Check for packs being loaded: lesson ‘ldr’

Check CNET: lesson ‘netmon’

Check NOW: lesson ‘nowprod’

Check catalog: lesson ‘catalog’
Developing Draft : Anthony Garone
3/9/2016
Functional Test Plan
Page 39
3/9/2016
6.3 End-User Tests
7 Appendix
7.1 List of Tests Performed in this Document
Below is a table of all tests documented in this test plan. An explanation of this table can be found
after the table.
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
Subsys
CP
CP
CP
CP
CP
CP
CP
CP
CP
NOW
NOW
NOW
NOW
Oracle
Oracle
Oracle
Oracle
Oracle
Oracle
MySQL
MySQL
MySQL
DMZ
DMZ
DMZ
26
27
28
29
30
31
32
33
34
35
36
37
38
39
DMZ
DMZ
DMZ
DMZ
DMZ
DMZ
DMZ
DMZ
DMZ
Telco
Telco
Telco
Telco
Telco
Test
Verify remote login to Alpha
Verify disk storage connectivity
Verify read/write to SAN
Verify Alpha connectivity to primary NovaNET components
Verify proper directory structure for Nova
Verify Nova executables' installation, configuration, and interoperability
Copy data and verify data matches between SANs on old and new Alphas
Verify Nova operation
Portal: Access notes files
Verify remote login to all NOW servers (cdc-1955 farm, Osiris, Horus)
Verify existence of NOW apps on all NOW servers
Verify connectivity between NOW servers and Alpha
Verify NOW performance: 'now' lesson
Verify remote login to all database servers and multiplexers
Verify functionality of Oracle installation
Verify connectivity and functionality of Oracle SAN
Migrate/Verify data from production to testing environment
Verify connectivity between NOW app servers and multiplexers
Verify connectivity between multiplexers and Oracle DB servers
Verify remote login to database server and multiplexer
Verify functionality of MySQL
Migrate/Verify data from production to testing environment
Verify remote login to all components
Verify serial card in ELAU
Verify configuration file on ELAU
Verify existence/proper location of run files, support scripts, and logs on
INIU
Verify 'nc' and other processes on INIU
Verify Lynx installation and connectivity to Internet
Verify functionality of Stargate
Verify internal connectivity to DMZ
Verify external connectivity to DMZ
Verify Stargate/Lynx in Portal: 'lynx' and 'internet' lessons
Verify Portal connectivity through ELAU(s)
Verify Portal connectivity through INIU(s) ***
Verify physical connections on telco equipment
Verify functionality status of telco equipment (hardware LEDs)
Verify functionality of NIUs (hardware LEDs)
Verify NIU software location and running status
Verify NIU configuration files
Developing Draft : Anthony Garone
Best
0
5
5
15
5
5
300
15
5
15
30
20
5
10
0
0
0
0
0
5
20
45
0
0
0
Optimal
5
10
10
30
15
15
360
30
10
25
45
45
10
20
0
0
0
0
0
10
40
90
0
0
0
Worst
60
60
60
60
30
120
720
60
30
45
90
60
45
45
0
0
0
0
0
45
60
120
0
0
0
0
0
0
0
0
0
5
5
5
60
10
10
20
0
0
0
0
0
0
0
10
10
10
90
30
30
40
0
0
0
0
0
0
0
30
30
30
180
120
120
60
0
Pha
2,3
2,3
2,3
2,3
2,3
2,3
2,3
2,3
2,3
2,3
Functional Test Plan
40
41
42
43
44
45
46
47
48
Telco
Telco
Telco
Telco
Telco
BU
BU
BU
BU
Page 40
Verify NIU VCOM boards
Verify hardware/software state of LUs
Verify telco via Portal: 'comstat' lesson
Verify Portal connectivity through telco
Loopback tests via Atlas
Verify existence/functionality of BU scripts on Alpha
Verify backup/restore procedure
Verify MySQL backups/restores
Verify Oracle backups/restores
3/9/2016
0
0
10
10
0
60
30
0
0
0
0
15
15
0
75
45
0
0
7.1.1 Table Legend
Subsys is short for “Subsystem”. The Subsystem abbreviations are as follows: CP => Central Processing
Subsystem, NOW => NOW Delivery Subsystem, Oracle => NOW Oracle Database Subsystem, MySQL =>
NOW MySQL Database Subsystem, DMZ => DMZ Customer-Facing Subsystem, Telco => Leased-Line
Subsystem, BU => Backup and Recovery.
The “Best,” “Optimal,” and “Worst” columns represent the estimated time (in minutes) for completion
of the related test.
The “Phases” column represents which phases of this project in which the test will be performed.
P2Round is the round, or collection, of tests to which that particular test belongs. Please see the
proceeding section (section 7.1.2) for more information on testing rounds.
Tester is a column representing who will perform each test. JH => John Hegarty, RT => Ray Thomsen,
SL => Steve Litt, FL => Frank L’Odense, RI => Renee Inman, AG => Anthony Garone, LK => Laryssa
Kachorowsky.
7.1.2 Phase II Implementation Rounds of Tests
The goal of Phase II is to replace the current production AlphaServer (Mesa) with the new ES80
AlphaServer. In order to do this, we will introduce the new AlphaServer into the development
environment by replacing Esker, which is the current development AlphaServer. Once we verify the
functionality of the new AlphaServer as a fully-functional development AlphaServer, we can qualify it
to run as the new production AlphaServer. Phase II implementation will occur on Presidents’ Day
weekend with a go-live date of February 18, 2008.
The benefit of changing nothing but the AlphaServer is that no configuration changes need to occur
except for those relating directly to the AlphaServer. For example, we will be changing the IP address
and server name on the machine, but none of the other components will know that the Alpha has
changed.
7.1.2.1 Round 1: Low-Level Alpha Verification
During Round 1 of Phase II implementation, we will verify that the AlphaServer is functional as a
standalone server. We will verify that we can remotely connect to the machine, verify that it is
connected to and functioning with the SAN, and verifying that the directory structure is in place to
install the Nova software.
7.1.2.2 Round 2: Alpha Connectivity and Configuration
During Round 2 of Phase II implementation, we will verify that the AlphaServer is able to communicate
over our network with all relevant NovaNET subsystems and components, is properly configured to run
the Nova software, has the Nova software installed with the software’s correct configuration, and has
meaningful production-level data to work with on its SAN.
Developing Draft : Anthony Garone
0
0
30
30
0
150
60
0
0
2,3
2,3
2,3
Functional Test Plan
Page 41
3/9/2016
7.1.2.3 Round 3: Nova Software
Round 3 of Phase II implementation will consist of running the Nova software on the AlphaServer
without running into any issues. Because the operating environment for the Nova software is very
technical and has some compiler dependencies, we need to make sure we can get the software running
before we connect any other subsystems and, thus, Portal clients to it.
7.1.2.4 Round 4: Portal Connectivity
Round 4 will ensure that all means of connecting to NovaNET through the Portal application are
available and functional. This involves connecting to NovaNET using a serial data connection (which
tests the NIUs), using an ELAU, and using an INIU. Because the only changed variable is the AlphaServer
in Phase II, any issues in this round of testing will likely have to do with configuration and not with the
communication components. This round of testing will not deal in any way with functionality of Portal
or accessing any features offered by NovaNET.
7.1.2.5 Round 5: Basic Portal/NovaNET Functionality
In Round 5, we will test functionality within NovaNET starting small and working outward. Some of the
more basic features of NovaNET are logging in, accessing lessons, and testing notes files. Some of the
more complicated features of NovaNET are the ‘comstat’ lesson (which employs EMP connectivity to
many NovaNET components), the ‘lynx’ or ‘internet’ lessons (which demonstrate the functionality of
the Stargate and Lynx servers), the ‘now’ lesson (which demonstrates the functionality of the NOW
Delivery subsystem), etc.
7.1.2.6 Round 6: Backup Testing
By the time we reach round 6, we will have verified that NovaNET is fully-function under the operation
of the new AlphaServer. We can now concern ourselves with backing up the restoring data since we
know we have a system that will not compromise the integrity of the data upon which it operates.
7.2 Central Processing Subsystem Software
7.2.1 NovaNET System Software Components
7.2.1.1 Development Tools
COMPASS: An assembler for CDC Cyber mainframe assembly language code (called COMPASS).
Developed as a drop-in replacement for the original NOS-based CDC assembler, with added facilities to
make some translation into native Alpha machine code and calls to native compiled HLL code possible.
It is used to assemble the hundreds of thousands of lines of code comprising the CONDEN and PLATO
programs into object files containing CDC mainframe machine code, which can be linked together into
programs by the LOADER. ~15000 lines of C++ code.
LOADER: A linker/loader for CDC mainframe binaries. Developed as a drop-in replacement for the
original NOS-based CDC loader. Links COMPASS object files together into valid NOS format binaries for
CDC mainframes, which can be loaded by the EMULATOR and executed. The loader has additions to
export portions of the code to be treated by the TRANSLATOR to create native Alpha machine code.
~2500 lines of C++ code.
TRANSLATOR: A program to translate CDC mainframe machine code into Alpha machine code, which
are then loaded into the EMULATOR as shared libraries and via special mechanisms in COMPASS and the
assembly code can be invoked from the emulated CDC machine language. ~5000 lines of C code.
Developing Draft : Anthony Garone
Functional Test Plan
Page 42
3/9/2016
7.2.1.2 Basic Delivery Components
ALARMD: Alarm daemon. This program forwards notifications of exceptional conditions to the remote
alarm board machine where the system operator can view them. ~250 lines of C code.
CONDEN: TUTOR condensor. Compiles TUTOR code – the language used to develop all software on the
NovaNET system – into COMPASS machine code and pcode. The condensor now runs within the
EMULATOR Unix process instead of being a process running under NOS on a CDC mainframe. ~80000
lines of COMPASS code.
EMULATOR: Provides for emulation/execution of NOS executables for CDC mainframes in the Tru64
Unix/Alpha environment. The programs CONDEN and PLATO run within the EMULATOR. ~2500 lines of C
code.
FORMAT: TUTOR output formatter. Originally written in COMPASS, this program has been rewritten in
C/C++ and turns TUTOR outcodes into a stream of the Level One Protocol for display on NovaNET
terminals, which mainly consists of microcomputers running Portal software for Windows and Macintosh
platforms. ~7000 lines of C&C++ code.
FRAMAT: Head end of forward channel of the NovaNET communications network. Takes output from the
FORMAT program, frames it and delivers it, via the NIU and telecommunications network, to site units
and ELAUs in the CERL X-51 protocol – a windowed error-correcting protocol developed for NovaNET at
the University of Illinois. In the reverse direction, the NIU handles the X-51 error correction and the
FRAMAT program takes input from terminals and distributes them to either PLATO via EM data
structures or via a UNIX socket to the OSIRISRXD program. ~10000 lines of C++ code & ~30000 lines of
libraries.
FRX: Serves as a stripped down FRAMAT substitute for users that are currently executing internet
activities (wx, gopher, telnet, and lynx.) ~3000 lines of C code.
IMSGUIDE: The NOW instructor system used by instructors to manage their students. ~125000 lines of
C++ code.
IMSLEARN: The NOW student system used by students to take lessons on NovaNET. ~45000 lines of C++
code.
MASTOR: Processes low-level (sector) disk I/O requests for the PLATO and CONDEN programs, and
provides an interface to NCD for PLATO. ~8000 lines of C code.
MULTIPLEXOR: Serves as platform-independent connection pooling software between Osiris applications
and our Oracle & MySQL databases. ~5000 lines of C++ code.
NCD: CNET daemon. Forwards CNET traffic for other physical hosts via UDP. CNET is the
communications protocol that was developed for TUTOR/NovaNET prior to TCP/IP protocol availability
on the hardware that implemented the system, and CNET is still the only mechanism available for
TUTOR software to communicate to other machines & processes. CNET is now implemented using UNIX
sockets and UDP. ~3000 lines of C code.
NIU: Network interface unit that interfaces to the telecommunications equipment via high-speed
synchronous serial cards, delivers all forward channel data and implements the reverse channel portion
of the X-51 protocol. ~17000 lines of C code.
Osiris library: Provides the GUI, system interfaces, and cross-platform frameworks for the Osiris
applications IMSGUIDE & IMSLEARN. ~130000 lines of C++ code.
Developing Draft : Anthony Garone
Functional Test Plan
Page 43
3/9/2016
OSIRISFILED: Provides access to NovaNET files to Osiris programs. ~4000 lines of C++ code & ~30000
lines of libaries.
OSIRISRXD: Provides an interface from the TUTOR/COMPASS world to off-host Osiris programs, allowing
them to do I/O to user terminals and communicate with TUTOR software programs. ~4000 lines of C++
code & ~30000 lines of libraries.
PLATO: TUTOR runtime. An operating system unto itself, it provides timesharing for thousands of
simultaneous users who run in their own virtual threads of TUTOR code within this program, as well as
a filesystem, memory facilities, timing, and other facitilies. Written in COMPASS and running within the
EMULATOR, certain heavily executed portions of the COMPASS image have been modified so they could
be translated into Alpha machine language by the TRANSLATOR and are loaded into the EMULATOR as
shared libraries. Other small portions have been implemented as C subroutines. The bulk of the code is
emulated COMPASS machine code. ~250000 lines of COMPASS code.
PORTAL: NovaNET terminal software that customers use to connect to and interact with the NovaNET
system. Different code bases for DOS, MacOS, Unix/X11, Windows, and Java. Each version is ~35000
lines of C++ code (except for the Java port, naturally.)
7.3 Hostnames and IP Addresses
7.4 Oracle Database Acceptance Checklist
Pre-Install Checks
Verify base environment
export ORACLE_HOME=/opt/oracle/product/product_version/db_#
export PATH=$ORACLE_HOME/bin:/usr/bin:/usr/ccs/bin:/usr/bin/X11:/usr/local/bin:$PATH
export LD_LIBRARY_PATH=/opt/oracle/product/9.2.0/lib
export ORACLE_TERM=xterm
Verify umask = 022 (umask)
Verify Kernel Settings
Get OS specific kernel settings from Installation document for the installation OS
Verify kernel settings
Standard Directories
Standard ORACLE_HOME /opt/oracle/product/version/db_n
Standard ORACLE_BASE /opt/oracle
Standard Backup Directories
/opt/orabkup
/opt/orabkup/$ORACLE_SID/exports
/opt/orabkup/$ORACLE_SID/HOT.date*
/opt/orabkup/$ORACLE_SID/ARCH.date*
Standard Database Directories
/u/u0n/oradata/${ORACLE_SID} - data
/u/u02/oradata/${ORACLE_SID}/arch - archived redos
$ORACLE_BASE/admin/$ORACLE_SID/bdump
$ORACLE_BASE/admin/$ORACLE_SID/udump
$ORACLE_BASE/admin/$ORACLE_SID/cdump
$ORACLE_BASE/admin/$ORACLE_SID/pfile
$ORACLE_BASE/admin/$ORACLE_SID/create
$ORACLE_BASE/admin/$ORACLE_SID/adump
$ORACLE_BASE/admin/$ORACLE_SID/scripts
Developing Draft : Anthony Garone
Functional Test Plan
Standard Script directories
/opt/oracle/admin/local/bin
/opt/oracle/admin/local/sql
/opt/oracle/admin/local/app
/opt/oracle/admin/local/log
Standard authentication scripts
on_${ORACLE_SID}.sql
exp.par
rman_incremental_hot_backup.parm
Oracle SW Install
oui location /opt/oracle/oraInventory
Enterprise Edition Install - all components
dba group as software owner
* entry in /etc/oratab for SW release
Server Boot Scripts - Unix/Linux - Non RAC
/etc/init.d/dbora file
/etc/rc3.d/s99dbora link
/etc/rc0.d/K10dbora
Standard Cron Jobs
analyze job
export job
RMAN Job
Jobs scheduled ?
Jobs tested ?
Monitoring
GRID Agents installed
GRID Agents reporting in to GRID Server
Verify with Admins that Nagios is setup
Other
Verify Admins sweeping backup directory
Database Creation
Listener configured
Listener resolves instance name
pedtech tnsname.ora update
Wallet update with ORACLE_SID and Server
Create_SYSTEM_NCS4S_Objects.sql
Create Procs for SYSTEM_NCS4S
Standard sys/system password
$ORACLE_HOME/dbs/password file
$ORACLE_HOME/dbs/spfile
Developing Draft : Anthony Garone
Page 44
3/9/2016