From Entrepreneurial to
Enterprise
IT Grows Up
Nate Baxley – ATLAS nbaxley@illinois.edu
Rami Dass – ATLAS rpd@illinois.edu
Learn@Illinois
• A partnership between Colleges of LAS and
Education
• Learning Management System (LMS) based on
Moodle
–
–
–
–
–
Open Source
Released in 2002
Currently at version 2.7
Over 80,000 installations worldwide
http://moodle.org
Learn@Illinois
A brief history
• Moodle came to LAS and Educaiton in 2003
– http://courses.las.illinois.edu
– http://learn.education.illinois.edu
• Implemented on single machines
• 2010 – Sharing a single developer
• 2012 – Service was combined
– http://learn.illinois.edu
• Statistics (Fall 2014)
– 500 courses
– 26,000 enrollments
20,000 daily logins
23,000 unique users
Early Days
Separate Installations
• A single web server in tandem with a separate
database server
• Every year courses were copied manually along
with a new database
• Rosters were managed with daily file uploads or
registration keys sent out via email
• Code was pushed as needed with minimal testing
• Changes were often implemented as soon as they
were ready
Early Days
Separate Installations (cont)
• Support was by email directly to the developers
or service managers
• Monitoring was done through home grown
script
• Logging was decentralized, each host kept logs
locally
• We lacked scalability and flexibility
Growing Up
The Partnership Evolves
• Opportunity to design infrastructure for
scalability and stability
• Code release procedures were tightened
• Support team was expanded
• Partnership became stronger
• Moodle 2.0 release coincided with partnership
– Major code changes
– Revisit old plugins
Growing Up
Evolution of Infrastructure
• Scalable, flexible, and redundant
• Multiple web front ends were deployed
behind a load balancer
• Offloaded scheduled task from the web front
ends to a dedicated server
• Moved from a single individual to a team to
better manage the rapid growth
• Transparent modifications to the systems with
minimal interruptions to the users
Growing Up
Dev & Release Process
•
•
•
•
Improved reliability
Thorough documentation of changes
Reduced the frequency of releases
Increased usage made ad hoc fixes less
desirable
• Established a stricter dev/test/prod
environment
Growing Up
Dev & Release Process (cont)
• Changes tracked in Redmine and bundled into
releases
• Development team expanded to support
other LMS
• CVS and GIT used for code versioning
• Code releases through direct file transfers
Today
Infrastructure
• The database runs on a MySQL cluster with
nodes spread across 3 data centers for high
availability.
• We have several web front ends spanned
across multiple data centers
• The file server replicates to a “hot spare”
server in another data center that can be
manually switched to.
Today
Infrastructure (cont)
• The framework is duplicated to varying
degrees for the dev, test, and shadow systems
• Use Zabbix for monitoring, and get a lot of
information on the health of the systems
keeping us ahead of the curve
• Logging is centralized using Graylog to review
the logs and event notifications
Today
Application/Service
• Code releases are done through GIT
• 5 issue severity levels ranging from “emergency
fix needed” to “annual major upgrades”
• Issues tracked from requirements gathering
through testing and resolution
• Coordination of support team between partners
• Instructional designers at LAS, Education, and
CITL help faculty design and build their courses
Future Growth
• Automatic failover for file server
• Implement configuration management to
automate, standardize, and
document infrastructure changes.
• Future performance increases to handle growing
enrollments.
• Dedicated "admin" web front end, tweaked to
handle power user requests
• Moving beyond building maintenance tools to
building teaching tools
Questions
• Nate Baxley
– College of LAS - ATLAS
– Client Relations Manager
– nbaxley@illinois.edu
• Rami Dass
– College of LAS – ATLAS
– Lead IT Infrastructure Engineer
– rpd@illinois.edu