How We Maintain a Healthy Bb
Environment
Sheila Chandler
Director, Learning Systems
Virginia Commonwealth University
www.ts.vcu.edu
Presentation Overview
• Background
– Who am I?
– Virginia Commonwealth University (VCU)
– VCU Blackboard Environment
____________________________________
How We Maintain a Healthy Bb Environment
Sharing Lessons Learned
Who am I?
• High School – Computer programming (IBM)
• University of Louisville
– Sociology Degree
– First programming job (PDP 11/45 - BASIC)
• Community Hospital of Central California (MUMPS,
BASIC)
• Virginia Commonwealth University (VCU) -Accounting
Degree
• Auditor of Public Accounts – CPA, CISA
• VCU
– Programmed on Web Course in a Box (1998 to 2000)
– Blackboard purchased WCB (2000)
About VCU
• Richmond, VA
• Students: 32,000+
• Faculty:
1,927
• Adjunct:
1,161
• Staff:
2,586
• Monroe Park, MCV, Qatar, NOVA
Bb App Environment
• Bb Application (2000 to present)
– Bb Classic Enterprise (7.3 -> v9 – May 2010)
– Bb Content Management System
• Building Blocks
–
–
–
–
–
–
–
–
Bb Faculty Tools (VCU developed)
CPS Connection (Clickers)
Echo360 Content Creator
Health Check (modified)
Help Desk Tools (VCU developed)
Wimba
SafeAssign
SoftChalk
Courses per Semester
3500
3000
2500
2000
1500
1000
500
0
# of Courses
Fall
Spring
Fall 2002 Spring Fall 2003 Spring Fall 2004 Spring Fall 2005 Spring Fall 2006 Spring Fall 2007 Spring Fall 2008
2001(Cour 2002 (Bb
(5)
2003 (5) (5.5) 2004 (5.5) (6)
2005 (6) (6.3) 2006 (6.3) (6.3) 2007 (6.3) (7.1) 2008 (7.1) (7.3)
seInfo4)
5)
656
612
956
1244
1447
1726
2070
2079
2569
2337
2777
2612
2987
2993
3170
Bb User Environment
Fall 2008
Spring 2009
3319 Courses
2031 Instructors
29688 Students (85%)
3099 Courses
1983 Instructors
28236 Students (87%)
723 Organizations
804 Organizations
Lesson Learned #1
Designate Bb Coordinator and
Support Team
• VCU Support Team (Tribe)
– Technology Services
•
•
•
•
Learning Systems
Help Desk
University Computer Center
Networking
– Center for Teaching Excellence
– Support staff in Schools (School of Nursing, Business, etc)
– Vendor (Blackboard)
Lesson Learned # 2
Recognize Mission Critical
• Path to Mission Critical
– Student demand (85%+)
– Faculty use (email, gradebook,
document storage)
– Administration/Staff found uses
– Keep statistics – show growth
Lesson Learned # 3 - Scale Out
Lesson Learned #4
Request System Audit
Lesson Learned # 5
Monitor and Take Action
• Hobbit (Internal)
• Pingdom (External)
• Custom Scripts
– Check db connections
– Search logs for Oracle errors
– Scan Snapshot files (header & footer)
– Monitor SAN space (set to expand at 90%)
#5 Monitor and Take Action
• System cleanup
– Rotate logs
– Run Bb Cleanbb once a week
– Restart app servers twice a week
• Track usage and react to trends
• Ask Users
– My VCU Portal Polls
– Annual Tech Fair suggestion box
– Talk to Students on Campus
– blackboard@vcu.edu
#5 Monitor and Take Action
• External Listserv
– ASU list
• (BBADMIN-L@lists.asu.edu)
– BIE list
• (BIEACAMDEMIC@lists.blackboard.com)
– Developers
• (BB-OPENSRC@lists.vcu.edu)
• Internal VCU Listserv
– Bb-core
– Bb-mgmt
– Bb faculty list
Lesson Learned #6
Document and Communicate
• Technical
– Bb Diary
– Internal wiki – notes
– BbCMS – Documentation
– Emergency Call list
• End-User
– Help Desk - blackboard@vcu.edu
– Blogs (Tech Services, Bb system-wide,
Bb faculty)
– Learning Systems Newsletter
– Technology Services Knowledge Base
(http://www.ts.vcu.edu/kb/blackboard.html)
#6 Document/Communicate
• Regular Meetings
– Weekly LS meeting
– Monthly Bb Tech meeting (agenda & minutes)
– Monthly Center for Teaching Excellence
meeting
– Periodic meetings with IT Directors of Schools
#6 Document/Communicate
• Special Meetings
– Special Task Force (iptables fix)
• Discovered that
net.ipv4.netfilter.ip_conntrack_tcp_be_liberal = 1
doesn't survive an iptables restart
– Bb TSM bi-weekly meetings
• Periodically request to review Bb tickets
Lesson Learned #7
“No such thing as a SMALL change”
• ALL Changes follow Written Procedures
– Development Environment (maybe multiple
times)
– Test Environment (always multiple times)
– Production
#7
“No such thing as a SMALL change”
• System Maintenance – Regularly Scheduled
– Second and Fourth Friday of the month
– Perform Downtime Checklist
• One week before downtime
• During downtime
• After downtime
– Group IM (Instant Messenger)
– Follow emergency communication plan (if needed)
Lessons Learned Summary
1. Designate Bb Coordinator and Support
Team
2. Recognize Mission Critical
3. Scale out
4. Request System Audit
5. Monitor and Take Action
6. Document and Communicate
7. “No Such Thing as a SMALL Change”
Additional Information
• Additional Information
– Sheila Chandler (sdchandl@vcu.edu)
– Bb Connections site
– http://healthybb.wetpaint.com
• Thank you to…
– Learning Systems Staff
– University Computer Center Staff
– Networking Services
– Center for Teaching Excellence