Deploying Office 365 At UC Merced
Nick Dugan and Todd Van Zandt
UC Merced Information Technology
UCCSC 2013
Introductions

Nick Dugan, Senior Systems Administrator
 ndugan@ucmerced.edu

Todd Van Zandt, Director of Academic Technology
and User Services
 tvanzandt@ucmerced.edu
Agenda
Background
 Selection and Pilot Process
 Infrastructure Deployment
 Migration
 User Support
 Next Steps and Conclusions

Background
UC Merced
Opened 2005
 Population in Fall 2012

 5,800
students
 140 faculty
 1,000 staff

10,000 students by 2020
Background

Technology
 ~8,000
active accounts
 ~10,000 alumni / former student accounts
 Lots of infrastructure from ~2005
 Sun
Messaging Server
 Oracle Calendar
 SPARC Hardware
Background

Sun -> Oracle == $$$
SPARC hardware refreshes
 Maintenance and license costs


Oracle Calendar EOL
Heavy use of Outlook with Oracle Calendar connector
 Incompatible with Outlook 2010
 Client software not supported in Mac OS X Mountain Lion


Email Growing Pains
Quotas, Compromised accounts, SMTP blacklisting
 Aging webmail client

Background

Minimally Utilized Windows Infrastructure
 Identity
Management: Sun Waveset
 LDAP: Sun Directory Server

Active Directory
 Fed
from IDM
 Lab support
 Printing
 NAS Authentication
 Minimal account attributes populated
Selection and Pilot Process

Selection Committee
 Convened
November 2011, charged with developing
functional requirements and recommending a solution
 Representatives from faculty, staff, and student
populations
 Locally hosted options ruled out
 Microsoft vs. Google
 Work completed March 2012
Selection and Pilot Process

Microsoft vs. Google
 No
clear consensus among committee members
 Subjective
and objective scoring gave a slight advantage to
Office 365
 Both
better than current solution
 High level of satisfaction with both
 High satisfaction with Lync among staff
 Final decision left to IT after consideration of committee
report and analysis of costs
Selection and Pilot Process

IT Review
 March
– May 2012
 Microsoft reduced price of A2 plan to $0
 Evaluated
technical requirements for
implementation
 Reviewed committee input
 Spoke with other UC CIOs and IT organizations
 Final recommendation for Office 365 delivered to
Chancellor and Provost
Infrastructure Deployment

Consultant Engagement
 Decision
made to employ professional services for
tenant deployment and migration support
 Interviewed and vetted approx. 10 companies in July
2012
 Selected Cloudbearing, initial engagement Sept. 2012
Infrastructure Deployment

Active Directory Enhancements
 Domain
Controllers in four physical locations
 (Merced

x2, Berkeley, Fresno)
ADFS Servers
 2x
ADFS, 2x ADFS proxy, load balanced across two
physical locations

DirSync Server
 HA
deployment not possible
Infrastructure Deployment

Why ADFS and DirSync?
 Apprehension
about pushing passwords to the cloud
 IDM Already fed Active Directory
 Infrastructure vs. custom development trade-offs made
sense
UC Merced AD FS Deployment
September 2012
Internal Domain
ucmerced.edu
Wpfread01
192.35.211.235
UC Merced
AD ID
Microsoft
Federation
Gateway
Wpdirsync01
169.236.80.175
Wpadfsp01
10.15.4.26
Wpadfs01
169.236.80.174
F5 BIG-IP 1600 NLB
adfs.ucmerced.edu
Inbound Mail Flow during pilot phase
Mail flow between on-premise and
cloud mail hosts
Wpadfs02
169.236.11.108
AD FS authorization traffic
Recipient synchronization from
on-premises to Office 365
Wpadfsp02
10.15.4.27
mail2
169.236.253.81
Object Custom Filter Sync
·
User Attribute Extension15=0365
·
OU=0365 (O365 Groups Container)
·
OU=People (User Container)
MX Record
lpvmg02.ucmerced.edu
lpvmg01.ucmerced.edu
mg03.ucmerced.edu
Auth
Provisioning
ADFS
Active
Directory
DirSync
Shibboleth
IDM
Portal
Office 365
PowerShell
On-site bulk mailer
Mailman
Clients
Mail
Infrastructure Deployment

Lesson – Time is everything
 ADFS
is sensitive to clock skew
 ADFS Proxies are not domain-joined by design
 No
automatic clock sync with DCs
 Small
time discrepancies lead to authentication outages
 Proxies
need reliable NTP
Infrastructure Deployment

Identity Management Integration
 Sun
Waveset (Java) on Solaris
 EOL, Replacement in 2014
 Office 365 == PowerShell
 IDM provision/deprovision events now trigger licensing
actions on a remote PowerShell server
 Lots of waiting built into process due to (un)timeliness of
events in o365 cloud
 Account provisioning went from real-time to “an hour or
two”
Infrastructure Deployment


Shibboleth / SSO integration
Goal: Never see the ADFS sign-on page, OWA
integration into campus portal
Infrastructure Deployment





Shibboleth configured as a relying party in ADFS
ADFS Home Realm Discovery page modified to
bypass IdP selection box
ADFS Logout handler modified to ignore SAML
errors and redirect to global logout page
We will help
office365@ucdavis.edu
Infrastructure Deployment

Dynamic Mailing Lists
Sun Messaging does this well
 Office 365 does this… differently.

Limited to default attributes
 We have dynamic lists based on lots of attributes we don’t push to
AD


Interim Solution: maintain Sun Messaging Server at
lists.ucmerced.edu


Distribution lists in o365 forward to @lists for expansion
and ultimate delivery back to o365
Long-term Solution: Next-Gen IDM with proper group
management
Migration - Email

Approach and timing
1.
2.
3.
4.
5.
Early adopters, opt-ins (early-mid December)
Faculty and Staff by affiliation/geographic location
(Jan 2-20)
Students alphabetically (Jan 14-25)
Mailing lists (Feb 2013)
Alumni (March-April 2013)
Migration – Email

Technical considerations
 Respecting
user settings
 Forwarding
 Vacation
 Messages
 Identify
messages (didn’t do)
that won’t migrate (>25 MB)
and notify
Migration - Email

Technical Process
 Set
DirSync flag in AD so accounts are pushed to Cloud
 Assign licenses using PowerShell (not too early)
 wait
 Copy
forwarding settings using PowerShell
 Change LDAP routing address
 user@merced.onmicrosoft.com
 Set
migration password
 Migrate mail
 Remove migration password
Migration – Email

Tool #1: Office 365 IMAP Migration Tool
 Pros:
 Free
 Fast
(not subject to the same bandwidth throttling as non-MS
tools)
 Easy
 Cons:
 ZERO
reporting/logging.
 Only provides a count of “skipped” messages for each
account – not which messages or why they were skipped.
Migration - Email


Lesson: Never trust a tool that has no logging.
Half-way through faculty/staff migration, number
of skipped messages went through the roof
 Hundreds
or thousands on some accounts, zero on others
 All attempts at resolution failed
 Session/resource
issue with local IMAP server? – No
 Too many concurrent migrations? – No
 Microsoft able to provide an explanation or resolution? – No
 Panic? – Yes
Migration – Email

Migration Tool #2 – MigrationWiz
 Contract
with Cloudbearing provided up to 300
mailbox licenses
 Pros:



Excellent Reporting Capabilities
Excellent Support
Easy to incorporate into migration workflow
 Cons:


$11/mailbox == $$$$
Subject to Microsoft inbound traffic throttling
Migration - Email

Lesson: Ask what other people are doing first.
 office365@ucdavis.edu
Migration - Email

Migration Tool #3: imapsync
http://imapsync.lamiral.info/
 Pros:
 $50
(might as well be free)
 Excellent support
 Great reference from UCSB
 Cons:
 Required
a bit of work to incorporate into our migration
workflow
 Subject to Microsoft inbound traffic throttling
Migration - Email

Schedule, bandwidth, timings
 “Just
in time” mailbox provisioning was difficult
 Start data migration by 10pm, run until done
 Largest migration batch was ~1,100 accounts
 40
simultaneous connections
 Generally finished by 8:00am
Migration - Email
Lesson: Your carefully constructed process will
experience its worst breakdown the day you migrate
the Chancellor.
Story
Migration – Email

Jan 3, 2013 – Migration of Chancellor’s office
 8:00am
– Departmental account identified that did not
provision successfully (sAMAccountName >20 chars)
 10:00am – Sysadmin attempts to rectify problem by
deleting and then re-provisioning account to Active
Directory
 Sysadmin accidentally deletes ou=People instead of
individual account
Migration – Email

Recovery
 First
thought – pull the (virtual) network plug on one of
the DCs before changes propagate
 Too
late
 Second
thought – shut down DirSync server NOW
 Preserved
accounts and mailboxes in cloud tenant
 ADFS non-functional, so as auth tokens expired, users lost
access
 Third
 No
thought – Reprovision from IDM?
IDM group management, share permissions would be lost
Migration - Email

Recovery
 Decision
made to restore from tape and rebuild AD
 Backups
were good, but VM restore process was failing
 3:00pm – Support case with Symantec
 Overnight – work case with Symantec, upgrade server and
client software
 Jan 4 morning – single DC restored
 Windows sysadmins rebuild AD forest
 Jan 5 9am – Full functionality restored
Migration – Email

Lessons:
 AD
Recycle Bin would have made this a non-issue
 Requires

2008 R2 domain functional level
UCM was at 2003
 Verify
backups (AND restores)
 Prepare for the worst when migrating campus
executives
Migration - Calendar


Cloudbearing contract included migration of
calendar data through third party (CalMover)
All calendar data migrated to Office 365 at the
same time
 Thursday,
Jan 17 5:00pm – Oracle Calendar database
extracted and uploaded to CalMover
 Friday, Jan 18 – receive PSTs from CalMover
 Fri-Sun – import PSTs to Office 365
 Monday Jan 22 – Office 365 official UCM Calednar
Migration - Calendar

Issues, side effects
 Modification
or cancelation of migrated meetings
would not notify attendees
 Contacts, tasks, and other non-calendar data was
excluded from import
 Resource delegates, sharing and booking permissions
did not migrate
Migration – Calendar

Timing, speed, historical events
 Cloudbearing
recommended limiting historical data to
12 months
 PST
import could be very slow, might not finish over weekend
window
 Individuals needing complete historical data would be
accommodated through other means of import/export
 Application/service
updates made the import much
quicker than anticipated
 Allocated
3 days, completed in 6 hours
User Support






Campus Communication
Pilot Support
Migration Support
Documentation
Training
Client Issues
User Support - Communication




Campuswide announcements
News articles
MSOs
Info sessions


Record and make available
Inform about
Benefits
 Cons / Changes (more important)
 Process , expectations


Direct emails to users at the time of migration
User Support - Communication

Website
 Large
Banner to draw attention
 O365 project section included
 Migration
schedule
 Configuration Guides

List by device, not just service or client
 Training
& Getting Started Guides
 Explanation of what is coming (large mailboxes, mobile
support) , but also limitations and differences
 Update
Guides
all old information pages, aside from Config
User Support - Pilot

Dedicated a technician to assist with installs
Did not provide a lot of help
Asked pilot group to review draft documentation
IT Student Employees
Engaged Resident Assistants (additional support)

Lessons Learned




 Needed
to have created formal testing plans
 Needed to have been more engaged in support
User Support - Migration


Get Extra Help!
We used temps, technically savvy dept staff, and IT
students and staff

Dedicated 2 (of 5) Desktop Support technicians to this effort
once migrations started


2 DSS departed as we were starting
Hired 4 temps to help with migration appointments
First used them to create and finalize documentation for various
client set ups
 Followed up on problems, after migrations completed for the day

Identify technical leads in departments and target for early
adoption
 Student Technology Consultants
 Interested IT staff

User Support - Migration
 Temps:
 The
Bad: Not as engaged, less reliable, had to replace two
 The Good: Easy to hire, able to release when not needed,
trained for the future, targeted at O365 tasks
 Suggestions: Interview more so you have extras “on-hand”;
Have projects lined up for them in downtime; Keep an eye
on them
User Support - Migration

Coordinated with Depts (opt-in)
 Took
a LOT of effort
 Difficulties
 60-120
with scheduling
per group;
 Only
so many depts on a day
 Plus additional one off early adopters
 Coordinating postponements for out of office users
 Allowed
us to test our processes and improve
documentation
User Support - Migration

Informed Depts for mass moves
 Chose
by location; focused support to an area
 Tables in the lobbies (rarely used)
 Self-help documentation was good



Morning appts for high profile groups
Most done within 4-6hrs
Everyone wanted help immediately
 OWA

was an option
Many users (faculty and lecturers) not present
User Support - Migration

Staff
 Very
interested; needed help and assistance
 Many Windows Outlook users
 Remove
Outlook Oracle Connector
 Reattach Personal Folders and Archives
 Import Address Books and Autocomplete lists
 Upgrade

to Office 2010 (became too lengthy)
Faculty
 Many
were out of office
 Many self-configured their clients
User Support - Migration

Students

Mostly web access users so just adapted to using OWA

Or still forwarded their email to Gmail, etc.
Switched webmail in Campus Portal if migrated
 Did not read email to understand process; if went to direct
to MyMail link


Migration of lots of students took longer for data to copy, new
mailbox looked empty except for new
Offered trainings, did not attend
 More critical for Grad students (instruction and research)


Finally moved them en masse; should’ve planned that from start
User Support - Documentation

Documentation
 Configuration
of Clients
 Outlook,
Thunderbird, MacMail, Mac Outlook
 What about calendar for Oracle Calendar Client users
 What about Lightning, Mac Calendar?
 Configuration
of Mobile Devices
 Use of new systems (OWA)
User Support - Documentation

Documentation
do well documenting for Outlook 2007  2010
 Didn’t do well documenting Outlook calendar for old
Oracle Calendar client users
 Documenting the use of Oracle Calendar web client for
early adopters who were Outlook users
 Didn’t
User Support - Training

Training
 Pre-migration
 What
to expect, limitations and differences, overview of
OWA and Mac/PC Outlook
 Post-migration
 OWA
 Calendar
 Lync
 iOS
 Dept
or Individual Request
 Specific
questions
User Support – Client Issues

Calendar

Mac Outlook & Outlook 2007
Viewing & Sharing calendars
 Acting on calendars in Mac Outlook (selected vs. checked)


Client differences for viewing multiple calendars

Overlay, Side-by-side, Both
Setting delegates; only Outlook client or powershell
 Needed to have done more testing with delegates and
room calendars to understand needs and functionality

The wonderful subculture of Exec Administrative Assistants
 Breaking bad processes established by deficiencies in previous
Calendar


Adding Holidays
User Support – Client Issues

Email
Forwarding to multiple users
 Away Messages when Forwarding is set
 Creation of Distribution Groups (limiting)
 Attachment / Message Size limits
 Limits on Mass Mailing to unique accounts
 MacMail – sync stops
 Thunderbird – archive issue
 iOS – disappearing messages
 Outlook – Unread Email and Sync logs & Modification
Resolution


Recovering deleted emails and calendar appointments
User Support – Client Issues

OWA
Different on mobile browsers
 Settings are limited on mobile


Lync

Mac client
Login issues
 Features not as robust


Considerations for default presence; All visible or All
offline?
All visible – encourages use of enterprise IM
 All offline – Faculty and Administrators not public
 Can’t separate students easily if all on the same tenant

What’s Next

“Wave 15” Service Upgrade
 Back-end
upgrade to 2013 Office products
 Significant web UI changes
 Just
 Some

in time to make our new documentation obsolete
time between now and end of year
Self-service web portal for resource management
and calendar delegation
 Changing
booking options for resources
 Additional means of managing calendar permissions
for non-Outlook users
Conclusions

What have we gained?
 Modern
messaging and calendaring platform
 25x quota increase
 Enterprise IM
 Proper mobile device support
 +1 critical enterprise service (ADFS)
Conclusions

Dollars and cents
1
FTE freed to work on other projects
 +6 Windows servers (VM)
 -3 SPARC servers
 $17,000 annual hardware/software maintenance
 Money we won’t spend
 Hardware
refreshes (servers, SAN)
 Ongoing (increasing) Oracle licensing costs
Conclusions

What have we lost?
 Control
 Upgrades
and changes (announced and otherwise) happen
when they happen
 Outages
Questions?

ndugan@ucmerced.edu
tvanzandt@ucmerced.edu

office365@ucdavis.edu
