Systems Integration and Data
Management
Presented By Greg Devine
Director, Higher Education
Blackboard Consulting
April 14, 2005
Welcome
• Session Objectives:
– Introduce 2-3 key points
• Innovation
– Insert
• Results/Outcomes
– Insert
Blackboard® Snapshot Integration
An overview of the process, tools and operations
For Blackboard Learning System™ (Release 6.2)
• What Data = Which Files
• Snapshot Components
• Snapshot Tool
– Modes, Operators, Syntax
•
•
•
•
Data Sources and Data Source Keys
Field Ownership
Remote Snapshots
Snapshot Controller
IMS Entities
• Integration Entities = Snapshot Files
• Enterprise Data
People
Memberships
Groups
COLLEGE
ENR_FA02
CRS_FA02
LAW
ENR_SP03
CRS_SP02
USR
ENR*
STAFF
ORGMEM
CRS
ORG
Other Data Types
• Additional Enterprise Data
Content
System
Roles
PORTAL_ROLE_MEM
Users
Portal Roles
Course and Organization Catalog
Courses & Organizations
Institution Roles
courselink_
FA02
category
courselink_
SP03
ORG_CATEGORY_MEM
CRS_CATEGORY_MEM
ORG_CATEGORY
CRS_CATEGORY
File Formats
• XML
<PERSON>
<SOURCEDID>
<SOURCE>Blackboard University</SOURCE>
<ID>39450210223</ID>
</SOURCEDID>
<USERID>swang</USERID>
<NAME>
<FN>Mr. Stanley Wang Jr.</FN>
<SORT>Wang,Stanley</SORT>
<NICKNAME>Wang,Stanley</NICKNAME>
<N>
<FAMILY>Wang</FAMILY>
<GIVEN>Stanley</GIVEN>
<OTHER>Franklin</OTHER>
<PREFIX>Mr. </PREFIX>
<SUFFIX>Jr.</SUFFIX>
</N>
</NAME>
<DEMOGRAPHICS>
<GENDER>2</GENDER>
<BDAY>1959-01-01</BDAY>
</DEMOGRAPHICS>
<EMAIL>Swang5@Blackboard_university.com</EMAIL>
<TEL teltype="1">3104591276</TEL>
<TEL teltype="2">3104591200</TEL>
<ADR>
<STREET>Twin Oaks Valley Rd</STREET>
<STREET>attn: S. Wang</STREET>
EXTERNAL_PERSON_KEY|USER_ID|SYSTEM_ROLE|INSTITUTION_ROLE|FIRSTNAME|LASTNAME|EMAIL
<LOCALITY>San Marcos</LOCALITY>
0142270|sait.tarhan|none|role_11|Sait|Tarhan|sait.tarhan@college.edu
<REGION>CA</REGION>
0001129|marcha.hunley|none|role_11|Marcha|Hunley|marcha.hunley@college.edu
<PCODE>92096-0001</PCODE>
0001613|steven.yelton|none|role_11|Steven|Yelton|steven.yelton@college.edu
<COUNTRY>US</COUNTRY>
0003970|dorothy.mann|none|role_11|Dorothy|Mann|dorothy.mann@college.edu
</ADR>
0099821|sherry.hutton|none|role_11|Sherry|Hutton|sherry.hutton@college.edu
<DATASOURCE>Blackboard University</DATASOURCE>
0000311|gary.webster|none|role_11|Gary|Webster|gary.webster@college.edu
<EXTENSION>
0173103|carla.gesell-streeter|none|role_11|Carla|Gesell|carla@college.edu
<X_BB_SYSTEMROLE>0</X_BB_SYSTEMROLE>
0000612|olivia.watts|none|role_11|Olivia|Watts|olivia.watts@college.edu
<X_BB_INSTITUTION_ROLE>0</X_BB_INSTITUTION_ROLE>
0000381|robert.eveslage|none|role_11|Robert|Eveslage|robert.eveslage@college.edu
<X_BB_STUDENTID>144532</X_BB_STUDENTID>
0000987|debra.lierl|none|role_11|Debra|Lierl|debra.lierl@college.edu
<X_BB_PASSWORD>rpeterson</X_BB_PASSWORD>
0225494|robert.nields|none|role_11|Robert|Nields|robert.nields@college.edu
</EXTENSION>
0000982|charalee.allen|none|role_11|Charalee|Allen|charalee.allen@college.edu
</PERSON>
0295293|david.hoctor|none|role_11|David|Hoctor|david.hoctor@college.edu
0001393|colleen.meyer|none|role_11|Colleen|Meyer|colleen.meyer@college.edu
00294456|clark.stull|none|role_11|Clark|Stull|clark.stull@college.edu
0000577|julie.webster|none|role_11|Julie|Webster|julie.webster@college.edu
0074937|george.armstrong|none|role_11|George|Armstrong|george.armstrong@college.edu
– All data in one file
– Reusable
– Standardized (IMS+)
• Delimited (preferred)
– Easy to Manipulate
– Multiple Files
– Smaller
Snapshot Integration Components
• 3 Primary Software Components
– Generator (client developed)
– Tool (Blackboard Learning System core)
– Controller (Blackboard Consulting deliverable)
Snapshot Controller
SIS
SIS
SIS
Snapshot
Generator
Snapshot
Files
Snapshot
Tool
Blackboard
Default Snapshot Configuration
Administrative Systems
Blackboard Academic Suite™
Snapshot Controller
SIS
SIS
SIS
Snapshot
Generator
FTP
Snapshot
Files
Snapshot
Tool
JDBC
Database
Remote Snapshot Operations
Campus System(s)
Blackboard Client
Blackboard Learning System
Snapshot Controller
SIS
SIS
SIS
Snapshot
Generator
FTP
Snapshot
Data
Files
Snapshot
Tool
Application
SOAP
Database
JDBC
On Campus
Remote (ASP)
Snapshot Security
Campus System(s)
Blackboard Client
Blackboard Learning System
Snapshot Controller
SIS
SIS
SIS
Snapshot
Generator
Snapshot
Data
Files
Snapshot
Tool
Application
HTTPS
SFTP, SCP
On Campus
Database
VPN
Remote (ASP)
Data States
• Available/Unavailable vs. Enabled/Disabled
– Availability (Y/N) is set via the GUI, Snapshot or
Event
– Row Status (enabled/disabled) is set via
snapshot only.
Available_Ind
Row_Status
User 1
Y
Enabled
User 2
N
Enabled
User 3
Y
Disabled
User 4
N
Disabled
See Integration Manual or Data Planning Workbook for
detailed behavior of users, enrollments and courses.
Snapshot Tool - Modes
• Manual (MANUAL)
– Simplest mode, adds and updates records
– Used to manage partial data sets.
• Snapshot (SNPSHT)
– Adds, updates and disables records
– Used to manage full data sets.
• Remove (REMOVE)
– Removes records
• Copy Into* (COPYINTO)
– Copies content between courses or organizations
Snapshot Tool – Modes
• Adding Records
User_ID
Available_Ind
Row_Status
User1
Y
enabled
User1
User2
Y
enabled
User1
User3
Y
enabled
Snapshot File
Adding records is the same in
both Snapshot and Manual modes
User_ID
ADD
Available_Ind
Row_Status
Bb
Blackboard Database
Snapshot Tool – Modes
• Updating Records
User_ID
Available_Ind
Row_Status
User1
Y
enabled
User2
Y
User3
User4
Updating records is the same in
both Snapshot and Manual modes
User_ID
Available_Ind
Row_Status
No Change
User1
Y
enabled
disabled
update
User2
Y
N
enabled
update
User3
N
Y
Y
enabled
No Change
User4
Y
Snapshot File
Bb
disabled
enabled
enabled
enabled
Blackboard Database
Snapshot Tool – Modes
The difference between Manual and
Snapshot is seen when records are no
longer found in the file.
• Missing Records
User_ID
Available_Ind
Row_Status
User_ID
Available_Ind
Row_Status
User1
Y
enabled
User1
Y
enabled
User2
Y
enabled
User2
Y
User3
Y
disabled
enabled
Bb
User4
Y
enabled
disable
ignore
User4
Y
Snapshot File
enabled
enabled
Blackboard Database
Manual mode just adds and updates.
Snapshot mode disables missing records.
Snapshot Tools
• ..\blackboard\apps\snapshot
– \bin
• DSM – data source manager
• SNAPSHOT – snapshot tool
• SNAPSHOT_OVERRIDE – snapshot tool
– \config
• ENV - connection info, class path and connection type
– \data
• SNAPSHOT.PROPERTIES – primary snapshot configuration
• IMS-EP01.dtd – DTD for XML snapshots
Snapshot Tools – Command Lines
snapshot -V <server name> -f <operation> -t <data file> -C <properties file>
 snapshot –V bbls.abc.edu –f CRS_SNAPSHOT –t
..\data\SIS_CRS_SP2003 –C
..\data\snapshot.properties
 snapshot –V bbls.abc.com –f USR_MANUAL –t
..\data\SIS_USR
-C ..\data\snapshot.properties
 snapshot –V bbls.abc.org –f STAFF_REMOVE –t
..\data\SIS_MEM_FA2003 -C..\data\snapshot.properties
 snapshot –V bbls.abc.gov –f CRS_COPYINTO –t
..\data\SIS_CRS_SU2002 –C
..\data\snapshot.properties
Snapshot Tools – Snapshot Override
– Snapshot override allows you to override settings
from the properties file.
 snapshot_override “-Ddata.source.key=DL_CRS” –V
bbls.abc.edu –f CRS_SNAPSHOT –t
..\data\SIS_CRS_SP2003 –C ..\data\snapshot.properties
 Snapshot_override “-Dmax.error.count=0” –V
bbls.abc.com –f USR_MANUAL –t ..\data\SIS_USR
-C ..\data\snapshot.properties
Snapshot Properties
• Operations Properties
– delimiter, batch size, error
count
– data source key
• Ownership
– bb.controlled fields
• Course Copy Operations
– content areas, settings,
memberships
# Maximum errors to process.
# If you want all valid records to be processed, use -1 which indicates unlimited
# If you want an all or none transaction, use 0
# Otherwise, pick some reasonable number of errors
max.error.count=-1
# Snapshot process settings
# delimteter used for parsing snapshot files
data.delimeter=|
# Used to denote datasource to use for operation.
data.source.key=SIS
# Determines if password field should be encrypted
encrypt.password=Y
# Denotes string used to mark the beginning of errors
error.delimeter=(!)
# Determines character used to escape delimeter
escape.character=/
# Designates flag to handle header validation for configuration property file
header.validation=Y
# Determines if logged information should be sent to console.
log.stdout=N
# Designates flag to establish handling of enumerated values.
parse.allow.default=N
# Designates maximum size of database transaction
snapshot.batch.size=300
# Used to configure pause between persistence actions
wait.length=-1
# Snapshot controlled field settings for Blackboard override of data
# fields for category entities
category.bb.controlled.fields=
Snapshot Properties - Ownership
• Ownership = Control
– Fields “owned” by Blackboard are not updated by
the snapshot process.
– Fields not owned by Blackboard may be
overwritten by the snapshot.
• Ownership properties are set and stored in
snapshot.properties
# Snapshot controlled field settings for Blackboard override of data
# fields for category entities
category.bb.controlled.fields=AVAILABLE_IND
– Fields owned by Blackboard
listed under “bb.controlled”
– Fields are Specific
for Each Entity
# fields for link entities
categorymembership.bb.controlled.fields=
# fields for group entities
group.bb.controlled.fields=AVAILABLE_IND,COURSE_NAME
# fields for user entities
person.bb.controlled.fields=INSTITUTION_ROLE
# fields for membership entities
membership.bb.controlled.fields=ROLE
Ownership - Example
– FirstName is owned by Blackboard
– LastName is owned by SIS
User_ID
FirstName
LastName
User1
Joseph
User2
FirstName
LastName
Smith
Joe
Smithe
Angela
Freedman
Freed
Angie
User3
Jennie
Brown
Jen
User4
Francis
Frank
Jones
Frank
Snapshot File
User_ID
Bb
Jones-Smith
Blackboard Database
1. Snapshot creates user records.
2. Data is updated via the UI.
3. Data is updated in SIS.
4. Snapshot runs again and last names are updated, first
names are left alone.
Data Sources and Data Source Keys
• Data Source Keys can be used to separate
and identify data from different sources.
SIS
HR
Alumni
Blackboard
Data Sources and Data Source Keys
• Data Source Keys can also be used to
separate and identify data from different
logical groups.
Spring
SIS
Summer
Fall
Data Sources and Data Source Keys
– Blackboard database records are mapped to a
particular data source key.
– Anything entered via the GUI is mapped to the
“SYSTEM” DSK.
course_ID
courseName
ext_crs_key
dataSourceKey
Course1
Biology
Bio01_02F2005
fa_crs_2005
Course2
History
Hist03_06F2005
fa_crs_2005
Course3
French
Fre01_05P2005
sp_crs_2005
Course4
French2
Fre02_01P2005
sp_crs_2005
Course5
Spanish
Course5*
SYSTEM
* Note: Course_ID & Ext_Crs_Key will be identical for GUI entered records.
Using Data Source Keys
111-11-1111
222-22-2222
888-88-8888
666-66-6666
DSK = “Medical”
555-55-5555
444-44-4444
777-77-7777
333-33-3333
DSK = “Law”
external_person_key
111-11-1111
222-22-2222
333-33-3333
444-44-4444
555-55-5555
666-66-6666
777-77-7777
888-88-8888
DSK
Medical
Medical
Law
Law
Law
Medical
Law
Medical
Bb
The DSK ensures that only the “right”
records are updated.
Without DSK the “Law” records
would be disabled when the “Medical”
snapshot is processed.
Snapshot Tools – Data Source Manager
DSM -V <server name> -f <operation] -b <data source key>
 dsm –V bbls.abc.edu –f CREATE –b BBCRS_FA2005 –d
Fa05_Courses
 dsm –V bbls.abc.com –f DISABLE_PERSON –b BBUSERS –t
20050305
 dsm –V bbls.abc.org –f PURGE_MEMBERSHIP –b
BBSTAFF_FA2005
 dsm –V bbls.abc.gov –f MODIFY –b BBCAT –r BBCAT_FA05
Only disabled records will be removed using the PURGE operation.
Closing Slide
• Innovating Together in ‘05:
– Recap top 2-3 points
• Resources Available:
– Any links, resources available for attendees to learn more
• Follow up Contact(s):
– Insert
• IF YOU ONLY REMEMBER 1 THING:
– Insert