University of Edinburgh
_____________________________________________________________________________________________
System Description
For
Unidesk CMDB: Hardware import
Document Version: 0.2
Date: 15/05/14
__________________________________________________________________________
Information Services - Template Revised May 2014
System Description
Version: 0.2
Unidesk CMDB: Hardware import
Contents
1
DOCUMENT MANAGEMENT ........................................................................... 3
1.1
Contributors ................................................................................................. 3
1.2
Version Control ............................................................................................ 3
2
OVERVIEW ....................................................................................................... 4
2.1
Service Description ..................................................................................... 4
2.2
Data Model ................................................................................................... 4
2.3
Reporting...................................................................................................... 5
2.4
Technology .................................................................................................. 5
2.4.1 DOS batch files ...................................................................................................... 5
2.4.2 PowerShell............................................................................................................. 5
2.4.3 SQL server agent job ............................................................................................. 6
2.5
Development Tools ...................................................................................... 7
2.6
Interfaces and services ............................................................................... 7
2.7
Access, Authentication and Authorisation ................................................ 7
2.8
Delivery......................................................................................................... 8
3
3.1
SUPPORT DETAILS ......................................................................................... 9
Third Party.................................................................................................... 9
3.2
Documentation............................................................................................. 9
3.2.1 Source code........................................................................................................... 9
3.2.2 SCP setup.............................................................................................................. 9
3.3
Standard tasks ............................................................................................. 9
3.4
Troubleshooting Guide ............................................................................... 9
3.4.1 Specify remote (source) download directory .......................................................... 9
3.4.2 Specify files to download from partner ..................................................................10
4
RELATED PROJECTS OR MAJOR SUPPORT WORK ................................. 13
5
DOCUMENT SIGN OFF .................................................................................. 13
Page 2 of 13
System Description
Version: 0.2
Unidesk CMDB: Hardware import
1 Document Management
General Guidance
1) Development services write a draft System Description Document by filling in the relevant
parts of the Document Template.
2) Development Services send the draft document to Application Management for review.
3) Development services and Application Management hold a handover meeting to discuss
any further additions or changes.
4) Development services complete any additions and/or changes agreed at the handover
meeting.
5) Development services handover the System Description Document to Application
Management.
6) Any further refinements to the document are completed by Applications Management.
7) Applications Management will provide final sign-off and from that point will take
responsibility for keeping the document up-to-date.
8) Until the System Description Document template is converted to a wiki template,
Applications Management will add the System Description Document to the insite Tech
Collab wiki.
1.1 Contributors
Please provide details of all contributors to this document
Role
Production Management
Support Coordinator
Systems Analyst Designer
Technical Architect
Project Manager
Project Sponsor
Business Analyst
Other document contributors
Unit
Name
ISG Dev
Geir Granum
1.2 Version Control
Please document all changes made to this document since initial distribution.
Date
12/05/14
15/05/14
Version
0.1
0.2
Author
GG
GG
Section
All
2,3
Amendment
Initial version
Changes suggested after review
meeting
Page 3 of 13
System Description
Version: 0.2
Unidesk CMDB: Hardware import
2 OVERVIEW
This UniDesk Incident, Problem and Change&Release processes would all be improved if
underpinned by a Configuration Management Database (CMDB).
Additionally, there is a need to have an inventory of key assets for the partners and business
units within the partners.
This document describes the loading of Hardware assets into the system’s ‘Holding table’.
2.1 Service Description
Sources for Hardware
CSV files
“G:\unidesk\data\
<partner>\hardware\
parts”
Directory
Manually
created and
uploaded
OR
Upload
Load CSV file into
temporary holding
table
Temporary
Holding table
Validated data
Holding table
Automatically
created and
uploaded
E.g:
bcp Holding table in master\
hardware.csv -T -f
hardware.xml
Validate data
2.2 Data Model
The contents of the table
 HARDWARE_temp
Is programmatically validated against the contents of the following tables:
 Object_id_suffix
o This validation is for data from Edinburgh only. It ensures that only data that
has been correctly identified as coming from a known Edinburgh data provider
is loaded into the Hardware holding table. This validation is: the suffix of the
object_id (the part of the object_id after the string ‘—‘) must match one of the
values in the valid_suffix column in the object_id_suffix table.
 Object_type_hardware
o This validation ensures that only hardware types that already exist in Unidesk
can be added when uploading a CSV file to the system. Without this test any
new hardware type found in the file would be created in Unidesk and this
behaviour was not wanted. Current values are found in the column: ref_soort
in the table: hardware in the main topdesk table.
A final validation is done on contents of the ‘link_location’ and ‘object_id’ columns. These
validations were added after reading the document describing the hardware holding table
from TopDesk at:
https://www.projects.ed.ac.uk/system/files/projectotherdocuments/Documentation%20Hardw
are%20Holding%20tables_0.html
1. ‘link_location’ and ‘object_id’ MUST have values.
2. If the value of ‘link_location’ in a row is ‘6’ then the corresponding ‘person’ column
MUST have a value.
3. If the value of ‘link_location’ in a row is ‘2’ then the corresponding ‘configuration_id’
column MUST have a value.
Page 4 of 13
System Description
Version: 0.2
Unidesk CMDB: Hardware import
2.3 Reporting
There is no reporting from the hardware import into the CMDB.
2.4 Technology
The Hardware import is implemented using the following technologies:
 DOS batch files
 PowerShell
 SQL server agent job (MS SQL)
2.4.1 DOS batch files
Dos batch files are used to:
1. For non-Edinburgh partners: pull down data using secure file copy to the
hardware\parts directory (upload2b.bat)
2. Load the data in the CSV files into the temporary holding table (upload4.bat)
2.4.1.1Upload2b.bat
This batch file takes 2 parameters: location and type:
1. Valid values for location are: ed, sta, abt, shu
2. Valid values for type are: hw
Using the location value the batch file sets the location where the CSV files should be found,
it the location indicates a remote partner SCP values are set and SCP is called to download
the CSV files and finally upload4.bat is called
2.4.1.2Upload4.bat
This batch file takes 3 parameters: location_of_CSV_file ,location and type:
1. location_of_CSV_file is derived in upload2b.bat
2. Valid values for location are: ed, sta, abt, shu
3. Valid values for type are: hw
Using these values the target database for upload and which format file should be used for
the upload are derived and a SQL server bulk insert command is called to load the data in
the CSV files(s) into the temporary holding table.
2.4.2 PowerShell
PowerShell is only used for the following early validation of Edinburgh data: The name of the
CSV file should match the suffixes in the object_id column in the file. If one or more records
in the file do not match this criterion a warning email is sent. The file is still processed. Any
invalid object_id suffixes are picked up by later validation.
Page 5 of 13
System Description
Version: 0.2
Unidesk CMDB: Hardware import
2.4.3 SQL server agent job
Start
Clear away old CSV
files
Success
No
Yes
Blank holding table
Success
No
Yes
Get new CSV files
Using PSCP
Success
No
Yes
For Edinburgh data:
suffix=filename
Load CSV files into
temporary table
table
Success
No
Yes
Validate content
Validated
No
Yes
Copy validated
content to Holding
table
Send success email
Send Failure email
End
The steps of the SQL server agent job
Page 6 of 13
System Description
Version: 0.2
Unidesk CMDB: Hardware import
2.5 Development Tools
For the database work; SQL server management studio was used.
2.6 Interfaces and services
The hardware import process neither calls nor is called by other systems.
2.7 Access, Authentication and Authorisation
The processes should be automated in SQL server (schedule TBD).
To manually run hardware import for a partner find the SQL server agent job named:
’Import Hardware <partner>’
and manually run it by right clicking on it and select: ‘Start Job at Step’


Details how authentication works
Details how authorisation works
Page 7 of 13
System Description
Version: 0.2
Unidesk CMDB: Hardware import
2.8 Delivery
State if the application is delivered via a Portal or embedded/part of a wider application (e.g.
MyEd)
Page 8 of 13
System Description
Version: 0.2
Unidesk CMDB: Hardware import
3 Support details
3.1 Third Party
The Holding table is provided by TopDesk. The specification can be found at:
https://www.projects.ed.ac.uk/system/files/projectotherdocuments/Documentation%20Hardw
are%20Holding%20tables_0.html
3.2 Documentation
3.2.1 Source code
The source code is found in SVN at:
https://svn.ecdf.ed.ac.uk/repo/is/apps/software/TopDesk_CMDB_data_import
3.2.2 SCP setup
To grab files from remote partners the same SCP setup as described in:
https://www.wiki.ed.ac.uk/display/insite/Unidesk+Partner+File+Transfers
is used, with data fetched from a different directory (configured with the ‘remdir1’ variable)
3.3 Standard tasks


Describe any tasks which are required for the delivery of the application which are not
automated and require manual intervention
Describe tasks which are required to be performed on regular basis as part of
business cycles, such as annual data roll overs, data clean-up, manual data extracts
3.4 Troubleshooting Guide
3.4.1 Specify remote (source) download directory
This is set in the upload2b.bat file. For hardware (‘hw’) download this should be ‘Hardware’
according to the SDS
IF [%2]==[hw] (
set "locdir1=hardware"
REM remdir1 should have trailing / if set
set "remdir1=Hardware"
)
IF [%2]==[sw] (
set "locdir1=software"
REM remdir1 should have trailing / if set
set "remdir1="
)
Page 9 of 13
System Description
Version: 0.2
Unidesk CMDB: Hardware import
3.4.2 Specify files to download from partner
The hardware CSV system is using the same method as the current Person CSV system.
The specification calls for a directory on the remote server called Hardware to be created
and that the hardware CSV file(s) that should be processed are placed in this directory. The
default configuration is to download ALL *.csv files found in the Hardware directory. If you
want to limit the download to a named CSV file then this can be configured in the
upload2b.bat file.
1. Find the section in the file corresponding to the site you want to configure. ‘abt’ =
Abertay and ‘sta’ = St. Andrews
IF "%1"=="abt" (
set "dir1=G:\unidesk\data\abertay\%locdir1%\parts"
set "scpaddr1=unidesk@kydd.uad.ac.uk:"
set "scpfrom1=./%remdir1%*.csv"
set "doscp1=1"
)
2. Change the setting of the variable ‘scrpfrom1’ so that only a specified, named file is
downloaded
set "scpfrom1=./%remdir1% AbertayTestHardware.csv"
All other values can be over-ridden in the partner sections if necessary (E.g if the partner
site do not put the hardware CSV files in a Hardwar directory this can easily be taken into
account by changing the part of the scpfrom1 variable that is currently reading /%remdir1%
3.4.3 Manual upload of CSV file to holding table
This script can be used as a basis for directly/manually adding a file to the holding table in
SQL server management studio.
-- Select the target database
USE [<database>]
GO
-- Clear the target tables
delete from [dbo].[HARDWARE_temp];
GO
delete from [dbo].[HARDWARE];
GO
-- Insert the data from the CSV file into the temp table
-- (repeat for each file you want to upload – without clearing out
-- the target tables)
bulk insert [dbo].[HARDWARE_temp] from
'G:\unidesk\data\<site>\hardware\parts\<filename>.csv' with
(formatfile='G:\unidesk\bin\CMDB_scripts\format_files\hardware.xml')
GO
-- Initialise the validation columns
UPDATE [dbo].[HARDWARE_temp]
SET IS_INVALID=0,
REASON=' ';
GO
-- Check for mandatory columns
UPDATE [dbo].[HARDWARE_temp]
SET IS_INVALID=1,
REASON = REASON+'Mandatory data missing.'
WHERE
(OBJECT_ID IS NULL) OR (LOCATION_LINK IS NULL)
-- check person column for location_link=6
UPDATE [dbo].[HARDWARE_temp]
SET IS_INVALID=1,
Page 10 of 13
System Description
Version: 0.2
Unidesk CMDB: Hardware import
REASON = REASON+'Person ID missing for location_link=6.'
WHERE
(PERSON IS NULL) AND (LOCATION_LINK=6)
-- check configuration_id column for location_link=2
UPDATE [dbo].[HARDWARE_temp]
SET IS_INVALID=1,
REASON = REASON+'Configuration_id missing for location_link=2.'
WHERE
(CONFIGURATION_ID IS NULL) AND (LOCATION_LINK=2)
GO
-- For Edinburgh content check for valid suffixes in the object_id column
-- valid suffixes are held in the object_id_suffix table
UPDATE [dbo].[HARDWARE_temp]
SET IS_INVALID=1,
REASON = REASON+'Invalid ID suffix.'
WHERE
LTRIM(SUBSTRING(OBJECT_ID, CHARINDEX('--',OBJECT_ID)+2,LEN(OBJECT_ID)-1)) not
in (select valid_suffix COLLATE Latin1_General_CS_AS from [dbo].[object_id_suffix])
GO
-- Check for valid/existing object types
UPDATE [dbo].[HARDWARE_temp]
SET IS_INVALID=1,
REASON = REASON+'Invalid object_type.'
WHERE
OBJECT_TYPE not in (select valid_type COLLATE Latin1_General_CS_AS from
[dbo].[object_type_hardware])
GO
-- Finally copy all valid data to the holding table
INSERT INTO [dbo].[HARDWARE]
([OBJECT_ID]
,[PURCHASE_PRICE]
,[PURCHASE_DATE]
,[NOTES]
,[DEPRECIATION]
,[WARRANTY_EXPIRES]
,[HOSTNAME]
,[IP_ADDRESS]
,[MAC_ADDRESS]
,[MODEL]
,[ORDER_NUMBER]
,[LOCATION_LINK]
,[RESIDUAL_VALUE]
,[SERIAL_NUMBER]
,[SPECIFICATION]
,[INSURED_UNTIL]
,[BUDGET_HOLDER]
,[CONFIGURATION_ID]
,[SUPPLIER]
,[MAKE]
,[PERSON]
,[OBJECT_TYPE]
,[STATUS])
SELECT [OBJECT_ID]
,[PURCHASE_PRICE]
,[PURCHASE_DATE]
,[NOTES]
,[DEPRECIATION]
,[WARRANTY_EXPIRES]
,[HOSTNAME]
,[IP_ADDRESS]
,[MAC_ADDRESS]
,[MODEL]
,[ORDER_NUMBER]
,[LOCATION_LINK]
,[RESIDUAL_VALUE]
,[SERIAL_NUMBER]
,[SPECIFICATION]
Page 11 of 13
System Description
Version: 0.2
Unidesk CMDB: Hardware import
,[INSURED_UNTIL]
,[BUDGET_HOLDER]
,[CONFIGURATION_ID]
,[SUPPLIER]
,[MAKE]
,[PERSON]
,[OBJECT_TYPE]
,[STATUS]
FROM [dbo].[HARDWARE_temp]
where not (IS_INVALID=1)
GO
Page 12 of 13
System Description
Version: 0.2
Unidesk CMDB: Hardware import
Related Projects or Major Support Work
The following table lists previous projects or major pieces of support work that have been
carried out on this service:
Project / Major Support Work
https://www.wiki.ed.ac.uk/display/insite/UniDesk
Analyst
Developer
4 Document Sign Off
Please add other sign off roles where required:
Production Management
Coordinator
Project Manager
Project Sponsor
Business Analyst
Systems Analyst Designer
Technical Architect
Name
Date
Name
Name
Name
Name
Name
Date
Date
Date
Date
Date
Page 13 of 13
Description
The InSite wiki page
for UniDesk