I. What is the CSV Import Feature? The CSV Import Feature is a Service Manager component that allows users to bulk-import instances of any class type or projection type defined in the Service Manager Configuration Management Database (SMCMDB). This feature can be used to: Create Configuration Item or Work Item instances from data stored in a tabular format Bulk-edit existing database instances Populate the SMCMDB with data exported from an external database Circumvent data entry through forms when many class instances must be created at once To import a set of instances using the CSV Import Feature, two files are required: 1. A data file that consists of a series of comma-delimited object instances. The data file must end with the ".csv" file extension. 2. A format file that specifies the class type or projection type of the instances present in the data file. Every instance in the data file is assumed to be of the this type. The format file also specifies (1) the subset of properties (and, in the case of projections, components) being imported for the indicated type and (2) the order in which those properties appear (as columns) in the associated data file. The format file must have the same file name as the CSV file that it describes, and it must end with the ".txt" file extension. II. Walkthrough: Using the CSV Import Feature Let's examine a typical use of the CSV Importer. Suppose that we have an Excel spreadsheet describing the 10 new computers that we have recently purchased. We would like to add these new Configuration Items to the SMCMDB. The entries in our spreadsheet are: Computer Name billing-comp-1 billing-comp-2 billing-comp-3 hr-24 hr-25 hr-30 hr-31 audit-1 audit-9 audit-11 IP Address 172.30.14.21 172.30.145.213 172.30.145.214 172.115.14.21 172.115.14.22 172.125.30.21 172.125.30.26 172.126.15.36 172.126.15.39 172.126.15.33 Domain Name DETROIT DETROIT DETROIT DALLAS DALLAS CHICAGO CHICAGO CHICAGO CHICAGO CHICAGO 1. Creating the Data File To use the CSV Importer, we first need to save the spreadsheet as a .csv file. The first row of the spreadsheet is assumed to contain an object instance (and not a header), so we remove this line and save the spreadsheet above (in CSV format) in a file named "newcomputers.csv". billing-comp-1, 172.30.14.21, DETROIT billing-comp-2, 172.30.145.213, DETROIT billing-comp-3, 172.30.145.214, DETROIT hr-24, 172.115.14.21, DALLAS hr-25, 172.115.14.22, DALLAS hr-30, 172.125.30.21, CHICAGO hr-31, 172.125.30.26, CHICAGO audit-1, 172.126.15.36, CHICAGO audit-9, 172.126.15.39, CHICAGO audit-11, 172.126.15.33, CHICAGO Figure 1: Contents of "newcomputers.csv" data file 2. Creating the Format File Next, we must construct a format file suitable for importing the rows contained within newcomputers.csv. The first step in writing a format file is identifying the class type or projection type to be used for the instances in the CSV file. Appendix A of this documentation provides the names of classes and projection types and the properties that can be imported for these types (see section IV of this document: Using Appendix A). (Note: The CSV Importer is not limited to importing data in the classes listed in Appendix A. It can also be used to import instances of classes and projection types defined in custom management packs created by the user.) Scanning the list of classes in Appendix A, we find that Microsoft.Windows.Computer is the most appropriate choice for the object type and property set that we wish to import. We begin by declaring the class of the object being imported: <CSVImportFormat> <Class Type="Microsoft.Windows.Computer"> ... </Class> </CSVImportFormat> Next, from Appendix A, we scan the available properties of the Microsoft.Windows.Computer class and selects those that best correspond to the columns being imported. We settle on the following property mapping: Column 1 → PrincipalName Column 2 → IPAddress Column 3 → DomainDnsName Using this mapping, we construct the format file as below. The properties in the format file are specified by name in the order in which they appear in the CSV file. We save the format file with the same file name as the data file but with the "txt" file extension. The format file must be saved in the directory where the CSV file is located. <CSVImportFormat> <Class Type="Microsoft.Windows.Computer"> <Property ID="PrincipalName"/> <Property ID="IPAddress"/> <Property ID="DomainDnsName"/> </Class> </CSVImportFormat> Figure 2: Contents of "newcomputers.txt" format file Note that any class properties that are marked as key or required (Appendix A makes note of such properties) must be included in the format file definition. Key properties uniquely identify an instance in the SMCMDB; required properties must be set to a non-null value. Furthermore, the values specified for a property in the data file must be of the type expected. The table below shows example values for the possible property types. Type binary bool enum Examples Binary data true, false, 1, 0 (enum-type-specific, see Appendix A) System.Knowledge.ArticleTypeEnum.HowTo, System.ServiceManagement.ServicePriority.Medium, int decimal double guid datetime richtext (or an enum guid, e.g.: 9192AAD4-5549-1a1d-dd83-5367AAAE2B66) 1,2, 124, 521 5, 200, 1004 20.125, 30.412, 1.002, 0.001 DDCFBE6D-EA53-4a71-BC9B-CC4F70EFF746, 27F5FA6C015E4720B6303848BC72BE20, {8B71AAD4-4049-4e3e-928F-13B720BE20D5} 05/01/2009 14:57:32.8, Fri, 15 May 2009 20:10:57 GMT, 1 May 2008 2:57:32.8 PM (Accepts DateTime formats accepted by the .NET method DateTime.TryParse() http://msdn.microsoft.com/en-us/library/ch92fbc1.aspx) Rich text format data {\rtf1\ansi\ansicpg1252\deff0\deflang1033{\fonttbl{\f0\fnil\fcharset0 Calibri;}} {\*\generator Msftedit 5.41.21.2510;}\viewkind4\uc1\pard\sa200\sl276\slmult1\lang9\f0\fs22 This is \b bold\b0 text.\par } string asdf, test string, This is a sentence, "test" (Note: Leading/trailing whitespace is trimmed by default. To preserve whitespace and newlines, enclose the field in double quotes (e.g.: " space test "). Any field that contains a double-quote character must be enclosed in quotes and the quote must be escaped by a double quote (e.g.: "Importing a "" character")). Figure 3: Examples of valid values for each property type 3. Importing the Data With the format and data files prepared, we are ready to use the CSV Import task. To access it, we click on the Administration Wunderbar as pictured below. Figure 4: Administration Wunderbar Selected, Import Task Visible We then select the "Import from CSV file..." task in the Tasks Pane (top right). The following dialog appears: Figure 5: CSV Import File Dialog We select the newcomputers.csv file and click the "Open" button. Upon doing so, the CSV Import code automatically locates the like-named format file in the same directory and proceeds to import the 10 computers in newcomputers.csv. These imported computers are now visible in the All Computers view. Figure 6: Imported Computer Instances Visible in All Computers View III. Importing Projection Types In some cases, it might appear as though rows in a CSV file can be imported as instances of a single class when , in fact, they cannot. Consider the case of an incident instance. An incident always includes an affected user, but this affected user cannot be represented by any one simple property (a string, an int, a bool, etc.). The affected user is really an instance of a separate class with its own set of properties -- a user class -- and the incident class merely establishes a reference to that instance. As viewed in our incident form, an incident spans multiple classes and several relationships in Service Manager's type system. If we wish to import an incident with its affected user, then we must somehow define a format file that specifies both the incident and user classes and the "affected user" relationship that exists between them. This relatively simple example hints at a more general problem: how do we import classes that have multiple relationships to other classes, and how do we know what classes and relationships to use? Service Manager simplifies this problem by supporting a kind of data type called a type projection. Type projections provide a convenient way to combine several classes and relationships into a single object type. When a projection instance is committed to the SMCMDB, our SDK does the work of creating the individual classes and relationships that compose the instance. Much can be written about authoring custom classes, relationships, and projection types in user-defined management packs. See the Authoring Guide for additional information. The CSV Importer can be used to import instances of user-defined classes and projections. For the remainder of this section, we'll focus on importing instances of a projection type that ships with Service Manager CTP2: System.Incident.ProjectionType. 1. Importing an Incident Projection It makes sense to first examine the projection type being used to import the data. Let's assume that the incident data that we wish to import contains only information about the incident itself and the user affected by the incident. Using Appendix A to determine the name of the type projection, we begin by defining a format file with the projection type only: <CSVImportFormat> <Projection Type="System.WorkItem.Incident.ProjectionType"> ... </Projection> </CSVImportFormat> Figure 7 We next examine the list of classes available in the incident projection type to determine the objects that we need to import. 2. Specifying the Seed Type Every type projection definition defines a seed class. The seed class "anchors" the projection to a class in the SM type space and allows the user to reference other class types through specific relationships on that seed class. As per the table in Appendix A, the System.WorkItem.Incident.ProjectionType class may have a seed type of System.WorkItem.Incident and an "AffectedUser" relationship that references a System.Domain.User object. It is this combination of two class instances and one relationship instance that we wish to import for each instance in our data file. We add the seed element to the format file as follows: <CSVImportFormat> <Projection Type="System.WorkItem.Incident.ProjectionType"> <Seed> <Class Type="System.WorkItem.Incident"> ... </Class> </Seed> </Projection> </CSVImportFormat> Figure 8 We must now determine the subset of properties of the System.WorkItem.Incident class to import with each incident projection instance. The syntax for specifying these properties is identical to the syntax used to specify properties for a single class. We add these to the seed element in the format file as follows. <CSVImportFormat> <Projection Type="System.WorkItem.Incident.ProjectionType"> <Seed> <Class Type="System.WorkItem.Incident"> <Property ID="Id"/> <Property ID="ContactMethod"/> <Property ID="ResolutionDescription"/> <Property ID="Impact"/> <Property ID="Urgency"/> </Class> </Seed> </Projection> </CSVImportFormat> Figure 9 Note that "Id" is a key property of the System.WorkItem class, the parent class of System.WorkItem.Incident. When specifying the properties to import for System.WorkItem.Incident, it is important that all key and required properties in the class hierarchy of System.WorkItem.Incident are included. Appendix A is designed to help you identify these key and required properties (they are marked in red). 3. Adding a Related Class Now we must update our format file to specify that we are adding a separate class instance -- an affected user -- with every incident imported. To do this, we add a "Component" node under the "Projection" node. This is similar to the syntax used to define projection types in management packs. The alias attribute, present in the component definition, identifies the relationship between that component and the seed instance of the projection. This alias name originates in the projection definition. Appendix A contains a list of available components for each projection type. <CSVImportFormat> <Projection Type="System.WorkItem.Incident.ProjectionType"> <Seed> <Class Type="System.WorkItem.Incident"> <Property ID="Id"/> <Property ID="ContactMethod"/> <Property ID="ResolutionDescription"/> <Property ID="Impact"/> <Property ID="Urgency"/> </Class> </Seed> <Component Alias="AffectedUser"> ... </Component> </Projection> </CSVImportFormat> Figure 10 Now that the AffectedUser component has been specified, we must now supply a class with which to import affected users. An AffectedUser relationship instance relates a System.WorkItem object to a System.User object. We cannot import instances of the System.User class, however, because it is defined as an abstract class (only its "concrete" child classes can be created). We must therefore use a class like the System.Domain.User class, a derived class not marked as abstract, to create an AffectedUser. Note that we could have specified any class deriving from System.WorkItem.Incident in the seed class type definition (e.g.: Microsoft.SystemCenter.WorkItem.SCOMIncident, for example). Lists of valid classes to use for seed and component types appear in each projection type entry in Appendix A. We add the System.Domain.User class -- and the properties we wish to specify -- to the projection format definition as follows: <CSVImportFormat> <Projection Type="System.WorkItem.Incident.ProjectionType"> <Seed> <Class Type="System.WorkItem.Incident"> <Property ID="Id"/> <Property ID="ContactMethod"/> <Property ID="ResolutionDescription"/> <Property ID="Impact"/> <Property ID="Urgency"/> </Class> </Seed> <Component Alias="AffectedUser"> <Seed> <Class Type="System.Domain.User"> <Property ID="FirstName"/> <Property ID="Domain"/> <Property ID="UserName"/> </Class> </Seed> </Component> </Projection> </CSVImportFormat> Figure 11 One subtlety of this updated definition is that we wrapped System.Domain.User class declaration inside of a "Seed" tag. While each "Component" node must have a "Seed" node that identifies the related class type being imported, it may also contain additional "Component" subnodes. These subnodes are used to import class types related to the component's seed. As an example, if we wanted to import a single file attachment instance and the user that added that file attachment with every incident in the data file, we could add the following nested components: The syntax would be: <CSVImportFormat> <Projection Type="System.WorkItem.Incident.ProjectionType"> <Seed> <Class Type="System.WorkItem.Incident"> <Property ID="Id"/> <Property ID="ContactMethod"/> <Property ID="ResolutionDescription"/> <Property ID="Impact"/> <Property ID="Urgency"/> </Class> </Seed> <Component Alias="AffectedUser"> <Seed> <Class Type="System.Domain.User"> <Property ID="FirstName"/> <Property ID="Domain"/> <Property ID="UserName"/> </Class> </Seed> </Component> <Component Alias="FileAttachments"> <Seed> <Class Type="System.FileAttachment"> <Property ID="Id" /> <Property ID="Extension" /> <Property ID="Description" /> </Class> </Seed> <Component Alias="FileAttachmentAddedBy"> <Seed> <Class Type="System.Domain.User"> <Property ID="FirstName"/> <Property ID="Domain"/> <Property ID="UserName"/> </Class> </Seed> </Component> </Component> </Projection> </CSVImportFormat> Figure 12 4. Ordering Properties in the CSV File Let's revisit the format file definition in Figure 11 that contains only the incident projection type, the incident class, and the affected user related class. Now that the format file is written, how do we order the properties in the CSV file? The order of properties specified in the CSV file is identical to the order in which they were defined in the format file. That is, the first comma-delimited field should contain a value for the Id property of System.WorkItem.Incident and the last comma-delimited field should contain a value for the UserName property of System.Domain.User. There should be 8 comma-delimited fields total for the CSV Format file in Figure 11. Together, the example format and CSV files are: <CSVImportFormat> <Projection Type="System.WorkItem.Incident.ProjectionType"> <Seed> <Class Type="System.WorkItem.Incident"> <Property ID="Id"/> <Property ID="ContactMethod"/> <Property ID="ResolutionDescription"/> <Property ID="Impact"/> <Property ID="Urgency"/> </Class> </Seed> <Component Alias="AffectedUser"> <Seed> <Class Type="System.Domain.User"> <Property ID="FirstName"/> <Property ID="Domain"/> <Property ID="UserName"/> </Class> </Seed> </Component> </Projection> </CSVImportFormat> Figure 13: "newincident.txt" Inc-{0}, e-mail, resolved on reboot, System.WorkItem.TroubleTicket.ImpactEnum.High, System.WorkItem.TroubleTicket.UrgencyEnum.High, Bob, DALLAS, bobsmith Inc-{0}, voicemail, network error, System.WorkItem.TroubleTicket.ImpactEnum.High, System.WorkItem.TroubleTicket.UrgencyEnum.High, James, DALLAS, jamesw Figure 14: "newincident.csv". The symbol indicates that the current line continues on the following line. The file above contains two instances. Note that when {0} appears in the value of an autoincrement property of type string (here, the "Id" property of System.WorkItem), it will be replaced with the next available ID. 5. Importing Multiple Related Instances under the Same Projection Component Some projection types in the SM type system allow a seed instance to be related to multiple class instances under the same component. For example, an incident may reference multiple file attachments (System.FileAttachment) in its "FileAttachments" component. To import multiple instances under the same projection component, the count attribute is used. Consider the simplified format file defined below: <CSVImportFormat> <Projection Type="System.WorkItem.Incident.ProjectionType"> <Seed> <Class Type="System.WorkItem.Incident"> <Property ID="Id"/> <Property ID="ContactMethod"/> <Property ID="ResolutionDescription"/> <Property ID="Impact"/> <Property ID="Urgency"/> </Class> </Seed> <Component Alias="FileAttachments" Count="3"> <Seed> <Class Type="System.FileAttachment"> <Property ID="Id" /> </Class> </Seed> <Component Alias="FileAttachmentAddedBy"> <Seed> <Class Type="System.Domain.User"> <Property ID="Domain"/> <Property ID="UserName"/> </Class> </Seed> </Component> </Component> </Projection> </CSVImportFormat> Figure 15: "multipleattachments.txt" The format file above indicates that three System.FileAttachment instances will be imported alongside each System.WorkItem.Incident instance. Since the count attribute of a component node in the format file is set to a value greater than 1, the number of properties expected in each row of the CSV file will be greater than the number of "Property" XML nodes appearing in the format file. There are 4 properties specified in the format file: 1 for System.Workitem.Incident 1 for System.FileAttachment 2 for System.Domain.User Since 3 file attachments are being imported (1 property per instance), each with a user (2 properties per instance), a total of 3*(1+2) = 9 properties are expected for the FileAttachments component. Adding the Id property of System.WorkItem.Incident brings the total property count to 10. In what order do these 10 properties appear in the CSV file? The properties are parsed in an in-order tree traversal of property nodes. One way of determining this order is to consider the order in which seed classes and components (not properties) appear in the format file, level by level. We can diagram (left-right) the order in which the seed and components appear in the projection definition. Starting at the projection type definition in Figure 15, we have the following seed (blue) and component types (red) at the first level: Incident FileAttachments Figure 16: Seed Class (blue) and Projection Component (red) Since the FileAttachments node has a count of 3, we repeat this component twice: Incident FileAttachments FileAttachments FileAttachments Figure 17 Our goal is to replace the red boxes (components) with blue boxes (class types in seed nodes) so that we can determine a property order. The components (red boxes) in our diagram can successively be replaced with their constituent seed and component types. By making these replacements, we will eventually have a diagram that contains only seed classes. The FileAttachments component contains a System.FileAttachment seed class and a FileAttachmentsAddedBy component. Making the substitution, we have: Incident FileAttachment FAAddedBy FileAttachment FAAddedBy FileAttachment FAAd Figure 18: After Descending into the FileAttachments Projection Component Repeating this step once more (descending to the FileAttachmentAddedBy component and replacing it with the System.Domain.User seed class) yields: Incident FileAttachment User FileAttachment User FileAttachment U Figure 19: After Descending into the File Attachment Added By Projection Component Each of these classes has a set of properties associated with it: 1. 2. 3. 4. 5. Incident Id ContactMethod ResolutionDescriptio n Impact Urgency FileAttachment 1. Id User 1. Domain 2. UserName FileAttachment 1. Id User 1. Domain 2. UserName Figure 20: Seed Class Display with Property Order The order in which properties are specified in the CSV file is the order in which they appear in the boxes above: read the boxes left-to-right, and at each box, enumerate the properties top-to-bottom. An example CSV file would be: Inc-{0}, e-mail, resolved on reboot, System.WorkItem.TroubleTicket.ImpactEnum.High, System.WorkItem.TroubleTicket.UrgencyEnum.High, FileAttachment (1), UserDomain1, User UserName(1), FileAttachment (2), UserDomain2, User UserName(2), FileAttachment (3), UserDomain3, User UserName(3) Figure 21: Example CSV File. The symbol indicates that the current line continues on the following line. The file above contains a single line of text. Note that the count attribute of any component should reflect the maximum number of instances under that component for all projection instances in the csv file. For example, if a CSV file contained two incidents, and one incident had 4 file attachments while another incident had 2 file attachments, the count attribute of the FileAttachments component would be 4. The incident with 2 file attachments would have a series of empty columns (i.e.: ",,") for the properties corresponding to the two absent file attachments. FileAttachmen 1. Id IV. Using Appendix A Appendix A is included in this zip file as AppendixA.html. This appendix contains a list of projection types, configuration item class types, and work item class types that can be imported using the CSV Importer. The projection types bolded in the index are the ones that are most frequently imported. Figure 22: Appendix A, Projection Types View Suppose we want to import knowledge articles. We can use the Projection Types section at the top of the Index to see what predefined classes and relationships exist in an SM knowledge article. Scanning the list, we see the bolded projection type System.Knowledge.ArticleProjection (#7). Clicking on the link, we arrive at the definition of this projection: Figure 23: Appendix A, System.Knowledge.ArticleProjection Projection Type The projections are organized hierarchically by component alias. Here, the first section, Seed, contains links to all classes that can be supplied as the seed type of the knowledge article projection. From the above picture, we see that System.KnowledgeArticle.Projection has at least three components. The aliases of these components are Rating, ArticleOwner, and RelatedConfigItem. The Rating component must have a seed class of type System.StarRating and there must be exactly one of these instances in the projection (minimum of 1 maximum of 1). The user format file must also include the ArticleOwner projection component (min: 1, max: 1) and it must have a seed type that is one of the following 4 classes System.Domain.User, Microsoft.AD.UserBase, Microsoft.Ad.Group, or Microsoft.AD.User. Let's define a format file with the information known thus far: <CSVImportFormat> <Projection Type="System.Knowledge.ArticleProjection"> <Seed> <Class Type="System.Knowledge.Article"> ... </Class> </Seed> <Component Alias="Rating"> <Seed> <Class Type="System.StarRating"> ... </Class> </Seed> </Component> <Component Alias="ArticleOwner"> <Seed> <Class Type="System.Domain.User"> ... </Class> </Seed> </Component> </Projection> </CSVImportFormat> Figure 24: Intermediate Format File We know the class types to import, but we don't know the properties. To discover these, we simply click on the link of the class that we're using. In the picture above, clicking on the "System.Knowledge.Article" hyperlink next to Seed takes us here: Figure 25: Appendix A, System.Knowledge.Article Class The table on the right contains a list of the properties in that class. Those that must be defined in the format file are colored in red. Any properties that are of an enumeration type contain a link to a list of valid enumeration values. For example, clicking "enum:System.Knowledge.ArticleTypeEnum" in the image above takes us to: Figure 26: Appendix A, System.Knowledge.ArticleTypeEnum Enumeration Here, we see that there are three valid values for this enum property: System.Knowledge.ArticleTypeEnum.HowTo, System.Knowledge.ArticleTypeEnum.Issue, and System.Knowledge.ArticleTypeEnum.FAQ. By clicking on the System.StarRating link and System.Domain.User links in the System.KnowledgeArticle.Projection (Figure 23), we can discover the properties for the remaining classes and create the format file below: Figure 27: Appendix A, System.StarRating Class Figure 28: Appendix A, System.Domain.User Class <CSVImportFormat> <Projection Type="System.Knowledge.ArticleProjection"> <Seed> <Class Type="System.Knowledge.Article"> <Property ID="ArticleId"/> <Property ID="ArticleType"/> </Class> </Seed> <Component Alias="Rating"> <Seed> <Class Type="System.StarRating"> <Property ID="Id"/> <Property ID="RatingAverage"/> </Class> </Seed> </Component> <Component Alias="ArticleOwner"> <Seed> <Class Type="System.Domain.User"> <Property ID="Domain"/> <Property ID="UserName"/> </Class> </Seed> </Component> </Projection> </CSVImportFormat> Figure 29: Appendix A, System.KnowledgeArticle.Projection Format File Note that the indentation of projection component aliases in the Projection Types section of Appendix A reflects the relationship between projection components: components are indented under their parent component (or projection type). Consider System.WorkItem.ChangeRequestProjection (#15). Figure 30: Appendix A, System.ChangeRequestProjection Projection Type The Activity component has a subcomponent with alias ActivityAssignedTo. An example format file using these three components would be: <CSVImportFormat> <Projection Type="System.WorkItem.ChangeRequestProjection"> <Seed> <Class Type="System.WorkItem.ChangeRequest"> <Property ID="Id"/> </Class> </Seed> <Component Alias="Activity"> <Seed> <Class Type="System.WorkItem.Activity.ManualActivity"> <Property ID="Id"/> </Class> </Seed> <Component Alias="ActivityAssignedTo"> <Seed> <Class Type="System.Domain.User"> <Property ID="Domain"/> <Property ID="UserName"/> </Class> </Seed> </Component> </Component> </Projection> </CSVImportFormat> Figure 31: Appendix A, System.ChangeRequestProjection Projection Type The class and projection types in Appendix A include sample XML fragments that can assist you in constructing the format file.