Haifa University Department of Computer Science Final Report Instructor: Dr. Hagit Hel-Or Table of Contents Bibliography ................................................................................................................. 4 Chapter 1 - Abstract .................................................................................................... 5 Chapter 2 - General Description ................................................................................ 7 Picture Maker Class ..................................................................................................... 10 Item_Package class ...................................................................................................... 10 DynamicTree Class ...................................................................................................... 11 Layers class .................................................................................................................. 11 Chapter 3 - Detailed Description .............................................................................. 12 PictureMaker class ....................................................................................................... 12 Item_Package Class ..................................................................................................... 14 Dynamic Tree class ...................................................................................................... 18 Layers class .................................................................................................................. 21 Chapter 4 - Summary ................................................................................................ 23 Appendix – Operating Instruction ............................................................................... 25 Staring drawing Icons in the Main Screen ................................................................... 25 Grouping ...................................................................................................................... 26 Setting As Part Of ........................................................................................................ 28 Setting Up a Relation ................................................................................................... 29 FG/BG Separator ......................................................................................................... 30 Extra Features of VOPD Application .......................................................................... 35 2 Table of Figures Figure 1- Application View ........................................................................................... 8 Figure 2 – Generic Class Diagram ................................................................................. 9 Figure 3 – Color Chooser ............................................................................................. 16 Figure 4 – Show Properties Screen .............................................................................. 16 Figure 5 – Correct Grouping ........................................................................................ 19 Figure 6 - Incorrect Grouping ...................................................................................... 19 Figure 7 – Move Layer Example ................................................................................. 22 Figure 8 – Result of Applying Move Layer Down ..................................................... 22 Figure 9 – Example of Dragging an Icon..................................................................... 25 Figure 10 – Grouping Screen ....................................................................................... 27 Figure 11 – Set as Part of Screen ................................................................................. 28 Figure 12 – Set Relation Screen................................................................................... 29 Figure 13 – Relation-Typing Screen ............................................................................ 30 Figure 14 – Set as Foreground Separator Screen ......................................................... 31 Figure 15 – Result of Applying Set as Foreground Separator Screen ......................... 32 Figure 16 – Move Layer Up Screen ............................................................................. 33 Figure 17 –Result of Applying Move Layer Up Screen .............................................. 34 Figure 18 – Show Properties Screen ............................................................................ 35 3 Bibliography "Semantic Content-Based Image Retrieval Using Object-Process Diagrams", D. Dori and H.Z. Hel-Or, in Advances in Pattern Recognition, Lecture Notes in Computer Science, Vol. 1451, Pages 230-241, Editors: A. Amin, D. Dori, P. Pudil and H. Freeman, 1998. 4 Chapter 1 - Abstract The increase in accessibility to on-line visual data has promoted the interest in browsing and retrieval of images from Image Data-bases. An image retrieval system has been previously developed, that is based on the semantic contents of the scene. In this retrieval system, images are represented using object-process diagrams called VOPD (Visual Object-Process Diagrams). These descriptions involve the objects in the scene and their inter- and intra-relationships. This allows for abstract, high-level representation of the layout of the scene, as well as a distinction between the dominant core of the scene and its background. Although graced with many advantages, a major drawback of the VOPD based Semantic Content Based Image Retrieval system is the need for the user or database manager to learn and be knowledgeable in the rules and structures of the VOPD. This is not trivial and in fact troublesome to the naive user. To overcome this drawback two approaches are suggested to provide an interface to the Image Retrieval system and to bypass the need of manually creating VOPD representations. The optimal solution would be to provide an automatic VOPD generator which given any input image, outputs its VOPD. This, however, would require very sophisticated Computer Vision and Image Understanding techniques, which are currently unavailable. Instead, two VOPD generators are suggested (in this project and in the Block World project) which relax the constraints and limitations of an automatic VOPD generator. In this project fully automatic characteristic of the system is relaxed. In this approach, an interface tool - The PictureMaker - is developed to guide the naive user into supplying the necessary details required to create the image VOPD. The interface tool is very intuitive and is based on the simple process of creating an image using visual 5 and graphic tools. The naive user requires almost no training in using this tool to create VOPDs from images or to create VOPD queries. The PictureMaker is based on objects being drawn in layers to create a schematic "drawing" of the original or query image. 6 Chapter 2 - General Description The main motivation of the application is to function as an interface between the user and a server that will contain all the VOPDs. The application has a friendly GUI and supplies comfortable tools for the user in order to create a VOPD. The application also supplies all the features (connections) that are mentioned in the VOPD article. The main features are: Grouping – An option to assign few objects as a group and in addition there is an option to nest groups. Set As Part Of – a feature that enables the user to set an object as part of another object. Nesting is also permitted here. Set Relation – An option to assign a relation between two objects. Other Attributes such as Truncated, Occluded, Color, Size and Location. In addition, the application divides the image into foreground and background using a separator line, that is a main idea in the VOPD concept. The application is divided into several frames as shown below: The Main Screen Frame - contains the drawn picture. VOPD Tree Frame - contains tree that will create a VOPD for a given picture. The Layers frame - represents the layers and their order. The Image Viewer Frame - will show the current selected icon. 7 Figure 1- Application View An object that is created during application run time appears in the Main Screen Frame, the Layers Frame and the VOPD Tree Frame. The application enables easy expanding the image directory that contains the icons that represent the object by adding new icons to the image directory. The application’s input is the user’s drawn picture. The output of the application is a simple to read text file, which reflects the drawn image. The file can be saved under any name and extension and can be loaded at any time to recreate the image again. Meanwhile, the user can see the reflection of this drawing by a well-organized VOPD tree and a list box that shows the layers of the drawn picture. Another option the application suggests is going back. This operation is done by enabling the user to delete an object or even restart the entire picture. All the features that are mention above are available through menu bars and popup windows. 8 Generic Class Diagram Item Package Class Picture Maker class PictureMaker’s Sub class Dynamic Tree Class Layers Class Figure 2 – Generic Class Diagram 9 Picture Maker Class The PictureMaker class is a shell class of the entire application. This class communicates with all the other classes and it manages the entire process. The class receives all the application’s input during its operation. The class has a data structure in a form of array of objects that are instances of the class that represents a drawn object in the picture (Item_Package class). The above data structure is the main data structure of the application. All updates and changes during application run time affect this data structure. This data structure keeps track and reflects the drawn image’s current state. This class is also responsible for the application’s Main Screen Frame. In addition, it is also in charge of the menu bar, organizes all the frames in the application and takes care of full coordinated state between all the frames in the application. Item_Package class The Item_Package class is a sub-class of the PictureMaker class. It represents an object, meaning that every object is an instance of the Item_Package class. Each object has the following attributes: Name Serial Number Location Size Color Other parameters like Part of relations, grouping relations, occluding, truncating, free text relations. In addition, each object has a popup menu that allows a few other features such as: Full Screen, Color Changing, Foreground Separator and properties showing. 10 DynamicTree Class The dynamic Tree class is a shell class for the VOPD Tree Frame. This class is the only class that recognizes objects like background, foreground and my image, therefore dividing the objects’ treatment to physical and logical. Physical handling is the object’s location in the tree. Logical handling is the object’s location in the data structure. In addition, the class enables the relation between several objects, which are represented by the Item_Package class, such as grouping, set as part of and set relation-features that are available through the popup menu. This popup menu also enables the user to ungroup and UnPart. Layers class The layers class is a shell class for the layers Frame. In addition to the objects, this class has a line that represents the separator, which represents the separation between the foreground and the background. Like the DynamicTree Class, the layers class also handles the objects in two planes: logical and physical, when the physical place of the object is affected by the separator line and the logical place of the object is not affected by the separator. In addition, the class enables another options like Move layers Up/Move layers Down and Set Separator through a popup menu. 11 Chapter 3 - Detailed Description PictureMaker class The PictureMaker class is a shell class of the entire application. This class communicates with all the other classes and it manages the entire process. The class receives all the application’s input during its operation. The class is in charge of dividing and organizing the frames in the application. The division’s algorithm is to retrieves a parameter that indicates the screen resolution and then assign proportionally the frames sizes. The class deals with the Drag and Drop issue between the Icon Viewer Frame and the Pad, that is the Main Screen Frame of the application. The Drag and Drop function operates by detecting a mouse pressing in the source frame (the Icon Viewer Frame) and then detecting a mouse release in the target (Main Screen Frame). When detecting the above drag and drop form, the object that is shown in the Image Viewer is drawn at the Main Screen Frame. The PictureMaker class takes care that ‘younger’ objects that are drawn at the Main Screen Frame will hide ‘older’ objects that were drawn before. The class has a Combo Box component, that is filled by icons’ names from the images directory every time the application starts. This means that adding an icon to the image directory will add another icon to the application the next time the application will be started. When the user chooses a filename (icon) in the Combo Box, the Image Viewer Frame will show the selected icon. An object that is created during run time appears in 3 frames: Main Screen Frame VOPD Tree Frame Layers Frame The picture maker class takes care of full coordination between those 3 frames. That means that when the user selects a certain object in every frame, a message is sent to the class and the PictureMaker class activates the selected object in all 3 frames. 12 The PictureMaker class keeps track of the number of objects that exist in the application and who is the current object that is selected by the user. The picture maker class is responsible of the coordination between all the classes, using shared memory that is actually a part of the class itself. This shared memory is an array of instances of Item_Package class. The class also keeps track of the location of the line separator at any moment and keeps a counter that counts the number of groups. The picture maker class contains also a menu bar with Load, Save and Restart options: Save –saving the current state of the application as a file name that is given by the user. Load – loading and translating of a picture maker file and reconstruct the saves state. Restart – a possibility to restart the application, this is done by resetting all the databases to the initial values. 13 Item_Package Class The Item_Package class is a sub-class of the PictureMaker class. It represents an object, which means that every object is an instance of the Item_Package class. Every object has several attributes as shown below: Class Item_Package { JInternalFrame icon_pack; Int SerialNumber; Int SonOf; Color color; Rectangle location; String FileName; JPopupMenu popup; Int num_of_relations; String [] relations; Int [] relate_to; Int num_of_groups; Int [] groups; . . . } Each object appears in 3 frames. It is displayed in the Main Frame by an internal frame that contains an image of the object icon. Each internal frame has a certain size and a certain location that can be changed by action listeners that is attached to the internal frame. The internal frame appears where the user has dropped the mouse and the initial size of the internal frame is 80*80 pixels. In addition, by double clicking the image icon, we eliminate the internal frame and drawing the image itself, where the internal frame was at the size of the formal internal frame. 14 This is reversible, since the drawn image has an action listener too, that enables the user to click the image and receive the internal frame. Every object’s color is created during the application’s running and assigned by initial value of 100,100,100 in RGB format. Every object can be a son of (part of) another object but only 1 object (a father can have several sons, but a son can have only 1 father). Therefore 1 parameter is more than enough to describe the son of status of an object. The SonOf parameter is assigned as –1 at the beginning and is changed during the application’s running. The value of the SonOf parameter indicates the SerialNumber of the father. For example, if we want to set object number 5 as son of object number 3, all we have to do is assign object number five’s SonOf parameter to 3. Every object can be also assigned to a group or groups. This is done by incrementing the num_of_groups parameter by 1 and set the number of the group in the group array of the object. The same algorithm is used to record a relation between two nodes. Each instance of Item_Package can have up to 10 relations and can be assigned to up to 10 groups. The Item_Package contains a popup with these items: Full Screen – allows the user to expand the internal frame of a certain object to the size of the Main Screen Frame. This option is used mainly in case that a uniform background is needed for example sky. Change Object Color – allows the user to change the value of the object color. Note that the color of the object is not changed, only the value in the Data Structure. The color changing is done by the user using a Color Chooser window as shown below: 15 Figure 3 – Color Chooser Show properties - allows the user to view a selected object properties and its attributes as shown in the below figure. Figure 4 – Show Properties Screen 16 o Serial Number – the serial number of the object in the array of objects and layers order o Name – indicating the object name as transferred from the images file. o Location & Size- parameters that are transferred by internal frame methods o Color – the current value of object’s color (default value or changed value) o Truncated - defines whether this object is lying on the Main Screen frame board. This parameter is a Boolean parameter. o Occluded - defines whether this object is occluded by another object it’s done by using a method that checks whether two objects (representing by internal frames) are coincided. Set as FG separator – allows the user to set a specific object as the object that separates between the background and foreground of the image. This operation is done by setting the line separator of the entire application according to the serial number of the object that selected. 17 Dynamic Tree class The dynamic Tree class is a shell class for the VOPD Tree Frame. The main object of the class is an instance of Jtree. This is a java API that supplies the tree form class with useful methods that simplify the treatment of trees and changes their construction according to the updating and changes that are occurring during application run time. While the application is running for the first time, the tree creates 3 items (before the user inserts objects and starts drawing his picture) The items are: My image FG (fore ground) BG (background) It is important to realize that these items are semi-virtual objects from the picturemaker class point of view, and do not exist in the logical view of objects that are created during run time. As long as there are no changes in the FG/BG separator, all of the previously created objects are displayed as apart of the FG item In case that there was a change in the separator, there is also a change at the BG and FG location in the tree. The appropriate objects are transferred under the BG item according to line separator’s new location. Among the many methods that the Java API JTree supplies, there are some methods that are more crucial to the application – methods that enable us to manage tracking on which node is currently selected by the user, if more than one node is selected and according to this data control what occurs and changes in the tree class. The dynamic tree class controls and operates the application’s central features such as: Grouping, Set as part of and set relation. These features are available using popup menus that appear when the application indicates a right mouse clicking on one or more items in the tree. Grouping – An option to assign a few objects as a group. In addition, there is also an option to nest groups. When the dynamic tree class detects that grouping is requested, the class ascertains that more than one item is selected 18 and also that the selected items are contiguous (in case one of the above does not occur, the application displays an error message that specifies exactly why the operation was aborted). The dynamic tree class updates the data structure so all the selected items will be assigned as the same group number. Nesting grouping is permitted only in items that belong to the same group, meaning that in any case of selecting the grouping option, the items must belong to the same group as can be shown below. Permitted and not permitted grouping situation are shown below. Figure 5 – Correct Grouping Figure 6 - Incorrect Grouping In the same popup menu there is also an un-grouping option – the reverse operation to grouping. This option affects the only items that belong to the last common group. Note that this option is possible also on clicking single item when the dynamic tree class recognizes his rest group members according to nesting level. 19 Set As Part Of – a feature that enables the user to set an object as part of another object. Nesting is also permitted here. The moment the dynamic tree class detects that this option is selected, the class ascertains that more than one item is selected and also that there were selected contiguous items (in case one of the above does not occur the application notify the user by an error message). The dynamic tree class updates the data structure in a form that the first selected item that was selected (the item with the lowest serial number) become the father of the rest of the item(s). Note that there is one and only one father to each item. At this point, by using refresh tree () method the tree with the updating data is redrawn by the application. UnPart – This option is available in the same popup menu and this is the reverse operation to ‘ set as part of’. The operation can be applied only to items that have no children and they are the youngest in the children chain of the item that was pointed as the father. In any other case, an error message will appear to the user. Set Relation – An option to assign a relation between two objects. This is a free text relation. The moment the dynamic tree class detects that this option was selected, the class updates the data structure (the objects array) in a form that the relation that was typed by the user will be inserted and displayed in the database as relation between those two nodes. Note that for two specific nodes there are up to 10 relations permitted by the application. In dynamic tree class, there is quite an important function: refresh tree (). This function role is to redrawn the tree according to the changes that took place during application run time and caused updating the data structure- objects array. 20 Layers class The layers class is a shell class for the layers Frame. The main object of this class is instance of java API JList that is a simple list of items structure that operates and control it’s selected items. The class keeps track of the order, which the objects of the application will be drawn on the main screen frame. The JList object contains all the objects and the separator item. As previously mentioned, we can say that from picture maker class point of view it is a virtual object and it is transparent. The separator initial location in the JList is 0. The entire operation of the layers class is managed and controlled via mouse listeners that belong to this class and enable the class to detect an item selection from the list of layers at any moment. The layers class enables several features on a single item from the list (it actually represents an object) such as: Move Layer Up, Move Layer Down and Set Separator. Move Layer Up/Down – option that allows the user to play with the order of the layers in the layers frame and by that also affect the order the picture will be drawn. The moment the class detects that this option is selected attaching to specific object is ascertains that only one item was selected and this item is not a father of any other item, the class searches the place, which this item will be put. It is done by locating the brother item – i.e. the first item that locating above or below the selected item (according to chosen operation) or in other words, the first item with the same father as the selected item. In case there is no common father i.e. that all the items in application are children of FG/BG (marked –1 in the ‘son of’ field) and they are in the same level. The selected node is copied temporally and assign in the end of the data structure and after lowering/rising all the items that should be moved one place down or up the item copies from the temporally place to the new place that was request by the user. In case the selected item is the separator after the class detects that the user selects the separator the class search for the first place that the separator can be assigned when impossible place is defined inside “family” of items or just in the edges of the list. 21 Note that after all the operation mentioned above, all the frames are updated according to the changes that were done through the operation. For example, if the sun icon is selected to 'move layer down' as shown below: Figure 7 – Move Layer Example Because the icon sun is above icon that has 2 Children, the sun icon will skip over the 3 icons and relocate right after them as shown below: Figure 8 – Result of Applying Move layer Down 22 Chapter 4 - Summary The main goal of the Picture Maker -VOPD was to built a friendly interface application, comfortable as possible to the user. This interface is due to be a part of a larger project that its main and final goal is building a visual search engine. This search engine will improve and correct the current situation, as mentioned in the abstract. The secondary goal is that by accomplishing the first goal, the application shall supply all the tools to build, set and indicate all the features and relations that are vital part of the visual object process diagram. In our opinion, we have achieves these main goals. In addition, we feel that the application is quite appropriate for future extensions for the bigger VOPD project, since application's output is well organized and simple to process, thanks to the file's organization of the data that is needed for further development from this point on. Another goal that was partially a background requirement was that the application would be written in JAVA language so that it could run and operate on any platform, especially on web compatible platform. By fulfilling this goal, this project provided us with much needed background and experience with a quite powerful object oriented language. Through the programming we became aware of the enormous capabilities of this language, especially in the graphic field. At this project's stage, we could have written the code for the same application in a visual C++ environment - something that would have probably accelerated the application's operation, but on the other hand, we could not have accomplished the goal of "any platform compatibility". This project gave us the opportunity to become more skillful in object-oriented programming and in the same time to get to know the problems in computer vision, especially objects detection in a given picture. The picture maker project was written at the same time with other project that use search algorithm in order to find the nearest picture to the requested picture we are looking for. 23 The next stage is to combine both projects - combination that would lead to completion of the bigger project of VOPD under the refinement that had taken about the fully automatic requirement of the system. 24 Appendix – Operating Instruction In the following pages we will try to get a quick and simple summary of the application’s capabilities and operation. In addition, we will introduce a short introduction of the best way to use this application combined with all the extra tools. Staring drawing Icons in the Main Screen In order to put icons in the Main Screen, all you have to do is drag an image from the Image Viewer Frame and drop it in the Main Screen Frame as shown below: Figure 9 – Example of Dragging an Icon 25 As you can see, the new icon was inserted to the main screen. The VOPD Tree Frame also reflects the added icon, as son of FG (ForeGround). This icon insertion is also displayed in the Layers Frame. Each icon can be closed, resized and moved in the main screen. During these operations, the icon will be displayed as shown above (as internal frame). Double clicking the image itself will hide the internal frame and will show only the frame’s inside image. The image will be scaled to the frame’s size. When another icon is selected or dragged to the main screen it will be considered as double clicking on the old icon. Grouping Option in the application that enable to group several nodes together. In the below example there is grouping of 3 trees icons that were drawn before. In order to group nodes together, you should: 1. Select several contiguous icons in the VOPD Tree Frame (using the CTRL key or the shift key). 2. Right click with the mouse. A new popup should be displayed. 3. Choose "Set As A Group". You should see a change in the VOPD tree frame (the group number will appear right after the name). 26 Figure 10 – Grouping Screen Notes: 1. You should select only contiguous nodes. Any other selected will cause an error message to appear. 2. The application enables group nesting. The nested group should be fully inside the "parent group". 3. Grouped objects are not stacked together, so moving a grouped icon will not affect the grouping. 4. Ungrouping option is also available. 5. In order to ungroup all you have to do is select a node (or more) in the group and select UnGroup in the popup menu. 27 Setting As Part Of Another option of the application is to set a ' part of ' relation between two nodes. In order to set a 'part of ' relation: 1. Select at least two contiguous nodes in the VOPD tree frame. 2. Right click on the mouse. A new popup should be displayed. 3. Choose " Set As Part Of ". The first selected node should be a 'father' of all the other selected nodes. Figure 11 – Set as Part of Screen Notes: 1. The order of the selection of the nodes is not important. The father will be the oldest object and not the first selected node. 2. The program enables nested 'Part Of' relations. 3. UnPart function is also available. 4. In Order to UnPart you should select only 1 node at a time. This node should be the 'youngest' node in the chain of the 'Part Of' relation. 28 Setting up a Relation There is an option to specify a relation between nodes. In order to set up a relation: 1. Select 2 nodes in the VOPD tree frame. The selection’s order is important. 2. Right click on the mouse. A new popup should be displayed. 3. Choose "Set Relation". Figure 12 – Set Relation Screen 4. A new frame will appear as shown below. 5. Type the relation between the two nodes. 29 Figure 13 – Relation-Typing Screen Notes: 1. The application enables up to 10 relations. 2. You can type the relation in Hebrew or in other language. FG/BG Separator The separator separates between the FG (ForeGround) and the BG (BackGround). There are 3 ways to set the separator. The first is done through the Main Screen Frame by selecting an object and choosing 'Set as foreground Separator' in the popup menu. This selected icon will become the new separator as shown below. The second way is choosing and icon in the Layers Frame, and choosing 'set separator' option in the popup menu. The third way to set the separator is through the Layers Frame by selecting the line separator (the dashed line in that frame) itself and selecting move up/move down from the layers popup menu (will be shown further). 30 Figure 14 – Set as Foreground Separator Screen The result of setting the separator: all the VOPD tree structure is changed according to the location of the separator selection. The Layers Frame also reflects the new situation. 31 Figure 15 – Result of Applying Set as Foreground Separator Screen Moving layers Up and Down There is an option to change the layers’ order in the layers frame. In Order to move the layers up and down: 1. Select an icon in the layers frame. 2. Choose 'Move Layer Up' or 'Move Layer Down' in the popup menu. 32 Figure 16 – Move Layer Up Screen As a result, the layer will go up or down. The order of layers will be changed in all frames. 33 Figure 17 –Result of Applying Move Layer Up Screen Notes: 1. If you choose the line separator, the line separator will go up or down and appear where it can (The line separator cannot be in the middle of 'part of' relation.) 2. A node will go up and down until it meets a 'brother node' or a 'father node'. That means that the new location of the selected icon will be above or below (according the chosen operation) the first icon, which its level is the same of the selected icon. In case icon selected to 'move layer up/down' and it’s located above or under group of icons there will be no skipping above the icons and the selected icon will squeeze among the grouped icons. 34 Extra features of vopd application There are several features the PictureMaker application supplies, that are available through popup menu that are displayed by right clicking on specific icon in the main screen as shown below: Figure 18 – Show Properties Screen Full screen: This option enables selected icon to be drawn and fill all the application’s main window. It is useful mainly for specific icons like sky etc. Change object color: The created objects that are drawn during the application’s running have a default color value in RGB format. This default value is 100,100,100. There is an option to change this color by pointing to 'change object color' item in the popup menu that will lead to appearing of this color chooser. Note: The object color isn't changed, only the value of its RGB value represented 35 Show item’s properties: There is an option to display the properties of a selected icon by choosing 'show item properties' item in the popup menu. The properties are organized in a frame, that is shown below: 36