Each call to vect_file consists of taking the words from a region in a file, stemming them, discarding common, meaningless words, and converting them to integer values (wordcodes). A binary tree is built up with the wordcodes as keys to the nodes, each node containing the number of occurences of the word in the document. When completed, the tree is flattened into an array by an in-order traversal, resulting in the desired docvec. Zeroes are inserted into the array for wordcodes which do not appear in the tree, but do appear in other documents in the collection. The wordcodes which have appeared globally, so to speak, are kept track of with a lookup tree, whose nodes are sorted by wordcode like the document trees above. In this case the nodes contain the index into a docvec of their word, e.g., if the entry at node "banana" in this tree is 87, then j_random_docvec is the number of occurences of "banana" in the document. When a document tree is flattened into a docvec, it is compared with this lookup tree so that zeroes can be inserted appropriately. Conversely, if the document contains a word not appearing in the lookup tree, a new node is created for it, containing the next highest index. init_collection() initializes the lookup tree. For now, the vectors returned will have a sparse representation. Densification may come later. A docvec will be a struct containing an int indicating number of entries (because this will grow as indexing progresses) and an array of ints, the vector itself. See vector.h for more info. We'll also have stuff in here for finding the covariance matrix of a bunch of docvecs. -------------------------------------------------There's some conceptually important stuff in here. We have two kinds of vectors, a DocVec and a WordVec. The first is associated with a particular document from the collection (or a query) and contains arrays of word hashes and weights, the frequencies of those words in the document. The second is associated with a particular word appearing in the collection, and contains an array of packed integers, each of which has a field to identify a document in which the word appears and a field that has the frequency of the word in that document. Basically, we can think of a document as a vector in "wordspace", a vector space whose basis vectors correspond to words. e.g., "Now is the time for all quick brown men to jump over their lazy country" is parsed into separate words and has the common words discarded, leaving "time quick brown men jump lazy country". This can be thought of as a vector whose "time" component is 1, "quick" component is 1, etc. If we perform this operation on all documents in a collection, we can represent the collection as a matrix, setting all these (column) vectors side by side. Originally, savant used only these document vectors; collections were saved as sequences of these vectors and retrieval was performed by sucking the whole thing into memory (since every single document vector has to be examined in a query). It turns out, however, it is much more convenient to store and retrieve the row vectors of the matrix mentioned above. These are the WordVec's. With these structures saved on disk, we need only load one at a time - as we are finding the best matches for a query, we are only looking at one word from it at a time, so we need only grab from disk the row vector of the matrix that corresponds to that word. We add the weights in this vector to a running total and discard it. Very light on memory, and still very fast.