ManifoldCF for Content
Acquisition
Karl Wright, Nokia Inc.
kwright@apache.org, 11/10/2011
What this presentation is about
 An introduction to ManifoldCF
 Presenter: Karl Wright, original
ManifoldCF developer
 Challenge: Getting content into a
search engine, keeping it up to date,
and securing it
Information about me
 My name is Karl Wright
 Principal Software Engineer at Nokia, Inc.
 Former Principal Software Engineer at
MetaCarta, Inc.
 Core committer for ManifoldCF
 Author of ManifoldCF in Action
ManifoldCF…
• Pulls documents from
disparate sources
• Writes documents into
the target(s) of your
choice
• Provides an end-user
authorization
mechanism
• Synchronizes, doesn’t
just crawl once!
• Has bounded memory
usage
• Is reasonably
performant
• Is extendible to new
kinds of repositories
• Shows you what it is
doing
• Is resilient against
restart
ManifoldCF vs. Nutch, Heritrix
Heritrix
Nutch
ManifoldCF
Tree operations?
No
Yes
Some
Web only?
Yes
Http, ftp, svn
All sorts of content
UI?
Yes
No
Yes
Restartable?
Painful
Uses Hadoop
Yes
Incremental?
Not really
Basic support
Yes
Max docs
“web scale”
100,000,000
Technically no limit;
10,000,000 tested
(using postgresql)
Docs/sec
80+ per instance
Scales as needed
80+ per instance
(using postgresql)
Memory bounded?
No
Uses Hadoop
Yes
Security model?
No
No
Yes
How does ManifoldCF fit?
Just how many kinds of document
repositories are out there?
•
•
•
•
•
•
•
•
File systems (CIFS too)
Windows shares
The Web (RSS too)
Wikis
Databases
CMIS repositories
SharePoint (Microsoft)
FileNet (IBM)
•
•
•
•
Documentum (EMC)
LiveLink (OpenText)
Meridio (Autonomy)
Many, many more
What is a connector?
• A connector is code implementing an interface
• ManifoldCF uses three kinds of ‘connector’
– “Authority connector” understands a specific
authorization entity, e.g. AD or LiveLink
– “Repository connector” understands a specific
content repository, e.g. Windows shares or
Documentum
– “Output connector” understands a specific output
destination, e.g. Apache Solr or OpenSearchServer
Connections and jobs
• A ‘connection’ is a configured instance of a
‘connector’ object
– Connections are pooled
– Max number of similar connections is configurable
• Jobs describe “what” and “when”, not “how”
– Has a repository connection and an output
connection
– Not really a task; but rather a set of documents
ManifoldCF Document Flow
ManifoldCF’s Crawling Models
• Push vs. Pull
– Observation: ‘Push’ model may require
notifications to be queued 1
– Observation: ‘Push’ is no longer an option if ANY
notification is overlooked 2
– Observation: There are no real-world systems I’ve
found that really support ‘push’!
• ManifoldCF uses ‘pull’ exclusively right now
ManifoldCF’s Crawling Models, ctd.
• For incremental ‘pull’:
– Need to periodically identify documents that have
‘changed’ within a given time window
– Changes include “add”, “modify”, or “delete”
– Only a few repositories can tell you about “delete”
• Connectors in ManifoldCF declare their ability
to detect different kinds of changes
Continuous vs. Periodic Crawling
• ‘Continuous’ crawling
– Can’t delete documents from index unless they are discovered
missing on refetch
– Can refetch or expire documents on a dynamic schedule
– Can reseed, also on a schedule
• ‘Periodic’ crawling
– MODEL_ADD_CHANGE_DELETE, MODEL_ADD, or MODEL_ALL
– A MODEL_ALL connector is “stupid”, a
MODEL_ADD_CHANGE_DELETE one is “brilliant”
– Two kinds of cycle: Seeding, discovery/processing/indexing,
(maybe) clean up
– Complex decision as to which kind happens, based on both
connector model and job state
Crawling models, graphic
Dealing with Disparate Systems
• Connection configuration stored as
XML in the database
• A job’s document specification and
output specification are also
stored as XML
• Connector-defined unlimited
strings for document identifier,
document version, output version,
access token
• Connector provides UI for editing
its configuration, specification
Example: File system job
… vs. Web Job
MCF Process Architecture
ManifoldCF Authorization
Requirements
• Observation: Every repository has its
own notion of document authorization
• Observation: Most repositories are
effectively ACL-based
• Observation: Active Directory handles
95% of enterprise authentication
• ManifoldCF idea: Enforce repository’s
existing security model, rather than
inventing something new
ManifoldCF Document Authorization
• Observation: A separate crawl for each end user
is not going to work
• Observation: Post-filtering of search results has
some nasty edge cases 1
• Observation: Document security doesn’t change
very often 2
• Observation: User changes should take effect
immediately 3
• ManifoldCF filters by search-engine query
– Document access tokens are passed to the target
– User access tokens are obtained at search time, via
the MCF Authority Service
MCF Security Architecture
Securing Documents from Multiple
Repositories
• You can define multiple authority connections in
ManifoldCF
• Each authority connection supplies its own access
tokens
• Every repository connection has an MCF
authority connection
• All access tokens from an authority are qualified
with the authority connection name
So how do I write a connector?
• Write a class implementing
an interface
– IOutputConnector
– IAuthorityConnector
– IRepositoryConnector
• Build and deploy
• Register it, or add it to
connectors.xml for the Quick
Start
• That’s it! You’re done!
• Read ManifoldCF in Action if
you want to do it right
Who has used ManifoldCF?
What’s new in the last 12 months?
•
•
•
•
•
Name change
Three releases
ManifoldCF in Action
Quick Start example
ManifoldCF API Service
(REST style, uses JSON)
• Scripting language
• Solr plugin distribution
• Hsqldb, Derby support
• Wiki connector
• CMIS repository
connector
• OpenSearchServer
output connector
What’s coming?
• Better scalability via NoSQL
(Voldemort?)
• Post-search document filtering
support
• Always more connectors and
performance improvements
• MySQL support
Shameless Plug for “ManifoldCF in
Action”
• Available as “early access”
from Manning Publishing
• Helpful for users, integrators,
and connector writers
• Won’t be put into production
until ManifoldCF grows, so
please help us to do that!
Resources
• ManifoldCF in Action, from Manning Publishing
– http://www.manning.com/wright
• ManifoldCF deployment instructions
– http://incubator.apache.org/connectors/how-tobuild-and-deploy.html
• ManifoldCF API documentation
– http://incubator.apache.org/connectors/programmati
c-operation.html
• ManifoldCF script language documentation
– http://incubator.apache.org/connectors/script.html
Contact
 Karl Wright
• kwright@apache.org
• http://manifoldcfinaction.blogspot.com