Open Data Policy and
Practice
Prof. Beth Simone Noveck
Dominic Mauro
Institute for Information Law and Policy
New York Law School
V 1.0 May 23, 2011
Edits and comments to
dominic.mauro@law.nyls.edu
Agenda
• To provide guidance, informed by various models, including
San Francisco, Portland and the US Federal Government,
about best practices for legislating and implementing open data.
1) Open Data Policy – suggestions for drafting open data
policy.
2) Open Data Practice – suggestions for crafting an
implementation strategy.
1) Open Data Policy
Open Data Policy: Suggestions for Legislation
Legislation should not be long or
complicated so long as there exists a
robust implementation plan.
•
Goals
•
Defining Data
• Legislation should be addressed to the
•
Open Formats
•
Implementation
•
public in plain English reserving
technical detail for subsequent
implementation.
• Topics to include:
• Setting Priorities
•
Deadlines and Milestones
•
Enforcement
•
Prizes and Challenges
•
Budget
Open Data Policy: The Value Proposition
• By explaining why open data matters, you will ease the path to
compliance. Communicate that:
• open data refers to more than data about the functioning of
government but includes the data that the city holds.
• open data legislation enables collaboration using data to the end of
solving problems, not on releasing data for its own sake.
• For example: “To amend the administrative code of the city of New York,
in relation to making appropriate data available to the public to the end of
enabling collaboration between the public and the public sector in
identifying data-driven efficiencies for government, innovative strategies
for social progress, and economic opportunities for all New Yorkers.”
Here’s An Example of What Others Have Said
• Portland
• WHEREAS, the City of Portland is committed to engaging the community by working with
citizens, soliciting their ideas, input, and creative energy; and
• WHEREAS, the City of Portland is committed to using technology to foster open, transparent, and
accessible government; and
• WHEREAS, by sharing data freely, the City of Portland seeks to develop opportunities for
economic development, commerce, increased investment, and civic engagement for Citizens of
the Portland region; and
• WHEREAS, publishing structured standardized data in machine readable formats creates new
opportunities for information from different sources to be combined and visualized in new and
unexpected ways, for niche markets to be identified and developed, and for Citizens to browse,
interpret and draw attention to trends or issues with greater efficiency; and
• WHEREAS the adoption of open standards improves transparency, access to public information,
and improved coordination and efficiencies among bureaus and partner organizations across the
public, non-profit and private sectors; and
• WHEREAS, the City seeks to encourage the local software community to develop software
applications and tools to collect, organize, and share public data in new and innovative ways;
Here’s An Example of What Others Have Said
• San Francisco: goal of legislation is to enhance government
transparency and accountability, develop new applications based on the
unique data the city provides, mobilize the city’s high-tech workforce at
no cost to the city, and create social and economic benefits based on
innovation in how residents interact with government.
• White House: open government directive “intended to direct executive
departments and agencies to take specific actions to implement the
principles of transparency, participation, and collaboration ... toward the
goal of creating a more open government.”
• Model: the stated goal is to justify city officials in directing resources
toward transparency projects and have no political excuse for not
complying.
Open Data Policy: Defining Data
• Legislation should mandate that all data be inventoried.
• Prioritize types of information to release.
• Defining “data” or “dataset” must not preclude textual
information such as the laws and ordinances. One of the most
successful open gov projects to date has been the revamped
Federal Register.
• Find the right balance between defining data in the legislation
and the implementation plan.
Here’s How Others Have Defined Data
• Federal Government Definition:
– High-value information is information that can be used to
increase agency accountability and responsiveness; improve
public knowledge of the agency and its operations; further the
core mission of the agency; create economic opportunity; or
respond to need and demand as identified through public
consultation.
• This general prescription to policymakers was followed by specific
instructions articulated by the CIO to the CIO Council with the input
of public interest and technology groups.
Open Data Policy: Open Formats
• Any legislation needs to require that data be released in
human- and machine-readable formats.
• Legislation can list examples of open formats but details of
domain specific metadata can be provided in subsequent
guidance to CIOs, especially standards for providing an API to
enable applications to continually use up-to-date information.
Here’s What Others Have Said
• Federal Open Government Directive:
– “Each agency shall take prompt steps to expand access to
information by making it available online in open formats.”
– “To the extent practicable and subject to valid restrictions,
agencies should publish information online in an open format
that can be retrieved, downloaded, indexed, and searched by
commonly used web search applications. An open format is
one that is platform independent, machine readable, and made
available to the public without restrictions that would impede
the re-use of that information.”
Here’s What Others Have Said
• Model Open Government Initiative:
– Strongly recommends the publication of multiple open formats
to maximize usefulness of information.
– Gives examples of open formats: HTML, XML, CSV, JSON,
RDF, or XHTML. Information must be human-readable as well
as machine-readable.
– If information is published in PDF format, it must be configured
to open with bookmarks for enhanced navigation on-screen.
Open Data Policy: Implementation
• The legislation is only as good as the implementation plan that follows it.
Legislation should mandate an open public process for developing the
implementation plan.
• Instead of directing one department to craft a plan to make a plan, legislation
should both: 1) appoint one department as the coordinator/enforcer to craft an
implementation plan; and 2) direct all agencies to develop their own open data
plans to inventory their data.
• There should be an aggressive timeline for progress towards a complete data
inventory, regular publication of data sets starting with some number within 30-45
days of enactment, and a mandatory and iterative public consultation process.
• The coordinating department should be mandated to develop technical
guidelines, provide support and convene the agencies but everyone should be
addressed.
Here is What Others Said
• Federal Open Government Directive:
– Individual agencies are explicitly directed to take action on their
own.
– Examples include publishing data sets, creating web portals,
developing their own open government plans, responding to
feedback from members of the public, and designating a senior
official to be responsible for the quality of published data sets.
• Model Open Government Directive:
– City departments are directed to publish information online and
develop a schedule for their release of data sets.
– Further, they are to develop their own open government web page,
including a mechanism to receive and respond to public feedback.
Open Data Policy: Setting Priorities
• An interagency working group of senior officials should develop
priorities together by identifying challenges (ie. reducing
homelessness, improving literacy, fighting obesity) and then
cataloging data each agency holds that can help solve them.
• Any information provided under local open records laws (FOIL)
should be published on the open data portal automatically.
• Information that has been the most often requested should
always be prioritized for publication.
Open Data Policy: Setting Priorities (cont’d)
• In addition to milestones for data publication, legislation should
mandate milestones for:
– public consultation in setting priorities citywide
– public consultation on an agency-by-agency basis in connection
with the data inventory
– public engagement in connection with using public data to
address priorities and problems
• All agencies should articulate examples in plain English of what they
hope the public will do with the data to the end of helping residents.
• All new regulations should be published with accompanying data on
which the regulation was based.
Open Data Policy: Deadlines and Milestones
• Release data sets early and often. At the federal level, the Open Gov Directive
mandated:
– Release of initial data in 45 days.
– Creating inter-agency working group to develop priorities and best practices for
data identification, dissemination, and public collaboration within 45 days.
– Establishment of every agency’s open data page within 60 days (to feed central
portal).
• It is reasonable to catalog public information already online within 90 days. There are
tools to automate this process. A year is reasonable for a full inventory.
• Mandate that support be provided to facilitate this process (see Implementation), ie. the
department should delineate the tools and resources available for agencies to publish
their data sets as soon as possible.
• Mandate that all data sets be released with examples of what the agency is hoping the
public will do with the data set, ie. we hope that someone will develop mashups using
this environmental quality data in order to help identify priority sites in need of clean-up.
• Deadlines to establish deadlines don’t count!
Here’s What Others Have Said
• Federal Open Government Directive:
– requires each agency to identify and publish in an open format three high-value
data sets within 45 days. Also within 45 days, each agency must designate a
senior official to be accountable to for the quality of the published data sets, and
the Federal OMB, CIO and CTO will establish a working group to focus on
transparency.
– Within 60 days, each agency must create an open government webpage; further,
the OMB will release a test data set about federal spending, and later assess
whether additional quality controls for released information are needed. Also within
60 days, the Federal CIO and CTO must create an open government dashboard.
Within 120 days, each agency must publish an open government plan on its web
site.
– There is a general agreement that in a matter of months, a portal should be set up
and an open data plan be published for each agency. Publishing an immediate
series of high-value data sets is also a common theme in the Model language and
the Federal language as a near-term goal.
Open Data Policy: Enforcement
• Agencies should be responsible to the public not simply to another
agency as that would only create a further bottleneck.
• Public consultation should be a central part of any open data legislation.
• For example, at the federal level each agency is required to set up a
mechanism for the public to:
– “Give feedback on and assessment of the quality of published
information;
– Provide input about which information to prioritize for publication.”
– And is required to give feedback to the public on a regular basis.
• In addition, there should be sticks and carrots for enforcing compliance
with deadlines.
Open Data Policy: Prizes
• If further legal authority is needed for agencies to use prizes and
challenges for use in running apps contests using open data then it
should be added to the open data legislation.
• Prizes and challenges have had high returns on investment.
• For prizes totaling $20,000, the NYC Big Apps competition
generated $4.25 million in apps that used data sets from the NYC
Data Mine.
• There should be challenges running 365 days a year, with a
different vertical focus each month and the ability to develop both
big and small, publicly and privately funded challenges.
• ‘The legislation should create the authority to explore use of cash
and non-cash prizes as well as contracts and in-kind gifts.
Open Data Policy: Budget
• Legislation should provide a modest budget to support:
–the work of the coordinating agency to hire an open data
expert to develop the implementation plan and advise the
agencies on their implementation process, especially how
to make data available to help achieve important goals.
–the acquisition of software for automating crawling
domains to “discover” data sets. With the right tools, we
can drastically reduce the effort required to extract data
and put it in the cloud.
–Public seed fund to match private sector donations in
support of prizes and challenges.
2) Open Data Practice
Open Data: Practice
• The slides that follow lay out suggestions for the plan to
implement the legislation.
• The basics of that plan should be in place prior to legislating to
ensure a pathway to implementation.
• A single agency should “own” coordination of implementation
and, in addition, every agency needs a senior official with
policymaking authority to take responsibility in the agency.
Open Data Practice: Release Early and Often
• Require immediate release of a small number of datasets.
• By publishing data sets immediately, this will accustom
agencies to the practice and help to identify problems and
opportunities early.
• The federal directive required the release of three high-value
datasets in the first 45 days.
Open Data Practice:
Require a Comprehensive Inventory
• Agencies must know what they have.
• Each agency should create a team comprising technical,
legal/FOIL, and policy to oversee the process of getting the
agency to inventory its data.
• Crawlers can help to automate the process of finding data on
the domain.
Open Data Practice: Publish Data on a Rolling
Schedule
• Start by “doing something.” Release data!
• Require a regular release of data on a schedule, ie. at least # of
data sets per quarter.
• Citizen enthusiasm and participation will decrease if data is only
published annually.
Open Data Practice: Articulate Priorities
• Adopt a regular schedule for articulating priorities, ie. reducing
childhood obesity, improving literacy or reducing homelessness.
• Every agency has to respond with whatever data it has or that
the public demands that could help achieve that goal.
• In addition to data in support of agreed upon priorities, prioritize:
1)accountability data (specify certain common data sets to start);
2)data requested by the public;
3)data designed to achieve the agency's core mission;
4)data to create economic opportunity.
Open Data Practice: Establish a Community of
Practice
• Convene an inter-agency community to exchange best
practices on doing the data inventory.
• Agencies to be represented by senior officials with policymaking
authority not only CIOs.
• With hundreds of thousands of data sets waiting to be
published, it’s important to have help with knowing how to take
stock of an agency’s information. Agencies can help one
another.
Open Data Practice: Interdisciplinary Teams in
Agencies
• Doing open data right requires legal, technical, and policy input.
Create teams from different disciplines within agencies.
• Use this opportunity to ensure that CIOs and FOIL officers,
technical and policy people, communications and new media
are all talking to one another about how to use data to achieve
the agency’s core mission.
Open Data Practice: Coordination
• One central department with technical expertise should provide
assistance, such as:
–a blanket purchase agreement for software to inventory
agency domains
–Developing model data standards and formats
Open Data Practice: Public-Private “SWAT” Teams
• Rather than have all the work fall to one Department, use large
community of advocates for Implementation and Accountability.
• Establish “SWAT” team with representative from coordinating
department and two organizations from the transparency
community. The swat team should include one open data techie
and one good government advocate.
• Swat team provides advice and support and helps to “market”
availability of new data sets to the community interested in
using them.
Open Data Practice: Establish Regular Mash-a-Thons
• Every quarter, Administration and agencies articulate a priority,
identify and release data relating to addressing that issue.
• An agency is selected to coordinate.
• Public-private partnerships established to fund prizes and
challenges in connection with that issue.
• Mash-a-thons scheduled to support development of data-driven
solutions.