CycleStreets gets a lot of feedback, currently running at a few hundred user reports a month. The current level and system for managing it is beyond the capacity of the current personnel to handle.
Levels are expected to increase considerably as mobile use expands and as developers implement the feedback API call (which enables route bug reporting directly from apps).
This paper outlines a proposed new specification for a feedback handling system. Once reviewed and approved, code (object-PHP/MySQL) can be written, mostly just integrating existing classes that handle things like map display, form handling, etc.
It is our aim to develop the current system to a more useful feedback triaging/handling system, which helps and facilitates the work of OSM people to improve the map.
The quality of reported bugs is actually generally fairly high – users tend to give enough detail to enable the problem to be investigated properly. Simple “Problem route” feedbacks are very rare.
However, this feedback needs triaging before it can be investigated, and potentially pushed out to other systems like MapDust, because often the problem can be engine-related (rather than datarelated), or is feedback on another part of the system (e.g. Photomap), or contains personal data that must be removed. Furthermore, discussion with the reporting user has often been shown to improve the quality of the data repair considerably.
The current backend is basically a database table that stores the itinerary ID and comments (plus name and e-mail etc). The category of feedback (e.g. route, general, photomap, etc.) is also stored.
Feedback reports also result in e-mail to an e-mail list.
For route feedback, which is the most heavily-submitted type of feedback and the main type needing a better system of management, following the link in the e-mail to the feedback list, and signing in with a signin username which has the 'feedback' privilege, results in viewing the standard itinerary page but with an additional form at the top.
This form for the feedback handler contains the feedback from the user, a preset category of feedback, and a box for entering the response (which includes a set of preset texts, with all shown, so irrelevant ones have to be deleted), a box to enter an alternative route number, and a publicly visible note.
A table containing all the feedbacks is available. This contains links to the itinerary page, resulting in the same as above.
1


No map-based view, so it is difficult to handle areas of the country not familiar to feedback handlers or to pick out feedbacks from familiar areas of the country. (Actually there is a map view of the feedback - which is handled in a similar way to a layer on the photomap - click on
"Ways for OSM users" at low zoom. Blue dots are where routes start, red dots are route starts with unresponded feedback, green when a response has been made. This is not very accessible, but at least forms the basis of code for further work.

Assumes only one piece of feedback per itinerary.

Doesn’t have a nice web view for handling non-route data.

Hard to see at a glance what has been dealt with.

No formal connection between the feedback system and the bug tracker.

There is no triaging of problem types.

Only a small number of people handling the feedback, and difficult to expand because of these deficiencies.
This project aims to create a new feedback system that builds on the existing infrastructure but which has the following characteristics:

Primarily: aims to help OSM data errors be fixed easily so that there is a virtuous cycle of improvements that benefits both CycleStreets and OSM (and therefore other routing engines and projects)

Is intended to be delegated: enables OSM people to ‘sign up’ to watch an area of the country

Demonstrate the effect of data improvements, e.g. the DfT cycling data now available.

Maintains personal privacy while enabling individual text submissions to be public, probably by using an initial triaging stage

Has a map view at its heart, as well as a tabular list. Most useful will be a list whose contents relate to a bounding-box or area.

Is not specific to route-handling, i.e. can deal with other types of feedback

For routing feedback, enables a clear distinction between problem types, namely (i) data errors, e.g. disconnected ways, (ii) lack of data in an areas (iii) data translation errors (iv) engine deficiencies

Enables splitting of an item into subparts that effectively count as separate bugs

Is intended to avoid the use of e-mail-based replies altogether, i.e. replies done via the site
(though an e-mail ingesting in-route could be considered).

Provides clear statistics on route-handling levels and problem types
2


Results in no loss of existing data

Entirely avoids use of e-mail for route feedback

Coded so that other routing sites could use the same code, i.e. not tightly coupled to our existing infrastructure, as they presumably face the same kinds of problems if they are handling feedback

Gives the original submitter a clear and ideally non-technical response to reassure them that their feedback was useful. It should mention in passing that they can become involved in
OSM if they wish to, but it should never give them any sense that ‘planning a route on
CycleStreets is hard work’ through inference or suggestion that they have to fix data bugs themselves.
The user submits a feedback as presently. There is also a API feedback route for bug reporting direct from mobile clients.
Dialog boxes relating to feedback submission will make clear that non-personal data (i.e. name/email, if the user has supplied these) will be made available to reviewers but not the public at large.
There will be a small and group of privileged ‘triagers’ who have signed a DPA agreement. There are only two purposes and responsibilities of this status.
The first is simply to ensure that the body text of the feedback becoming viewable to others or publicly contains no personal data. Once each item has been reviewed for personal data, it becomes visible to reviewers and potentially the public.
The second responsibility is to check the overall type of feedback. The available types are:

Route feedback (problematic route)

Photomap

Mobile

User-interface

Feature suggestions

Business proposals

Spam

Duplicate of another feedback ID
For this data protection filtering stage, it is proposed to implement this as a simple HTML table which can be easily scanned through and which by default approves all the shown items. Items which do not comply would have their checkbox unchecked to exempt them from approval, and
3

they can then be edited to strip out personal data. This clears a personal-data-unchecked flag in the database associated with the feedback entry.
Note that journey start/finish points are not considered personal data for the purposes of the feedback system since they are already publicly viewable at the /journey/<id>/ URL. (This is considered acceptable because there is no complete listing of all journeys which could result in user patterns being spotted and give rise to data protection concerns.)
The HTML table will include a select control for each entry in order to set the type, so that it can be manipulated directly en-bloc rather than than having to into each item first.
Where a signed in reviewer is a triager, they should also be able to do the personal data filtering stage at the same time as the actual feedback handling. However, this will result in an additional tick-box that the user must tick to confirm they have eliminated any personal data in the body of the feedback. Doing so will clear the personal-data-unchecked flag.
As a result, by this point, data will be correctly categorised and the route feedback items will be DPAsafe for future stages of manipulation by ordinary reviewers.
OSM experts and CycleStreets developers can enrol for the feedback system via the signin system, giving them ‘reviewer’ status, enabling them to view and ideally respond to the route feedback. A quick webform will be available for them to apply for the reviewer status once signed in with a validated account.
The journey feedback data (not other types) should be visible without being signed in, to encourage browsing by OSM people to increase the number of reviewers, as long as the personal data has definitely been stripped. Having the dialogue (responses) viewable would also be very useful.
New reviewers are first shown a page giving a set of expected norms, to which they must agree in order to proceed further. This gives (a) a brief outline of the feedback handling process; (b) the importance of a polite response at all times, even where the user's feedback was very negative; (c) the importance of never giving the reporting user the impression that problems in the data are the user's responsibility to fix (i.e. the reviewer's job is fundamentally that of a proxy between the user who experienced the problem and the developers or OSM data).
One existing reviewer has suggested that perhaps new reviewers could enjoy a probationary phase during which any responses they make are first moderated by a full reviewer. This is worth considering given that some sites that CycleStreets runs are on behalf of local Councils or other bodies.
Reviewers are then introduced to the settings screen where they can select the (bounding-box) area(s) of the country of interest to them, can select checkboxes for the default types of feedback they wish to know about, and can see stats relating to their activity. These settings are presented at first-run and must be entered before anything else becomes available, but the settings page is also available at any later time. Bounding box area contents can also be listed separately, e.g. Edinburgh
4

and Penge, so that reviewers who are familiar with more than one area will find it mentally easier to parse through the responses.
The feedback reviewer will see a selection list of types of feedback (e.g. route feedback, photomap, other). Selecting one shows a summary of that kind of feedback. They can choose between two ways of viewing the summary:
1.
A tabular format, paginated. This is available for all types of feedback.
2.
A map format. This is available for the route feedback type only (since other types of feedback are non-geographical in nature).
The reviewer can follow links from each report shown in the list/map and then fill out a response
(see later section below). This results in the response being saved in the database, and its removal from the table/map of non-responded feedbacks.
Items marked as duplicates will be treated in the same way as feedback that has been dealt with, i.e. hidden by default. The item number that it duplicates will be shown.
There will be a control for marking things as form-spam directly and quickly, as per the current system.
For the tabular view, earliest (oldest) non-responded feedback is always shown first, on the earliest pages, to encourage its clearance. A button will be available to make others visible.
The tabular view is paginated appropriately, defaulting to 10 per page but this can be changed on the feedback reviewer's user settings page.
As noted above, feedback which is of a non-geographical type, i.e. everything other than route feedback, is available only in this tabular format, paginated. This tends to involve things like
Photomap bugs, feature requests, business proposals, etc., which are not for public display or of relevance to OSM experts.
The map view is a similar map-and-markers format to the Photomap facility at /photomap/. Shown on this are markers where feedback exists. At the point of each login session, this will default to the bounding box of the area (or first of the areas) being watched. (The bounding-box setting will dominate over the user’s last-saved bounding box from elsewhere in the site.)
Consideration will be given whether to limit feedback to a particular area when in the subdomain mode of the site. This would need to be done in such a way that the code is not tied down to that model, to avoid preventing its use on other sites not run by CycleStreets.
5

For each feedback item, the start and finish markers will be placed on the map of the underlying journey being referenced. Ideally, there would be a clickable line in place – experimentation will be done if time permits.
Clicking on a marker brings up a bubble. (If a bubble is already present it will be closed, i.e. one on the screen at a time; usability testing for the Photomap found this to be preferable than having to close bubbles explicitly.)
The bubble contains the main text of the feedback, resized to show as much of the text as possible.
It shows the journey number, and a link to the feedback handling page which is basically a box added to the main public itinerary page (as at present). However the latter code needs to work the other way round; the box should be followed by a load of the itinerary, with a URL relating to the feedback
ID rather than the itinerary ID, e.g. /feedback/2741/ . In this way, multiple feedbacks can be submitted against one route. Other feedbacks relating to the same journey should be accessible, to improve the context when judging the problem.
Analysis of feedback is a skilled task and can take time. Route problems tend to have one of four main causes:
1.
Small data problems such as misconnected ways or one-way streets in the wrong direction;
2.
Absence of data in an area, meaning that the network is not sufficiently complete to enable a good route solution to be created;
3.
Engine deficiencies such as over-wiggly routes or poor handling of busy roads;
4.
Mis-translation of OSM data to the format used internally by the CycleStreets routing engine.
Each of these can take time to determine.
Subject to implementation time and complexity, at the point that a reviewer enters the feedback item's review page, and for the next 30 minutes, other users will be informed that another reviewer
(giving their name) has recently opened it and thus may be editing it. This does not prevent others doing anything, as it is merely a notification on the HTML page.
The feedback response page is to be structured so that notes can be left for further investigation, so that reviewers can gradually chip away at a problematic case and leave notes as they go, i.e. save a partially-completed feedback.
Each route feedback page should have links to common OSM facilities, such as OpenStreetBugs,
MapDust, other map bases, ITOworld visualisations, etc.
Resolution of a feedback requires both a response to the user (where they have left their e-mail address) and an internal note about the resolution, plus an overall categorisation status for use in future analysis.
6

A traffic-light system will be in use, with the colour shown on the table/map listings and on the feedback page itself. Red indicates a non-dealt-with feedback. Yellow indicates a feedback that has been both opened and had internal notes left against it, i.e. being dealt with. Green indicates a feedback that has been cleared. (Green would not normally be shown unless the reviewer has gone to the version of the table/map which shows these.)
Ideally, there will be a facility to filter out older, stale feedback.
The list at http://www.cyclestreets.net/journey/help/faq/ should be auto-generated.
A specific objective of the new system is that it should not involve e-mail in any way, except for the non-route types and except for once-a-day notifications.
The only e-mail should be a daily reminder to reviewers to log in, giving statistics of unhandled/outstanding feedback and relevant link(s).
The outgoing e-mail from the system to the reporting user should be from an address which, if sent to, can be ingested by the server. A PHP class already exists to enable this to be coded easily, though some mailserver settings need to be changed. However, this uses direct ingesting, rather than mailbox retrieval, and so may be unsuitable. The e-mail should include the name of the reviewer but not their personal/role e-mail address.
The correspondence trail is then shown in a conversation-like fashion showing the incoming/outgoing mails (with dates, names and e-mail addresses shown, with a response box always being the final item in the list enabling a reviewer to send further correspondence.) All this ensures that a whole trail of correspondence is available to other reviewers.
All feedback and response trails are available to other reviewers. They can view / return to feedbacks that have been dealt with at any time, by unhiding them from table/map view and following the link to them.
As noted above, feedback relating to non-route items is handled differently. These are generally bug reports, feature requests, funding opportunities, and so on which are effectively just e-mails for the attention of the developers. (As with the rest of the proposed system, the code will not be tightly coupled with CycleStreets assumptions, instead using settings or API access methods, so that the code is more easily renewable.)
These are sent via e-mail to the developers.
They can be handled using the HTML table method also (though they are not mixed in with route items; instead they are shown separately) and the response handled there, with the reply being stored in the usual way. The developers should get in the habit of copying in the server-ingest mail so that the replies are also captured into the database. Alternatively, they can flag 'transferred to e-
7

mail' which is then shown in the table to other reviewers; the name of the developer/reviewer who has picked it up is shown also.
An API-end to the data should be added if time permits, enabling pushing of post-triaged data to other systems.
In an ideal world, existing e-mail based replies, currently archived to a mailman list, would be incorporated into the new system so that a single database exists.
8