Data Collection Process
Guide
ScotXed Unit
Contents
Purpose
2
Roles
2
Data Collection Process Maps
3
High Level
3
Stage 1- Data Requirements & Stage 2 – Development (New Collection)
4
Stage 1- Data Requirements & Stage 2 – Modification (Existing Collection)
5
Stage 3 - Configuration
6
Stage 4 – Data Management
7
Stage 5 & 6 – Data Release and Review
8
Stage 1 – Data Requirements
9
Stage 2 - New Collection – Development
10
Stage 2 - Existing Collection – Modification
10
Stage 3 – Review
11
Stage 3 – Configuration
12
Stage 3 - Generation
13
Stage 4 – Data Management
15
Stages 4 & 5 – Data Release
16
Stage 6 – Review
17
Technical Support
18
The User Experience
19
Pre-survey Meeting Template
20
1|Page
Purpose
To document the data collection process for ScotXed projects. Information is provided at both a high level via the process map diagrams or at a more detailed level via
desk instructions. In order to obtain more information from the process maps – click on the darker blue icons to go to a lower level of detail for that particular process.
This guide cannot cover all eventualities, if you have any questions at any stage of the process please speak with an experienced colleague within ScotXed.
Process Maps key:
Process
Document
Decision
Roles
Roles defined within the process maps as follows:
 Customer - regarded as the owner of the collection. Usually (but not always) the statistician responsible for the publication of the data.

Project Officer/Manager – ScotXed representative who is the person responsible for the implementation of the collection. The project officer/manager may make
use of support office staff to manage particular tasks within the project.

Technical Support – Likely to be a representative from ScotXed’s Professional Services Team (PST). However, in certain circumstances PST may recommend
you get technical support from Lockheed Martin developers. Should this be the case, they would be responsible for the development of the plug-in and possibly
the xml Excel template and data transform file. PST would remain to have responsibility for providing DbXed data extracts/data releases.
2|Page
ProcXed.NET
Data Collection Process Maps
High Level
Yes
Stage 1
New Data
Requirements
Stage 2
Develop
new plug-in
New to
ProcXed.NET?
No
Stage 3
Configuration
Stage 1
Review current Data
Requirements
Stage 4
Data
Management
Stage 5
Data Release
Stage 6
Review
Stage 2
Modify
3|Page
ProcXed.NET
New Data Collection Process
Stage 1- Data Requirements & Stage 2 – Development Map
Generate:
-Data Spec
-G.Notes
Pre-survey Meeting
Tech Support
Project
Manager
Customer
ScotXed Data Collection Process – New ProcXed.NET Collection
Agree:
-Requirements
-Roles &
Responsibilities
-Milestones/Dates
-Validation
-Reports
-Organisations
-Users
-Pre-pop?
-Data Release Dates
-Wash Up Meeting
Outputs
Sign-Off
Provide:
-Test Data
-Pre-pop file
(if required)
Sign-Off
Yes
Test Module
(inc pre-pop if
required and
Reports!)
Generate:
-Sample Form
Layout
Create:
Project Plan
Match
Requirements?
Yes
No
Okay with layout?
STAGE 1 – DATA REQUIREMENTS
Develop:
--Module
Module
-Template
-Template
-Transform
-Transform
No
STAGE 2 – DEVELOPMENT
Back to High level Plan
4|Page
ProcXed.NET
Existing Data Collection Process
Stage 1- Data Requirements & Stage 2 – Modification Map
ScotXed Data Collection Process – Existing Collection
Pre-survey Meeting
Tech Support
Project
Manager
Customer
Revise:
-Data Spec
-G.Notes
-Template
-Form Layout
Agree:
-Requirements
-Roles &
Responsibilities
-Milestones/Dates
-Validation
-Reports
-Organisations
-Users
-Pre-pop?
-Data Release
Dates
-Wash Up Meeting
Outputs
Sign-Off
Sign-Off
Yes
Update:
Project Plan
Change required to
existing module?
Yes
No
Re-Test Existing
Module
(Inc Reports!)
Match
Requirements?
Modify:
Modify:
-Module
-Module
-Transform
-Templates
-Templates
No
STAGE 1 – DATA REQUIREMENTS
STAGE 2 – MODIFICATION
Back to High level Plan
5|Page
ProcXed.NET
New & Existing Data Collection Process
Stage 3 - Configuration
Project
Manager
Customer
ScotXed Data Collection Process – Stage 3 Configuration
Stage 2 Outputs:
-All Supporting
Documentation
-XSLT File (Transform)
-Schema
-Template (If required)
Review:
Review:
-Users
-Organisations
-Users
-Organisations
Configure:
-Live Data Collection
Generate:
Generate:
Data
Collection
-Data Collection
Add:Add:
-Users
-Users
-Organisations
-Organisations
Is pre-pop
required?
No
Review:
-Configuration
(timescales,
organisations etc)
Tech Support
Yes
Upload:
-Transform
Upload:
-Data for Pre-pop
Back to High level Plan
6|Page
ProcXed.NET
New & Existing Data Collection Process
Stage 4 – Data Management
Project Manager
Customer
ScotXed Data Collection Process – Stage 4 – Data Management
Analyse:
-Data Released
Comms:
Comms:
-1st Reminder
-2nd Reminder
Yes
Publish:
-All Supporting
Documentation
(incl Data Specs, GNotes, Templates,
Schema,
Manage:
Manage:
Comms:
-E-mail users
-Support
-Support
-Data
-DataSubmissions
Subs
-Status Reports
-Status
Reports
-Survey
Log
-Survey Log
Manage:
Approve:
Validation
Correct?
-Data
Submission
-Support
-Data Submissions
-Status Reports
-Survey Log
2nd Stage
Validation Correct?
Approve:
Yes
-Data
Submission
Yes
No
No
Comms:
-Contact User
Comms:
-Contact User
Tech Support
Review:
-No of
Submissions
Approved
Develop:
Release:
-DbXed Export
-Data in DbXed
Back to High level Plan
7|Page
ProcXed.NET
New & Existing Data Collection Process
Stage 5 & 6 – Data Release and Review
Customer
ScotXed Data Collection Process – Stages 5&6 – Data Release/Analysis & Review
Analyse:
-Data Released
Data Quality Fit for
Publish?
Yes
Sign Off:
-Data
Comms:
-Contact User
Arrange:
-Wash-up Meeting
Review:
-Lessons learned
-Issues Log
-Pre-survey
meeting date
Circulate
Agree?
Project Manager
No
Update:
-Project Plan
(roll on one
year)
Cross Check:
Tech Support
-No of approved
organisations
Release:
-Final data to
DbXed
STAGE 5 – Data Release & Analysis
STAGE 6 – Review
Back to High level Plan
8|Page
Stage 1 – Data Requirements
Event - Pre-Survey
Time: Minimum one hour duration
Attendees present: Customer(s),ScotXed Project Officer, Technical Support (PST
Representative), Administration Support if required
Outputs: Agreed roles and responsibilities, updated data spec and guidance
The process for capturing requirements should be the same whether this is a new or
existing collection.
The pre-survey meeting is used to elicit and document the customers’ requirements,
and should be arranged up to 4 months in advance of a suspected go-live date
(longer if you know significant changes are to be made). Suggested agenda template
to use. The key points to cover are:









Review issues from previous year (performance, validation, improvements)
Are there any changes required this year (questions, validation, reports)? If
so, you have to decide who will update the Data Specification and Guidance
Note (usually the customer).
Improvements - discuss previous performance to identify any improvements
Pre-population of data – is there a requirement? If so, does a spreadsheet go
to users or data uploaded to ProcXed?
What input method is being used – XML, spreadsheet, keyed data?
Summary Tables – Are Summary Tables to be included? Did they work fine?
Will proposed changes have impact on existing tables?
Agree dates – publication, survey live/end, reporting period
Organisations and Users – are they new to the system and/or are there any
changes? (see stage 3 for further details)
Responsibilities - agree who and what will be carried out throughout collection
cycle, remember to include timescales. It is important that this is not
overlooked. This should be documented as part of the meeting write up and
reflected within the BrightWork Project Plan.
Further information on validation can be found here
The customer is responsible for the Data Specification and Guidance Note. Both
documents and the sample form should be signed off by the customer in a suitable
format ready for publication. ScotXed can provide recommended templates for
consistency.
9|Page
Specifics about a new data collection
Eliciting the requirements of a new collection can be complicated. It is important that
at least two team members from ScotXed are at the initial meetings, and that at least
one of these is experienced in creating new collections. Ensure that you are fully
prepared to run through all requirements with the customer using the suggested
template.
Back to Stage 1 detailed plan
Stage 2 - New Collection – Development
The data collection module/plug-in will either be developed by Lockheed Martin or inhouse by PST.
Lockheed Martin Development
If Lockheed Martin is carrying out this work, there will be the need to provide the
Data Specification to them. Please remember to include the Summary
Tables/Reports required for the collection. This work is carried out jointly by the
Project Manager and customer.
PST Development
Provide PST with as much supporting documentation as possible. This would
include:
 Draft Data Specification – this can initially be in the basic form of a list with
field formats and validation checks. The customer will help provide this
information.
 Examples of form layout – how you envisage the form to look when users log
in (you can use existing collection forms as a starting point)
 Examples of Summary Tables/Reports to be included (you can use existing
collection tables/reports as a starting point)
 Confirm the requirement for an Excel template and corresponding xslt file
(transform) if appropriate
Stage 2 - Existing Collection – Modification
For existing data collections which already have in place a ProcXed.NET module
and changes are required. Provide PST/Lockheed Martin with as much supporting
documentation as possible:
 List of changes from last year to this year
 Draft Data Specification
 Examples of how form layout would change
 Examples of new Summary Tables/Reports to be included
 Confirm the requirement remains for an Excel template and corresponding
xslt file (transform). It is likely that a new template will be required if the
module is changing.
10 | P a g e
Testing the Development
Once the module has been completed, there is the need to fully test what has been
developed to ensure it matches the requirements from the Data Specification and the
other information provided at the time. It is important to generate a test plan and
have dummy data to hand to replicate a “live” situation as best as possible.
Remember to test form layout, uploads, data validation and reports (if reports are
required). The outcome from each test case should be recorded within the test plan.
Please ensure that you record anomalies in a way that allows the developer to easily
reference the issue being queried.
It is important to ensure that the customer is involved with testing where possible. It
is recommended that the customer signs off the development to complete work on
Stage 2.
Back to Stage 2 detailed plan
Stage 3 – Review
Pre-population data
At this stage you should request pre-pop data from the customer if appropriate. It is
important to get this data in advance to allow time to upload and/or reformat as
necessary. Please discuss pre-pop with PST if you are unsure.
Review Organisations and Users
For all collections, a task should be included in BrightWork for admin support to email all users at least two weeks before go-live date to ensure they are the correct
contact for the survey. Data Collection contacts can be found on the contact spread
sheet <link> which is updated by ScotXed admin support.
Establish if the organisations to be assigned to the collection already exist within
ProcXed.NET. To do this:
 Menu
 Organisations - Manage Organisations
 Check the “advanced search” box
 Enter organisation name (you can just enter part of the name)
 If listed then there is no need to insert again. If not, please refer to this
instruction for further advice.
11 | P a g e
Similar to organisations, there is a need to establish if the users already exist in
ProcXed.NET.
To do this:
 Menu
 Security
 Manage Users
 Check the “advanced search” box
Stage 3 – Configuration
Very Important! Always perform in test environment first.
Adding New Organisations
Once you have established that a new organisation is required. Please refer to the
organisational hierarchy structure [insert link] to review where the new organisation
would be located within the hierarchy. If you are unsure, please contact the ScotXed
IT Support desk accordingly.
To add the new organisations, follow these steps:
 Menu
o Manage Organisations (within Organisations)
 You can enter them in individually but it is recommended to
complete the spread sheet template [insert link to template] for
uploading a batch of organisations. Note – you may need to get
support from the ScotXed IT Support Desk again!
o Batch Upload Organisations
 Browse to your completed template, click on upload
“organisations”.
 You should get a message displaying a successful upload or indicating
something that needs attention.
12 | P a g e
Adding New Users
If new user accounts are required:
 Menu
 Security
o Manage Users
 Click on “advanced search”
 enter the relevant organisation you want the user aligned to
 Enter details
 Like organisations, it is recommended to complete a template
for a batch upload of user account information.
o Batch Upload Users
o Browse to your completed template and check the appropriate boxes
for your choice of user account set-up.
Note that unless specified otherwise, ProcXed.Net will issue an e-mail to the user
when the account is created. Please consider this.
Stage 3 - Generation
For new and existing collections and in Test or Live
Set up a collection in ProcXed.NET



Menu – Data Collection – Manage Data Collection
Choose relevant collection
Add new collection
 Start date – is the “go live” date. Note that to prepopulate with data, the
start date must be in the future
 End date – when you would like the organisation to have submitted
their data e.g. 6 weeks from ‘start date’
 Real end date – allowance for organisations submitting their data late
e.g. 10 weeks from ‘start date’. Collection will no longer be viewable in
ProcXed by Organisations after this date
 Remind yourself in advance of the real end date just in case this needs
to be changed. If the real end date passes, PST will need to amend the
collection dates.
 Period start date – reporting period start date
 Period end date – reporting period end date
 End date reminder period (days) – this is used when you want to send
out automated reminders. Good for large collections such as Pre-school
Census. Smaller collections, leave this blank and contact organisations
directly
 Real date reminder period (days) – as above - used for sending out
automated reminders
13 | P a g e
Add organisations to the collection





Menu – Data Collection – Manage Collection Organisations
Choose relevant collection
Use the “Name Filter” or the advanced search option to search for
organisations. For example – if it was local authorities being used for a
collection:
o For collections under education, type “education - ” in “Name Filter”
o For collections under social work, type “social work - ” in “Name Filter”
When an organisation is selected it moves to right of screen. Check that all
your organisations are listed on the right.
Save
Add transform file/s (if required) for the collection






Obtain any transform files from the Project Manager or PST
Menu – Data Collection – Manage upload data transformers
Select the Data Collection Type using the drop down
Add new upload data transformer
Name the transform – E.g. ‘<Collection Name> Excel to XML’ and browse to
where you saved the transform file
Save upload data transformer
The collection is now set up
QA check
Check to make sure that all expected organisations are included and that their status
is at “initialised” and the upload transformer is there
 Select your new collection within Data Returns
 select one of the organisations
 click upload and check that the transform is there. Then click “back”.
 ensure that the status has remained at “initialised”
Prepopulate the collection (if required)

PST will already know that pre-pop data is required from pre-survey meeting
requirements
 Data must be in “xml code” format which means you are likely going to require
support from PST to complete the file
Once pre-population file has been created
 Menu – Data Collection – Upload Prepopulation Data
 Select collection
 Upload xml file
 Menu – Data Collection - Manage Data Returns. View a return for an
organisation to check that it has been prepopulated
Back to Stage 3 Detailed Plan
14 | P a g e
Stage 4 – Data Management
Managing the collection when live
Review the status of each of the organisations expected to submit
 Menu
 Data Returns
 Manage Data Returns
 All organisations will be listed within a table. Please monitor their status
regularly.
Data Transform
It is likely that the data will need to be transformed before being uploaded into
ProcXed.Net (The user will need to check the box transform data first box) The
transform is carried out automatically by the system. If a user is having difficulties
you should check the status of the transform:
 Menu
 Data Returns
 Manage Upload Data Transform Jobs
If anything is pending for more than 20 minutes, speak to ScotXed IT Support as
it
might need to be pushed through. An upload that is pending for a long time will
prevent other users from uploading (for ANY collection, not just your own). If the
upload has failed, check the customer is using the correct upload template.
Validation when a return has been submitted
There are two types of validation, 1st stage and 2nd stage. 1st stage validation is the
responsibility of the project officer, the purpose is to ensure any validation errors are
fixed or commented by the organisation responsible for the data (eg local
authorities). The project officer must liaise with the organisations until all validation
errors are clear/commented. It is best practice to check for validation errors as soon
as a data return has been submitted.
1 st stage validation
There are two ways to check for validation errors. Within the form (menu – data
returns – manage data return – collection type – select and edit) there is a drop
down menu at the top of the form that will take you directly to validation errors. Use
the next/previous buttons to go between errors:
The second option is to use the specific validation area within ProcXed: Menu - data
returns – validation errors and select the collection. You can use advance search to
only look at one organisation if preferred. Once you have selected an error you can
‘edit’ it directly on the form on comment and clear the error.
The organisation is responsible for fixing or commenting errors.
2nd stage validation occurs later in the process
15 | P a g e
Reports
This allows users to download their data from ProcXed to carry out their own
summary checks before submitting. A user will only see their own organisation’s
data. We see any submitted data.
 Go to menu-reports-view reports
 Select the collection– summary
 The reports can be viewed in html or exported in other formats including
Word, Excel or PDF.
Back to Stage 4 Detailed Plan
Stages 4 & 5 – Data Release
When data has been submitted, there is a need to make this available for further
analysis. These are the suggested procedures to follow:
 Quality Assure the data submitted. Do this by running the validation reports
and ensuing that the data is of good quality
 Review and list the organisations that has the submitted status
 Send data release request to the ScotXed IT Support mailbox enclosing the
list of expected organisations
 Ask
o when the data will likely be made available
o to cross-check against list provided (there should always be a match)
 In addition to the data, ScotXed IT Support will provide a list of any issues that
are present with the data
 Notify customer of data release.
2 nd stage validation
The customer will analyse the data and may have questions that need to go back to
the organisation (for example significant changes from last year’s submission). It is
the responsibility of the project officer to co-ordinate the questions and get answers
from the organisations.
The questions and answers can be recorded on a spreadsheet for ease of access,
however this must be stored within the appropriate file in eRDM for future reference.
Where possible, the answers should also be reflected within the data collection,
either as a comment or by updating specific data fields.
16 | P a g e
Stage 6 – Review
It is important to review the collection that has just passed allowing enough time for
all the data to be submitted, quality checked and published. This means that the full
process can be reviewed.
Wash-up Meeting





Discuss and agree what went well and what can be improved for next
collection period
Discuss and agree any issues that require attention
Document these by means of a Lessons Learned Log
Circulate this log to all involved to agree content and potential action required
– save in eRDM
Update BrightWork should any tasks require changing and roll dates forward
to next anticipated timescales.
17 | P a g e
Technical Support
It is important that a member of the Professional Services Team (PST) is present at
the Pre-survey meeting. This ensures that should there be any discussion in relation
to the existing way the collection is being managed or should there be a required
change or this is a new collection - a technical perspective can be provided at the
outset and fully documented accordingly.
The requirements gathered at the Pre-survey meeting should be circulated to all
present and signed off by the customer. Specific tasks and timescales should be
made clear to the PST representative and subsequently agreed. It might not be the
case that PST are able to “assign” someone in particular to this collection, if this is
the case all tasks in BrightWork should be assigned to the PST team leader who will
then allocate accordingly.
It is recommended that any changes are updated within the Data Specification and
Guidance Note using tracked changes. Ideally, the customer/owner of the collection
would update these documents. However, it may be prudent to update the
documents yourself and ensure that the customer signs off the changes. In addition
to the updated Data Spec and Guidance Notes, it is recommended to generate a
one-page document to list the changes from the previous year’s module. Examples
of possible changes:



Year references to be amended from previous to this year’s reporting period
A variable is to be changed to mandatory
The order of variables on the form is to be changed to help flow
A new variable is to be added to the collection
Please consider - whenever a change has been implemented, there may be the
need to update the reports (aka Summary Tables) accordingly.
Technical Support will be required at the following points in the data collection
process:







Pre-survey Meeting – Stage 1
Development of new or modification of existing module – Stage 2
Creation of Spread sheet template and data transform file (xslt) – Stage 2
Pre-pop upload (if required) – Stage 3
Creation of DbXed extract/export file (this allows data to be released to
DbXed for customer to analyse) – Stage 4
Reviewing of data submitted to date – Stage 4
Data releases – Stages 4 & 5
Lessons Learned – Stage 6
All the above must be included within the project plan to help PST plan and coordinate appropriate support throughout all collections.
18 | P a g e
The User Experience
What they see when they log in:
The user is taken straight to the Data Return Components (i.e. the data collections)
that they are expected to submit data on. The user should be encouraged to click on
“Overview” to open up more detailed tables on the collection and provides them with
more actions to carry out such as enter or upload.
19 | P a g e
Pre-survey Meeting Template
Suggested template to document ALL Data Collection Requirements (new or
changes to an existing ProcXed.NET collection)
1. Data Requirements
a. Data Variables
b. Validation – Both 1st and 2nd Stage
c. Pre-population of data required?
2. Milestones/Timescales
3. Input Method (spreadsheet/XML/data entry)
4. Summary Tables/Reports
5. Organisations
6. Users
7. Responsibilities
Back to Pre Survey Meeting detail
Go to new Collection Stage 1 Map
Go to existing Collection Stage 1 Map
20 | P a g e