EHC Data Extraction Automation
These notes are to guide someone through the steps of how to setup the automation of their EHC
data extraction process through the use of a SQL job. The ‘EHC Data Extraction’ process runs the EHC
data extract stored procedure, outputs this data to a CSV file, and uploads this export to the chosen
sFTP (UAT/PROD) via the use of the software WinSCP.
Process prerequisites:



SQL Server Database (backend of Authority/Pathway/Property & Rating).
SQL Server Agent to be running on said SQL Server.
EHC data extraction script.
The following steps have been setup and tested on a Windows 7 machine with SQL Server 2012. The
setup should be very similar across previous version of SQL Server (2005/2008).
1. Install sFTP software WinSCP
Software can be installed from the following URL:
http://winscp.net/eng/download.php
Install as per default instructions.
2. Configure sFTP software
Both UAT and PROD connection profiles require creating before they can be referenced in the
Windows PowerShell.


UAT IP address: 150.207.159.200
Prod IP address: 143.119.208.6
The old IP addresses for UAT (143.119.247.54) and Prod (143.119.252.174) have been
decommissioned (March 2015).
Enter the Council specific sFTP details for UAT (150.207.159.200) and click ‘Save’. Ensure you tick the
box to save the password and always trust the connection:
150.207.159.200
Enter the session name as ‘EHCUAT’:
150.207.159.200
Enter the Council specific sFTP details for PROD (143.119.208.6) and click ‘Save’. Ensure you tick the
box to save the password and always trust the connection:
143.119.208.6
Enter the session name as ‘EHCPROD’:
143.119.208.6
Once both profiles have been setup, enter to the console and click the ‘Preferences’ option through
‘Options’:
Navigate to the ‘Storage’ tab, and select the radio button for ‘INI file (WinSCP ini) under
‘Configuration registry’. This ensure that the EHC data extraction transferred by the sFTP software
transfer in the correct format:
The final configuration is setup WinSCP to ensure it can be used in Windows Powershell.
Open ‘System Properties’ and click’ Environment Variables’:
Locate and highlight ‘Path’ under ‘System variables’ and click ‘Edit’:
Add the text ‘;C:\Program Files\WinSCP’ to the end of the ‘Variable value’ string. Click ‘Ok’ to save:
Don’t forget to test both connections, and report to the EHC team if there are any outstanding
issues.
3. Configuration of PowerShell on SQL Server
Open SQL Server and run the following T-SQL on the database that the EHC extract will be running
from. This will allow PowerShell queries to be run within SQL Server.
-- To allow advanced options to be changed.
EXEC sp_configure 'show advanced options', 1
GO
-- To update the currently configured value for advanced options.
RECONFIGURE
GO
-- To enable the feature.
EXEC sp_configure 'xp_cmdshell', 1
GO
-- To update the currently configured value for this feature.
RECONFIGURE
GO
4. Creation of data extract export stored procedure
The first step is to ensure you have a stored procedure setup on the SQL Server to export the EHC
extract in the correct CSV format. Create the ‘sp_Generate_csv_with_columns’ stored procedure
using the following code. This SP outputs the EHC extract in the correct format ready for sending to
the sFTP. Please note the variables passed through to the SP will differ from Council and PIS:
--EXEC sp_Generate_csv_with_columns 'DBNAME',
'TABLENAME','COUNCILNAME','OUTPUT LOCATION'
--EXEC sp_Generate_csv_with_columns 'CurrentGLSUAT',
'##tblEHCDataExtract_Final','BANKSTOWN','D:\'
ALTER PROCEDURE sp_Generate_csv_with_columns
(
@db_name VARCHAR(100),
@table_name VARCHAR(100),
@council_name VARCHAR(100),
@output_location VARCHAR(100)
)
AS
BEGIN
--Generate column names as a recordSET
DECLARE @columns VARCHAR(8000), @sql VARCHAR(8000), @data_file
VARCHAR(100), @file_name VARCHAR(100)
--Set file name and output location
SELECT @file_name = @output_location + 'EHC_' + @council_name + '_' +
CAST(CONVERT(CHAR(8), GETDATE(), 112) AS VARCHAR(8)) + '.csv'
SELECT
@columns = COALESCE(@columns+',','') + column_name + ' AS ' +
column_name
FROM
-- Set to tempdb as sp_Authority_SQL_EHC_Data_Extract generates
a temporary file as the output (##tblEHCDataExtract_Final)
tempdb.INFORMATION_SCHEMA.COLUMNS
WHERE
TABLE_NAME = REPLACE(REPLACE(@table_name,']',''),'[','')
SELECT @columns = ''''''+REPLACE(REPLACE(@columns,' AS ',''''' AS
'),',',',''''')
--Create a dummy file to have actual data
SELECT @data_file = SUBSTRING(@file_name,1,LEN(@file_name) CHARINDEX('\',REVERSE(@file_name))) + '\data_file.xls'
--Generate column names in the passed EXCEL file
SET @sql = 'EXEC ' + @db_name + '..xp_cmdshell ''bcp " SELECT * FROM
(SELECT ' + @columns+') AS t" queryout "' + @file_name + '" -c -t"|" -T S'''
EXEC(@sql)
--Generate data in the dummy file.
SET @sql = 'EXEC ' + @db_name + '..xp_cmdshell ''bcp "SELECT * FROM '
+ @db_name + '..' + @table_name + '" queryout "' + @data_file + '" -c -t"|"
-T -S'''
EXEC(@sql)
--Copy dummy file to passed EXCEL file
SET @sql = 'EXEC ' + @db_name + '..xp_cmdshell ''type ' + @data_file
+ ' >> "' + @file_name + '"'''
EXEC(@sql)
--Delete dummy file
SET @sql= 'EXEC ' + @db_name + '..xp_cmdshell ''del ' + @data_file +
''''
EXEC(@sql)
END
5. Creation of sFTP stored procedure
Copy this SQL into Query analyser and run to create the stored procedure used for the sFTP process.
Notable changes per Council include the variable of the EHC data extract location @WorkDir.
--EXEC sp_SQL_EHC_sFTP 'UAT', 'LIVERPOOLPLAINS', 'authlive'
/*
***************************************************************************
*******************
-- Author:
Iain Gardiner (Wingecarribee Shire Council)
-- Create date: 11/09/2012
-- Description:
This stored procedure is used to FTP the file to the site
that is
-required depending on the environment we are running in
PROD and TEST
-to upload into the EHC system.
Change Log:
-- James Boulton - 09/10/2013: Updated for use across different PIS
systems.
***************************************************************************
********************/
ALTER PROCEDURE sp_SQL_EHC_sFTP
-- Parameter of Environment To Run FTP Procedure
@ENVT CHAR(4), -- Accepted values of 'UAT' or 'PROD'
@CouncilName AS VARCHAR(50), -- Council name
@db_name VARCHAR(100) -- DB name
AS
BEGIN
-- SET NOCOUNT ON added to prevent extra result sets from
-- interfering with SELECT statements.
SET NOCOUNT ON;
-- Declare Variables Required
DECLARE @FileName AS VARCHAR(255)
DECLARE @FTPProfile AS VARCHAR(255)
DECLARE @FTPCommand AS VARCHAR(255)
DECLARE @WorkDIR AS VARCHAR(255)
DECLARE @ExtractDate AS VARCHAR(8)
SET @ExtractDate = CAST(CONVERT(CHAR(8), GETDATE(), 112) AS
VARCHAR(8))
-- Lets See What Environment We Are Running
IF @ENVT = 'UAT'
BEGIN
-- Set The Working Directory when the EHC extract files
will be transferred from
SET @WorkDIR = 'D:\'
-- Set FTP Variables
SET @FTPProfile = 'EHCUAT'
END
ELSE IF @ENVT = 'PROD'
BEGIN
-- Set The Working Directory when the EHC extract files
will be transferred from
SET @WorkDIR = 'D:\'
-- Set FTP Variables
SET @FTPProfile = 'EHCPROD'
END
-- Set The File Name Required To FTP Up As
SET @FileName = 'EHC_' + @CouncilName + '_' + @ExtractDate + '.csv'
-- Set The FTP Command
SET @FTPCommand = 'C:\Progra~2\WinSCP\winscp.com /command '+
'"option confirm off" "open "' + @FTPProfile + '""' + ' "put ' + @WorkDIR +
@FileName + ' ' + @FileName + '" "exit"'
-- Execute The Command. Change name to DB name
EXEC authlive..xp_cmdshell @FTPCommand
SET NOCOUNT OFF;
END
6. Creation of SQL Job
Check to see if the ‘SQL Server Agent’ is running. If not, click ‘Start’.
Right click ‘Jobs’ and select ‘New Job...’.
Enter ‘EHC_Data_Extraction’ as name.
Select ‘Steps’ from the left hand menu, and click ‘New...’.
Enter ‘Step name’ as ‘EHC Data Extract’ and enter the following T-SQL to run all 3 of the required
stored procedures for the EHC data extraction automation process:
EXEC sp_Authority_SQL_EHC_Data_Extract
EXEC sp_Generate_csv_with_columns 'CurrentGLSUAT', '##tblEHCDataExtract_Final','COUNCIL','D:\'
EXEC sp_SQL_EHC_sFTP 'UAT', 'COUNCIL', 'authlive'
This is the only step you will need to run in your SQL job. Stored procedures are all run within the
same step to ensure the temporary table created in ‘sp_Authority_SQL_EHC_Data_Extract’ can be
referenced in the subsequent SP’s.
Click ‘Schedules’ from the left hand side bar.
Set up schedule to occur every weekday outside of office hours. Set schedule without an end date,
and at a time after close of business eg. between 18:00 and 05:00.
In the SQL Management Studio, right click the ‘Operators’ folder and select ‘New Operator...’.
Enter details of operator. This email will be used as a notification if the SQL job fails.
Back in the SQL Job, select ‘Notifications’ from the left hand menu, tick the box for Email, select the
recently created operator, and ensure ‘When the job fails’ is selected in the right hand dropdown.
Click ‘OK’ to complete the creation of the SQL job. To test job, right click job and click ‘Start Job at
Step...’ and choose step 1 and run.