Getting started with the Data Exchange Agent
The Data Exchange Agent (DEA) is an open source tool that accommodates periodic upload of details
from a local database to a remote server. Designed to be installed local to a data-source, the DEA
can be configured to automatically extract data from the data-source and upload it as XML to an
HTTP resource such as a REST web-service. The advantage of the DEA is that the process of
extracting and uploading data can be set to run automatically on a timed schedule and using
qualified queries so that once the bulk of data has been uploaded, updates and changes to individual
records can be selected and uploaded automatically to keep remote records up to date. The DEA can
load details from XML, CSV, ODBC and JDBC based data sources.
Running the Data Exchange Agent
The DEA is a Java servlet-base web-application and so must be installed on a system with both a Java
run-time environment and a servlet container, such as Apache Tomcat, installed. When choosing
where to install the DEA it is important to consider the access the tool will need, locally to reach the
data-source and outward access to the server that will receive the data.
The Java run-time environment is available from http://java.com/en/download/index.jsp
Apache Tomcat is available from http://tomcat.apache.org/
Receiving data from the DEA
The DEA formats data into an XML document and posts that XML to an HTTP resource as the value of
a parameter ‘xml’. Servers receiving uploads from the DEA are expected to read the posted contents
of the ‘xml’ parameter and process them accordingly. To manage connection times, if more than
500 records of data are to be sent, the DEA will break the upload into batches of up to 500 records
and send each batch in sequence. Batches are marked with the identifier of the originating upload
category and the receiving server is expected to treat all batches with the same id as part of the
same upload if they are received within 15 minutes of the first batch or until a batch with the
SubmissionComplete=”true” flag set. Therefore, the receiving server should wait until all batches of
a particular upload have been received before processing the uploaded data.
Below is an example of the XML format in which a SotD inspection would be uploaded
<?xml version="1.0" encoding="UTF-8"?>
<fsn_inspections destination="3144" ext_id="01224" password="jHH45t4rD">
<operatorproprietor>Mr. P Smith</operatorproprietor>
<establishmentaddress>183 High Street</establishmentaddress>
<establishmentphoneno>01975 334 921</establishmentphoneno>
<comments>No comments made</comments>
In uploads containing multiple records, the ‘fsn_inspection’ element is repeated for each record.
The attributes of the root ‘fsn_inspections’ element identify the Local Authority by id in the
‘destination’ attribute and authenticate the source of the upload as valid by providing an ‘ext_id’
and ‘password’ pair.
Installing the DEA web-app
To deploy the DEA web-application in to a servlet container, copy ‘dea.war’ in to the web-app
directory e.g. ‘/apache-tomcat-6.0/webapps/’. When the servlet container is started, the .war file
will be automatically deployed into a ‘dea’ subfolder. Copy the file web.xml in to the newly created
‘dea/WEB-INF’ directory and the DEA should be ready for use.
To access the Data Exchange Agent in SotD mode, navigate to the following URL in a web-browser
This should present the DEA welcome page.
Configuring the DEA
The Data Exchange Agent is used to extract data from a local data-source and upload that data to a
remote HTTP resource, any number of such uploads can be configured and are referred to as upload
categories. Setting up the DEA for use is split in to two separate sections, general options and
defaults are configured on the ‘Destination’ page under the ‘General settings’ section of the menu.
Individual uploads of SotD inspection records are configured via the Food safety inspections page.
On the ‘Destination’ page the following details can be configured
Destination resource: The HTTP resource to which data will be posted during an upload. It is
possible to configure a destination resource in the settings of an individual upload so that each
upload can be sent to a different endpoint, but any uploads that do not have a destination set will
use this default endpoint. The URL entered here will have upload data posted to it as a parameter
Access key: The access key is sent with each XML upload to identify this DEA installation as the
source of the data. It is possible to configure an access key in the settings of an individual upload but
any that do not have an access key set will use this default.
Secret key: The secret key is sent with each XML upload to authenticate the upload as valid. It
should be part of a matched pair, along with the access key, known by the server so that only
authenticated uploads are accepted. It is possible to configure a secret key in the setting of an
individual upload but any that do not have a secret key set will use this default.
Notify email: If a notify email is set, and is supported by the receiving server, each upload sent will
be acknowledged with an email message sent to this address.
Proxy server, port, user, and password: If the DEA must send uploads via a proxy to allow access to
the receiving server then details of that proxy should be configured here. The machine name or IP
address in the ‘server’ field, the port number in the ‘port’ field and, if required by the proxy, a
username and password to authenticate the proxy request.
Configuring an upload
Each upload category can connect to a different data-source to extract data from and can send that
data to a different receiving server. Details of the data-source to connect to, the data to extract and
the destination server to upload to are configured via a wizard interface. To create your first ‘Food
safety inspection’ upload, navigate to ‘Food safety inspections’ on the left hand menu and the click
the ‘Click here to add new category’ link. This will launch the category wizard.
On the first page of the wizard basic details for the upload are set
ID: The ID field should be set to the Local Authority id of the authority providing the data that will be
loaded. It is used to identify all uploads sent through this category.
Display name: The display name of the category is not included with the data when uploaded; it is
used simply to allow users of the DEA to easily distinguish between different upload categories that
have been set-up on an installation of the tool.
Datasource type: The DEA can extract data from several types of data-source; this list shows the
available connection types that can be used. The data-source connected here will influence how a
connection is made to extract data (e.g. via JDBC connection to RDBMS or direct file access of XML
or CSV files) and will also alter options later in the wizard according to the type of data-source being
Destination resource, access key, secret key: As described above, each upload category can have
these options configured so that uploads can be sent to different receiving servers and using
different key-pairs from the same DEA installation. If these values are not set, the default settings
from the ‘Destination’ page are used.
The second page of the wizard is used to configure the data elements that will be picked up from the
configured data-source and included in the upload. The fields shown on the page are those that will
be included in the XML uploaded, the page is used to configure where in the data-source data for
each field should be pulled. For RDBMS based data-sources, the details to be configured are the
name of the database table and columns from which each data element will be pulled, along with a
SQL-style ‘where’ clause to restrict the selection of records. For a CSV data-source, the details to be
set are simply the names of data columns in the file from which each data element will be pulled.
For an XML-based data-source, the details to be configured are the elements in the XML feed from
which contents will be retrieved, specified in X-Path format.
The third page of the wizard is used to configure more specific details of the data-source to be used.
For JDBC based connections that would mean inputting the database connect string, username and
password used to access the database. For CSV files, simply the location of the file in the file system.
For XML based data-sources, the file location is specified along with the name of a base-element
from which all the X-Path queries configured on page two are rooted (see the file XPath tips for an
example of how XML data-sources should be queried).
This page also provides settings to schedule the upload category to be run automatically. The timer
options allow you to set the time at which the upload will first be triggered and then the interval
between subsequent uploads of the same category.
Once the category wizard is submitted and saved, you will be redirected back to the page listing
existing Food safety inspection uploads, and your new upload category should be visible in the list.
For each upload configured, this page gives access to four options. The ‘edit’ link will re-launch the
category wizard to allow settings to be altered. The ‘remove’ link will delete the configured category
and remove it from the upload schedule. For testing purposes, the ‘test’ link will run the configured
category against the data-source but will show the extracted data on screen rather than sending it to
the destination server. The ‘send’ link will immediately run the upload, extracting data and sending
it to the receiving server.
It is recommended that each newly created upload category is tested using the ‘test’ link to check
that the data-source has been configured correctly and that data is being extracted in to the right
upload fields etc. Testing the upload will also highlight any problems with the data found such as
exceeding expected length and format.