Get documents about "Data Integration Streamlined
Document Sample
Data Integration Streamlined
C A S E S T U D Y ’ S F E B U A R Y 2 0 0 9
Introduction
One of the greatest challenges facing many data personnel is the integration of data from various
sources including applications that source data on a “multi-dimensional” level. Management has
the decision of choosing the correct package to suit budget requirements but also achieve the busi-
ness goals. Technical capabilities also need to be taken into account. The product chosen needs to
comply with standards, allow for data profiling and data quality management and sometimes even
more importantly, the time that it takes to deploy the integration product and the return on invest-
ments.
In this portfolio of case studies we will highlight to you the various methods that were used during
the projects and the challenges that are faced by the businesses. We introduce a case study that
immediately showed how i-Con can produce results when integrated into an environment and im-
prove the workflow process with a drastic improve-
ment on efficiencies. We then move onto two stud-
ies that show how i-Con assisted in the once-off
migration of data, using various techniques for en-
suring data quality, profiling, governance and com-
pliance.
Our first case study gained momentum when the business needed to converge all fuel card data
into a centralised and manageable system. As organisations grow, many disjoint functions need to
be incorporated and reporting on such systems becomes vital for resource administration.
The second study focuses on the migration of data from a multi-valued UniVerse database into an
Oracle database. An enormous amount of effort went into understanding the data, the logic around
the loaders as well as the translation code for the conversions.
In our last discussion we will highlight the technical requirements needed when migrating policy
information using the ACCORD XMLife standard.
Please note that due to privacy, we cannot disclose the names of the organisations.
INTELLIGENT INTEGRATION Convergence of information with centralised and managed
systems
i-Con provides the developer the ability to map data.
Leverage existing IT infrastructure
i-Con has helped many business reduce the amount of
code that is written Cost effective
Monitoring of interfaces Multi-dimensional database integration
Data Integration Streamlined
C A S E S T U D Y ’ S F E B U A R Y 2 0 0 9
Text files to UniVerse - Fuel card uploads
The financing company, as with most organisations needing to collate and converge data, required
the expertise to reduce the amount of time spent on importing data and populating a database. Not
only did i-Con reduce a working day spent on importing data to a few minutes, but i-Con also auto-
mated the process. (The figure in the middle of this page depicts the typical interface used for this project)
Flat files being received from an FTP server needed to be verified and imported into the UniVerse
database. If there was an error upon receiving the file from the FTP server, the administrator would
be notified and actively address the problem. The verification of the flat file determined the event
and process that followed. The organisation needed to know whether files imported were dupli-
cated and if so a notification of the
duplication would be sent. If no du-
plications occurred, files would be
uploaded into the UV database and
marked for processing.
A second action was performed to
process the new data and move files
into specific folders depending on
whether an error occurred or if they
needed to be archived.
i-Con assisted in providing the nec-
essary components to perform spe-
cific actions and respond to prede-
fined events. Instead of writing code
to move files between directories,
verifying outcomes and perform the
necessary check that the business
required, i-Con was able to perform
all of this and more. i-Con also took
the files that needed to be archived,
compressed them and sent them to
the archive server via FTP to complete the process. Again the administrator was notified of any
errors that occurred.
“i-Con assisted in providing the necessary
The organisation acquired the confi- components to perform specific actions
dence in their systems to handle the and respond to predefined events. “
new influx of data as well as future in-
creases. This meant the convergence of data to a central service point enabling effective reporting
and effective provision of services to their clients.
INTELLIGENT INTEGRATION Efficient logging of information
Interface designed on palette No writing of code
To concurrently run processes efficiently, the organization Scalable and flexible
required the resources of their existing IT environment.
Email/SMS notifications
Process automation
Data Integration Streamlined
C A S E S T U D Y ’ S F E B U A R Y 2 0 0 9
UniVerse to Oracle - Multi-value migration
In this second study, we will focus on the once-off migration of data from a multi-valued UniVerse
database into an Oracle database. The objective was to provide a migration plan to the Fleet man-
agement company who required a smooth transition of their legacy data into a new format and onto
a different database in order to facilitate the growth of the business.
The solution design included the i-Con setup, a high level analysis of all the applications involved
and an architect solution. The solution build was comprised of:
Analysing the defined source and destination application
data requirements
Defining the extraction and migration methods
Creating the migration process from the templates to
populate the migration database.
Defining data validation and tolerance
Configuration of i-Con according to requirements
Building the extraction process from the migration data-
base to produce required application input requirements
Develop reports to validate process
Iterations of uploading the legacy data
The diagram above depicts the typical migration process that was used during the migration of
transactional data from the primary source through to the data’s final Oracle database. The data
was obtained from a variety of sources and applications that used UniVerse as the database.
The migration database contained tables for each of the applications that will be updated in the
new application that used Oracle. To assist with the auditing of data before it was uploaded in the
new system, the data was cleansed and mapped to each of the tables in the migration DB.
The data migration was finalised with
“i-Con was able to map all data requirements solution testing incorporating data
to improve overall translation of data.“ analysis and the modification of the
solution based on the analysis of the
data. Various Risk Management techniques were used as well as progress reports on issues and
the current status of each phase of the migration.
The Fleet management company achieved all goals from budgeting to time to complete as well as
covering all risk migration issues.
INTELLIGENT INTEGRATION Data validations
Easy to use developer interface similar to Visual Studio Speed of deployment
Efficient mapping of data UniVerse and Oracle mulit-value integration
Data mapping and cleansing
Data Integration Streamlined
C A S E S T U D Y ’ S F E B U A R Y 2 0 0 9
XML to UniVerse - Life Insurance
The approach to the project was to capitalise to the maximum the knowledge of the Policy Manage-
ment System environment from previous experience and technology expertise. Added to this, the
architecture of the solution was such that the interface was applicable to the movement of data
from any Life Insurance Policy Management system to the new database. The interface will be well
documented to facilitate a smooth hand-over to the designated capability within the Life Insurance
organisation.
The scope of the migration included data mapping, data schemes and interface requirements for
client, policy, benefit, payment, loan and cash withdrawals. Being in the Life Insurance Industry,
Accord XMLife was the standard for the message formatting to import data into the target Policy
Management System database.
To ensure the completeness of the migrated data, hash totals were kept on each of the systems.
As records are processed, hash totals were accumulated and inserted with each of the batch files.
The client kept the totals before and after the data was processed. If the totals did not correspond,
the problem was investigated and a re-run was scheduled with exceptions reconciled and docu-
mented.
i-Con was used as the tool to easily map the data in an XMLife XML document into the final Policy
Management system database. i-Con could either poll a directory on the system of an XML docu-
ment, or the XML document can be delivered to the system via a socket call. The validity of the
document was checked and an error, via email or text file, was returned if the file was not valid.
The data from the document was then translated into a valid format so that it may be imported into
the new database. The XMLife standard requires that certain codes are generated for certain
nodes in the XML document.
These codes were replaced “i-Con was integrated seamlessly into our migra-
with codes utilized in the new tion strategy and achieved our goal of smoothly
system. Lastly i-Con mapped
XMLife documents into the moving our data, with minimal impact to our busi-
new UniVerse database. ness operations.“
INTELLIGENT INTEGRATION Great return on investment
XML translations
ACCORD XMLife standard
Pro Solutions and i-Con
About i-Con
i-Con is used as a System Integration tool, with the ability to integrate data from various sources,
including multi-dimensional databases, allowing you to leave your heterogeneous environments as
they are. i-Con will help you leverage your existing IT assets, enabling you to face the necessary
challenges and get more from your current infrastructure. i-Con provides a more flexible solution to
your integration dilemma. Future business scenarios are supported by i-Con as well as the solu-
tions that utilize enterprise services to blend functionality from multiple applications.
Reduce the need for costly system interfaces and let i-Con to do the Intelligent Integration.
Pro Solutions
Pro Solutions is a privately owned Business Systems Development and Professional Services
Group, based in Johannesburg, South Africa. Since 1986, Pro Solutions has been a leading soft-
ware development and services company delivering world-class solutions to local and international
organisations.
A trusted business partner and adviser to our customers, Pro Solutions leverages a powerful blend
of the best industry-proven practices and leading standards, refined business acumen and deep
market understanding. This coupled with profound technical skills and extensive hands-on experi-
ence enable us to meet the toughest challenges that our clients face. Helping them compete suc-
cessfully in the dynamically changing IT marketplace and achieve the maximum return on invest-
ment.
When you choose Pro Solutions, you have the freedom to focus on your core business while a
dedicated team of qualified experts is committed to deliver a solution taking on all and every as-
pect of your IT needs, starting with strategy consulting and concept development, design and ar-
chitecture, continuing to deployment, training, maintenance and future enhancement.
Our business philosophy is to provide you with high quality personal involvement in everything we
do.
For more info:
call: +27 11 259 6400
visit: www.i-deal.co.za
email :info@i-deal.co.za
