30 Jan 20031Information Management Framework Data Quality2What is qualityQuality is dynamic concept that is continuously changing to respond to changing customer requirements Defined in 3 ways:Conformance to specifications (DQA)Fitness for use (Surveys)3Quality issuesProblems can result from:Human errorMachine errorProcess error4PurposeStorageAccess andUse Archive andCollectionInformation lifecycle phasesDisposalData QualityAssessmentData EntryVerificationInternal qualitychecksData StoreData CollectionData AccessHistoric dataUser FeedbackAnalysis andFitness for useApproval topublish (vetting)30 Jan 20035Conformance to specifications: Quality Plan Data Quality Assessments6Data Quality AssessmentsQuality Investigation Module Update MetadataData QualityAssessmentprocessesPrior auditsStandardsUse/analyse dataDefine BusinessRequirementsDefineBusiness/datarulesQuality Plan(Benchmarks)System SpecData Quality management cycleObtain FeedbackStartUpdate DataManagement PlanOriginal useDefine Purposeof DataUpdateBenchmarks7Data StoreData CollectionData AccessHistoric dataStorageAccess &UseArchive/DisposalCollectionDQ Assessment and Remediation ProcessAudit RecommendationsData Quality AssessmentApproval /Priority ProcessData RemediationPart 3 Data Management Plan Information lifecycle phases8Recording quality -ANZLICTotal Input ErrorsLinageVertical accuracyHorizonal accuracyPositional accuracyAttribute accuracyAttribute consistancyLogical consistancySpatial CoverageTemporal CoverageClassificationCompletenessQuality9Business rulesEach business rule should have an expected outcome (benchmark)Business rules need to align to quality ANZLIC elements 10Findings -DQ ProcessesThe processes and guidelines are good!The Data Management Plan is importantNeeds to be completed by all data sets prior to AssessmentBenchmarks for quality established with Data Managers before DQA11Soil ProfileVery large and varied data set (millions of soil properties)Where Data exists -is mostly goodMany missing valuesData Transformation ErrorsData on forms different to values in databaseMissing values set to default values in load program.12Data Analysis –Soil PropertiesExamples of problems:Location Accuracy -Invalid grid references for a grid zoneMandatory Fields missing dataNature of Exposure-1269 records missing valueLogical InconsistenciesIf Horizon Code begins with 'B' And ACS Order is 'SO' (Sodosol)Then pH >= 5.5238 records in error.13Data Analysis –Ground WaterMinimal spatial data (point locations only)Data where present is mostly goodMany missing values14Examples of problemsInvalid Key fieldsWork Number of non standard format Location AccuracyInvalid grid references for a grid zoneLogical InconsistenciesJobs completed before they startedHole depth of 36kmMandatory Fields missing dataWork Type Code -1503 records missing value.15Data Analysis –Ground WaterDatabase Issues:No Load or creation date in database (only update date)Impossible to apply date based business rulesGW licenses mandatory from 2001 onwards.Logical Inconsistencies:License Form A received and no GDS record (1000’s)Needs investigationRegion CodeRegion NameGW Licenses in LASGW Licenses in GDSGW Licenses not in GDSPercentage Missing10Sydney -South Coast362234202025%20Hunter2000128072036%30North Coast32013162391%40Murrumbidgee191218071055%50Murray2350911143961%60Lower Murray /Darling84424250%70Lachlan1913137154228%80Macquarie -Western2345200234314%90Barwon45264445812%16Data AnalysisAction ListsGenerated for each data setScope of RemediesImproving data quality goes beyond the identifying, measuring and fixing the data in the IT systems. Improve data capture–Train entry staff–Replace entry processes–Provide meaningful feedback–Change motivations to encourage qualityAdd defensive checkers, Periodic DQ asssessments, Data cleansing17Data Quality ReportingData Quality PortalGeneral DQ informationStatistical Reporting and MonitoringData Quality Exception ReportingManagement of Data Quality issues30 Jan 200318Fitness for use -User needs covered later in day30 Jan 200319Improving quality20Ways of improving qualityTackle quality at source, not downstream in the lifecycleTraining data collectors in importance on getting it rightContinual improvement with quality method 21Planning ProcessControlling process (check)Initiating processClosing processExecuting process (do)Links among Process Groups in a Phase(Arrows represent flow of information)( PMBOK 2000 Fig 3-1 p31)2223