Title

Data position and profiling in domain-independent warehouse cleaning

Document Type

Conference Paper

Publication Date

2003

Publication Title

5th International Conference on Enterprise Information Systems, ICEIS 2003

Volume

1

First Page

232

Last Page

238

Abstract

A major problem that arises from integrating different databases is the existence of duplicates. Data cleaning is the process for identifying two or more records within the database, which represent the same real world object (duplicates), so that a unique representation for each object is adopted. Existing data cleaning techniques rely heavily on full or partial domain knowledge. This paper proposes a positional algorithm that achieves domain independent de-duplication at the attribute level. The paper also proposes a technique for field weighting through data profiling, which, when used with the positional algorithm, achieves domain-independent cleaning at the record level. Experiments show that the positional algorithm achieves more accurate de-duplication than existing algorithms.