Data position and profiling in domain-independent warehouse cleaning
5th International Conference on Enterprise Information Systems, ICEIS 2003
A major problem that arises from integrating different databases is the existence of duplicates. Data cleaning is the process for identifying two or more records within the database, which represent the same real world object (duplicates), so that a unique representation for each object is adopted. Existing data cleaning techniques rely heavily on full or partial domain knowledge. This paper proposes a positional algorithm that achieves domain independent de-duplication at the attribute level. The paper also proposes a technique for field weighting through data profiling, which, when used with the positional algorithm, achieves domain-independent cleaning at the record level. Experiments show that the positional algorithm achieves more accurate de-duplication than existing algorithms.
Ezeife, C. I. and Udechukwu, A. O.. (2003). Data position and profiling in domain-independent warehouse cleaning. 5th International Conference on Enterprise Information Systems, ICEIS 2003, 1, 232-238.