Reducing the loss of information through annealing text distortion Articles uri icon

publication date

  • January 2011

start page

  • 1090

end page

  • 1102


  • 7


  • 23

International Standard Serial Number (ISSN)

  • 1041-4347

Electronic International Standard Serial Number (EISSN)

  • 1558-2191


  • Compression distances have been widely used in knowledge discovery and data mining. They are parameter-free, widely applicable, and very effective in several domains. However, little has been done to interpret their results or to explain their behavior. In this paper, we take a step toward understanding compression distances by performing an experimental evaluation of the impact of several kinds of information distortion on compression-based text clustering. We show how progressively removing words in such a way that the complexity of a document is slowly reduced helps the compression-based text clustering and improves its accuracy. In fact, we show how the nondistorted text clustering can be improved by means of annealing text distortion. The experimental results shown in this paper are consistent using different data sets, and different compression algorithms belonging to the most important compression families: Lempel-Ziv, Statistical and Block-Sorting. © 2006 IEEE.


  • Computer Science
  • Information Science


  • clustering by compression data compression information distortion kolmogorov complexity normalized compression distance block-sorting clustering by compression compression algorithms data sets experimental evaluation information distortion knowledge discovery and data minings kolmogorov complexity normalized compression distance text clustering cluster analysis data mining data compression