Duplicate detection

Search engines may apply different levels of duplicate detection. Exact duplicates means the same document, but located in different repositories. Next level of duplicate detection is typically to look for documents with equal visible content (excluding metadata). Certain applications may want to apply even more aggressive duplicate detection, e.g. based on a set of fields that are equal.See also Field collapsing.