MinHash for twin resources

The text similarity that helps recognise (4 passages, no note) the forms (52 passages, no note) of one document (note: Document) works on extracted text (note: Extracted text), never on binary (7 passages, no note) content (43 passages, no note), and never compares every pair (47 passages, no note). Each text goes to its comparison form (note: Comparison form), then to shingles (5 passages, no note) of five words, then to a MinHash signature (3 passages, no note) of 128 functions under a fixed seed (4 passages, no note); LSH banding (3 passages, no note), four rows (41 passages, no note) per band (3 passages, no note), enumerates the candidate pairs (3 passages, no note), two signatures (8 passages, no note) that share no band being never compared, so that the full matrix is never built. In auto mode (15 passages, no note) the pairs estimated (3 passages, no note) at 0.5 or more have their exact Jaccard index (3 passages, no note) recomputed on the full shingle sets, and the finding (note: Finding) gives the share of lines in common; a pair whose word counts differ by more than half is an inclusion (3 passages, no note) rather than a duplicate (10 passages, no note) and its content signal is capped (3 passages, no note). Resources (note: Resource) are sorted by identifier (note: Identifier) before anything else, so two runs give the same result (48 passages, no note).

• Resource
• Representation
• Duplicate candidate
• Build