Cleaning Line Lists: Duplicates and Version Differences
Product codes, task lists and exported records may contain repeated lines. Cleaning should remove unwanted repetition without losing distinct records.
Last updated:
Define a duplicate first
Consider three lines: A-001, A-002, A-001. Removing exact duplicate lines leaves two distinct codes. But A-001 and a-001 are not necessarily the same identifier. Decide whether leading or trailing spaces matter before normalizing text.
A controlled cleanup workflow
Keep the original. Test a small subset and compare input and output line counts. Inspect codes with leading zeros, blank lines and case differences. Use the cleaned list only after checking these details.
Compare two versions
Place the old text in one field and the new text in the other. Review additions and deletions yourself: a highlighted difference is not automatically an error. Changing 00123 to 123 may look minor but can identify a different record.
Use table tools for structured data
When data contains comma-separated or tab-separated columns, removing a line removes the entire record. Use table tools when duplicates must be identified by one column; do not assume a plain-text operation preserves column relationships.