Duplicate document detection in a client file means comparing what each page says, not what each file is called. A file hash catches byte-for-byte copies, such as a forwarded attachment. Rescans and phone photos need a check on the issuer, form type, period and key amounts. Flag the second copy and keep it, so nothing is entered twice.
How duplicate documents get into a client file
Duplicate documents reach a client file through three routes: the client sends the same thing twice, the client sends the same document in two formats, or the firm creates a copy while handling the file. Each route produces a different kind of duplicate, and each needs a different check to catch it.
Client resends are the most common. A client emails a 1099-INT on Monday, hears nothing, and emails it again on Thursday. Another client replies to the firm's request list and attaches everything they sent last week, plus the two new items. A spouse uploads the household's documents without knowing the other spouse already did. In every case the second file is often an exact byte-for-byte copy of the first, because it is the same attachment sent again.
Format changes are harder. A client photographs a brokerage statement with a phone, then later downloads the same statement as a PDF from the broker's website. A bookkeeping client forwards a vendor's emailed invoice and also scans the paper copy that came in the mail. The content is the same document; the files share nothing. The difference between a scanned copy and an emailed original, and why each reads differently, is the subject of the post on scanned invoices versus emailed PDF invoices.
Portal downloads add a twist. Many payers let clients download a statement as often as they like, and some regenerate the PDF each time. Two downloads of the same statement can then differ in tiny ways, such as a print date in the footer, while every figure is the same.
Firm-side copies are the last route. Staff save an attachment to the client folder and also drag the whole email in. Someone splits a combined PDF and keeps the combined file next to the parts. A preparer scans the paper a client dropped off without checking that the client had already uploaded it. These duplicates are avoidable, but only if someone looks for them.
Exact copies versus rescanned duplicate documents
An exact copy is a second file whose bytes are identical to the first. A rescanned duplicate is a new image of the same paper, or a new export of the same data, so its bytes differ even though the document is the same. A file hash detects the first kind reliably and misses the second kind entirely.
A file hash is a short fingerprint calculated from a file's contents. The Microsoft documentation for the PowerShell Get-FileHash cmdlet explains both halves of why this works and where it stops. It says that "if the hash values of two files are identical, the contents of the files are also identical," and that file names and extensions can change without changing the hash. It also says "changing even a single character in the contents of a file changes the hash value."
The first statement makes hashing ideal for resent attachments. A 1099-DIV forwarded twice, renamed by the email client to "1099-DIV(1).pdf", still hashes to the same value as the original. Comparing hashes across a client folder finds every such pair, whatever the files are called.
The second statement is why hashing fails on rescans. A phone photo taken from a slightly different angle is a different file. So is a second scan of the same paper on the office scanner, and so is a portal download with a new print timestamp. Each one produces a new hash, and a hash-only check reports no duplicate.
Page image similarity sits between the two. It compares how pages look, which can catch a rescan of the same paper, but it struggles when one copy is a clean PDF and the other is a tilted photo, and it can wrongly match two different months of the same bank's statement, which share a layout.
The W-2 version of this problem, including the boxes that settle whether two scans are one form, has its own post: which W-2 boxes prove two scans are the same form.
How do you detect duplicate PDF files by content?
To detect duplicate PDF files by content, read the identifying fields from each document and compare those fields instead of the files. Two documents are duplicates when they share the same document type, the same issuer, the same period and the same key amounts. Those four fields survive rescanning, renaming and format changes.
Document type comes first because it narrows the comparison. A 1099-INT is only ever a duplicate of another 1099-INT. A bank statement is only a duplicate of another statement from the same bank.
Issuer is the second field. On a tax form, that means the payer or employer name and its identification number. On a statement, it means the institution and the last four digits of the account. On an invoice, it means the vendor. Two forms of the same type from different issuers are never duplicates.
Period is the third. A tax form carries a tax year. A statement carries a start and end date. A receipt or invoice carries a date. A December statement and a January statement from the same account are different documents with almost the same layout, which is exactly the case where image similarity goes wrong and field comparison does not.
Key amounts are the tiebreaker. For a W-2, the wage and withholding boxes. For a 1099-INT, the interest figure. For a statement, the opening and closing balances. For an invoice, the invoice number and total; the fields that settle an invoice match are set out in which invoice fields identify a duplicate invoice.
Which detection method catches which duplicate
The table compares four methods against five scenarios a firm meets every season. The scenarios use fictional examples.
| Scenario (example) | File name | File hash | Page image similarity | Extracted fields |
|---|---|---|---|---|
| 1099-INT from Harbor Federal emailed twice, second copy renamed "1099-INT(1).pdf" | Misses | Catches | Catches | Catches |
| W-2 from Lakeshore Dental sent as a phone photo, then as a portal PDF | Misses | Misses | Often misses | Catches |
| Same bank statement downloaded twice, footer print date differs | Sometimes | Misses | Catches | Catches |
| December and January statements for account XXXX-4821, same layout | Misses | Misses (correctly) | May match wrongly | Keeps apart |
| Original 1099-B and a copy with the CORRECTED box checked | Misses | Misses (correctly) | May match wrongly | Keeps apart, flags the correction |
File names catch almost nothing, because clients and email programs rename files freely. Hashes are fast and certain for exact copies but blind to everything else. Field comparison is the only method that both finds format-changed copies and keeps lookalike non-duplicates apart.
Lookalike documents that are not duplicates
Lookalike documents are pairs that share an issuer and a layout but are both supposed to be in the client file. Treating one as a duplicate removes real information. The usual lookalikes are consecutive statements, corrected forms, multiple accounts at one institution, and documents for each spouse.
Consecutive statements are the most frequent. Twelve monthly statements from one checking account look nearly identical as images. Only the period and the balances differ, which is why the period field matters so much.
Corrected forms are next. The IRS General Instructions for Certain Information Returns tell filers to mark the "CORRECTED" checkbox only when correcting a form already filed with the IRS or furnished to the recipient. A 1099 with that box marked replaces the figures on the original. It is a new document, not a copy, and both should stay in the file with the original marked as superseded.
Multiple accounts at one institution can fool a quick review. A client with a checking account ending XXXX-4821 and a savings account ending XXXX-7730 at the same credit union receives two statements each month that share a letterhead. The account number decides.
Spouses and dependents round out the list. A married couple can each receive a 1099-INT from the same bank for separate accounts. The recipient name and identification number on each form tell them apart.
When a lookalike pair cannot be resolved from the page, ask the client. One short question costs less than an amended return.
What should happen to a flagged duplicate document?
A flagged duplicate document should be kept, linked to the copy it repeats, and excluded from entry. Deleting it loses the record that the client sent it, and leaving it unmarked invites a second entry. A named reviewer confirms each flag, because a few flags will turn out to be lookalikes rather than copies.
Keep, do not delete. A firm that deletes the second copy cannot later show what the client sent, and cannot answer a client who asks whether a resend arrived. Move confirmed duplicates to a subfolder or mark them in an index. Either way, the file still exists and points to the copy that was used.
Link the pair. A duplicate note should say which document it repeats: "Duplicate of Harbor Federal 1099-INT, received Feb 2" is useful months later. "Dup" is not.
Decide who confirms. In a small firm, the preparer who owns the client reviews flags before data entry starts. In a larger firm, the person who handles intake confirms exact copies and passes field-match flags to the preparer, because those are the ones most likely to be a lookalike.
Choose the copy to keep as the source. When one copy is a clean PDF and the other a phone photo, use the PDF for entry. When one copy is an original and the other a correction, the correction is not a duplicate at all, so it goes back to the lookalike rules above.
Check before entry, not after. A duplicate caught while the folder is being sorted costs a minute. The same duplicate caught at review, after its figures went into the return, means finding every entry it touched and backing each one out. Run the comparison each time a new batch arrives, so a resend in March is compared against everything that came in since January, not only against the files in the same email.
Track the flag in the client's status list. A duplicate status alongside received and missing keeps the second copy visible without letting it count toward the expected documents. Schema CPA works this way: each incoming page is labelled by type, issuer and period, and a repeat is shown as a duplicate in the client's list instead of being added again. The client tracker demo shows the three statuses side by side.