Bash script to delete duplicate files in system

Below command will find and list all duplicate files by its size and md5hash which you can later delete on your choice.
This command finds duplicate files by comparing size first, then md5sum, it doesn’t delete anything, just lists them.

find -not -empty -type f -printf "%s\n" | sort -rn | uniq -d | xargs -I{} -n1 find -type f -size {}c -print0 | xargs -0 md5sum | sort | uniq -w32 --all-repeated=separate | cut -d" " -f3

Example:

[rks@localhost temp]$ find -not -empty -type f -printf "%s\n" | sort -rn | uniq -d | xargs -I{} -n1 find -type f -size {}c -print0 | xargs -0 md5sum | sort | uniq -w32 --all-repeated=separate
ae474fbc97b9d6ff6e1fb37c2b5c0a1d  ./abc
ae474fbc97b9d6ff6e1fb37c2b5c0a1d  ./abc2

[rks@localhost temp]$ find -not -empty -type f -printf "%s\n" | sort -rn | uniq -d | xargs -I{} -n1 find -type f -size {}c -print0 | xargs -0 md5sum | sort | uniq -w32 --all-repeated=separate | cut -d" " -f3
./abc
./abc2

Below command will find and list all duplicate files by its md5sum which you can later delete on your choice.
Find Duplicate Files (based on MD5 hash)
Calculates md5 sum of files. sort. uniq based on only the hash. use cut to remove the hash from the result.

find -type f -exec md5sum '{}' ';' | sort | uniq --all-repeated=separate -w 33 | cut -c 35-

[rks@localhost temp]$ find -type f -exec md5sum '{}' ';' | sort | uniq --all-repeated=separate -w 33 | cut -c 35-
./abc
./abc2

Below script searches and removes ALL duplicate files(doesn’t keep a sample of duplicate deletes all). So please test it on sample files before real use.

#!/bin/bash
#Filename: rmdups.sh
#Description: Find and remove duplicate files.
ls -lS | awk 'BEGIN {
getline;getline;
name1=$8; size=$5
}
{ name2=$8;
if (size==$5)
{
"md5sum "name1 | getline; csum1=$1;
"md5sum "name2 | getline; csum2=$1;
if ( csum1==csum2 )
{print name1; print name2 }
};
size=$5; name1=name2;
}' | sort -u > duplicate_files
cat duplicate_files | xargs -I {} md5sum {} | sort | uniq -w 32 | awk '{ print "^"$2"$" }' | sort -u > duplicate_sample
echo Removing..
comm duplicate_files duplicate_sample -2 -3 | tee /dev/stderr | xargs rm
echo Removed duplicates files successfully.

In case of any ©Copyright or missing credits issue please check CopyRights page for faster resolutions.