I am not sure about the duplicate.
But if you want to keep the stuff in front of the ] then it could work like this:
Action of the type "Replace with regular expression"
Search string: (.]).
Replace string: $1
EDIT:
Here is a link that looks for duplicate words and removes them except on: /t/16289/1