remove all the duplicate data and filter the error data.
remove all the duplicate data and filter the error data.