To reduce the oversampling rate of baseband signals, decimation is widely used in digital communication systems. Polyphase filters (PPFs) can be used to efficiently implement decimators. SRAM-based FPGAs provide large amounts of resources combined with flexibility and are a popular option for the implementation of communication receivers. However, they are sensitive to soft errors, which limit their application in harsh environments, such as space. An initial reliability study on SRAM-based FPGA implemented decimation shows that the soft errors on around 5% of the critical bits in the configuration memory of the decimator would degrade the decimated signal dramatically. Based on this result, this paper proposes an efficient fault tolerance scheme, in which the high correlation between adjacent PPFs outputs is utilized to tolerate the fault of a single-phase filter, and a duplicate and comparison structure is used to protect the fault tolerance logic. Hardware implementation and fault injection experiments show that the proposed scheme can drastically reduce the number of critical bits that cause severe output degradation with 1.5x resource usage and 0.75x maximum frequency relative to the unprotected decimator. Therefore, the proposed scheme can be an alternative to Triple Modular Redundancy that more than triples the use of resources.