Passive early-type galaxies (ETGs) provide an ideal laboratory for studying the interplay between dust formation around evolved stars and its subsequent destruction in a hot gas. Using Spitzer-IRS and Herschel data we compare the dust production rate in the envelopes of evolved AGB stars with a constraint on the total dust mass. Early-type galaxies which appear to be truly passively evolving are not detected by Herschel. We thus derive a distance independent upper limit to the dust grain survival time in the hostile environment of ETGs of <46 ± 25 Myr for amorphous silicate grains. This implies that ETGs which are detected at far-infrared wavelengths have acquired a cool dusty medium via interaction. Given likely time-scales for ram-pressure stripping, this also implies that only galaxies with dust in a cool (atomic) medium can release dust into the intra-cluster medium.