It is often suggested that oxygen vacancies (V O ) are involved in fatigue and pinning of domain walls in ferroelectric (FE) materials, but generally without definite evidence or models. Here the progress of damage induced by the coexistence of FE and antiferroelectric (AFE) domains in the absence of electric cycling is probed by monitoring the Young’s modulus, which may undergo more than fourfold softenings without significant changes in the Raman spectra, but may end with the disaggregation of PZT with ∼5% Ti. At these compositions, the FE and AFE phases coexist at room temperature, as also observed with micro-Raman, and hence the observations are interpreted in terms of the aggregation of V O at the interfaces between FE and AFE domains, which are sources of internal electric and stress fields. The V O would coalesce into planar defects whose extension grows with time but can be dissolved by annealing above 600 K, which indeed restores the original stiffness. The observed giant softening is interpreted by assimilating the planar aggregations of V O to flat inclusions with much reduced elastic moduli, due to the missing Zr/Ti−O bonds. A relationship between the coalescence of a fixed concentration of V O into planar defects and softening is then obtained from the existing literature on the effective elastic moduli of materials with inclusions of various shapes.