We introduce a kinetically constrained spin model with a local softness parameter, such that spin flips can violate the kinetic constraint with an (annealed) site-dependent rate. We show that adding MC swap moves to this model can dramatically accelerate structural relaxation. We discuss the connection of this observation with the fact that swap moves are also able to accelerate relaxation in structural glasses. We analyse the rates of relaxation in the model. We also show that the extent of dynamical heterogeneity is strongly suppressed by the swap moves.