The model of elementary magnets explains that when the magnet is removed from iron, the alignment of the “mini-magnets“ (elementary magnets) disarranges again over time. The elementary magnets lose their parallel structure and align themselves randomly, so that the piece of iron is not magnetic any longer.
The elementary magnets of a magnet, however, permanently point in the same direction. As long as there are no bigger impacts from outside, a magnet keeps its effects.

If elementary magnets in a real magnet realigned themselves, we could not mark their north and south poles (green and red colour). With each realignment, the polarity of the magnet would change as well.