I designed a small plug, which is superglued or epoxied to the hole and holds the magnet in the right position.
