Wisps spawning near the end of a map

I've had this happen too many times, when I'd have nearly or be practically finished with exploring the map, and only by the "exit" of the map, I see the NPC for the Wisp.

And there would be no Wisp because I'd have killed all the rare mobs.

This happens during the campaign and during Atlas mapping.

And all in all it just adds to the feeling of lameness with this whole Wisp mechanic.

Can't something be done to stop this from happening?

It's just...lame.