You can't really prevent it. Based on the scene setup the engine considers the green X to be closest to the destination. The question is, is it really such a big problem? Normally the player only clicks the active spots, which send the actor to a specific position. This whole "go to the closest position" situtation should be relatively infrequent.