The Outdoor Boys explains why his son doesn't come on camps as often