Asteroids yes, comets not so much. A long period comet could hit the Earth with a warning time of only a few years, and could easily be large enough to destroy human civilization.
Pluto is effectively a comet. There are objects just as large or larger in the Kuiper Belt and Oort cloud. It's likely that the larger an object is the less likely it is to be jostled by interstellar forces and have its orbit perturbed enough to fall into the inner Solar System (if we're lucky) but we don't really have any clue on what the size limits are for such processes. It would suck to find out by way of the entire human race being eliminated.
