Southern purple mint moth caterpillars can make rosemary look awful.
Rosemary damaged by southern purple mint moth caterpillars and disfigured by their webbing and frass.

Photo by Matt Bertone, NC State University