A Fine Line between Funny and Offensive

Humor can be clever like a play on words or crude like toilet humor.  More often, humor is found in misery.  Other people's misery, to be specific.  Slapstick comedy's main focus is literally when a character endures some sort of physical pain.  Probably the most popular example of this would be The Three Stooges.  There isn't much more to Larry, Curly, and Mo than poking out eyes, and slapping the backs of heads.  Humor is also found in making fun of appearances, gestures, and actions.  If someone talks funny, has a big nose, or has odd interests, they become perfect candidates to be the butt of a joke.  But when do caricatures of life as a whole become offensive?  The phrase "too soon?" often follows a snarky remark, taken too seriously.  Do we need to be cautious and sensitive to everyone's feelings?  Or do people need to relax, and take things less seriously?  Offensive comedy is prevalent and isn't going anywhere.  Family Guy and Southpark will continue to push the envelop to see how offensive they can be.  It just makes me wonder if there's a line, and if so, where do we draw it?