Regarding appropriateness:
One thing that is really hard for people who heavily rely on using reason, discussion, and negotiation, is using any of these things on someone who is incapable of them. I've known this for a long time but it really, really hit me at the Milo event in UW when I did my damnedest to get any Antifa idiot to talk to me ("But tell me why I'm a fascist!"), and I was simply rewarded with blows and paint on my head. You literally cannot reason with some people.
Unfortunately, there are people who can fake reason. These are known as sophists. Whether they are really good at undermining your argument, or just not shutting the fuck up so no other information can get through, they will often appear to be reasonable to others watching. Trolling then is an effective method for frustrating them and sabotaging their efforts, allowing you to get more wholesome information out. The goal is to shut them up, waste their energy, or get them on something unimportant, in the same way they set out to waste your energy by pretending to be reasonable.
Regarding being a jerk:
The initiator who pretends reason is the jerk. When you troll, you will appear as a jerk to people who can't reason, and a savior to people who can. Focus on impressing the important of the two groups. I noticed this when Stef had a call-in show a few months back with a guy, I think named Fritz. Fritz was an annoying sophist and Stef literally spent 40 patient minutes with him. I could tell within 30 seconds Fritz was a jerk, and after 40 minutes when Stef finally lost his patience, he received a HUGE amount of feedback that HE was the jerk. Stef actually had to do another call-in show to explain exactly what happened and why Fritz was the jerk and not him. I noticed it right off the bat, and was actually frustrated with how patient Stef was. If you identify sophistry, don't worry about being a jerk.
How to know when you're going to far:
When you're trolling people who are being reasonable to begin with? When you can't stop trolling?
Hope that helps.