I haven't moderated in a very long time, but the one thing that sticks out for me from my phpbb forum days is the paradox of tolerance.
Basically--you generally don't want to be a jerk. You do want to be fair and tolerant to the best of your ability.
However, if you let THAT GUY run around doing shitty things, your community will end up polluted and toxic and it will drive away the people who would/could've made your community a nice place.
So be kind--but not so kind it allows losers to rampage unchecked.
It can really feel shitty to use the banhammer, but sometimes it's important to pull the trigger for the sake of the community. If you don't, the good people will leave, and the fuckwads will stay and fuck around.
The best-moderated community I've seen in recent memory is actually the Raised by Narcissists sub on Reddit. Probably because the mods there are all trauma survivors and have hawk-eyes for ALL the weasly shit verbally-manipulative people can get up to. So they lay out the rules plainly and fairly--but they also don't fuck around when things need to be shut down. Most of the time the troll/drama posts don't even get out of New.
I don't know what sort of sub you're moderating here, it's possible it won't attract the same BS that THAT sub attracts by its very nature, but as a mod (if you want to be a good/fair one) you'll start to develop a second sense for when someone is using outwardly-reasonable words to try to defend what they're doing, while continuing to shit-stir below the surface. You'll get an eye for overall patterns.
Trust your gut--if you feel something's wrong but can't put a finger on why because everyone's using polite words, ponder it for a while and see if you can figure out what's up. Often where there's smoke, there's fire.
But yeah. Don't be a dick, explain your rules clearly, but when someone's fucking around on your sub? Let 'em find out.
Oh...and if you have been modding for a while, but find yourself getting stressed as fuck or angry and snappish in real life? It's totally cool to step back from a modding job. I don't mod anymore even though I know "how" to because I have C-PTSD and sub drama puts me into legit raging flashbacks. It's terrible for my health, and not fun for others if my temper slips, so I don't do it anymore.