Every small-to-medium sized community trying to build will have a handful of users like this. I managed to discover the serial downvoters on my old lemm.ee comm and when I banned them (about 4 of them?) it had a huge impact. They didn't all downvote /everything/ but they downvoted a lot of things, and no contribution. And if they got in early, they could sink new threads.
Made a huge difference.
Season 2 is a better season. The weaker plotlines aren't as weak at all, and the Empire genetic dynasty carries the day just as much.