That's like saying that only using high security locks with various security pins in them to protect your house is a bad idea, and you should throw in some secured with padlocks too just to change things up.
And if some of them are shitty masterlocks, well, you're changing things up.
That's really not how security works.
Yes, pass phrases can have large amounts of entropy attached. But unless you are picking your pass phrases truly randomly, with a large dictionary, and using unique pass phrases per site, and the sites are not silently truncating the password input (such as bcrypt which truncates to 72 bytes), you are not actually getting that large amount of entropy.
Where as a 16 character password that randomly uses the ASCII printable range, excluding spaces, gives you 93^16 possible combinations. That's 31313180170800116587336013460801 passwords.
Or, very roughly, 104.6 bits of entropy. (104.6265409777285022441578006899739 bits of entropy if you want to be downright absurd about it.)
Knowing that you're doing that simply doesn't help the attacker in any meaningful way.
Bumping that to 20 characters gives you over 130 bits of entropy, or 2342388736625917052139104541473924426001 possible combinations.
This is quite simply not a viable attack surface.
Where as saying 'use pass phrases for some things' means that it is quite likely that some of your pass phrases are going to be much less secure than this.
But let's give the same numbers for properly generated random passphrases.
The xkcdpass utility can help us here.
Even picking entirely randomly, out of a large word list of 7227 words, a 6 word pass phrase only gives roughly 76 bits of entropy.
Going up to 8 words gives us roughly 102 bits of entropy, that helps a ton... Except that some of those passphrases are going to be longer than 72 bytes. So you're almost certainly losing bits of entropy.
That best case still gives you fewer bits of entropy than a 20 character randomly generated password. Unless you're trying to memorize your password, there are no benefits to alternating between randomly generated passwords with good generation settings and passphrases.
And if you're trying to memorize your passwords, you are definitely doing it wrong.
Mastodon absolutely does have a weakness of making it more difficult to find people that you want to follow based on what you have already engaged with.
And from a purely user perspective, that is a weakness.
But it's also a very distinct choice. Because having enough data to be able to meaningfully make such recommendations means having a central database of every user interaction by every user.
And it also means making choices and value judgements which, almost by definition, can not be value neutral.
If the creators of the algorithm are good, they will actually be aware of the choices and value judgements being made, if not, well... They will still be making them, just not in nearly as educated of a way.
On the whole, I really hope that we eventually come up with answers to these problems that make it possible for a user to make those choices, and to have the amount of recommendations that they want, while somehow not having anyone have the huge database of user interactions. I'm not sure if that's even possible, most especially if you assume that there will be entities on the fediverse that are fudging their data to get recommended in ways that other users don't want.
But it sure would be interesting to try.