I've used matrix for a year now and it works, but it seems slow.
Lots of people tried to self-host it and reported it uses too much RAM for what it does. (It allegedly uses 1GB or more of ram even if it only has 1-2 users)
Efficient software is a must. Software must not waste resources simply because "they are there". That's my biggest gripe with matrix.
Disclaimer: i've not tried to host matrix myself, so i could be wrong here.