speaking as a former matrix user and fan, I hate it because:

• the protocol is absurdly complex and bloated
• despite the promises of an open standard making interoperable clients possible, Element is the only client that supports most of the features in my experience, and it has constant bugs.
• It's also slow and has some really annoying interface design issues, like not being able to adjust the width of the sidebar to where I want it (to be fair, many other messengers, like Signal, also have this problem)
• I had several experiences inviting someone to become a new Matrix user and having to immediately tell them that I couldn't decrypt their messages and the first 3 things we tried to resolve it didn't work. Made me feel like an ass for inviting them
• It has many security flaws (for example, all sorts of metadata and even some kinds of message content, like emoji reactions, are unencrypted)
• The whole "verify your device" nag thing is incredibly annoying and when I try to actually do it, it has almost never worked for me.

Overall, I view it much like I do XMPP. it takes basically the same approach to everything and has all of the same downsides, except E2EE is easier to get working I guess