[...] Is that “nonbinary” is not the most accurate term for people because it, seems to me, to imply that gender is binary.
It does not imply that. If it did, it would defeat itself, because there is no place for nonbinary in binary. It implies that a slice of gender can be seen as binary (Men and Women), but there is more to it.
I can see how people could have problems with that, yes. It is just not what I understood from the comment.