Several people have mentioned the difficulty of Japanese grammar, and I want to expand on that a little. If you have already learned a second language that's significantly different from your first language, then you probably won't find Japanese grammar to be too confusing. It turns out that there aren't that many different ways for most world languages to organize themselves, so what we often see is that people learning their third or fourth language don't actually find the grammar to be too difficult.
If you want to learn the language efficiently, you need large amounts of input and output, because you have to memorize so many words and patterns and example sentences. And then from time to time you study some grammar to make sure that you know why those example sentences mean what they mean.
For example, other commenters have mentioned that the order of the parts of speech in the sentences for Japanese is quite different from English. That's true. But actually that's not a stumbling block, because you should often be forming sentences by creating variations on sentences that you've already memorized. And if you're doing that, it doesn't take very long to get used to the switched order.
Some new language learners intuitively decide that they should try to formulate the perfect sentence in English first, and then ponder how to translate that to Japanese, in which case they feel like the switched order is problematic... Here, the real problem is their intuition. If they can learn to stop doing that, they will find it much easier to proceed in their studies. But sometimes that's easier said than done.