A native speaker of Spanish, for example, will find Portuguese much easier to learn than a native speaker of Chinese, for example, because Portuguese is very similar to Spanish, while Chinese is very different, so first language can affect learning a second language. The greater the differences between the second language and our first, the harder it will be for most people to learn. Many people answer that Chinese is the hardest language to learn, possibly influenced by the thought of learning the Chinese writing system, and the pronunciation of Chinese does appear to be very difficult for many foreign learners. However, for Japanese speakers, who already use Chinese characters in their own language, learning writing will be less difficult than for speakers of languages using the Roman alphabet.
Some people seem to learn languages readily, while others find it very difficult. Teachers and the circumstances in which the language is learned also play an important role, as well as each learner's motivation for learning. If people learn a language because they need to use it professionally, they often learn it faster than people studying a language that has no direct use in their day to day life.
Apparently, British diplomats and other embassy staff have found that the second hardest language is Japanese, which will probably come as no surprise to many, but the language that they have found to be the most problematic is Hungarian, which has 35 cases (forms of a nouns according to whether it is subject, object, genitive, etc). This does not mean that Hungarian is the hardest language to learn for everyone, but it causes British diplomatic personnel, who are generally used to learning languages, the most difficulty. However, Tabassaran, a Caucasian language has 48 cases, so it might cause more difficulty if British diplomats had to learn it.
Different cultures and individuals from those cultures will find different languages more difficult. In the case of Hungarian for British learners, it is not a question of the writing system, which uses a similar alphabet, but the grammatical complexity, though native speakers of related languages may find it easier, while struggling with languages that the British find relatively easy.
No language is easy to learn well, though languages which are related to our first language are easier. Learning a completely different writing system is a huge challenge, but that does not necessarily make a language more difficult than another. In the end, it is impossible to say that there is one language that is the most difficult language in the world.