Imho, any kind of fiction is imagination, dream, escape, and is without limit. The setting and the 'reality' it's based upon may change in one form of fiction compared to another but that doesn't make them different (or less daring) anymore than, say, me changing clothe.
So to answer your question, I think fantasy like most other literary labels is mostly a term coined to allow a few people to shine a little more than they would otherwise do, and maybe to help publishers create/manage more niche markets. And since it works quite well, it started being used by readers to feel assured that they would get the kind of 'imagination' they're used too.
I love reading fiction a lot more than I care about reading any genre... despite having my own favorite genres too, don't get me wrong. My bookshelves are a mess of various type of books and genres and the only semblance of order one will find in it is how much I appreciate one author compared to the others ;)
Un crayon et du papier + une photo/scan avec le smartphone? Tu pouuras même annoter le plan, sans changer de crayon, pour le rendre encore plus informatif ;)