Anon, evening. It's worth mentioning that our community isn't that small really, if you look at our Twitch community, as well as the Nico and Afreeca communities. I wouldn't suggest that popularity makes something good, but I'll try to explain why there are many people interested in these games after all these years.
1. Sword tackled a lot of specifics that I won't repeat so I'll go in a different direction here. I really like playing as the Kid. I don't think I've ever enjoyed the feel of a video game character more than him in my years of gaming, especially in the engines which improved the physics beyond the original. He's why you can almost always have a good time playing random fangames from the wiki, even if they're bad. You can call me a lazy video gamer if you like, but there's something nice about going into a fangame and not have to get used to playing as a new type of character, especially when it's a character you really enjoy playing as. That's probably one reason that gets people hooked into these games, and leads someone to say "I'm gonna download some random fangame of dubious quality" rather than scour Steam for the latest hot release in your preferred genre. You might argue that it's a more rewarding experience to master a completely new game from the ground up, but this is how I and many others like to spend our time.
Another benefit of playing a bunch of games with the same exact character and physics is the level mastery you can obtain. To master most other video games, you'd have to play it over and over again, but with fangames you can continue to develop your skills in a wide variety of games that challenge the player in many ways. This leads to there being a lot of super talented players in our community who are amazing to watch and do amazing things (not that I consider myself one of them).
Besides that, fangames are a really fun for live streams. Going back to the whole "they're fun to play even when they're bad," that's especially true when you're with people who can laugh with you at the absurdity of whatever stupid game you're trying. That's why there are a lot of popular fangame streamers. One fun thing our community likes to do is Wiki Roulette, where someone uses an RNG to pick random fangames for a night, that a bunch of streamers race while Skyping with each other. It's always a great time, even as we struggle through the most ridiculous bullshit you can imagine.
2. I acknowledge that the laziness of most fangame creators lead to them lacking interesting ideas and lacking the novelty of most "real games." That is the trade off of playing games that are mostly sculpted out of a simple engine. Also, I like games with good stories, which most (all) of these don't have.
3. Yes, but I tend to put such games down after beating them, and not return. Even though games like SMB are legendary for their great physics, I still prefer the simplicity of the Kid, and the many random adventures he is sadistically forced to go through.
4. There is an analogue to what you're talking about that I can think of: the last words in Touhou 8. For the record, I never played through that one, but I found the idea of a few really high level attacks to be done in isolation from the rest of the game, with infinite tries of course, to be a fascinating challenge.
5. Yes, but then the other one will kill you. It's really a Catch 22 situation.
To answer your follow ups, I don't see playing fangames as anything like grinding. Most fangames don't just emphasize memorizing traps, but in fact require you to develop your skill, master jump strats, learn attacks, etc., which is not monotonous at all if it's a legitimate challenge. If you watch my streams, you see the process by which I tackle a fangame challenge, and it's really enjoyable to me. In an RPG, you develop the skill of an avatar, which is boring and monotonous. And I don't have the slightest interest in sandboxes.
I'm not really sure what to say about your last question, at least in the context of fangames. You probably think "a bunch of small challenges" is what a fangame is, but in the ones I play (really high level), a single "little" challenge can take me many months. And I find that very rewarding, when I succeed.