Withholding the "real" Super Mario Bros 2 from the Western Market due to its difficulty and instead releasing a character swapped clone of Yume Kōjō: Doki Doki Panic was indeed an inexplicable act on the part of Nintendo--for several reasons.
In retrospect, the notion that "this game is too difficult for Westerners" is rather bizarre (and not to mention coddling and condescending) and contradicts the notion of "Nintendo hard" adopted by numerous other games of the time, i.e. the numerous far more difficult NES games that were released to the Western Market: Silver Surfer, BattleToads (its hover-bike levels in particular), Ninja Gaiden, Ghosts N' Goblins, Teenage Mutant Ninja Turtles (its timed, electrified underwater bombing diffusing level in particular), The Adventures of Bayou Billy, etc. Making such games so difficult was often one the major goals in their development, to keep people challenged by them, not tired of them, and re-renting them in the rent driven video game market of the time. (That principle makes a far amount of sense, especially in light of the opposite: far too easy games like Kirby in Dreamland that could be mastered in a day often seem like a waste of money in retrospect, given the lack of challenge and re-play value!)
As for why the "real" Super Mario Bros 2 was so difficult in the first place, especially when compared to the other Mario games: I can see why they would up the challenge in the first, and must assume that they decided it was a bit of a miss-step during Super Mario Bros 3's development. A difficulty setting may have been a better approach that would have satisfied the broadest audience, letting them tailor the challenge to their level--but perhaps that was beyond the cartridges' limited ROM.
Also, why take another, completely unrelated and quite different series unknown to the Western Market and replace its protagonists with Mario and co? Among other things, it robs Yume Kōjō of the potential of itself being ported to the Western Market, unaltered--people would just dismiss it as a weird clone of Super Mario Bros 2, rather than what is in truth the reverse! Say what you well about Square releasing Final Fantasy VI as Final Fantasy III in the Western Market (disguising the fact that there were many other Final Fantasy games that we weren't receiving), but at least was in truth still a Final Fantasy game!