Why didn't Nintendo want to release the original "Super Mario Bros. 2" in America in 1986?

When people in the west hear the title "Super Mario Bros. 2", they usually think of this game, which was essentially a slightly improved version of "Doki Doki Panic" (the Japanese title of this game is "Super Mario USA").
But the original Japanese "Super Mario Bros. 2" title (which is known as "Super Mario Bros.: The Lost Levels" in the west) was very similar to the original "Super Mario Bros." - it had the exactly same game engine, graphical style, music, gameplay etc - but it was much harder and could sometimes be fairly tricky and confusing;
it pretty much starts on the same difficulty as World 3 and World 4 in the original title, and then the difficulty gets more and more insane from there.
The game even has 4 bonus worlds that can be unlocked by getting to the end of the ordinary game 8 times (which is realistic, since the game keeps track of all playthroughs even after the console is shut off - obviously, those 4 special worlds are super-hard).
And this is apparently the reason why it was not released outside of Japan - it was considered "too hard".

However - as far as I know, this game was actually very popular in Japan, and was even ranked "the number one Famicom game" some time after its release.
Of course, this was probably largely thanks to the popularity of its precursor, but it really does seem like "The Lost Levels" remained very popular among Japanese gamers.
So why were Nintendo so afraid of releasing it outside of Japan if it was so successful in their own country?
What is the difference between a Japanese and a western audience in this case?

I have heard some rumours that Japanese people have a particularly strong passion for really challenging games, and have a generally higher tolerance for brutal challenges;
could this be the reason?

Short answer: Money.

Long answer:
I've been in the video game industry for over a decade and I can report that it is a very cynical, profit- and marketing-oriented business. The goal is to suck as much money out of gamers as possible, using the most efficient means as possible.

Now, let's flashback 30 years, to the time of Super Mario 2. The 8-bit Nintendo was popular both in Japan and the US. Games were sold as individual cartridges; they were effectively hardware. And they were sold in retail stores, which take their own cut of profit. Overall, the cost of production was very very high (even ignoring the cost of the software development). Thus, the risk of releasing a game that became a flop was a very hazardous economic outcome.

So, Japanese manufacturers were very careful with their plans. Japan is a reasonably sized market but the US is huge. Remember that, to sell the game, they need to manufacture the expensive cartridges. How many could they sell? That was their main concern. So they were very conservative with their decisions. As you suggest, Super Mario 2 met with critical acclaim. But does critical acclaim translate to profit? Ah. That is the central conflict of creative arts in the contemporary world.

Yes, I just read somewhere that one of the reasons why they didn't want to release the Japanese SMB2 in the west was because they thought it would be too expensive to turn a FDS disc into a cartridge, and apparently they also felt that the game was too similar to SMB1 and weren't interested in giving that particular game to the rest of the world.

Personally I really like the Japanese SMB2, though;
it may not be quite as polished as SMB1 and probably not as good as the American SMB2, but it is very stimulating if you have mastered SMB1.
I admit that some parts are quite frustrating and mildly ridiculous, like for example the parts where you have to jump on a series of vertically flying Koopa Paratroopas in order to clear certain pits, or where you have to catapult far above the top of the screen to clear huge chasms and then estimate where you end up when you fall back down.
However, all of those parts are at least reasonably fair in that they can be mastered enough to beat consistently, and they aren't anyhere near as unfair and extreme as things like that Kaizô Mario World hack, for example.

Video games are 99% business and 1% art. The same is true for many crafts. The art percentage is antithetical to pop culture and consumerism. Somehow, it all works out. Companies sell their games. A few creative people navigate the bullshit to express their art. The world keeps turning.