When Firefox became popular, I believe it was supported by DONATIONS. And it was used by people with a certain set of preferences.
LOL no, the money was always in getting paid for being the default search engine. The market back then was quite easy, it was Microsoft with IE in one corner and basically everyone else in the other corner. Everyone that's developed for IE6 knows it absolutely sucked and was cheering for Firefox to bring standards compliance to the web, because Firefox actually passed the ACID1/2/3 tests. All kinds of web applications, CMS systems etc. could drop all their IE hacks. So a lot of people cheered for and promoted Firefox, but the money mostly came from the Google.
And honestly, that's kinda all the plan they had. After the Microsoft monopoly was broken it was like "Ding Dong! The Witch Is Dead" and they floundered. Google decided to do their own thing with Chrome while Firefox seemed to go off on ten different side projects while ignoring their core product. And over time what might once have been acceptable degree of crashes, memory leaks and lock-ups became annoying. Having to manage a bunch of different extensions of varying quality to get functionality that was getting integrated and worked out of the box in other browsers got tedious.
I have a love-hate relationship to the death of XUL. It's ability to plug in pretty much everywhere is probably one of the leading causes for why the whole project slowed to a crawl and died on the vine. On the other hand, when they finally cut it they also cut the branch they were sitting on that occasionally made Firefox still worth using. It's no doubt that Electrolysis made Firefox a much better browser and if it had happened five years earlier it'd be a success but by 2016 you mostly ended up with a poorer version of Chrome. Today I feel it's maybe back to even but still lacking any clear benefit.
