WWDC Keynote WiFi woes may have been due to iPhone 4 drivers

“Ladies and gentleman, put down your MiFis, and step away from your laptops, or there’s going to be trouble.”

Apple’s CEO didn’t actually utter these words during his demonstration of the iPhone 4’s higher-density screen at the Worldwide Developer’s Conference (WWDC) on Monday. But the seeming failure of the iPhone 4 to access an Apple Wi-Fi network caused a stir.

Jobs said from stage, and an Apple engineer later told InfoWorld, that over 500 WiFi networks were in operation. Not 500 WiFi adapters or connections—in an audience of 5,000, there could have been thousands—but 500 distinct networks.

Most of these networks were apparently generated by people using a MiFi, a cellular gateway from Novatel Wireless, sold by Verizon Wireless and Sprint Nextel, which relays 3G networks over WiFi.

However, after examining the video from the event and discussing it with two veteran WiFi gurus, it seems almost certain that the MiFi was only part of the problem. A flaw in the pre-release iPhone 4 iOS was clearly another element. Apple’s public relations confirmed receipt of a request for comment, but none was forthcoming.

That’s not to say that having hundreds of WiFi base stations doesn’t cause trouble. In fact, the iPhone 4’s putative driver problem likely arose from the multitude of network signals. But neither the ocean of signals nor the iPhone 4’s performance can be looked at entirely in isolation.

Checking the replay

Let’s take a close look at the video, which shows an iPhone 3GS on the left and an iPhone 4 on the right. What’s confusing in watching the video at first is that Apple had a backup set of devices cued up and ready to go. Steve Jobs’ backstage folks switch between the first set of phones and a second, and then back to the first during the first part of the referenced clip.

Here’s the replay in slow-motion:

0:06: The iPhone 3GS has mostly loaded the New York Times, but the iPhone 4 displays just a title bar, and no content on the page. A WiFi icon is in the iPhone 4’s status bar at the top.

0:17: Jobs brings up the backup set of devices, and the backup iPhone 4 is on 3G (switch to 720p in YouTube to see that more distinctly). 

0:21: You can see the 3G indicator clearly on screen here on the backup iPhone 4. 

0:28: The iPhone 4 can’t acquire a 3G connection, which isn’t surprising given the indoor location and thousands of AT&T devices in the audience. The audience laughs, and a wag suggests Jobs try Verizon. 

0:47: The demo screen fades subtly back to the original pair of devices, in which the WiFi icon is shown on the iPhone 4, but it still has not retrieved the page. 

0:59: Jobs presses the Home button, then relaunches Safari, and a WiFi icon is clearly visible, but the page doesn’t load. 

1:10: The page finally starts to load, but Jobs gives up and moves on.

So what happened here? I spoke to two people with a long career in wireless networking: Phil Belanger and Phil Kearney. Belanger’s name is on the 802.11b spec and he helped pick the brand name WiFi, while Kearney headed up Apple’s networking group (including AirPort) from 2002 to late 2008.

Neither was present at the keynote, but after seeing the video and discussing what appeared to happen, both agreed separately that the iPhone 4 was having trouble. Belanger said, “It seems more like there’s something funky with the iPhone software.” Kearney said, “My experience in the wireless space leads me to believe that there may be a bug in the firmware or the driver for the WiFi chip in the iPhone 4.” (Kearney and his group provided advice for the iPhone and iPod touch engineers, and he consulted with Apple’s WiFi chipmaker on the chip in question.)

Ten years of testing WiFi devices led me to the same conclusion. I’m the tester companies both like and hate: I’ve found hundreds of problems with adapters and drivers, which companies have (usually) fixed after I’ve reported them. This includes a number of issues uncovered with Apple’s AirPort products. I’m used to analyzing and diagnosing WiFi oddities.

How did the two Phils and I make this supposition? Let’s start with what seemed to be the problem initially, too many WiFi networks.

Mobile Hotspots Proliferate

The MiFi was the first truly mobile and widespread deployment of a mobile hotspot. It’s a battery-powered, shirt-pocket-sized device that picks up a 3G network and shares that access over WiFi with up to five other computers or gadgets.

While there were several similar gateways available before the MiFi, it was the first to combine portability with direct carrier resale to a mass audience. The MiFi requires a two-year contract and is free from Sprint or $100 from Verizon, with a $60-per-month contract plus five cents per MB for usage above 5 GB each month.

The MiFi defined the category of “mobile hotspot.” Before the MiFi, 3G-based mobile hotspots were typically limited to either tethering (using a cell phone as a kind of modem via a USB “tether” or through a Bluetooth modem profile), or using a 3G modem plugged into a laptop with Internet sharing enabled over the laptop’s WiFi card. Tethering is fine, but limited; 3G modem sharing is awkward.

A mobile hotspot creates an access point (AP) in precisely the same way as a dedicated WiFi router. This includes having a unique BSSID (Basic Service Set Identifier), which is like a MAC address for a base station, and an SSID (Service Set Identifier), the human-readable name that’s presented by software that lets you connect.

Now, we’re going to see massive numbers of mobile hotspots in the near future as phones supplant and supplement MiFis. The Palm Pre Plus and Pixi Plus offered the feature months ago, although Verizon is the only U.S. carrier to support it. And the just-out Sprint HTC Evo 4G, also includes this service, although it’s only enabled over WiMax networks right now, and Sprint will charge $30 per month for use (over 3G or 4G) starting in July.

What will really open up the floodgates, though, is Android 2.2, which includes a mobile hotspot feature in the basic operating system setup. Carriers may choose to disable or charge for this option, but it will set up a very different expectation from phones.

Oddly enough, the iPhone 4 doesn’t include a mobile hotspot option. iPhone OS 3 allowed tethering, which some carriers (not AT&T) enabled for a fee. The iOS 4 release will bring with it AT&T tethered support, but not WiFi-based sharing of a connection.

Contention in the Air

What makes the MiFi and other mobile hotspots into a congestion problem? After all, thousands of base stations can also work in concert—as at colleges, corporations, and large outdoor deployments like that of Cablevision—where centralized management varies elements such as power output and channel choice.

That’s because a set of WiFi adapters connecting to a network of base stations that expose themselves as the same network (or set of networks with virtual SSIDs) don’t contend for access. The adapters use a variety of techniques to share the available spectrum slice (a channel) allotted to their communications, with the base station acting as a mediator.

The problem with huge numbers of uncoordinated base stations is that each is trying to carve out its own use, and each (along with associated devices) has to be only mildly respectful of all other WiFi gear in the same and even adjacent channels.

Without base station coordination, each network takes a hit because it doesn’t know precisely when a device on another network will start broadcasting. Get enough of those, and every device is backing off from talking, because it’s overlapping with someone else.

Phil Belanger explained, “Rather than interference, it’s just congestion. It’s not a foreign signal, it’s well known; all the clients can decode there’s something going on.”

With enough devices, some possibly set manually to certain channels and others automatically trying to find the least-used channel, traffic at the conference center was likely spread across the entire band.

Adjacent channels produce a slightly different problem. The 2.4 GHz band, the only one in which current MiFi models and other mobile hotspots work, has just 11 legal and fully supported channels in the United States. Other countries have slightly more or fewer.

These 2.4 GHz channels center power in the middle of a roughly 20 MHz range, but are offset by 5 MHz. That is, channel 2 is only 5 MHz higher in the band than channel 1, overlapping most of the signal. Channels 1, 6, and 11 are considered mostly clear of each other, but no WiFi base station is required to use only those channels.

With enough devices, some possibly set manually to certain channels and others automatically trying to find the least-used channel, traffic at the conference center was likely spread across the entire band. WiFi devices interact differently with signals on adjacent channels, handling interaction more poorly than signals on the same channel.

The big difference between a MiFi or similar device and a dedicated WiFi router is that the mobile hotspot uses much less power by design and necessity. Because devices are designed to operate from a battery and serve small groups, the signal need only be strong enough to reach a short distance, while being weak enough to keep the battery alive for a useful period of time.

In the Moscone convention center where the WWDC keynote took place, most of the devices broadcasting were likely too weak to see each other, even if Apple’s engineers could pick up the signals using devices with a higher receive sensitivity.

However, Apple was also running public and private WiFi networks, which would broadcast at far higher power, thus interfering with mobile hotspots, as well as capturing signals from them.

Remarkably, all of this congestion should simply slow WiFi networking without keeping it from working altogether. A lot of air time is wasted bowing and retreating, but eventually conversation on each network takes place.

Belanger said, “It is my professional opinion that having that many WiFi networks would cause congestion, and the visible, noticeable experience a user would have is that whatever they were trying to do would run slower.”

Charlotte Rubin, the senior director public relations at MiFi’s maker, Novatel Wireless, said, “We haven’t heard about any people who haven’t connected via their own WiFi solution who have had issues other than, yes, it wasn’t as fast as usual because they were sharing bandwidth with an awful lot of people.”

Rubin was surprisingly candid for someone speaking on behalf of a firm making a portable hotspot, and she admitted that the company sees congestion at “every trade show we go to.” But, as she points out, “That’s the nature of WiFi and the WiFi spectrum; we see this as a very simple issue, it’s the nature of interference.”

People on the ground at WWDC confirmed that they were able to get their own connections in the keynote, or use their MiFis or other devices, despite the on-stage difficulties.

In sessions, however, congestion did get overwhelming. Mac and iPhone OS developer Craig Hockenberry of Iconfactory said via email that it was nearly impossible to use the public WiFi networks Apple had set up in session rooms because of all the mobile hotspots in use. He noted that one person in frustration set up a network called “TURN OFF YOUR F___KING MIFI!!!”

Of course, if you tried to use the public WiFi and couldn’t get a consistent connection, and you had a MiFi or Android 2.2 phone, you’d turn it on. Which would make it harder for everyone else to make a connection, leading those who could to turn their mobile hotspots on. And the vicious cycle perpetuates itself.

The iPhone 4’s Potential Driver Flaw

Let’s circle back around to the beginning now. The video seems to reveal a flaw in the iPhone 4’s ability to function in a congestion environment over WiFi.

The salient points:

• Both of the iPhone 3GS models used in the demo were able to access the WiFi network and were in close proximity to the two iPhone 4s (the original and backup units).
• The first iPhone 4 was apparently able to connect to a WiFi network, but was unable to load the page after several attempts.
• The backup iPhone 4 was unable to get onto the WiFi network at all, or was perhaps (inconceivably) not configured to join a WiFi network.
• The iPhone 4 uses an 802.11n chip in 2.4GHz, which is different than the iPad’s 802.11n radio (which works in 2.4 and 5GHz), and from the previous models of iPhone which only included 802.11g support. An 802.11n radio should have performed far better than the 802.11g radio in the iPhone 3GS. (It’s worth noting the many reports of WiFi network problems with the initial iPads, too. Apple hasn’t yet released an update promised in a support note on the company’s site.)
• The in-house network engineer who does WiFi for Apple’s events has many, many years in the field, and is known for his technical savvy. Unless Apple changed the way it sets up its event networks, the WWDC environment wasn’t set up by an outsourced IT firm; rather, it would’ve been set up by a team that would be prepared for a challenging radio frequency situation.

Phil Kearney said, “My experience in the wireless space leads me to believe that there may be a bug in the firmware or the driver for the WiFi chip in the iPhone 4.” Belanger said of what he saw of the iPhone 4 in the video, “The failure mode was different” than what typically happens with congestion or a connection problem, like “there’s something funky like having that many SSIDs.”

Kearney and Belanger are in a position to offer remote diagnosis of what happened. Both have been working to develop WiFi and wireless networking products for over 15 years. Belanger is a veteran of Aironet, Wayport, Vivato, and BelAir; he’s now a principal at Novarum, an outdoor WiFi, 3G, and 4G measurement and testing firm.

Before Kearney headed up Apple’s networking division for seven years, he worked at InterCon, Ascend, Lucent, and Agere (each firm acquired by the next). His current company creates mobile application platforms for smartphones. Kearney was Apple’s point person in developing specs with its WiFi chip vendor for the radio circuits used in the 3GS, iPad, and iPhone 4, as well.

I can imagine that there are teams of engineers working around the clock to duplicate the problem that happened in the keynote. If the analysis of what happened is correct, and a bug is found, Apple would also be checking to see how likely most users were to encounter it.

If the bug only manifests in situations with hundreds of base stations, then Apple might wait and fold the fix into a 4.0.1 update. If there is a bug that’s readily triggered, however, there will be a mad dash to re-flash devices in the production stream.

The Future

Even if a bug triggered Jobs’ keynote kerfuffle, we are all likely to encounter more of this kind of congestion in the future.

One way out is heavier use of the 5GHz band in public places and at events. I don’t know whether or not Apple’s public network used 5GHz or only 2.4GHz at the keynote, but one can imagine that we’ll see more 5 GHz in play.