Pokémon plays Twitch: How a robot got IRC running on an unmodified SNES

The highlight of last year’s Awesome Games Done Quick (AGDQ), the annual weeklong marathon of video game speedrunning, was easily the block run by TASBot (short for tool-assisted speedrun robot). The moment when that emulator-fueled robot hacked into an unmodified Super Mario World cartridge and hard-coded simple versions of Pong and Snake on the fly, using nothing but standard button inputs, was truly surprising and bewildering to behold.

This year, though, the team behind TASBot outdid themselves, showing a new level of mastery over decades-old gaming hardware. As organizer Allan Cecil (a.k.a. DwangoAC) put it during the livestreamed demonstration Sunday night, “We took over Pokémon Red. Then we took over the Super Game Boy. Then we took over the entire Super NES.”

“Then we started streaming [IRC-based] Twitch chat directly through the controller cable.”

You can watch that entire incredible demonstration in action using this Twitch archive link. If you’re hoping to fully follow how the TASBot team used a Game Boy cartridge to get arbitrary code running on a Super NES, though, we should probably back up a bit.

Super Mario Inception

Members of the TASBot team smile as Super Mario Bros. runs through Super Mario World during AGDQ Sunday night.

While TASBot is mounted to a classic NES R.O.B. unit, the key to the system is a specially designed “replay board” that can enter inputs into a console at frame-level speed (i.e. 30 or 60 times per second). The inputs being fed into the system are first developed, tested, recorded, and re-recorded on emulators, then played back to the board through a Linux box (in this case a Macbook). While TASBot can enter inputs more quickly and precisely than any human, it’s only sending standard controller signals to the console; there isn’t any hardware-level hacking or memory alteration beyond what is possible through the games and hardware as they originally existed.

As a preamble during its AGDQ block Sunday night, the TASBot team used its godlike input powers to set up a particularly hazard-filled Choco Mountain course in Mario Kart 64. After that, the team staged a modified repetition of last year’s Super Mario World demonstration, taking “total control” of the game using a series of exploits that got the system to treat the controller input buffer as raw assembly code.

This year, once total control was achieved, the team decided to code in a fully functional copy of Super Mario Bros. onto the Super NES through faster-than-human button presses, essentially writing the game to the system’s 128KB of RAM in real time at a rate of 384 bytes per frame (23 KB/s).

Adapting the original Super Mario Bros. into a ROM that would run on an SNES wasn’t a simple task. TASBot team members going by p4plus2 and Masterjun didn’t directly emulate the raw NES game, which may well have been difficult on hardware as underpowered as the SNES, but instead ported the raw engine from one platform to another line by line. In doing so, the team had to get around a number of system-level challenges, including differences in sprite tile sets that had the entire game looking puke green at one point.

There was at least one happy, effort-saving coincidence during the conversion, however. “When Masterjun later took p4plus2’s standalone .smc ROM file of SMB on an SNES and ported it to execute inside of SMW, he initially did not include the SPC initialization to write the audio data, [which] thus pointed to whatever SMW left there,” TASBot organizer Allan Cecil explained to Ars in an e-mailed interview. “The result was extremely surprising—nearly every single sound effect from SMW matched up to SMB perfectly, with some minor exceptions such as the flagpole. … it’s possible to run SMB on SNES standalone, but we did discover by pure chance that the SMW developers apparently re-used the same general [sound] mappings from SMB.”

A close up of the mess of wires that need to be fiddled with to get TASBot running.

The team also ran into some real-world problems during its live demonstration of the Super Mario Bros. takeover during AGDQ Sunday night. TASBot’s first attempt failed seemingly thanks to physical input wires that crossed over each other, messing with the precise electrical timing necessary for the in-game exploits to work. “When you’re running on actual hardware things can get a little interesting because you’re dealing with real physics in terms of electrical interference and the like,” TASBot team member micro500 explained during the demonstration. “Refrigerators mess with it.” As Cecil put it to Ars later, “The metal rolling cart acted like a giant antenna funneling EMI right at our unshielded cables.”

The second attempt left the timing for the hard-coded speedrun of Super Mario Bros. failing, and graphical glitches plaguing the game-within-a-game. Hey, even robots can make mistakes…

Struggling with the Super Game Boy

As impressive as it is to get Super Mario Bros. to run inside a copy of Super Mario World, getting an IRC channel to run inside a copy of Pokémon Red running inside a Super Game Boy running on an SNES was a decent deal more impressive and difficult to pull off.

The core of this hack comes from an existing “total control” exploit method for Pokémon Yellow, which has been known publicly in the TAS community since at least 2011. The full details of how this method works get pretty deep into the vagaries of Pokémon memory management, but the script essentially uses a precisely timed reset to corrupt the game’s memory during a save. This fools the game into thinking the player has 255 Pokémon, overflowing the memory buffer used to store that monster data and giving indirect access to the game’s main ROM memory. By carefully switching Pokémon positions and throwing out set numbers of artificially created Pokémon and items to decrement memory values, the script can essentially code whatever it wants into that ROM, then execute that code by jumping to a memory location.

While the Pokémon exploit was already well known and had been demonstrated on emulators, showing it off on real hardware live during the AGDQ event presented a number of challenges. For one, as Cecil pointed out during the AGDQ demo Sunday night, “It’s very hard to connect a bot to something without a controller port,” such as a Game Boy. By going through a Super Game Boy plugged into a Super NES, the team could use the existing TASBot controller interface, which had already been used on Super Mario World and theoretically feed in the existing Pokémon Yellow takeover script that had already been shown to work on an emulator.

But the Super Game Boy brought its own issues for TASBot. For one, the SNES add-on contains code that cancels out the input if the player presses “up” and “down” or “left” and “right” on the control pad at the same time. This isn’t an issue for standard human gameplay (where pushing opposite directions simultaneously is practically impossible at any rate), but is key for a total control exploit that uses those input combinations to represent certain key opcodes in its on-the-fly assembly programming. Getting around this required splitting the code payload into half-byte “nibbles” every frame, reducing the effective throughput for the initial code insertion to an extremely low 30 bytes/second.

To make matters worse, the Super Game Boy apparently adds a slight, almost undetectable clock skew when compared to the original Game Boy, duplicating a single frame of input once every 56 or 57 frames and messing with the precisely timed code insertion. Slight accuracy issues between the existing Super Game Boy emulation core and the actual Super Game Boy made this difference even more difficult to track down, but the team eventually found that adding a seven frame delay to its scripting fixed things. Past that, accuracy issues concerning the emulation of Pokémon Yellow forced the team to switch to Pokémon Red, recoding the total control exploit from scratch to deal with that game’s slight differences in memory management.

One of the final challenges was how to get TASBot to perform the precisely timed hardware reset that’s required to initially corrupt the memory on the Pokémon Red cartridge. This is trivial on an emulator, but a bit of a challenge in the real world, where the reset switch has to be pushed with split-second timing that would be difficult for a human to pull off and impossible for TASBot to pull off using its existing connection through the controller ports.

“My initial plan involved a solenoid from one of my pinball machines strapped to the SNES reset switch, to gently tap it at exactly the right time,” Cecil told Ars. “When I started bench testing this idea, I quickly discovered that I was far more likely to launch the reset button across the room and embed it into a wall than move it the quarter-inch we required. I eventually opted to use the reset pin on the expansion port of the SNES; a single wire, shorted to ground by the replay board, can immediately reset the console with the precision required for the save game reset glitch to work.”

IRC stage magic

The crowd in the AGDQ ballroom broke into a standing ovation when the IRC hack started up.

With all those problems out of the way, the team was finally able to reliably run arbitrary code through a real Pokémon Red cartridge plugged in to a Super Game Boy. From there, jumping to more serious control of the Super NES hardware was practically trivial. “There are a number of references in the Super Game Boy documentation that basically imply that there is no [memory] protection,” Cecil told Ars. “With what we have now we could do just about anything we want to the SGB including changing the border or triggering built-in sound effects. There’s nothing particularly of interest there, so we just bypass it altogether and go straight for the jugular by writing directly to SNES memory and having the SNES execute that instead.”

But how did the team get the live data from an IRC chat onto Super NES that doesn’t even have a modem (add-ons like XBand notwithstanding)? As Cecil puts it, that process is just a bit of “stage magic.” The IRC data is actually being read by the Macbook controlling TASBot, which converts it via a Python script into sanitized, encoded button presses that can be sent by the control board and read by a program previously inserted onto the Super NES. The code repository that makes it all possible is available on GitHub, if you care to examine the special five-bit character mapping the team used to increase their data throughput (with special codes built in that allow for those popular Twitch emoticons).