Sarah Edwards: Poking the Bear - Teasing out Apple’s Secrets Through Dynamic Forensic Testing and Analysis

Sarah Edwards speaking at the Jailbreak Brewing Company Security Summit on Friday, October 11, 2019. 

If I come across a useful piece of data on macOS or iOS I do not just assume I know what it means - especially if my whole case depends on it. My experience with Apple data is that it is consistently inconsistent. They certainly do some questionable things. Testing is the only way to get that warm fuzzy feeling that the awesome piece of data you found truly means what you think it means. Yes, testing takes time. Yes, testing can be tedious. However, testing can make or break cases. This talk will go through my testing processes on Mac and IOS platforms to show that sometimes a quick test really is a quick test. A 30 second test may be well worth the investment in the long run. I will also show how more intensive testing can be implemented to tease out the strange oddities of native and 3rd party data stored in various SQLite databases using some of my APOLLO modules as examples.

(Source: Jailbreak Brewing Company)

Transcript

Sarah Edwards: [00:00:46:22] This is Poking the Bear: Teasing out Apple's Secrets Through Dynamic Forensic Analysis. We do have a reverse engineering theme going on here, if anybody's already caught that? I am not a reverse engineer. I can find strings in IDA, or Hopper, or some other things, but that's about it. Maybe figure out a function, here and there, but frankly I'm a forensics person - straight up. I do a little bit of reverse engineering, but I also do a lot of dynamic analysis. I'm specifically a Mac person. I don't touch Windows. I actually try to actively avoid Windows, so I'm going to go through some Mac and iOS stuff here, kind of focusing on iOS, because it does tend to be a little bit more difficult.

Sarah Edwards: [00:01:30:20] My contact information is up here and if you want a copy of the presentation it's going to be uploaded to my website in a couple of weeks. I'm presenting this later at BsidesNola, so I really only go to conferences where there's booze, at least in the month of October. So, Bear Essentials - what am I going to be talking about? I'm going to do a presentation of stuff that I do, on a daily basis. I do forensics testing. I do research. I do development. I want to test how things work on Mac devices. I do a little bit of Android as well, but this one's mostly focusing on iOS and Mac OS.

Sarah Edwards: [00:02:10:20] Why do testing? It sounds boring! I understand. But you want to be as accurate as possible. When you're doing forensics, whether it's for legal purposes, whether it's for intelligence, whether it's just for reverse engineering, you want to be accurate. There has been a lot of things that I've seen, specifically on Mac and iOS devices that just don't make sense. Apple does not put the most logical sense into some of their data files, and I'll show you some examples of that as we go along.

Sarah Edwards: [00:02:37:19] So, as far as testing goes - it doesn't have to take a long time. You want to test out where one toggle switch is being stored in the Plist file somewhere? It can take approximately 30 seconds to figure out what Plist file that actually it. It doesn't have to be difficult. It doesn't have to use advanced tools. I'm using free and open source tools. I didn't pay for anything, other than the Apple devices, of course, which comes with that Mac tax. It also doesn't have to be expensive. This is pretty bare bones - quick and dirty testing and I want to show some examples of that.

Sarah Edwards: [00:03:10:05] I tell people, students - I also teach full-time, just a little background - government contractors, big surprise in this area. I'm in Virginia, for the most part. The only way they get me to come to Maryland is for me to present at a conference, because it's so far away. My second full-time job is teaching for the SANS Corporation. I teach Mac and iOS forensic analysis for them. So I am constantly doing this sort of testing; Mac, iOS devices - it doesn't matter, it's pretty much the same. The iOS set up is a little bit more difficult, but I'm going to go through that, so everybody in here could pretty much get set up with the basic amount of tools to do this type of testing.

Sarah Edwards: [00:03:51:08] As far as the scope goes. I'm going to go through a little bit of the device prep. I'm going to go through how to search for different applications. One of the things that I do for day work is I do application tear downs. I'm looking at things like Telegram - I want to know where that data's stored. How is it being stored? Is it actually encrypted on disk? Is it not encrypted? Can I find keys? Can I do this? Can I do that? I am doing a lot of different research. I also do tend to look at a lot of the Native stuff. I have a tool, that I'll talk a little bit about later on, called Apollo - the Apple Pattern Of Life Lazy Outputter. I think it's actually a really good name - I'm very proud of that. It's a pattern of life tool. It's getting really creepy, into everything that I do on iOS devices. That might also include your iOS devices, if you hand them to me.

Sarah Edwards: [00:04:37:08] We're going to get into some of the how I do some of the research to come up with the different Apollo modules. How I do SQLite analysis. How I do Plist analysis and how I put all the different pieces together.

Sarah Edwards: [00:04:49:16] A little bit about the prep. First off, let's talk about Macs. As far as prep goes, for some quick testing, you've probably already done this. You've probably created a VM. If you had no idea that you could create VMs out of Mac systems, you absolutely can. You can have two virtualized instances. It has to be on Mac hardware. Can you not have it on Mac hardware? Yes! But I'm not supposed to tell you that! There's VM hackintoshes that are very much out there. But all the different tools that you might have used before: Fusion, VMware. Parallels, VirtualBox, if you've got some extra patience - but it is free, so you don't have to pay for it. A couple of recommendations that I would make. You've probably done some malware analysis before? Set up a VM? You have certain tools that you put on there. You create snapshots. A few of the other Mac specific things.

Sarah Edwards: [00:05:38:19] When you're doing some of the testing that I'm going to show you, especially on the Mac side, you probably want to turn of SIP, System Integrity Protection. It's only going to cause you trouble and frustration. Get into recovery mode, CSR Utility Disable. Maybe make some clean snapshots, from a VMware standpoint or if you are just doing it on a live system, I do plenty of quick and dirty analysis on my own live system. I might do a tmutil snapshot. So it basically takes a copy of the file system metadata and stores that off within the file system. Then we'll go down a little bunch of binary. I'm pretty much going to use the same binaries that I do on iOS, as I do on Mac OS so I'm going to save some of the preparation for that.

Sarah Edwards: [00:06:22:10] The iOS side - this is where it gets to be a bit trickier. You need a jailbroken device, and that's hard sometimes, so it's all about finding one. I have gone to the random Best Buy, the random store, and picked a device that has the most dust on it, all the way back on the shelf there, to find a device that is actually jailbreakable. It's not just a random one you can pick up at the Apple Store - it's a little bit more hard to find a jailbreakable device. It's very dependent on the hardware. It is very dependent on the iOS release. It's very dependent on the point release of the iOS version running on the device.

Sarah Edwards: [00:07:00:11] If I got to the Apple store, right now, pick up an iPhone, that's got 13.1.2, if we're at that now, but that is not jailbreakable. I can get that device, and I can basically just hold onto it, not update it, until a jailbreak actually exists. Will there be one available? Maybe, but maybe not. So I actually keep a library of different devices, on different point levels, and some of them are jailbroken, some of them are not. Sometimes I'm just waiting, or updating and hoping that they're actually jailbreakable.

Sarah Edwards: [00:07:31:17] I've got an example here of one of my go-to websites to figure out, I have this device. What can I do with it? It is iphonewiki.com. This is a nice matrix of what devices, on what point levels, can actually be jailbroken. An example that I'm using here - this one is an iPhone 6. It is currently on 12.4. 12.4 got jailbroken a few weeks ago, and it was currently on 12.4 - that almost never happens. I went and just basically jailbroke everything that I could get my hands on. That's a magical jailbreaking moment. It's something you don't lose, right there.

Sarah Edwards: [00:08:07:04] Another issue - I get a lot of folks saying, "Hey, can you downgrade?" No! Not very easily, anyway. Maybe if you save SHSH blobs, and all that stuff. But if you just have a device, in your hands right now, you cannot downgrade - unless it is currently being signed by Apple. This screenshot was taken a couple of days ago, from my iPhone 6. The current version, that I could potentially downgrade to, if I was at iOS 13, would be 12.4.2, which you'll notice they actually tell you, you cannot jailbreak that at this point in time. So it's a nice place to keep an eye out.

Sarah Edwards: [00:08:42:21] This is why I do keep a library of devices, on different point levels - you have to get it at the exact moment. And then, once you do jailbreak, you pray that you don't upgrade it, because there is nothing worse than losing your jailbreak. I've done it before, many different times. I've got one here - it's on 12.4. It's going to stay on 12.4, probably for the lifetime of it. I've got an iPhone 6, it is a little bit older, you can't even install 13 on it. So this one's going to be, hopefully, permanently jailbroken - unless I accidentally upgrade it to 12.4.2. Here's hoping I do not!

Sarah Edwards: [00:09:15:11] This website is another one of my go-tos. This is ipsw.me. This is going to show you what the sign in window is, for all of these different devices.

Sarah Edwards: [00:09:24:17] The next thing is to actually jailbreak. Some folks ask me, what's my favorite jailbreak? I'm like, whichever one works for my device! I am not here to be picky. There's currently two jailbreaks out for the iOS 12.4 devices: Chimera and Uncover. Most of these public jailbreaks - they should be free. I have gone, and I have downloaded some random jailbreaks - just doing a 12.4 iOS 6 jailbreak, or iPhone 6 jailbreak. You're going to get some shady links. You want to be absolutely sure you're downloading a real jailbreak, or it might come with a little something extra. I've done this a couple of times in the past. I've now cautioned myself on this, plenty of times. I will usually use the links off of jailbreakwiki.com, to find a legitimate site to get the files that I need to use.

Sarah Edwards: [00:10:16:03] A good indicator, that you downloaded not a legit version - if they're asking you to donate money to their Paypal account, or through Bitcoin, that is usually pretty obvious. They are not going to ask you for money. All the public jailbreaks, right now, should not ask you for money. Maybe, donations here, tag with this hashtag, and all that stuff, but not say, hey, send me money to this Bitcoin address.

Sarah Edwards: [00:10:40:09] Other things: follow the directions. Every jailbreak is a little bit different. Sometimes you need WiFi on. Sometimes off. Sometimes Find my iPhone needs to be on, or off. Whatever they're exploiting, you need the perfect scenario to get those exploits to work. So read the directions. Most of the time the different jailbreaks, as soon as they come out, there's some person doing YouTube videos, or online instructions, with blogs and stuff. Certainly read through a couple of those - be able to expect some of the problems, some of the issues that might come into play here.

Sarah Edwards: [00:11:10:12] Once you do that, and most of the more modern jailbreaks, about iOS 10, 11, and 12, are the semi-untethered jailbreaks, which means you're downloading an IPA file, or an app file, and side loading that to the device that you are going to jailbreak. You're likely going to do this with Cydia Impactor, from Saurik. I've got an example on the screen here. You'll want to have some sort of developer credentials. It depends on if you want to pay $99, or not. If you are going to be jailbreaking things, all the time, and I literally jailbreak anything I can get my hands on, it is absolutely worth the $99 for the paid developer creds. If you just have unpaid credentials you'll have to re-jailbreak or re-sign this every week, or so, and that's extremely frustrating.

Sarah Edwards: [00:11:54:09] I deal with these devices all day long. I don't want to have to go through, every week, and re-sideload my jailbreak. If you have a paid developer account, it's 365 days. At that point you probably already have the jailbreak, or maybe a new jailbreak has come out, you've updated your device. So it does save you a little bit of time. $99 can go quite well.

Sarah Edwards: [00:12:15:22] Next up, jailbreaking. So you sideload this application - I'm using Chimera. Just figure out which ones you like. I could use Uncover. I like the pretty colors, honestly, in Chimera - it's seriously the only reason I'm using it here as an example. Click on the app. Click Chimera. There is a big jailbreak button. It's pretty obvious what I need to do to jailbreak it. Now each different jailbreak might have a few different options. If you're into the tweaking, and all that stuff - that's a jailbreak thing, not a drug thing, I promise - I don't really get into all of that. Just give me that access to the device. But make sure that you read ahead. Maybe some of those things might be helpful for you. I don't really get into that.

Sarah Edwards: [00:13:00:02] Press the big jailbreak button, and hope for the best. Cross your fingers. Some jailbreaks are more stable than others. Some exploits are more stable than others. I've had some, on some devices, work every single time - first time. I've had others where I'm clicking that jailbreak button about 30 different times and it's rebooting every single time. It's a little frustrating. So, you find one that works for you, for your particular device - stick with it, until you need another jailbreak for whatever purpose you might need. For this example I am using Chimera.

Sarah Edwards: [00:13:35:20] Next up, let's talk about SSH. So most of the new semi-untethered jailbreaks are coming with some sort of SSH server - whether it's OpenSSH, whether it's Dropbear - I don't particular care, just have it installed for me. Some of the older jailbreaks, and potentially some of the new ones coming out, they may or may not actually have a built-in SSH server to the system. So you may have to install it. The old school way of doing it is through Cydia. Sileo is the new Cydia, give or take, so you might have to do it through them. Just keep your options open. Again, not every jailbreak is the same, so you have to work with it.

Sarah Edwards: [00:14:08:23] SSH may not actually be on Port 22. There's one jailbreak, I want to say it was for an iOS 10, or 11 device, and I honestly can't remember which one, but that one ran on 22.22. I was like, why can't I get into my device? Over and over and over. I was like, man, what is going on here? I was actually port scanning my device, to find what SSH port it was. Now, had I read the documentation for this particular jailbreak, just scrolling down a little bit, it says, oh, we run Dropbear in port 22.22. I'm like, oh, yeah, sure. RTFM, I kind of failed that one.

Sarah Edwards: [00:14:42:18] First thing I'd recommend doing, after you jailbreak - change the passwords. What's the default password for all iOS devices? Alpine. It's been the same password since 2007. So people know the passwords to these things. So as soon as you jailbreak it, I can log into your device, assuming that's it online, route, mobile, Alpine, and mess up your device. So if you are doing this online, and you don't necessarily have to do this online, you absolutely want to change that password.

Sarah Edwards: [00:15:14:05] You want to remember the password. I've run into some devices that I think I've changed the password, but I don't ever document the password. If it's a test device, change it to just no name password, whatever, throw it on the back of a device with a label maker. That makes your life a little easier. Or use a standard password across all your devices. Not your actual daily driver device, but all of your test devices. I ran into one incidence, where I forgot the password, and I tried every password I thought I would ever put on a device. Turns out it was just Alpine. Yeah, I never actually changed that. So, do as I say, not as I do.

Sarah Edwards: [00:15:50:20] It's not impossible to get device back, once you have SSH access to it. You'll notice that this particular device has an application FileZilla SCP, or something like that, where you can get access to the master password. You can actually read it. You can put the different password in there, just in case you ever need to get back onto the device. Don't forget the password. It's not impossible to get back, but it makes your life a little bit difficult.

Sarah Edwards: [00:16:17:09] Next up is getting access to that. Now I mentioned you can do this over WiFi. If you have an access point, you're connect it, SSH 192.168, whatever your IP address is. I don't tend to do that. I do work in environments where there's not a whole lot of wireless stuff going on. I'd use iproxy to get access to that - libimobiledevice - it's a suite of tools. Highly recommend it. Again, it's free. I do a Brew install, libimobile device. And I'm using iprocess to be tethered to it, through lightning cable, here. So just USB access. So that's where that -p4242 comes into play - that is me setting up the port for iproxy, to tether it over my lightning cable.

Sarah Edwards: [00:16:56:00] Now we get onto the device. We need to start putting stuff on there. Different jailbreaks will put different tools in there. Your basic install of iOS does not have a lot of your normal Unix binaries that you have been accustomed to actually using. So Jonathan Levin, author of all these fantastic books, which I highly recommend purchasing, he's created the binpack. Binpack is probably one of my favorite suite of tools. It's got most of the Unix utilities that you would expect to have on a normal Unix-based system. You plop that on the device, over SCP, however you want to get on to the device. A few of the tools that I'm going to be using are some of his own: Jtool and JUtil, as well as some of the other ones, fs_usage, xxd, and SQLite, to get some of my analysis done.

Sarah Edwards: [00:17:42:10] There's quite a few more utilities on there, so I highly recommend perusing that particular website and seeing what you can use in your analysis. A couple of other third party tools. I use cda and fsmon, on a daily basis. cda, I don't even build it - I just extract it from the dead file that they provide on GitHub. Open that up. I use the N-Archiver to open up dead files, and plop that cda binary over to my device. Fsmon, on the other hand, is a little bit more cross-platform, so I use this on Mac, Linux, Android, iOS - pretty much anything with a command line you can use fsmon. You can either build your own or they do have the pre-compiled binaries as well. So they have one for all those different platforms, so just download and use them as needed.

Sarah Edwards: [00:18:41:02] Now there's a little bit more prep - you can't just take that binary, throw it on your device. You'll notice in the screenshot here that cda and fsmon are both Fat binaries - Fat Mach-O Universal Binaries, meaning they have multiple architectures embedded into them. I have found that thinning them to just arm64, assuming that you're on an arm64 device, everything 5S and newer, that you should just use that small one. So you keep the footprint a little bit smaller and it also just tends to be a little bit easier to use once you're on the device itself.

Sarah Edwards: [00:19:11:03] So we've got a couple of Fat binaries - we need to thin them. How do you thin something? Lipo. Oh Apple, you're so funny! So lipo is built into the Mac system, but I give you a couple of different tools here. Pretty much doing the same thing - lipo is Apple's version of this. Jtool is Jonathan Levin's version of this. Lipo, mostly just on Mac OS, versus Jtool, which you can do on either the device itself. So if you've already loaded it onto the device, you can already thin it on that iOS device.

Sarah Edwards: [00:19:46:13] A couple of examples of that. On the top there we have lipo running on the Mac itself. I don't know if you can do this on a Windows Box? I don't also care if you can do this on a Windows Box, to be perfectly honest with you. But I'm just using this on cda. I'm thinning it out to the arm64, and I'm calling it a cda64. So once you output it, I do tend to call it either 64, or non-64, so I know which one it is and I'll load onto the device in just a bit. Second one there, on the screenshot at the bottom, is using jtool. The flags are a little bit different, but again I'm extracting the 64 bit version of this. It's being called .arch._arm64. So I will likely rename that one, to make usage a little bit easier.

Sarah Edwards: [00:20:36:22] SCP these to the device. Again, that -P4242 is from iproxy. If you are not using iproxy you don't need to specify the ports. I load them right into the root directory. So var route, drop them there. However they can't be run from there. So, a couple of errors that you might come across. If you seen operation not permitted, very likely it's because the executable is in a location where it can't be executed. You probably want to throw that the system partition - iOS is made up of two partitions: system and data. We now have that on Catalina - so that's fun. I usually throw them into /bin, /sbin, whatever your favorite bin directory is. I don't think it actually matters, but some old-school Unix neckbeardsmight freak out, how I deal with that. I don't care, as long as it works.

Sarah Edwards: [00:21:29:02] If you have permission denied - check the permissions. When you upload them some of the permissions may not be executable. So I want to show you a couple of these errors, because some of the times I get these errors. I'm like, what the hell is going on here? You want to absolutely make sure that they are in the proper place, with the proper permissions, to actually be able to run. I move cda and fsmon to my /bin directory.

Sarah Edwards: [00:21:53:06] Next thing, entitlements, code signing, all sorts of fun stuff. Theoretically, you're not supposed to be able to run any unsigned code or unentitled code onto iOS devices, but our fun jailbreakers have made that possible for us, which makes me even happier, so I don't have to figure out how to do that.

Sarah Edwards: [00:22:10:16] The first thing I usually do is just test it. Does it just work? You know, Apple, it just works. Sometimes it just works. Fsmon, in this case, because I downloaded the precompiled iOS binary, it also had the proper entitlements and those are highlighted in the other screenshot. So it's got the platform application entitlement on there. I don't have to add it, I don't have to sign it, it's good to go, as it is. And I am using jtool --sig --ent, to actually get that information out there. So jtool is being on the device itself.

Sarah Edwards: [00:22:43:14] Next up, cda. This is what you might find, if you do have to add entitlements and signature. You're going to get killed. It just kills it. So I'm doing jtool -sig - ent, and I want to see if it does have the proper signature and entitlements. This is an example of one that does not. So I do have to do a little bit additional work on here. A couple more command lines, I want to extract the entitlements from a working binary. In this case, I just chose fsmon. You can pretty much just choose anything in /bin. And I save them off to my root home directory, as ent.xml. It's nice to just have that waiting and available if you just need to add other binaries to the device.

Sarah Edwards: [00:23:26:01] Second command line - I'm using -sign inplace ent, provided those entitlement xml file and I'm providing it the cda binary. So this is going to do it's thing. If I try to run it, it still gets killed. This is where it can get a little bit tricky, did the command work, or did it not work? If you use --sig and ent, if I go back one, it should look a little bit something more like fsmon here. But if you do have those, you see them there - just try rebooting the device. A lot of the code signing, and cd# stuff, and trust cache, and all that good stuff, kind of needs to work into play there. I'm certain there's more people who are more informant of that in this particular conference here - that is not me. I'm just trying to get this stuff to work. I find a good old reboot and re-jailbreak does tend to get that to work again.

Sarah Edwards: [00:24:18:12] You might have some SSH stuff here and sometimes SSH isn't the most favorable process, so you might have to give it a couple of tries. Finally, right at the bottom there, in the purple, run cda, and it shows me the help output for that. Alright, so good to go on those particular things.

Sarah Edwards: [00:24:35:17] So Mac versus iOS set up. IOS, little bit trickier. This is why I do tend to keep my devices around, unjailbroken, already prepped and ready to go, whenever I need them for my analysis testing. How do I do this analysis? I might do some demos, we'll see if I've got time, but let's see how this goes.

Sarah Edwards: [00:24:57:15] First things first, I'm going to use my cda binary. This one, I'm only doing on iOS - there really isn't a comparative tool on Mac OS. This is going to help me find certain applications. So if I'm doing a third party app tear down, say Telegram or WhatsApp, or something, I need to know where to look in the file path. Now if you've ever looked on physical iOS devices, it's all over the place. You get these private var containers and mobile containers, data shared, shared app group, these gooids - these are not standard. These gooids are assigned at install time, or upgrade time. So this is not always going to be the path for WhatsApp - not on my device, not on your device, not on somebody else's device. They are dynamically-generated.

Sarah Edwards: [00:25:40:12] So I need to find these paths. Now the ones I'm mostly interested in, for me, unless I was doing reverse engineering, I usually don't bother with the bundle ID there. You'll have your encrypted binaries in there, so you can potentially decrypt those. I'm interested in the data and group ones. So this is where my application data's going to be stored: my contacts, my pictures, my media, my messages, all that useful third party data that I want to take a look at. Now, WhatsApp is a really good example of a group container. So WhatsApp is owned by Facebook. If I am looking for WhatsApp specific data, I may want to also look at the Facebook data. So I can share data between the two applications. We do see those shared app groups in there as well - one for Facebook and one for WhatsApp.

Sarah Edwards: [00:26:26:10] So if you're looking for third party app data, don't just look for emobile containers data application. Sometimes, depending on developer, they might throw it into another directory, and WhatsApp is particularly one of those that tends to throw the stuff all around in different directories. A few other examples here: Wegmans, best grocery store on the planet. I know I'm in the right area here for that. We've got the bundle and the data. That one is just storing its application data. If you need to look at Wegmans data, I don't know, somebody might, you'll find that it's just being stored in this one path here. And, of course, you can also do the native stuff. So it's not just about third parties. You might want to look at some native applications as well.

Sarah Edwards: [00:27:09:19] The photos application has a weird bundle ID. So bundle IDs are very important. And when we're looking at some of these different pieces of data, to find the data that's associated, photos is com.apple.mobileslideshow. Sure - makes perfect sense. This is a great way of being able to find the bundle IDs for that. On the Mac side, finding the data's a little bit more tricky. Just using the Find command is usually my go-to. It's the most efficient way for me to be able to do that. But it's all about knowing the application name or the bundle ID for that application. I'm using Signal here, as just an example. Fortunately, the word signal is actually in the bundle ID. Sometimes it's not, depending on which application you're looking for.

Sarah Edwards: [00:27:55:17] So I'm really just doing a find here in my library directory where it's most likely going to be stored, for signal, that's going to catch on as the signal directory, in application support, as well as anything that potentially has that bundle ID in there. A lot of the data on Mac and iOS does have the bundle ID stored in the storage path, so it's a pretty good way of finding the associated directories and files.

Sarah Edwards: [00:28:20:18] Next thing we have is actually getting into file system monitoring. I'm dynamically looking at this. I will install an application, Telegram, Signal, whatever my application of interest might be, and I am poking around there, sending messages, making calls, taking pictures, doing whatever I need to do to figure out where is that data being stored on the device itself? From a forensics perspective I need to know where can I pull this particular data from? How is it being stored? Where do I look for it? So fsmon is my initial triage, lower go-to, file system monitor. It's free. It's from Now Secure. It's multi-platform. It's awesome. I use this on my Android devices too. I'm not going to talk about Android here, but a lot of the same techniques are used for that as well.

Sarah Edwards: [00:29:09:17] I have an example here. It's probably somewhat tricky to read, if you're not the first group here in the front, but this is me taking a picture. It does say, taking photo. You see the DCIMdirectory, some of the misc incoming, some of the iCloud related stuff here, some photo data. And all of sudden down here, you see private/var/mobile/media/dcim/100apple/image_0065.jpg. This is actually everything that's creating the jpeg picture. So I like to use this, because this is potentially another place where I can find other forensic artifacts of specific events on a device. What I often do, Mac or iOS, is I stop and start fsmon hundreds of times. It's pretty verbose. You want to capture that action very specifically.

Sarah Edwards: [00:29:55:06] I might run it for two seconds, toggle something, maybe wait a few more seconds just to make sure it writes to a file, or something like that, and then stop that and try to look at the file paths to see where does it make sense for this particular action to store data associated to it?

Sarah Edwards: [00:30:11:21] Mac OS pretty much looks the same way - it's cross-platform. But you do have to run Sudo, so make sure that you are running with an admin account, to be able to do this. I also notice that there is a lot of extraneous stuff going on here. So this is looking at the fs events, activity, the records, with all the file system transactions happening on this particular file system. You're not going to get just the stuff that you're looking for, you're going to get a lot more as well. So if you've got a lot of other applications running in the background, try to kill off those as much as you can just so you can focus on the actions of interest.

Sarah Edwards: [00:30:50:15] Very often you're going to see a lot of log D stuff - that's the system unified, or the Mac unified logging, that was introduced a couple of versions ago. It is extremely loud and verbose and you're always going to see it there. So people are always like, what is this trace V3 thing? This is the malware, right? I'm like, no, that's just logging. So this one is just an example of me creating a file: test.text in my home directory. So you see file creates, file deletes, also very useful for reverse engineering malware.

Sarah Edwards: [00:31:22:01] If you do need a little bit more detail, this gets extremely verbose. This is fsusage. This is going to be pretty much every system call that's happening on the device. Again, I would not keep running this for about a day because you're going to have a lot of output to go through. You're running this in very incremental stages, testing various pieces of different actions on these particular devices. So this one is another example of me just taking a picture, but it shows every system call on the system and what file, what's being written, what's being outputted - all that good stuff. So if you do need a little bite more detail, I do recommend using fsusage.

Sarah Edwards: [00:32:03:07] On the Mac side, it just comes in handy, Watch. Not installed by default, but just do a Brew install Watch. I tend to use this for monitoring specific directories. I want to see if I'm doing something on the device, sending a message. I want to see which databases it writes to or where the media is or whatever action I might be performing on the device. Watch is nice and useful and if I just run LS/LA, I'll see the file system right times be updated and that's going to help me target. I want to look at that database versus the ten others that I really don't care about. So it does help me target specific files.

Sarah Edwards: [00:32:42:19] Finally, we have plist files. We get a lot of the files. We're going to try to go into these app directories and we want to find which application of interest and we want to find different plist files and databases. Now we need to be able to look at these. So if you've ever dealt with plist files before, there's XML plist files and then there's binary plist files. It's the binary ones that cause a lot of folks trouble. They tend to open those up in a non-binary plist viewer like Text Edit or Notepad++, and, yes, you can see some strings in there and some other stuff, but you don't get the context of it. You really need a good viewer for that.

Sarah Edwards: [00:33:18:02] If you're on the iOS device itself, I use jlutil, another utility from Jonathan Levin. That's part of his binpack to actually output this to a somewhat readable format. This one is just an example from Wegmans. It's one of the configuration plist files. All of this is test data. It's not my own personal data, just for the record.

Sarah Edwards: [00:33:39:05] On the Mac side I like to use Xcode or plutil. Xcode has got a great built in plist viewer. Plutil's more or less on the command line, so -p for print. Plutil-p, this one is for 0xED. It's a GUI-based text editor showing me the contents of that particular plist file. Why look at plist files, especially for applications? You're gonna find that there's usernames, there's passwords, there's server, there's key information - there's all sorts of juicy stuff depending on which application's data you're looking at. Hex editor, you know, who cares? But you can tell which root directory I was in, when I was last using that hex editor. That's for my Protobuf Blog, that I just released about a week ago.

Sarah Edwards: [00:34:20:07] Next up, SQLite. This can be a little bit tricky. Databases are consistently inconsistent. But there are tiny databases, one, two tables, maybe a few columns in each one and then there's gigantic databases - we're talking about the power log database. That one has, I don't know, 150 different tables. Some of those tables have 100 different columns in there. That's when, maybe doing command line stuff, it's a little bit more difficult to take a look at that. So I kind of bounce between command line, SQLite 3, from the binpack, or GUI, using DB browser for SQLite. I like it. It's free. It's open source and it is fairly cross-platform. I'll show you an example of that in just a little bit.

Sarah Edwards: [00:35:04:11] I'm going to break out into a little demo here. If it works. Here's hoping. So I've got my iOS device here, and it is tethered. If you ever need it for a presentation, quicktime, so I'm not doing this to the audience - that never works out really well. I am jailbroken. Hopefully it hasn't updated in the time that I've been talking. I am running iproxy here - iproxy 4242, and then port 22, to tether it to my lightning cable. Test the device. SSH into it.

Sarah Edwards: [00:36:08:11] So we're in the device. I want to run this script. I don't actually memorize any of these paths, ever. So this is the KnowledgeC database. KnowledgeC database is keeping track of a ton of information about your activity on your iOS device. Actually, your Mac too, but this one is specifically for iOS. I've got my KnowledgeC database here, it's a SQLite database. I'm just going to run SQLite 3. I extracted SQLite 3 from the binpack. Just get into SQLite viewer. So, show me the tables. Now some databases are a little bit tricker than others. I'm looking for some stuff in here. I know Z object and Z source are probably two of my main ones. Run a quick query.

Sarah Edwards: [00:37:31:07] That's not pretty. I can go up here, there's a lot of data here. There's quite a few columns in this particular database and this may not be the best way of doing this. It's good for an initial triage. I will often do this on my own stuff if I'm looking at a new application that I just installed. But I might want to look at the headers here. So I'm going to turn headers on, so the column names, and I'm also going to turn it to move column. Re-run this. So it still looks like crap. Make it teeny-tiny - you can't read this, but no problem. It's going to format it a little bit better. This is much better done on a giant monitor at work, not so much as a demo. Just a few different things in here, that we can try to make this pretty.

Sarah Edwards: [00:38:24:12] So we zoom back in. I have some pre-built queries for this particular one. I've done quite a bit of research into this database for my Apollo project. This is one of my Apollo projects - Apple Pattern Of Life Lazy Outputter. It doesn't even have to be Apple stuff, but that really works with a great name. So, iOS or Mac, don't have a lot of support for Mac right now, but this is going to contain a bunch of different modules. So this is more or less a crappy Python script, that takes a bunch of SQLite databases, and performs a bunch of different queries. Outputs that into a csv or a SQLite database for analysis. So it's more or less a way for me to not do 100 different queries on 20 different databases but able to extract all of that data and correlate it across one single database. It's really good,

Sarah Edwards: [00:39:15:01] A lot of these different databases hold some interesting data points. Some of my Apollo databases, if I run them across the whole iOS dump, we're talking about a few million, possibly into the tens of millions different datapoints. Everything from location to data usage, to health information, all sorts of stuff. It really depends on what you're pointing it to. So I don't even know how many things I have anymore, but if I just scroll through here. If you've got your health database on your iOS, I can tell you what your heart rate was five years ago. That's not creepy. I can see your workout information. I can see when you're lazy or not.

Sarah Edwards: [00:39:57:10] Your knowledge stuff - so this is a lot of your application usage. I'm going to specifically call out Knowledge app in focus. So this is the one that I'm going to use as an example. This is one of the modules, it's got some metadata information that shows me which versions I can run it on, the time stamp that I'm using to key off of. So it throws it into the database, so I can sort by time, so I know exactly when you were doing anything on your device. And this is my query. I make a lot of times very specific queries for very specific things, for the Apollo project. This is a query that extracts what application was in focus on your iOS device. This is a great one to see, when were they using Facebook? How long were they on Twitter for? All that great information.

Sarah Edwards: [00:40:42:22] I'm going to use this one. I've already uploaded it to the device. I copied this, saved it as a .sql file and thrown it into my home directory on this iOS device here. So it makes this output quite a bit prettier. Let me do SQLite 3, KnowledgeC... It doesn't have the headers, it doesn't have the formatting, I can change that up a little bit. But it does show me things like the bundle ID, some time stamps.

Sarah Edwards: [00:41:43:19] This is, more or less, the bundle ID of the application in use, preferences, Safari. So the ones at the bottom here, I'm sorting this by time, those are the ones that have been most frequently used. I just checked Chimera, just to show you that it was jailbroken. This column over here, this is number of seconds being used. There's a start time and an end time here, these two time stamps, I'm calculating these out within the query to show me how long it's actually being used in seconds. For some reason, KnowledgeC also has some day information as well as some timezone information. So good to see when this device has been traveling in different timezones.

Sarah Edwards: [00:42:28:08] But for this example, I'm focusing on these first two columns here. If I go back to my device and open Telegram. I was in that one for a couple of seconds. Let's see, what else? Secure browser, good to show this one, if I have some extra time left. So secure browser is just a crappy secure browser. I'll use the term very loosely.

Sarah Edwards: [00:43:45:16] Let's see. Starbucks. Again, test device, not my own information. Just in case anything embarrassing comes up. It's not me.

Sarah Edwards: [00:44:08:21] So with that interaction this database should have been updated. Now sometimes databases do take some time to update. Transactions will be pushed to the wall file first, the redder head log, before they're pushed into the main database. So sometimes you do need a little bit of patience. But this is something that I'm constantly doing - I am running these SQL queries on a live device and seeing what things are being flipped, seeing what changes are being made to these different items here. Pretty much everything, from here down are the new entries to this. So we've got telegro, which is telegram bundle ID. There's mirmay downloader 3 is that secure browser.

Sarah Edwards: [00:44:48:07] CoreAuth, remember I did the touch ID? Then we got Starbucks, right at the end. Again, time in seconds here. So I was in that secure browser for quite a bit of time, 42 seconds, before I flipped to Starbucks. But this does show me some useful information. It shows me pretty much exact times. Now what I have found is that I don't want to necessarily trust time stamps. These time stamps are particularly accurate. Another database that I want to show you, if we do have some time, is the power log which may or may not actually have accurate time stamps. So this is why I do dynamic testing. I do something, I notice the time, I look at the time stamp in the database. If it's accurate, great. I can report on that, do whatever forensic stuff I need to do. If it's not, well, why? Or how can I make it more accurate? So that is KnowledgeC, for that one.

Sarah Edwards: [00:45:46:19] Before I go to a different database, let me just do an example of cda. I want to go look at that secure browser. I'm just going to throw a keyword in there. It finds it - Mirmay Downloader free. I want to go take a look at some of the data associated with that activity.

Sarah Edwards: [00:46:19:13] We've got some local storage stuff going on here. I'm not surprised to see that in the browser. I usually try looking at a lot of the preferences plists and those are likely going to be stored in library/preferences. So most third party apps have a very similar structure into them. I've got a few different plists. Most applications will have one with their bundle ID in them. Likely the most interesting of all of them. I'm going to use jlutil on that plist, and see if I find anything of interest. Now you notice that it did have a PIN code when I opened it up? So you can hide your browsing activity. I might just do searches for PIN code. No. Let me do a search for pass code. Unlock true. There we go. This is a good example for third party apps.

Sarah Edwards: [00:47:44:13] But there are some third party apps I'm going to specifically call out: the Hyatt, the room key app - whatever that's call? The Entry app. That stores your Hyatt credentials, in plain text, in this particular plist file. So there's absolutely reasons to take a look at these plist files. From a forensics perspective, I will generally just throw in keyword searches for PIN, pass code, user password, pass - all sorts of different iterations of that, to find useful information like this. Maybe I don't care about that Hyatt mobile entry app? But if I can get user credentials, nobody's ever reused credentials before in their life, ever, I might be able to use that for something else. Maybe opening up key chains or other protected containers. So it can be quite useful.

Sarah Edwards: [00:48:28:13] So this is just a nice little way of saying, hey, are they storing their data properly? In this case, absolutely not. What else? Let's look at the database. If I'm not sure which database this is - this is a great example of using fsmon.

Sarah Edwards: [00:49:13:03] I've got my jailbreak stuff here. Yes, I'm over 21, thank you. I can do all my fun stuff - I'm browsing. In the meantime, all of this stuff is being populated. So this is all the processes, this is everything. It's not just this application here. Now I'm just going to kill it and kind of browse. So it's creating, it's deleting, but there is access to this. That container's data application path, if you know which one you've already looked for, you might just maybe grabber filter it on that particular path of interest there. I'm going to take a look at that VD.db.

Sarah Edwards: [00:49:58:06] It's a SQLite database. The round databases are coming into play now, so always fun to keep up with all the different artifacts. My first level triage - check out the tables. History - that looks like a good one. If they're overly complicated I might scp this off and bring them into a different viewer altogether. But you can see some of my other examples here. Put some headers in here. Let me re-run this to make it pretty.

Sarah Edwards: [00:50:54:14] So we've got History, URL title, whatever, whatever, last visited, number of visits. This is great for data validation. So I went to my website yesterday, apparently 587 times. I don't love myself that much. I think it's a great website, I think it's a creative name, but this is just something weird and I feel like forensic analysts who don't do this sort of testing is going to be like, yes, this chick she loves herself. She went to her own website 587 times. No. I went there once and maybe it's recording all the different other things, that it's caching images, javascript, whatever the other crap is on there to make it look pretty. But this is where validation comes into play.

Sarah Edwards: [00:51:36:06] In this case, jailbreakbrewing.com. You saw me do this live. Did I go their six times? Definitely not. So I don't know why it's recording this? What's it's doing? Maybe I need to do a little bit more true reverse engineering with the binary to figure out why it's calculating the visits out here. But this is great for some quick validation. If you see something in your data that just doesn't make sense, don't trust it. I've been burned too many times to actually trust what third party, or frankly what Apple, even puts out there.

Sarah Edwards: [00:52:07:02] So that's what we do with the plist, with the databases. What else? I want to do one more database here. I want to go to power log. Power log keeps track of a crazy amount of stuff, like 100+ different tables. I have queries for a lot of these in my Apollo project. Do I have stuff for every single one of them? Absolutely not. I look at some of these tables, I'm like, I have no clue what's that's even tracking. Could be quite useful if you happen to find some that I don't support and you know a little bit more about it, please reach out. Do a pull request. Email me. Twitter me. But I do have quite a few different ones in here. One of them that's tracking is the flashlight. We've got headers in here. We've got date and time. I'm doing a little bit of parsing here, but it's mostly just a select star from this particular table, ID for this, time stamp, bundle ID and level.

Sarah Edwards: [00:53:37:06] Just as another example, again, I'm not joking when I say live testing. We're just going to sit here for a couple of seconds. This is what my life is, it's just this, and talking to myself on a variety of different devices. Turning on. Turning off. Turning on. Turning off. Re-run that again. Now this one does take a little bit of time to update... Notice these time stamps, here? Mmhm, Is it 1970? It's not.

Sarah Edwards: [00:54:26:10] This is honestly my life. It's just re-running SQL queries over and over and over until the database updates, which is probably not going to happen live.

Sarah Edwards: [00:54:38:14] While I'm waiting for that to do its thing, I have built another query. Let me just pop over to my Apollo stuff first. This is the last demo I'm doing to get you to lunch on time. This particular query is probably one of my most difficult queries. This takes three different selects, from two different tables in here. So there is a time offset table, don't ask me why they do this, it's weird. But a lot of the different time stamps within power log, every single one of them is not trustworthy, there is an offset time. So, three different selects. There is this one, this is the one I was actually showing you before, showing you on and off, on and off. And this one here, this time offset table, which for whatever reason is keeping a time offset. So I have to kind of calculate that together to figure out what realistic time these events actually occurred.

Sarah Edwards: [00:55:37:21] This is not perfect yet. If anybody does a lot of SQL query stuff, this particular one sometimes is off by one second, which really makes me angry, but when you come down to forensics, time stamps need to be absolutely accurate. So that does really bother me. So three different selects to make that work. There might be a better way of doing this? Please let me know. But that's what I have for it right now. So let me run this one. Let's see, did this update yet? No, of course not. Why would it update during a live demo?

Sarah Edwards: [00:56:14:22] If it doesn't, it doesn't matter. So I took that one, from Apollo SQL and then I've just added a little bit more stuff to it. We have this adjusted time stamp here. Oh, now it's added all of them, now you decide to do it, great. So this is me turning it on and off, on and off. I have my adjusted time stamp here. This is where I'm taking that other time off that table, adding that in there. I keep the original one in here, this is purely for forensics on Apollo. I have the offset time stamp, so that value that I basically have to add to make it a reasonable accurate time stamp. And then, finally, this information here. For a lot of my Apollo stuff we also have another column right at the end. I'm just going to show you the original entry in the database. You can always go back and validate your information there.

Sarah Edwards: [00:57:14:02] Instead of zeros and ones, I changed it to on and off - makes a little bit more sense. Some accurate time stamps. So we do get that information. Of course, this is showing it to you in UTC time, because UTC are GTFO when it comes to forensics, but you do want to be accurate with this information. So don't necessarily trust your time stamps. Don't necessarily trust your data. Test it. I'm not going to say it's going to take quick stuff like this. This particular set of queries here took days to try to figure out how to do that. Once I figure it out for one table, fortunately I can more or less copy and paste, fill in the bits that I need to do for a lot of the other power log tables in there. But this does take a long time, especially when you put some of this information into the queries itself.

Sarah Edwards: [00:58:28:11] Just to wrap it up, this was just a very basic example but I end up using these tools, these scenarios, these kind of techniques every single day. Whenever I'm doing my day work, when I'm doing my SANS work, if I'm on a MAC, if I'm on iOS, if I'm doing Android stuff - pretty much the same stuff there applies. I'm doing these same exact pieces here. A few other things that I do look at from the Mac side, I absolutely use the log utility to look at the unified logs, because it's constantly running in the background and shows me a lot of system and user activity. I can't really do that easily, dynamically, on iOS device, because there is no log binary for that. If anybody wants to create one, I will be very happy to test that for you.

Sarah Edwards: [00:59:11:07] Other things I could do: network analysis. If I want to see where these applications are calling out to, different processes, what it's doing there. And, of course, we can get into the objection and fridaand maybe a little bit more of the true reverse engineering, dynamic reverse engineering. There are so many other different things that you can do to kind of step up your analysis here. But we certainly do not have time for that.

Sarah Edwards: [00:59:32:04] Thanks for coming. The too long didn't read of this presentation is: testing isn't difficult. Don't look at me the wrong way when I say, hey, just test it. I'm not going to tell you all the answers. I get emails all the time saying, hey, where is this artifact stored? Or where is this plist? Or this key? I don't know - test it. Actually try some work before you come to me. All of this is due to jailbreakers and researchers and all that stuff. I cannot live, I cannot do my day job or frankly any job without a jailbreak, so I'm very dependent on them. So, thank your jailbreaker, thank your researcher, buy him a beer, give him a high-five, don't when, ETA them - they don't like that. I know this, because I tried to joke with Jonathan last time - he did not find that funny. Don't do that.

Sarah Edwards: [01:00:14:21] Of course, it says purchase of Mac or iOS device, that's your true expense right there. If you want more information you want to take a class with me, certainly the same sort of sarcasm, snarkiness, comes out in my class. That's forensics 518, for SANS. I'm on Twitter a lot, that's probably my main communication point. I'm not really great at email. I don't even put it up here anymore. But if you can't find my email, well, you shouldn't be on the Internet, because you can definitely find it - I just won't answer it. Twitter, my blog, GitHub and specifically my Apple Pattern of Life Lazy Outputter, is out there as well. I've got the logos up there, if you want stickers of either one of those come and find me a little bit later on - happy to do that.

Sarah Edwards: [01:00:55:13] If you do have any questions, I will be here all day. And now I feel like I need a beer. So thank you.