I finally figured out how to build a standalone executable after an Alice in Wonderland run through redistributable libraries, py2exe, and Windows installers. There are still some issues, but it works well for the most part. Check the Download section on www.integriography.com.
Some tools that helped me turn a Python script into something that can run on any (most?) Windows systems are:
- py2exe – http://www.py2exe.org/ – Read the Tutorial page for some really good help with the .dlls
- Dependency Walker – http://dependencywalker.com/ – A great tool for determining what modules your application depends on
- Inno Setup – http://www.jrsoftware.org/isinfo.php – A very simple yet powerful tool to build installation packages
At the request of Harlan Carvey and Rob Lee I made some changes to analyzeMFT and fixed a few bugs along the way.
- Version 1.1: Split parent folder reference and sequence into two fields. I’m still trying to figure out the significance of the parent folder sequence number, but I’m convinced that what some documentation refers to as the parent folder record number is really two values – the parent folder record number and the parent folder sequence number.
- Version 1.2:
- Fixed problem with non-printable characters in filenames. Any Unicode character is legal in a filename, including newlines. This presented some problems in my output. Characters that do not render well are now converted to hex and a note is added to the Notes column indicating this.
- Added “compile time” flag to turn off the inclusion of any GUI related modules and libraries for systems missing tk/tcl support. (Set noGUI to True in the code)
- Version 1.3: Added new column to hold log entries relating to each record. For example, a note stating that some characters in the filename were converted to hex as they could not be printed
The code and more details are available at www.integriography.com
Quick note on $MFT sequence numbers:
Microsoft tells us that each record in the $MFT has a FILE_RECORD_SEGMENT_HEADER Structure. Within this structure is a sequence number, defined as follows:
“This value is incremented each time that a file record segment is freed; it is 0 if the segment is not used.”
Ok, that’s pretty straightforward. At least until you look at teh first 16 entries in any $MFT as all of their sequence numbers match their record number. I’ve been told that since these files can never be deleted, repurposing the sequence number adds an additional sanity check and disaster recovery option. However, I’ve found one volume where this behavior continues for 12,000 records or more. Still looking into that one.
One of the best sources for NTFS documentation isn’t Microsoft, it comes from the Linux NTFS developers and is available here.
It is considered very good practice to make two copies of any image collected, particularly in the field. On one very long collection trip we did this by collecting to one set of drives during the day and running Robocopy over night to duplicate the image set. FTK allows writing to two destinations, and the various versions of dd have always allowed this via one means or another. But these all require either time or precious IO bandwidth.
So, I thought, is there any way to create two images in real time without pushing the data down the pipe twice? Isn’t that what RAID1 is supposed to provide? But, are two drives in a hardware RAID 1 *really* identical? Turns out, that at least in my test case, they are.
I bought a vAGE220-SAU two drive, USB 2.0/eSATA, RAID0/1 external enclosure. ($275 @ Amazon.) It’s fairly well constructed, compact, and easy to use. The instructions weren’t clearly translated but were sufficient unto the task. Once I flipped the dip switches correctly and waited a few hours for it to do the initial mirroring, I was good to go.
I hooked my source drive up to one port on my field laptop’s eSATA card and the RAID enclosure up to the other one. Fired off FTK (but dd, or EnCase, or whatever would have done just as well.) Imaged the drive and it ran at near expected speeds. The process finished and the image was verified.
Now the test. I pulled both drives and hashed them via a writeblocker. The hashes matched. I had two identical, forensically sound, images of my source drive. This required less time that imaging to two destinations using the hardware available on my field laptop, and a lot less time than running a copy overnight.
I need to try this a few more times and do some more performance measurements, but I’m pretty happy with the outcome. I wish there was a drop in drive dock with RAID1 capability. That would eliminate the need to open the enclosure up when changing disks.
Three elements combined last week to inspire me to write a tool to deconstruct the Windows NTFS $MFT file:
- I’ve been wanting to learn Python for quite awhile. (I found a “Learning Python” book on my shelf published in 1999.
- Mark Menz’s MFT Ripper started me wondering about the significance of the MFT sequence number.
- I’d been trying to get through the SANS 508.1 book but couldn’t bear to read about NTFS structures yet again.
- I wanted to start building a framework for doing more detailed timeline analysis.
So, last week I sat down and wrote analyzeMFT.py. Please keep in mind that this is a novice Python programmer’s code and is definitely a work in progress. A simple project page and a link to the source can be found here.
If you have any comments, suggestions, or improvements, please do let me know. I’d like to keep building on this and making it as useful as possible.
I bought this months ago as it seemed like something that would be really useful to own. Then I got busy and it sat until someone asked me about doing a review. I pulled the unit out and immediately attached it to a WD 320GB Passport. No go, not enough power. Wiebetech very promptly sent me a very well constructed dual USB cable to provide power and connectivity. Good to go, or so I thought.
I really want this thing to work, and it just doesn’t.
- It doesn’t work at all with OS X. (Correction: It works once, but subsequent drives are not recognized.)
- The drivers failed to install on Windows 7. It looks like this can be addressed with a certain amount of “reset and reboot”. (Rebooting and reinstalling cleared the problem. It works with Windows 7 32bit and 64bit.)
- It requires a lot of “fiddling” to get it working on XP. Sometimes you need to hit the reset button, sometimes the order that you plug things in matters, and all too often you need to reboot your system.
- It doesn’t work with EnCase at all. EnCase locks up trying to read the first two sectors of the device. If you hit the reset button, EnCase returns an error and drops back to the GUI so it isn’t completely locked up, but you cannot add any evidence attached to the write blocker. X-Ways, FTK Imager, and FTK all work, taking the “fiddling” into account. (This is an EnCase v6.15 problem. If you turn off “Detect FastBloc”, EnCase v6.14 and .15 recognize the device.)
I tried it with FAT and NTFS formatted thumb drives as well as the WD 320GB Passport – same results.
The packaging is great and the concept is good but it simply doesn’t work consistently enough or with enough operating systems to make it worth adding to my kit.
Since posting this, I’ve spoken with several other users. Some have experiences very similar to mine, some haven’t had any issues at all. I’m also talking with CRU-DataPort and they’re actively working through identifying and fixing the problems. I’ll hang on to my unit and hope they get these problems sorted out soon.
- The USBWB apparently works with versions of EnCase prior to V6.14.
- Disabling “Detect FastBloc” in the later versions of EnCase allows. EnCase to successfully recognize the USBWB attached drive.
- USBWB doesn’t work on OS X 10.4, does work on 10.5, and hasn’t been tested on 10.6. It seems to work once on 10.6 but not with subsequent drives, at least not without a reboot.
- USBWB works on Windows 7 64bit after a reboot. Drivers fail to install, reboot, drivers install.
In several Guidance classes, I’ve heard fellow students ask “Can you suggest a standard workflow for using EnCase?” The exact workflow will vary from case to case, but I’ve put together one possible workflow with some help from other contributors to the Forensic Focus forum. Please bear in mind that this is a guideline, a suggestion, just one possible way to work through a case using EnCase. You should clearly understand what each of these steps entails and adjust the workflow to suit your style, your written processes, and the case you are working on.
- Create case – Ensure that you have all relevant information – custodians, clients, case name, etc.
- Change storage paths as appropriate. I set everything to go to a volume or folder dedicated to the case.
- Save All.
- Add evidence – E01, LEFs, loose files, etc. Each time you add evidence, you should consider rerunning several of the following steps.
- Confirm disk geometry, sector count, partitions. You’re checking to see if everything is accounted for. There may be hidden partitions, for example.
- Run Partition Finder if indicated
- Run Recover Deleted Folders
- Search case – hash and signature analysis. You will probably repeat this each time you add new evidence.
- Run File Mounter – recursive, not persistent, create LEF, add LEF to case.
- Run Case Processor -> File Finder. Export results, add back in as LEF.
- Search case – hash and signature analysis.
- Search for encrypted or protected files. Address as appropriate.
- Extract registry hives. This can happen at any point really and they’ll be fed to RegRipper.
- Index case.
Depending on the case:
- Analyze LNK files and INFO2 records
- Extract browser history and carve browser history from unallocated
- Parse the event logs into a CSV format.
Other tasks performed outside of EnCase:
- Mount image and scan for viruses. Use several different products and never assume that they’re 100% accurate.
- Mount image and run triage tool(s) against it
- Run image in LiveView or VFC to see system as user experienced it
- Run RegRipper and RipXP against registry hives
- Run MFT Ripper against an extracted MFT
- etc, etc, etc
An article on PoliceProfessional.Com (original article has vanished and been replaced with new content) contains the following statement:
“ACPO is currently working on a new software tool that will allow forensic officers to operate locally and uncover information almost instantaneously. “What we’re very keen on doing is looking for a forensic triage tool that police officers or forensic officers can use locally. One that is quite simple, one they can ask questions of, such as, ‘in this computer is there the following…?’,” said Ms Williams. “The triage tool can pull that out for them.” She said the current backlog is one of e-crime’s biggest problems and that ACPO is close to identifying the right product to handle it.”
[Note: I've been told that the ACPO is looking to the vendor community for this solution. Rereading this quote, I suspect I should focus less on "working on new software" and focus more on "identifying the right product". I'll leave the post as originally written but will insert commentary.]
The apparent expectation that a tool will significantly address the backlog is rather disturbing for three reasons:
1) The tool will not provide context. It may indicate the presence of an encrypted file container on the system but cannot determine its contents. Or that file sharing is present, but not what it was used for. Or that seven different chat programs are in use, but not the information going through them. As several people have pointed out, these PBF tools will get the low hanging fruit and gather disparate facts but cannot put do any analysis to show relationships, or lack thereof. Further, we’ll need to err on the conservative side and may well end up with a lot of false positives.
2) Technology, and the criminal’s use of technology, advances rapidly, often more rapidly than the tools. This is why DriveProphet’s author is very willing to add new capabilities as issues are reported to him. It is why Digital Detective Group’s Blade product has plug in modules that they can develop and release as new capability is required. Keeping a triage tool current requires ongoing investment by the developer and ongoing training for the users. A one time investment in the technology and training will quickly lead to a situation where the triage tool is missing relevant information. [Note: ACPO's looking to a vendor solution should address the support issue. Keep in mind maintenance costs when investing in a tool. Some vendors charge upwards of 20% of the initial investment each year for maintenance.]
3) I’ve not seen any well researched study on the LE computer forensics backlog that we can use to determine where resources should be spent. The ACPO and others believe that the the backlog is in the triage stage. This appears to be valid, particularly for getting evidence back to the owners, but I suspect that “fixing” the triage stage will simply move the backlog further downstream, even more so if the number of false positives is high.
I also wonder why the ACPO is working on a new tool rather than working with a vendor of an existing tool to tune it to their particular needs. A number of good, well supported, triage tools already exist – Drive Phrophet, Blade, EnCase Portable, e-fense’s suite (now Access Data’s?), to name a few. The ACPO money might be better spent creating a fund to provide training on these existing tools rather than bringing another tool to an already crowded market. [Note: This point is moot given the feedback I received, noted above.]
Triage is an incredibly valuable process, particularly in time critical situations where limited resources are available. Triage, in the medical environment, is performed by trained specialists using diagnostic tools. Computer forensics triage tools often are designed to be used by anyone with minimal training. Witness the Microsoft press release about COFFE – “According to a Microsoft spokesperson ‘an officer with even minimal computer experience can be tutored—in less than 10 minutes—to use a pre-configured COFEE device.’” I believe there is value in this sort of tool when used as part of a well designed forensics process. I fear that, due to vendor marketing, budget issues, and backlog pressures, these tools will be deployed without the necessary framework to properly support them.
Allow me to close with some questions:
- Why is the ACPO creating a new tool rather than using an existing one? [Note: Addressed by feedback, noted above.]
- Who will use these triage tools and how much training will they get? If they’re designed for lab use to address the backlog will they stay in the lab? Can they safely be deployed earlier in the process?
- Are there any well documented studies on the LE computer forensics backlog?
- What other options are available for addressing the backlog? Anyone who knows me also knows that I’m very interested in finding ways for the private sector to assist LE with computer forensics and this would be one option.
My post about the value of push button forensics produced a number of interesting comments for which I am quite thankful. A common thread in many of the remarks was that someone needs to understand the the science, logic, and art behind the PBF tools. I absolutely agree. Anyone depending on a technician and a tool alone is doing a disservice to their clients, and will likely fail spectacularly in court.
As one reader put it:
“I think the point that is being missed is this – at the end of the day the goal is to produce admissible evidence. The fact remains that our system generally looks to an expert to introduce digital evidence into court. “
Harlan made a similar comment, and really got to the heart of the matter:
“The fact is that the questions being asked by customers…was data exfiltrated, did the malware make data exfiltration possible, etc…cannot be answered by a $50/hr “analyst” with a dongle. This approach will work for low hanging fruit, but even a relatively unsophisticated compromise will be improperly and incompletely investigated in this sort of environment.”
A $50/hour analyst with a PBF dongle should not testify in court and their findings alone should not be presented to a client as they lack context and perspective. Their results are only pieces of the larger construct, a construct that should be built and signed off on by people with significantly more experience. A senior examiner can guide a team of less experienced staff using a wide variety of tools, interpret and combine the results into a well constructed report, and sign off on the team’s work product.
Law firms and private investigation firms are but two of many examples of organizations that employ associates to perform many of the simpler tasks involved in preparing cases. Doing so distributes the workload, frees senior staff up for more complex tasks, provides associates with opportunities to learn on the job under the supervision of senior staff, and ensures that work product is reviewed and approved by someone in the firm who is responsible for presenting the case to the court or to the client. The same can hold true in a computer forensics firm, lab, or department. In fact, any firm with more than a few examiners needs to operate in this manner simply for coordination and responsibility purposes. I’m just proposing that the same structure works well to mitigate the risks of using push button forensics.
We build everything from airplanes to software applications to roads out of component parts that are designed to accomplish a specific task but that, standing on their own, have little value. Organizations work in a similar manner, utilizing human components along with their associated skills and tools to streamline many processes and produce better results than one person standing alone could accomplish. Integrate PBF tools and less experienced people into your organization, manage them appropriately, validate the tools, review the results, and let the senior examiners do the heavy lifting with the complex problems, clients, and courts.
Also, I suspect if most people looked around their organization, they’ll see technicians using push button tools as part of the computer forensic process already. Do you have Voom Hard Copy II or a Talon or one of the other hardware imaging solutions? How many button presses does it take to image a drive, and who is usually pushing those buttons? Do you really believe that you’ll need to explain to a client or a court how the Talon creates an E01 image? Your report will say “Imaged the suspect’s drive with a Talon, serial number XXXXX. The hash values reported by the Talon were XXXX and they matched. The Talon was certified to be operating normally during our regular maintenance, conducted per our SOPs.” It is pretty likely that the imaging was performed by a technician, and as was the regularly scheduled testing.
Push button forensics tools are here to stay and they’re already in use in most of our organizations. There clearly are risks to using PBF and inexperienced examiners inappropriately but through sound business practices they can safely contribute to our projects and improve our efficiency in the process.
Access Data recently entered into a partnership with e-fense. In the announcement, they wrote: “Digital investigations are no longer the exclusive domain of highly trained experts.” I don’t think Access Data is wrong, and I think the forensics community needs to accept that “push button forensics” is here to stay. Further, I think it can be an important part of our future.
(Two notes: 1) For the purpose of this article, forensics and e-discovery are essentially interchangeable. 2) I’m using “technician” to describe someone with basic to moderate technical skills but lacking in deep forensics and/or e-discovery experience.)
“Push button forensics” (PBF) is often derided by computer forensics professionals. We rail against it, occasionally joke about it, and have even made “Find Evidence” buttons to stick on our keyboards. Certain facts suggest that we should embrace it, though perhaps while wearing PPE.
- Tool vendors have a vested interest in selling forensics and e-discovery tools that can be used by people without forensics experience and certifications. If you can make a tool that any technician, lawyer, or IT person can use in a legally defensible manner, you will expand your potential market considerably. We are no match for the combined weight of the marketing departments of the vendors whose tools we are using.
- Corporations, LE agencies, law firms, and other consumers of computer forensics services have a financial interest in acquiring tools that will perform complex forensics and e-discovery tasks and that can be used by technicians rather than by experts. The cost per hour of computer forensics services in the San Francisco Bay Area is around $250. There is a lot of appeal in buying a tool and using a $50 per hour in house technician if you can get the same results.
- The volume and complexity of digital evidence is growing, and growing faster than we can cope with it. LE agencies at all levels have significant computer forensics backlogs, made worse by current budget issues. Corporate legal departments and law firms are under pressure to sift through enormous volumes of data more quickly, and more efficiently, than ever before. The number of people available who can manually sort through the complex evidence isn’t keeping pace, and the explosion in new computer forensics certification and degree programs will not solve the problem any time soon.
In addition to the facts that suggest we need to accept PBF into our environments, I’d like to suggest that, properly integrated, it can be very good for us personally and for our businesses. Here’s one example:
I’ve quite enjoyed following the development of Harlan Carvey’s timeline analysis tools and procedures. I’ve learned a lot from working through his examples, and I’d strongly encourage others to do so. But, the process is currently far too time consuming to use on any project with any significant pressure. We will need more automation, more “push buttoness”, to effectively employ it. And once it is “push button” AND validated, why can’t I farm that part of the process out to a technician? In doing so, I will:
- Acquire useful information in a more timely manner, speeding the investigation and saving the client money.
- Distribute the workload among more junior staff, enhancing their ability to contribute and decreasing the bottleneck on senior resources.
- Free up senior staff for tasks that truly require more experience and knowledge.
Put another way, from a consulting perspective, I can save my clients money, free up experienced people to work on more difficult problems, and safely incorporate people with less experience. The clients will be happy – better results for less money; the senior people will be happy – real challenges, less grunt work; and the junior people will be happy – more opportunity to gain experience.
Our forums are full of discussions about how to use an enormous number of tools, many of which automate and greatly simplify our processes.
- Anyone proficient with EnCase, FTK, X-Ways, or Sleuthkit could replicate Drive Prophet’s results but it would take hours longer, and the chance of missing something is greater.
- Similar point for web browser analysis – if there wasn’t a need to automate this, why do we have Mandiant Web Historian, Gaijin Historian, Cache Back, Pasco, Fox Analysis, NirSoft Mozilla History View, and Passcape History Viewer to name a few?
- With Mount Image Pro, I can provide a forensically sound image to a reviewer to examine with tools they’re comfortable with – Outlook, Explorer, dtSearch – without any risk that they’ll modify the evidence. This can save me a lot of back and forth to produce directory listings, copies of the My Documents folder, and .pst files.
If we look back through the archives of out discussion forums we’ll see that we’ve been automating and simplifying computer forensics processes since the dawn of the profession. In doing so we’ve made the profession more accessible to new practitioners, more valuable to our clients, and more interesting to ourselves. This mimics developments in the rest of the computer industry, and in every aspect of our lives. We’ve got push button cooking, push button flying (auto-land capability), push button navigation, push button photography, …. Push button forensics is here to stay. Accepting the fact and incorporating it into our processes and companies seems wise.
Mind you, I say this with several important assumptions in mind:
- The tools work as advertised, their behavior and results are well understood, and the process and results can be verified.
- The tools are verified internally.
- The use of the tools is supervised by experienced staff.
“Push Button Forensics” has a place in our business toolkits. Digital investigations are no longer the exclusive domain of highly trained experts. Validated PBF tools in the hands of properly trained and supervised technicians can be a very powerful combination for law enforcement agencies, law firms, corporations, and consulting firms.
I’d like to leave you with perhaps the most important point, one that is frequently overlooked or assumed – Finding the evidence is only a small part of the process. Tools can find keywords, put together a timeline, or show you the CP images. They cannot put any of that information in context. Interpreting the information, whether found manually or by PBF tools, still falls squarely in the pervue of a trained and experienced computer forensics investigator.