Matt also added some new options, and the full list of options is now:

Options:
  --version             show program's version number and exit
  -h, --help            show this help message and exit
  -f FILENAME, --filename=FILENAME
                         [Required] Name of the MFT file to process.
  -d, --debug            [Optional] Turn on debugging output.
  -p, --fullpath         [Optional] Print full paths in output (see comments
                        in code).
  -n, --fntimes          [Optional] Use MAC times from FN attribute instead of
                        SI attribute.
  -a, --anomaly          [Optional] Turn on anomaly detection.
  -b BODYFILE, --bodyfile=BODYFILE
                         [Optional] Write MAC information in mactimes format
                        to this file.
  -m MOUNTPOINT, --mountpoint=MOUNTPOINT
                         [Optional] The mountpoint of the filesystem that held
                        this MFT.
  -g, --gui              [Optional] Use GUI for file selection.
  -o OUTPUT, --output=OUTPUT
                         [Optional] Write analyzeMFT results to this file.

The project is now hosted on Google Code, here.

http://code.google.com/p/opensourceforensics/

It is currently the only project there, but I will be adding a new project hopefully this week and others are encouraged to make this their home as well.

The site has all the bells and whistles required to support collaborative development of open source DFIR tools – a wiki, a Mercurial source code repository (and Mercurial really seems easier to grok than git), an issue tracker, and a download page for binaries and other packages.

As part of the move, I finally built a current binary using bb-freeze. (Hat tip to @bbaskin for the pointer to it.)

As mentioned elsewhere, I’m just starting on a new project to build a loose framework of dfir utilities and their supporting libraries in Python. The first release should go up on the site this week.

First off, I went through college earning most of my CS degree on Linux. The command line is an old friend, and stringing processes together with utilities is second nature. But even if you’re fresh out of college and have never seen Linux you will quickly find that the GUI driven tools just don’t cover all of your needs, and probably never will. This is particularly true if you’re working with a client on a small budget or a client who lacks in house litigation support. Why? You can’t deliver your work via load files or an expensive review platform. Instead, you need to send over zip files and massage the contents so they can be reviewed with commonly available applications. But even in large ediscovery and forensics projects, the GUI driven tools don’t give you 100% coverage.

Case in point. Using dtSearch I had identified 700 files spanning four volumes mounted using FTK Imager. The list of files was in a single text file. I needed to pack all of these files up in multiple zip containers due to bandwidth issues for delivery to a client without modifying their MAC times. And, by the way, the filenames weren’t unique so I couldn’t just zip them up, and I couldn’t copy them to one location and then zip up that location. I also couldn’t put them in a traditional evidence container using FTK Imager because the client didn’t have FTK Imager or MIP.

I eventually wrote my own utility that drove xxcopy ’cause robocopy is designed for directories, not files and xcopy doesn’t preserve MAC times and neither of them will take a list of files to work on as a command line option. It got the job done, but I spent a lot of time thrashing around before I stumbled on this.

Enter Dan Mares and the upcopy utility. It has an incredible number of useful options, but for my purposes, three really stood out:

1. It preserves MAC times
2. The –flatten option will take a tree structure and copy all the files to a single directory
3. The –nodupe option will detect duplicate files that would result in name collisions and add a unique suffix to each duplicate file

http://www.dmares.com/index.htm (follow the various links in the direct links section.)

Please note that, despite the disclaimer, Dan is still actively supporting his tools and is still very active in the community.

1) As data volumes and the number of devices increase, clients may need to be willing to pay more for the analysis. The cost of the work isn’t nearly proportional to the number of custodians these days. Just because data volumes are increasing doesn’t mean that the work doesn’t need to be done. The successful practitioners will be the ones who figure out how to process all that data while keeping their clients happy.

2) Then again, does all the data need to be processed immediately? The successful practitioner may also be the ones who successfully triage the problem and can defend those triage decisions to their client and in court. Just because you don’t process all the data immediately doesn’t mean you cannot go for a deeper look later when justified.

3) Approaching the problem as a team rather than as an individual will yield better results. In addition to splitting the problem over multiple cores (technical solution), split the problem over multiple people (organizational solution), each with deep domain knowledge and appropriate skills. The amount of work done by each individual may go down a bit, the total work done by the team will scale with the volume of data and number of devices, and there will be some additional overhead due to coordination. The overall efficiency, given a good team, should increase quite a bit. I know I’m much more efficient with additional eyes on the problem working in concert. The solo practitioner may need to limit the jobs they take on, or form partnerships that allow them to share the work efficiently.

The problem is hardly insurmountable, and in any such challenge there are opportunities. We can wail and gnash our teeth or we can quietly (or, if you’re in marketing, noisily) step up and meet the challenge, ensuring quality services for our clients and a secure job for ourselves.

1. WinFE with FTK Imager, IEF, and X-Ways. This successfully imaged a Vaio laptop with dual SSDs in a RAID configuration without a hitch.
2. Tableau TD1 – if this thing would write to multiple destination drives simultaneously, I’d kiss it. Even without the dual destinations, it is a rock solid imaging solution. (Bring a USB keyboard to make things a bit easier.)
3. FTK Imager CLI – Ok, I know how to use dd and its brethren, but FTK is a bit more full featured, and being able to use one software tool across all the platforms was great.
4. FTK Imager – FTK Imager doing logical folder collections made packaging the loose files very easy. And, again, one software tool.

WinFE

> diskpart (to run DiskPart)
> list disks (to see the media connected to the system)
> select disk “N” (where “N” is number of your destination drive)
> online disk (to bring the disk online)
> attributes disk clear READONLY (to allow writing to the disk)
> list volume (in order to choose the volume on the destination disk to write)
> select volume “V” (where “V” is the volume number to your destination disk)
> attributes volume clear READONLY (to allow writing to the volume)
> assign letter=Z (any letter you choose, to which your image will be written

Of course, there are all sorts of other things in my collection kit – two Pelican cases full of stuff, in fact, but everything mentioned here will fit in one case and will allow you to handle quite a bit of what might be thrown at you.

In those five years I’ve noticed that the computer forensics community has become *less* supportive, not more supportive. This runs contrary to trends to other communities such as software engineering tools, web frameworks, and startups. I have some feelings and thoughts on why this is. I wish I had some good ideas on how to turn this trend around.

I think there are four major problems:

1) Fragmentation of the sites supporting the community.

When I showed up, there was Forensic Focus, the CCE list, and HTCIA. (And other people probably had their three or four sources that don’t overlap with mine.) Now, I’ve got Forensic Focus, CCE, HTCIA, HTCC, DFCB, wn4n6s, and a host of OS and tool specific sites. Then there is LinkedIn, with an almost one to one mapping of all the external groups, plus subgroups, plus additional new groups not represented elsewhere.It seems that everyone wants their own lawn to play on rather than contributing to the health of an existing lawn. How often have you seen a post along the lines of “Hey, I set up a new forensics wiki! Come check it out and help it grow!” Or found yet another computer forensics LinkedIn group?

This leads to two related problems: Where do you post, and where do you go looking for information? I belong to a lot of the mailing lists and use my personal mail archive as a research tool when I have questions, but that doesn’t reach into the various web based forums. And if I want to post a question, where does it go? Some people blast every mailing list they’re on, hoping for an answer. And the more we balkanize, the more likely those questions are to go unanswered.

I still use FF and the CCE list mostly, but then there are items #2 an #3.

2) Web of trust.

When I joined the CCE list with certification #832. There’s no way I’d ever meet all 832 people, but by proxy, we knew of most people on the list. It was a small, tight community. Forensic Focus was similar – it was a place where we had a pretty good sense of most of the people posting, and most of the new people took some time to get up to speed on the community.I don’t know how many CCEs there are on the list now, but it seems that I  know fewer of the people who are posting now that I did two years ago. People I used to see regularly on Forensic Focus are rarely seen, often replaced by very new people who are unfamiliar with the community. Many of these new posters seems to be looking for a solution to some university project. There are now people on the HTCC  list posting anonymously.

3) Archiving, auditing, and reach of social media.

The growth in the number of forums, and the number of participants in those forums, greatly increases the number of potential employers, detractors, auditors, etc. Five years ago I felt pretty comfortable about asking stupid questions on the CCE list (a closed list) and even on Forensic Focus. Now, I’m very reluctant to ask anything that might display a lack of knowledge in an area where I am an expert.

We all know that none of us knows everything, and we’re all better for the support and feedback of our community. But when those questions can be spun, taken out of context, or turned back on us in some way, it makes us wonder if the potential downsides are worth it. Since there are almost always other people with the same question who aren’t speaking up, our failure to ask those questions means the entire community is worse off for these questions not being asked.

4) Pointing out that the Emperor might not be wearing any clothes is discouraged, actively and passively.

Some of this is due to “there but for the grace of God go I”, some  due to over sensitivity to political correctness, some due to fear of legal action, and some due to fear of getting dragged into the mud. (“Never wrestle with a pig: You both get all dirty, and the pig likes it.)The end result is that bad information lingers in the community, bad behavior persists, and people get fed up and move on to other places to invest their time and energy. And once you lose people, getting them to come back is often very hard.

I know I’ve become far more of a content consumer than generator over the last few years, though I still go through bouts of trying to contribute. My solution was to grow a small group of people I can trust to bounce ideas off of and I’ll turn to them rather than the larger community.

I am poorer for this fragmentation, and if you aggregate the loss of many people such as myself, the community is poorer as well.

Take a moment and do some research on the ADAMS problem. If you’ve any experience with ediscovery, or complex computer forensics cases, you might begin to think that you’ve seen this problem before on a smaller scale. Note that the ADAMS announcement specifies that the providers must provide test data – the providers need to prove that their products work in a controlled, instrumented environment before they’re released into the wild. Further, the people running the project must see the results before the solutions are accepted.

Hmm. What if we could do the same for ediscovery? What if you could have three vendors on site and compare them, on known data, head to head?

And, what if you could run known data through an ediscovery tool or process and accurately measure that process? What if, in so doing, you found that the process was flawed? If it is your process? Your vendor’s process? Your opponent’s process?

Oddly enough, we’re developing tools to help you answer some of those “What if”s.

In the course of the interview, I came up with an analogy for our process which the reporter captured quite well – we’re creating virtual crime scenes. Crime scenes that can be adjusted, wiped clean, rebuilt, or used over and over again. Further, we’re populating these entirely electronic crime scenes with real evidence – documents with accurate metadata, email messages with legitimate headers, SMS messages with topical content.

To digress a bit, the last item is the most difficult, and the most interesting. It is easy to sanitize existing content, and fairly easy to generate responsive content wrapped in digital noise, but can we create a reasonable approximation of human generated content, and keep it on topic? Can we create, out of the whole cloth, email conversations that appear to discuss a particular business topic in a manner that ensures they will be, or will not be, responsive to particular criteria?

No, not immediately, but we’re on the right path. And please don’t get too distracted by our desire to include natural language processing at some point as there is an enormous amount of value we can add now, and in the near future.

We already build virtual crime scenes, or digital corpus representing the corporate computing environment to be processed by ediscovery tools. And knowing how the corpus was built down to the last byte allows us to determine the accuracy of the ediscovery process, down to the last byte.

Stay tuned, interesting times are coming for the ediscovery world.

Unfortunately, I do not have a Tableau eSATA writeblocker so I cannot include TIM in this research. (While I understand the benefits of tight integration between software and hardware, particularly in this case, it would be nice if  TIM had a “degraded mode” that worked with non-Tableau writeblockers.) So here’s my test environment:

Test components:

• System: Dell Precision 690, Dual 64 bit Xeon CPUs, 8GB RAM, 64 bit Windows 7, internal RAID 5
• Drives:Western Digital VelociRaptor 160GB drives, one wiped with “00″ and one formatted and cloned with real world data.
• Writeblocker: WiebeTech UltraDock via eSATA interface to drive and to system
• Tools: FTK Imager 3.0.0.1443 and EnCase 6.17.0.90

To see what the various tools might do with different types of data on the test disks, I wiped two disks, one with a pattern of “00″,  and one with standard Windows formatting and then added real world data to it. These are listed as 00 and RW in the results.

All tests were conducted using a WiebeTech Ultradock connected via eSATA to both the drive and to the system.

All tests used 1500MB chunks, MD5 hashes,  and had verification turned off.

Imaging times:

Compression results (size of resulting images):

FTK, full E01 compression, ’00′ drive – 262MB
EnCase, full E01 compression, ’00′ drive – 524MB

FTK, full E01 compression, real world drive – 61.5GB
EnCase, full E01 compression, real world drive – 62.3GB

Conclusions:

Please bear in mind that this is a really limited data set, ok? As I write this, I’m imagining all the comments of the form “But if you did X, then ….” My suggestion to those people is “Why don’t you try doing X and let us know how it turns out?”

1. Using compression will add significantly to your imaging times
2. EnCase’s E01 compression is faster than FTK Imager’s.
3. Both tools do equally well on compression
4. The randomness of the data on the drive affects the time required to image the drive if compression is turned on.

Observations:

1. There are a lot of variables that affect imaging speed – drive, drive interface, write blocker, write blocker interface, system IO bus, CPU type and speed, and target drive to name the big ones. If you’re looking for performance, you can’t control the drive characteristics but you can invest in the other components. If you’re not using compression, the biggest bottleneck will be your IO bus so go with eSATA whenever possible.
2. If you’re imaging for archival purposes, compressing while imaging makes sense. Otherwise, consider leaving the image uncompressed until you want to archive it.

Further research:

1. If I had more time and hardware resources, I’d love to rerun these tests while adjusting each of the variables identified above.

For a brief primer from a school’s department that assists with submitting proposals, check out this link.

Quoting from that document:

SBIR One to three announcements per year Schedule: www.zyn.com,

Phase I

– $75K – $100K (or more) award + Options – 6 months duration – Feasibility Study – Can sub-contract SBIRs up to 33.3%

Phase II

– $750K award (typical) – 18 mo – 24 months

Phase III

– Unfunded commercializationA Brief SBIR/STTR BAA Primer

STTR
• Same award value
• Prime must perform at least 40% of the work
• Research partner must perform at least 30% of the work
• A maximum of 60% can be subcontracted
• Small business must submit
• Much smaller funding pool

BAA – Broad Agency Announcements
• A description of needed research and technology
• For projects not supported by current programs
• Initiated by a white paper
• Funding not always available!
• Award amounts typically $600K – $850K

Some relevant points from my own experiences with these mechanisms:

• Long lag between proposal submission and funding
• Highly structured proposal format (which is a plus in my book)
• No commercial restrictions on products developed with the funding
• Must give product to government for free. (They paid for it with the funding.)

The last bullet point is the source of my crow lunch. With funding comes strings and if you want to get a product to market, you need to make some compromises.

So if you’re looking for funding for computer forensics products, you might want to keep an eye on the SBIRs and BAAs. Go read up on the requirements and proposal formats. Think about possible partners that will add value to your proposal. Plan ahead.

Not a sure thing, but a possibility, and there are other similar programs out there.

• Version 1.5:
  • Fixed date/time reporting. I wasn’t reporting useconds at all.
  • Added anomaly detection, with many thanks to Greg Kelley. Adds two columns:
    • std-fn-shift:  If Y, entry’s FN create time is after the STD create time
    • usec-zero: If Y, entry’s STD create time’s usec value is zero
• Version 1.6: Various bug fixes
• Version 1.7: Bodyfile support, with thanks to Dave Hull

The anomaly detection isn’t perfect by any stretch of the imagination, it simply helps reduce the noise a bit.

• On the $MFT from a volume on a workstation with 110593 total records, checking for FN creation times greater than STF creation times resulted in 19649 flagged records. Pretty significant reduction.
• On the same file, checking to see if the STF creation time microseconds are zero resulted in 14571 flagged records.
• Turning both on resulted in 2157 flagged records. Most appear to be benign. (I hope they all are!)

That’s still 2157 (or 19,649, or 14571) files that you need to check by other means, but it is a lot less than 110593.

If there’s some feature you’d like to see in analyzeMFT, please, do drop me a note.

You can find the source and more details here….

There’s also a great post on how to install Python and run analyzeMFT’s source code here….