Remote Forensic Investigation Platform

Posted: September 2, 2016 18:47:26 • 769 words
  • Started: March 2013 (Ongoing project)
  • Role: Team Leader, Architect, Back-End Developer
  • Team Size: 6
  • Tech: Django, Python, PHP

In 2013, I took on the biggest project of my career: building a large-scale remote forensic investigation platform. This process required some of the most advanced, innovative, and demanding development I've ever done, while also challenging my architectural and leadership skills, an effort which continued long after the initial launch. Since the specifics of the platform and the missions it supports are classified, I cannot go into great detail, but I can describe the project and my role in general terms. The project's charter was to build a series of new digital forensic tools for evidence collections and analysis, united by a centralized application. The central module remotely managed and communicated with a wide variety of tools (some custom-built, some pre-existing), controlling their operations and retrieving their data, with extensive use of parallel processing to handle the heavy encryption, large amounts of data, and wide selection of tools. It also offered controls for the selection of tools, and displayed the resulting data, in a user-friendly interface for our agents.

Python was the foundation of the project; I built the user interface and central management/control components with Django, and I used non-Django Python for many of the forensic tools controlled by the central application. Because of my expertise as a software developer and my past work as a network security administrator, I was able to write many of the tools myself, augmenting the tools created by other developers. Linking all the pieces together so that they could communicate with the control module was the biggest challenge; some of the pre-existing in-house tools were easy to modify, but most required more extensive modification before they could properly communicate with the control module. In addition to integrating new and existing in-house components into the platform, I had to integrate several third-party tools by writing an inter-operation layer that could normalize the differences between their widely-varying API designs. All of this communication had to be heavily encrypted and secured, to preserve operational security and evidence integrity/chain of custody, and to prevent the possibility of data leaks if a component were compromised.

In addition to the code-centric challenges, most of the agents using this system weren't very tech-savvy: I invested significant time into UX testing and design, alongside my team's UX specialist, to put powerful, advanced digital forensic capabilities into a tool that any agent could operate and understand with minimal training. As a result of that effort, the most common feedback I received from our investigative teams was that this platform was the single most useful application ever built within the agency. Several even joked that the only thing missing was a "Click Here to Arrest Subject" button. We had successfully taken a pie-in-the-sky idea from an ambitious manager, and turned it into a platform that exceeded all expectations, giving our investigators and field agents brand-new tools and opening up new ways of investigating crimes.

As team leader, I initially coordinated the efforts of two other forensic developers, a UX engineer, and two network security engineers. I also routinely met with three different investigative teams to learn more about the criminal investigation process, to understand their work and determine the best way to meet their needs. This project was unlike anything the agency had ever built, and we had very few pre-existing tools to work from, so early on, I was in charge of developing the architecture of the system's components, data models, and inter-component communication methods. Once we had that, development could begin; I built the control-side components, and developed several of the forensic components, to demonstrate what worked and to create some example code for the other developers to work from. I then delegated the rest of the forensic components to other developers, with frequent collaboration and penetration tests from the network security engineers, while I handled user/management demos and ongoing development/performance adjustment of the central control-side components.

Over time, members of the development team came and went, and post-launch work continued, with incremental improvements and adjustments to handle edge-cases and corner-cases that didn't come up during initial testing, plus routine check-in meetings to ensure that we were still serving the needs of the investigative teams. In Spring 2016, I led a 6-month project to build more modern versions of some of the older and more outdated components (including replacing one particular legacy tool that a former member of my team built with ColdFusion), and to add new features that were too substantial to patch in as incremental improvements.