Leveraging Cloud Technologies to Process Massive Data
Metron has developed specialized methods for quickly mining massive data repositories to identify suspicious and anomalous entities, subgraphs, and transaction patterns. We have successfully demonstrated the ability to track, detect, and characterize networks of interest in a massive sea of transactional noise.
We have implemented these techniques on Hadoop-based architectures and successfully performed analyses within distributed cloud environments. By converting our algorithms into MapReduce tasks and running on our in-house 18-node, 250-core computing cluster, we have applied these methods to multiple scales. On the low end, we successfully conducted network analyses on repositories of 200 million Twitter messages. On the upper end, we applied MapReduce methods to parallel-process billions of TSA records to learn complex passenger risk indicators and screening rules, 30 times faster than the previous non-Hadoop implementation.