Retro-Future: The Retrospective Future in the Internet

Project Description

New network security events, from new zero-day attacks to insider threats, are only apparent after they have occurred. By definition, these events work in unexpected ways and only through hindsight may one learns what should have been collected and analyzed. To address these events, one either needs to predict the unexpected or travel back in time to replay the event. The premise of this project is to record enough network state to replay network security events and effectively travel back in time–in effect, to provide an Internet Digital Video Recorder (DVR).

The challenge in this effort is to provide Internet time-travel that is efficient, maximizing the effective history that is saved, to be cost-effective commodity hardware and software, and most importantly, to accommodate permission and privacy constraints that are necessary to deploy this system.

We expect that the resulting Internet DVR will help aid development and testing of network defenses.

Retro-future is a joint research effort of USC Information Sciences Institute Colorado State University’s Network Security lab, and Los Alamos National Laboratory. It is part of the ANT: the Analysis of Network Traffic research group.

This work supported (2012-2016) by Department of Homeland Security Science and Technology Directorate, Cyber Security Division , via SPAWAR Systems Center Pacific under Contract No. N66001-13-C-3001. Any opinions, findings and conclusions or recommendations expressed in this material are those of the author(s) and do not necessarily reflect the views of SSC-Pacific.

It is also supported (starting in 2016) by the Department of Homeland Security (DHS) Science and Technology Directorate, Cyber Security Division (DHS S&T/CSD) via contract number HHSP233201600010C. The views and conclusions contained herein are those of the authors and should not be interpreted as necessarily representing the official policies or endorsements, either expressed or implied, of the Department of Homeland Security.

People

  • Calvin Ardi, PhD student (USC CS Dept. and ISI)
  • Manaf Gharaibeh, PhD student (CSU)
  • John Heidemann, PI on this project, project leader and professor (USC/ISI)
  • Christos Papadopoulos, co-PI on this project, professor (Colorado State University CS Dept.) christos (at) cs.colostate.edu
  • Yuri Pradkin, researcher (USC/ISI)
  • Abdul Qadeer, PhD student (USC CS Dept. and ISI)

Alumni

  • Daniel Byrne, graduate research assistant (LANL)
  • Xun Fan, USC PhD graduate (2015) (USC CS Dept. and ISI)
  • Paul Ferrell, researcher (LANL)
  • Gina Fisk, project lead (LANL)
  • Mike Fisk, co-PI on this project, project leader (LANL)
  • Andy Gu, undergraduate researcher (USC CS Dept. and ISI)
  • Zi Hu, USC CS MS graduate (2014) (USC/ISI)
  • Dan Massey, professor (University of Colorado Computer Science Dept.)
  • Neale Pickett, researcher (LANL)
  • Anant Shah, Colorado State U. CS PhD graduate (2018) (CSU)
  • Shannon (Shane) Steinfadt, researcher and project lead (LANL)
  • Ben Uphoff, researcher (LANL)
  • James Wernicke, researcher (LANL)
  • Han Zhang, PhD student (Colorado State University CS Dept.)

In addition, we thank Dan Massey for his involvement early in the effort. He is currently on leave from Colorado State and this project.

Publications

  • Liang Zhu and John Heidemann 2018. LDplayer: DNS Experimentation at Scale. Proceedings of the ACM Internet Measurement Conference (Boston, Massachusetts, USA, Oct. 2018), to appear. [DOI] [PDF] [Code] Details
  • Abdul Qadeer and John Heidemann 2018. Plumb: Efficient Processing of Multi-Users Pipelines (Extended). Technical Report ISI-TR-727. USC/Information Sciences Institute. [PDF] Details
  • Hang Guo and John Heidemann 2018. IP-Based IoT Device Detection. Proceedings of the ACM SIGCOMM Workshop on IoT Security and Privacy (Budapest, Hungary, Aug. 2018), 36–42. [DOI] [PDF] [Dataset] Details
  • Calvin Ardi and John Heidemann 2018. Leveraging Controlled Information Sharing for Botnet Activity Detection. Proceedings of the ACM SIGCOMM Workshop on Traffic Measurements for Cybersecurity (Budapest, Hungary, Aug. 2018), 14–20. [DOI] [PDF] Details
  • John Heidemann 2018. Internet Outages: Reliablity and Security. Invited talk at University of Oregon Cybersecurity Day. [PDF] Details
  • Hang Guo and John Heidemann 2018. Detecting ICMP Rate Limiting in the Internet. Proceedings of the Passive and Active Measurement Workshop (Berlin, Germany, Mar. 2018), to appear. [PDF] Details
  • John Heidemann 2018. Outage Clustering: From Leaves to Trees. Talk at CAIDA Active Internet Measurement Workshop (AIMS). [PDF] Details
  • Basileal Imana, Aleksandra Korolova and John Heidemann 2018. Enumerating Privacy Leaks in DNS Data Collected Above the Recursive. Proceedings of the ISOC NDSS Workshop on DNS Privacy (San Diego, California, USA, Feb. 2018). [PDF] [Dataset] Details
  • Lan Wei and John Heidemann 2018. Does Anycast Hang up on You (UDP and TCP)? IEEE Transactions on Network and Service Management. 15, 2 (Feb. 2018), 707–717. [PDF] Details
  • John Heidemann 2018. Internet Reliability, from Addresses to Outages. Talk at MIT CSAIL. [PDF] Details
  • John Heidemann, Yuri Pradkin and Aqib Nisar 2018. Back Out: End-to-end Inference of Common Points-of-Failure in the Internet (extended). Technical Report ISI-TR-724. USC/Information Sciences Institute. [PDF] Details
  • Liang Zhu and John Heidemann 2017. LDplayer: DNS Experimentation at Scale. Technical Report 722. USC/Information Sciences Institute. [PDF] [Code] Details
  • Wouter B. de Vries, Ricardo de O. Schmidt, Wes Hardaker, John Heidemann, Pieter-Tjerk de Boer and Aiko Pras 2017. Verfploeter: Broad and Load-Aware Anycast Mapping. Proceedings of the ACM Internet Measurement Conference (London, UK, 2017), 477–488. [DOI] [PDF] [Dataset] Details
  • Moritz Müller, Giovane C. M. Moura, Ricardo de O. Schmidt and John Heidemann 2017. Recursives in the Wild: Engineering Authoritative DNS Servers. Proceedings of the ACM Internet Measurement Conference (London, UK, 2017), 489–495. [DOI] [PDF] [Dataset] Details
  • Kensuke Fukuda, John Heidemann and Abdul Qadeer 2017. Detecting Malicious Activity with DNS Backscatter Over Time. ACM/IEEE Transactions on Networking. 25, 5 (Aug. 2017), 3203–3218. [DOI] [PDF] [Dataset] Details
  • Liang Zhu and John Heidemann 2017. LDplayer: DNS Experimentation at Scale (abstract with poster). Technical Report ISI-TR-2017-721. USC/Information Sciences Institute. [PDF] [Code] Details
  • Liang Zhu and John Heidemann 2017. LDplayer: DNS Experimentation at Scale (poster abstract). Proceedings of the SIGCOMM Posters and Demos (Aug. 2017), 60–62. [DOI] [PDF] [Code] Details
  • Lan Wei and John Heidemann 2017. Does Anycast Hang up on You? IEEE. [DOI] [PDF] Details
  • Jelena Mirkovic, Genevieve Bartlett, John Heidemann, Hao Shi and Xiyue Deng 2017. Do You See Me Now? Sparsity in Passive Observations of Address Liveness. IEEE International Conference on Traffic Monitoring and Analysis (Dublin, Ireland, Jul. 2017), 1–9. [DOI] [PDF] Details
  • John Heidemann 2017. Digging in to Ground Truth in Network Measurements. Talk at the Network Traffic Measurement and Analysis PhD School. [PDF] Details
  • Moritz Müller, Giovane C. M. Moura, Ricardo de O. Schmidt and John Heidemann 2017. Recursives in the Wild: Engineering Authoritative DNS Servers. Technical Report ISI-TR-720. USC/Information Sciences Institute. [PDF] Details
  • Wouter B. de Vries, Ricardo de O. Schmidt, Wes Hardaker, John Heidemann, Pieter-Tjerk de Boer and Aiko Pras 2017. Verfploeter: Broad and Load-Aware Anycast Mapping. Technical Report ISI-TR-719. USC/Information Sciences Institute. [PDF] [Dataset] Details
  • Hang Guo and John Heidemann 2017. Detecting ICMP Rate Limiting in the Internet (extended). Technical Report ISI-TR-717. USC/Information Sciences Institute. [PDF] Details
  • Ricardo de O. Schmidt, John Heidemann and Jan Harm Kuipers 2017. Anycast Latency: How Many Sites Are Enough? Proceedings of the Passive and Active Measurement Workshop (Sydney, Australia, Mar. 2017), to appear. [PDF] Details
  • John Heidemann 2017. Collecting and Visualizing Outages Over the Long Haul. Talk at CAIDA Active Internet Measurement Workshop (AIMS). [PDF] Details
  • Lan Wei and John Heidemann 2017. Does Anycast Hang up on You? (extended). Technical Report ISI-TR-716. USC/Information Sciences Institute. [PDF] Details
  • Anant Shah, Romain Fontugne and Christos Papadopoulos 2016. Towards Characterizing International Routing Detours. Proceedings of the 12th Asian Internet Engineering Conference (AINTEC) (Bangkok, Thailand, Nov. 2016), to appear. Details
  • Giovane C. M. Moura, Ricardo de O. Schmidt, John Heidemann, Wouter B. de Vries, Moritz Müller, Lan Wei and Christian Hesselman 2016. Anycast vs. DDoS: Evaluating the November 2015 Root DNS Event. Proceedings of the ACM Internet Measurement Conference (Nov. 2016). [DOI] [PDF] Details
  • John Heidemann, Ricardo de O. Schmidt and Jan Harm Kuipers 2016. Anycast Latency: How Many Sites are Enough? Presentation at DNS-OARC Meeting. [PDF] Details
  • John Heidemann, Giovane C. M. Moura, Ricardo de O. Schmidt, and Wouter B. de Vries, Moritz Muller, Lan Wei and Christian Hesselman 2016. Anycast vs. DDoS: Evaluating Nov. 30. Presentation at DNS-OARC Meeting. [PDF] Details
  • Jelena Mirkovic, Genevieve Bartlett, John Heidemann, Hao Shi and Xiyue Deng 2016. Do You See Me Now? Sparsity in Passive Observations of Address Liveness (extended). Technical Report ISI-TR-2016-710. USC/Information Sciences Institute. [PDF] Details
  • Giovane C. M. Moura, Ricardo de O. Schmidt, John Heidemann, Wouter B. de Vries, Moritz Müller, Lan Wei and Christian Hesselman 2016. Anycast vs. DDoS: Evaluating the November 2015 Root DNS Event (extended). Technical Report ISI-TR-2016-709b. USC/Information Sciences Institute. [PDF] Details
  • Ricardo de O. Schmidt, John Heidemann and Jan Harm Kuipers 2016. Anycast Latency: How Many Sites Are Enough? Technical Report ISI-TR-2016-708. USC/Information Sciences Institute. [PDF] Details
  • Z. Hu, L. Zhu, J. Heidemann, A. Mankin, D. Wessels and P. Hoffman 2016. Specification for DNS over Transport Layer Security (TLS) . Technical Report 7858. Internet Request For Comments. [DOI] [PDF] Details
  • Abdul Qadeer, John Heidemann and Kensuke Fukuda 2016. Improving Long-term Accuracy of DNS Backscatter for Monitoring of Internet-Wide Malicious Activity (poster). Technical Report ISI-TR-2016-707. USC/Information Sciences Institute. [PDF] [Dataset] Details
  • Manaf Gharaibeh, Han Zhang, Christos Papadopoulos and John Heidemann 2016. Assessing Co-Locality of IP Blocks. Proceedings of the 19th IEEE Global Internet Symposium (San Francisco, CA, USA, Apr. 2016). [PDF] Details
  • Han Zhang, Manaf Gharaibeh, Spiros Thanasoulas and Christos Papadopoulos 2016. BotDigger: Detecting DGA Bots in a Single Network. Proceedings of the IEEE International Conference on Traffic Monitoring and Analysis (Louvain La Neuve, Belgium, Apr. 2016), 16–21. [DOI] Details
  • Liang Zhu, Zi Hu, John Heidemann, Duane Wessels, Allison Mankin and Nikita Somaiya 2016. T-DNS: Connection-Oriented DNS to Improve Privacy and Security (poster abstract). Technical Report ISI-TR-2016-706. USC/Information Sciences Institute. [PDF] Details
  • Calvin Ardi and John Heidemann 2016. AuntieTuna: Personalized Content-Based Phishing Detection. Proceedings of the NDSS Workshop on Usable Security (San Diego, California, USA, Feb. 2016). [PDF] [Code] Details
  • Han Zhang, Manaf Gharaibeh, Spiros Thanasoulas and Christos Papadopoulos 2016. BotDigger: Detecting DGA Bots in a Single Network. Technical Report CS-16-101. Colorado State University . Details
  • Manaf Gharaibeh, Han Zhang, Christos Papadopoulos and John Heidemann 2015. Assessing Co-Locality of IP Blocks. Technical Report CS-15-103. Colorado State University Department of Computer Science . [PDF] Details
  • Kensuke Fukuda and John Heidemann 2015. Detecting Malicious Activity with DNS Backscatter. Proceedings of the ACM Internet Measurement Conference (Tokyo, Japan, Oct. 2015), 197–210. [DOI] [PDF] [Dataset] Details
  • Kensuke Fukuda and John Heidemann 2015. Detecting Malicious Activity with DNS Backscatter (extended). Technical Report ISI-TR-2015-704. USC/Information Sciences Institute. [PDF] [Dataset] Details
  • Abdulla Alwabel, John Healy, John Heidemann, Brian Luu, Yuri Pradkin and Rasoul Safavian. 2015. Evaluating Externally Visible Outages. Technical Report ISI-TR-701. USC/Information Sciences Institute. [PDF] Details
  • Calvin Ardi and John Heidemann 2015. Poster: Lightweight Content-based Phishing Detection. Technical Report ISI-TR-2015-698. USC/Information Sciences Institute. [PDF] Details
  • Liang Zhu, Zi Hu, John Heidemann, Duane Wessels, Allison Mankin and Nikita Somaiya 2015. Connection-Oriented DNS to Improve Privacy and Security. Proceedings of the 36thIEEE Symposium on Security and Privacy (San Jose, Californa, USA, May 2015), 171–186. [DOI] [PDF] [Code] [Dataset] Details
  • Xun Fan, Ethan Katz-Bassett and John Heidemann 2015. Assessing Affinity Between Users and CDN Sites. Proceedings of the 7th IEEE International Workshop on Traffic Monitoring and Analysis (Barcelona, Spain, Apr. 2015). [DOI] [PDF] [Dataset] Details
  • Liang Zhu, Zi Hu, John Heidemann, Duane Wessels, Allison Mankin and Nikita Somaiya 2015. Connection-Oriented DNS to Improve Privacy and Security (extended). Technical Report ISI-TR-2015-695. USC/Information Sciences Institute. [PDF] [Code] Details
  • Liang Zhu, Zi Hu and John Heidemann 2015. Evaluation of Future DNSSEC Response Sizes at a Root and a TLD Server. [PDF] Details
  • Lin Quan, John Heidemann and Yuri Pradkin 2014. When the Internet Sleeps: Correlating Diurnal Networks With External Factors. Proceedings of the ACM Internet Measurement Conference (Vancouver, BC, Canada, Nov. 2014), 87–100. [DOI] [PDF] Details
  • John Heidemann 2014. Internet Populations (Good and Bad): Measurement, Estimation, and Correlation. Presentation at ICERM Workshop on Cybersecurity. [PDF] Details
  • Liang Zhu, Zi Hu, John Heidemann, Duane Wessels, Allison Mankin and Nikita Somaiya 2014. T-DNS: Connection-Oriented DNS to Improve Privacy and Security (extended). Technical Report ISI-TR-2014-693. USC/Information Sciences Institute. [PDF] [Code] Details
  • Lin Quan, John Heidemann and Yuri Pradkin 2014. When the Internet Sleeps: Correlating Diurnal Networks With External Factors (extended). Technical Report ISI-TR-2014-691b. USC/Information Sciences Institute. [PDF] Details
  • Lin Quan, John Heidemann and Yuri Pradkin 2014. When the Internet Sleeps: Correlating Diurnal Networks With External Factors (extended). Technical Report ISI-TR-2014-691. USC/Information Sciences Institute. [PDF] Details
  • Michael E. Fisk and Curtis L. Hash 2014. FileMap: Map-Reduce Program Execution on Loosely-Coupled Distributed Systems. Proceedings of the 4th International Workshop on Cloud Data and Platforms (Amsterdam, the Netherlands, Apr. 2014), to appear. [DOI] [Code] Details
  • Zi Hu, Liang Zhu, Calvin Ardi, Ethan Katz-Bassett, Harsha V. Madhyastha, John Heidemann and Minlan Yu 2014. The Need for End-to-End Evaluation of Cloud Availability. Proceedings of the Passive and Active Measurement Workshop (Marina del Rey, California, USA, Mar. 2014), 119–130. [DOI] [PDF] Details
  • Alefiya Hussain, Yuri Pradkin and John Heidemann 2013. Replay of Malicious Traffic in Network Testbeds. Proceedings of the 13th IEEE Conference on Technologies for Homeland Security (HST) (Waltham, Massachusetts, USA, Nov. 2013), (to appear). [PDF] Details
  • Matt Calder, Xun Fan, Zi Hu, Ethan Katz-Bassett, John Heidemann and Ramesh Govindan 2013. Mapping the Expansion of Google’s Serving Infrastructure. Proceedings of the ACM Internet Measurement Conference (Barcelona, Spain, Oct. 2013), 313–326. [PDF] Details
  • Matt Calder, Xun Fan, Zi Hu, Ethan Katz-Bassett, John Heidemann and Ramesh Govindan 2013. Mapping the Expansion of Google’s Serving Infrastructure. Technical Report TR 13-935. University of Southern California Computer Science Department. [PDF] Details
  • Lin Quan, John Heidemann and Yuri Pradkin 2013. Poster Abstract: Towards Active Measurements of Edge Network Outages. Proceedings of the Passive and Active Measurement Workshop (Hong Kong, China, Mar. 2013), 276–279. [DOI] [PDF] Details
  • John Heidemann 2013. Third-Party Measurement of Network Outages in Hurricane Sandy. Proceedings of the FCC Workshop on Network Resiliency (Brooklyn, New York, USA, Feb. 2013). [PDF] Details

For related publications, please see the ANT publications web page.

Software

  • antlink Manage a tree of git or other VC repositories with funky symlinks
  • babarchive Manage babarchives, checksumed directory trees that can be validated
  • babarchive Manage babarchives, checksumed directory trees that can be validated
  • cryptopANT CryptopANT is a C library for IP address anonymization using crypto-PAn algorithm, originally defined by Georgia Tech. The library supports anonymization and de-anonymization (provided you possess a secret key) of IPv4, IPv6, and MAC addresses. The software release includes sample utilities that anonymize IP addresses in text, but we expect most use of the library will be as part of other programs. The Crypto-PAn anonymization scheme was developed by Xu, Fan, Ammar, and Moon at Georgia Tech and described in "Prefix-Preserving IP Address Anonymization", Computer Networks, Volume 46, Issue 2, 7 October 2004, Pages 253-272, Elsevier. Our library is independent (and not binary compatible) of theirs.
  • dag scrubber Dag Scrubber is our tool for scrubbing packets of user data and optionally doing IP address anonymization. It supports both pcap and ERF format ("dag", giving the legacy name).
  • dnsanon extract DNS traffic from pcap to text with optionally anonymization
  • dnsanon_rssac Dnsanon_rssac is an implementation of RSSAC-002v2 processing for DNS statistics
  • LDplayer/dns-replay-client Replay DNS queries against a DNS server with correct timing and optionally log timing or latency.
  • LDplayer/dns-replay-controller Distribute DNS query stream and to queriers (dns-replay-client).
  • LDplayer/dns-query-mutator Change DNS queries in a network trace file and generate binary input for dns-replay-{controller,client}.
  • LDplayer/dns-replay-proxy A proxy that helps to emulate DNS hierarchy in DNS trace replay.
  • LDplayer/dns-route-setup A set of scripts that set up port-based routing and dns-replay-proxy for replaying queries against a recursive server in LDplayer.
  • LDplayer/dns-zone-constructor A set of scripts that generate zone files in order to replay queries against a recursive server in LDplayer.
  • LANDER Trace Software LANDER Trace Capture software handles for packet capture, scrubbing, and triggering user-provided scripts
  • timefind and indexer Software to handle indexing and selection of multiple network data types based on a given time range.

See also the ANT software web page.

Datasets

The primary goal of Retro-Future is to develop new network measurement capabilities. When we generate new datasets we can release as a side-effect of this process we will announce them here.