{"id":1760,"date":"2021-07-22T08:43:00","date_gmt":"2021-07-22T15:43:00","guid":{"rendered":"https:\/\/ant.isi.edu\/blog\/?p=1760"},"modified":"2022-01-07T11:46:03","modified_gmt":"2022-01-07T19:46:03","slug":"new-conference-paper-efficient-processing-of-streaming-data-using-multiple-abstractions","status":"publish","type":"post","link":"https:\/\/ant.isi.edu\/blog\/?p=1760","title":{"rendered":"new conference paper \u201cEfficient Processing of Streaming Data using Multiple Abstractions\u201d at IEEE Cloud"},"content":{"rendered":"\n

We have published a new paper \u201cEfficient Processing of Streaming Data using Multiple Abstractions<\/a>\u201d at the IEEE Cloud 2021 conference. (to be available at\u00a0https:\/\/conferences.computer.org\/cloud\/2021\/<\/a>)<\/p>\n\n\n\n

\"\"<\/a>
We show that one framework can efficiently support multiple abstractions. We provide three abstractions of Block, Windowed, and Stateful streaming and demonstrate that many application classes can be developed with ease, correctness, and low processing latency.<\/figcaption><\/figure>\n\n\n\n

From the abstract of our paper:<\/p>\n\n\n\n

Large websites and distributed systems employ sophisticated analytics to evaluate successes to celebrate and problems to be addressed. As analytics grow, different teams often require different frameworks, with dozens of packages supporting with streaming and batch processing, SQL and no-SQL. Bringing multiple frameworks to bear on a large, changing dataset often create challenges where data transitions\u2014these impedance mismatches can create brittle glue logic and performance problems that consume developer time. We propose Plumb, a meta-framework that can bridge three different abstractions to meet the needs of a large class of applications in a common workflow. Large-block streaming (Block-Streaming) is suitable for single-pass applications that care about the temporal and spatial locality. Windowed-Streaming allows applications to process a group of data and many reductions. Stateful-Streaming enables applications to keep a long-term state and always-on behavior. We show that it is possible to bridge abstractions, with a common, high-level workflow specification, while the system transitions data batch processing and block- and record-level streaming as required. The challenge in bridging abstractions is to minimize latency while allowing applications to select between sequential and parallel operation, while handling out-of-order data delivery, component failures, and providing clear semantics in the face of missing data. We demonstrate these abstractions evaluating a 10-stage workflow of DNS analytics that has been in production use with Plumb for 2 years, comparing to a brittle hand-built system that has run for more than 3 years.<\/em><\/p><\/blockquote>\n\n\n\n

This conference paper is joint work of Abdul Qadeer and  John Heidemann from USC\/ISI.<\/p>\n\n\n\n

Plumb is open source software and will be available at: https:\/\/ant.isi.edu\/software\/plumb\/index.html<\/a><\/p>\n\n\n\n

Update 2021-09-26: This paper was given a “special paper award<\/a>” at IEEE Conference on Cloud Computing 2021! Congratulations, Abdul!<\/em><\/p>\n","protected":false},"excerpt":{"rendered":"

We have published a new paper \u201cEfficient Processing of Streaming Data using Multiple Abstractions\u201d at the IEEE Cloud 2021 conference. (to be available at\u00a0https:\/\/conferences.computer.org\/cloud\/2021\/) From the abstract of our paper: Large websites and distributed systems employ sophisticated analytics to evaluate successes to celebrate and problems to be addressed. As analytics grow, different teams often require […]<\/p>\n","protected":false},"author":634,"featured_media":0,"comment_status":"closed","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[284,282,1],"tags":[214,85,24,219,122,58,8,318,29,273,16,57],"class_list":["post-1760","post","type-post","status-publish","format-standard","hentry","category-papers-publications","category-publications","category-uncategorized","tag-bigdata","tag-cloud","tag-conference","tag-gawseed","tag-ieee","tag-isi","tag-lander","tag-lanic","tag-network-traffic","tag-plumb","tag-software","tag-usc"],"_links":{"self":[{"href":"https:\/\/ant.isi.edu\/blog\/index.php?rest_route=\/wp\/v2\/posts\/1760","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/ant.isi.edu\/blog\/index.php?rest_route=\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/ant.isi.edu\/blog\/index.php?rest_route=\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/ant.isi.edu\/blog\/index.php?rest_route=\/wp\/v2\/users\/634"}],"replies":[{"embeddable":true,"href":"https:\/\/ant.isi.edu\/blog\/index.php?rest_route=%2Fwp%2Fv2%2Fcomments&post=1760"}],"version-history":[{"count":7,"href":"https:\/\/ant.isi.edu\/blog\/index.php?rest_route=\/wp\/v2\/posts\/1760\/revisions"}],"predecessor-version":[{"id":1827,"href":"https:\/\/ant.isi.edu\/blog\/index.php?rest_route=\/wp\/v2\/posts\/1760\/revisions\/1827"}],"wp:attachment":[{"href":"https:\/\/ant.isi.edu\/blog\/index.php?rest_route=%2Fwp%2Fv2%2Fmedia&parent=1760"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/ant.isi.edu\/blog\/index.php?rest_route=%2Fwp%2Fv2%2Fcategories&post=1760"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/ant.isi.edu\/blog\/index.php?rest_route=%2Fwp%2Fv2%2Ftags&post=1760"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}