Stream Processing | Event Driven | Real Time

Conference Program

September 3
Training
September 4 & 5
Conference
  • Ecosystem
  • Keynote
  • Operations
  • Technology Deep Dive
  • Use Case
  • Research
  • Other

Flink Forward Berlin 2018 continues on September 4-5 with two days of keynotes and technical talks including Apache Flink® use cases, internals, growth of the Flink ecosystem, and many more topics on stream processing and real-time analytics. After an inspiring day of technical sessions we invite you to join our Flink Fest in the evening on September 4.

TUESDAY September 4

Location
Kesselhaus
Maschinenhaus
Moon Lounge
Palais Atelier
Kino 3
8:30am

Check-In & Breakfast

8:30am - 9:30amMaschinenhaus, Palais

Check-In & Breakfast

September 4: 8:30am - 9:30am
Speakers

Go to: Speakers or Schedule

9:30am

Keynote sessions - Coming soon

9:30am - 11:00amKino 3

Keynote sessions - Coming soon

September 4: 9:30am - 11:00am
Speakers

Go to: Speakers or Schedule

11:00am

Coffee Break

11:00am - 11:30amPalais

Coffee Break

September 4: 11:00am - 11:30am
Speakers

Go to: Speakers or Schedule

11:30am
12:00pm
12:30pm

Failure is not fatal: what is your recovery story?

Steven Wu, Netflix

11:30am - 12:10pmKesselhaus

Failure is not fatal: what is your recovery story?

Failures are inevitable. How can we recover a Flink job from outage? How do we reprocess data from outage period? What are the implications to downstream consumers? These are important questions that we need to answer when running Flink for critical data processing applications. We implemented two solutions for our stream processing platform: (1) use data warehouse, like Hive, as backfill source (2) rewind Flink job using external checkpoint. We will describe both solutions in details, and discuss the pros and cons of each approach. We will also take a look at some of the caveats to watch out for.

Level: Intermediate

September 4: 11:30am - 12:10pm
Speakers
Steven Wu Netflix

Steven Wu is a software engineer at Netflix. He is working on real-time data infrastructure that powers massive data ingestion pipeline and stream processing platform. Previously he was working on cloud platform that builds the foundation for Netflix’s cloud-native microservice architecture. He is passionate about building scalable distributed system and empowering people with data.

Technology Deep Dive

Failure is not fatal: what is your recovery story?

Go to: Speakers or Schedule

Deploying a secured Flink cluster on Kubernetes

Edward Alexander Rojas Clavijo, IBM

11:30am - 12:10pmMaschinenhaus

Managing Flink operations at GoJek

Ravi Suhag, GoJek
Sumanth Nakshatrithaya, GoJek

11:30am - 12:10pmMoon Lounge

Using a sharded Akka distributed data cache as a Flink pipelines integration buffer

Andrew Torson, Walmart Labs

11:30am - 12:10pmPalais Atelier

September 4:
Speakers

Go to: Speakers or Schedule

Exploiting Apache Flink's Stateful Operators

Olga Slenders, ING
Gijsbert van Vliet, ING

12:20pm - 1:00pmKesselhaus

Upgrading Apache Flink Applications: State of the Union

Gordon Tai, data Artisans

12:20pm - 1:00pmMaschinenhaus

Our successful journey with Flink

Lasse Nedergaard, Trackunit A/S

12:20pm - 1:00pmMoon Lounge

Ask-Me-Anything

Interactive session

12:20pm - 1:00pmPalais Atelier

Ask-Me-Anything

September 4: 12:20pm - 1:00pm
Speakers

Go to: Speakers or Schedule

1:00pm

Lunch Break

1:00pm - 2:00pmPalais

Lunch Break

September 4: 1:00pm - 2:00pm
Speakers

Go to: Speakers or Schedule

2:00pm

Taming large-state to join datasets for Personalization

Shriya Arora, Netflix

2:00pm - 2:20pmKesselhaus

Taming large-state to join datasets for Personalization

Streaming engines like Apache Flink are redefining ETL and data processing. Data can be extracted, transformed, filtered and written out in real-time with an ease matching that of batch processing. However the real challenge of matching the prowess of batch ETL remains in doing joins, in maintaining state and to have the data be paused or rested dynamically. Netflix has a microservices architecture. Different microservices serve and record different kind of user interactions with the product. Some of these live services generate millions of events per second, all carrying meaningful but often partial information. Things start to get exciting when we want to combine the events coming from one high-traffic microservice to another. Joining these raw events generates rich datasets that are used to train the machine learning models that serve Netflix recommendations. Historically we have done this joining of large volume data-sets in batch. However we asked ourselves if the data is being generated in real-time, why must it not be processed downstream in real time? Why wait a full day to get information from an event that was generated a few mins ago? In this talk, we will share how we solved a complex join of two high-volume event streams using Flink. We will talk about maintaining large state, fault tolerance of a stateful application and strategies for failure recovery.

Level: Intermediate

September 4: 2:00pm - 2:20pm
Speakers
Shriya Arora Netflix

Shirya works on the Data engineering team for Personalization. Which, among other things, delivers recommendations made for each user. The team is responsible for the data that goes into training and scoring of the various machine learning models that power the Netflix homepage. They have been working on moving some of our core datasets from being processed in a once-a-day daily batch ETL to being processed in near-real time using Apache Flink. Before Netflix, she was at Walmart Labs, where she helped build and architect the new generation item-setup, moving from batch processing to stream. They used Storm-Kafka to enable a micro-services architecture that can allow for products to be updated near real-time as opposed to once-a-day update on the legacy framework.

Use Case

Taming large-state to join datasets for Personalization

Go to: Speakers or Schedule

Monitoring Flink with Prometheus

Maximilian Bode, TNG Tech

2:00pm - 2:20pmMaschinenhaus

Efficient Window Aggregation with Stream Slicing

Jonas Traub, TU Berlin
Philipp Grulich, DFKI

2:00pm - 2:20pmMoon Lounge

Efficient Window Aggregation with Stream Slicing

Computing aggregates over windows is at the core of virtually every stream processing job. Typical stream processing applications involve overlapping windows and, therefore, cause redundant computations. Several techniques prevent this redundancy by sharing partial aggregates among windows. However, these techniques do not support out-of-order processing and session windows. Out-of-order processing is a key requirement to deal with delayed tuples in case of source failures such as temporary sensor outages. Session windows are widely used to separate different periods of user activity from each other. Current versions of Apache Flink use Window Buckets to process stream aggregations with session windows and out-of-order tuples. This Approach does not share partial aggregates among overlapping windows. In our talk, we present Scotty, a high throughput operator for window discretization and aggregation in Apache Flink. Scotty splits streams into non-overlapping slices and computes partial aggregates per slice. These partial aggregates are shared among all overlapping windows including session windows. Scotty introduces the first slicing technique which (1) enables stream slicing for session windows in addition to tumbling and sliding windows and (2) processes out-of-order tuples efficiently. Scotty was first published at ICDE 2018 (http://www.user.tu-berlin.de/powibol/assets/publications/traub-scotty-icde-2018.pdf).

Level: Intermediate

September 4: 2:00pm - 2:20pm
Speakers
Jonas Traub TU Berlin

Jonas is a Research Associate at TU Berlin and a PhD candidate supervised by Volker Markl. His research interests include data stream processing, sensor data analysis, and data acquisition from sensor nodes. All his publications are available on http://www.user.tu-berlin.de/powibol/. He wrote my master thesis during a year abroad at the Royal Institute of Technology (KTH) and the Swedish Institute of Computer Science (SICS) / RISE in Stockholm under Supervision of Seif Haridi and Volker Markl and advised by Paris Carbone and Asterios Katsifodimos. He graduated with a M.Sc. in computer science in April 2015 at TU-Berlin. Prior to that, he received his B.Sc degree at Baden-Württemberg Cooperative State University (DHBW Stuttgart) and worked several years at IBM in Germany and the USA. He is a participant of Software Campus and Alumnus of Studienstiftung des deutschen Volkes and Deutschlandstipendium.

Research

Efficient Window Aggregation with Stream Slicing

Philipp Grulich German Research Centre for Artificial Intelligence

Philipp is a computer science Master’s student at the Technische Universität Berlin, specializing in big data analytics systems. Besides the university, he has worked for several companies and collected experiences in frontend and backend software development. At the German Research Center for Artificial Intelligence, he joined a streaming systems oriented research project involving Apache Flink as a research assistant.

Research

Efficient Window Aggregation with Stream Slicing

Go to: Speakers or Schedule

Data lossless event time streaming processing for revenue calculation

Oleksandr Nitavskyi, Criteo

2:00pm - 2:20pmPalais Atelier

Data lossless event time streaming processing for revenue calculation

One of the main characteristics of the good streaming pipeline is correctness for event time processing. Real challenges become when such pipeline should be resilient to different types of failures. In this talk, we describe how Criteo runs Flink on one of the biggest Yarn clusters in Europe and computes 100k messages per second to acknowledge revenue of our platform within the delay of 5 minutes. Real-time revenue monitoring system calculates data under 1% of discrepancies and minimizes business impact in case of revenue anomalies.

Level: Beginner

September 4: 2:00pm - 2:20pm
Speakers

Staffs Software Engineer in Criteo, Currently technical lead of Invalid Traffic detection team. Worked for Grammarly in the past. Likes JVM and functional programming. Fun of improving development productivity.

Use Case

Data lossless event time streaming processing for revenue calculation

Go to: Speakers or Schedule

September 4:
Speakers

Go to: Speakers or Schedule

2:30pm

Running Flink Data-Connectors at Scale

Vipul Singh, Yelp

2:30pm - 2:50pmKesselhaus

Matchmaking in multiplayer games with Apache Flink

Vladimír Schäfer, King

2:30pm - 2:50pmMaschinenhaus

Stream Loops on Flink: Reinventing the wheel for the streaming era

Paris Carbone, KTH

2:30pm - 2:50pmMoon Lounge

Stream Loops on Flink: Reinventing the wheel for the streaming era

You have probably heard that stream processing subsumes batch workloads, a valid but not yet fully implemented claim. Our lab research aims to fulfil this dream and delve further into the deep world of iterative processes, a fundamental building block for graph and machine learning algorithms. Yet, a building block that is missing from your stream pipelines today. In this talk, we will investigate why bulk and stale synchronous iterative models are nothing more than a special case of out-of-order stream processing, the paradigm behind your ultra fast watermark-based window aggregations on Flink and Beam. Next, we will examine how watermarks can be extended to incorporate more metrics for tracking iterative progress as well as the necessary structured graph modifications (spoiler alert: loops) that can make our lives easier. Finally, we will demonstrate how on top of these primitives we can execute scalable multi-pass window aggregations with purgeable and persistent managed state as well as robust flow control and several domain specific applications such as Vertex-centric graph aggregations and Stochastic Gradient Descent on stream windows.

Level: Advanced

September 4: 2:30pm - 2:50pm
Speakers
Paris Carbone KTH Royal Institute of Technology in Stockholm

Paris Carbone is a Flink committer and a senior computer scientist within the special intersection of distributed systems, data management and programming systems. Paris is currently the tech lead of the ‘Continuous Deep Analytics’ project at KTH and RISE SICS in Sweden, investigating how intermediate programming languages and hardware acceleration will make data streaming the dominant end-to-end architecture for critical and complex decision making. At night, you can catch Paris performing with his jazz quintet at the oldest neighbourhoods of Stockholm.

Research

Stream Loops on Flink: Reinventing the wheel for the streaming era

Go to: Speakers or Schedule

Using Apache Flink for Smart Cities: Warsaw case study

Piotr Wawrzyniak, Orange Polska
Jarosław Legierski, Orange Polska

2:30pm - 2:50pmPalais Atelier
3:00pm
3:30pm
4:00pm
4:30pm

Tuning Flink for Robustness and Performance

Stefan Richter, data Artisans

3:00pm - 3:40pmKesselhaus

Patterns of Streaming Applications

Level: Intermediate

3:00pm - 3:40pmMaschinenhaus

Patterns of Streaming Applications

Stream processing engines are becoming pivotal in analyzing data. They have evolved beyond a data transport and simple processing machinery, to one that's capable of complex processing. The necessary features and building blocks of these engines are well known. And most capable engines have a familiar Dataflow based programming model. As with any new paradigm, building streaming applications requires a different mindset and approach. Hence there is a need for identifying and describing patterns and anti-patterns for building these applications. Currently this mindshare is scarce. Drawn from my experience working with several engineers within and outside of Netflix, the talk will present the following: A blueprint for streaming data architecture and a review of desirable features of a streaming engine Streaming Application patterns and anti-patterns Use case and concrete examples using Flink.

Level: Intermediate

September 4: 3:00pm - 3:40pm
Speakers
Monal Daxini Netflix

Monal Daxini is the Tech Lead for Stream Processing platform for business insights at Netflix. He helped build the petabyte scale Keystone pipeline running on this Flink powered platform. He also helped define the vision for this platform. He has over 16 years of experience building scalable distributed systems. He has a Masters in Computer Science, and has shared his thoughts and learnings at several conferences.

Use Case

Patterns of Streaming Applications

Go to: Speakers or Schedule

Stream Join in Flink: from Discrete to Continuous

Xingcan Cui, Shandong University

3:00pm - 3:40pmMoon Lounge

Stream Join in Flink: from Discrete to Continuous

As a distributed stream processing engine, Flink provides users with convenient operators to manipulate data on the fly. Among all these operators, join could be the most complicated one as it requires the capability to cross-analyze various sources simultaneously. In this talk, we aim to give a comprehensive introduction to the stream join in Flink. Specifically, we'll first provide an overview of the different join types and which of them are currently supported by Flink DataStream and Table & SQL APIs. Then we'll discuss some key points when performing distributed stream join. After that, we'd like to focus the rationale and implementation details of the time-windowed join launched in version 1.4. Since there are still a lot of improvements can be made, we'll end our talk by sharing some proposals for the future work.

Level: Intermediate

September 4: 3:00pm - 3:40pm
Speakers
Xingcan Cui Shandong University

Xingcan Cui, who is interested in database and stream processing, is a committer of the Apache Flink project. He has just finished his Ph.D. study under the supervision of Prof. Xiaohui Yu at Shandong University, China and will continue his research as a postdoc at York University, Canada.

Research

Stream Join in Flink: from Discrete to Continuous

Go to: Speakers or Schedule

"Play it again, Sam": Bookmarking, Slicing, and Replaying Unbounded Data Streams for Analytics Applications

Raúl Gracia-Tinedo, DellEMC

3:00pm - 3:40pmPalais Atelier

"Play it again, Sam": Bookmarking, Slicing, and Replaying Unbounded Data Streams for Analytics Applications

Pravega is a novel storage system that exposes data stream as a first-class abstraction as opposed to objects and files. With Pravega, a stream is a consistently ordered, durable, available and elastic series of data events. Pravega is designed to ingest, store and serve potentially unbounded data streams with high performance while adapting to workload fluctuations thanks to auto-scaling. Developers can extract value and insights out of stream data by connecting Pravega with a stream processor: Apache Flink is a strong candidate due to its advanced stream processing features. We provide a ready-to-use connector that enables Flink jobs to process data stored in Pravega in stream (ordered) or batch (unordered) fashion. For instance, reading in batch a slice of old data events, rewinding/fast-forwarding parts of a stream, or bookmarking a specific point of a stream that is being read are common needs for developers that become simple tasks with Pravega. In this talk, we present the main abstraction to support such operations over streams: stream cut, a compact data structure that represents an event boundary across a collection of streams. Developers can easily instruct applications to work with stream cuts to go back and forth on a data stream; they are cheap to create and allow efficient seeks in a stream. In Pravega, developers may use pairs of stream cuts to replay arbitrarily old stream slices in a more natural way than batch loading data from traditional file- and object-based storage systems. Stream cuts can be created to bookmark a stream based on time (e.g., data created in a single day in a company), event references (e.g., series of events for which an anomaly has been detected), or any other aspect. Moreover, stream cuts complement Flink features such as savepoints; developers that snapshot the state of an application can also capture the precise range of events used as input to reach such a state. In this talk, we illustrate how this simple yet powerful abstraction can be exploited with examples, including Flink samples.

Level: Intermediate

September 4: 3:00pm - 3:40pm
Speakers

Raúl Gracia-Tinedo is a senior software engineer at DellEMC working for Pravega: a novel distributed storage system for data streams. Prior to joining DellEMC, he has worked as a postdoc in the context of European research projects (FP7 CloudSpaces, H2020 IOStack) and as intern at IBM Research and Tel-Aviv University. He holds a Ph.D. in Computer Engineering (2015, outstanding thesis award) from Universitat Rovira i Virgili (Spain). Raúl is a highly motivated researcher and engineer interested in distributed systems, cloud storage, and data analytics, with more than 20 papers.

Ecosystem

“Play it again, Sam”: Bookmarking, Slicing, and Replaying Unbounded Data Streams for Analytics Applications

Go to: Speakers or Schedule

Coffee Break in Palais

3:40pm - 4:10pmPalais

Coffee Break in Palais

September 4: 3:40pm - 4:10pm
Speakers

Go to: Speakers or Schedule

Threading Needles in a Haystack: Sessionizing the Uber firehose in realtime

Amey Chaugule, Uber

4:10pm - 4:30pmKesselhaus

Threading Needles in a Haystack: Sessionizing the Uber firehose in realtime

One Uber's Marketplace team we're tasked with efficiently matching our riders and driver partners in real time. To that end, we we employ various systems within the ride sharing marketplace such as dynamic pricing (popularly known as surge), demand modeling and forecasting systems and health monitoring systems in order to ensure optimal marketplace efficiency. One of the key data systems underpinning Uber's Marketplace is the Rider Sessions pipeline which tracks the lifetime of a singular Uber trip in realtime, capturing rider interaction with various Uber systems, from the pricing engine to dispatch systems all the way until a trip end. In our talk we discuss the evolution of the Rider Session state machine at Uber and challenges involved in managing realtime stateful streaming pipeline across half a dozen event streams and millions of riders who use Uber every day. We plan to delve into various aspects of the running the job in production such as managing state checkpointing, monitoring, experiences moving the job from Spark Streaming to Flink and ensuring low latency to downstream systems.

Level: Intermediate

September 4: 4:10pm - 4:30pm
Speakers

Amey is a Senior Software Engineer on Uber’s Marketplace Data Intelligence team where he works on the stateful streaming and geo-spatial data systems that power various applications ranging from health monitoring, forecasting to dynamic pricing within Uber’s rider sharing marketplace. He’s been dealing with thorny issues around streaming pipelines and state ever since he started his career working on Yahoo’s ad tech systems in 2011 when Apache Pig and Storm were the state-of-the-art. He holds a B.S/M.S in Electrical and Computer Engineering from the University of Illinois at Urbana-Champaign.

Use Case

Threading Needles in a Haystack: Sessionizing the Uber firehose in realtime

Go to: Speakers or Schedule

TBA

4:10pm - 4:30pmMaschinenhaus

TBA

September 4: 4:10pm - 4:30pm
Speakers

Go to: Speakers or Schedule

Approximate standing queries on Stream Processing

Tobias Lindener, RISE/KTH

4:10pm - 4:30pmMoon Lounge

Approximate standing queries on Stream Processing

Data analytics in its infancy has taken off with the development of SQL. Yet, at web-scale, even simple analytics queries can prove challenging within (Distributed-) Stream Processing environments. Two such examples are Count and Count Distinct. Because of the key-oriented nature of these queries, traditionally such queries would result in ever increasing memory demand. Through approximation techniques with fixed-size memory consumption, said tasks are feasible and potentially more resource efficient within streaming systems. This is demonstrated by integrating Yahoo Data Sketches on Apache Flink. The evaluation highlights the resource efficiency as well as the challenges of approximation techniques (e.g. varying accuracy) and potential for tuning depending on the dataset. Furthermore, challenges in integrating the components within the existing Streaming interfaces(e.g. Table API) and stateful processing are presented.

Level: Intermediate

September 4: 4:10pm - 4:30pm
Speakers
Tobias Lindener RISE/KTH

Tobias previously worked as an IT-Consultant for Internet of Things at IBM before he started a pan-european Master Program in Data Science

Research

Approximate standing queries on Stream Processing

Go to: Speakers or Schedule

Interactive session

4:40pm - 5:00pmPalais Atelier

Interactive session

September 4: 4:40pm - 5:00pm
Speakers

Go to: Speakers or Schedule

Activities: TBC

4:10 - 4:30pm

Activities: TBC

September 4: 4:10 - 4:30pm
Speakers

Go to: Speakers or Schedule

A streaming Quantitative Analytics engine

Dr Raj Subramani, Flumaion Ltd

4:40pm - 5:00pmKesselhaus

A streaming Quantitative Analytics engine

The application of Quantitative Analytics to trades for the generation of Risk and P&L metrics has traditionally followed a batch based approach. Regulatory changes impose increasing demand for compute on financial institutions along with a growing demand for real time analytics due to increased volumes in eTrading across all asset classes

The talk is based on a use case for pricing Interest Rate Swaps, using Apache Beam, with a call to an external C++ analytics process. It describes the performance characteristics when operating in a non-cloud environment using Apache Flink as opposed to Google Cloud Dataflow

The talk will touch upon the subtle difference when operating across multiple runners. It will make suggestions on approaches to portability when architecting for a multi-runner operational environment.

Level: Intermediate

September 4: 4:40pm - 5:00pm
Speakers
Dr Raj Subramani Flumaion Ltd

Raj has over 20 years’ experience in Investment Banking. Raj has worked primarily in the Fixed Income business at HSBC, J P Morgan and Deutsche Bank. He started his career in FX Options at Citibank. Raj has a PhD in Engineering from the University of Newcastle upon Tyne and holds a certificate in Quantitative Finance from the CQF Institute in London.

Use Case

A streaming Quantitative Analytics engine

Go to: Speakers or Schedule

Real-time driving score service using flink

Kim Dongwon, SK Telecom

4:40pm - 5:00pmMaschinenhaus

MERA: Trading precision for performance

Niklas Semmler, TU Berlin

4:40pm - 5:00pmMoon Lounge

MERA: Trading precision for performance

Sudden spikes in load can be a source of disaster for stream processors. These spikes can reveal latent bottlenecks in otherwise well-balanced configurations and through them introduce backpressure, increase latency and reduce overall throughput. This problem is far from being solved. While the prevailing solution of dynamic scaling (i.e. the process of re-deploying the analytic job on an increased set of resources) offers relief in some cases, it has the disadvantage of requiring free resources to be available during the spike and additionally risks violating existing SLAs during the re-deployment phase. As an alternative, we have implemented MERA. MERA prioritizes items based on their value for the result of an analytic job. When a job is in risk of creating backpressure, MERA sheds items of lower priority at selected locations in the job graph, ensuring that the job stays within its SLA. MERA relies on Flink's internal metrics to restrict its overhead during normal operations to a minimum. In this talk we present MERA, a framework for trading in precision for performance. We demonstrate MERA via a custom performance monitor for Apache Flink.

Level: Intermediate

September 4: 4:40pm - 5:00pm
Speakers
Niklas Semmler TU Berlin

Niklas is doing his PhD with Prof. Anja Feldmann (TU Berlin). During his masters degree he became intrigued with the mechanics of large-scale networks. In his research he investigates means to speed up the analysis of network traces and develops optimizations for stream processors. He participated in the research project Berlin Big Data Center (bbdc.berlin) and has supervised multiple bachelor and master thesis in the environment of Apache Flink.

Research

MERA: Trading precision for performance

Go to: Speakers or Schedule

Interactive session

4:40pm - 5:00pmPalais Atelier

Interactive session

September 4: 4:40pm - 5:00pm
Speakers

Go to: Speakers or Schedule

5:00pm
5:30pm

Elastic Streams at Scale

Till Rohrmann, data Artisans
Joerg Schad, Mesosphere

5:10pm - 5:50pmKesselhaus

Elastic Streams at Scale

One of the big operational challenges when running streaming applications is to cope with varying workloads. Variations, e.g. daily cycles, seasonal spikes or sudden events, require that allocated resources are constantly adapted. Otherwise, service quality deteriorates or money is wasted. Apache Flink 1.5 includes a lot of enhancements to support full resource elasticity on cluster management frameworks such as Apache Mesos. With the latest version, it is now possible to build elastic applications which can be programmatically scaled up or down in order to react to changing workloads. In this talk, we will discuss recent improvements to Flink's deployment model which also enables full resource elasticity. In particular, we will discuss how Flink leverages cluster management frameworks, e.g. Mesos, and already-introduced features like scalable state to support elastic streaming applications. We will conclude the presentation with a short demo showing how a stateful Flink application can be rescaled on top of Mesos.

Level: Intermediate

September 4: 5:10pm - 5:50pm
Speakers
Till Rohrmann data Artisans

Till is a PMC member of Apache Flink and software engineer at dataArtisans. His main work focuses on enhancing Flink’s scalability as a distributed system. Till studied computer science at TU Berlin, TU Munich and École Polytechnique where he specialized in machine learning and massively parallel dataflow systems.

Technology Deep Dive

Elastic Streams at Scale

Flink as a Library (and still as a Framework)

Joerg Schad Mesosphere

Jörg is the technical lead for community projects at Mesosphere in San Francisco. In his previous life he implemented distributed and in memory databases and conducted research in the Hadoop and Cloud area during his PhD. His speaking experience includes various Meetups, international conferences, and lecture halls.

Technology Deep Dive

Elastic Streams at Scale

Go to: Speakers or Schedule

Runtime Improvement for Flink Batch Processing

Feng Wang, Alibaba

5:10pm - 5:50pmMaschinenhaus

Hardware-efficient Stream Processing

George Theodorakis

5:10pm - 5:30pmMoon Lounge

Hardware-efficient Stream Processing

In the era of big data and AI, many data-intensive applications, such as streaming, exhibit requirements that cannot be satisfied by traditional batch processing models. In response, distributed stream processing systems, such as Spark Streaming or Apache Flink, exploit the resources of a compute cluster for streaming applications. As with any distributed system, this raises the question of how efficiently these systems utilise the available hardware resources on each node. At the same time, with highly-parallel heterogeneous architectures becoming commonplace in data centres, stream processing systems can exploit previously unseen levels of parallel processing even from single nodes. A step towards these changes is our engine SABER, which exploits the parallelism that both multi-core CPUs and GPGPUs offer in a single node, to achieve high processing throughput while maintaining low latency. We use the modified version of Yahoo Streaming Benchmark to measure the performance of aforementioned systems and compare the performance of distributed and centralised computation. SABER processes 79 million tuples per second with 8 CPU cores, outperforming Flink (3x), Spark Streaming (7x) and StreamBox (7x). It exhibits better performance than a cluster-based deployment with 40 CPU cores.

However, even these results are not satisfactory, as there is still a large performance gap between handwritten code and current stream processing systems. By comparing a handwritten C++ program with the single-core implementations of these systems, we notice more than (2x) speedup that is hindered by the systems' design. Thus, we have to recalibrate the way we approach stream processing and focus on hardware-conscious techniques. In our current research, we have begun designing highly efficient streaming operator implementations that exploit superscalar execution and SIMD parallelism. We also envision to introduce compilation-based techniques to keep data in CPU registers as long as possible, while taking into consideration the non-uniform memory access (NUMA) caused by multiple CPU sockets on modern scale-up architectures.

Level: Beginner

September 4: 5:10pm - 5:30pm
Speakers
George Theodorakis Imperial College

George is a Ph.D. student in the Large-Scale Distributed Systems (LSDS) group @ Imperial College London, under the supervision of Dr. Peter Pietzuch. His Ph.D. is supported by a CDT HiPEDS scholarship. Prior to this, he was an undergraduate student in the Electrical and Computer Engineering department of National Technical University of Athens and conducted his thesis in affiliation with CSLab.

Research

Hardware-efficient Stream Processing

Go to: Speakers or Schedule

Ask-Me-Anything

Interactive session

5:00pm - 5:40pmPalais Atelier

Ask-Me-Anything

September 4: 5:00pm - 5:40pm
Speakers

Go to: Speakers or Schedule

6:00pm

Flink Fest

6:00pm - 9:00pmPalais

Wednesday September 5

Location
Kesselhaus
Maschinenhaus
Moon Lounge
Palais Atelier
Kino 3
9:00am

Check-In and Breakfast

9:00am - 10:00amMaschinenhaus, Palais

Check-In and Breakfast

September 5: 9:00am - 10:00am
Speakers

Go to: Speakers or Schedule

10:00am

Keynote sessions - Coming soon

10:00am - 11:00amKino 3

Keynote sessions - Coming soon

September 5: 10:00am - 11:00am
Speakers

Go to: Speakers or Schedule

11:00am

Coffee Break

11:00am - 11:30amPalais

Coffee Break

September 5: 11:00am - 11:30am
Speakers

Go to: Speakers or Schedule

11:30am
12:00pm
12:30pm

Python Streaming Pipelines with Beam on Flink

Thomas Weise, Lyft
Aljoscha Krettek, data Artisans

11:30am - 12:10pmKesselhaus

Real-time Processing of Noisy Data from Connected Vehicles

Robin Slomkowski, HERE

11:30am - 12:10pmMaschinenhaus

Real-time Processing of Noisy Data from Connected Vehicles

Modern vehicles are capable of producing large volumes of data from dozens of sensors. We will demonstrate the use of map matching to deal with noisy GPS information, and how to develop and deploy real-time sensor data processing applications on Flink using HERE’s Open Location Platform.

Level: Intermediate

September 5: 11:30am - 12:10pm
Speakers
Robin Slomkowski HERE Technologies

American living in Berlin, working with distributed systems since 1992, alternating between technical operations and software engineering. These days focusing on translating business needs into technology implementations both in developing HERE’s platform and helping HERE’s customers.

Use Case

Real-time Processing of Noisy Data from Connected Vehicles

Go to: Speakers or Schedule

Democratizing data in GO-JEK

Rohil Surana, GoJek
Prakhar Mathur, GoJek

11:30am - 12:10pmMoon Lounge

Democratizing data in GO-JEK

At GO-JEK, we build products that help millions of Indonesians commute, shop, eat and pay, daily. Data at GO-JEK doesn’t grow linearly with the business, but exponentially, as people start building new products and logging new activities on top of the growth of the business. We currently see 6 Billion events daily and rising. GO-JEK currently has 18+ products. Each and every team publishes events as Protobuf messages to Kafka clusters in order to have a well-defined schema and to ensure backward compatibility. This makes data available to all teams for different use-cases. In order to make sense out of this raw data, we needed to have some data aggregation pipeline. We found Flink to be useful. First use-case/requirement for real-time aggregation : We needed to implement Dynamic Surge Pricing. In order to do this, we needed real-time data of booking being created and drivers available to accept bookings per min per s2Id (http://s2geometry.io/) . We created two Flink jobs to achieve this. What are daggers? After the successful implementation of Surge pricing, we realised that real-time data aggregation can solve a lot of problems. So instead of creating different jobs for these use-cases by ourselves, we came up with a DIY solution for creating Flink jobs. We created a generic application knows as DAGGERS on top of Flink that could take parameters like the topic from which the user wants to read the data along with some options including watermark intervals, delays and parallelism. What is Datlantis? In order to give a DIY interface to the user, we created a portal called Datlantis which allows users to create and deploy massive, production-grade real-time data aggregation pipelines within minutes. Datlantis uses Flink's Monitoring REST API for communicating with the Flink cluster to monitor current jobs and deploying new ones. Now the users can just select Kafka topics from all Kafka clusters and write a simple SQL query on the UI which will spawn a new Flink job. Users also have the option to select one more Kafka data-stream in order to write JOINS query. This Flink job pushes data to InfluxDB, that enables the user to visualize their data on Grafana dashboards. Once the logic of the SQL is verified using the dashboard, the Flink job is then promoted to push the data to Kafka. The users can manage their Flink jobs on Datlantis. They can edit the jobs, stop or restart the jobs or change the job configurations. They can also see logs of their Flink jobs on Datlantis itself. The reasoning behind pushing data back to Kafka is so that the aggregated data is available for all the other teams. Our application FIREHOSE takes care of consuming this data from Kafka and pushing it to different sinks like relational DB sink, HTTP sink, GRPC sink, Influx sink, Redis sink etc. This data is then pushed to our cold storage which enables us to do historical analysis. Data Pipeline: Producer Apps → Kafka → Deserialization → DataStream → SQL → Result → Serialize → Kafka → Consumer Apps/InfluxDB This DIYness enabled not only the developers, but data analysts and even Product Managers to write simple queries and solve complex use cases like : - System uptime - Real-time customer segmentation - Fraud control - Dynamic surge pricing - Allocation metrics for city managers Deployment : We have different Kafka clusters depending upon the type and throughput of data, for example, transactional data, driver location pings, API logs etc. We have deployed multiple Flink clusters for different teams that have different configurations for resources allocated to task managers. On Datlantis, users have the option to create DAGGERS on their team’s Flink cluster. We used to deploy Flink on YARN using CHEF but have moved to Kubernetes using Helm charts as it is easier to scale. For Checkpointing on YARN based Flink clusters, we used the underlying HDFS, while for Kubernetes Flink clusters we use GlusterFS. Datlantis also takes care of automatically scaling the number of task managers on Kubernetes depending upon the number of jobs. Our build pipelines take care of having the latest production version of Dagger’s JAR available on Flink clusters. In case of JobManager failures, we have implemented a Kubernetes controller that listens for a JobManager restart and re-uploads the latest JAR to it. Metrics -> We use Influx, Grafana and Telegraf for monitoring For monitoring purposes, we use the StastD Reporter provided for Flink. Our monitoring stack consists of Telegraf as the StatsD Agent, InfluxDB as the time series database, Grafana for visualization and Kapacitor for alerts. Datlantis takes care of alert creation at the time of creation of a new Job. Users are provided with a Health dashboard that they can use to track the health of their Job. Alerts are sent to specific teams via their slack channels and pager duties.

Level: Intermediate

September 5: 11:30am - 12:10pm
Speakers
Rohil Surana GoJek

Rohil Surana works at Go-Jek as a Product Engineer in the Data-Engineering team. He has been solving problems on data streaming and data warehousing at Go-Jek. He prefers a hands-on aprroach for solving problems while learning at the same time. He loves to travel and learn about different places and try their food.

Ecosystem

Democratizing data in GO-JEK

Prakhar Mathur has completed his bachelors from Indian Institute of Technology, Jodhpur. He is currently working at GO-JEK as a Product Engineer with the Data Engineering team. He is working with the team solving problems regarding data publishing and making data easily available to the organisation.

Ecosystem

Democratizing data in GO-JEK

Go to: Speakers or Schedule

Detecting Patterns in Event Streams with Flink SQL

Dawid Wysakowicz, data Artisans

11:30am - 12:10pmPalais Atelier

September 5:
Speakers

Go to: Speakers or Schedule

Flink SQL in Action

Timo Walther, data Artisans

12:20pm - 1:00pmKesselhaus

September 5:
Speakers

Go to: Speakers or Schedule

Streaming Digital Fingerprints

Sebastian Czarnota, Centrum Bezpieczeństwa Cyfrowego S.A.

12:20pm - 1:00pmMoon Lounge

Streaming Digital Fingerprints

Authorisation is typically associated with a single act of “logging in”. But nobody likes to login too often, so most websites have a “remember me” option. But it’s not very safe to be constantly logged in. How to accommodate contradictory goals of good UX and high security? At Digital Fingerprints, we believe it can be solved using passive biometrics in tandem with stream processing. During the talk you can learn what technical challenges occured during designing and implementing continuous authorisation solution that uses thousands of machine learning models to provide seamless user experience with increased security. Key takeaways: • What is passive biometrics? • How continuous authorisation works? • How to meet tight latency requirements? • Is it possible to serve thousands of machine learning models? • Ways to provide privacy of data and privacy of models. • How Apache Flink fits into that kind of system?

Level: Intermediate

September 5: 12:20pm - 1:00pm
Speakers
Sebastian Czarnota Centrum Bezpieczeństwa Cyfrowego S.A.

Sebastian Czarnota is a Principal Streaming Architect at Digital Fingerprints (Centrum Bezpieczeństwa Cyfrowego S.A.), Polish cybersecurity startup working on continous authorisation solution (aka behavioural biometry) for financial sector. Sebastian is responsible for both system architecture and data flow design. His experience includes working on acquisition, analysis and presentation of sec.gov data as well as research of algorithmic trading on fundamental data retrieved from aforementioned source. Sebastian also worked in Samsung R&D on secure bootloader project. He finished Military University of Technology with speciality in Cryptology with distinction. He loves feeding Flink with Acorns.

Use Case

Streaming Digital Fingerprints

Go to: Speakers or Schedule

Running Flink at Enterprise scale

Jorn Eilander, ING

12:20pm - 1:00pmPalais Atelier
1:00pm

Lunch Break

1:00pm - 2:00pmPalais

Lunch Break

September 5: 1:00pm - 2:00pm
Speakers

Go to: Speakers or Schedule

2:00pm
2:30pm
3:00pm
3:30pm
4:00pm
4:30pm

Anomaly Detection Engine for Cloud Activities using Flink

Yonatan Most, Microsoft
Avihai Berkovitz, Microsoft

2:00pm - 2:40pmKesselhaus

Streaming topic model training and inference with Apache Flink

Suneel Marthi, Amazon
Joey Frazee, Databricks

2:00pm - 2:40pmMaschinenhaus

The Apache Way! ... ?

Isabel Drost-Fromm, Europace AG

2:00pm - 2:40pmMoon Lounge

The Apache Way! ... ?

To quote http://www.apache.org/foundation - “The mission of the Apache Software Foundation (ASF) is to provide software for the public good. We do this by providing services and support for many like-minded software project communities of individuals who choose to join the ASF.” The ASF provides a home to several hundred OSS projects. Likely, if you are developing software on a daily basis you did run into one of those projects at some point in time. If you have been digging a bit deeper, you may have encountered the phrase “The Apache Way” - often used in relation to how projects at the ASF should work. Depending on who you ask, very likely you will get different answers as to what actual meaning of “The Apache Way” is. This talk tries to give a glimpse behind the scenes - with a strong focus on what the implications of best practices advertised at the ASF are for downstream users of it’s projects. It's all about pulling you in to become active and contribute yourself.

Level: Beginner

September 5: 2:00pm - 2:40pm
Speakers
Isabel Drost-Fromm Europace AG

Isabel Drost-Fromm is Open Source Strategist at Europace AG Germany. She’s a member of the Apache Software Foundation, co-founder of Apache Mahout and mentored several incubating projects. Isabel is interested in all things FOSS, search and text mining with a decent machine learning background. True to the nature of people living in Berlin she loves having friends fly in for a brief visit – as a result she co-founded and is still one of the creative heads behind Berlin Buzzwords, a tech conference on all things search, scale and storage.

Ecosystem

The Apache Way! … ?

Go to: Speakers or Schedule

Assisting millions of active users in real-time

Krzysztof Zarzycki, GetInData
Alexey Brodovshuk, Kcell

2:00pm - 2:40pmPalais Atelier

Assisting millions of active users in real-time

Nowadays many companies become data rich and intensive. They have millions of users generating billions of interactions and events per day. These massive streams of complex events can be processed and reacted upon to e.g. offer new products, next best actions, communicate to users or detect frauds, and quicker we can do it, the higher value we can generate. Our presentation will be based on our recent experience in building a real-time data analytics platform for telco events. This platform has been jointly built by GetInData and Kcell - the leading telco in Kazakhstan - in just a few months and it currently runs in production at the scale of 10M subscribers and 160K events per second on a still small cluster. It's used as a backbone for personalized marketing campaigns, detecting frauds, cross-sell & up-sell by following the behavior of millions of users in real-time and reacting to it instantly. We will share how we build such platform using current best of breed open-source projects like Flink, Kafka, and Nifi. We won't skimp on the details how we designed and optimized our Flink applications for high load and performance. We will also describe challenges that we faced during development and try to provide some tips what one should pay attention to when developing similar solutions, not only for telco, but also for banks, e-commerce, IoT and other industries.

Level: Intermediate

September 5: 2:00pm - 2:40pm
Speakers
Krzysztof Zarzycki GetInData

Krzysztof is an architect, engineer and researcher of solutions that take advantage of Big Data technologies, like advanced analytics, decision automation systems or recommendation engines. He is a Big Data geek working with those technologies, HPC, distributed systems and machine learning for over 7 years, previously in companies like Netezza/IBM, Hadapt/Teradata and now as GetInData experts team member. He likes to work full-stack going from architecting solution through engineering down to installing, troubleshooting and monitoring. Now specializes in scalable real-time analytics solutions.

Use Case

Assisting millions of active users in real-time

Ten years of experience in software development field for one of the largest telecom operators in Kazakhstan. Interested in high load and big data projects. Technical lead for real time event processing and data lake projects in Kcell.

Use Case

Assisting millions of active users in real-time

Go to: Speakers or Schedule

September 5:
Speakers

Go to: Speakers or Schedule

Upshot: distributed tracing using Flink

Brian Wolfe, Airbnb

2:50pm - 3:10pmKesselhaus

TBA

2:50pm - 3:10pmMaschinenhaus

TBA

September 5: 2:50pm - 3:10pm
Speakers

Go to: Speakers or Schedule

Flink Positive/Flinking Positive

Caito Scherr, New Relic
Nikolas Davis, New Relic

2:50pm - 3:10pmMoon Lounge

Ask-Me-Anything

Interactive session

2:50pm - 3:10pmPalais Atelier

Ask-Me-Anything

September 5: 2:50pm - 3:10pm
Speakers

Go to: Speakers or Schedule

September 5:
Speakers

Go to: Speakers or Schedule

Automating Flink Deployments to Kubernetes

Marc Rooding, ING
Niels Dennissen, ING

3:20pm - 3:40pmKesselhaus

Alert DSL on Flink for eBay monitoring platform

Yu Huang, eBay
Garrett Li, eBay

3:20pm - 3:40pmMaschinenhaus

Lessons learned from Migrating to a Stateful Streaming Framework

Tony Wei, Appier

3:20pm - 3:40pmMoon Lounge

Lessons learned from Migrating to a Stateful Streaming Framework

In modern applications of streaming frameworks, stateful streaming is arguably one of the most important usage cases. Flink, as a well-supported streaming framework for stateful streaming, readily helps developers spend less efforts on system deployment and focus more on the business logic. Nevertheless, upgrading from an existing production system to a new one with stateful streaming can still be a challenging task for any development team. In this talk, we will share our experience in migrating an existing system at Appier (an AI-based startup specialized with B2B solutions) to stateful streaming with Flink. We will first discuss how stateful streaming matches our business logic and its potential benefits. Then, we review the obstacles that we have encountered during migration, and present our solutions to conquer them. We hope that our experience and tips shared in this talk hints future users to prepare themselves towards applying Flink in their production systems more painlessly.

Level: Beginner

September 5: 3:20pm - 3:40pm
Speakers

Wei-Che(Tony) Wei is a softwore engineer on Data Platform Team at Appier. He works on providing general facilities for internal users to access data by leveraging different open sources, such as Flink, Spark and Kafka. Recently, he focuses on building a streaming platform to let users benefit from the advantage of stateful streaming framework. And he has been contributing to Flink as well.

Use Case

Lessons learned from Migrating to a Stateful Streaming Framework

Go to: Speakers or Schedule

September 5:
Speakers

Go to: Speakers or Schedule

Coffee Break

3:40pm - 4:10pmPalais

Coffee Break

September 5: 3:40pm - 4:10pm
Speakers

Go to: Speakers or Schedule

How to keep our flock happy with Apache Flink on AWS

Henri Heiskanen, Rovio

4:10pm - 4:50pmKesselhaus

Streaming ETL with Flink and Elasticsearch

Jared Stehler, Intellify Learning

4:30pm - 4:50pmMaschinenhaus

Flink as a Library (and still as a Framework)

Till Rohrmann, data Artisans

4:10pm - 4:50pmMoon Lounge

September 5:
Speakers

Go to: Speakers or Schedule

5:00pm

Closing session

5:00pm - 5:10pm

Closing session

September 5: 5:00pm - 5:10pm
Speakers

Go to: Speakers or Schedule

Apache Flink, Flink and Apache are either registered trademarks or trademarks of the Apache Software Foundation in the United States and/or other countries, and are used with permission. The Apache Software Foundation has no affiliation with and does not endorse, or review the materials provided at this event.

Apache Flink, Flink and Apache are either registered trademarks or trademarks of the Apache Software Foundation in the United States and/or other countries, and are used with permission. The Apache Software Foundation has no affiliation with and does not endorse, or review the materials provided at this event.