📌 相关文章
- Apache Kafka-与Spark集成
- Apache Kafka-与Spark集成(1)
- 使用Python 的Hadoop Streaming – 字数问题
- 使用Python 的Hadoop Streaming – 字数问题(1)
- Spark-安装(1)
- Spark-安装
- Kafka流处理(1)
- Kafka流处理
- Kafka安全(1)
- Kafka安全
- 在Linux上安装Kafka(1)
- 在Linux上安装Kafka
- Kafka主题(1)
- Kafka主题
- kafka nodejs 示例 (1)
- Spark-简介(1)
- Spark-简介
- Spark SQL-数据框(1)
- Spark SQL-数据框
- Kafka使用Java编程(1)
- Kafka使用Java编程
- 安装Apache Kafka
- 安装Apache Kafka(1)
- Kafka实时示例
- Kafka实时示例(1)
- 在 python 中创建 spark 数据框(1)
- 将 spark 连接到 postgres；将 spark 连接到数据库 - Python (1)
- L'entente cordiale streaming - CSS (1)
- kafka 删除主题 (1)
📜 Kafka Streams与Spark Streaming
📅 最后修改于: 2021-01-05 03:04:43 🧑 作者: Mango
Kafka Streams与火花流
Apache Spark
Apache Spark是一个分布式的通用处理系统，可以一次处理PB级的数据。它主要用于流传输和处理数据。它分布在数千个虚拟服务器之间。大型组织使用Spark处理大量数据集。 Apache Spark允许使用大约80个高级运算符更快地构建应用程序。通过查询优化器，物理执行引擎和DAG调度程序，它可以获得流和批处理数据的高性能。因此，它的速度快了一百倍。
火花流
Apache spark通过Spark Streaming启用大型数据集的流传输。 Spark Streaming是核心Spark API的一部分，可让用户处理实时数据流。它从不同的数据源获取数据，并使用复杂的算法对其进行处理。最后，将处理后的数据推送到实时仪表板，数据库和文件系统中。
kafka 流
一个客户端库，用于处理和分析存储在Kafka中的数据。 Kafka流使用户能够构建应用程序和微服务。此外，将输出存储在Kafka集群中。除了Kafka之外，它对系统没有任何外部依赖性。它一次只处理一条记录。
Kafka Streams与火花流
| Parameters | Apache Kafka | Apache Spark |
|---|---|---|
| Developers | Originally developed by LinkedIn. Later, donated to Apache Software Foundation. | Originally developed at the University of California. Later, it was donated to Apache Software Foundation. |
| Infrastructure | It is a Java client library. Thus, it can execute wherever Java is supported. | It executes on the top of the Spark stack. It can be either Spark standalone, YARN, or container-based. |
| Data Sources | It processes data from Kafka itself via topics and streams. | Spark ingest data from various files, Kafka, Socket source, etc. |
| Processing Model | It processes the events as it arrives. Thus, it uses Event-at-a-time (continuous) processing model. | It has a micro-batch processing model. It splits the incoming streams into small batches for further processing. |
| Latency | It has low latency than Apache Spark | It has a higher latency. |
| ETL Transformation | It is not supported in Apache Kafka. | This transformation is supported in Spark. |
| Fault-tolerance | Fault-tolerance is complex in Kafka. | Fault-tolerance is easy in Spark. |
| Language Support | It supports Java mainly. | It supports multiple languages such as Java, Scala, R, Python. |
| Use Cases | The New York Times, Zalando, Trivago, etc. use Kafka Streams to store and distribute data. | Booking.com, Yelp (ad platform) uses Spark streams for handling millions of ad requests per day. |