成为VIP
统一声明：
1.本站联系方式QQ：1754646538 TG：@UXWNET 如果有其他人通过本站链接联系您导致被骗，本站一律不负责! 2.需要付费搭建请联系站长QQ：1754646538 TG：@UXWNET 3.国外免备案服务器- 游侠云服务 4.免实名域名注册购买- 游侠云域名 5.免实名国外服务器购买- 游侠网云服务资源名称：Spark内核设计的艺术架构设计与实现
内容简介：
本书赞誉
前言
第1章 环境准备 ········································1
1.1 运行环境准备 ···········································2
1.1.1 安装JDK ·········································2
1.1.2 安装Scala ········································2
1.1.3 安装Spark ·······································3
1.2 Spark初体验 ···································4
1.2.1 运行spark-shell ·······························4
1.2.2 执行word count ······························5
1.2.3 剖析spark-shell ·······························9
1.3 阅读环境准备 ·········································14
1.3.1 安装SBT ·······································15
1.3.2 安装Git ·········································15
1.3.3 安装Eclipse Scala IDE插件 ········15
1.4 Spark源码编译与调试 ·························17
1.5 小结 ···························23
第2章 设计理念与基本架构 ···············24
2.1 初识Spark ··································25
2.1.1 Hadoop MRv1的局限···················25
2.1.2 Spark的特点 ·································26
2.1.3 Spark使用场景 ·····························28
2.2 Spark基础知识 ······································29
2.3 Spark基本设计思想 ·····························31
2.3.1 Spark模块设计 ·····························32
2.3.2 Spark模型设计 ·····························34
2.4 Spark基本架构 ···································36
2.5 小结 ·································38
第3章 Spark基础设施 ·························39
3.1 Spark配置 ········································40
3.1.1 系统属性中的配置 ·······················40
3.1.2 使用SparkConf配置的API ·········41
3.1.3 克隆SparkConf配置 ····················42
3.2 Spark内置RPC框架 ····························42
3.2.1 RPC配置TransportConf ··············45
3.2.2 RPC客户端工厂Transport- ClientFactory ·······················47
3.2.3 RPC服务端TransportServer ········53
3.2.4 管道初始化 ···································56
3.2.5 TransportChannelHandler详解 ·····57
3.2.6 服务端RpcHandler详解 ··············63
3.2.7 服务端引导程序Transport-ServerBootstrap ·····················68
3.2.8 客户端TransportClient详解 ········71
3.3 事件总线 ····································78
3.3.1 ListenerBus的继承体系 ···············79
3.3.2 SparkListenerBus详解 ··················80
3.3.3 LiveListenerBus详解 ····················83
3.4 度量系统 ···········································87
3.4.1 Source继承体系 ···························87
3.4.2 Sink继承体系 ·······························89
3.5 小结 ·········································92
第4章 SparkContext的初始化 ·········93
4.1 SparkContext概述 ·································94
4.2 创建Spark环境 ·····································97
4.3 SparkUI的实现 ····································100
4.3.1 SparkUI概述 ·······························100
4.3.2 WebUI框架体系 ·························102
4.3.3 创建SparkUI ·······························107
4.4 创建心跳接收器 ··································111
4.5 创建和启动调度系统··························112
4.6 初始化块管理器BlockManager ·······114
4.7 启动度量系统 ·······························114
4.8 创建事件日志监听器··························115
4.9 创建和启动ExecutorAllocation-Manager ··························116
4.10 ContextCleaner的创建与启动 ········120
4.10.1 创建ContextCleaner ·················120
4.10.2 启动ContextCleaner ·················120
4.11 额外的SparkListener与启动事件总线 ··························122
4.12 Spark环境更新 ··································123
4.13 SparkContext初始化的收尾 ···········127
4.14 SparkContext提供的常用方法 ·······128
4.15 SparkContext的伴生对象················130
4.16 小结 ····································131
第5章 Spark执行环境 ························132
5.1 SparkEnv概述 ·································133
5.2 安全管理器SecurityManager ············133
5.3 RPC环境 ·········································135
5.3.1 RPC端点RpcEndpoint ···············136
5.3.2 RPC端点引用RpcEndpointRef ···139
5.3.3 创建传输上下文TransportConf ···142
5.3.4 消息调度器Dispatcher ···············142
5.3.5 创建传输上下文Transport-Context ·························154
5.3.6 创建传输客户端工厂Transport-ClientFactory ····················159
5.3.7 创建TransportServer ···················160
5.3.8 客户端请求发送 ·························162
5.3.9 NettyRpcEnv中的常用方法 ·······173
5.4 序列化管理器SerializerManager ·····175
5.5 广播管理器BroadcastManager ·········178
5.6 map任务输出跟踪器 ··························185
5.6.1 MapOutputTracker的实现 ··········187
5.6.2 MapOutputTrackerMaster的实现原理 ·······················191
5.7 构建存储体系 ·······································199
5.8 创建度量系统 ·······································201
5.8.1 MetricsCon?g详解 ·····················203
5.8.2 MetricsSystem中的常用方法 ····207
5.8.3 启动MetricsSystem ····················209
5.9 输出提交协调器 ··································211
5.9.1 OutputCommitCoordinator-Endpoint的实现 ··················211
5.9.2 OutputCommitCoordinator的实现 ··························212
5.9.3 OutputCommitCoordinator的工作原理 ························216
5.10 创建SparkEnv ····································217
5.11 小结 ·····································217
第6章 存储体系 ·····································219
6.1 存储体系概述 ·······································220
6.1.1 存储体系架构 ·····························220
6.1.2 基本概念 ·····································222
6.2 Block信息管理器 ································227
6.2.1 Block锁的基本概念 ···················227
6.2.2 Block锁的实现 ···························229
6.3 磁盘Block管理器 ······························234
6.3.1 本地目录结构 ·····························234
6.3.2 DiskBlockManager提供的方法 ···························236
6.4 磁盘存储DiskStore ·····························239
6.5 内存管理器 ·····································242
6.5.1 内存池模型 ·································243
6.5.2 StorageMemoryPool详解 ···········244
6.5.3 MemoryManager模型 ················247
6.5.4 Uni?edMemoryManager详解 ····250
6.6 内存存储MemoryStore ······················252
6.6.1 MemoryStore的内存模型 ··········253
6.6.2 MemoryStore提供的方法 ··········255
6.7 块管理器BlockManager ····················265
6.7.1 BlockManager的初始化 ·············265
6.7.2 BlockManager提供的方法 ·········266
6.8 BlockManagerMaster对Block-Manager的管理 ·················285
6.8.1 BlockManagerMaster的职责 ······285
6.8.2 BlockManagerMasterEndpoint详解 ·································286
6.8.3 BlockManagerSlaveEndpoint详解 ·····························289
6.9 Block传输服务 ····································290
6.9.1 初始化NettyBlockTransfer-Service ···························291
6.9.2 NettyBlockRpcServer详解 ·········292
6.9.3 Shuf?e客户端 ·····························296
6.10 DiskBlockObjectWriter详解 ···········305
6.11 小结 ·······································308
第7章 调度系统 ·····································309
7.1 调度系统概述 ·······································310
7.2 RDD详解 ·····································312
7.2.1 为什么需要RDD ························312
7.2.2 RDD实现的初次分析 ················313
7.2.3 RDD依赖 ····································316
7.2.4 分区计算器Partitioner················318
7.2.5 RDDInfo ······································320
7.3 Stage详解 ········································321
7.3.1 ResultStage的实现 ·····················322
7.3.2 Shuf?eMapStage的实现 ·············323
7.3.3 StageInfo ······································324
7.4 面向DAG的调度器DAGScheduler ···326
7.4.1 JobListener与JobWaiter ·············326
7.4.2 ActiveJob详解 ····························328
7.4.3 DAGSchedulerEventProcessLoop的简要介绍 ·······················328
7.4.4 DAGScheduler的组成 ················329
7.4.5 DAGScheduler提供的常用方法 ···330
7.4.6 DAGScheduler与Job的提交 ····334
7.4.7 构建Stage····································337
7.4.8 提交ResultStage ························341
7.4.9 提交还未计算的Task ·················343
7.4.10 DAGScheduler的调度流程 ······347
7.4.11 Task执行结果的处理 ··············348
7.5 调度池Pool ······································351
7.5.1 调度算法 ·······························352
7.5.2 Pool的实现 ·································354
7.5.3 调度池构建器 ·····························357
7.6 任务集合管理器TaskSetManager ···363
7.6.1 Task集合 ·····································363
7.6.2 TaskSetManager的成员属性 ······364
7.6.3 调度池与推断执行 ·····················366
7.6.4 Task本地性 ·································370
7.6.5 TaskSetManager的常用方法 ······373
7.7 运行器后端接口LauncherBackend ···383
7.7.1 BackendConnection的实现 ········384
7.7.2 LauncherBackend的实现 ···········386
7.8 调度后端接口SchedulerBackend ····389
7.8.1 SchedulerBackend的定义 ··········389
7.8.2 LocalSchedulerBackend的实现分析 ································390
7.9 任务结果获取器TaskResultGetter ···394
7.9.1 处理成功的Task ·························394
7.9.2 处理失败的Task ·························396
7.10 任务调度器TaskScheduler ··············397
7.10.1 TaskSchedulerImpl的属性 ·····397
7.10.2 TaskSchedulerImpl的初始化 ···399
7.10.3 TaskSchedulerImpl的启动 ·····399
7.10.4 TaskSchedulerImpl与Task的提交 ·······················400
7.10.5 TaskSchedulerImpl与资源分配 ···························402
7.10.6 TaskSchedulerImpl的调度流程 ······························405
7.10.7 TaskSchedulerImpl对执行结果的处理 ·····························406
7.10.8 TaskSchedulerImpl的常用方法 ···409
7.11 小结 ·······································412
第8章 计算引擎 ·····································413
第9章 部署模式 ········································520
第10章 Spark API ································647
资源截图：
感谢您的支持，我会继续努力的!
1. 本站所有资源来源于用户上传和网络，因此不包含技术服务请大家谅解！如有侵权请邮件联系客服！17131833@qq.com 2. 本站不保证所提供下载的资源的准确性、安全性和完整性，资源仅供下载学习之用！如有链接无法下载、失效或广告，请联系客服处理，有奖励！ 3. 您必须在下载后的24个小时之内，从您的电脑中彻底删除上述内容资源！如用于商业或者非法用途，与本站无关，一切后果请用户自负！ 4. 如果您也有好的资源或教程，您可以投稿发布，成功分享后有站币奖励和额外收入！
源雷 云计算教程 Spark内核设计的艺术架构设计与实现 https://www.mubanke.com/58288.html
2. 分享目的仅供大家学习和交流，您必须在下载后24小时内删除！
3. 不得使用于非法商业用途，不得违反国家法律。否则后果自负！
4. 本站提供的源码、模板、插件等等其他资源，都不包含技术服务请大家谅解！
5. 如有链接无法下载、失效或广告，请联系管理员处理！
6. 本站资源售价只是赞助，收取费用仅维持本站的日常运营所需！
7. 如遇到加密压缩包，请使用WINRAR解压,如遇到无法解压的请联系管理员！
8. 精力有限，不少源码未能详细测试（解密），不能分辨部分源码是病毒还是误报，所以没有进行任何修改，大家使用前请进行甄别！
站长QQ：1754646538 站长邮箱：1754646538@qq.com
YUANLEI
