chenjl/canal
1
0
Fork 0
mirror of https://github.com/alibaba/canal.git synced 2026-11-04 00:30:15 +08:00
Code Issues Packages Projects Releases Wiki Activity

Created Performance (markdown)

agapple
2018-07-20 14:33:48 +08:00
parent 17a4500e89
commit 652a34ef57
1 changed files with 135 additions and 0 deletions
Download Patch File Download Diff File Expand all files Collapse all files

135
Performance.md Normal file

@@ -0,0 +1,135 @@
# 测试环境
| 类型 | 配置 |
| --- | --- |
| MySQL A | Intel(R) Xeon(R) CPU E5-2430 0 @ 2.20GHz (24core 96G) |
| MySQL B | Intel(R) Xeon(R) CPU E5-2430 0 @ 2.20GHz (24core 96G) 日常业务库 |
| Canal Server | Intel(R) Xeon(R) CPU E5-2430 0 @ 2.20GHz (24core 96G) |
| Canal Client | Intel(R) Xeon(R) CPU E5-2430 0 @ 2.20GHz (24core 96G) |
# 测试方式
为了更加精准的描述canal的性能，会从整个流程上进行优化和分析.   具体优化思路参考：[https://github.com/alibaba/canal/issues/726](https://github.com/alibaba/canal/issues/726)
1. 整个canal流程处理binlog，分为这么5步操作(4+1)，整个优化和评测对这5步分别进行.
![image.png | left | 747x237](https://cdn.yuque.com/lark/0/2018/png/5565/1532057042091-3ad852d4-6346-4c7d-b94b-327baba85010.png "")
2. 构造了两个测试场景，批量Insert/Update/Delete 和 普通业务DB(单条操作为主)
# 测试数据
<div class="bi-table">
<table>
<colgroup>
<col width="90px" />
<col width="auto" />
<col width="auto" />
<col width="auto" />
</colgroup>
<tbody>
<tr height="34px">
<td rowspan="1" colSpan="1">
<div data-type="p">序号</div>
</td>
<td rowspan="1" colSpan="1">
<div data-type="p">阶段</div>
</td>
<td rowspan="1" colSpan="1">
<div data-type="p">批量操作Insert/Update/Delete (导入业务)</div>
</td>
<td rowspan="1" colSpan="1">
<div data-type="p">单条操作 (普通业务)</div>
</td>
</tr>
<tr height="34px">
<td rowspan="1" colSpan="1">
<div data-type="p">1</div>
</td>
<td rowspan="1" colSpan="1">
<div data-type="p">Binlog接收</div>
</td>
<td rowspan="1" colSpan="1">
<div data-type="p">200w TPS (网络 117MB/s)</div>
</td>
<td rowspan="1" colSpan="1">
<div data-type="p">71w TPS (网络 112MB/s)</div>
</td>
</tr>
<tr height="34px">
<td rowspan="1" colSpan="1">
<div data-type="p">2</div>
</td>
<td rowspan="1" colSpan="1">
<div data-type="p">Binlog Event解析</div>
</td>
<td rowspan="1" colSpan="1">
<div data-type="p">200w TPS (网络 117MB/s)</div>
</td>
<td rowspan="1" colSpan="1">
<div data-type="p">70w TPS (网络 110MB/s)</div>
</td>
</tr>
<tr height="34px">
<td rowspan="1" colSpan="1">
<div data-type="p">3</div>
</td>
<td rowspan="1" colSpan="1">
<div data-type="p">Insert/Update/Delete深度解析</div>
</td>
<td rowspan="1" colSpan="1">
<div data-type="p">200w TPS (网络 117MB/s)</div>
</td>
<td rowspan="1" colSpan="1">
<div data-type="p">65w TPS (网络 105MB/s)</div>
</td>
</tr>
<tr height="34px">
<td rowspan="1" colSpan="1">
<div data-type="p">4</div>
</td>
<td rowspan="1" colSpan="1">
<div data-type="p">生成CanalEntry (存储到memory store)</div>
</td>
<td rowspan="1" colSpan="1">
<div data-type="p">130w TPS (网络 75MB/s)</div>
</td>
<td rowspan="1" colSpan="1">
<div data-type="p">50w TPS (网络 90MB/s)</div>
</td>
</tr>
<tr height="34px">
<td rowspan="1" colSpan="1">
<div data-type="p">5</div>
</td>
<td rowspan="1" colSpan="1">
<div data-type="p">client接收</div>
</td>
<td rowspan="1" colSpan="1">
<div data-type="p">20w TPS 1.canal server机器网络 11MB/s</div>
<div data-type="p">2.canal client机器网络 75MB/s</div>
<div data-type="p">
binlog膨胀率为 1:6.8</div>
</td>
<td rowspan="1" colSpan="1">
<div data-type="p">10w TPS 1.canal server网络 22MB/s</div>
<div data-type="p">2.canal client网络 42MB/s</div>
<div data-type="p"></div>
<div data-type="p">binlog膨胀率为 1:1.9</div>
</td>
</tr>
</tbody>
</table>
</div>
# 小结
  从最开始接收(跑满网络带宽)到最后client机器收到格式化的binlog数据，binlog解析的5个阶段是一个漏斗形的性能。目前整个阶段4->阶段5，性能下降比较明显主要是因为网络传输、序列化的代价影响，binlog接收为了保序采用了串行方式，所以串行里的每个代码逻辑处理都会影响最后吞吐。
so. 如果基于canal做额外的数据扩展，比如对接到MQ系统，可以在步骤3、4阶段介入，最大化的吞吐.
结论数据：
1. 1.0.26经过优化之后的性能，从业务binlog入库到canal client拿到数据，基本可以达到10~20w的TPS.
2. 单纯的binlog解析能力可以跑到60w ~ 200w的TPS，相当于100MB/s的解析速度