mirror of
https://github.com/alibaba/canal.git
synced 2026-11-04 00:30:15 +08:00
Updated 简介 (markdown)
26
简介.md
26
简介.md
@@ -22,14 +22,14 @@
|
||||
<p style="font-size: 14px;"> </p>
|
||||
<h2>工作原理</h2>
|
||||
<h3 style="font-size: 14px;">mysql主备复制实现</h3>
|
||||
<p><img src="http://dl.iteye.com/upload/attachment/0080/3086/468c1a14-e7ad-3290-9d3d-44ac501a7227.jpg" alt=""><br> 从上层来看,复制分成三步:</p>
|
||||
<p><img src="https://static.sitestack.cn/projects/canal-v1.1.4/c521db7b157d0e920a69adfe1724e3c9.jpeg" alt=""><br> 从上层来看,复制分成三步:</p>
|
||||
<ol>
|
||||
<li>master将改变记录到二进制日志(binary log)中(这些记录叫做二进制日志事件,binary log events,可以通过show binlog events进行查看);</li>
|
||||
<li>slave将master的binary log events拷贝到它的中继日志(relay log);</li>
|
||||
<li>slave重做中继日志中的事件,将改变反映它自己的数据。</li>
|
||||
</ol>
|
||||
<h3>canal的工作原理:</h3>
|
||||
<p><img width="590" src="http://dl.iteye.com/upload/attachment/0080/3107/c87b67ba-394c-3086-9577-9db05be04c95.jpg" alt="" height="273"></p>
|
||||
<p><img width="590" src="https://static.sitestack.cn/projects/canal-v1.1.4/4a2dbea434ebf5ec6016c4c802b393e3.jpeg" alt="" height="273"></p>
|
||||
<p>原理相对比较简单:</p>
|
||||
<ol>
|
||||
<li>canal模拟mysql slave的交互协议,伪装自己为mysql slave,向mysql master发送dump协议</li>
|
||||
@@ -37,7 +37,7 @@
|
||||
<li>canal解析binary log对象(原始为byte流)</li>
|
||||
</ol>
|
||||
<h1>架构</h1>
|
||||
<p><img width="548" src="http://dl.iteye.com/upload/attachment/0080/3126/49550085-0cd2-32fa-86a6-f676db5b597b.jpg" alt="" height="238" style="line-height: 1.5;"></p>
|
||||
<p><img width="548" src="https://static.sitestack.cn/projects/canal-v1.1.4/cb60bc8fe2dd0c07734a8de174de9988.jpeg" alt="" height="238" style="line-height: 1.5;"></p>
|
||||
<p style="color: #333333; background-image: none; margin-top: 10px; margin-bottom: 10px; font-family: Arial, Helvetica, FreeSans, sans-serif;">说明:</p>
|
||||
<ul style="line-height: 1.5; color: #333333; font-family: Arial, Helvetica, FreeSans, sans-serif;">
|
||||
<li>server代表一个canal运行实例,对应于一个jvm</li>
|
||||
@@ -73,7 +73,7 @@
|
||||
<p> </p>
|
||||
<h2>EventParser设计</h2>
|
||||
<p>大致过程:</p>
|
||||
<p><img width="707" src="http://dl.iteye.com/upload/attachment/0080/3143/7951c169-f7df-3cb3-aebb-d924f57311cb.jpg" alt="" height="406"></p>
|
||||
<p><img width="707" src="https://static.sitestack.cn/projects/canal-v1.1.4/ee844ab2f20b7ca63e16d87b67e1ed32.jpeg" alt="" height="406"></p>
|
||||
<p>整个parser过程大致可分为几步:</p>
|
||||
<ol>
|
||||
<li>Connection获取上一次解析成功的位置 (如果第一次启动,则获取初始指定的位置或者是当前数据库的binlog位点)</li>
|
||||
@@ -84,7 +84,7 @@
|
||||
<li>存储成功后,定时记录Binaly Log位置</li>
|
||||
</ol>
|
||||
<p>mysql的Binlay Log网络协议:</p>
|
||||
<p><img width="667" src="http://dl.iteye.com/upload/attachment/0080/3173/638a83ae-3b5c-3f83-9722-2c1912e7cac6.png" alt="" height="352"></p>
|
||||
<p><img width="667" src="https://static.sitestack.cn/projects/canal-v1.1.4/2d0de5d690ea4ae344c4720a285a1613.png" alt="" height="352"></p>
|
||||
<p> </p>
|
||||
<p>说明:</p>
|
||||
<ul>
|
||||
@@ -100,7 +100,7 @@
|
||||
</li>
|
||||
</ul>
|
||||
<h2>EventSink设计</h2>
|
||||
<p><img width="647" src="http://dl.iteye.com/upload/attachment/0080/3214/0a2fd671-d6e9-3ed7-b110-6a3b738a3cb0.jpg" alt="" height="240"></p>
|
||||
<p><img width="647" src="https://static.sitestack.cn/projects/canal-v1.1.4/c1ff8ff7d572e4104507c0d0be819004.jpeg" alt="" height="240"></p>
|
||||
<p>说明:</p>
|
||||
<ul>
|
||||
<li>数据过滤:支持通配符的过滤模式,表名,字段内容等</li>
|
||||
@@ -122,21 +122,21 @@
|
||||
<li style="line-height: 1.5; margin-bottom: 0px; margin-left: 0px;">2. 借鉴了Disruptor的RingBuffer的实现思路</li>
|
||||
</ul>
|
||||
<p>RingBuffer设计:</p>
|
||||
<p><img width="331" src="http://dl.iteye.com/upload/attachment/0080/3237/063e8480-15c8-3e66-bbd3-9c44def09c8f.jpg" alt="" height="303"></p>
|
||||
<p><img width="331" src="https://static.sitestack.cn/projects/canal-v1.1.4/3ab10f59d35166c43b6baa8ba05253f1.jpeg" alt="" height="303"></p>
|
||||
<p>定义了3个cursor</p>
|
||||
<ul>
|
||||
<li>Put : Sink模块进行数据存储的最后一次写入位置</li>
|
||||
<li>Get : 数据订阅获取的最后一次提取位置</li>
|
||||
<li>Ack : 数据消费成功的最后一次消费位置</li>
|
||||
</ul>
|
||||
<p>借鉴Disruptor的RingBuffer的实现,将RingBuffer拉直来看:<br><img width="620" src="http://dl.iteye.com/upload/attachment/0080/3239/4f58912d-7c8e-37de-b7b8-10dd21aa6c22.jpg" alt="" height="165"></p>
|
||||
<p>借鉴Disruptor的RingBuffer的实现,将RingBuffer拉直来看:<br><img width="620" src="https://static.sitestack.cn/projects/canal-v1.1.4/81b165d55be9db4cb7a45558098527a9.jpeg" alt="" height="165"></p>
|
||||
<p>实现说明:</p>
|
||||
<ul>
|
||||
<li>Put/Get/Ack cursor用于递增,采用long型存储</li>
|
||||
<li>buffer的get操作,通过取余或者与操作。(与操作: cusor & (size - 1) , size需要为2的指数,效率比较高)</li>
|
||||
</ul>
|
||||
<h2>Instance设计</h2>
|
||||
<p><br><img width="651" src="http://dl.iteye.com/upload/attachment/0080/3247/5de1c9af-7798-3d42-bc43-5c54d82c4dbf.jpg" alt="" height="415"></p>
|
||||
<p><br><img width="651" src="https://static.sitestack.cn/projects/canal-v1.1.4/2693dc620b84508d18e474d35e41147b.jpeg" alt="" height="415"></p>
|
||||
<p> </p>
|
||||
<p>instance代表了一个实际运行的数据队列,包括了EventPaser,EventSink,EventStore等组件。</p>
|
||||
<p>抽象了CanalInstanceGenerator,主要是考虑配置的管理方式:</p>
|
||||
@@ -145,14 +145,14 @@
|
||||
<li>spring方式:基于spring xml + properties进行定义,构建spring配置. </li>
|
||||
</ul>
|
||||
<h2>Server设计</h2>
|
||||
<p><br><img width="498" src="http://dl.iteye.com/upload/attachment/0080/3257/f4df38ba-59e2-398e-b5eb-1bbfbecc0676.jpg" alt="" height="254"></p>
|
||||
<p><br><img width="498" src="https://static.sitestack.cn/projects/canal-v1.1.4/6cfd86444401611a3afe1b57cedeefe2.jpeg" alt="" height="254"></p>
|
||||
<p>server代表了一个canal的运行实例,为了方便组件化使用,特意抽象了Embeded(嵌入式) / Netty(网络访问)的两种实现</p>
|
||||
<ul>
|
||||
<li>Embeded : 对latency和可用性都有比较高的要求,自己又能hold住分布式的相关技术(比如failover)</li>
|
||||
<li>Netty : 基于netty封装了一层网络协议,由canal server保证其可用性,采用的pull模型,当然latency会稍微打点折扣,不过这个也视情况而定。(阿里系的notify和metaq,典型的push/pull模型,目前也逐步的在向pull模型靠拢,push在数据量大的时候会有一些问题) </li>
|
||||
</ul>
|
||||
<h2>增量订阅/消费设计</h2>
|
||||
<p><img width="427" src="http://dl.iteye.com/upload/attachment/0080/3297/9d7ed13e-6a86-386d-92f4-852238c475bf.jpg" alt="" height="524"></p>
|
||||
<p><img width="427" src="https://static.sitestack.cn/projects/canal-v1.1.4/20adf0becbb39d551e1cc53be1135452.jpeg" alt="" height="524"></p>
|
||||
<p>具体的协议格式,可参见:<a href="https://github.com/alibaba/canal/blob/master/protocol/src/main/java/com/alibaba/otter/canal/protocol/CanalProtocol.proto">CanalProtocol.proto</a></p>
|
||||
<p>get/ack/rollback协议介绍:</p>
|
||||
<ul>
|
||||
@@ -168,7 +168,7 @@
|
||||
<li>get获取数据后,业务消费存在瓶颈或者需要多进程/多线程消费时,可以不停的轮询get数据,不停的往后发送任务,提高并行化. (作者在实际业务中的一个case:业务数据消费需要跨中美网络,所以一次操作基本在200ms以上,为了减少延迟,所以需要实施并行化)</li>
|
||||
</ul>
|
||||
<p>流式api设计:</p>
|
||||
<p><img width="441" src="http://dl.iteye.com/upload/attachment/0080/3300/7f7986e5-d8c6-3a17-8796-71166e6bc2dc.jpg" alt="" height="277"></p>
|
||||
<p><img width="441" src="https://static.sitestack.cn/projects/canal-v1.1.4/c06bf7b162cc467e487f5ae0d1879ec2.jpeg" alt="" height="277"></p>
|
||||
<ul>
|
||||
<li>每次get操作都会在meta中产生一个mark,mark标记会递增,保证运行过程中mark的唯一性</li>
|
||||
<li>每次的get操作,都会在上一次的mark操作记录的cursor继续往后取,如果mark不存在,则在last ack cursor继续往后取</li>
|
||||
@@ -217,7 +217,7 @@ Column
|
||||
<p>整个HA机制的控制主要是依赖了zookeeper的几个特性,watcher和EPHEMERAL节点(和session生命周期绑定),可以看下我之前zookeeper的相关文章。</p>
|
||||
<p> </p>
|
||||
<p>Canal Server: </p>
|
||||
<p><img width="624" src="http://dl.iteye.com/upload/attachment/0080/3303/d3202c26-e954-35c0-a319-57604102c57d.jpg" alt="" height="341"><br>大致步骤:</p>
|
||||
<p><img width="624" src="https://static.sitestack.cn/projects/canal-v1.1.4/9bd4a70b5b068a736b7ad82feb1f4881.jpeg" alt="" height="341"><br>大致步骤:</p>
|
||||
<ol>
|
||||
<li><span style="line-height: 21px;">canal server要启动某个canal instance时都先向zookeeper进行一次尝试启动判断 (实现:创建EPHEMERAL节点,谁创建成功就允许谁启动)</span></li>
|
||||
<li><span style="line-height: 21px;">创建zookeeper节点成功后,对应的canal server就启动对应的canal instance,没有创建成功的canal instance就会处于standby状态</span></li>
|
||||
|
||||
Reference in New Issue
Block a user