Compare commits
126 Commits
canal-1.1.
...
canal-1.1.
| Author | SHA1 | Date | |
|---|---|---|---|
|
64862503ba | ||
|
|
f9e0dc7daa | ||
|
|
0c399509a9 | ||
|
|
a09cd9adb5 | ||
|
|
1369418c6a | ||
|
|
75174a63af | ||
|
|
17f4ee13e7 | ||
|
|
f0417502f8 | ||
|
|
a7720b51e4 | ||
|
|
21904ef1dc | ||
|
|
94425223bd | ||
|
|
7857df9701 | ||
|
|
64ea4ff222 | ||
|
|
85df89f777 | ||
|
|
a0f8e182ec | ||
|
|
33721d0394 | ||
|
|
91bddfeed6 | ||
|
|
25feab4af8 | ||
|
|
f66baeabf1 | ||
|
|
b1dc3821ea | ||
|
|
f5f79104eb | ||
|
|
e8d138d3bd | ||
|
|
32d9e58915 | ||
|
|
58448e559d | ||
|
|
fede3b3d1d | ||
|
|
5d8ec31bc4 | ||
|
|
82b18ebe62 | ||
|
|
66408782b0 | ||
|
|
846c8870ae | ||
|
|
8cfb07f07e | ||
|
|
d741425895 | ||
|
|
20add85ff6 | ||
|
|
a5c09c521e | ||
|
|
c94276b1ac | ||
|
|
272d575c78 | ||
|
|
693f8d501a | ||
|
|
bb1b6b9862 | ||
|
|
8ed82caaef | ||
|
|
de1dd70609 | ||
|
|
09429c241f | ||
|
|
9a332940d7 | ||
|
|
362974b1c2 | ||
|
|
435db30b7b | ||
|
|
e7de8289cd | ||
|
|
5065fea963 | ||
|
|
6989eb3d2b | ||
|
|
df24596092 | ||
|
|
43c23ffb97 | ||
|
|
e7658879bc | ||
|
|
df60eda679 | ||
|
|
b81e9dfeb8 | ||
|
|
b79af46054 | ||
|
|
90bf9e6de5 | ||
|
|
182163c3bd | ||
|
|
e4726972de | ||
|
|
985892cf7e | ||
|
|
746d51e621 | ||
|
|
cdd077b758 | ||
|
|
72ae7ba8d7 | ||
|
|
b82c8bca83 | ||
|
|
576406a4ce | ||
|
|
b5b08e92e3 | ||
|
|
9f6021cf36 | ||
|
|
0e3d719a39 | ||
|
|
afbf125231 | ||
|
|
b2b2a82554 | ||
|
|
30a1b067a8 | ||
|
|
beaca165d0 | ||
|
|
c21c18458b | ||
|
|
5e761e14fa | ||
|
|
18418ebd98 | ||
|
|
92f04f86cb | ||
|
|
0d5505aa2c | ||
|
|
bd56438eca | ||
|
|
b9a157657f | ||
|
|
523e8b85e6 | ||
|
|
ab74b2c82c | ||
|
|
265a8e8594 | ||
|
|
4fc7e5f7d9 | ||
|
|
9cc8caa834 | ||
|
|
a71375dad5 | ||
|
|
70d8a379f2 | ||
|
|
b6333f018f | ||
|
|
31acfc0101 | ||
|
|
6d9ea57186 | ||
|
|
c4c0255073 | ||
|
|
1fa9020599 | ||
|
|
dd7554374f | ||
|
|
65b5325ca6 | ||
|
|
b64a3e301a | ||
|
|
31562919bb | ||
|
|
9c75e25da7 | ||
|
|
7d91850be6 | ||
|
|
4af3b24b69 | ||
|
|
e4e66b143c | ||
|
|
dd93e2b0c4 | ||
|
|
d505b5ab9d | ||
|
|
0fb717a97f | ||
|
|
c634f98fca | ||
|
|
460479390a | ||
|
|
26013e239d | ||
|
|
46fe40ed47 | ||
|
|
0c27bcf73e | ||
|
|
d86b0e6e67 | ||
|
|
fff623acb1 | ||
|
|
f45c0206da | ||
|
|
61726c7e05 | ||
|
|
79552e0a37 | ||
|
|
5ec2087257 | ||
|
|
5214257cd4 | ||
|
|
365ca4d01f | ||
|
|
419a73b3de | ||
|
|
c093fbb8f6 | ||
|
|
31bb85bea3 | ||
|
|
803565456f | ||
|
|
520f5771d8 | ||
|
|
5fbd6f6606 | ||
|
|
bc9c7f0f1d | ||
|
|
08a2f0aa44 | ||
|
|
a0387faa67 | ||
|
|
1dd4674ccb | ||
|
|
60e26d1ddd | ||
|
|
ecdab76705 | ||
|
|
403fdf997c | ||
|
|
a9e83463e0 | ||
|
|
2bde4327c7 |
36
README.md
@@ -5,11 +5,13 @@
|
||||
[](http://isitmaintained.com/project/alibaba/canal "average time to resolve an issue")
|
||||
[](http://isitmaintained.com/project/alibaba/canal "percentage of issues still open")
|
||||
|
||||
## 背景
|
||||
## 简介
|
||||
|
||||
早期，阿里巴巴 B2B 公司因为存在杭州和美国双机房部署，存在跨机房同步的业务需求 ，主要是基于trigger的方式获取增量变更。从 2010 年开始，公司开始逐步尝试数据库日志解析，获取增量变更进行同步，由此衍生出了增量订阅和消费业务，从此开启一段新纪元。
|
||||
|
||||
|
||||
当前的 canal 支持源端 MySQL 版本包括 5.1.x , 5.5.x , 5.6.x , 5.7.x , 8.0.x
|
||||
**canal [kə'næl]**，译意为水道/管道/沟渠，主要用途是基于 MySQL 数据库增量日志解析，提供增量数据订阅和消费
|
||||
|
||||
早期阿里巴巴因为杭州和美国双机房部署，存在跨机房同步的业务需求，实现方式主要是基于业务 trigger 获取增量变更。从 2010 年开始，业务逐步尝试数据库日志解析获取增量变更进行同步，由此衍生出了大量的数据库增量订阅和消费业务。
|
||||
|
||||
基于日志增量订阅和消费的业务包括
|
||||
- 数据库镜像
|
||||
@@ -18,12 +20,7 @@
|
||||
- 业务 cache 刷新
|
||||
- 带业务逻辑的增量数据处理
|
||||
|
||||
## 项目介绍
|
||||
|
||||
#### 名称：canal [kə'næl]
|
||||
#### 译意： 水道/管道/沟渠
|
||||
#### 产品定位： 基于数据库增量日志解析，提供增量数据订阅和消费
|
||||
#### 关键词： MySQL binlog parser / real-time / queue&topic / index build
|
||||
当前的 canal 支持源端 MySQL 版本包括 5.1.x , 5.5.x , 5.6.x , 5.7.x , 8.0.x
|
||||
|
||||
## 工作原理
|
||||
|
||||
@@ -35,7 +32,6 @@
|
||||
- MySQL slave 重放 relay log 中事件，将数据变更反映它自己的数据
|
||||
|
||||
#### canal 工作原理
|
||||
|
||||
|
||||
- canal 模拟 MySQL slave 的交互协议，伪装自己为 MySQL slave ，向 MySQL master 发送dump 协议
|
||||
- MySQL master 收到 dump 请求，开始推送 binary log 给 slave (即 canal )
|
||||
@@ -43,7 +39,7 @@
|
||||
|
||||
## 重要版本更新说明
|
||||
|
||||
canal 1.1.x 版本（[release_note](https://github.com/alibaba/canal/releases)）,性能与功能层面有较大的突破,重要提升包括:
|
||||
1. canal 1.1.x 版本（[release_note](https://github.com/alibaba/canal/releases)）,性能与功能层面有较大的突破,重要提升包括:
|
||||
|
||||
- 整体性能测试&优化,提升了150%. #726 参考: [Performance](https://github.com/alibaba/canal/wiki/Performance)
|
||||
- 原生支持prometheus监控 #765 [Prometheus QuickStart](https://github.com/alibaba/canal/wiki/Prometheus-QuickStart)
|
||||
@@ -51,6 +47,8 @@ canal 1.1.x 版本（[release_note](https://github.com/alibaba/canal/releases)
|
||||
- 原生支持aliyun rds的binlog订阅 (解决自动主备切换/oss binlog离线解析) 参考: [Aliyun RDS QuickStart](https://github.com/alibaba/canal/wiki/aliyun-RDS-QuickStart)
|
||||
- 原生支持docker镜像 #801 参考: [Docker QuickStart](https://github.com/alibaba/canal/wiki/Docker-QuickStart)
|
||||
|
||||
2. canal 1.1.4版本，迎来最重要的WebUI能力，引入canal-admin工程，支持面向WebUI的canal动态管理能力，支持配置、任务、日志等在线白屏运维能力，具体文档：[Canal Admin Guide](https://github.com/alibaba/canal/wiki/Canal-Admin-Guide)
|
||||
|
||||
## 文档
|
||||
|
||||
- [Home](https://github.com/alibaba/canal/wiki/Home)
|
||||
@@ -60,6 +58,11 @@ canal 1.1.x 版本（[release_note](https://github.com/alibaba/canal/releases)
|
||||
- [Canal Kafka/RocketMQ QuickStart](https://github.com/alibaba/canal/wiki/Canal-Kafka-RocketMQ-QuickStart")
|
||||
- [Aliyun RDS for MySQL QuickStart](https://github.com/alibaba/canal/wiki/aliyun-RDS-QuickStart)
|
||||
- [Prometheus QuickStart](https://github.com/alibaba/canal/wiki/Prometheus-QuickStart)
|
||||
- Canal Admin
|
||||
- [Canal Admin QuickStart](https://github.com/alibaba/canal/wiki/Canal-Admin-QuickStart)
|
||||
- [Canal Admin Guide](https://github.com/alibaba/canal/wiki/Canal-Admin-Guide)
|
||||
- [Canal Admin ServerGuide](https://github.com/alibaba/canal/wiki/Canal-Admin-ServerGuide)
|
||||
- [Canal Admin Docker](https://github.com/alibaba/canal/wiki/Canal-Admin-Docker)
|
||||
- [AdminGuide](https://github.com/alibaba/canal/wiki/AdminGuide)
|
||||
- [ClientExample](https://github.com/alibaba/canal/wiki/ClientExample)
|
||||
- [ClientAPI](https://github.com/alibaba/canal/wiki/ClientAPI)
|
||||
@@ -81,20 +84,19 @@ canal 特别设计了 client-server 模式，交互协议使用 protobuf 3.0 , c
|
||||
- canal go客户端: [https://github.com/CanalClient/canal-go](https://github.com/CanalClient/canal-go)
|
||||
- canal php客户端: [https://github.com/xingwenge/canal-php](https://github.com/xingwenge/canal-php)
|
||||
- canal Python客户端：[https://github.com/haozi3156666/canal-python](https://github.com/haozi3156666/canal-python)
|
||||
- canal Rust客户端：[https://github.com/laohanlinux/canal-rs](https://github.com/laohanlinux/canal-rs)
|
||||
|
||||
canal 作为 MySQL binlog 增量获取和解析工具，可将变更记录投递到 MQ 系统中，比如 Kafka/RocketMQ，可以借助于 MQ 的多语言能力
|
||||
|
||||
- 参考文档: [Canal Kafka/RocketMQ QuickStart](https://github.com/alibaba/canal/wiki/Canal-Kafka-RocketMQ-QuickStart)
|
||||
|
||||
## 相关开源
|
||||
## 相关开源&产品
|
||||
|
||||
- [canal 消费端开源项目: Otter](http://github.com/alibaba/otter)
|
||||
- [阿里巴巴去 Oracle 数据迁移同步工具: yugong](http://github.com/alibaba/yugong)
|
||||
- [阿里巴巴离线同步开源项目 DataX](https://github.com/alibaba/datax)
|
||||
- [阿里巴巴数据库连接池开源项目 Druid](https://github.com/alibaba/druid)
|
||||
- [阿里巴巴实时数据同步工具 DTS](https://www.aliyun.com/product/dts)
|
||||
|
||||
## 问题反馈
|
||||
- 邮件: jianghang115@gmail.com
|
||||
- 报告 issue: [github issues](https://github.com/alibaba/canal/issues)
|
||||
|
||||
## 其他资料
|
||||
|
||||
- [ ADC 阿里技术嘉年华分享 ppt ( google docs)](https://docs.google.com/presentation/d/1MkszUPYRDkfVPz9IqOT1LLT5d9tuwde_WC8GZvjaDRg/edit?usp=sharing)
|
||||
|
||||
BIN
admin/admin-ui/node/node
Executable file
@@ -1,14 +1,14 @@
|
||||
<?xml version="1.0" encoding="UTF-8"?>
|
||||
<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
|
||||
<parent>
|
||||
<artifactId>canal-admin</artifactId>
|
||||
<artifactId>canal.admin</artifactId>
|
||||
<groupId>com.alibaba.otter</groupId>
|
||||
<version>1.1.4</version>
|
||||
<version>1.1.5</version>
|
||||
</parent>
|
||||
<modelVersion>4.0.0</modelVersion>
|
||||
|
||||
<artifactId>canal-admin-ui</artifactId>
|
||||
|
||||
<artifactId>canal.admin-ui</artifactId>
|
||||
<name>canal admin-ui module for otter ${project.version}</name>
|
||||
<properties>
|
||||
<project.build.sourceEncoding>UTF-8</project.build.sourceEncoding>
|
||||
<project.reporting.outputEncoding>UTF-8</project.reporting.outputEncoding>
|
||||
|
Before Width: | Height: | Size: 6.2 KiB After Width: | Height: | Size: 6.2 KiB |
|
Before Width: | Height: | Size: 66 KiB After Width: | Height: | Size: 66 KiB |
|
Before Width: | Height: | Size: 21 KiB After Width: | Height: | Size: 21 KiB |
|
Before Width: | Height: | Size: 96 KiB After Width: | Height: | Size: 96 KiB |
|
Before Width: | Height: | Size: 4.7 KiB After Width: | Height: | Size: 4.7 KiB |
|
Before Width: | Height: | Size: 2.3 KiB After Width: | Height: | Size: 2.3 KiB |
|
Before Width: | Height: | Size: 497 B After Width: | Height: | Size: 497 B |
|
Before Width: | Height: | Size: 1.3 KiB After Width: | Height: | Size: 1.3 KiB |
|
Before Width: | Height: | Size: 944 B After Width: | Height: | Size: 944 B |
|
Before Width: | Height: | Size: 2.4 KiB After Width: | Height: | Size: 2.4 KiB |
|
Before Width: | Height: | Size: 285 B After Width: | Height: | Size: 285 B |
|
Before Width: | Height: | Size: 821 B After Width: | Height: | Size: 821 B |
|
Before Width: | Height: | Size: 623 B After Width: | Height: | Size: 623 B |
|
Before Width: | Height: | Size: 597 B After Width: | Height: | Size: 597 B |
|
Before Width: | Height: | Size: 1.8 KiB After Width: | Height: | Size: 1.8 KiB |
|
Before Width: | Height: | Size: 440 B After Width: | Height: | Size: 440 B |