Compare commits

..

28 Commits

Author SHA1 Message Date
lcybo
67a3a1d955 Merge pull request #827 from lcybo/metrics_support
Merge master updates
2018-08-09 22:50:32 +08:00
Chuanyi Li
a857b87388 Merge branch 'master' into metrics_support 2018-08-09 22:44:39 +08:00
七锋
b57ac0bcdf fixed merge and blob encoding example 2018-08-09 16:56:13 +08:00
charles.lin
ea6391d1c3 Merge branch 'branch-rds-support' 2018-08-09 16:44:12 +08:00
lcybo
a2eb128698 Merge pull request #826 from lcybo/metrics_support
Metrics support
2018-08-08 22:56:44 +08:00
Chuanyi Li
760f2dda80 Enhance heartbeat, rowscount. 2018-08-08 22:53:08 +08:00
Chuanyi Li
bf9dc0073d Merge branch 'master' into metrics_support 2018-08-08 21:01:22 +08:00
Chuanyi Li
d4797a8fb9 Merge master branch. 2018-08-08 20:35:14 +08:00
Chuanyi Li
060bd8e31d Reformat code. 2018-08-08 20:10:34 +08:00
Chuanyi Li
88de747ed5 Improve sink blocking time resolution. 2018-08-08 07:53:44 +08:00
lcybo
725d36aa98 Merge pull request #819 from lcybo/metrics_support
canal 性能监控改动版,merge到开发分支(metrics_support)
2018-08-07 23:02:37 +08:00
Chuanyi Li
d24e10fb3e Canal metrics snapshot version. 2018-08-07 22:52:30 +08:00
Chuanyi Li
fdae646e26 Merge branch 'master' into metrics_support 2018-08-07 19:46:30 +08:00
Chuanyi Li L
40c3e65341 Refactor metrics code. 2018-08-07 17:17:33 +08:00
Chuanyi Li
b5a1898cec Prometheus refactor. 2018-08-07 08:18:01 +08:00
Chuanyi Li L
680faea2bb Add listener to write. 2018-08-06 17:22:28 +08:00
winger
1cd77bf551 fix bug: kafka get row data for performance 2018-08-06 15:29:14 +08:00
qmzhang
9786faba9a instance由manager管理时,CanalParameter#MetaMode增加本地文件方式,本地meta.dat存储位点信息; 2018-08-06 15:29:14 +08:00
七锋
325fe89aed fixed issue #801 , optimizer docker 2018-08-06 15:29:13 +08:00
七锋
ffeec8fc86 fixed issue #801 , support canal docker 2018-08-06 15:29:13 +08:00
winger
7662c9241e 支持GTID模式下,通过获取当前gtid值 2018-08-06 15:29:13 +08:00
Chuanyi Li L
d01195ac82 Merge branch 'master' into metrics_support 2018-08-06 15:26:44 +08:00
Chuanyi Li
8b186d1330 canal server metrics refactor. 2018-08-06 08:12:59 +08:00
Chuanyi Li L
b7431eb7d0 Reformat code. 2018-08-03 17:42:57 +08:00
Chuanyi Li L
c7fc149e7e Netty server/client profiler skeleton. 2018-08-03 17:32:48 +08:00
Chuanyi Li L
368554166c Builder of Aggregator. 2018-08-02 17:36:22 +08:00
Chuanyi Li L
c4ee350717 Merge branch 'master' of https://github.com/alibaba/canal 2018-08-02 14:55:01 +08:00
Chuanyi Li
ae0f43d1c7 netty3 future listener for metrics. 2018-08-01 08:17:55 +08:00
242 changed files with 12974 additions and 24284 deletions

View File

@@ -1,6 +1,14 @@
<div class="blog_content">
<div class="iteye-blog-content-contain">
<h3>最新更新</h3>
<ol>
<li>canal QQ讨论群已经建立,群号:161559791 ,欢迎加入进行技术讨论。</li>
<li>canal消费端项目开源: Otter(分布式数据库同步系统),地址:<a href="https://github.com/alibaba/otter">https://github.com/alibaba/otter</a></li>
<li>Canal已在阿里云推出商业化版本 <a href="https://www.aliyun.com/product/dts?spm=a2c4g.11186623.cloudEssentials.80.srdwr7">数据传输服务DTS</a>, 开通即用,免去部署维护的昂贵使用成本。DTS针对阿里云RDS、DRDS等产品进行了适配,解决了Binlog日志回收,主备切换、VPC网络切换等场景下的订阅高可用问题。同时,针对RDS进行了针对性的性能优化。出于稳定性、性能及成本的考虑,强烈推荐阿里云用户使用DTS产品。<a href="https://help.aliyun.com/document_detail/26592.html?spm=a2c4g.11174283.6.539.t1Y91E">DTS产品使用文档</a></li>
DTS支持阿里云RDS&DRDS的Binlog日志实时订阅,现推出首月免费体验,限时限量,<a href="https://common-buy.aliyun.com/?commodityCode=dtspre&request=%7b%22dts_function%22%3a%22data_subscribe%22%7d">立即体验>>></a>
</ol>
<h1>背景</h1>
<p style="font-size: 14px;"> 早期,阿里巴巴B2B公司因为存在杭州和美国双机房部署,存在跨机房同步的业务需求。不过早期的数据库同步业务,主要是基于trigger的方式获取增量变更,不过从2010年开始,阿里系公司开始逐步的尝试基于数据库的日志解析,获取增量变更进行同步,由此衍生出了增量订阅&amp;消费的业务,从此开启了一段新纪元。</p>
<p style="font-size: 14px;"> ps. 目前内部版本已经支持mysql和oracle部分版本的日志解析,当前的canal开源版本支持5.7及以下的版本(阿里内部mysql 5.7.13, 5.6.10, mysql 5.5.18和5.1.40/48)</p>
@@ -38,16 +46,6 @@
<li>canal解析binary log对象(原始为byte流)</li>
</ol>
<h1>重要版本更新说明</h1>
canal 1.1.x系列,参考release文档:<a href="https://github.com/alibaba/canal/releases">版本发布信息</a>
1. 整体性能测试&优化,提升了150%. #726 参考: 【[Performance](https://github.com/alibaba/canal/wiki/Performance)】
2. 原生支持prometheus监控 #765 【[Prometheus QuickStart](https://github.com/alibaba/canal/wiki/Prometheus-QuickStart)】
3. 原生支持kafka消息投递 #695 【[Canal Kafka/RocketMQ QuickStart](https://github.com/alibaba/canal/wiki/Canal-Kafka-RocketMQ-QuickStart)】
4. 原生支持aliyun rds的binlog订阅 (解决自动主备切换/oss binlog离线解析) 参考: 【[Aliyun RDS QuickStart](https://github.com/alibaba/canal/wiki/aliyun-RDS-QuickStart)】
5. 原生支持docker镜像 #801 参考: 【[Docker QuickStart](https://github.com/alibaba/canal/wiki/Docker-QuickStart)】
<h1>相关文档</h1>
See the wiki page for : <a href="https://github.com/alibaba/canal/wiki" >wiki文档</a>
@@ -56,41 +54,20 @@ See the wiki page for : <a href="https://github.com/alibaba/canal/wiki" >wiki文
<ul>
<li><a class="internal present" href="https://github.com/alibaba/canal/wiki/Home">Home</a></li>
<li><a class="internal present" href="https://github.com/alibaba/canal/wiki/Introduction">Introduction</a></li>
<li>
<a class="internal present" href="https://github.com/alibaba/canal/wiki/QuickStart">QuickStart</a>
<ul>
<li><a class="internal present" href="https://github.com/alibaba/canal/wiki/Docker-QuickStart">Docker QuickStart</a></li>
<li><a class="internal present" href="https://github.com/alibaba/canal/wiki/Canal-Kafka-RocketMQ-QuickStart">Canal Kafka/RocketMQ QuickStart</a></li>
<li><a class="internal present" href="https://github.com/alibaba/canal/wiki/aliyun-RDS-QuickStart">Aliyun RDS QuickStart</a></li>
<li><a class="internal present" href="https://github.com/alibaba/canal/wiki/Prometheus-QuickStart">Prometheus QuickStart</a></li>
</ul>
</li>
<li><a class="internal present" href="https://github.com/alibaba/canal/wiki/AdminGuide">AdminGuide</a></li>
<li><a class="internal present" href="https://github.com/alibaba/canal/wiki/QuickStart">QuickStart</a></li>
<li><a class="internal present" href="https://github.com/alibaba/canal/wiki/ClientExample">ClientExample</a></li>
<li><a class="internal present" href="https://github.com/alibaba/canal/wiki/AdminGuide">AdminGuide</a></li>
<li><a class="internal present" href="https://github.com/alibaba/canal/wiki/ClientAPI">ClientAPI</a></li>
<li><a class="internal present" href="https://github.com/alibaba/canal/wiki/Performance">Performance</a></li>
<li><a class="internal present" href="https://github.com/alibaba/canal/wiki/DevGuide">DevGuide</a></li>
<li><a class="internal present" href="https://github.com/alibaba/canal/wiki/BinlogChange%28mysql5.6%29">BinlogChange(Mysql5.6)</a></li>
<li><a class="internal present" href="https://github.com/alibaba/canal/wiki/BinlogChange%28MariaDB%29">BinlogChange(MariaDB)</a></li>
<li><a class="internal present" href="https://github.com/alibaba/canal/wiki/TableMetaTSDB">TableMetaTSDB</a></li>
<li><a href="http://alibaba.github.com/canal/release.html">ReleaseNotes</a></li>
<li><a href="https://github.com/alibaba/canal/releases">Download</a></li>
<li><a class="internal present" href="/alibaba/canal/wiki/FAQ">FAQ</a></li>
</ul>
<h1>多语言业务</h1>
1. canal整体交互协议设计上使用了protobuf3.0,理论上可以支持绝大部分的多语言场景,欢迎大家提交多客户端的PR
* canal java客户端: <a href="https://github.com/alibaba/canal/wiki/ClientExample"> https://github.com/alibaba/canal/wiki/ClientExample </a>
* canal c#客户端开源项目地址:<a href="https://github.com/CanalClient/CanalSharp"> https://github.com/CanalSharp/CanalSharp </a>
* canal go客户端开源项目地址:<a href="https://github.com/CanalClient/canal-go"> https://github.com/CanalClient/canal-go </a>
2. canal作为MySQL binlog的增量获取工具,可以将数据投递到MQ系统中,比如Kafka/RocketMQ,可以借助于MQ的多语言能力
* 参考文档: [Canal Kafka/RocketMQ QuickStart](https://github.com/alibaba/canal/wiki/Canal-Kafka-RocketMQ-QuickStart)
<h1>相关资料</h1>
* ADC阿里技术嘉年华分享ppt (放在google docs上,可能需要翻墙): <a href="https://docs.google.com/presentation/d/1MkszUPYRDkfVPz9IqOT1LLT5d9tuwde_WC8GZvjaDRg/edit?usp=sharing">ppt下载</href>
* [与阿里巴巴的RocketMQ配合使用](https://github.com/apache/RocketMQ)
* [与阿里巴巴的RocketMQ配合使用](https://github.com/alibaba/RocketMQ)
<h1>相关开源</h1>
<ol>
@@ -114,13 +91,9 @@ See the wiki page for : <a href="https://github.com/alibaba/canal/wiki" >wiki文
<li>报告issue:<a href="https://github.com/alibaba/canal/issues">issues</a></li>
</ol>
<h3>最新更新</h3>
<ol>
<li>canal发布重大版本更新1.1.0,具体releaseNode参考:<a href="https://github.com/alibaba/canal/releases/tag/canal-1.1.0">https://github.com/alibaba/canal/releases/tag/canal-1.1.0</a></li>
<li>canal c#客户端开源项目地址:<a href="https://github.com/CanalClient/CanalSharp"> https://github.com/CanalClient/CanalSharp </a>,推荐! </li>
<li>canal QQ讨论群已经建立,群号:161559791 ,欢迎加入进行技术讨论。</li>
<li>canal消费端项目开源: Otter(分布式数据库同步系统),地址:<a href="https://github.com/alibaba/otter">https://github.com/alibaba/otter</a></li>
<li>Canal已在阿里云推出商业化版本 <a href="https://www.aliyun.com/product/dts?spm=a2c4g.11186623.cloudEssentials.80.srdwr7">数据传输服务DTS</a>, 开通即用,免去部署维护的昂贵使用成本。DTS针对阿里云RDS、DRDS等产品进行了适配,解决了Binlog日志回收,主备切换、VPC网络切换等场景下的订阅高可用问题。同时,针对RDS进行了针对性的性能优化。出于稳定性、性能及成本的考虑,强烈推荐阿里云用户使用DTS产品。<a href="https://help.aliyun.com/document_detail/26592.html?spm=a2c4g.11174283.6.539.t1Y91E">DTS产品使用文档</a></li>
DTS支持阿里云RDS&DRDS的Binlog日志实时订阅,现推出首月免费体验,限时限量,<a href="https://common-buy.aliyun.com/?commodityCode=dtspre&request=%7b%22dts_function%22%3a%22data_subscribe%22%7d">立即体验>>></a>
</ol>
<pre>
【招聘】阿里巴巴中间件团队招聘JAVA高级工程师
岗位主要为技术型内容(非业务部门),阿里中间件整个体系对于未来想在技术上有所沉淀的同学还是非常有帮助的
工作地点:杭州、北京均可. ps. 阿里待遇向来都是不错的,有意者可以QQ、微博私聊.
具体招聘内容:https://job.alibaba.com/zhaopin/position_detail.htm?positionId=32666
</pre>

View File

@@ -1,251 +0,0 @@
## 基本说明
canal 1.1.1版本之后, 增加客户端数据落地的适配及启动功能, 目前支持功能:
* 客户端启动器
* 同步管理REST接口
* 日志适配器, 作为DEMO
* HBase的数据同步(表对表同步), ETL功能
* (后续支持) ElasticSearch多表数据同步,ETL功能
## 环境版本
* 操作系统:无要求
* java版本: jdk1.8 以上
* canal 版本: 请下载最新的安装包,本文以当前v1.1.1 的canal.deployer-1.1.1.tar.gz为例
* MySQL版本 :5.7.18
* HBase版本: Apache HBase 1.1.2, 若和服务端版本不一致可自行替换客户端HBase依赖
## 一、适配器启动器
client-adapter分为适配器和启动器两部分, 适配器为多个fat jar, 每个适配器会将自己所需的依赖打成一个包, 以SPI的方式让启动器动态加载
启动器为 SpringBoot 项目, 支持canal-client启动的同时提供相关REST管理接口, 运行目录结构为:
```
canal-adapter-launcher.jar
- lib
client-adapter.logger-1.1.1-jar-with-dependencies.jar
client-adapter.hbase-1.1.1-jar-with-dependencies.jar
- config
application.yml
- hbase
mytest_person2.yml
```
以上目录结构最终会打包成 canal-adapter-launcher.tar.gz 压缩包
## 二、启动器
### 2.1 启动器配置 application.yml
#### canal相关配置部分说明
```
canal.conf:
canalServerHost: 127.0.0.1:11111 # 对应单机模式下的canal server的ip:port
zookeeperHosts: slave1:2181 # 对应集群模式下的zk地址, 如果配置了canalServerHost, 则以canalServerHost为准
bootstrapServers: slave1:6667 # kafka或rocketMQ地址, 与canalServerHost不能并存
flatMessage: true # 扁平message开关, 是否以json字符串形式投递数据, 仅在kafka/rocketMQ模式下有效
canalInstances: # canal实例组, 如果是tcp模式可配置此项
- instance: example # 对应canal destination
groups: # 对应适配器分组, 分组间的适配器并行运行
- outAdapters: # 适配器列表, 分组内的适配串行运行
- name: logger # 适配器SPI名
- name: hbase
properties: # HBase相关连接参数
hbase.zookeeper.quorum: slave1
hbase.zookeeper.property.clientPort: 2181
zookeeper.znode.parent: /hbase
mqTopics: # MQ topic租, 如果是kafka或者rockeMQ模式可配置此项, 与canalInstances不能并存
- mqMode: kafka # MQ的模式: kafak/rocketMQ
topic: example # MQ topic
groups: # group组
- groupId: g2 # group id
outAdapters: # 适配器列表, 以下配置和canalInstances中一样
- name: logger
```
#### 适配器相关配置部分说明
```
adapter.conf:
datasourceConfigs: # 数据源配置列表, 数据源将在适配器中用于ETL、数据同步回查等使用
defaultDS: # 数据源 dataSourceKey, 适配器中通过该值获取指定数据源
url: jdbc:mysql://127.0.0.1:3306/mytest?useUnicode=true
username: root
password: 121212
adapterConfigs: # 适配器内部配置列表
- hbase/mytest_person2.yml # 类型/配置文件名, 这里示例就是对应HBase适配器hbase目录下的mytest_person2.yml文件
```
## 2.2 同步管理REST接口
#### 2.2.1 查询所有订阅同步的canal destination或MQ topic
```
curl http://127.0.0.1:8081/destinations
```
#### 2.2.2 数据同步开关
```
curl http://127.0.0.1:8081/syncSwitch/example/off -X PUT
```
针对 example 这个canal destination/MQ topic 进行开关操作. off代表关闭, 该destination/topic下的同步将阻塞或者断开连接不再接收数据, on代表开启
注: 如果在配置文件中配置了 zookeeperHosts 项, 则会使用分布式锁来控制HA中的数据同步开关, 如果是单机模式则使用本地锁来控制开关
#### 2.2.3 数据同步开关状态
```
curl http://127.0.0.1:8081/syncSwitch/example
```
查看指定 canal destination/MQ topic 的数据同步开关状态
#### 2.2.4 手动ETL
```
curl http://127.0.0.1:8081/etl/hbase/mytest_person2.yml -X POST -d "params=2018-10-21 00:00:00"
```
导入数据到指定类型的库
#### 2.2.5 查看相关库总数据
```
curl http://127.0.0.1:8081/count/hbase/mytest_person2.yml
```
### 2.3 启动canal-adapter示例
#### 2.3.1 启动canal server (单机模式), 参考: [Canal QuickStart](https://github.com/alibaba/canal/wiki/QuickStart)
#### 2.3.2 修改config/application.yml为:
```
server:
port: 8081
logging:
level:
com.alibaba.otter.canal.client.adapter.hbase: DEBUG
spring:
jackson:
date-format: yyyy-MM-dd HH:mm:ss
time-zone: GMT+8
default-property-inclusion: non_null
canal.conf:
canalServerHost: 127.0.0.1:11111
flatMessage: true
canalInstances:
- instance: example
adapterGroups:
- outAdapters:
- name: logger
```
启动 canal-adapter-launcher.jar
```
java -jar canal-adapter-launcher.jar
```
## 三、HBase适配器
### 3.1 修改启动器配置: application.yml
```
server:
port: 8081
logging:
level:
com.alibaba.otter.canal.client.adapter.hbase: DEBUG
spring:
jackson:
date-format: yyyy-MM-dd HH:mm:ss
time-zone: GMT+8
default-property-inclusion: non_null
canal.conf:
canalServerHost: 127.0.0.1:11111
flatMessage: true
canalInstances:
- instance: example
adapterGroups:
- outAdapters:
- name: hbase
properties:
hbase.zookeeper.quorum: slave1
hbase.zookeeper.property.clientPort: 2181
zookeeper.znode.parent: /hbase
adapter.conf:
datasourceConfigs:
defaultDS:
url: jdbc:mysql://127.0.0.1:3306/mytest?useUnicode=true
username: root
password: 121212
adapterConfigs:
- hbase/mytest_person.yml
```
其中指定了一个HBase表映射文件: mytest_person.yml
### 3.2 适配器表映射文件
修改 config/hbase/mytest_person.yml文件:
```
dataSourceKey: defaultDS # 对应application.yml中的datasourceConfigs下的配置
hbaseMapping: # mysql--HBase的单表映射配置
mode: STRING # HBase中的存储类型, 默认统一存为String, 可选: #PHOENIX #NATIVE #STRING
# NATIVE: 以java类型为主, PHOENIX: 将类型转换为Phoenix对应的类型
destination: example # 对应 canal destination/MQ topic 名称
database: mytest # 数据库名/schema名
table: person # 表名
hbaseTable: MYTEST.PERSON # HBase表名
family: CF # 默认统一Column Family名称
uppercaseQualifier: true # 字段名转大写, 默认为true
commitBatch: 3000 # 批量提交的大小, ETL中用到
#rowKey: id,type # 复合字段rowKey不能和columns中的rowKey并存
# 复合rowKey会以 '|' 分隔
columns: # 字段映射, 如果不配置将自动映射所有字段,
# 并取第一个字段为rowKey, HBase字段名以mysql字段名为主
id: ROWKE
name: CF:NAME
email: EMAIL # 如果column family为默认CF, 则可以省略
type: # 如果HBase字段和mysql字段名一致, 则可以省略
c_time:
birthday:
```
如果涉及到类型转换,可以如下形式:
```
...
columns:
id: ROWKE$STRING
...
type: TYPE$BYTE
...
```
类型转换涉及到Java类型和Phoenix类型两种, 分别定义如下:
```
#Java 类型转换, 对应配置 mode: NATIVE
$DEFAULT
$STRING
$INTEGER
$LONG
$SHORT
$BOOLEAN
$FLOAT
$DOUBLE
$BIGDECIMAL
$DATE
$BYTE
$BYTES
```
```
#Phoenix 类型转换, 对应配置 mode: PHOENIX
$DEFAULT 对应PHOENIX里的VARCHAR
$UNSIGNED_INT 对应PHOENIX里的UNSIGNED_INT 4字节
$UNSIGNED_LONG 对应PHOENIX里的UNSIGNED_LONG 8字节
$UNSIGNED_TINYINT 对应PHOENIX里的UNSIGNED_TINYINT 1字节
$UNSIGNED_SMALLINT 对应PHOENIX里的UNSIGNED_SMALLINT 2字节
$UNSIGNED_FLOAT 对应PHOENIX里的UNSIGNED_FLOAT 4字节
$UNSIGNED_DOUBLE 对应PHOENIX里的UNSIGNED_DOUBLE 8字节
$INTEGER 对应PHOENIX里的INTEGER 4字节
$BIGINT 对应PHOENIX里的BIGINT 8字节
$TINYINT 对应PHOENIX里的TINYINT 1字节
$SMALLINT 对应PHOENIX里的SMALLINT 2字节
$FLOAT 对应PHOENIX里的FLOAT 4字节
$DOUBLE 对应PHOENIX里的DOUBLE 8字节
$BOOLEAN 对应PHOENIX里的BOOLEAN 1字节
$TIME 对应PHOENIX里的TIME 8字节
$DATE 对应PHOENIX里的DATE 8字节
$TIMESTAMP 对应PHOENIX里的TIMESTAMP 12字节
$UNSIGNED_TIME 对应PHOENIX里的UNSIGNED_TIME 8字节
$UNSIGNED_DATE 对应PHOENIX里的UNSIGNED_DATE 8字节
$UNSIGNED_TIMESTAMP 对应PHOENIX里的UNSIGNED_TIMESTAMP 12字节
$VARCHAR 对应PHOENIX里的VARCHAR 动态长度
$VARBINARY 对应PHOENIX里的VARBINARY 动态长度
$DECIMAL 对应PHOENIX里的DECIMAL 动态长度
```
如果不配置将以java对象原生类型默认映射转换
### 3.3 启动HBase数据同步
#### 创建HBase表
在HBase shell中运行:
```
create 'MYTEST.PERSON', {NAME=>'CF'}
```
#### 启动canal-adapter启动器
```
java -jar canal-adapter-launcher.jar
```
#### 验证
修改mysql mytest.person表的数据, 将会自动同步到HBase的MYTEST.PERSON表下面, 并会打出DML的log

View File

@@ -1,36 +0,0 @@
<?xml version="1.0" encoding="UTF-8"?>
<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
<parent>
<artifactId>canal.client-adapter</artifactId>
<groupId>com.alibaba.otter</groupId>
<version>1.1.1</version>
</parent>
<modelVersion>4.0.0</modelVersion>
<groupId>com.alibaba.otter</groupId>
<artifactId>client-adapter.common</artifactId>
<packaging>jar</packaging>
<name>canal client adapter common module for otter ${project.version}</name>
<dependencies>
<dependency>
<groupId>com.alibaba.otter</groupId>
<artifactId>canal.protocol</artifactId>
<version>1.1.1</version>
</dependency>
<dependency>
<groupId>joda-time</groupId>
<artifactId>joda-time</artifactId>
<version>2.9.4</version>
</dependency>
<dependency>
<groupId>com.alibaba</groupId>
<artifactId>druid</artifactId>
<version>1.1.9</version>
</dependency>
<dependency>
<groupId>mysql</groupId>
<artifactId>mysql-connector-java</artifactId>
<version>5.1.40</version>
</dependency>
</dependencies>
</project>

View File

@@ -1,68 +0,0 @@
package com.alibaba.otter.canal.client.adapter;
import java.util.List;
import java.util.Map;
import com.alibaba.otter.canal.client.adapter.support.Dml;
import com.alibaba.otter.canal.client.adapter.support.EtlResult;
import com.alibaba.otter.canal.client.adapter.support.OuterAdapterConfig;
import com.alibaba.otter.canal.client.adapter.support.SPI;
/**
* 外部适配器接口
*
* @author reweerma 2018-8-18 下午10:14:02
* @version 1.0.0
*/
@SPI("logger")
public interface OuterAdapter {
/**
* 外部适配器初始化接口
*
* @param configuration 外部适配器配置信息
*/
void init(OuterAdapterConfig configuration);
/**
* 往适配器中同步数据
*
* @param dml 数据包
*/
void sync(Dml dml);
/**
* 外部适配器销毁接口
*/
void destroy();
/**
* Etl操作
*
* @param task 任务名, 对应配置名
* @param params etl筛选条件
*/
default EtlResult etl(String task, List<String> params) {
throw new UnsupportedOperationException("unsupported operation");
}
/**
* 计算总数
*
* @param task 任务名, 对应配置名
* @return 总数
*/
default Map<String, Object> count(String task) {
throw new UnsupportedOperationException("unsupported operation");
}
/**
* 通过task获取对应的destination
*
* @param task 任务名, 对应配置名
* @return destination
*/
default String getDestination(String task) {
return null;
}
}

View File

@@ -1,41 +0,0 @@
package com.alibaba.otter.canal.client.adapter.support;
import java.util.HashMap;
import java.util.LinkedHashSet;
import java.util.Map;
import java.util.Set;
/**
* 适配器配置集合, 用于配置加载, 线程不安全
*
* @author rewerma @ 2018-10-20
* @version 1.0.0
*/
public class AdapterConfigs {
/**
* 类型下对应所有配置名, 如:
* hbase
* ┗━ mytest_person.yml
* ┗━ mytest_role.yml
* ┗━ mytest_department.yml
*/
private static Map<String, Set<String>> configs = new HashMap<>();
public static void put(String key, String value) {
Set<String> values = configs.get(key);
if (values == null) {
values = new LinkedHashSet<>();
}
values.add(value);
configs.put(key, values);
}
public static Set<String> get(String key) {
return configs.get(key);
}
public static void clear() {
configs.clear();
}
}

View File

@@ -1,170 +0,0 @@
package com.alibaba.otter.canal.client.adapter.support;
import java.util.ArrayList;
import java.util.List;
/**
* 配置信息类
*
* @author rewerma 2018-8-18 下午10:40:12
* @version 1.0.0
*/
public class CanalClientConfig {
private String canalServerHost; // 单机模式下canal server的 ip:port
private String zookeeperHosts; // 集群模式下的zk地址, 如果配置了单机地址则以单机为准!!
private String bootstrapServers; // kafka or rocket mq 地址
private Boolean flatMessage = true; // 是否已flatMessage模式传输, 只适用于mq模式
private List<MQTopic> mqTopics; // mq topic 列表
private List<CanalInstance> canalInstances; // tcp 模式下 canal 实例列表, 与mq模式不能共存!!
public String getCanalServerHost() {
return canalServerHost;
}
public void setCanalServerHost(String canalServerHost) {
this.canalServerHost = canalServerHost;
}
public String getZookeeperHosts() {
return zookeeperHosts;
}
public void setZookeeperHosts(String zookeeperHosts) {
this.zookeeperHosts = zookeeperHosts;
}
public String getBootstrapServers() {
return bootstrapServers;
}
public void setBootstrapServers(String bootstrapServers) {
this.bootstrapServers = bootstrapServers;
}
public List<MQTopic> getMqTopics() {
return mqTopics;
}
public void setMqTopics(List<MQTopic> mqTopics) {
this.mqTopics = mqTopics;
}
public Boolean getFlatMessage() {
return flatMessage;
}
public void setFlatMessage(Boolean flatMessage) {
this.flatMessage = flatMessage;
}
public List<CanalInstance> getCanalInstances() {
return canalInstances;
}
public void setCanalInstances(List<CanalInstance> canalInstances) {
this.canalInstances = canalInstances;
}
public static class CanalInstance {
private String instance; // 实例名
private List<Group> groups; // 适配器分组列表
public String getInstance() {
return instance;
}
public void setInstance(String instance) {
if (instance != null) {
this.instance = instance.trim();
}
}
public List<Group> getGroups() {
return groups;
}
public void setGroups(List<Group> groups) {
this.groups = groups;
}
}
public static class Group {
private List<OuterAdapterConfig> outAdapters; // 适配器列表
public List<OuterAdapterConfig> getOutAdapters() {
return outAdapters;
}
public void setOutAdapters(List<OuterAdapterConfig> outAdapters) {
this.outAdapters = outAdapters;
}
}
public static class MQTopic {
private String mqMode; // mq模式 kafka or rocketMQ
private String topic; // topic名
private List<MQGroup> groups = new ArrayList<>(); // 分组列表
public String getMqMode() {
return mqMode;
}
public void setMqMode(String mqMode) {
this.mqMode = mqMode;
}
public String getTopic() {
return topic;
}
public void setTopic(String topic) {
this.topic = topic;
}
public List<MQGroup> getGroups() {
return groups;
}
public void setGroups(List<MQGroup> groups) {
this.groups = groups;
}
}
public static class MQGroup {
private String groupId; // group id
private List<OuterAdapterConfig> outAdapters; // 适配器配置列表
public String getGroupId() {
return groupId;
}
public void setGroupId(String groupId) {
this.groupId = groupId;
}
public List<OuterAdapterConfig> getOutAdapters() {
return outAdapters;
}
public void setOutAdapters(List<OuterAdapterConfig> outAdapters) {
this.outAdapters = outAdapters;
}
}
}

View File

@@ -1,81 +0,0 @@
package com.alibaba.otter.canal.client.adapter.support;
import java.util.Map;
import java.util.concurrent.ConcurrentHashMap;
import com.alibaba.druid.pool.DruidDataSource;
/**
* 数据源配置
*
* @author rewerma @ 2018-10-20
* @version 1.0.0
*/
public class DatasourceConfig {
public final static Map<String, DruidDataSource> DATA_SOURCES = new ConcurrentHashMap<>(); // key对应的数据源
private String driver = "com.mysql.jdbc.Driver"; // 默认为mysql jdbc驱动
private String url; // jdbc url
private String database; // jdbc database
private String type = "mysql"; // 类型, 默认为mysql
private String username; // jdbc username
private String password; // jdbc password
private Integer maxActive = 3; // 连接池最大连接数,默认为3
public String getDriver() {
return driver;
}
public void setDriver(String driver) {
this.driver = driver;
}
public String getUrl() {
return url;
}
public void setUrl(String url) {
this.url = url;
}
public String getDatabase() {
return database;
}
public void setDatabase(String database) {
this.database = database;
}
public String getType() {
return type;
}
public void setType(String type) {
this.type = type;
}
public String getUsername() {
return username;
}
public void setUsername(String username) {
this.username = username;
}
public String getPassword() {
return password;
}
public void setPassword(String password) {
this.password = password;
}
public Integer getMaxActive() {
return maxActive;
}
public void setMaxActive(Integer maxActive) {
this.maxActive = maxActive;
}
}

View File

@@ -1,118 +0,0 @@
package com.alibaba.otter.canal.client.adapter.support;
import java.io.Serializable;
import java.util.List;
import java.util.Map;
/**
* DML操作转换对象
*
* @author rewerma 2018-8-19 下午11:30:49
* @version 1.0.0
*/
public class Dml implements Serializable {
private static final long serialVersionUID = 2611556444074013268L;
private String destination; // 对应canal的实例或者MQ的topic
private String database; // 数据库或schema
private String table; // 表名
private String type; // 类型: INSERT UPDATE DELETE
// binlog executeTime
private Long es; // 执行耗时
// dml build timeStamp
private Long ts; // 同步时间
private String sql; // 执行的sql, dml sql为空
private List<Map<String, Object>> data; // 数据列表
private List<Map<String, Object>> old; // 旧数据列表, 用于update, size和data的size一一对应
public String getDestination() {
return destination;
}
public void setDestination(String destination) {
this.destination = destination;
}
public String getDatabase() {
return database;
}
public void setDatabase(String database) {
this.database = database;
}
public String getTable() {
return table;
}
public void setTable(String table) {
this.table = table;
}
public String getType() {
return type;
}
public void setType(String type) {
this.type = type;
}
public Long getTs() {
return ts;
}
public void setTs(Long ts) {
this.ts = ts;
}
public String getSql() {
return sql;
}
public void setSql(String sql) {
this.sql = sql;
}
public List<Map<String, Object>> getData() {
return data;
}
public void setData(List<Map<String, Object>> data) {
this.data = data;
}
public List<Map<String, Object>> getOld() {
return old;
}
public void setOld(List<Map<String, Object>> old) {
this.old = old;
}
public Long getEs() {
return es;
}
public void setEs(Long es) {
this.es = es;
}
public void clear() {
database = null;
table = null;
type = null;
ts = null;
es = null;
data = null;
old = null;
sql = null;
}
@Override
public String toString() {
return "Dml{" + "destination='" + destination + '\'' + ", database='" + database + '\'' + ", table='" + table
+ '\'' + ", type='" + type + '\'' + ", es=" + es + ", ts=" + ts + ", sql='" + sql + '\'' + ", data="
+ data + ", old=" + old + '}';
}
}

View File

@@ -1,44 +0,0 @@
package com.alibaba.otter.canal.client.adapter.support;
import java.io.Serializable;
/**
* ETL的结果对象
*
* @author rewerma @ 2018-10-20
* @version 1.0.0
*/
public class EtlResult implements Serializable {
private static final long serialVersionUID = 4250522736289866505L;
private boolean succeeded = false;
private String resultMessage;
private String errorMessage;
public boolean getSucceeded() {
return succeeded;
}
public void setSucceeded(boolean succeeded) {
this.succeeded = succeeded;
}
public String getResultMessage() {
return resultMessage;
}
public void setResultMessage(String resultMessage) {
this.resultMessage = resultMessage;
}
public String getErrorMessage() {
return errorMessage;
}
public void setErrorMessage(String errorMessage) {
this.errorMessage = errorMessage;
}
}

View File

@@ -1,574 +0,0 @@
package com.alibaba.otter.canal.client.adapter.support;
import java.io.*;
import java.net.MalformedURLException;
import java.net.URL;
import java.net.URLClassLoader;
import java.nio.file.Paths;
import java.util.*;
import java.util.concurrent.ConcurrentHashMap;
import java.util.concurrent.ConcurrentMap;
import java.util.regex.Pattern;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
/**
* SPI 类加载器
*
* @author rewerma 2018-8-19 下午11:30:49
* @version 1.0.0
*/
public class ExtensionLoader<T> {
private static final Logger logger = LoggerFactory
.getLogger(ExtensionLoader.class);
private static final String SERVICES_DIRECTORY = "META-INF/services/";
private static final String CANAL_DIRECTORY = "META-INF/canal/";
private static final String DEFAULT_CLASSLOADER_POLICY = "internal";
private static final Pattern NAME_SEPARATOR = Pattern
.compile("\\s*[,]+\\s*");
private static final ConcurrentMap<Class<?>, ExtensionLoader<?>> EXTENSION_LOADERS = new ConcurrentHashMap<>();
private static final ConcurrentMap<Class<?>, Object> EXTENSION_INSTANCES = new ConcurrentHashMap<>();
private final Class<?> type;
private final String classLoaderPolicy;
private final ConcurrentMap<Class<?>, String> cachedNames = new ConcurrentHashMap<>();
private final Holder<Map<String, Class<?>>> cachedClasses = new Holder<>();
private final ConcurrentMap<String, Holder<Object>> cachedInstances = new ConcurrentHashMap<>();
private String cachedDefaultName;
private ConcurrentHashMap<String, IllegalStateException> exceptions = new ConcurrentHashMap<>();
private static <T> boolean withExtensionAnnotation(Class<T> type) {
return type.isAnnotationPresent(SPI.class);
}
public static <T> ExtensionLoader<T> getExtensionLoader(Class<T> type) {
return getExtensionLoader(type, DEFAULT_CLASSLOADER_POLICY);
}
@SuppressWarnings("unchecked")
public static <T> ExtensionLoader<T> getExtensionLoader(Class<T> type, String classLoaderPolicy) {
if (type == null) throw new IllegalArgumentException("Extension type == null");
if (!type.isInterface()) {
throw new IllegalArgumentException("Extension type(" + type + ") is not interface!");
}
if (!withExtensionAnnotation(type)) {
throw new IllegalArgumentException("Extension type(" + type + ") is not extension, because WITHOUT @"
+ SPI.class.getSimpleName() + " Annotation!");
}
ExtensionLoader<T> loader = (ExtensionLoader<T>) EXTENSION_LOADERS.get(type);
if (loader == null) {
EXTENSION_LOADERS.putIfAbsent(type, new ExtensionLoader<T>(type, classLoaderPolicy));
loader = (ExtensionLoader<T>) EXTENSION_LOADERS.get(type);
}
return loader;
}
private ExtensionLoader(Class<?> type, String classLoaderPolicy){
this.type = type;
this.classLoaderPolicy = classLoaderPolicy;
}
public String getExtensionName(T extensionInstance) {
return getExtensionName(extensionInstance.getClass());
}
public String getExtensionName(Class<?> extensionClass) {
return cachedNames.get(extensionClass);
}
public ConcurrentHashMap<String, IllegalStateException> getExceptions() {
return exceptions;
}
/**
* 返回扩展点实例,如果没有指定的扩展点或是还没加载(即实例化)则返回<code>null</code>注意:此方法不会触发扩展点的加载
* <p/>
* 一般应该调用{@link #getExtension(String)}方法获得扩展,这个方法会触发扩展点加载
*
* @see #getExtension(String)
*/
@SuppressWarnings("unchecked")
public T getLoadedExtension(String name) {
if (name == null || name.length() == 0) throw new IllegalArgumentException("Extension name == null");
Holder<Object> holder = cachedInstances.get(name);
if (holder == null) {
cachedInstances.putIfAbsent(name, new Holder<>());
holder = cachedInstances.get(name);
}
return (T) holder.get();
}
/**
* 返回已经加载的扩展点的名字
* <p/>
* 一般应该调用{@link #getSupportedExtensions()}方法获得扩展,这个方法会返回所有的扩展点
*
* @see #getSupportedExtensions()
*/
public Set<String> getLoadedExtensions() {
return Collections.unmodifiableSet(new TreeSet<>(cachedInstances.keySet()));
}
/**
* 返回指定名字的扩展
*
* @param name
* @return
*/
@SuppressWarnings("unchecked")
public T getExtension(String name) {
if (name == null || name.length() == 0) throw new IllegalArgumentException("Extension name == null");
if ("true".equals(name)) {
return getDefaultExtension();
}
Holder<Object> holder = cachedInstances.get(name);
if (holder == null) {
cachedInstances.putIfAbsent(name, new Holder<>());
holder = cachedInstances.get(name);
}
Object instance = holder.get();
if (instance == null) {
synchronized (holder) {
instance = holder.get();
if (instance == null) {
instance = createExtension(name);
holder.set(instance);
}
}
}
return (T) instance;
}
/**
* 返回缺省的扩展,如果没有设置则返回<code>null</code>
*/
public T getDefaultExtension() {
getExtensionClasses();
if (null == cachedDefaultName || cachedDefaultName.length() == 0 || "true".equals(cachedDefaultName)) {
return null;
}
return getExtension(cachedDefaultName);
}
public boolean hasExtension(String name) {
if (name == null || name.length() == 0) throw new IllegalArgumentException("Extension name == null");
try {
return getExtensionClass(name) != null;
} catch (Throwable t) {
return false;
}
}
public Set<String> getSupportedExtensions() {
Map<String, Class<?>> clazzes = getExtensionClasses();
return Collections.unmodifiableSet(new TreeSet<String>(clazzes.keySet()));
}
/**
* 返回缺省的扩展点名,如果没有设置缺省则返回<code>null</code>
*/
public String getDefaultExtensionName() {
getExtensionClasses();
return cachedDefaultName;
}
/**
* 编程方式添加新扩展点
*
* @param name 扩展点名
* @param clazz 扩展点类
* @throws IllegalStateException 要添加扩展点名已经存在
*/
public void addExtension(String name, Class<?> clazz) {
getExtensionClasses(); // load classes
if (!type.isAssignableFrom(clazz)) {
throw new IllegalStateException("Input type " + clazz + "not implement Extension " + type);
}
if (clazz.isInterface()) {
throw new IllegalStateException("Input type " + clazz + "can not be interface!");
}
if (name == null || "".equals(name)) {
throw new IllegalStateException("Extension name is blank (Extension " + type + ")!");
}
if (cachedClasses.get().containsKey(name)) {
throw new IllegalStateException("Extension name " + name + " already existed(Extension " + type + ")!");
}
cachedNames.put(clazz, name);
cachedClasses.get().put(name, clazz);
}
/**
* 编程方式添加替换已有扩展点
*
* @param name 扩展点名
* @param clazz 扩展点类
* @throws IllegalStateException 要添加扩展点名已经存在
* @deprecated 不推荐应用使用,一般只在测试时可以使用
*/
@Deprecated
public void replaceExtension(String name, Class<?> clazz) {
getExtensionClasses(); // load classes
if (!type.isAssignableFrom(clazz)) {
throw new IllegalStateException("Input type " + clazz + "not implement Extension " + type);
}
if (clazz.isInterface()) {
throw new IllegalStateException("Input type " + clazz + "can not be interface!");
}
if (name == null || "".equals(name)) {
throw new IllegalStateException("Extension name is blank (Extension " + type + ")!");
}
if (!cachedClasses.get().containsKey(name)) {
throw new IllegalStateException("Extension name " + name + " not existed(Extension " + type + ")!");
}
cachedNames.put(clazz, name);
cachedClasses.get().put(name, clazz);
cachedInstances.remove(name);
}
@SuppressWarnings("unchecked")
private T createExtension(String name) {
Class<?> clazz = getExtensionClasses().get(name);
if (clazz == null) {
throw new IllegalStateException("Extension instance(name: " + name + ", class: " + type
+ ") could not be instantiated: class could not be found");
}
try {
T instance = (T) EXTENSION_INSTANCES.get(clazz);
if (instance == null) {
EXTENSION_INSTANCES.putIfAbsent(clazz, (T) clazz.newInstance());
instance = (T) EXTENSION_INSTANCES.get(clazz);
}
return instance;
} catch (Throwable t) {
throw new IllegalStateException("Extension instance(name: " + name + ", class: " + type
+ ") could not be instantiated: " + t.getMessage(),
t);
}
}
private Class<?> getExtensionClass(String name) {
if (type == null) throw new IllegalArgumentException("Extension type == null");
if (name == null) throw new IllegalArgumentException("Extension name == null");
Class<?> clazz = getExtensionClasses().get(name);
if (clazz == null)
throw new IllegalStateException("No such extension \"" + name + "\" for " + type.getName() + "!");
return clazz;
}
private Map<String, Class<?>> getExtensionClasses() {
Map<String, Class<?>> classes = cachedClasses.get();
if (classes == null) {
synchronized (cachedClasses) {
classes = cachedClasses.get();
if (classes == null) {
classes = loadExtensionClasses();
cachedClasses.set(classes);
}
}
}
return classes;
}
private String getJarDirectoryPath() {
URL url = Thread.currentThread().getContextClassLoader().getResource("");
if (url == null) {
throw new IllegalStateException("failed to get class loader resource");
}
String dirtyPath = url.toString();
String jarPath = dirtyPath.replaceAll("^.*file:/", ""); // removes
// file:/ and
// everything
// before it
jarPath = jarPath.replaceAll("jar!.*", "jar"); // removes everything
// after .jar, if .jar
// exists in dirtyPath
jarPath = jarPath.replaceAll("%20", " "); // necessary if path has
// spaces within
if (!jarPath.endsWith(".jar")) { // this is needed if you plan to run
// the app using Spring Tools Suit play
// button.
jarPath = jarPath.replaceAll("/classes/.*", "/classes/");
}
return Paths.get(jarPath).getParent().toString(); // Paths - from java 8
}
private Map<String, Class<?>> loadExtensionClasses() {
final SPI defaultAnnotation = type.getAnnotation(SPI.class);
if (defaultAnnotation != null) {
String value = defaultAnnotation.value();
if ((value = value.trim()).length() > 0) {
String[] names = NAME_SEPARATOR.split(value);
if (names.length > 1) {
throw new IllegalStateException("more than 1 default extension name on extension " + type.getName()
+ ": " + Arrays.toString(names));
}
if (names.length == 1) cachedDefaultName = names[0];
}
}
Map<String, Class<?>> extensionClasses = new HashMap<String, Class<?>>();
// 1. lib folder,customized extension classLoader (jar_dir/lib)
String dir = File.separator + this.getJarDirectoryPath() + File.separator + "lib";
logger.info("extension classpath dir: " + dir);
File externalLibDir = new File(dir);
if (!externalLibDir.exists()) {
externalLibDir = new File(
File.separator + this.getJarDirectoryPath() + File.separator + "canal_client" + File.separator + "lib");
}
if (externalLibDir.exists()) {
File[] files = externalLibDir.listFiles(new FilenameFilter() {
@Override
public boolean accept(File dir, String name) {
return name.endsWith(".jar");
}
});
if (files != null) {
for (File f : files) {
URL url = null;
try {
url = f.toURI().toURL();
} catch (MalformedURLException e) {
throw new RuntimeException("load extension jar failed!", e);
}
ClassLoader parent = Thread.currentThread().getContextClassLoader();
URLClassLoader localClassLoader;
if (classLoaderPolicy == null || "".equals(classLoaderPolicy)
|| DEFAULT_CLASSLOADER_POLICY.equalsIgnoreCase(classLoaderPolicy)) {
localClassLoader = new URLClassLoader(new URL[] { url }, parent) {
@Override
public Class<?> loadClass(String name) throws ClassNotFoundException {
Class<?> c = findLoadedClass(name);
if (c != null) {
return c;
}
if (name.startsWith("java.") || name.startsWith("org.slf4j.")
|| name.startsWith("org.apache.logging")
|| name.startsWith("org.apache.commons.logging.")) {
// || name.startsWith("org.apache.hadoop."))
// {
c = super.loadClass(name);
}
if (c != null) return c;
try {
// 先加载jar内的class,可避免jar冲突
c = findClass(name);
} catch (ClassNotFoundException e) {
c = null;
}
if (c != null) {
return c;
}
return super.loadClass(name);
}
@Override
public Enumeration<URL> getResources(String name) throws IOException {
@SuppressWarnings("unchecked")
Enumeration<URL>[] tmp = (Enumeration<URL>[]) new Enumeration<?>[2];
tmp[0] = findResources(name); // local class
// path first
// tmp[1] = super.getResources(name);
return new CompoundEnumeration<>(tmp);
}
};
} else {
localClassLoader = new URLClassLoader(new URL[] { url }, parent);
}
loadFile(extensionClasses, CANAL_DIRECTORY, localClassLoader);
loadFile(extensionClasses, SERVICES_DIRECTORY, localClassLoader);
}
}
}
// 2. load inner extension class with default classLoader
ClassLoader classLoader = findClassLoader();
loadFile(extensionClasses, CANAL_DIRECTORY, classLoader);
loadFile(extensionClasses, SERVICES_DIRECTORY, classLoader);
return extensionClasses;
}
public static class CompoundEnumeration<E> implements Enumeration<E> {
private Enumeration<E>[] enums;
private int index = 0;
public CompoundEnumeration(Enumeration<E>[] enums){
this.enums = enums;
}
private boolean next() {
while (this.index < this.enums.length) {
if (this.enums[this.index] != null && this.enums[this.index].hasMoreElements()) {
return true;
}
++this.index;
}
return false;
}
public boolean hasMoreElements() {
return this.next();
}
public E nextElement() {
if (!this.next()) {
throw new NoSuchElementException();
} else {
return this.enums[this.index].nextElement();
}
}
}
private void loadFile(Map<String, Class<?>> extensionClasses, String dir, ClassLoader classLoader) {
String fileName = dir + type.getName();
try {
Enumeration<URL> urls;
if (classLoader != null) {
urls = classLoader.getResources(fileName);
} else {
urls = ClassLoader.getSystemResources(fileName);
}
if (urls != null) {
while (urls.hasMoreElements()) {
URL url = urls.nextElement();
try {
BufferedReader reader = null;
try {
reader = new BufferedReader(new InputStreamReader(url.openStream(), "utf-8"));
String line = null;
while ((line = reader.readLine()) != null) {
final int ci = line.indexOf('#');
if (ci >= 0) line = line.substring(0, ci);
line = line.trim();
if (line.length() > 0) {
try {
String name = null;
int i = line.indexOf('=');
if (i > 0) {
name = line.substring(0, i).trim();
line = line.substring(i + 1).trim();
}
if (line.length() > 0) {
Class<?> clazz = classLoader.loadClass(line);
// Class<?> clazz =
// Class.forName(line, true,
// classLoader);
if (!type.isAssignableFrom(clazz)) {
throw new IllegalStateException(
"Error when load extension class(interface: " + type
+ ", class line: " + clazz.getName()
+ "), class " + clazz.getName()
+ "is not subtype of interface.");
} else {
try {
clazz.getConstructor(type);
} catch (NoSuchMethodException e) {
clazz.getConstructor();
String[] names = NAME_SEPARATOR.split(name);
if (names != null && names.length > 0) {
for (String n : names) {
if (!cachedNames.containsKey(clazz)) {
cachedNames.put(clazz, n);
}
Class<?> c = extensionClasses.get(n);
if (c == null) {
extensionClasses.put(n, clazz);
} else if (c != clazz) {
cachedNames.remove(clazz);
throw new IllegalStateException(
"Duplicate extension " + type.getName() + " name "
+ n + " on "
+ c.getName() + " and "
+ clazz.getName());
}
}
}
}
}
}
} catch (Throwable t) {
IllegalStateException e = new IllegalStateException(
"Failed to load extension class(interface: " + type + ", class line: "
+ line + ") in " + url
+ ", cause: "
+ t.getMessage(),
t);
exceptions.put(line, e);
}
}
} // end of while read lines
} finally {
if (reader != null) {
reader.close();
}
}
} catch (Throwable t) {
logger.error("Exception when load extension class(interface: " + type + ", class file: " + url
+ ") in " + url,
t);
}
} // end of while urls
}
} catch (Throwable t) {
logger.error(
"Exception when load extension class(interface: " + type + ", description file: " + fileName + ").",
t);
}
}
private static ClassLoader findClassLoader() {
return ExtensionLoader.class.getClassLoader();
}
@Override
public String toString() {
return this.getClass().getName() + "[" + type.getName() + "]";
}
private static class Holder<T> {
private volatile T value;
private void set(T value) {
this.value = value;
}
private T get() {
return value;
}
}
}

View File

@@ -1,149 +0,0 @@
package com.alibaba.otter.canal.client.adapter.support;
import java.math.BigDecimal;
import java.math.BigInteger;
import java.sql.*;
import org.joda.time.DateTime;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
/**
* 类型转换工具类
*
* @author rewerma 2018-8-19 下午06:14:23
* @version 1.0.0
*/
public class JdbcTypeUtil {
private static Logger logger = LoggerFactory.getLogger(JdbcTypeUtil.class);
public static Object getRSData(ResultSet rs, String columnName, int jdbcType) throws SQLException {
if (jdbcType == Types.BIT || jdbcType == Types.BOOLEAN) {
return rs.getByte(columnName);
} else {
return rs.getObject(columnName);
}
}
public static Class<?> jdbcType2javaType(int jdbcType) {
switch (jdbcType) {
case Types.BIT:
case Types.BOOLEAN:
// return Boolean.class;
case Types.TINYINT:
return Byte.TYPE;
case Types.SMALLINT:
return Short.class;
case Types.INTEGER:
return Integer.class;
case Types.BIGINT:
return Long.class;
case Types.DECIMAL:
case Types.NUMERIC:
return BigDecimal.class;
case Types.REAL:
return Float.class;
case Types.FLOAT:
case Types.DOUBLE:
return Double.class;
case Types.CHAR:
case Types.VARCHAR:
case Types.LONGVARCHAR:
return String.class;
case Types.BINARY:
case Types.VARBINARY:
case Types.LONGVARBINARY:
case Types.BLOB:
return byte[].class;
case Types.DATE:
return java.sql.Date.class;
case Types.TIME:
return Time.class;
case Types.TIMESTAMP:
return Timestamp.class;
default:
return String.class;
}
}
public static Object typeConvert(String columnName, String value, int sqlType, String mysqlType) {
if (value == null || value.equals("")) {
return null;
}
try {
Object res;
switch (sqlType) {
case Types.INTEGER:
res = Integer.parseInt(value);
break;
case Types.SMALLINT:
res = Short.parseShort(value);
break;
case Types.BIT:
case Types.TINYINT:
res = Byte.parseByte(value);
break;
case Types.BIGINT:
if (mysqlType.startsWith("bigint") && mysqlType.endsWith("unsigned")) {
res = new BigInteger(value);
} else {
res = Long.parseLong(value);
}
break;
// case Types.BIT:
case Types.BOOLEAN:
res = !"0".equals(value);
break;
case Types.DOUBLE:
case Types.FLOAT:
res = Double.parseDouble(value);
break;
case Types.REAL:
res = Float.parseFloat(value);
break;
case Types.DECIMAL:
case Types.NUMERIC:
res = new BigDecimal(value);
break;
case Types.BINARY:
case Types.VARBINARY:
case Types.LONGVARBINARY:
case Types.BLOB:
res = value.getBytes("ISO-8859-1");
break;
case Types.DATE:
if (!value.startsWith("0000-00-00")) {
value = value.trim().replace(" ", "T");
DateTime dt = new DateTime(value);
res = new Date(dt.toDate().getTime());
} else {
res = null;
}
break;
case Types.TIME:
value = "T" + value;
DateTime dt = new DateTime(value);
res = new Time(dt.toDate().getTime());
break;
case Types.TIMESTAMP:
if (!value.startsWith("0000-00-00")) {
value = value.trim().replace(" ", "T");
dt = new DateTime(value);
res = new Timestamp(dt.toDate().getTime());
} else {
res = null;
}
break;
case Types.CLOB:
default:
res = value;
}
return res;
} catch (Exception e) {
logger.error("table: {} column: {}, failed convert type {} to {}", columnName, value, sqlType);
return value;
}
}
}

View File

@@ -1,173 +0,0 @@
package com.alibaba.otter.canal.client.adapter.support;
import java.util.ArrayList;
import java.util.HashSet;
import java.util.LinkedHashMap;
import java.util.List;
import java.util.Map;
import java.util.Set;
import com.alibaba.otter.canal.protocol.CanalEntry;
import com.alibaba.otter.canal.protocol.FlatMessage;
import com.alibaba.otter.canal.protocol.Message;
/**
* Message对象解析工具类
*
* @author rewerma 2018-8-19 下午06:14:23
* @version 1.0.0
*/
public class MessageUtil {
public static void parse4Dml(String destination, Message message, Consumer<Dml> consumer) {
if (message == null) {
return;
}
List<CanalEntry.Entry> entries = message.getEntries();
for (CanalEntry.Entry entry : entries) {
if (entry.getEntryType() == CanalEntry.EntryType.TRANSACTIONBEGIN
|| entry.getEntryType() == CanalEntry.EntryType.TRANSACTIONEND) {
continue;
}
CanalEntry.RowChange rowChange;
try {
rowChange = CanalEntry.RowChange.parseFrom(entry.getStoreValue());
} catch (Exception e) {
throw new RuntimeException("ERROR ## parser of eromanga-event has an error , data:" + entry.toString(),
e);
}
CanalEntry.EventType eventType = rowChange.getEventType();
final Dml dml = new Dml();
dml.setDestination(destination);
dml.setDatabase(entry.getHeader().getSchemaName());
dml.setTable(entry.getHeader().getTableName());
dml.setType(eventType.toString());
dml.setEs(entry.getHeader().getExecuteTime());
dml.setTs(System.currentTimeMillis());
dml.setSql(rowChange.getSql());
List<Map<String, Object>> data = new ArrayList<>();
List<Map<String, Object>> old = new ArrayList<>();
if (!rowChange.getIsDdl()) {
Set<String> updateSet = new HashSet<>();
for (CanalEntry.RowData rowData : rowChange.getRowDatasList()) {
if (eventType != CanalEntry.EventType.INSERT && eventType != CanalEntry.EventType.UPDATE
&& eventType != CanalEntry.EventType.DELETE) {
continue;
}
Map<String, Object> row = new LinkedHashMap<>();
List<CanalEntry.Column> columns;
if (eventType == CanalEntry.EventType.DELETE) {
columns = rowData.getBeforeColumnsList();
} else {
columns = rowData.getAfterColumnsList();
}
for (CanalEntry.Column column : columns) {
row.put(column.getName(),
JdbcTypeUtil.typeConvert(column.getName(),
column.getValue(),
column.getSqlType(),
column.getMysqlType()));
// 获取update为true的字段
if (column.getUpdated()) {
updateSet.add(column.getName());
}
}
if (!row.isEmpty()) {
data.add(row);
}
if (eventType == CanalEntry.EventType.UPDATE) {
Map<String, Object> rowOld = new LinkedHashMap<>();
for (CanalEntry.Column column : rowData.getBeforeColumnsList()) {
if (updateSet.contains(column.getName())) {
rowOld.put(column.getName(),
JdbcTypeUtil.typeConvert(column.getName(),
column.getValue(),
column.getSqlType(),
column.getMysqlType()));
}
}
// update操作将记录修改前的值
if (!rowOld.isEmpty()) {
old.add(rowOld);
}
}
}
if (!data.isEmpty()) {
dml.setData(data);
}
if (!old.isEmpty()) {
dml.setOld(old);
}
}
consumer.accept(dml);
}
}
public static Dml flatMessage2Dml(String destination, FlatMessage flatMessage) {
if (flatMessage == null) {
return null;
}
Dml dml = new Dml();
dml.setDestination(destination);
dml.setDatabase(flatMessage.getDatabase());
dml.setTable(flatMessage.getTable());
dml.setType(flatMessage.getType());
dml.setTs(flatMessage.getTs());
dml.setEs(flatMessage.getEs());
dml.setSql(flatMessage.getSql());
if (flatMessage.getSqlType() == null || flatMessage.getMysqlType() == null) {
throw new RuntimeException("SqlType or mysqlType is null");
}
List<Map<String, String>> data = flatMessage.getData();
if (data != null) {
dml.setData(changeRows(data, flatMessage.getSqlType(), flatMessage.getMysqlType()));
}
List<Map<String, String>> old = flatMessage.getOld();
if (old != null) {
dml.setOld(changeRows(old, flatMessage.getSqlType(), flatMessage.getMysqlType()));
}
return dml;
}
private static List<Map<String, Object>> changeRows(List<Map<String, String>> rows, Map<String, Integer> sqlTypes,
Map<String, String> mysqlTypes) {
List<Map<String, Object>> result = new ArrayList<>();
for (Map<String, String> row : rows) {
Map<String, Object> resultRow = new LinkedHashMap<>();
for (Map.Entry<String, String> entry : row.entrySet()) {
String columnName = entry.getKey();
String columnValue = entry.getValue();
Integer sqlType = sqlTypes.get(columnName);
if (sqlType == null) {
continue;
}
String mysqlType = mysqlTypes.get(columnName);
if (mysqlType == null) {
continue;
}
Object finalValue = JdbcTypeUtil.typeConvert(columnName, columnValue, sqlType, mysqlType);
resultRow.put(columnName, finalValue);
}
result.add(resultRow);
}
return result;
}
public interface Consumer<T> {
void accept(T t);
}
}

View File

@@ -1,52 +0,0 @@
package com.alibaba.otter.canal.client.adapter.support;
import java.util.Map;
/**
* 外部适配器配置信息类
*
* @author rewerma 2018-8-18 下午10:15:12
* @version 1.0.0
*/
public class OuterAdapterConfig {
private String name; // 适配器名称, 如: logger, hbase, es
private String hosts; // 适配器内部的地址, 比如对应es该参数可以填写es的server地址
private String zkHosts; // 适配器内部的ZK地址, 比如对应HBase该参数可以填写HBase对应的ZK地址
private Map<String, String> properties; // 其余参数, 可填写适配器中的所需的配置信息
public String getName() {
return name;
}
public void setName(String name) {
this.name = name;
}
public String getHosts() {
return hosts;
}
public void setHosts(String hosts) {
this.hosts = hosts;
}
public Map<String, String> getProperties() {
return properties;
}
public void setProperties(Map<String, String> properties) {
this.properties = properties;
}
public String getZkHosts() {
return zkHosts;
}
public void setZkHosts(String zkHosts) {
this.zkHosts = zkHosts;
}
}

View File

@@ -1,57 +0,0 @@
package com.alibaba.otter.canal.client.adapter.support;
import java.io.Serializable;
import java.util.Date;
/**
* 用于rest的结果返回对象
*
* @author rewerma @ 2018-10-20
* @version 1.0.0
*/
public class Result implements Serializable {
private static final long serialVersionUID = -3276409502352405716L;
public Integer code = 20000;
public Object data;
public String message;
public Date sysTime;
public static Result createSuccess(String message) {
Result result = new Result();
result.setMessage(message);
return result;
}
public Integer getCode() {
return code;
}
public void setCode(Integer code) {
this.code = code;
}
public Object getData() {
return data;
}
public void setData(Object data) {
this.data = data;
}
public String getMessage() {
return message;
}
public void setMessage(String message) {
this.message = message;
}
public Date getSysTime() {
return sysTime;
}
public void setSysTime(Date sysTime) {
this.sysTime = sysTime;
}
}

View File

@@ -1,22 +0,0 @@
package com.alibaba.otter.canal.client.adapter.support;
import java.lang.annotation.Documented;
import java.lang.annotation.ElementType;
import java.lang.annotation.Retention;
import java.lang.annotation.RetentionPolicy;
import java.lang.annotation.Target;
/**
* SPI装载器注解
*
* @author rewerma @ 2018-10-20
* @version 1.0.0
*/
@Documented
@Retention(RetentionPolicy.RUNTIME)
@Target({ ElementType.TYPE })
public @interface SPI {
// Default SPI name
String value() default "";
}

View File

@@ -1,68 +0,0 @@
<?xml version="1.0" encoding="UTF-8"?>
<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
<parent>
<artifactId>canal.client-adapter</artifactId>
<groupId>com.alibaba.otter</groupId>
<version>1.1.1</version>
</parent>
<modelVersion>4.0.0</modelVersion>
<groupId>com.alibaba.otter</groupId>
<artifactId>client-adapter.hbase</artifactId>
<packaging>jar</packaging>
<name>canal client adapter hbase module for otter ${project.version}</name>
<dependencies>
<dependency>
<groupId>com.alibaba.otter</groupId>
<artifactId>client-adapter.common</artifactId>
<version>${project.version}</version>
<scope>provided</scope>
</dependency>
<dependency>
<groupId>org.yaml</groupId>
<artifactId>snakeyaml</artifactId>
<version>1.19</version>
<scope>provided</scope>
</dependency>
<dependency>
<groupId>org.apache.hbase</groupId>
<artifactId>hbase-client</artifactId>
<version>1.1.2</version>
<exclusions>
<exclusion>
<groupId>org.slf4j</groupId>
<artifactId>slf4j-log4j12</artifactId>
</exclusion>
</exclusions>
</dependency>
<dependency>
<groupId>com.google.guava</groupId>
<artifactId>guava</artifactId>
<version>12.0.1</version>
</dependency>
</dependencies>
<build>
<plugins>
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-assembly-plugin</artifactId>
<version>2.4</version>
<configuration>
<descriptorRefs>
<descriptorRef>jar-with-dependencies</descriptorRef>
</descriptorRefs>
</configuration>
<executions>
<execution>
<id>make-assembly</id>
<phase>package</phase>
<goals>
<goal>single</goal>
</goals>
</execution>
</executions>
</plugin>
</plugins>
</build>
</project>

View File

@@ -1,177 +0,0 @@
package com.alibaba.otter.canal.client.adapter.hbase;
import java.io.IOException;
import java.util.HashMap;
import java.util.LinkedHashMap;
import java.util.List;
import java.util.Map;
import javax.sql.DataSource;
import org.apache.commons.lang.StringUtils;
import org.apache.hadoop.conf.Configuration;
import org.apache.hadoop.hbase.HBaseConfiguration;
import org.apache.hadoop.hbase.TableName;
import org.apache.hadoop.hbase.client.*;
import org.apache.hadoop.hbase.client.Result;
import org.apache.hadoop.hbase.filter.FirstKeyOnlyFilter;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
import com.alibaba.otter.canal.client.adapter.OuterAdapter;
import com.alibaba.otter.canal.client.adapter.hbase.config.MappingConfig;
import com.alibaba.otter.canal.client.adapter.hbase.config.MappingConfigLoader;
import com.alibaba.otter.canal.client.adapter.hbase.service.HbaseEtlService;
import com.alibaba.otter.canal.client.adapter.hbase.service.HbaseSyncService;
import com.alibaba.otter.canal.client.adapter.hbase.support.HbaseTemplate;
import com.alibaba.otter.canal.client.adapter.support.*;
/**
* HBase外部适配器
*
* @author machengyuan 2018-8-21 下午8:45:38
* @version 1.0.0
*/
@SPI("hbase")
public class HbaseAdapter implements OuterAdapter {
private static Logger logger = LoggerFactory.getLogger(HbaseAdapter.class);
private static volatile Map<String, MappingConfig> hbaseMapping = null; // 文件名对应配置
private static volatile Map<String, MappingConfig> mappingConfigCache = null; // 库名-表名对应配置
private Connection conn;
private HbaseSyncService hbaseSyncService;
private HbaseTemplate hbaseTemplate;
@Override
public void init(OuterAdapterConfig configuration) {
try {
if (mappingConfigCache == null) {
synchronized (MappingConfig.class) {
if (mappingConfigCache == null) {
hbaseMapping = MappingConfigLoader.load();
mappingConfigCache = new HashMap<>();
for (MappingConfig mappingConfig : hbaseMapping.values()) {
mappingConfigCache.put(StringUtils.trimToEmpty(mappingConfig.getHbaseMapping().getDestination())
+ "." + mappingConfig.getHbaseMapping().getDatabase() + "."
+ mappingConfig.getHbaseMapping().getTable(),
mappingConfig);
}
}
}
}
Map<String, String> propertites = configuration.getProperties();
Configuration hbaseConfig = HBaseConfiguration.create();
propertites.forEach(hbaseConfig::set);
conn = ConnectionFactory.createConnection(hbaseConfig);
hbaseTemplate = new HbaseTemplate(conn);
hbaseSyncService = new HbaseSyncService(hbaseTemplate);
} catch (Exception e) {
throw new RuntimeException(e);
}
}
@Override
public void sync(Dml dml) {
if (dml == null) {
return;
}
String destination = StringUtils.trimToEmpty(dml.getDestination());
String database = dml.getDatabase();
String table = dml.getTable();
MappingConfig config = mappingConfigCache.get(destination + "." + database + "." + table);
hbaseSyncService.sync(config, dml);
}
@Override
public EtlResult etl(String task, List<String> params) {
EtlResult etlResult = new EtlResult();
MappingConfig config = hbaseMapping.get(task);
if (config != null) {
DataSource dataSource = DatasourceConfig.DATA_SOURCES.get(config.getDataSourceKey());
if (dataSource != null) {
return HbaseEtlService.importData(dataSource, hbaseTemplate, config, params);
} else {
etlResult.setSucceeded(false);
etlResult.setErrorMessage("DataSource not found");
return etlResult;
}
} else {
DataSource dataSource = DatasourceConfig.DATA_SOURCES.get(task);
if (dataSource != null) {
StringBuilder resultMsg = new StringBuilder();
boolean resSucc = true;
// ds不为空说明传入的是datasourceKey
for (MappingConfig configTmp : hbaseMapping.values()) {
// 取所有的datasourceKey为task的配置
if (configTmp.getDataSourceKey().equals(task)) {
EtlResult etlRes = HbaseEtlService.importData(dataSource, hbaseTemplate, configTmp, params);
if (!etlRes.getSucceeded()) {
resSucc = false;
resultMsg.append(etlRes.getErrorMessage()).append("\n");
} else {
resultMsg.append(etlRes.getResultMessage()).append("\n");
}
}
}
if (resultMsg.length() > 0) {
etlResult.setSucceeded(resSucc);
if (resSucc) {
etlResult.setResultMessage(resultMsg.toString());
} else {
etlResult.setErrorMessage(resultMsg.toString());
}
return etlResult;
}
}
}
etlResult.setSucceeded(false);
etlResult.setErrorMessage("Task not found");
return etlResult;
}
@Override
public Map<String, Object> count(String task) {
MappingConfig config = hbaseMapping.get(task);
String hbaseTable = config.getHbaseMapping().getHbaseTable();
long rowCount = 0L;
try {
HTable table = (HTable) conn.getTable(TableName.valueOf(hbaseTable));
Scan scan = new Scan();
scan.setFilter(new FirstKeyOnlyFilter());
ResultScanner resultScanner = table.getScanner(scan);
for (Result result : resultScanner) {
rowCount += result.size();
}
} catch (IOException e) {
logger.error(e.getMessage(), e);
}
Map<String, Object> res = new LinkedHashMap<>();
res.put("hbaseTable", hbaseTable);
res.put("count", rowCount);
return res;
}
@Override
public void destroy() {
if (conn != null) {
try {
conn.close();
} catch (IOException e) {
throw new RuntimeException(e);
}
}
}
@Override
public String getDestination(String task) {
MappingConfig config = hbaseMapping.get(task);
if (config != null && config.getHbaseMapping() != null) {
return config.getHbaseMapping().getDestination();
}
return null;
}
}

View File

@@ -1,365 +0,0 @@
package com.alibaba.otter.canal.client.adapter.hbase.config;
import java.util.*;
/**
* HBase表映射配置
*
* @author rewerma 2018-8-21 下午06:45:49
* @version 1.0.0
*/
public class MappingConfig {
private String dataSourceKey; // 数据源key
private HbaseMapping hbaseMapping; // hbase映射配置
public String getDataSourceKey() {
return dataSourceKey;
}
public void setDataSourceKey(String dataSourceKey) {
this.dataSourceKey = dataSourceKey;
}
public HbaseMapping getHbaseMapping() {
return hbaseMapping;
}
public void setHbaseMapping(HbaseMapping hbaseMapping) {
this.hbaseMapping = hbaseMapping;
}
public void validate() {
if (hbaseMapping.database == null || hbaseMapping.database.isEmpty()) {
throw new NullPointerException("hbaseMapping.database");
}
if (hbaseMapping.table == null || hbaseMapping.table.isEmpty()) {
throw new NullPointerException("hbaseMapping.table");
}
if (hbaseMapping.hbaseTable == null || hbaseMapping.hbaseTable.isEmpty()) {
throw new NullPointerException("hbaseMapping.hbaseTable");
}
if (hbaseMapping.mode == null) {
throw new NullPointerException("hbaseMapping.mode");
}
if (hbaseMapping.rowKey != null && hbaseMapping.rowKeyColumn != null) {
throw new RuntimeException("已配置了复合主键作为RowKey,无需再指定RowKey列");
}
}
@Override
public boolean equals(Object o) {
if (this == o) return true;
if (o == null || getClass() != o.getClass()) return false;
MappingConfig config = (MappingConfig) o;
return hbaseMapping != null ? hbaseMapping.equals(config.hbaseMapping) : config.hbaseMapping == null;
}
@Override
public int hashCode() {
return hbaseMapping != null ? hbaseMapping.hashCode() : 0;
}
public static class ColumnItem {
private boolean isRowKey = false;
private String column;
private String family;
private String qualifier;
private String type;
public boolean isRowKey() {
return isRowKey;
}
public void setRowKey(boolean rowKey) {
isRowKey = rowKey;
}
public String getColumn() {
return column;
}
public void setColumn(String column) {
this.column = column;
}
public String getFamily() {
return family;
}
public void setFamily(String family) {
this.family = family;
}
public String getQualifier() {
return qualifier;
}
public void setQualifier(String qualifier) {
this.qualifier = qualifier;
}
public String getType() {
return type;
}
public void setType(String type) {
this.type = type;
}
@Override
public boolean equals(Object o) {
if (this == o) return true;
if (o == null || getClass() != o.getClass()) return false;
ColumnItem that = (ColumnItem) o;
return Objects.equals(column, that.column);
}
@Override
public int hashCode() {
return Objects.hash(column);
}
}
public enum Mode {
STRING("STRING"), NATIVE("NATIVE"), PHOENIX("PHOENIX");
private String type;
public String getType() {
return type;
}
Mode(String type){
this.type = type;
}
}
public static class HbaseMapping {
private Mode mode = Mode.STRING; // hbase默认转换格式
private String destination; // canal实例或MQ的topic
private String database; // 数据库名或schema名
private String table; // 表面名
private String hbaseTable; // hbase表名
private String family = "CF"; // 默认统一column family
private boolean uppercaseQualifier = true; // 是否转大写
private boolean autoCreateTable = false; // 同步时HBase中表不存在的情况下自动建表
private String rowKey; // 指定复合主键为rowKey
private Map<String, String> columns; // 字段映射
private List<String> excludeColumns; // 不映射的字段
private ColumnItem rowKeyColumn; // rowKey字段
private String etlCondition; // etl条件sql
private Map<String, ColumnItem> columnItems = new LinkedHashMap<>(); // 转换后的字段映射列表
private Set<String> families = new LinkedHashSet<>(); // column family列表
private int readBatch = 5000;
private int commitBatch = 5000; // etl等批量提交大小
public Mode getMode() {
return mode;
}
public void setMode(Mode mode) {
this.mode = mode;
}
public String getDestination() {
return destination;
}
public void setDestination(String destination) {
this.destination = destination;
}
public String getDatabase() {
return database;
}
public void setDatabase(String database) {
this.database = database;
}
public String getTable() {
return table;
}
public void setTable(String table) {
this.table = table;
}
public String getHbaseTable() {
return hbaseTable;
}
public void setHbaseTable(String hbaseTable) {
this.hbaseTable = hbaseTable;
}
public Map<String, String> getColumns() {
return columns;
}
public boolean isAutoCreateTable() {
return autoCreateTable;
}
public void setAutoCreateTable(boolean autoCreateTable) {
this.autoCreateTable = autoCreateTable;
}
public int getReadBatch() {
return readBatch;
}
public void setReadBatch(int readBatch) {
this.readBatch = readBatch;
}
public int getCommitBatch() {
return commitBatch;
}
public void setCommitBatch(int commitBatch) {
this.commitBatch = commitBatch;
}
public String getRowKey() {
return rowKey;
}
public void setRowKey(String rowKey) {
this.rowKey = rowKey;
}
public String getEtlCondition() {
return etlCondition;
}
public void setEtlCondition(String etlCondition) {
this.etlCondition = etlCondition;
}
public void setColumns(Map<String, String> columns) {
this.columns = columns;
if (columns != null) {
for (Map.Entry<String, String> columnField : columns.entrySet()) {
String field = columnField.getValue();
String type = null;
if (field != null) {
// 解析类型
int i = field.indexOf("$");
if (i > -1) {
type = field.substring(i + 1);
field = field.substring(0, i);
}
}
ColumnItem columnItem = new ColumnItem();
columnItem.setColumn(columnField.getKey());
columnItem.setType(type);
if ("rowKey".equalsIgnoreCase(field)) {
columnItem.setRowKey(true);
rowKeyColumn = columnItem;
} else {
if (field == null || field.equals("")) {
columnItem.setFamily(family);
columnItem.setQualifier(columnField.getKey());
} else {
int len = field.indexOf(":");
if (len > -1) {
columnItem.setFamily(field.substring(0, len));
columnItem.setQualifier(field.substring(len + 1));
} else {
columnItem.setFamily(family);
columnItem.setQualifier(field);
}
}
if (uppercaseQualifier) {
columnItem.setQualifier(columnItem.getQualifier().toUpperCase());
}
families.add(columnItem.getFamily());
}
columnItems.put(columnField.getKey(), columnItem);
}
} else {
this.columns = new LinkedHashMap<>();
}
}
public List<String> getExcludeColumns() {
return excludeColumns;
}
public void setExcludeColumns(List<String> excludeColumns) {
this.excludeColumns = excludeColumns;
}
public String getFamily() {
return family;
}
public void setFamily(String family) {
this.family = family;
if (family == null) {
this.family = "CF";
}
}
public boolean isUppercaseQualifier() {
return uppercaseQualifier;
}
public void setUppercaseQualifier(boolean uppercaseQualifier) {
this.uppercaseQualifier = uppercaseQualifier;
}
public ColumnItem getRowKeyColumn() {
return rowKeyColumn;
}
public void setRowKeyColumn(ColumnItem rowKeyColumn) {
this.rowKeyColumn = rowKeyColumn;
}
public Map<String, ColumnItem> getColumnItems() {
return columnItems;
}
public void setColumnItems(Map<String, ColumnItem> columnItems) {
this.columnItems = columnItems;
}
public Set<String> getFamilies() {
return families;
}
public void setFamilies(Set<String> families) {
this.families = families;
}
@Override
public boolean equals(Object o) {
if (this == o) return true;
if (o == null || getClass() != o.getClass()) return false;
HbaseMapping hbaseMapping = (HbaseMapping) o;
if (table != null ? !table.equals(hbaseMapping.table) : hbaseMapping.table != null) return false;
return hbaseTable != null ? hbaseTable.equals(hbaseMapping.hbaseTable) : hbaseMapping.hbaseTable == null;
}
@Override
public int hashCode() {
int result = table != null ? table.hashCode() : 0;
result = 31 * result + (hbaseTable != null ? hbaseTable.hashCode() : 0);
return result;
}
}
}

View File

@@ -1,149 +0,0 @@
package com.alibaba.otter.canal.client.adapter.hbase.config;
import java.io.File;
import java.io.FileInputStream;
import java.io.IOException;
import java.io.InputStream;
import java.util.Collection;
import java.util.LinkedHashMap;
import java.util.Map;
import org.apache.commons.lang.StringUtils;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
import org.yaml.snakeyaml.Yaml;
import com.alibaba.otter.canal.client.adapter.support.AdapterConfigs;
/**
* HBase表映射配置加载器
*
* @author rewerma 2018-8-21 下午06:45:49
* @version 1.0.0
*/
public class MappingConfigLoader {
private static Logger logger = LoggerFactory.getLogger(MappingConfigLoader.class);
private static final String BASE_PATH = "hbase";
/**
* 加载HBase表映射配置
*
* @return 配置名/配置文件名--对象
*/
public static Map<String, MappingConfig> load() {
logger.info("## Start loading mapping config ... ");
Map<String, MappingConfig> result = new LinkedHashMap<>();
Collection<String> configs = AdapterConfigs.get("hbase");
for (String c : configs) {
if (c == null) {
continue;
}
c = c.trim();
if (c.equals("") || c.startsWith("#")) {
continue;
}
MappingConfig config;
String configContent = null;
if (c.endsWith(".yml")) {
configContent = readConfigContent(BASE_PATH + "/" + c);
}
// 简单配置database.table@datasourcekey?rowKey=key1,key2
if (StringUtils.isEmpty(configContent)) {
String[] mapping = c.split("\\?");
String params = mapping.length == 2 ? mapping[1] : null;
String rowKey = null;
String srcMeta = mapping[0];
//
if (params != null) {
for (String entry : params.split("&")) {
if ("rowKey".equals(entry.split("=")[0])) {
rowKey = entry.split("=")[1];
}
}
}
String dsKey = srcMeta.split("@").length == 2 ? srcMeta.split("@")[1] : null;
String[] dbTable;
if (dsKey == null) {
dbTable = srcMeta.split("\\.");
} else {
dbTable = srcMeta.split("@")[0].split("\\.");
}
if (dbTable.length == 2) {
config = new MappingConfig();
MappingConfig.HbaseMapping hbaseMapping = new MappingConfig.HbaseMapping();
hbaseMapping.setHbaseTable(dbTable[0].toUpperCase() + "." + dbTable[1].toUpperCase());
hbaseMapping.setAutoCreateTable(true);
hbaseMapping.setDatabase(dbTable[0]);
hbaseMapping.setTable(dbTable[1]);
hbaseMapping.setMode(MappingConfig.Mode.PHOENIX);
hbaseMapping.setRowKey(rowKey);
// 有定义rowKey
if (rowKey != null) {
MappingConfig.ColumnItem columnItem = new MappingConfig.ColumnItem();
columnItem.setRowKey(true);
columnItem.setColumn(rowKey);
hbaseMapping.setRowKeyColumn(columnItem);
}
config.setHbaseMapping(hbaseMapping);
config.setDataSourceKey(dsKey);
} else {
throw new RuntimeException(String.format("配置项[%s]内容为空, 或格式不符合database.table", c));
}
} else { // 配置文件配置
config = new Yaml().loadAs(configContent, MappingConfig.class);
}
try {
config.validate();
} catch (Exception e) {
throw new RuntimeException("ERROR Config: " + c + " " + e.getMessage(), e);
}
result.put(c, config);
}
logger.info("## Mapping config loaded");
return result;
}
public static String readConfigContent(String config) {
InputStream in = null;
try {
// 先取本地文件,再取类路径
File configFile = new File("config/" + config);
if (configFile.exists()) {
in = new FileInputStream(configFile);
} else {
in = MappingConfigLoader.class.getClassLoader().getResourceAsStream(config);
}
if (in == null) {
throw new RuntimeException("Config file not found.");
}
byte[] bytes = new byte[in.available()];
in.read(bytes);
return new String(bytes, "UTF-8");
} catch (IOException e) {
throw new RuntimeException("Read ds-config.yml or hbase-mappings.conf error. ", e);
} finally {
try {
if (in != null) {
in.close();
}
} catch (IOException e) {
// ignore
}
}
}
}

View File

@@ -1,385 +0,0 @@
package com.alibaba.otter.canal.client.adapter.hbase.service;
import java.sql.*;
import java.text.SimpleDateFormat;
import java.util.ArrayList;
import java.util.Date;
import java.util.List;
import java.util.concurrent.ExecutorService;
import java.util.concurrent.Executors;
import java.util.concurrent.Future;
import java.util.concurrent.atomic.AtomicBoolean;
import java.util.concurrent.atomic.AtomicLong;
import java.util.function.Function;
import javax.sql.DataSource;
import org.apache.hadoop.hbase.util.Bytes;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
import com.alibaba.otter.canal.client.adapter.hbase.config.MappingConfig;
import com.alibaba.otter.canal.client.adapter.hbase.support.*;
import com.alibaba.otter.canal.client.adapter.support.EtlResult;
import com.alibaba.otter.canal.client.adapter.support.JdbcTypeUtil;
import com.google.common.base.Joiner;
/**
* HBase ETL 操作业务类
*
* @author rewerma @ 2018-10-20
* @version 1.0.0
*/
public class HbaseEtlService {
private static Logger logger = LoggerFactory.getLogger(HbaseEtlService.class);
public static Object sqlRS(DataSource ds, String sql, Function<ResultSet, Object> fun) throws SQLException {
Connection conn = null;
Statement stmt = null;
ResultSet rs = null;
try {
conn = ds.getConnection();
stmt = conn.createStatement(ResultSet.TYPE_FORWARD_ONLY, ResultSet.CONCUR_READ_ONLY);
stmt.setFetchSize(Integer.MIN_VALUE);
rs = stmt.executeQuery(sql);
return fun.apply(rs);
} finally {
if (rs != null) {
try {
rs.close();
} catch (SQLException e) {
logger.error(e.getMessage(), e);
}
}
if (stmt != null) {
try {
stmt.close();
} catch (SQLException e) {
logger.error(e.getMessage(), e);
}
}
if (conn != null) {
try {
conn.close();
} catch (SQLException e) {
logger.error(e.getMessage(), e);
}
}
rs = null;
stmt = null;
conn = null;
}
}
/**
* 建表
*
* @param hbaseTemplate
* @param config
*/
public static void createTable(HbaseTemplate hbaseTemplate, MappingConfig config) {
try {
// 判断hbase表是否存在,不存在则建表
MappingConfig.HbaseMapping hbaseMapping = config.getHbaseMapping();
if (!hbaseTemplate.tableExists(hbaseMapping.getHbaseTable())) {
hbaseTemplate.createTable(hbaseMapping.getHbaseTable(), hbaseMapping.getFamily());
}
} catch (Exception e) {
logger.error(e.getMessage(), e);
throw new RuntimeException(e);
}
}
/**
* 导入数据
*
* @param ds 数据源
* @param hbaseTemplate hbaseTemplate
* @param config 配置
* @param params 筛选条件
* @return 导入结果
*/
public static EtlResult importData(DataSource ds, HbaseTemplate hbaseTemplate, MappingConfig config,
List<String> params) {
EtlResult etlResult = new EtlResult();
AtomicLong successCount = new AtomicLong();
List<String> errMsg = new ArrayList<>();
String hbaseTable = "";
try {
if (config == null) {
logger.error("Config is null!");
etlResult.setSucceeded(false);
etlResult.setErrorMessage("Config is null!");
return etlResult;
}
MappingConfig.HbaseMapping hbaseMapping = config.getHbaseMapping();
hbaseTable = hbaseMapping.getHbaseTable();
long start = System.currentTimeMillis();
if (params != null && params.size() == 1 && "rebuild".equalsIgnoreCase(params.get(0))) {
logger.info(hbaseMapping.getHbaseTable() + " rebuild is starting!");
// 如果表存在则删除
if (hbaseTemplate.tableExists(hbaseMapping.getHbaseTable())) {
hbaseTemplate.disableTable(hbaseMapping.getHbaseTable());
hbaseTemplate.deleteTable(hbaseMapping.getHbaseTable());
}
params = null;
} else {
logger.info(hbaseMapping.getHbaseTable() + " etl is starting!");
}
createTable(hbaseTemplate, config);
// 拼接sql
String sql = "SELECT * FROM " + config.getHbaseMapping().getDatabase() + "." + hbaseMapping.getTable();
// 拼接条件
if (params != null && params.size() == 1 && hbaseMapping.getEtlCondition() == null) {
AtomicBoolean stExists = new AtomicBoolean(false);
// 验证是否有SYS_TIME字段
sqlRS(ds, sql, rs -> {
try {
ResultSetMetaData rsmd = rs.getMetaData();
int cnt = rsmd.getColumnCount();
for (int i = 1; i <= cnt; i++) {
String columnName = rsmd.getColumnName(i);
if ("SYS_TIME".equalsIgnoreCase(columnName)) {
stExists.set(true);
break;
}
}
} catch (Exception e) {
// ignore
}
return null;
});
if (stExists.get()) {
sql += " WHERE SYS_TIME >= '" + params.get(0) + "' ";
}
} else if (hbaseMapping.getEtlCondition() != null && params != null) {
String etlCondition = hbaseMapping.getEtlCondition();
int size = params.size();
for (int i = 0; i < size; i++) {
etlCondition = etlCondition.replace("{" + i + "}", params.get(i));
}
sql += " " + etlCondition;
}
// 获取总数
String countSql = "SELECT COUNT(1) FROM ( " + sql + ") _CNT ";
long cnt = (Long) sqlRS(ds, countSql, rs -> {
Long count = null;
try {
if (rs.next()) {
count = ((Number) rs.getObject(1)).longValue();
}
} catch (Exception e) {
logger.error(e.getMessage(), e);
}
return count == null ? 0 : count;
});
// 当大于1万条记录时开启多线程
if (cnt >= 10000) {
int threadCount = 3;
long perThreadCnt = cnt / threadCount;
ExecutorService executor = Executors.newFixedThreadPool(threadCount);
List<Future<Boolean>> futures = new ArrayList<>(threadCount);
for (int i = 0; i < threadCount; i++) {
long offset = i * perThreadCnt;
Long size = null;
if (i != threadCount - 1) {
size = perThreadCnt;
}
String sqlFinal;
if (size != null) {
sqlFinal = sql + " LIMIT " + offset + "," + size;
} else {
sqlFinal = sql + " LIMIT " + offset + "," + cnt;
}
Future<Boolean> future = executor
.submit(() -> executeSqlImport(ds, sqlFinal, hbaseMapping, hbaseTemplate, successCount, errMsg));
futures.add(future);
}
for (Future<Boolean> future : futures) {
future.get();
}
executor.shutdown();
} else {
executeSqlImport(ds, sql, hbaseMapping, hbaseTemplate, successCount, errMsg);
}
logger.info(
hbaseMapping.getHbaseTable() + " etl completed in: " + (System.currentTimeMillis() - start) / 1000 + "s!");
etlResult.setResultMessage("导入HBase表 " + hbaseMapping.getHbaseTable() + " 数据:" + successCount.get() + " 条");
} catch (Exception e) {
logger.error(e.getMessage(), e);
errMsg.add(hbaseTable + " etl failed! ==>" + e.getMessage());
}
if (errMsg.isEmpty()) {
etlResult.setSucceeded(true);
} else {
etlResult.setErrorMessage(Joiner.on("\n").join(errMsg));
}
return etlResult;
}
/**
* 执行导入
*
* @param ds
* @param sql
* @param hbaseMapping
* @param hbaseTemplate
* @param successCount
* @param errMsg
* @return
*/
private static boolean executeSqlImport(DataSource ds, String sql, MappingConfig.HbaseMapping hbaseMapping,
HbaseTemplate hbaseTemplate, AtomicLong successCount, List<String> errMsg) {
try {
sqlRS(ds, sql, rs -> {
int i = 1;
try {
boolean complete = false;
List<HRow> rows = new ArrayList<>();
String[] rowKeyColumns = null;
if (hbaseMapping.getRowKey() != null) {
rowKeyColumns = hbaseMapping.getRowKey().trim().split(",");
}
while (rs.next()) {
int cc = rs.getMetaData().getColumnCount();
int[] jdbcTypes = new int[cc];
Class<?>[] classes = new Class[cc];
for (int j = 1; j <= cc; j++) {
int jdbcType = rs.getMetaData().getColumnType(j);
jdbcTypes[j - 1] = jdbcType;
classes[j - 1] = JdbcTypeUtil.jdbcType2javaType(jdbcType);
}
HRow row = new HRow();
if (rowKeyColumns != null) {
// 取rowKey字段拼接
StringBuilder rowKeyVale = new StringBuilder();
for (String rowKeyColumnName : rowKeyColumns) {
Object obj = rs.getObject(rowKeyColumnName);
if (obj != null) {
rowKeyVale.append(obj.toString());
}
rowKeyVale.append("|");
}
int len = rowKeyVale.length();
if (len > 0) {
rowKeyVale.delete(len - 1, len);
}
row.setRowKey(Bytes.toBytes(rowKeyVale.toString()));
}
for (int j = 1; j <= cc; j++) {
String columnName = rs.getMetaData().getColumnName(j);
Object val = JdbcTypeUtil.getRSData(rs, columnName, jdbcTypes[j - 1]);
if (val == null) {
continue;
}
MappingConfig.ColumnItem columnItem = hbaseMapping.getColumnItems().get(columnName);
// 没有配置映射
if (columnItem == null) {
String family = hbaseMapping.getFamily();
String qualifile = columnName;
if (hbaseMapping.isUppercaseQualifier()) {
qualifile = qualifile.toUpperCase();
}
if (MappingConfig.Mode.STRING == hbaseMapping.getMode()) {
if (hbaseMapping.getRowKey() == null && j == 1) {
row.setRowKey(Bytes.toBytes(val.toString()));
} else {
row.addCell(family, qualifile, Bytes.toBytes(val.toString()));
}
} else if (MappingConfig.Mode.NATIVE == hbaseMapping.getMode()) {
Type type = Type.getType(classes[j - 1]);
if (hbaseMapping.getRowKey() == null && j == 1) {
row.setRowKey(TypeUtil.toBytes(val, type));
} else {
row.addCell(family, qualifile, TypeUtil.toBytes(val, type));
}
} else if (MappingConfig.Mode.PHOENIX == hbaseMapping.getMode()) {
PhType phType = PhType.getType(classes[j - 1]);
if (hbaseMapping.getRowKey() == null && j == 1) {
row.setRowKey(PhTypeUtil.toBytes(val, phType));
} else {
row.addCell(family, qualifile, PhTypeUtil.toBytes(val, phType));
}
}
} else {
// 如果不需要类型转换
if (columnItem.getType() == null || "".equals(columnItem.getType())) {
if (val instanceof java.sql.Date) {
SimpleDateFormat dateFmt = new SimpleDateFormat("yyyy-MM-dd");
val = dateFmt.format((Date) val);
} else if (val instanceof Timestamp) {
SimpleDateFormat datetimeFmt = new SimpleDateFormat("yyyy-MM-dd HH:mm:ss");
val = datetimeFmt.format((Date) val);
}
byte[] valBytes = Bytes.toBytes(val.toString());
if (columnItem.isRowKey()) {
row.setRowKey(valBytes);
} else {
row.addCell(columnItem.getFamily(), columnItem.getQualifier(), valBytes);
}
} else {
PhType phType = PhType.getType(columnItem.getType());
if (columnItem.isRowKey()) {
row.setRowKey(PhTypeUtil.toBytes(val, phType));
} else {
row.addCell(columnItem.getFamily(),
columnItem.getQualifier(),
PhTypeUtil.toBytes(val, phType));
}
}
}
}
if (row.getRowKey() == null) throw new RuntimeException("RowKey 值为空");
rows.add(row);
complete = false;
if (i % hbaseMapping.getCommitBatch() == 0 && !rows.isEmpty()) {
hbaseTemplate.puts(hbaseMapping.getHbaseTable(), rows);
rows.clear();
complete = true;
}
i++;
successCount.incrementAndGet();
if (logger.isDebugEnabled()) {
logger.debug("successful import count:" + successCount.get());
}
}
if (!complete && !rows.isEmpty()) {
hbaseTemplate.puts(hbaseMapping.getHbaseTable(), rows);
}
} catch (Exception e) {
logger.error(hbaseMapping.getHbaseTable() + " etl failed! ==>" + e.getMessage(), e);
errMsg.add(hbaseMapping.getHbaseTable() + " etl failed! ==>" + e.getMessage());
// throw new RuntimeException(e);
}
return i;
});
return true;
} catch (Exception e) {
logger.error(e.getMessage(), e);
return false;
}
}
}

View File

@@ -1,426 +0,0 @@
package com.alibaba.otter.canal.client.adapter.hbase.service;
import java.util.*;
import org.apache.hadoop.hbase.util.Bytes;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
import com.alibaba.otter.canal.client.adapter.hbase.config.MappingConfig;
import com.alibaba.otter.canal.client.adapter.hbase.support.*;
import com.alibaba.otter.canal.client.adapter.support.Dml;
/**
* HBase同步操作业务
*
* @author rewerma 2018-8-21 下午06:45:49
* @version 1.0.0
*/
public class HbaseSyncService {
private Logger logger = LoggerFactory.getLogger(this.getClass());
private HbaseTemplate hbaseTemplate; // HBase操作模板
public HbaseSyncService(HbaseTemplate hbaseTemplate){
this.hbaseTemplate = hbaseTemplate;
}
public void sync(MappingConfig config, Dml dml) {
try {
if (config != null) {
String type = dml.getType();
if (type != null && type.equalsIgnoreCase("INSERT")) {
insert(config, dml);
} else if (type != null && type.equalsIgnoreCase("UPDATE")) {
update(config, dml);
} else if (type != null && type.equalsIgnoreCase("DELETE")) {
delete(config, dml);
}
if (logger.isDebugEnabled()) {
String res = dml.toString();
logger.debug(res);
}
}
} catch (Exception e) {
logger.error(e.getMessage(), e);
}
}
/**
* 插入操作
*
* @param config 配置项
* @param dml DML数据
*/
private void insert(MappingConfig config, Dml dml) {
List<Map<String, Object>> data = dml.getData();
if (data == null || data.isEmpty()) {
return;
}
MappingConfig.HbaseMapping hbaseMapping = config.getHbaseMapping();
// if (!validHTable(config)) {
// logger.error("HBase table '{}' not exists",
// hbaseMapping.getHbaseTable());
// return;
// }
int i = 1;
boolean complete = false;
List<HRow> rows = new ArrayList<>();
for (Map<String, Object> r : data) {
HRow hRow = new HRow();
// 拼接复合rowKey
if (hbaseMapping.getRowKey() != null) {
String[] rowKeyColumns = hbaseMapping.getRowKey().trim().split(",");
String rowKeyVale = getRowKeys(rowKeyColumns, r);
// params.put("rowKey", Bytes.toBytes(rowKeyVale));
hRow.setRowKey(Bytes.toBytes(rowKeyVale));
}
convertData2Row(hbaseMapping, hRow, r);
if (hRow.getRowKey() == null) {
throw new RuntimeException("empty rowKey");
}
rows.add(hRow);
complete = false;
if (i % config.getHbaseMapping().getCommitBatch() == 0 && !rows.isEmpty()) {
hbaseTemplate.puts(hbaseMapping.getHbaseTable(), rows);
rows.clear();
complete = true;
}
i++;
}
if (!complete && !rows.isEmpty()) {
hbaseTemplate.puts(hbaseMapping.getHbaseTable(), rows);
}
}
/**
* 将Map数据转换为HRow行数据
*
* @param hbaseMapping hbase映射配置
* @param hRow 行对象
* @param data Map数据
*/
private static void convertData2Row(MappingConfig.HbaseMapping hbaseMapping, HRow hRow, Map<String, Object> data) {
Map<String, MappingConfig.ColumnItem> columnItems = hbaseMapping.getColumnItems();
int i = 0;
for (Map.Entry<String, Object> entry : data.entrySet()) {
if (hbaseMapping.getExcludeColumns() != null && hbaseMapping.getExcludeColumns().contains(entry.getKey())) {
continue;
}
if (entry.getValue() != null) {
MappingConfig.ColumnItem columnItem = columnItems.get(entry.getKey());
byte[] bytes = typeConvert(columnItem, hbaseMapping, entry.getValue());
if (columnItem == null) {
String familyName = hbaseMapping.getFamily();
String qualifier = entry.getKey();
if (hbaseMapping.isUppercaseQualifier()) {
qualifier = qualifier.toUpperCase();
}
if (hbaseMapping.getRowKey() == null && i == 0) {
hRow.setRowKey(bytes);
} else {
hRow.addCell(familyName, qualifier, bytes);
}
} else {
if (columnItem.isRowKey()) {
// row.put("rowKey", bytes);
hRow.setRowKey(bytes);
} else {
hRow.addCell(columnItem.getFamily(), columnItem.getQualifier(), bytes);
}
}
}
i++;
}
}
/**
* 更新操作
*
* @param config
* @param dml
*/
private void update(MappingConfig config, Dml dml) {
List<Map<String, Object>> data = dml.getData();
List<Map<String, Object>> old = dml.getOld();
if (old == null || old.isEmpty() || data == null || data.isEmpty()) {
return;
}
MappingConfig.HbaseMapping hbaseMapping = config.getHbaseMapping();
// if (!validHTable(config)) {
// logger.error("HBase table '{}' not exists",
// hbaseMapping.getHbaseTable());
// return;
// }
MappingConfig.ColumnItem rowKeyColumn = hbaseMapping.getRowKeyColumn();
int index = 0;
int i = 1;
boolean complete = false;
List<HRow> rows = new ArrayList<>();
out: for (Map<String, Object> r : data) {
byte[] rowKeyBytes;
if (hbaseMapping.getRowKey() != null) {
String[] rowKeyColumns = hbaseMapping.getRowKey().trim().split(",");
// 判断是否有复合主键修改
for (String updateColumn : old.get(index).keySet()) {
for (String rowKeyColumnName : rowKeyColumns) {
if (rowKeyColumnName.equalsIgnoreCase(updateColumn)) {
// 调用删除插入操作
deleteAndInsert(config, dml);
continue out;
}
}
}
String rowKeyVale = getRowKeys(rowKeyColumns, r);
rowKeyBytes = Bytes.toBytes(rowKeyVale);
} else if (rowKeyColumn == null) {
Map<String, Object> rowKey = data.get(0);
rowKeyBytes = typeConvert(null, hbaseMapping, rowKey.values().iterator().next());
} else {
rowKeyBytes = typeConvert(rowKeyColumn, hbaseMapping, r.get(rowKeyColumn.getColumn()));
}
if (rowKeyBytes == null) throw new RuntimeException("rowKey值为空");
Map<String, MappingConfig.ColumnItem> columnItems = hbaseMapping.getColumnItems();
HRow hRow = new HRow(rowKeyBytes);
for (String updateColumn : old.get(index).keySet()) {
if (hbaseMapping.getExcludeColumns() != null && hbaseMapping.getExcludeColumns().contains(updateColumn)) {
continue;
}
MappingConfig.ColumnItem columnItem = columnItems.get(updateColumn);
if (columnItem == null) {
String family = hbaseMapping.getFamily();
String qualifier = updateColumn;
if (hbaseMapping.isUppercaseQualifier()) {
qualifier = qualifier.toUpperCase();
}
Object newVal = r.get(updateColumn);
if (newVal == null) {
hRow.addCell(family, qualifier, null);
} else {
hRow.addCell(family, qualifier, typeConvert(null, hbaseMapping, newVal));
}
} else {
// 排除修改id的情况
if (columnItem.isRowKey()) continue;
Object newVal = r.get(updateColumn);
if (newVal == null) {
hRow.addCell(columnItem.getFamily(), columnItem.getQualifier(), null);
} else {
hRow.addCell(columnItem.getFamily(),
columnItem.getQualifier(),
typeConvert(columnItem, hbaseMapping, newVal));
}
}
}
rows.add(hRow);
complete = false;
if (i % config.getHbaseMapping().getCommitBatch() == 0 && !rows.isEmpty()) {
hbaseTemplate.puts(hbaseMapping.getHbaseTable(), rows);
rows.clear();
complete = true;
}
i++;
index++;
}
if (!complete && !rows.isEmpty()) {
hbaseTemplate.puts(hbaseMapping.getHbaseTable(), rows);
}
}
private void delete(MappingConfig config, Dml dml) {
List<Map<String, Object>> data = dml.getData();
if (data == null || data.isEmpty()) {
return;
}
MappingConfig.HbaseMapping hbaseMapping = config.getHbaseMapping();
// if (!validHTable(config)) {
// logger.error("HBase table '{}' not exists",
// hbaseMapping.getHbaseTable());
// return;
// }
MappingConfig.ColumnItem rowKeyColumn = hbaseMapping.getRowKeyColumn();
boolean complete = false;
int i = 1;
Set<byte[]> rowKeys = new HashSet<>();
for (Map<String, Object> r : data) {
byte[] rowKeyBytes;
if (hbaseMapping.getRowKey() != null) {
String[] rowKeyColumns = hbaseMapping.getRowKey().trim().split(",");
String rowKeyVale = getRowKeys(rowKeyColumns, r);
rowKeyBytes = Bytes.toBytes(rowKeyVale);
} else if (rowKeyColumn == null) {
// 如果不需要类型转换
Map<String, Object> rowKey = data.get(0);
rowKeyBytes = typeConvert(null, hbaseMapping, rowKey.values().iterator().next());
} else {
Object val = r.get(rowKeyColumn.getColumn());
rowKeyBytes = typeConvert(rowKeyColumn, hbaseMapping, val);
}
if (rowKeyBytes == null) throw new RuntimeException("rowKey值为空");
rowKeys.add(rowKeyBytes);
complete = false;
if (i % config.getHbaseMapping().getCommitBatch() == 0 && !rowKeys.isEmpty()) {
hbaseTemplate.deletes(hbaseMapping.getHbaseTable(), rowKeys);
rowKeys.clear();
complete = true;
}
i++;
}
if (!complete && !rowKeys.isEmpty()) {
hbaseTemplate.deletes(hbaseMapping.getHbaseTable(), rowKeys);
}
}
private void deleteAndInsert(MappingConfig config, Dml dml) {
List<Map<String, Object>> data = dml.getData();
List<Map<String, Object>> old = dml.getOld();
if (old == null || old.isEmpty() || data == null || data.isEmpty()) {
return;
}
MappingConfig.HbaseMapping hbaseMapping = config.getHbaseMapping();
String[] rowKeyColumns = hbaseMapping.getRowKey().trim().split(",");
int index = 0;
int i = 1;
boolean complete = false;
Set<byte[]> rowKeys = new HashSet<>();
List<HRow> rows = new ArrayList<>();
for (Map<String, Object> r : data) {
// 拼接老的rowKey
List<String> updateSubRowKey = new ArrayList<>();
for (String rowKeyColumnName : rowKeyColumns) {
for (String updateColumn : old.get(index).keySet()) {
if (rowKeyColumnName.equalsIgnoreCase(updateColumn)) {
updateSubRowKey.add(rowKeyColumnName);
}
}
}
if (updateSubRowKey.isEmpty()) {
throw new RuntimeException("没有更新复合主键的RowKey");
}
StringBuilder oldRowKey = new StringBuilder();
StringBuilder newRowKey = new StringBuilder();
for (String rowKeyColumnName : rowKeyColumns) {
newRowKey.append(r.get(rowKeyColumnName).toString()).append("|");
if (!updateSubRowKey.contains(rowKeyColumnName)) {
// 从data取
oldRowKey.append(r.get(rowKeyColumnName).toString()).append("|");
} else {
// 从old取
oldRowKey.append(old.get(index).get(rowKeyColumnName).toString()).append("|");
}
}
int len = newRowKey.length();
newRowKey.delete(len - 1, len);
len = oldRowKey.length();
oldRowKey.delete(len - 1, len);
byte[] newRowKeyBytes = Bytes.toBytes(newRowKey.toString());
byte[] oldRowKeyBytes = Bytes.toBytes(oldRowKey.toString());
rowKeys.add(oldRowKeyBytes);
HRow row = new HRow(newRowKeyBytes);
convertData2Row(hbaseMapping, row, r);
rows.add(row);
complete = false;
if (i % config.getHbaseMapping().getCommitBatch() == 0 && !rows.isEmpty()) {
hbaseTemplate.deletes(hbaseMapping.getHbaseTable(), rowKeys);
hbaseTemplate.puts(hbaseMapping.getHbaseTable(), rows);
rowKeys.clear();
rows.clear();
complete = true;
}
i++;
index++;
}
if (!complete && !rows.isEmpty()) {
hbaseTemplate.deletes(hbaseMapping.getHbaseTable(), rowKeys);
hbaseTemplate.puts(hbaseMapping.getHbaseTable(), rows);
}
}
/**
* 根据对应的类型进行转换
*
* @param columnItem 列项配置
* @param hbaseMapping hbase映射配置
* @param value 值
* @return 复合字段rowKey
*/
private static byte[] typeConvert(MappingConfig.ColumnItem columnItem, MappingConfig.HbaseMapping hbaseMapping,
Object value) {
if (value == null) {
return null;
}
byte[] bytes = null;
if (columnItem == null || columnItem.getType() == null || "".equals(columnItem.getType())) {
if (MappingConfig.Mode.STRING == hbaseMapping.getMode()) {
bytes = Bytes.toBytes(value.toString());
} else if (MappingConfig.Mode.NATIVE == hbaseMapping.getMode()) {
bytes = TypeUtil.toBytes(value);
} else if (MappingConfig.Mode.PHOENIX == hbaseMapping.getMode()) {
PhType phType = PhType.getType(value.getClass());
bytes = PhTypeUtil.toBytes(value, phType);
}
} else {
if (hbaseMapping.getMode() == MappingConfig.Mode.STRING) {
bytes = Bytes.toBytes(value.toString());
} else if (hbaseMapping.getMode() == MappingConfig.Mode.NATIVE) {
Type type = Type.getType(columnItem.getType());
bytes = TypeUtil.toBytes(value, type);
} else if (hbaseMapping.getMode() == MappingConfig.Mode.PHOENIX) {
PhType phType = PhType.getType(columnItem.getType());
bytes = PhTypeUtil.toBytes(value, phType);
}
}
return bytes;
}
/**
* 获取复合字段作为rowKey的拼接
*
* @param rowKeyColumns 复合rowK对应的字段
* @param data 数据
* @return
*/
private static String getRowKeys(String[] rowKeyColumns, Map<String, Object> data) {
StringBuilder rowKeyValue = new StringBuilder();
for (String rowKeyColumnName : rowKeyColumns) {
Object obj = data.get(rowKeyColumnName);
if (obj != null) {
rowKeyValue.append(obj.toString());
}
rowKeyValue.append("|");
}
int len = rowKeyValue.length();
if (len > 0) {
rowKeyValue.delete(len - 1, len);
}
return rowKeyValue.toString();
}
}

View File

@@ -1,84 +0,0 @@
package com.alibaba.otter.canal.client.adapter.hbase.support;
import java.util.ArrayList;
import java.util.List;
/**
* HBase操作对象类
*
* @author machengyuan 2018-8-21 下午10:12:34
* @version 1.0.0
*/
public class HRow {
private byte[] rowKey;
private List<HCell> cells = new ArrayList<>();
public HRow(){
}
public HRow(byte[] rowKey){
this.rowKey = rowKey;
}
public byte[] getRowKey() {
return rowKey;
}
public void setRowKey(byte[] rowKey) {
this.rowKey = rowKey;
}
public List<HCell> getCells() {
return cells;
}
public void setCells(List<HCell> cells) {
this.cells = cells;
}
public void addCell(String family, String qualifier, byte[] value) {
HCell hCell = new HCell(family, qualifier, value);
cells.add(hCell);
}
public static class HCell {
private String family;
private String qualifier;
private byte[] value;
public HCell(){
}
public HCell(String family, String qualifier, byte[] value){
this.family = family;
this.qualifier = qualifier;
this.value = value;
}
public String getFamily() {
return family;
}
public void setFamily(String family) {
this.family = family;
}
public String getQualifier() {
return qualifier;
}
public void setQualifier(String qualifier) {
this.qualifier = qualifier;
}
public byte[] getValue() {
return value;
}
public void setValue(byte[] value) {
this.value = value;
}
}
}

View File

@@ -1,163 +0,0 @@
package com.alibaba.otter.canal.client.adapter.hbase.support;
import java.io.IOException;
import java.util.ArrayList;
import java.util.List;
import java.util.Set;
import org.apache.hadoop.hbase.HColumnDescriptor;
import org.apache.hadoop.hbase.HTableDescriptor;
import org.apache.hadoop.hbase.TableName;
import org.apache.hadoop.hbase.client.Connection;
import org.apache.hadoop.hbase.client.Delete;
import org.apache.hadoop.hbase.client.HBaseAdmin;
import org.apache.hadoop.hbase.client.HTable;
import org.apache.hadoop.hbase.client.Put;
import org.apache.hadoop.hbase.util.Bytes;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
/**
* HBase操作模板
*
* @author machengyuan 2018-8-21 下午10:12:34
* @version 1.0.0
*/
public class HbaseTemplate {
private Logger logger = LoggerFactory.getLogger(this.getClass());
private Connection conn;
public HbaseTemplate(Connection conn){
this.conn = conn;
}
public boolean tableExists(String tableName) {
try (HBaseAdmin admin = (HBaseAdmin) conn.getAdmin()) {
return admin.tableExists(TableName.valueOf(tableName));
} catch (IOException e) {
throw new RuntimeException(e);
}
}
public void createTable(String tableName, String... familyNames) {
try (HBaseAdmin admin = (HBaseAdmin) conn.getAdmin()) {
HTableDescriptor desc = new HTableDescriptor(TableName.valueOf(tableName));
// 添加列簇
if (familyNames != null) {
for (String familyName : familyNames) {
HColumnDescriptor hcd = new HColumnDescriptor(familyName);
desc.addFamily(hcd);
}
}
admin.createTable(desc);
} catch (IOException e) {
throw new RuntimeException(e);
}
}
public void disableTable(String tableName) {
try (HBaseAdmin admin = (HBaseAdmin) conn.getAdmin()) {
admin.disableTable(tableName);
} catch (IOException e) {
logger.error(e.getMessage(), e);
throw new RuntimeException(e);
}
}
public void deleteTable(String tableName) {
try (HBaseAdmin admin = (HBaseAdmin) conn.getAdmin()) {
if (admin.isTableEnabled(tableName)) {
disableTable(tableName);
}
admin.deleteTable(tableName);
} catch (IOException e) {
logger.error(e.getMessage(), e);
throw new RuntimeException(e);
}
}
/**
* 插入一行数据
*
* @param tableName 表名
* @param hRow 行数据对象
* @return 是否成功
*/
public Boolean put(String tableName, HRow hRow) {
boolean flag = false;
try {
HTable table = (HTable) conn.getTable(TableName.valueOf(tableName));
Put put = new Put(hRow.getRowKey());
for (HRow.HCell hCell : hRow.getCells()) {
put.addColumn(Bytes.toBytes(hCell.getFamily()), Bytes.toBytes(hCell.getQualifier()), hCell.getValue());
}
table.put(put);
flag = true;
} catch (Exception e) {
logger.error(e.getMessage(), e);
}
return flag;
}
/**
* 批量插入
*
* @param tableName 表名
* @param rows 行数据对象集合
* @return 是否成功
*/
public Boolean puts(String tableName, List<HRow> rows) {
boolean flag = false;
try {
HTable table = (HTable) conn.getTable(TableName.valueOf(tableName));
List<Put> puts = new ArrayList<>();
for (HRow hRow : rows) {
Put put = new Put(hRow.getRowKey());
for (HRow.HCell hCell : hRow.getCells()) {
put.addColumn(Bytes.toBytes(hCell.getFamily()),
Bytes.toBytes(hCell.getQualifier()),
hCell.getValue());
}
puts.add(put);
}
if (!puts.isEmpty()) {
table.put(puts);
}
flag = true;
} catch (Exception e) {
logger.error(e.getMessage(), e);
}
return flag;
}
/**
* 批量删除数据
*
* @param tableName 表名
* @param rowKeys rowKey集合
* @return 是否成功
*/
public Boolean deletes(String tableName, Set<byte[]> rowKeys) {
boolean flag = false;
try {
HTable table = (HTable) conn.getTable(TableName.valueOf(tableName));
List<Delete> deletes = new ArrayList<>();
for (byte[] rowKey : rowKeys) {
Delete delete = new Delete(rowKey);
deletes.add(delete);
}
if (!deletes.isEmpty()) {
table.delete(deletes);
}
flag = true;
} catch (Exception e) {
logger.error(e.getMessage(), e);
}
return flag;
}
}

View File

@@ -1,73 +0,0 @@
package com.alibaba.otter.canal.client.adapter.hbase.support;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
import java.math.BigDecimal;
import java.math.BigInteger;
import java.sql.Time;
import java.sql.Timestamp;
import java.util.Date;
/**
* Phoenix类型
*
* @author machengyuan 2018-8-21 下午06:12:34
* @version 1.0.0
*/
public enum PhType {
DEFAULT, UNSIGNED_INT, UNSIGNED_LONG, UNSIGNED_TINYINT, UNSIGNED_SMALLINT, UNSIGNED_FLOAT,
UNSIGNED_DOUBLE, INTEGER, BIGINT, TINYINT, SMALLINT, FLOAT, DOUBLE, DECIMAL, BOOLEAN, UNSIGNED_TIME,
UNSIGNED_DATE, UNSIGNED_TIMESTAMP, TIME, DATE, TIMESTAMP, VARCHAR, VARBINARY;
private static Logger logger = LoggerFactory.getLogger(PhType.class);
public static PhType getType(Class<?> javaType) {
if (javaType == null) return DEFAULT;
PhType phType;
if (Integer.class == javaType || int.class == javaType) {
phType = INTEGER;
} else if (Long.class == javaType || long.class == javaType) {
phType = BIGINT;
} else if (Byte.class == javaType || byte.class == javaType) {
phType = TINYINT;
} else if (Short.class == javaType || short.class == javaType) {
phType = SMALLINT;
} else if (Float.class == javaType || float.class == javaType) {
phType = FLOAT;
} else if (Double.class == javaType || double.class == javaType) {
phType = DOUBLE;
} else if (Boolean.class == javaType || boolean.class == javaType) {
phType = BOOLEAN;
} else if (java.sql.Date.class == javaType) {
phType = DATE;
} else if (Time.class == javaType) {
phType = DATE;
} else if (Timestamp.class == javaType) {
phType = TIMESTAMP;
} else if (Date.class == javaType) {
phType = DATE;
} else if (byte[].class == javaType) {
phType = VARBINARY;
} else if (String.class == javaType) {
phType = VARCHAR;
} else if (BigDecimal.class == javaType) {
phType = DECIMAL;
} else if (BigInteger.class == javaType) {
phType = UNSIGNED_LONG;
} else {
phType = DEFAULT;
}
return phType;
}
public static PhType getType(String type) {
if (type == null) return DEFAULT;
try {
return PhType.valueOf(type.toUpperCase());
} catch (Exception e) {
logger.error(e.getMessage(), e);
return DEFAULT;
}
}
}

View File

@@ -1,620 +0,0 @@
package com.alibaba.otter.canal.client.adapter.hbase.support;
import java.math.BigDecimal;
import java.math.BigInteger;
import java.math.MathContext;
import java.math.RoundingMode;
import java.sql.Timestamp;
import java.util.Date;
import org.apache.hadoop.hbase.util.Bytes;
import org.joda.time.DateTime;
import com.google.common.math.LongMath;
/**
* Phoenix类型转换工具类
*
* @author machengyuan 2018-8-21 下午06:14:26
* @version 1.0.0
*/
public class PhTypeUtil {
public static byte[] toBytes(Object v, PhType phType) {
if (v == null) return null;
byte[] b = null;
if (phType == PhType.DEFAULT) {
PhType phType1 = PhType.getType(v.getClass());
if (phType1 != null && phType1 != PhType.DEFAULT) {
toBytes(v, phType1);
}
} else if (phType == PhType.INTEGER) {
b = new byte[Bytes.SIZEOF_INT];
encodeInt(((Number) v).intValue(), b, 0);
} else if (phType == PhType.UNSIGNED_INT) {
b = new byte[Bytes.SIZEOF_INT];
encodeUnsignedInt(((Number) v).intValue(), b, 0);
} else if (phType == PhType.BIGINT) {
b = new byte[Bytes.SIZEOF_LONG];
encodeLong(((Number) v).longValue(), b, 0);
} else if (phType == PhType.UNSIGNED_LONG) {
b = new byte[Bytes.SIZEOF_LONG];
encodeUnsignedLong(((Number) v).longValue(), b, 0);
} else if (phType == PhType.SMALLINT) {
b = new byte[Bytes.SIZEOF_SHORT];
encodeShort(((Number) v).shortValue(), b, 0);
} else if (phType == PhType.UNSIGNED_SMALLINT) {
b = new byte[Bytes.SIZEOF_SHORT];
encodeUnsignedShort(((Number) v).shortValue(), b, 0);
} else if (phType == PhType.TINYINT) {
b = new byte[Bytes.SIZEOF_BYTE];
encodeByte(((Number) v).byteValue(), b, 0);
} else if (phType == PhType.UNSIGNED_TINYINT) {
b = new byte[Bytes.SIZEOF_BYTE];
encodeUnsignedByte(((Number) v).byteValue(), b, 0);
} else if (phType == PhType.FLOAT) {
b = new byte[Bytes.SIZEOF_FLOAT];
encodeFloat(((Number) v).floatValue(), b, 0);
} else if (phType == PhType.UNSIGNED_FLOAT) {
b = new byte[Bytes.SIZEOF_FLOAT];
encodeUnsignedFloat(((Number) v).floatValue(), b, 0);
} else if (phType == PhType.DOUBLE) {
b = new byte[Bytes.SIZEOF_DOUBLE];
encodeDouble(((Number) v).doubleValue(), b, 0);
} else if (phType == PhType.UNSIGNED_DOUBLE) {
b = new byte[Bytes.SIZEOF_DOUBLE];
encodeUnsignedDouble(((Number) v).doubleValue(), b, 0);
} else if (phType == PhType.BOOLEAN) {
if ((Boolean) v) {
b = new byte[] { 1 };
} else {
b = new byte[] { 0 };
}
} else if (phType == PhType.TIME || phType == PhType.DATE) {
b = new byte[Bytes.SIZEOF_LONG];
encodeDate(v, b, 0);
} else if (phType == PhType.TIMESTAMP) {
b = new byte[Bytes.SIZEOF_LONG + Bytes.SIZEOF_INT];
encodeTimestamp(v, b, 0);
} else if (phType == PhType.UNSIGNED_TIME || phType == PhType.UNSIGNED_DATE) {
b = new byte[Bytes.SIZEOF_LONG];
encodeUnsignedDate(v, b, 0);
} else if (phType == PhType.UNSIGNED_TIMESTAMP) {
b = new byte[Bytes.SIZEOF_LONG + Bytes.SIZEOF_INT];
encodeUnsignedTimestamp(v, b, 0);
} else if (phType == PhType.VARBINARY) {
b = (byte[]) v;
} else if (phType == PhType.VARCHAR) {
b = Bytes.toBytes(v.toString());
} else if (phType == PhType.DECIMAL) {
if (v instanceof BigDecimal) {
b = encodeDecimal(v);
} else if (v instanceof Number) {
b = encodeDecimal(new BigDecimal(v.toString()));
}
}
return b;
}
public static Object toObject(byte[] b, PhType phType) {
if (b == null) return null;
Object v = null;
if (phType == PhType.INTEGER) {
v = decodeInt(b, 0);
} else if (phType == PhType.UNSIGNED_INT) {
v = decodeUnsignedInt(b, 0);
} else if (phType == PhType.BIGINT) {
v = decodeLong(b, 0);
} else if (phType == PhType.UNSIGNED_LONG) {
v = decodeUnsignedLong(b, 0);
} else if (phType == PhType.SMALLINT) {
v = decodeShort(b, 0);
} else if (phType == PhType.UNSIGNED_SMALLINT) {
v = decodeUnsignedShort(b, 0);
} else if (phType == PhType.TINYINT) {
v = decodeByte(b, 0);
} else if (phType == PhType.UNSIGNED_TINYINT) {
v = decodeUnsignedByte(b, 0);
} else if (phType == PhType.FLOAT) {
v = decodeFloat(b, 0);
} else if (phType == PhType.UNSIGNED_FLOAT) {
v = decodeUnsignedFloat(b, 0);
} else if (phType == PhType.DOUBLE) {
v = decodeDouble(b, 0);
} else if (phType == PhType.UNSIGNED_DOUBLE) {
v = decodeUnsignedDouble(b, 0);
} else if (phType == PhType.BOOLEAN) {
checkForSufficientLength(b, 0, Bytes.SIZEOF_BOOLEAN);
if (b[0] == 1) {
v = true;
} else if (b[0] == 0) {
v = false;
}
} else if (phType == PhType.TIME || phType == PhType.DATE) {
v = new Date(decodeLong(b, 0));
} else if (phType == PhType.TIMESTAMP) {
long millisDeserialized = decodeLong(b, 0);
Timestamp ts = new Timestamp(millisDeserialized);
int nanosDeserialized = decodeUnsignedInt(b, Bytes.SIZEOF_LONG);
ts.setNanos(nanosDeserialized < 1000000 ? ts.getNanos() + nanosDeserialized : nanosDeserialized);
v = ts;
} else if (phType == PhType.UNSIGNED_TIME || phType == PhType.UNSIGNED_DATE) {
v = new Date(decodeUnsignedLong(b, 0));
} else if (phType == PhType.UNSIGNED_TIMESTAMP) {
long millisDeserialized = decodeUnsignedLong(b, 0);
Timestamp ts = new Timestamp(millisDeserialized);
int nanosDeserialized = decodeUnsignedInt(b, Bytes.SIZEOF_LONG);
ts.setNanos(nanosDeserialized < 1000000 ? ts.getNanos() + nanosDeserialized : nanosDeserialized);
v = ts;
} else if (phType == PhType.VARBINARY) {
v = b;
} else if (phType == PhType.VARCHAR || phType == PhType.DEFAULT) {
v = Bytes.toString(b);
} else if (phType == PhType.DECIMAL) {
v = decodeDecimal(b, 0, b.length);
}
return v;
}
private static int decodeInt(byte[] bytes, int o) {
checkForSufficientLength(bytes, o, Bytes.SIZEOF_INT);
int v;
v = bytes[o] ^ 0x80; // Flip sign bit back
for (int i = 1; i < Bytes.SIZEOF_INT; i++) {
v = (v << 8) + (bytes[o + i] & 0xff);
}
return v;
}
private static int encodeInt(int v, byte[] b, int o) {
checkForSufficientLength(b, o, Bytes.SIZEOF_INT);
b[o + 0] = (byte) ((v >> 24) ^ 0x80); // Flip sign bit so that INTEGER
// is binary comparable
b[o + 1] = (byte) (v >> 16);
b[o + 2] = (byte) (v >> 8);
b[o + 3] = (byte) v;
return Bytes.SIZEOF_INT;
}
private static int decodeUnsignedInt(byte[] b, int o) {
checkForSufficientLength(b, o, Bytes.SIZEOF_INT);
int v = Bytes.toInt(b, o);
if (v < 0) {
throw new RuntimeException();
}
return v;
}
private static int encodeUnsignedInt(int v, byte[] b, int o) {
checkForSufficientLength(b, o, Bytes.SIZEOF_INT);
if (v < 0) {
throw new RuntimeException();
}
Bytes.putInt(b, o, v);
return Bytes.SIZEOF_INT;
}
private static long decodeLong(byte[] bytes, int o) {
checkForSufficientLength(bytes, o, Bytes.SIZEOF_LONG);
long v;
byte b = bytes[o];
v = b ^ 0x80; // Flip sign bit back
for (int i = 1; i < Bytes.SIZEOF_LONG; i++) {
b = bytes[o + i];
v = (v << 8) + (b & 0xff);
}
return v;
}
private static int encodeLong(long v, byte[] b, int o) {
checkForSufficientLength(b, o, Bytes.SIZEOF_LONG);
b[o + 0] = (byte) ((v >> 56) ^ 0x80); // Flip sign bit so that INTEGER
// is binary comparable
b[o + 1] = (byte) (v >> 48);
b[o + 2] = (byte) (v >> 40);
b[o + 3] = (byte) (v >> 32);
b[o + 4] = (byte) (v >> 24);
b[o + 5] = (byte) (v >> 16);
b[o + 6] = (byte) (v >> 8);
b[o + 7] = (byte) v;
return Bytes.SIZEOF_LONG;
}
private static long decodeUnsignedLong(byte[] b, int o) {
checkForSufficientLength(b, o, Bytes.SIZEOF_LONG);
long v = 0;
for (int i = o; i < o + Bytes.SIZEOF_LONG; i++) {
v <<= 8;
v ^= b[i] & 0xFF;
}
if (v < 0) {
throw new RuntimeException();
}
return v;
}
private static int encodeUnsignedLong(long v, byte[] b, int o) {
checkForSufficientLength(b, o, Bytes.SIZEOF_LONG);
if (v < 0) {
throw new RuntimeException();
}
Bytes.putLong(b, o, v);
return Bytes.SIZEOF_LONG;
}
private static short decodeShort(byte[] b, int o) {
checkForSufficientLength(b, o, Bytes.SIZEOF_SHORT);
int v;
v = b[o] ^ 0x80; // Flip sign bit back
for (int i = 1; i < Bytes.SIZEOF_SHORT; i++) {
v = (v << 8) + (b[o + i] & 0xff);
}
return (short) v;
}
private static int encodeShort(short v, byte[] b, int o) {
checkForSufficientLength(b, o, Bytes.SIZEOF_SHORT);
b[o + 0] = (byte) ((v >> 8) ^ 0x80); // Flip sign bit so that Short is
// binary comparable
b[o + 1] = (byte) v;
return Bytes.SIZEOF_SHORT;
}
private static short decodeUnsignedShort(byte[] b, int o) {
checkForSufficientLength(b, o, Bytes.SIZEOF_SHORT);
short v = Bytes.toShort(b, o);
if (v < 0) {
throw new RuntimeException();
}
return v;
}
private static int encodeUnsignedShort(short v, byte[] b, int o) {
checkForSufficientLength(b, o, Bytes.SIZEOF_SHORT);
if (v < 0) {
throw new RuntimeException();
}
Bytes.putShort(b, o, v);
return Bytes.SIZEOF_SHORT;
}
private static byte decodeByte(byte[] b, int o) {
checkForSufficientLength(b, o, Bytes.SIZEOF_BYTE);
int v;
v = b[o] ^ 0x80; // Flip sign bit back
return (byte) v;
}
private static int encodeByte(byte v, byte[] b, int o) {
checkForSufficientLength(b, o, Bytes.SIZEOF_BYTE);
b[o] = (byte) (v ^ 0x80); // Flip sign bit so that Short is binary
// comparable
return Bytes.SIZEOF_BYTE;
}
private static byte decodeUnsignedByte(byte[] b, int o) {
checkForSufficientLength(b, o, Bytes.SIZEOF_BYTE);
byte v = b[o];
if (v < 0) {
throw new RuntimeException();
}
return v;
}
private static int encodeUnsignedByte(byte v, byte[] b, int o) {
if (v < 0) {
throw new RuntimeException();
}
Bytes.putByte(b, o, v);
return Bytes.SIZEOF_BYTE;
}
private static float decodeFloat(byte[] b, int o) {
checkForSufficientLength(b, o, Bytes.SIZEOF_INT);
int value;
value = Bytes.toInt(b, o);
value--;
value ^= (~value >> Integer.SIZE - 1) | Integer.MIN_VALUE;
return Float.intBitsToFloat(value);
}
private static int encodeFloat(float v, byte[] b, int o) {
checkForSufficientLength(b, o, Bytes.SIZEOF_FLOAT);
int i = Float.floatToIntBits(v);
i = (i ^ ((i >> Integer.SIZE - 1) | Integer.MIN_VALUE)) + 1;
Bytes.putInt(b, o, i);
return Bytes.SIZEOF_FLOAT;
}
private static float decodeUnsignedFloat(byte[] b, int o) {
checkForSufficientLength(b, o, Bytes.SIZEOF_FLOAT);
float v = Bytes.toFloat(b, o);
if (v < 0) {
throw new RuntimeException();
}
return v;
}
private static int encodeUnsignedFloat(float v, byte[] b, int o) {
checkForSufficientLength(b, o, Bytes.SIZEOF_FLOAT);
if (v < 0) {
throw new RuntimeException();
}
Bytes.putFloat(b, o, v);
return Bytes.SIZEOF_FLOAT;
}
private static double decodeDouble(byte[] bytes, int o) {
checkForSufficientLength(bytes, o, Bytes.SIZEOF_LONG);
long l;
l = Bytes.toLong(bytes, o);
l--;
l ^= (~l >> Long.SIZE - 1) | Long.MIN_VALUE;
return Double.longBitsToDouble(l);
}
private static int encodeDouble(double v, byte[] b, int o) {
checkForSufficientLength(b, o, Bytes.SIZEOF_LONG);
long l = Double.doubleToLongBits(v);
l = (l ^ ((l >> Long.SIZE - 1) | Long.MIN_VALUE)) + 1;
Bytes.putLong(b, o, l);
return Bytes.SIZEOF_LONG;
}
private static double decodeUnsignedDouble(byte[] b, int o) {
checkForSufficientLength(b, o, Bytes.SIZEOF_DOUBLE);
double v = Bytes.toDouble(b, o);
if (v < 0) {
throw new RuntimeException();
}
return v;
}
private static int encodeUnsignedDouble(double v, byte[] b, int o) {
checkForSufficientLength(b, o, Bytes.SIZEOF_DOUBLE);
if (v < 0) {
throw new RuntimeException();
}
Bytes.putDouble(b, o, v);
return Bytes.SIZEOF_DOUBLE;
}
private static int encodeDate(Object v, byte[] b, int o) {
if (v instanceof Date) {
encodeLong(((Date) v).getTime(), b, 0);
} else if (v instanceof String) {
String dateStr = (String) v;
Date date;
try {
date = parseDatetime(dateStr);
if (date != null) {
encodeLong(date.getTime(), b, 0);
}
} catch (Exception e) {
throw new RuntimeException(e);
}
}
return Bytes.SIZEOF_LONG;
}
private static int encodeTimestamp(Object v, byte[] b, int o) {
if (v instanceof Timestamp) {
Timestamp ts = (Timestamp) v;
encodeLong(ts.getTime(), b, o);
Bytes.putInt(b, Bytes.SIZEOF_LONG, ts.getNanos() % 1000000);
} else {
encodeDate(v, b, o);
}
return Bytes.SIZEOF_LONG + Bytes.SIZEOF_INT;
}
private static int encodeUnsignedDate(Object v, byte[] b, int o) {
if (v instanceof Date) {
encodeUnsignedLong(((Date) v).getTime(), b, 0);
} else if (v instanceof String) {
String dateStr = (String) v;
Date date;
try {
date = parseDatetime(dateStr);
if (date != null) {
encodeUnsignedLong(date.getTime(), b, 0);
}
} catch (Exception e) {
throw new RuntimeException(e);
}
}
return Bytes.SIZEOF_LONG;
}
private static int encodeUnsignedTimestamp(Object v, byte[] b, int o) {
if (v instanceof Timestamp) {
Timestamp ts = (Timestamp) v;
encodeUnsignedLong(ts.getTime(), b, o);
Bytes.putInt(b, Bytes.SIZEOF_LONG, ts.getNanos() % 1000000);
} else {
encodeUnsignedDate(v, b, o);
}
return Bytes.SIZEOF_LONG + Bytes.SIZEOF_INT;
}
private static byte[] encodeDecimal(Object object) {
if (object == null) {
return new byte[0];
}
BigDecimal v = (BigDecimal) object;
v = v.round(DEFAULT_MATH_CONTEXT).stripTrailingZeros();
int len = getLength(v);
byte[] result = new byte[Math.min(len, 21)];
decimalToBytes(v, result, 0, len);
return result;
}
private static BigDecimal decodeDecimal(byte[] bytes, int offset, int length) {
if (length == 1 && bytes[offset] == ZERO_BYTE) {
return BigDecimal.ZERO;
}
int signum = ((bytes[offset] & 0x80) == 0) ? -1 : 1;
int scale;
int index;
int digitOffset;
long multiplier = 100L;
int begIndex = offset + 1;
if (signum == 1) {
scale = (byte) (((bytes[offset] & 0x7F) - 65) * -2);
index = offset + length;
digitOffset = POS_DIGIT_OFFSET;
} else {
scale = (byte) ((~bytes[offset] - 65 - 128) * -2);
index = offset + length - (bytes[offset + length - 1] == NEG_TERMINAL_BYTE ? 1 : 0);
digitOffset = -NEG_DIGIT_OFFSET;
}
length = index - offset;
long l = signum * bytes[--index] - digitOffset;
if (l % 10 == 0) { // trailing zero
scale--; // drop trailing zero and compensate in the scale
l /= 10;
multiplier = 10;
}
// Use long arithmetic for as long as we can
while (index > begIndex) {
if (l >= MAX_LONG_FOR_DESERIALIZE || multiplier >= Long.MAX_VALUE / 100) {
multiplier = LongMath.divide(multiplier, 100L, RoundingMode.UNNECESSARY);
break; // Exit loop early so we don't overflow our multiplier
}
int digit100 = signum * bytes[--index] - digitOffset;
l += digit100 * multiplier;
multiplier = LongMath.checkedMultiply(multiplier, 100);
}
BigInteger bi;
// If still more digits, switch to BigInteger arithmetic
if (index > begIndex) {
bi = BigInteger.valueOf(l);
BigInteger biMultiplier = BigInteger.valueOf(multiplier).multiply(ONE_HUNDRED);
do {
int digit100 = signum * bytes[--index] - digitOffset;
bi = bi.add(biMultiplier.multiply(BigInteger.valueOf(digit100)));
biMultiplier = biMultiplier.multiply(ONE_HUNDRED);
} while (index > begIndex);
if (signum == -1) {
bi = bi.negate();
}
} else {
bi = BigInteger.valueOf(l * signum);
}
// Update the scale based on the precision
scale += (length - 2) * 2;
BigDecimal v = new BigDecimal(bi, scale);
return v;
}
private static int getLength(BigDecimal v) {
int signum = v.signum();
if (signum == 0) { // Special case for zero
return 1;
}
return (signum < 0 ? 2 : 1) + (v.precision() + 1 + (v.scale() % 2 == 0 ? 0 : 1)) / 2;
}
private static final int MAX_PRECISION = 38;
private static final MathContext DEFAULT_MATH_CONTEXT = new MathContext(MAX_PRECISION, RoundingMode.HALF_UP);
private static final Integer MAX_BIG_DECIMAL_BYTES = 21;
private static final byte ZERO_BYTE = (byte) 0x80;
private static final byte NEG_TERMINAL_BYTE = (byte) 102;
private static final int EXP_BYTE_OFFSET = 65;
private static final int POS_DIGIT_OFFSET = 1;
private static final int NEG_DIGIT_OFFSET = 101;
private static final BigInteger MAX_LONG = BigInteger.valueOf(Long.MAX_VALUE);
private static final BigInteger MIN_LONG = BigInteger.valueOf(Long.MIN_VALUE);
private static final BigInteger ONE_HUNDRED = BigInteger.valueOf(100);
private static final long MAX_LONG_FOR_DESERIALIZE = Long.MAX_VALUE / 1000;
private static int decimalToBytes(BigDecimal v, byte[] result, final int offset, int length) {
int signum = v.signum();
if (signum == 0) {
result[offset] = ZERO_BYTE;
return 1;
}
int index = offset + length;
int scale = v.scale();
int expOffset = scale % 2 * (scale < 0 ? -1 : 1);
int multiplyBy;
BigInteger divideBy;
if (expOffset == 0) {
multiplyBy = 1;
divideBy = ONE_HUNDRED;
} else {
multiplyBy = 10;
divideBy = BigInteger.TEN;
}
// Normalize the scale based on what is necessary to end up with a base
// 100
// decimal (i.e. 10.123e3)
int digitOffset;
BigInteger compareAgainst;
if (signum == 1) {
digitOffset = POS_DIGIT_OFFSET;
compareAgainst = MAX_LONG;
scale -= (length - 2) * 2;
result[offset] = (byte) ((-(scale + expOffset) / 2 + EXP_BYTE_OFFSET) | 0x80);
} else {
digitOffset = NEG_DIGIT_OFFSET;
compareAgainst = MIN_LONG;
// Scale adjustment shouldn't include terminal byte in length
scale -= (length - 2 - 1) * 2;
result[offset] = (byte) (~(-(scale + expOffset) / 2 + EXP_BYTE_OFFSET + 128) & 0x7F);
if (length <= MAX_BIG_DECIMAL_BYTES) {
result[--index] = NEG_TERMINAL_BYTE;
} else {
// Adjust length and offset down because we don't have enough
// room
length = MAX_BIG_DECIMAL_BYTES;
index = offset + length;
}
}
BigInteger bi = v.unscaledValue();
// Use BigDecimal arithmetic until we can fit into a long
while (bi.compareTo(compareAgainst) * signum > 0) {
BigInteger[] dandr = bi.divideAndRemainder(divideBy);
bi = dandr[0];
int digit = dandr[1].intValue();
result[--index] = (byte) (digit * multiplyBy + digitOffset);
multiplyBy = 1;
divideBy = ONE_HUNDRED;
}
long l = bi.longValue();
do {
long divBy = 100 / multiplyBy;
long digit = l % divBy;
l /= divBy;
result[--index] = (byte) (digit * multiplyBy + digitOffset);
multiplyBy = 1;
} while (l != 0);
return length;
}
private static void checkForSufficientLength(byte[] b, int offset, int requiredLength) {
if (b.length < offset + requiredLength) {
throw new RuntimeException("Expected length of at least " + requiredLength + " bytes, but had "
+ (b.length - offset));
}
}
private static Date parseDatetime(String dateStr) {
Date date = null;
int len = dateStr.length();
if (len == 10 && dateStr.charAt(4) == '-' && dateStr.charAt(7) == '-') {
date = new DateTime(dateStr).toDate();
} else if (len == 8 && dateStr.charAt(2) == ':' && dateStr.charAt(5) == ':') {
date = new DateTime("T" + dateStr).toDate();
} else if (len >= 19 && dateStr.charAt(4) == '-' && dateStr.charAt(7) == '-' && dateStr.charAt(13) == ':'
&& dateStr.charAt(16) == ':') {
date = new DateTime(dateStr.replace(" ", "T")).toDate();
}
return date;
}
}

View File

@@ -1,67 +0,0 @@
package com.alibaba.otter.canal.client.adapter.hbase.support;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
import java.math.BigDecimal;
import java.math.BigInteger;
import java.util.Date;
/**
* Java类型
*
* @author machengyuan 2018-8-21 下午06:11:36
* @version 1.0.0
*/
public enum Type {
DEFAULT, STRING, INTEGER, LONG, SHORT, BOOLEAN, FLOAT, DOUBLE, BIGDECIMAL, DATE, BYTE, BYTES;
private static Logger logger = LoggerFactory.getLogger(Type.class);
public static Type getType(String type) {
if (type == null) {
return DEFAULT;
}
try {
return Type.valueOf(type);
} catch (Exception e) {
logger.error(e.getMessage(), e);
return DEFAULT;
}
}
public static Type getType(Class<?> javaType) {
if (javaType == null) {
return DEFAULT;
}
Type type;
if (Integer.class == javaType || int.class == javaType) {
type = INTEGER;
} else if (Long.class == javaType || long.class == javaType) {
type = LONG;
} else if (Byte.class == javaType || byte.class == javaType) {
type = BYTE;
} else if (Short.class == javaType || short.class == javaType) {
type = SHORT;
} else if (Float.class == javaType || float.class == javaType) {
type = FLOAT;
} else if (Double.class == javaType || double.class == javaType) {
type = DOUBLE;
} else if (Boolean.class == javaType || boolean.class == javaType) {
type = BOOLEAN;
} else if (Date.class == javaType) {
type = DATE;
} else if (byte[].class == javaType) {
type = BYTES;
} else if (String.class == javaType) {
type = STRING;
} else if (BigDecimal.class == javaType) {
type = BIGDECIMAL;
} else if (BigInteger.class == javaType) {
type = LONG;
} else {
type = DEFAULT;
}
return type;
}
}

View File

@@ -1,189 +0,0 @@
package com.alibaba.otter.canal.client.adapter.hbase.support;
import java.math.BigDecimal;
import java.math.BigInteger;
import java.sql.Time;
import java.sql.Timestamp;
import java.util.Date;
import org.apache.hadoop.hbase.util.Bytes;
/**
* Java类型转换工具类
*
* @author machengyuan 2018-8-21 下午06:12:34
* @version 1.0.0
*/
public class TypeUtil {
public static byte[] toBytes(Object obj) {
if (obj == null) {
return null;
}
byte[] bytes;
Class<?> clazz = obj.getClass();
if (clazz == String.class) {
bytes = Bytes.toBytes((String) obj);
} else if (clazz == Integer.class || clazz == int.class) {
bytes = Bytes.toBytes((Integer) obj);
} else if (clazz == Long.class || clazz == long.class) {
bytes = Bytes.toBytes((Long) obj);
} else if (clazz == Short.class || clazz == short.class) {
bytes = Bytes.toBytes((Short) obj);
} else if (clazz == Boolean.class || clazz == boolean.class) {
bytes = Bytes.toBytes((Boolean) obj);
} else if (clazz == Float.class || clazz == float.class) {
bytes = Bytes.toBytes((Float) obj);
} else if (clazz == Double.class || clazz == double.class) {
bytes = Bytes.toBytes((Double) obj);
} else if (clazz == Byte.class || clazz == byte.class) {
bytes = new byte[] { (byte) obj };
} else if (clazz == BigDecimal.class) {
bytes = Bytes.toBytes((BigDecimal) obj);
} else if (clazz == BigInteger.class) {
bytes = Bytes.toBytes(((BigInteger) obj).longValue());
} else if (clazz == Date.class) {
bytes = Bytes.toBytes(((Date) obj).getTime());
} else if (clazz == byte[].class) {
bytes = (byte[]) obj;
} else {
// 其余类型统一转换为string
bytes = Bytes.toBytes(obj.toString());
}
return bytes;
}
public static byte[] toBytes(Object v, Type type) {
if (v == null) {
return null;
}
byte[] b = null;
if (type == Type.DEFAULT) {
Type type1 = Type.getType(v.getClass());
if (type1 != null && type1 != Type.DEFAULT) {
b = toBytes(v, type1);
}
} else if (type == Type.STRING) {
b = Bytes.toBytes(v.toString());
} else if (type == Type.INTEGER) {
b = Bytes.toBytes(((Number) v).intValue());
} else if (type == Type.LONG) {
b = Bytes.toBytes(((Number) v).longValue());
} else if (type == Type.SHORT) {
b = Bytes.toBytes(((Number) v).shortValue());
} else if (type == Type.BYTE) {
b = Bytes.toBytes(((Number) v).byteValue());
} else if (type == Type.FLOAT) {
b = Bytes.toBytes(((Number) v).floatValue());
} else if (type == Type.DOUBLE) {
b = Bytes.toBytes(((Number) v).doubleValue());
} else if (type == Type.BOOLEAN) {
b = Bytes.toBytes(((Boolean) v));
} else if (type == Type.DATE) {
b = Bytes.toBytes(((Date) v).getTime());
} else if (type == Type.BYTES) {
b = (byte[]) v;
} else if (type == Type.BIGDECIMAL) {
if (v instanceof BigDecimal) {
b = Bytes.toBytes((BigDecimal) v);
} else {
b = Bytes.toBytes(new BigDecimal(v.toString()));
}
}
return b;
}
@SuppressWarnings("unchecked")
public static <T> T toObject(byte[] bytes, Class<T> clazz) {
if (bytes == null) {
return null;
}
Object res;
if (String.class == clazz) {
res = Bytes.toString(bytes);
} else if (Integer.class == clazz || int.class == clazz) {
res = Bytes.toInt(bytes);
} else if (Long.class == clazz || long.class == clazz) {
res = Bytes.toLong(bytes);
} else if (Short.class == clazz || short.class == clazz) {
res = Bytes.toShort(bytes);
} else if (Boolean.class == clazz || boolean.class == clazz) {
res = Bytes.toBoolean(bytes);
} else if (Float.class == clazz || float.class == clazz) {
res = Bytes.toFloat(bytes);
} else if (Double.class == clazz || double.class == clazz) {
res = Bytes.toDouble(bytes);
} else if (Byte.class == clazz || byte.class == clazz) {
res = bytes[0];
} else if (BigDecimal.class == clazz) {
res = Bytes.toBigDecimal(bytes);
} else if (BigInteger.class == clazz) {
res = Bytes.toLong(bytes);
} else if (java.sql.Date.class == clazz) {
long ts = Bytes.toLong(bytes);
res = new java.sql.Date(ts);
} else if (Time.class == clazz) {
long ts = Bytes.toLong(bytes);
res = new Time(ts);
} else if (Timestamp.class == clazz) {
long ts = Bytes.toLong(bytes);
res = new Timestamp(ts);
} else if (Date.class == clazz) {
long ts = Bytes.toLong(bytes);
res = new Date(ts);
} else {
throw new IllegalArgumentException("mismatch class type");
}
return (T) res;
}
@SuppressWarnings("unchecked")
public static <T> T toObject(byte[] bytes, Type type) {
if (bytes == null) {
return null;
}
Object res = null;
if (type == Type.STRING || type == Type.DEFAULT) {
res = Bytes.toString(bytes);
} else if (type == Type.INTEGER) {
if (bytes.length == Bytes.SIZEOF_INT) {
res = Bytes.toInt(bytes);
}
} else if (type == Type.LONG) {
if (bytes.length == Bytes.SIZEOF_LONG) {
res = Bytes.toLong(bytes);
}
} else if (type == Type.SHORT) {
if (bytes.length == Bytes.SIZEOF_SHORT) {
res = Bytes.toShort(bytes);
}
} else if (type == Type.BYTE) {
if (bytes.length == Bytes.SIZEOF_BYTE) {
res = bytes[0];
}
} else if (type == Type.FLOAT) {
if (bytes.length == Bytes.SIZEOF_FLOAT) {
res = Bytes.toFloat(bytes);
}
} else if (type == Type.DOUBLE) {
if (bytes.length == Bytes.SIZEOF_DOUBLE) {
res = Bytes.toDouble(bytes);
}
} else if (type == Type.BOOLEAN) {
if (bytes.length == Bytes.SIZEOF_BOOLEAN) {
res = Bytes.toBoolean(bytes);
}
} else if (type == Type.DATE) {
if (bytes.length == Bytes.SIZEOF_LONG) {
res = new Date(Bytes.toLong(bytes));
}
} else if (type == Type.BYTES) {
res = bytes;
} else if (type == Type.BIGDECIMAL) {
res = Bytes.toBigDecimal(bytes);
} else {
throw new IllegalArgumentException("mismatch class type");
}
return (T) res;
}
}

View File

@@ -1 +0,0 @@
hbase=com.alibaba.otter.canal.client.adapter.hbase.HbaseAdapter

View File

@@ -1,60 +0,0 @@
dataSourceKey: defaultDS
hbaseMapping:
mode: PHOENIX #NATIVE #STRING
destination: example
database: mytest # 数据库名
table: person2 # 数据库表名
hbaseTable: MYTEST.PERSON2 # HBase表名
family: CF # 默认统一Family名称
uppercaseQualifier: true # 字段名转大写, 默认为true
commitBatch: 3000 # 批量提交的大小
#rowKey: id,type # 复合字段rowKey不能和columns中的rowKey重复
columns:
# 数据库字段:HBase对应字段
id: ROWKEY$UNSIGNED_LONG
name: NAME
email: EMAIL
type: $DECIMAL
c_time: C_TIME$UNSIGNED_TIMESTAMP
birthday: BIRTHDAY$DATE
excludeColumns:
- lat # 忽略字段
# -- NATIVE类型
# $DEFAULT
# $STRING
# $INTEGER
# $LONG
# $SHORT
# $BOOLEAN
# $FLOAT
# $DOUBLE
# $BIGDECIMAL
# $DATE
# $BYTE
# $BYTES
# -- PHOENIX类型
# $DEFAULT 对应PHOENIX里的VARCHAR
# $UNSIGNED_INT 对应PHOENIX里的UNSIGNED_INT 4字节
# $UNSIGNED_LONG 对应PHOENIX里的UNSIGNED_LONG 8字节
# $UNSIGNED_TINYINT 对应PHOENIX里的UNSIGNED_TINYINT 1字节
# $UNSIGNED_SMALLINT 对应PHOENIX里的UNSIGNED_SMALLINT 2字节
# $UNSIGNED_FLOAT 对应PHOENIX里的UNSIGNED_FLOAT 4字节
# $UNSIGNED_DOUBLE 对应PHOENIX里的UNSIGNED_DOUBLE 8字节
# $INTEGER 对应PHOENIX里的INTEGER 4字节
# $BIGINT 对应PHOENIX里的BIGINT 8字节
# $TINYINT 对应PHOENIX里的TINYINT 1字节
# $SMALLINT 对应PHOENIX里的SMALLINT 2字节
# $FLOAT 对应PHOENIX里的FLOAT 4字节
# $DOUBLE 对应PHOENIX里的DOUBLE 8字节
# $BOOLEAN 对应PHOENIX里的BOOLEAN 1字节
# $TIME 对应PHOENIX里的TIME 8字节
# $DATE 对应PHOENIX里的DATE 8字节
# $TIMESTAMP 对应PHOENIX里的TIMESTAMP 12字节
# $UNSIGNED_TIME 对应PHOENIX里的UNSIGNED_TIME 8字节
# $UNSIGNED_DATE 对应PHOENIX里的UNSIGNED_DATE 8字节
# $UNSIGNED_TIMESTAMP 对应PHOENIX里的UNSIGNED_TIMESTAMP 12字节
# $VARCHAR 对应PHOENIX里的VARCHAR 动态长度
# $VARBINARY 对应PHOENIX里的VARBINARY 动态长度
# $DECIMAL 对应PHOENIX里的DECIMAL 动态长度

View File

@@ -1,207 +0,0 @@
<?xml version="1.0" encoding="UTF-8"?>
<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
<parent>
<artifactId>canal.client-adapter</artifactId>
<groupId>com.alibaba.otter</groupId>
<version>1.1.1</version>
</parent>
<modelVersion>4.0.0</modelVersion>
<groupId>com.alibaba.otter</groupId>
<artifactId>client-adapter.launcher</artifactId>
<packaging>jar</packaging>
<name>canal client adapter launcher module for otter ${project.version}</name>
<dependencyManagement>
<dependencies>
<dependency>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-dependencies</artifactId>
<version>2.0.1.RELEASE</version>
<type>pom</type>
<scope>import</scope>
</dependency>
</dependencies>
</dependencyManagement>
<dependencies>
<dependency>
<groupId>com.alibaba.otter</groupId>
<artifactId>client-adapter.common</artifactId>
<version>${project.version}</version>
</dependency>
<dependency>
<groupId>com.alibaba.otter</groupId>
<artifactId>canal.client</artifactId>
<version>1.1.1</version>
</dependency>
<dependency>
<groupId>org.yaml</groupId>
<artifactId>snakeyaml</artifactId>
<version>1.19</version>
</dependency>
<dependency>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-starter-web</artifactId>
</dependency>
<dependency>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-configuration-processor</artifactId>
<optional>true</optional>
</dependency>
<dependency>
<groupId>org.apache.curator</groupId>
<artifactId>curator-recipes</artifactId>
<version>2.10.0</version>
</dependency>
<!-- 单独引入rocketmq依赖 -->
<dependency>
<groupId>org.apache.rocketmq</groupId>
<artifactId>rocketmq-client</artifactId>
<version>4.3.0</version>
</dependency>
<!-- 单独引入kafka依赖 -->
<dependency>
<groupId>org.apache.kafka</groupId>
<artifactId>kafka-clients</artifactId>
<version>1.1.1</version>
</dependency>
<!-- outer adapter jar with dependencies-->
<dependency>
<groupId>com.alibaba.otter</groupId>
<artifactId>client-adapter.logger</artifactId>
<version>${project.version}</version>
<exclusions>
<exclusion>
<artifactId>*</artifactId>
<groupId>*</groupId>
</exclusion>
</exclusions>
<classifier>jar-with-dependencies</classifier>
<optional>true</optional>
</dependency>
<dependency>
<groupId>com.alibaba.otter</groupId>
<artifactId>client-adapter.hbase</artifactId>
<version>${project.version}</version>
<exclusions>
<exclusion>
<artifactId>*</artifactId>
<groupId>*</groupId>
</exclusion>
</exclusions>
<classifier>jar-with-dependencies</classifier>
<optional>true</optional>
</dependency>
</dependencies>
<build>
<finalName>canal-adapter-launcher</finalName>
<plugins>
<plugin>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-maven-plugin</artifactId>
<version>2.0.1.RELEASE</version>
<configuration>
<excludes>
<exclude>
<groupId>com.alibaba.otter</groupId>
<artifactId>client-adapter.hbase</artifactId>
</exclude>
<exclude>
<groupId>com.alibaba.otter</groupId>
<artifactId>client-adapter.logger</artifactId>
</exclude>
</excludes>
</configuration>
<executions>
<execution>
<goals>
<goal>repackage</goal>
</goals>
</execution>
</executions>
</plugin>
<plugin>
<artifactId>maven-antrun-plugin</artifactId>
<executions>
<execution>
<phase>package</phase>
<goals>
<goal>run</goal>
</goals>
<configuration>
<tasks>
<copy todir="${project.basedir}/target/config" overwrite="true">
<fileset dir="${project.basedir}/src/main/resources" erroronmissingdir="true">
<include name="*.yml" />
</fileset>
</copy>
<copy todir="${project.basedir}/target/config/hbase" overwrite="true">
<fileset dir="${project.basedir}/../hbase/src/main/resources/hbase" erroronmissingdir="true">
<include name="*.yml" />
</fileset>
</copy>
</tasks>
</configuration>
</execution>
</executions>
</plugin>
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-dependency-plugin</artifactId>
<version>2.10</version>
<executions>
<execution>
<id>copy-dependencies-to-canal-client-service</id>
<phase>package</phase>
<goals>
<goal>copy-dependencies</goal>
</goals>
<configuration>
<includeClassifiers>jar-with-dependencies</includeClassifiers>
<outputDirectory>${project.basedir}/target/lib</outputDirectory>
</configuration>
</execution>
</executions>
</plugin>
</plugins>
</build>
<profiles>
<profile>
<id>release</id>
<activation>
<property>
<name>env</name>
<value>release</value>
</property>
</activation>
<build>
<plugins>
<plugin>
<artifactId>maven-assembly-plugin</artifactId>
<configuration>
<appendAssemblyId>false</appendAssemblyId>
<descriptors>
<descriptor>${basedir}/src/main/assembly/release.xml</descriptor>
</descriptors>
<finalName>${project.artifactId}-${project.version}</finalName>
<outputDirectory>${project.basedir}/../../target</outputDirectory>
</configuration>
<executions>
<execution>
<id>make-assembly</id>
<phase>package</phase>
<goals>
<goal>single</goal>
</goals>
</execution>
</executions>
</plugin>
</plugins>
</build>
</profile>
</profiles>
</project>

View File

@@ -1,34 +0,0 @@
<assembly xmlns="http://maven.apache.org/plugins/maven-assembly-plugin/assembly/1.1.0"
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xsi:schemaLocation="http://maven.apache.org/plugins/maven-assembly-plugin/assembly/1.1.0 http://maven.apache.org/xsd/assembly-1.1.0.xsd">
<id>canal-adapter</id>
<formats>
<format>tar.gz</format>
</formats>
<fileSets>
<fileSet>
<directory>${project.build.directory}</directory>
<outputDirectory>/</outputDirectory>
<includes>
<include>*.jar</include>
</includes>
</fileSet>
<fileSet>
<directory>${project.build.directory}/lib</directory>
<outputDirectory>/lib/</outputDirectory>
<includes>
<include>*.jar</include>
</includes>
</fileSet>
<fileSet>
<directory>${project.build.directory}/config</directory>
<outputDirectory>/config/</outputDirectory>
<includes>
<include>**</include>
</includes>
</fileSet>
</fileSets>
<baseDirectory>/</baseDirectory>
</assembly>

View File

@@ -1,18 +0,0 @@
package com.alibaba.otter.canal.adapter.launcher;
import org.springframework.boot.autoconfigure.SpringBootApplication;
import org.springframework.boot.builder.SpringApplicationBuilder;
/**
* 启动入口
*
* @author rewerma @ 2018-10-20
* @version 1.0.0
*/
@SpringBootApplication
public class CanalAdapterApplication {
public static void main(String[] args) {
new SpringApplicationBuilder(CanalAdapterApplication.class).run(args);
}
}

View File

@@ -1,120 +0,0 @@
package com.alibaba.otter.canal.adapter.launcher.common;
import java.util.Map;
import java.util.concurrent.ConcurrentHashMap;
import java.util.concurrent.TimeUnit;
import java.util.concurrent.locks.ReentrantLock;
import javax.annotation.PostConstruct;
import javax.annotation.Resource;
import org.apache.curator.framework.CuratorFramework;
import org.apache.curator.framework.recipes.locks.InterProcessMutex;
import org.springframework.stereotype.Component;
import com.alibaba.otter.canal.adapter.launcher.config.CuratorClient;
/**
* Etl 同步锁
*
* @author rewerma @ 2018-10-20
* @version 1.0.0
*/
@Component
public class EtlLock {
private static final Map<String, ReentrantLock> LOCAL_LOCK = new ConcurrentHashMap<>();
private static final Map<String, InterProcessMutex> DISTRIBUTED_LOCK = new ConcurrentHashMap<>();
private static Mode mode = Mode.LOCAL;
@Resource
private CuratorClient curatorClient;
@PostConstruct
public void init() {
CuratorFramework curator = curatorClient.getCurator();
if (curator != null) {
mode = Mode.DISTRIBUTED;
} else {
mode = Mode.LOCAL;
}
}
private ReentrantLock getLock(String key) {
ReentrantLock lock = LOCAL_LOCK.get(key);
if (lock == null) {
synchronized (EtlLock.class) {
lock = LOCAL_LOCK.get(key);
if (lock == null) {
lock = new ReentrantLock();
LOCAL_LOCK.put(key, lock);
}
}
}
return lock;
}
private InterProcessMutex getRemoteLock(String key) {
InterProcessMutex lock = DISTRIBUTED_LOCK.get(key);
if (lock == null) {
synchronized (EtlLock.class) {
lock = DISTRIBUTED_LOCK.get(key);
if (lock == null) {
lock = new InterProcessMutex(curatorClient.getCurator(), key);
DISTRIBUTED_LOCK.put(key, lock);
}
}
}
return lock;
}
public void lock(String key) throws Exception {
if (mode == Mode.LOCAL) {
getLock(key).lock();
} else {
InterProcessMutex lock = getRemoteLock(key);
lock.acquire();
}
}
public boolean tryLock(String key, long timeout, TimeUnit unit) {
try {
if (mode == Mode.LOCAL) {
return getLock(key).tryLock(timeout, unit);
} else {
InterProcessMutex lock = getRemoteLock(key);
return lock.acquire(timeout, unit);
}
} catch (Exception e) {
return false;
}
}
public boolean tryLock(String key) {
try {
if (mode == Mode.LOCAL) {
return getLock(key).tryLock();
} else {
InterProcessMutex lock = getRemoteLock(key);
return lock.acquire(500, TimeUnit.MILLISECONDS);
}
} catch (Exception e) {
return false;
}
}
public void unlock(String key) {
if (mode == Mode.LOCAL) {
getLock(key).unlock();
} else {
InterProcessMutex lock = getRemoteLock(key);
try {
lock.release();
} catch (Exception e) {
// ignore
}
}
}
}

View File

@@ -1,6 +0,0 @@
package com.alibaba.otter.canal.adapter.launcher.common;
public enum Mode {
LOCAL, // 本地模式
DISTRIBUTED // 分布式
}

View File

@@ -1,213 +0,0 @@
package com.alibaba.otter.canal.adapter.launcher.common;
import java.nio.charset.StandardCharsets;
import java.util.Map;
import java.util.concurrent.ConcurrentHashMap;
import java.util.concurrent.TimeUnit;
import java.util.concurrent.TimeoutException;
import javax.annotation.PostConstruct;
import javax.annotation.Resource;
import org.apache.curator.framework.CuratorFramework;
import org.apache.curator.framework.recipes.cache.NodeCache;
import org.apache.zookeeper.CreateMode;
import org.apache.zookeeper.data.Stat;
import org.springframework.stereotype.Component;
import com.alibaba.otter.canal.adapter.launcher.config.AdapterCanalConfig;
import com.alibaba.otter.canal.adapter.launcher.config.CuratorClient;
import com.alibaba.otter.canal.common.utils.BooleanMutex;
/**
* 同步开关
*
* @author rewerma @ 2018-10-20
* @version 1.0.0
*/
@Component
public class SyncSwitch {
private static final String SYN_SWITCH_ZK_NODE = "/sync-switch/";
private static final Map<String, BooleanMutex> LOCAL_LOCK = new ConcurrentHashMap<>();
private static final Map<String, BooleanMutex> DISTRIBUTED_LOCK = new ConcurrentHashMap<>();
private static Mode mode = Mode.LOCAL;
@Resource
private AdapterCanalConfig adapterCanalConfig;
@Resource
private CuratorClient curatorClient;
@PostConstruct
public void init() {
CuratorFramework curator = curatorClient.getCurator();
if (curator != null) {
mode = Mode.DISTRIBUTED;
DISTRIBUTED_LOCK.clear();
for (String destination : adapterCanalConfig.DESTINATIONS) {
// 对应每个destination注册锁
BooleanMutex mutex = new BooleanMutex(true);
initMutex(curator, destination, mutex);
DISTRIBUTED_LOCK.put(destination, mutex);
startListen(destination, mutex);
}
} else {
mode = Mode.LOCAL;
LOCAL_LOCK.clear();
for (String destination : adapterCanalConfig.DESTINATIONS) {
// 对应每个destination注册锁
LOCAL_LOCK.put(destination, new BooleanMutex(true));
}
}
}
private synchronized void startListen(String destination, BooleanMutex mutex) {
try {
String path = SYN_SWITCH_ZK_NODE + destination;
CuratorFramework curator = curatorClient.getCurator();
final NodeCache nodeCache = new NodeCache(curator, path);
nodeCache.start();
nodeCache.getListenable().addListener(() -> initMutex(curator, destination, mutex));
} catch (Exception e) {
throw new RuntimeException(e.getMessage());
}
}
private synchronized void initMutex(CuratorFramework curator, String destination, BooleanMutex mutex) {
try {
String path = SYN_SWITCH_ZK_NODE + destination;
Stat stat = curator.checkExists().forPath(path);
if (stat == null) {
if (!mutex.state()) {
mutex.set(true);
}
} else {
String data = new String(curator.getData().forPath(path), StandardCharsets.UTF_8);
if ("on".equals(data)) {
if (!mutex.state()) {
mutex.set(true);
}
} else {
if (mutex.state()) {
mutex.set(false);
}
}
}
} catch (Exception e) {
throw new RuntimeException(e.getMessage());
}
}
public synchronized void off(String destination) {
if (mode == Mode.LOCAL) {
BooleanMutex mutex = LOCAL_LOCK.get(destination);
if (mutex != null && mutex.state()) {
mutex.set(false);
}
} else {
try {
String path = SYN_SWITCH_ZK_NODE + destination;
try {
curatorClient.getCurator()
.create()
.creatingParentContainersIfNeeded()
.withMode(CreateMode.PERSISTENT)
.forPath(path, "off".getBytes(StandardCharsets.UTF_8));
} catch (Exception e) {
curatorClient.getCurator().setData().forPath(path, "off".getBytes(StandardCharsets.UTF_8));
}
} catch (Exception e) {
throw new RuntimeException(e);
}
}
}
public synchronized void on(String destination) {
if (mode == Mode.LOCAL) {
BooleanMutex mutex = LOCAL_LOCK.get(destination);
if (mutex != null && !mutex.state()) {
mutex.set(true);
}
} else {
try {
String path = SYN_SWITCH_ZK_NODE + destination;
try {
curatorClient.getCurator()
.create()
.creatingParentContainersIfNeeded()
.withMode(CreateMode.PERSISTENT)
.forPath(path, "on".getBytes(StandardCharsets.UTF_8));
} catch (Exception e) {
curatorClient.getCurator().setData().forPath(path, "on".getBytes(StandardCharsets.UTF_8));
}
} catch (Exception e) {
throw new RuntimeException(e);
}
}
}
public synchronized void release(String destination) {
if (mode == Mode.LOCAL) {
BooleanMutex mutex = LOCAL_LOCK.get(destination);
if (mutex != null && !mutex.state()) {
mutex.set(true);
}
}
if (mode == Mode.DISTRIBUTED) {
BooleanMutex mutex = DISTRIBUTED_LOCK.get(destination);
if (mutex != null && !mutex.state()) {
mutex.set(true);
}
}
}
public Boolean status(String destination) {
if (mode == Mode.LOCAL) {
BooleanMutex mutex = LOCAL_LOCK.get(destination);
if (mutex != null) {
return mutex.state();
} else {
return null;
}
} else {
BooleanMutex mutex = DISTRIBUTED_LOCK.get(destination);
if (mutex != null) {
return mutex.state();
} else {
return null;
}
}
}
public void get(String destination) throws InterruptedException {
if (mode == Mode.LOCAL) {
BooleanMutex mutex = LOCAL_LOCK.get(destination);
if (mutex != null) {
mutex.get();
}
} else {
BooleanMutex mutex = DISTRIBUTED_LOCK.get(destination);
if (mutex != null) {
mutex.get();
}
}
}
public void get(String destination, long timeout, TimeUnit unit) throws InterruptedException, TimeoutException {
if (mode == Mode.LOCAL) {
BooleanMutex mutex = LOCAL_LOCK.get(destination);
if (mutex != null) {
mutex.get(timeout, unit);
}
} else {
BooleanMutex mutex = DISTRIBUTED_LOCK.get(destination);
if (mutex != null) {
mutex.get(timeout, unit);
}
}
}
}

View File

@@ -1,51 +0,0 @@
package com.alibaba.otter.canal.adapter.launcher.config;
import java.util.LinkedHashSet;
import java.util.List;
import java.util.Set;
import org.springframework.boot.context.properties.ConfigurationProperties;
import org.springframework.stereotype.Component;
import com.alibaba.otter.canal.client.adapter.support.CanalClientConfig;
/**
* canal 的相关配置类
*
* @author rewerma @ 2018-10-20
* @version 1.0.0
*/
@Component
@ConfigurationProperties(prefix = "canal.conf")
public class AdapterCanalConfig extends CanalClientConfig {
public final Set<String> DESTINATIONS = new LinkedHashSet<>();
@Override
public void setCanalInstances(List<CanalInstance> canalInstances) {
super.setCanalInstances(canalInstances);
if (canalInstances != null) {
synchronized (DESTINATIONS) {
DESTINATIONS.clear();
for (CanalInstance canalInstance : canalInstances) {
DESTINATIONS.add(canalInstance.getInstance());
}
}
}
}
@Override
public void setMqTopics(List<MQTopic> mqTopics) {
super.setMqTopics(mqTopics);
if (mqTopics != null) {
synchronized (DESTINATIONS) {
DESTINATIONS.clear();
for (MQTopic mqTopic : mqTopics) {
DESTINATIONS.add(mqTopic.getTopic());
}
}
}
}
}

View File

@@ -1,86 +0,0 @@
package com.alibaba.otter.canal.adapter.launcher.config;
import java.sql.SQLException;
import java.util.List;
import java.util.Map;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
import org.springframework.boot.context.properties.ConfigurationProperties;
import org.springframework.stereotype.Component;
import com.alibaba.druid.pool.DruidDataSource;
import com.alibaba.otter.canal.client.adapter.support.AdapterConfigs;
import com.alibaba.otter.canal.client.adapter.support.DatasourceConfig;
/**
* 适配器数据源及配置文件列表配置类
*
* @author rewerma @ 2018-10-20
* @version 1.0.0
*/
@Component
@ConfigurationProperties(prefix = "adapter.conf")
public class AdapterConfig {
private static Logger logger = LoggerFactory.getLogger(AdapterConfig.class);
private Map<String, DatasourceConfig> datasourceConfigs;
private List<String> adapterConfigs;
public List<String> getAdapterConfigs() {
return adapterConfigs;
}
public Map<String, DatasourceConfig> getDatasourceConfigs() {
return datasourceConfigs;
}
public void setDatasourceConfigs(Map<String, DatasourceConfig> datasourceConfigs) {
this.datasourceConfigs = datasourceConfigs;
if (datasourceConfigs != null) {
for (Map.Entry<String, DatasourceConfig> entry : datasourceConfigs.entrySet()) {
DatasourceConfig datasourceConfig = entry.getValue();
// 加载数据源连接池
DruidDataSource ds = new DruidDataSource();
ds.setDriverClassName(datasourceConfig.getDriver());
ds.setUrl(datasourceConfig.getUrl());
ds.setUsername(datasourceConfig.getUsername());
ds.setPassword(datasourceConfig.getPassword());
ds.setInitialSize(1);
ds.setMinIdle(1);
ds.setMaxActive(datasourceConfig.getMaxActive());
ds.setMaxWait(60000);
ds.setTimeBetweenEvictionRunsMillis(60000);
ds.setMinEvictableIdleTimeMillis(300000);
ds.setPoolPreparedStatements(false);
ds.setMaxPoolPreparedStatementPerConnectionSize(20);
ds.setValidationQuery("select 1");
try {
ds.init();
} catch (SQLException e) {
logger.error("ERROR ## failed to initial datasource: " + datasourceConfig.getUrl(), e);
}
DatasourceConfig.DATA_SOURCES.put(entry.getKey(), ds);
}
}
}
public void setAdapterConfigs(List<String> adapterConfigs) {
this.adapterConfigs = adapterConfigs;
if (adapterConfigs != null) {
AdapterConfigs.clear();
for (String adapterConfig : adapterConfigs) {
int idx = adapterConfig.indexOf("/");
if (idx > -1) {
String type = adapterConfig.substring(0, idx);
String ymlFile = adapterConfig.substring(idx + 1);
AdapterConfigs.put(type, ymlFile);
}
}
}
}
}

View File

@@ -1,42 +0,0 @@
package com.alibaba.otter.canal.adapter.launcher.config;
import javax.annotation.PostConstruct;
import javax.annotation.Resource;
import org.apache.curator.framework.CuratorFramework;
import org.apache.curator.framework.CuratorFrameworkFactory;
import org.apache.curator.retry.ExponentialBackoffRetry;
import org.springframework.stereotype.Component;
/**
* curator 配置类
*
* @author rewerma @ 2018-10-20
* @version 1.0.0
*/
@Component
public class CuratorClient {
@Resource
private AdapterCanalConfig adapterCanalConfig;
private CuratorFramework curator = null;
@PostConstruct
public void init() {
if (adapterCanalConfig.getZookeeperHosts() != null) {
curator = CuratorFrameworkFactory.builder()
.connectString(adapterCanalConfig.getZookeeperHosts())
.retryPolicy(new ExponentialBackoffRetry(1000, 3))
.sessionTimeoutMs(6000)
.connectionTimeoutMs(3000)
.namespace("canal-adapter")
.build();
curator.start();
}
}
public CuratorFramework getCurator() {
return curator;
}
}

View File

@@ -1,34 +0,0 @@
package com.alibaba.otter.canal.adapter.launcher.config;
import org.springframework.beans.BeansException;
import org.springframework.context.ApplicationContext;
import org.springframework.context.ApplicationContextAware;
import org.springframework.stereotype.Component;
/**
* spring util配置类
*
* @author rewerma @ 2018-10-20
* @version 1.0.0
*/
@Component
public class SpringContext implements ApplicationContextAware {
private static ApplicationContext context;
/*
* 注入ApplicationContext
*/
public void setApplicationContext(final ApplicationContext context) throws BeansException {
// 在加载Spring时自动获得context
SpringContext.context = context;
}
public static Object getBean(final String beanName) {
return SpringContext.context.getBean(beanName);
}
public static Object getBean(final Class<?> clz) {
return context.getBean(clz);
}
}

View File

@@ -1,158 +0,0 @@
package com.alibaba.otter.canal.adapter.launcher.loader;
import java.util.ArrayList;
import java.util.List;
import java.util.concurrent.ExecutionException;
import java.util.concurrent.ExecutorService;
import java.util.concurrent.Executors;
import java.util.concurrent.Future;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
import com.alibaba.otter.canal.adapter.launcher.common.SyncSwitch;
import com.alibaba.otter.canal.adapter.launcher.config.SpringContext;
import com.alibaba.otter.canal.client.adapter.OuterAdapter;
import com.alibaba.otter.canal.client.adapter.support.Dml;
import com.alibaba.otter.canal.client.adapter.support.MessageUtil;
import com.alibaba.otter.canal.protocol.FlatMessage;
import com.alibaba.otter.canal.protocol.Message;
/**
* 适配器工作线程抽象类
*
* @author rewerma 2018-8-19 下午11:30:49
* @version 1.0.0
*/
public abstract class AbstractCanalAdapterWorker {
protected final Logger logger = LoggerFactory.getLogger(this.getClass());
protected String canalDestination; // canal实例
protected List<List<OuterAdapter>> canalOuterAdapters; // 外部适配器
protected ExecutorService groupInnerExecutorService; // 组内工作线程池
protected volatile boolean running = false; // 是否运行中
protected Thread thread = null;
protected Thread.UncaughtExceptionHandler handler = (t, e) -> logger.error("parse events has an error", e);
protected SyncSwitch syncSwitch;
public AbstractCanalAdapterWorker(List<List<OuterAdapter>> canalOuterAdapters){
this.canalOuterAdapters = canalOuterAdapters;
this.groupInnerExecutorService = Executors.newFixedThreadPool(canalOuterAdapters.size());
syncSwitch = (SyncSwitch) SpringContext.getBean(SyncSwitch.class);
}
protected void writeOut(final Message message) {
List<Future<Boolean>> futures = new ArrayList<>();
// 组间适配器并行运行
canalOuterAdapters.forEach(outerAdapters -> {
final List<OuterAdapter> adapters = outerAdapters;
futures.add(groupInnerExecutorService.submit(() -> {
try {
// 组内适配器穿行运行,尽量不要配置组内适配器
adapters.forEach(adapter -> {
long begin = System.currentTimeMillis();
MessageUtil.parse4Dml(canalDestination, message, adapter::sync);
if (logger.isDebugEnabled()) {
logger.debug("{} elapsed time: {}",
adapter.getClass().getName(),
(System.currentTimeMillis() - begin));
}
});
return true;
} catch (Exception e) {
return false;
}
}));
// 等待所有适配器写入完成
// 由于是组间并发操作,所以将阻塞直到耗时最久的工作组操作完成
futures.forEach(future -> {
try {
if (!future.get()) {
logger.error("Outer adapter write failed");
}
} catch (InterruptedException | ExecutionException e) {
// ignore
}
});
});
}
protected void writeOut(final FlatMessage flatMessage) {
List<Future<Boolean>> futures = new ArrayList<>();
// 组间适配器并行运行
canalOuterAdapters.forEach(outerAdapters -> {
futures.add(groupInnerExecutorService.submit(() -> {
try {
// 组内适配器穿行运行,尽量不要配置组内适配器
outerAdapters.forEach(adapter -> {
long begin = System.currentTimeMillis();
Dml dml = MessageUtil.flatMessage2Dml(canalDestination, flatMessage);
adapter.sync(dml);
if (logger.isDebugEnabled()) {
logger.debug("{} elapsed time: {}",
adapter.getClass().getName(),
(System.currentTimeMillis() - begin));
}
});
return true;
} catch (Exception e) {
return false;
}
}));
// 等待所有适配器写入完成
// 由于是组间并发操作,所以将阻塞直到耗时最久的工作组操作完成
futures.forEach(future -> {
try {
if (!future.get()) {
logger.error("Outer adapter write failed");
}
} catch (InterruptedException | ExecutionException e) {
// ignore
}
});
});
}
public void start() {
if (!running) {
thread = new Thread(this::process);
thread.setUncaughtExceptionHandler(handler);
thread.start();
running = true;
}
}
protected abstract void process();
public void stop() {
try {
if (!running) {
return;
}
running = false;
syncSwitch.release(canalDestination);
logger.info("destination {} is waiting for adapters' worker thread die!", canalDestination);
if (thread != null) {
try {
thread.join();
} catch (InterruptedException e) {
// ignore
}
}
groupInnerExecutorService.shutdown();
logger.info("destination {} adapters worker thread dead!", canalDestination);
canalOuterAdapters.forEach(outerAdapters -> outerAdapters.forEach(OuterAdapter::destroy));
logger.info("destination {} all adapters destroyed!", canalDestination);
} catch (Exception e) {
logger.error(e.getMessage(), e);
}
}
}

View File

@@ -1,93 +0,0 @@
package com.alibaba.otter.canal.adapter.launcher.loader;
import java.util.List;
import java.util.concurrent.TimeUnit;
import org.apache.kafka.clients.consumer.CommitFailedException;
import org.apache.kafka.common.errors.WakeupException;
import com.alibaba.otter.canal.client.adapter.OuterAdapter;
import com.alibaba.otter.canal.client.kafka.KafkaCanalConnector;
import com.alibaba.otter.canal.client.kafka.KafkaCanalConnectors;
import com.alibaba.otter.canal.protocol.FlatMessage;
import com.alibaba.otter.canal.protocol.Message;
/**
* kafka对应的client适配器工作线程
*
* @author rewerma 2018-8-19 下午11:30:49
* @version 1.0.0
*/
public class CanalAdapterKafkaWorker extends AbstractCanalAdapterWorker {
private KafkaCanalConnector connector;
private String topic;
private boolean flatMessage;
public CanalAdapterKafkaWorker(String bootstrapServers, String topic, String groupId,
List<List<OuterAdapter>> canalOuterAdapters, boolean flatMessage){
super(canalOuterAdapters);
this.topic = topic;
this.canalDestination = topic;
this.flatMessage = flatMessage;
this.connector = KafkaCanalConnectors.newKafkaConnector(bootstrapServers, topic, null, groupId, flatMessage);
// connector.setSessionTimeout(1L, TimeUnit.MINUTES);
}
@Override
protected void process() {
while (!running)
;
while (running) {
try {
syncSwitch.get(canalDestination);
logger.info("=============> Start to connect topic: {} <=============", this.topic);
connector.connect();
logger.info("=============> Start to subscribe topic: {} <=============", this.topic);
connector.subscribe();
logger.info("=============> Subscribe topic: {} succeed <=============", this.topic);
while (running) {
try {
Boolean status = syncSwitch.status(canalDestination);
if (status != null && !status) {
connector.disconnect();
break;
}
List<?> messages;
if (!flatMessage) {
messages = connector.getListWithoutAck(100L, TimeUnit.MILLISECONDS);
} else {
messages = connector.getFlatListWithoutAck(100L, TimeUnit.MILLISECONDS);
}
if (messages != null) {
for (final Object message : messages) {
if (message instanceof FlatMessage) {
writeOut((FlatMessage) message);
} else {
writeOut((Message) message);
}
}
}
connector.ack();
} catch (CommitFailedException e) {
logger.warn(e.getMessage());
} catch (Throwable e) {
logger.error(e.getMessage(), e);
TimeUnit.SECONDS.sleep(1L);
}
}
} catch (Exception e) {
logger.error(e.getMessage(), e);
}
}
try {
connector.unsubscribe();
} catch (WakeupException e) {
// No-op. Continue process
}
connector.disconnect();
logger.info("=============> Disconnect topic: {} <=============", this.topic);
}
}

View File

@@ -1,152 +0,0 @@
package com.alibaba.otter.canal.adapter.launcher.loader;
import java.net.InetSocketAddress;
import java.net.SocketAddress;
import java.util.ArrayList;
import java.util.HashMap;
import java.util.List;
import java.util.Map;
import java.util.concurrent.ExecutorService;
import java.util.concurrent.Executors;
import org.apache.commons.lang.StringUtils;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
import com.alibaba.otter.canal.client.adapter.OuterAdapter;
import com.alibaba.otter.canal.client.adapter.support.CanalClientConfig;
import com.alibaba.otter.canal.client.adapter.support.ExtensionLoader;
import com.alibaba.otter.canal.client.adapter.support.OuterAdapterConfig;
/**
* 外部适配器的加载器
*
* @version 1.0.0
*/
public class CanalAdapterLoader {
private static final Logger logger = LoggerFactory.getLogger(CanalAdapterLoader.class);
private CanalClientConfig canalClientConfig;
private Map<String, CanalAdapterWorker> canalWorkers = new HashMap<>();
private Map<String, AbstractCanalAdapterWorker> canalMQWorker = new HashMap<>();
private ExtensionLoader<OuterAdapter> loader;
public CanalAdapterLoader(CanalClientConfig canalClientConfig){
this.canalClientConfig = canalClientConfig;
}
/**
* 初始化canal-client
*/
public void init() {
loader = ExtensionLoader.getExtensionLoader(OuterAdapter.class);
String canalServerHost = this.canalClientConfig.getCanalServerHost();
SocketAddress sa = null;
if (canalServerHost != null) {
String[] ipPort = canalServerHost.split(":");
sa = new InetSocketAddress(ipPort[0], Integer.parseInt(ipPort[1]));
}
String zkHosts = this.canalClientConfig.getZookeeperHosts();
// 初始化canal-client的适配器
if (canalClientConfig.getCanalInstances() != null) {
for (CanalClientConfig.CanalInstance instance : canalClientConfig.getCanalInstances()) {
List<List<OuterAdapter>> canalOuterAdapterGroups = new ArrayList<>();
for (CanalClientConfig.Group connectorGroup : instance.getGroups()) {
List<OuterAdapter> canalOutConnectors = new ArrayList<>();
for (OuterAdapterConfig c : connectorGroup.getOutAdapters()) {
loadConnector(c, canalOutConnectors);
}
canalOuterAdapterGroups.add(canalOutConnectors);
}
CanalAdapterWorker worker;
if (sa != null) {
worker = new CanalAdapterWorker(instance.getInstance(), sa, canalOuterAdapterGroups);
} else if (zkHosts != null) {
worker = new CanalAdapterWorker(instance.getInstance(), zkHosts, canalOuterAdapterGroups);
} else {
throw new RuntimeException("No canal server connector found");
}
canalWorkers.put(instance.getInstance(), worker);
worker.start();
logger.info("Start adapter for canal instance: {} succeed", instance.getInstance());
}
}
// 初始化canal-client-mq的适配器
if (canalClientConfig.getMqTopics() != null) {
for (CanalClientConfig.MQTopic topic : canalClientConfig.getMqTopics()) {
for (CanalClientConfig.MQGroup group : topic.getGroups()) {
List<List<OuterAdapter>> canalOuterAdapterGroups = new ArrayList<>();
List<OuterAdapter> canalOuterAdapters = new ArrayList<>();
for (OuterAdapterConfig config : group.getOutAdapters()) {
loadConnector(config, canalOuterAdapters);
}
canalOuterAdapterGroups.add(canalOuterAdapters);
if (StringUtils.isBlank(topic.getMqMode()) || "rocketmq".equalsIgnoreCase(topic.getMqMode())) {
CanalAdapterRocketMQWorker rocketMQWorker = new CanalAdapterRocketMQWorker(canalClientConfig.getBootstrapServers(),
topic.getTopic(),
group.getGroupId(),
canalOuterAdapterGroups,
canalClientConfig.getFlatMessage());
canalMQWorker.put(topic.getTopic() + "-rocketmq-" + group.getGroupId(), rocketMQWorker);
rocketMQWorker.start();
} else if ("kafka".equalsIgnoreCase(topic.getMqMode())) {
CanalAdapterKafkaWorker canalKafkaWorker = new CanalAdapterKafkaWorker(canalClientConfig.getBootstrapServers(),
topic.getTopic(),
group.getGroupId(),
canalOuterAdapterGroups,
canalClientConfig.getFlatMessage());
canalMQWorker.put(topic.getTopic() + "-kafka-" + group.getGroupId(), canalKafkaWorker);
canalKafkaWorker.start();
}
logger.info("Start adapter for canal-client mq topic: {} succeed",
topic.getTopic() + "-" + group.getGroupId());
}
}
}
}
private void loadConnector(OuterAdapterConfig config, List<OuterAdapter> canalOutConnectors) {
try {
OuterAdapter adapter = loader.getExtension(config.getName());
ClassLoader cl = Thread.currentThread().getContextClassLoader();
// 替换ClassLoader
Thread.currentThread().setContextClassLoader(adapter.getClass().getClassLoader());
adapter.init(config);
Thread.currentThread().setContextClassLoader(cl);
canalOutConnectors.add(adapter);
logger.info("Load canal adapter: {} succeed", config.getName());
} catch (Exception e) {
logger.error("Load canal adapter: {} failed", config.getName(), e);
}
}
/**
* 销毁所有适配器 为防止canal实例太多造成销毁阻塞, 并行销毁
*/
public void destroy() {
if (canalWorkers.size() > 0) {
ExecutorService stopExecutorService = Executors.newFixedThreadPool(canalWorkers.size());
for (CanalAdapterWorker canalAdapterWorker : canalWorkers.values()) {
stopExecutorService.submit(canalAdapterWorker::stop);
}
stopExecutorService.shutdown();
}
if (canalMQWorker.size() > 0) {
ExecutorService stopMQWorkerService = Executors.newFixedThreadPool(canalMQWorker.size());
for (AbstractCanalAdapterWorker canalAdapterMQWorker : canalMQWorker.values()) {
stopMQWorkerService.submit(canalAdapterMQWorker::stop);
}
stopMQWorkerService.shutdown();
}
logger.info("All canal adapters destroyed");
}
}

View File

@@ -1,89 +0,0 @@
package com.alibaba.otter.canal.adapter.launcher.loader;
import java.util.List;
import java.util.concurrent.TimeUnit;
import org.apache.kafka.common.errors.WakeupException;
import com.alibaba.otter.canal.client.adapter.OuterAdapter;
import com.alibaba.otter.canal.client.rocketmq.RocketMQCanalConnector;
import com.alibaba.otter.canal.client.rocketmq.RocketMQCanalConnectors;
import com.alibaba.otter.canal.protocol.FlatMessage;
import com.alibaba.otter.canal.protocol.Message;
/**
* rocketmq对应的client适配器工作线程
*
* @version 1.0.0
*/
public class CanalAdapterRocketMQWorker extends AbstractCanalAdapterWorker {
private RocketMQCanalConnector connector;
private String topic;
private boolean flatMessage;
public CanalAdapterRocketMQWorker(String nameServers, String topic, String groupId,
List<List<OuterAdapter>> canalOuterAdapters, boolean flatMessage){
super(canalOuterAdapters);
this.topic = topic;
this.flatMessage = flatMessage;
this.canalDestination = topic;
this.connector = RocketMQCanalConnectors.newRocketMQConnector(nameServers, topic, groupId, flatMessage);
logger.info("RocketMQ consumer config topic:{}, nameServer:{}, groupId:{}", topic, nameServers, groupId);
}
@Override
protected void process() {
while (!running)
;
while (running) {
try {
syncSwitch.get(canalDestination);
logger.info("=============> Start to connect topic: {} <=============", this.topic);
connector.connect();
logger.info("=============> Start to subscribe topic: {}<=============", this.topic);
connector.subscribe();
logger.info("=============> Subscribe topic: {} succeed<=============", this.topic);
while (running) {
try {
Boolean status = syncSwitch.status(canalDestination);
if (status != null && !status) {
connector.disconnect();
break;
}
List<?> messages;
if (!flatMessage) {
messages = connector.getListWithoutAck(100L, TimeUnit.MILLISECONDS);
} else {
messages = connector.getFlatListWithoutAck(100L, TimeUnit.MILLISECONDS);
}
if (messages != null) {
for (final Object message : messages) {
if (message instanceof FlatMessage) {
writeOut((FlatMessage) message);
} else {
writeOut((Message) message);
}
}
}
connector.ack();
} catch (Throwable e) {
logger.error(e.getMessage(), e);
TimeUnit.SECONDS.sleep(1L);
}
}
} catch (Exception e) {
logger.error(e.getMessage(), e);
}
}
try {
connector.unsubscribe();
} catch (WakeupException e) {
// No-op. Continue process
}
connector.disconnect();
logger.info("=============> Disconnect topic: {} <=============", this.topic);
}
}

View File

@@ -1,77 +0,0 @@
package com.alibaba.otter.canal.adapter.launcher.loader;
import javax.annotation.PostConstruct;
import javax.annotation.PreDestroy;
import javax.annotation.Resource;
import com.alibaba.otter.canal.adapter.launcher.common.SyncSwitch;
import com.alibaba.otter.canal.adapter.launcher.config.SpringContext;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
import org.springframework.stereotype.Component;
import com.alibaba.druid.pool.DruidDataSource;
import com.alibaba.otter.canal.adapter.launcher.config.AdapterCanalConfig;
import com.alibaba.otter.canal.adapter.launcher.config.AdapterConfig;
import com.alibaba.otter.canal.client.adapter.support.DatasourceConfig;
/**
* 适配器启动业务类
*
* @author rewerma @ 2018-10-20
* @version 1.0.0
*/
@Component
public class CanalAdapterService {
private static final Logger logger = LoggerFactory.getLogger(CanalAdapterService.class);
private static CanalAdapterLoader adapterLoader;
@Resource
private AdapterCanalConfig adapterCanalConfig;
// 注入bean保证优先注册
@Resource
private AdapterConfig adapterConfig;
@Resource
private SpringContext springContext;
@Resource
private SyncSwitch syncSwitch;
@PostConstruct
public void init() {
if (adapterLoader == null) {
try {
logger.info("## start the canal client adapters.");
adapterLoader = new CanalAdapterLoader(adapterCanalConfig);
adapterLoader.init();
logger.info("## the canal client adapters are running now ......");
} catch (Throwable e) {
logger.error("## something goes wrong when starting up the canal client adapters:", e);
System.exit(0);
}
}
}
@PreDestroy
public void destroy() {
try {
logger.info("## stop the canal client adapters");
if (adapterLoader != null) {
adapterLoader.destroy();
}
for (DruidDataSource druidDataSource : DatasourceConfig.DATA_SOURCES.values()) {
try {
druidDataSource.close();
} catch (Exception e) {
logger.error(e.getMessage(), e);
}
}
} catch (Throwable e) {
logger.warn("## something goes wrong when stopping canal client adapters:", e);
} finally {
logger.info("## canal client adapters are down.");
}
}
}

View File

@@ -1,162 +0,0 @@
package com.alibaba.otter.canal.adapter.launcher.loader;
import java.net.SocketAddress;
import java.util.List;
import java.util.concurrent.TimeUnit;
import java.util.concurrent.TimeoutException;
import com.alibaba.otter.canal.client.CanalConnector;
import com.alibaba.otter.canal.client.CanalConnectors;
import com.alibaba.otter.canal.client.adapter.OuterAdapter;
import com.alibaba.otter.canal.client.impl.ClusterCanalConnector;
import com.alibaba.otter.canal.client.impl.SimpleCanalConnector;
import com.alibaba.otter.canal.protocol.Message;
/**
* 原生canal-server对应的client适配器工作线程
*
* @author rewrema 2018-8-19 下午11:30:49
* @version 1.0.0
*/
public class CanalAdapterWorker extends AbstractCanalAdapterWorker {
private static final int BATCH_SIZE = 50;
private static final int SO_TIMEOUT = 0;
private CanalConnector connector;
/**
* 单台client适配器worker的构造方法
*
* @param canalDestination canal实例名
* @param address canal-server地址
* @param canalOuterAdapters 外部适配器组
*/
public CanalAdapterWorker(String canalDestination, SocketAddress address,
List<List<OuterAdapter>> canalOuterAdapters){
super(canalOuterAdapters);
this.canalDestination = canalDestination;
connector = CanalConnectors.newSingleConnector(address, canalDestination, "", "");
}
/**
* HA模式下client适配器worker的构造方法
*
* @param canalDestination canal实例名
* @param zookeeperHosts zookeeper地址
* @param canalOuterAdapters 外部适配器组
*/
public CanalAdapterWorker(String canalDestination, String zookeeperHosts,
List<List<OuterAdapter>> canalOuterAdapters){
super(canalOuterAdapters);
this.canalDestination = canalDestination;
connector = CanalConnectors.newClusterConnector(zookeeperHosts, canalDestination, "", "");
((ClusterCanalConnector) connector).setSoTimeout(SO_TIMEOUT);
}
@Override
protected void process() {
while (!running)
; // waiting until running == true
while (running) {
try {
syncSwitch.get(canalDestination);
logger.info("=============> Start to connect destination: {} <=============", this.canalDestination);
connector.connect();
logger.info("=============> Start to subscribe destination: {} <=============", this.canalDestination);
connector.subscribe();
logger.info("=============> Subscribe destination: {} succeed <=============", this.canalDestination);
while (running) {
try {
syncSwitch.get(canalDestination, 1L, TimeUnit.MINUTES);
} catch (TimeoutException e) {
break;
}
if (!running) {
break;
}
// server配置canal.instance.network.soTimeout(默认: 30s)
// 范围内未与server交互,server将关闭本次socket连接
Message message = connector.getWithoutAck(BATCH_SIZE); // 获取指定数量的数据
long batchId = message.getId();
try {
int size = message.getEntries().size();
if (batchId == -1 || size == 0) {
Thread.sleep(1000);
} else {
if (logger.isDebugEnabled()) {
logger.debug("destination: {} batchId: {} batchSize: {} ",
this.canalDestination,
batchId,
size);
}
long begin = System.currentTimeMillis();
writeOut(message);
if (logger.isDebugEnabled()) {
logger.debug("destination: {} batchId: {} elapsed time: {} ms",
this.canalDestination,
batchId,
System.currentTimeMillis() - begin);
}
}
connector.ack(batchId); // 提交确认
} catch (Exception e) {
connector.rollback(batchId); // 处理失败, 回滚数据
logger.error("sync error!", e);
Thread.sleep(500);
}
}
} catch (Exception e) {
logger.error("process error!", e);
} finally {
connector.disconnect();
logger.info("=============> Disconnect destination: {} <=============", this.canalDestination);
}
if (running) { // is reconnect
try {
Thread.sleep(1000);
} catch (InterruptedException e) {
// ignore
}
}
}
}
@Override
public void stop() {
try {
if (!running) {
return;
}
if (connector instanceof ClusterCanalConnector) {
((ClusterCanalConnector) connector).stopRunning();
} else if (connector instanceof SimpleCanalConnector) {
((SimpleCanalConnector) connector).stopRunning();
}
running = false;
syncSwitch.release(canalDestination);
logger.info("destination {} is waiting for adapters' worker thread die!", canalDestination);
if (thread != null) {
try {
thread.join();
} catch (InterruptedException e) {
// ignore
}
}
groupInnerExecutorService.shutdown();
logger.info("destination {} adapters worker thread dead!", canalDestination);
canalOuterAdapters.forEach(outerAdapters -> outerAdapters.forEach(OuterAdapter::destroy));
logger.info("destination {} all adapters destroyed!", canalDestination);
} catch (Exception e) {
logger.error(e.getMessage(), e);
}
}
}

View File

@@ -1,172 +0,0 @@
package com.alibaba.otter.canal.adapter.launcher.rest;
import java.util.*;
import javax.annotation.PostConstruct;
import javax.annotation.Resource;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
import org.springframework.web.bind.annotation.*;
import com.alibaba.otter.canal.adapter.launcher.common.EtlLock;
import com.alibaba.otter.canal.adapter.launcher.common.SyncSwitch;
import com.alibaba.otter.canal.adapter.launcher.config.AdapterCanalConfig;
import com.alibaba.otter.canal.client.adapter.OuterAdapter;
import com.alibaba.otter.canal.client.adapter.support.EtlResult;
import com.alibaba.otter.canal.client.adapter.support.ExtensionLoader;
import com.alibaba.otter.canal.client.adapter.support.Result;
/**
* 适配器操作Rest
*
* @author rewerma @ 2018-10-20
* @version 1.0.0
*/
@RestController
public class CommonRest {
private static Logger logger = LoggerFactory.getLogger(CommonRest.class);
private static final String ETL_LOCK_ZK_NODE = "/sync-etl/";
private ExtensionLoader<OuterAdapter> loader;
@Resource
private SyncSwitch syncSwitch;
@Resource
private EtlLock etlLock;
@Resource
private AdapterCanalConfig adapterCanalConfig;
@PostConstruct
public void init() {
loader = ExtensionLoader.getExtensionLoader(OuterAdapter.class);
}
/**
* ETL curl http://127.0.0.1:8081/etl/hbase/mytest_person2.yml -X POST
*
* @param type 类型 hbase, es
* @param task 任务名对应配置文件名 mytest_person2.yml
* @param params etl where条件参数, 为空全部导入
* @return
*/
@PostMapping("/etl/{type}/{task}")
public EtlResult etl(@PathVariable String type, @PathVariable String task,
@RequestParam(name = "params", required = false) String params) {
boolean locked = etlLock.tryLock(ETL_LOCK_ZK_NODE + type + "-" + task);
if (!locked) {
EtlResult result = new EtlResult();
result.setSucceeded(false);
result.setErrorMessage(task + " 有其他进程正在导入中, 请稍后再试");
return result;
}
try {
OuterAdapter adapter = loader.getExtension(type);
String destination = adapter.getDestination(task);
Boolean oriSwithcStatus = null;
if (destination != null) {
oriSwithcStatus = syncSwitch.status(destination);
syncSwitch.off(destination);
}
try {
List<String> paramArr = null;
if (params != null) {
String[] parmaArray = params.trim().split(";");
paramArr = Arrays.asList(parmaArray);
}
return adapter.etl(task, paramArr);
} finally {
if (destination != null && oriSwithcStatus != null && oriSwithcStatus) {
syncSwitch.on(destination);
}
}
} finally {
etlLock.unlock(ETL_LOCK_ZK_NODE + type + "-" + task);
}
}
/**
* 统计总数 curl http://127.0.0.1:8081/count/hbase/mytest_person2.yml
*
* @param type 类型 hbase, es
* @param task 任务名对应配置文件名 mytest_person2.yml
* @return
*/
@GetMapping("/count/{type}/{task}")
public Map<String, Object> count(@PathVariable String type, @PathVariable String task) {
OuterAdapter adapter = loader.getExtension(type);
return adapter.count(task);
}
/**
* 返回所有实例 curl http://127.0.0.1:8081/destinations
*/
@GetMapping("/destinations")
public List<Map<String, String>> destinations() {
List<Map<String, String>> result = new ArrayList<>();
Set<String> destinations = adapterCanalConfig.DESTINATIONS;
for (String destination : destinations) {
Map<String, String> resMap = new LinkedHashMap<>();
Boolean status = syncSwitch.status(destination);
String resStatus = "none";
if (status != null && status) {
resStatus = "on";
} else if (status != null && !status) {
resStatus = "off";
}
resMap.put("destination", destination);
resMap.put("status", resStatus);
result.add(resMap);
}
return result;
}
/**
* 实例同步开关 curl http://127.0.0.1:8081/syncSwitch/example/off -X PUT
*
* @param destination 实例名称
* @param status 开关状态: off on
* @return
*/
@PutMapping("/syncSwitch/{destination}/{status}")
public Result etl(@PathVariable String destination, @PathVariable String status) {
if (status.equals("on")) {
syncSwitch.on(destination);
logger.info("#Destination: {} sync on", destination);
return Result.createSuccess("实例: " + destination + " 开启同步成功");
} else if (status.equals("off")) {
syncSwitch.off(destination);
logger.info("#Destination: {} sync off", destination);
return Result.createSuccess("实例: " + destination + " 关闭同步成功");
} else {
Result result = new Result();
result.setCode(50000);
result.setMessage("实例: " + destination + " 操作失败");
return result;
}
}
/**
* 获取实例开关状态 curl http://127.0.0.1:8081/syncSwitch/example
*
* @param destination 实例名称
* @return
*/
@GetMapping("/syncSwitch/{destination}")
public Map<String, String> etl(@PathVariable String destination) {
Boolean status = syncSwitch.status(destination);
String resStatus = "none";
if (status != null && status) {
resStatus = "on";
} else if (status != null && !status) {
resStatus = "off";
}
Map<String, String> res = new LinkedHashMap<>();
res.put("stauts", resStatus);
return res;
}
}

View File

@@ -1,54 +0,0 @@
server:
port: 8081
logging:
level:
com.alibaba.otter.canal.client.adapter.hbase: DEBUG
spring:
jackson:
date-format: yyyy-MM-dd HH:mm:ss
time-zone: GMT+8
default-property-inclusion: non_null
hbasezookeeper.quorum: 127.0.0.1
hbase.zookeeper.property.clientPort: 2181
hbase.zookeeper.znode.parent: /hbase
canal.conf:
canalServerHost: 127.0.0.1:11111
# zookeeperHosts: slave1:2181
# bootstrapServers: slave1:6667 #or rocketmq nameservers:host1:9876;host2:9876
flatMessage: true
canalInstances:
- instance: example
groups:
- outAdapters:
- name: logger
# - name: hbase
# properties:
# hbase.zookeeper.quorum: ${hbase.zookeeper.quorum}
# hbase.zookeeper.property.clientPort: ${hbase.zookeeper.property.clientPort}
# zookeeper.znode.parent: ${hbase.zookeeper.znode.parent}
# mqTopics:
# - mqMode: kafka
# topic: example
# groups:
# - groupId: g2
# outAdapters:
# - name: logger
# mqTopics:
# - mqMode: rocketmq
# topic: example
# groups:
# - groupId: g2
# outAdapters:
# - name: logger
#adapter.conf:
# datasourceConfigs:
# defaultDS:
# url: jdbc:mysql://127.0.0.1:3306/mytest?useUnicode=true
# username: root
# password: 121212
# adapterConfigs:
# - hbase/mytest_person2.yml

View File

@@ -1,46 +0,0 @@
<?xml version="1.0" encoding="UTF-8"?>
<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
<parent>
<artifactId>canal.client-adapter</artifactId>
<groupId>com.alibaba.otter</groupId>
<version>1.1.1</version>
</parent>
<modelVersion>4.0.0</modelVersion>
<groupId>com.alibaba.otter</groupId>
<artifactId>client-adapter.logger</artifactId>
<packaging>jar</packaging>
<name>canal client adapter logger example module for otter ${project.version}</name>
<dependencies>
<dependency>
<groupId>com.alibaba.otter</groupId>
<artifactId>client-adapter.common</artifactId>
<version>${project.version}</version>
<scope>provided</scope>
</dependency>
</dependencies>
<build>
<plugins>
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-assembly-plugin</artifactId>
<version>2.4</version>
<configuration>
<descriptorRefs>
<descriptorRef>jar-with-dependencies</descriptorRef>
</descriptorRefs>
</configuration>
<executions>
<execution>
<id>make-assembly</id>
<phase>package</phase>
<goals>
<goal>single</goal>
</goals>
</execution>
</executions>
</plugin>
</plugins>
</build>
</project>

View File

@@ -1,37 +0,0 @@
package com.alibaba.otter.canal.client.adapter.logger;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
import com.alibaba.otter.canal.client.adapter.OuterAdapter;
import com.alibaba.otter.canal.client.adapter.support.OuterAdapterConfig;
import com.alibaba.otter.canal.client.adapter.support.Dml;
import com.alibaba.otter.canal.client.adapter.support.SPI;
/**
* 外部适配器示例
*
* @author machengyuan 2018-8-19 下午11:45:38
* @version 1.0.0
*/
@SPI("logger")
// logger参数对应CanalOuterAdapterConfiguration配置中的name
public class LoggerAdapterExample implements OuterAdapter {
private Logger logger = LoggerFactory.getLogger(this.getClass());
@Override
public void init(OuterAdapterConfig configuration) {
}
@Override
public void sync(Dml dml) {
logger.info(dml.toString());
}
@Override
public void destroy() {
}
}

View File

@@ -1 +0,0 @@
logger=com.alibaba.otter.canal.client.adapter.logger.LoggerAdapterExample

View File

@@ -1,41 +0,0 @@
<?xml version="1.0" encoding="UTF-8"?>
<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
<modelVersion>4.0.0</modelVersion>
<groupId>com.alibaba.otter</groupId>
<artifactId>canal.client-adapter</artifactId>
<version>1.1.1</version>
<packaging>pom</packaging>
<name>canal client adapter module for otter ${project.version}</name>
<properties>
<project.build.sourceEncoding>UTF-8</project.build.sourceEncoding>
<maven.test.skip>true</maven.test.skip>
<downloadSources>true</downloadSources>
<java_source_version>1.8</java_source_version>
<java_target_version>1.8</java_target_version>
<file_encoding>UTF-8</file_encoding>
<canal_version>1.1.1-SNAPSHOT</canal_version>
</properties>
<modules>
<module>common</module>
<module>logger</module>
<module>hbase</module>
<module>launcher</module>
</modules>
<build>
<plugins>
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-compiler-plugin</artifactId>
<version>3.7.0</version>
<configuration>
<source>${java_source_version}</source>
<target>${java_target_version}</target>
<encoding>${file_encoding}</encoding>
</configuration>
</plugin>
</plugins>
</build>
</project>

View File

@@ -3,7 +3,7 @@
<parent>
<groupId>com.alibaba.otter</groupId>
<artifactId>canal</artifactId>
<version>1.1.1</version>
<version>1.0.26-SNAPSHOT</version>
<relativePath>../pom.xml</relativePath>
</parent>
<groupId>com.alibaba.otter</groupId>
@@ -15,180 +15,15 @@
<groupId>com.alibaba.otter</groupId>
<artifactId>canal.protocol</artifactId>
<version>${project.version}</version>
<optional>true</optional>
</dependency>
<dependency>
<groupId>com.google.protobuf</groupId>
<artifactId>protobuf-java</artifactId>
</dependency>
<!-- https://mvnrepository.com/artifact/io.netty/netty-all -->
<dependency>
<groupId>io.netty</groupId>
<artifactId>netty-all</artifactId>
</dependency>
<!-- zk -->
<dependency>
<groupId>org.apache.zookeeper</groupId>
<artifactId>zookeeper</artifactId>
</dependency>
<dependency>
<groupId>com.101tec</groupId>
<artifactId>zkclient</artifactId>
</dependency>
<!-- external -->
<dependency>
<groupId>commons-io</groupId>
<artifactId>commons-io</artifactId>
</dependency>
<dependency>
<groupId>commons-lang</groupId>
<artifactId>commons-lang</artifactId>
</dependency>
<dependency>
<groupId>commons-codec</groupId>
<artifactId>commons-codec</artifactId>
</dependency>
<dependency>
<groupId>com.alibaba</groupId>
<artifactId>fastjson</artifactId>
</dependency>
<dependency>
<groupId>com.google.guava</groupId>
<artifactId>guava</artifactId>
</dependency>
<!-- log -->
<dependency>
<groupId>ch.qos.logback</groupId>
<artifactId>logback-core</artifactId>
</dependency>
<dependency>
<groupId>ch.qos.logback</groupId>
<artifactId>logback-classic</artifactId>
</dependency>
<dependency>
<groupId>org.slf4j</groupId>
<artifactId>jcl-over-slf4j</artifactId>
</dependency>
<dependency>
<groupId>org.slf4j</groupId>
<artifactId>slf4j-api</artifactId>
</dependency>
<dependency>
<groupId>org.springframework</groupId>
<artifactId>spring-core</artifactId>
</dependency>
<dependency>
<groupId>org.springframework</groupId>
<artifactId>spring-aop</artifactId>
</dependency>
<dependency>
<groupId>org.springframework</groupId>
<artifactId>spring-context</artifactId>
</dependency>
<dependency>
<groupId>org.springframework</groupId>
<artifactId>spring-jdbc</artifactId>
</dependency>
<dependency>
<groupId>org.springframework</groupId>
<artifactId>spring-orm</artifactId>
</dependency>
<dependency>
<groupId>org.springframework</groupId>
<artifactId>spring-test</artifactId>
<version>${spring_version}</version>
<scope>test</scope>
</dependency>
<!-- 客户端要使用请单独引入rocketmq-client依赖 -->
<dependency>
<groupId>org.apache.rocketmq</groupId>
<artifactId>rocketmq-client</artifactId>
<version>4.3.0</version>
<scope>provided</scope>
</dependency>
<!-- 客户端要使用请单独引入kafka-clients依赖 -->
<dependency>
<groupId>org.apache.kafka</groupId>
<artifactId>kafka-clients</artifactId>
<version>1.1.1</version>
<scope>provided</scope>
</dependency>
<!-- junit -->
<dependency>
<groupId>junit</groupId>
<artifactId>junit</artifactId>
<scope>test</scope>
</dependency>
</dependencies>
<build>
<plugins>
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-shade-plugin</artifactId>
<version>2.4.1</version>
<configuration>
<createDependencyReducedPom>false</createDependencyReducedPom>
<artifactSet>
<excludes>
<exclude>org.apache.zookeeper:zookeeper</exclude>
<exclude>com.101tec:zkclient</exclude>
<exclude>commons-io:commons-io</exclude>
<exclude>commons-lang:commons-lang</exclude>
<exclude>commons-codec:commons-codec</exclude>
<exclude>com.alibaba:fastjson</exclude>
<exclude>ch.qos.logback:logback-core</exclude>
<exclude>ch.qos.logback:logback-classic</exclude>
<exclude>org.slf4j:jcl-over-slf4j</exclude>
<exclude>org.slf4j:slf4j-api</exclude>
<exclude>org.springframework:*</exclude>
<exclude>org.jboss.netty:netty</exclude>
<exclude>commons-logging:commons-logging</exclude>
<exclude>aopalliance:aopalliance</exclude>
<exclude>com.google.protobuf:protobuf-java</exclude>
<exclude>io.netty:*</exclude>
<exclude>junit:junit</exclude>
</excludes>
</artifactSet>
</configuration>
<executions>
<execution>
<phase>package</phase>
<goals>
<goal>shade</goal>
</goals>
<configuration>
<relocations>
<relocation>
<pattern>com.google.guava</pattern>
<shadedPattern>com.alibaba.google.guava</shadedPattern>
</relocation>
<relocation>
<pattern>org.joda</pattern>
<shadedPattern>com.alibaba.google.joda</shadedPattern>
</relocation>
<relocation>
<pattern>com.google.common</pattern>
<shadedPattern>com.alibaba.google.common</shadedPattern>
</relocation>
<relocation>
<pattern>com.google.thirdparty</pattern>
<shadedPattern>com.alibaba.google.thirdparty</shadedPattern>
</relocation>
</relocations>
<transformers>
<transformer implementation="org.apache.maven.plugins.shade.resource.ManifestResourceTransformer" />
</transformers>
</configuration>
</execution>
</executions>
</plugin>
</plugins>
</build>
<profiles>
<profile>
<id>dev</id>
@@ -256,7 +91,7 @@
<content>${project.build.directory}/apidocs</content>
<skipDeletedFiles>true</skipDeletedFiles>
<pubScmUrl>scm:git:git@github.com:alibaba/canal.git</pubScmUrl>
<scmBranch>gh-pages</scmBranch>
<scmBranch>gh-pages</scmBranch>
</configuration>
</plugin>
</plugins>

View File

@@ -149,4 +149,11 @@ public interface CanalConnector {
* @throws CanalClientException
*/
void rollback() throws CanalClientException;
/**
* 中断的阻塞,用于优雅停止client
*
* @throws CanalClientException
*/
void stopRunning() throws CanalClientException;
}

View File

@@ -1,91 +0,0 @@
package com.alibaba.otter.canal.client;
import java.util.List;
import java.util.concurrent.TimeUnit;
import com.alibaba.otter.canal.protocol.FlatMessage;
import com.alibaba.otter.canal.protocol.Message;
import com.alibaba.otter.canal.protocol.exception.CanalClientException;
/**
* canal MQ数据操作客户端
*
* <pre>
* 1. canal server写入MQ消息,考虑性能会合并多条数据写入为一个MQ消息,一个Message对应一个MQ消息
* 2. canal client消费MQ消息,因为client性能会弱于server的写入,MQ数据获取时会拿到堆积的多条MQ消息,会拿到List<Message>
* 3. client的ack/rollback,都是和MQ直接交互,不存在对应的batchId概念
* </pre>
*
* @author agapple 2018年10月28日 下午6:42:27
* @since 1.1.1
*/
public interface CanalMQConnector extends CanalConnector {
/**
* 获取数据,自动进行确认,设置timeout时间直到拿到数据为止
*
* <pre>
* 该方法返回的条件:
* a. 如果timeout=0,有多少取多少,不会阻塞等待
* b. 如果timeout不为0,尝试阻塞对应的超时时间,直到拿到数据就返回
* </pre>
*
* @return
* @throws CanalClientException
*/
List<Message> getList(Long timeout, TimeUnit unit) throws CanalClientException;
/**
* 获取数据,设置timeout时间直到拿到数据为止
*
* <pre>
* 该方法返回的条件:
* a. 如果timeout=0,有多少取多少,不会阻塞等待
* b. 如果timeout不为0,尝试阻塞对应的超时时间,直到拿到数据就返回
* </pre>
*
* @throws CanalClientException
*/
List<Message> getListWithoutAck(Long timeout, TimeUnit unit) throws CanalClientException;
/**
* 获取数据,自动进行确认,设置timeout时间直到拿到数据为止
*
* <pre>
* 该方法返回的条件:
* a. 如果timeout=0,有多少取多少,不会阻塞等待
* b. 如果timeout不为0,尝试阻塞对应的超时时间,直到拿到数据就返回
* </pre>
*
* @return
* @throws CanalClientException
*/
List<FlatMessage> getFlatList(Long timeout, TimeUnit unit) throws CanalClientException;
/**
* 获取数据,设置timeout时间直到拿到数据为止
*
* <pre>
* 该方法返回的条件:
* a. 如果timeout=0,有多少取多少,不会阻塞等待
* b. 如果timeout不为0,尝试阻塞对应的超时时间,直到拿到数据就返回
* </pre>
*
* @throws CanalClientException
*/
List<FlatMessage> getFlatListWithoutAck(Long timeout, TimeUnit unit) throws CanalClientException;
/**
* 消费确认。
*
* @throws CanalClientException
*/
void ack() throws CanalClientException;
/**
* 回滚到未进行 {@link #ack} 的地方,下次fetch的时候,可以从最后一个没有 {@link #ack} 的地方开始拿
*
* @throws CanalClientException
*/
void rollback() throws CanalClientException;
}

View File

@@ -1,55 +0,0 @@
package com.alibaba.otter.canal.client;
import com.alibaba.otter.canal.protocol.CanalEntry;
import com.alibaba.otter.canal.protocol.CanalPacket;
import com.alibaba.otter.canal.protocol.CanalPacket.Ack;
import com.alibaba.otter.canal.protocol.CanalPacket.Compression;
import com.alibaba.otter.canal.protocol.Message;
import com.alibaba.otter.canal.protocol.exception.CanalClientException;
import com.google.protobuf.ByteString;
public class CanalMessageDeserializer {
public static Message deserializer(byte[] data) {
return deserializer(data, false);
}
public static Message deserializer(byte[] data, boolean lazyParseEntry) {
try {
if (data == null) {
return null;
} else {
CanalPacket.Packet p = CanalPacket.Packet.parseFrom(data);
switch (p.getType()) {
case MESSAGES: {
if (!p.getCompression().equals(Compression.NONE)
&& !p.getCompression().equals(Compression.COMPRESSIONCOMPATIBLEPROTO2)) {
throw new CanalClientException("compression is not supported in this connector");
}
CanalPacket.Messages messages = CanalPacket.Messages.parseFrom(p.getBody());
Message result = new Message(messages.getBatchId());
if (lazyParseEntry) {
// byteString
result.setRawEntries(messages.getMessagesList());
} else {
for (ByteString byteString : messages.getMessagesList()) {
result.addEntry(CanalEntry.Entry.parseFrom(byteString));
}
}
return result;
}
case ACK: {
Ack ack = Ack.parseFrom(p.getBody());
throw new CanalClientException("something goes wrong with reason: " + ack.getErrorMessage());
}
default: {
throw new CanalClientException("unexpected packet type: " + p.getType());
}
}
}
} catch (Exception e) {
throw new CanalClientException("deserializer failed", e);
}
}
}

View File

@@ -218,7 +218,8 @@ public class ClusterCanalConnector implements CanalConnector {
return;
} catch (Throwable t) {
logger.warn(String.format("something goes wrong when rollbacking data from server:%s",
currentConnector != null ? currentConnector.getAddress() : "null"), t);
currentConnector != null ? currentConnector.getAddress() : "null"),
t);
times++;
restart();
logger.info("restart the connector for next round retry.");
@@ -235,7 +236,8 @@ public class ClusterCanalConnector implements CanalConnector {
return;
} catch (Throwable t) {
logger.warn(String.format("something goes wrong when rollbacking data from server:%s",
currentConnector != null ? currentConnector.getAddress() : "null"), t);
currentConnector != null ? currentConnector.getAddress() : "null"),
t);
times++;
restart();
logger.info("restart the connector for next round retry.");
@@ -253,7 +255,8 @@ public class ClusterCanalConnector implements CanalConnector {
return;
} catch (Throwable t) {
logger.warn(String.format("something goes wrong when acking data from server:%s",
currentConnector != null ? currentConnector.getAddress() : "null"), t);
currentConnector != null ? currentConnector.getAddress() : "null"),
t);
times++;
restart();
logger.info("restart the connector for next round retry.");

View File

@@ -18,13 +18,13 @@ import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
import com.alibaba.otter.canal.client.CanalConnector;
import com.alibaba.otter.canal.client.CanalMessageDeserializer;
import com.alibaba.otter.canal.client.impl.running.ClientRunningData;
import com.alibaba.otter.canal.client.impl.running.ClientRunningListener;
import com.alibaba.otter.canal.client.impl.running.ClientRunningMonitor;
import com.alibaba.otter.canal.common.utils.AddressUtils;
import com.alibaba.otter.canal.common.utils.BooleanMutex;
import com.alibaba.otter.canal.common.zookeeper.ZkClientx;
import com.alibaba.otter.canal.protocol.CanalEntry.Entry;
import com.alibaba.otter.canal.protocol.CanalPacket.Ack;
import com.alibaba.otter.canal.protocol.CanalPacket.ClientAck;
import com.alibaba.otter.canal.protocol.CanalPacket.ClientAuth;
@@ -32,6 +32,7 @@ import com.alibaba.otter.canal.protocol.CanalPacket.ClientRollback;
import com.alibaba.otter.canal.protocol.CanalPacket.Compression;
import com.alibaba.otter.canal.protocol.CanalPacket.Get;
import com.alibaba.otter.canal.protocol.CanalPacket.Handshake;
import com.alibaba.otter.canal.protocol.CanalPacket.Messages;
import com.alibaba.otter.canal.protocol.CanalPacket.Packet;
import com.alibaba.otter.canal.protocol.CanalPacket.PacketType;
import com.alibaba.otter.canal.protocol.CanalPacket.Sub;
@@ -158,7 +159,7 @@ public class SimpleCanalConnector implements CanalConnector {
}
//
Handshake handshake = Handshake.parseFrom(p.getBody());
supportedCompressions.add(handshake.getSupportedCompressions());
supportedCompressions.addAll(handshake.getSupportedCompressionsList());
//
ClientAuth ca = ClientAuth.newBuilder()
.setUsername(username != null ? username : "")
@@ -319,7 +320,33 @@ public class SimpleCanalConnector implements CanalConnector {
private Message receiveMessages() throws IOException {
byte[] data = readNextPacket();
return CanalMessageDeserializer.deserializer(data, lazyParseEntry);
Packet p = Packet.parseFrom(data);
switch (p.getType()) {
case MESSAGES: {
if (!p.getCompression().equals(Compression.NONE)) {
throw new CanalClientException("compression is not supported in this connector");
}
Messages messages = Messages.parseFrom(p.getBody());
Message result = new Message(messages.getBatchId());
if (lazyParseEntry) {
// byteString
result.setRawEntries(messages.getMessagesList());
} else {
for (ByteString byteString : messages.getMessagesList()) {
result.addEntry(Entry.parseFrom(byteString));
}
}
return result;
}
case ACK: {
Ack ack = Ack.parseFrom(p.getBody());
throw new CanalClientException("something goes wrong with reason: " + ack.getErrorMessage());
}
default: {
throw new CanalClientException("unexpected packet type: " + p.getType());
}
}
}
public void ack(long batchId) throws CanalClientException {

View File

@@ -98,7 +98,7 @@ public class ClientRunningMonitor extends AbstractCanalLifeCycle {
String path = ZookeeperPathUtils.getDestinationClientRunning(this.destination, clientData.getClientId());
zkClient.unsubscribeDataChanges(path, dataListener);
releaseRunning(); // 尝试一下release
// Fix issue #697
//Fix issue #697
if (delayExector != null) {
delayExector.shutdown();
}

View File

@@ -1,288 +0,0 @@
package com.alibaba.otter.canal.client.kafka;
import java.util.ArrayList;
import java.util.Collections;
import java.util.List;
import java.util.Properties;
import java.util.concurrent.TimeUnit;
import org.apache.kafka.clients.consumer.ConsumerRecord;
import org.apache.kafka.clients.consumer.ConsumerRecords;
import org.apache.kafka.clients.consumer.KafkaConsumer;
import org.apache.kafka.common.TopicPartition;
import org.apache.kafka.common.serialization.StringDeserializer;
import com.alibaba.fastjson.JSON;
import com.alibaba.otter.canal.client.CanalMQConnector;
import com.alibaba.otter.canal.client.impl.SimpleCanalConnector;
import com.alibaba.otter.canal.protocol.FlatMessage;
import com.alibaba.otter.canal.protocol.Message;
import com.alibaba.otter.canal.protocol.exception.CanalClientException;
import com.google.common.collect.Lists;
/**
* canal kafka 数据操作客户端
*
* <pre>
* 注意点:
* 1. 相比于canal {@linkplain SimpleCanalConnector}, 这里get和ack操作不能有并发, 必须是一个线程执行get后,内存里执行完毕ack后再取下一个get
* </pre>
*
* @author machengyuan @ 2018-6-12
* @version 1.1.1
*/
public class KafkaCanalConnector implements CanalMQConnector {
private KafkaConsumer<String, Message> kafkaConsumer;
private KafkaConsumer<String, String> kafkaConsumer2; // 用于扁平message的数据消费
private String topic;
private Integer partition;
private Properties properties;
private volatile boolean connected = false;
private volatile boolean running = false;
private boolean flatMessage;
public KafkaCanalConnector(String servers, String topic, Integer partition, String groupId, boolean flatMessage){
this.topic = topic;
this.partition = partition;
this.flatMessage = flatMessage;
properties = new Properties();
properties.put("bootstrap.servers", servers);
properties.put("group.id", groupId);
properties.put("enable.auto.commit", false);
properties.put("auto.commit.interval.ms", "1000");
properties.put("auto.offset.reset", "latest"); // 如果没有offset则从最后的offset开始读
properties.put("request.timeout.ms", "40000"); // 必须大于session.timeout.ms的设置
properties.put("session.timeout.ms", "30000"); // 默认为30秒
properties.put("max.poll.records", "100");
properties.put("key.deserializer", StringDeserializer.class.getName());
if (!flatMessage) {
properties.put("value.deserializer", MessageDeserializer.class.getName());
} else {
properties.put("value.deserializer", StringDeserializer.class.getName());
}
}
/**
* 打开连接
*/
public void connect() {
if (connected) {
return;
}
connected = true;
if (kafkaConsumer == null && !flatMessage) {
kafkaConsumer = new KafkaConsumer<String, Message>(properties);
}
if (kafkaConsumer2 == null && flatMessage) {
kafkaConsumer2 = new KafkaConsumer<String, String>(properties);
}
}
/**
* 关闭链接
*/
public void disconnect() {
if (kafkaConsumer != null) {
kafkaConsumer.close();
kafkaConsumer = null;
}
if (kafkaConsumer2 != null) {
kafkaConsumer2.close();
kafkaConsumer2 = null;
}
connected = false;
}
private void waitClientRunning() {
running = true;
}
public boolean checkValid() {
return true;// 默认都放过
}
/**
* 订阅topic
*/
public void subscribe() {
waitClientRunning();
if (!running) {
return;
}
if (partition == null) {
if (kafkaConsumer != null) {
kafkaConsumer.subscribe(Collections.singletonList(topic));
}
if (kafkaConsumer2 != null) {
kafkaConsumer2.subscribe(Collections.singletonList(topic));
}
} else {
TopicPartition topicPartition = new TopicPartition(topic, partition);
if (kafkaConsumer != null) {
kafkaConsumer.assign(Collections.singletonList(topicPartition));
}
if (kafkaConsumer2 != null) {
kafkaConsumer2.assign(Collections.singletonList(topicPartition));
}
}
}
/**
* 取消订阅
*/
public void unsubscribe() {
waitClientRunning();
if (!running) {
return;
}
if (kafkaConsumer != null) {
kafkaConsumer.unsubscribe();
}
if (kafkaConsumer2 != null) {
kafkaConsumer2.unsubscribe();
}
}
@Override
public List<Message> getList(Long timeout, TimeUnit unit) throws CanalClientException {
waitClientRunning();
if (!running) {
return Lists.newArrayList();
}
List<Message> messages = getListWithoutAck(timeout, unit);
if (messages != null && !messages.isEmpty()) {
this.ack();
}
return messages;
}
@Override
public List<Message> getListWithoutAck(Long timeout, TimeUnit unit) throws CanalClientException {
waitClientRunning();
if (!running) {
return Lists.newArrayList();
}
ConsumerRecords<String, Message> records = kafkaConsumer.poll(unit.toMillis(timeout)); // 基于配置,最多只能poll到一条数据
if (!records.isEmpty()) {
List<Message> messages = new ArrayList<>();
for (ConsumerRecord<String, Message> record : records) {
messages.add(record.value());
}
return messages;
}
return Lists.newArrayList();
}
@Override
public List<FlatMessage> getFlatList(Long timeout, TimeUnit unit) throws CanalClientException {
waitClientRunning();
if (!running) {
return Lists.newArrayList();
}
List<FlatMessage> messages = getFlatListWithoutAck(timeout, unit);
if (messages != null && !messages.isEmpty()) {
this.ack();
}
return messages;
}
@Override
public List<FlatMessage> getFlatListWithoutAck(Long timeout, TimeUnit unit) throws CanalClientException {
waitClientRunning();
if (!running) {
return Lists.newArrayList();
}
ConsumerRecords<String, String> records = kafkaConsumer2.poll(unit.toMillis(timeout));
if (!records.isEmpty()) {
List<FlatMessage> flatMessages = new ArrayList<>();
for (ConsumerRecord<String, String> record : records) {
String flatMessageJson = record.value();
FlatMessage flatMessage = JSON.parseObject(flatMessageJson, FlatMessage.class);
flatMessages.add(flatMessage);
}
return flatMessages;
}
return Lists.newArrayList();
}
@Override
public void rollback() throws CanalClientException {
}
/**
* 提交offset,如果超过 session.timeout.ms 设置的时间没有ack则会抛出异常,ack失败
*/
public void ack() {
waitClientRunning();
if (!running) {
return;
}
if (kafkaConsumer != null) {
kafkaConsumer.commitSync();
}
if (kafkaConsumer2 != null) {
kafkaConsumer2.commitSync();
}
}
@Override
public void subscribe(String filter) throws CanalClientException {
throw new CanalClientException("mq not support this method");
}
@Override
public Message get(int batchSize) throws CanalClientException {
throw new CanalClientException("mq not support this method");
}
@Override
public Message get(int batchSize, Long timeout, TimeUnit unit) throws CanalClientException {
throw new CanalClientException("mq not support this method");
}
@Override
public Message getWithoutAck(int batchSize) throws CanalClientException {
throw new CanalClientException("mq not support this method");
}
@Override
public Message getWithoutAck(int batchSize, Long timeout, TimeUnit unit) throws CanalClientException {
throw new CanalClientException("mq not support this method");
}
@Override
public void ack(long batchId) throws CanalClientException {
throw new CanalClientException("mq not support this method");
}
@Override
public void rollback(long batchId) throws CanalClientException {
throw new CanalClientException("mq not support this method");
}
/**
* 重新设置sessionTime
*
* @param timeout
* @param unit
*/
public void setSessionTimeout(Long timeout, TimeUnit unit) {
long t = unit.toMillis(timeout);
properties.put("request.timeout.ms", String.valueOf(t + 60000));
properties.put("session.timeout.ms", String.valueOf(t));
}
}

View File

@@ -1,31 +0,0 @@
package com.alibaba.otter.canal.client.kafka;
import java.util.Map;
import org.apache.kafka.common.serialization.Deserializer;
import com.alibaba.otter.canal.client.CanalMessageDeserializer;
import com.alibaba.otter.canal.protocol.Message;
/**
* Kafka Message类的反序列化
*
* @author machengyuan @ 2018-6-12
* @version 1.0.0
*/
public class MessageDeserializer implements Deserializer<Message> {
@Override
public void configure(Map<String, ?> configs, boolean isKey) {
}
@Override
public Message deserialize(String topic1, byte[] data) {
return CanalMessageDeserializer.deserializer(data);
}
@Override
public void close() {
// nothing to do
}
}

View File

@@ -1,48 +0,0 @@
package com.alibaba.otter.canal.client.rocketmq;
import java.util.List;
import java.util.concurrent.CountDownLatch;
import java.util.concurrent.TimeUnit;
public class ConsumerBatchMessage<T> {
private final List<T> data;
private CountDownLatch latch;
private boolean hasFailure = false;
public ConsumerBatchMessage(List<T> data){
this.data = data;
latch = new CountDownLatch(1);
}
public boolean waitFinish(long timeout) throws InterruptedException {
return latch.await(timeout, TimeUnit.MILLISECONDS);
}
public boolean isSuccess() {
return !hasFailure;
}
public List<T> getData() {
return data;
}
/**
* Countdown if the sub message is successful.
*/
public void ack() {
latch.countDown();
}
/**
* Countdown and fail-fast if the sub message is failed.
*/
public void fail() {
hasFailure = true;
// fail fast
long count = latch.getCount();
for (int i = 0; i < count; i++) {
latch.countDown();
}
}
}

View File

@@ -1,263 +0,0 @@
package com.alibaba.otter.canal.client.rocketmq;
import java.util.List;
import java.util.concurrent.BlockingQueue;
import java.util.concurrent.LinkedBlockingQueue;
import java.util.concurrent.TimeUnit;
import org.apache.commons.lang.StringUtils;
import org.apache.rocketmq.client.consumer.DefaultMQPushConsumer;
import org.apache.rocketmq.client.consumer.listener.ConsumeOrderlyContext;
import org.apache.rocketmq.client.consumer.listener.ConsumeOrderlyStatus;
import org.apache.rocketmq.client.consumer.listener.MessageListenerOrderly;
import org.apache.rocketmq.client.exception.MQClientException;
import org.apache.rocketmq.common.message.MessageExt;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
import com.alibaba.fastjson.JSON;
import com.alibaba.otter.canal.client.CanalMQConnector;
import com.alibaba.otter.canal.client.CanalMessageDeserializer;
import com.alibaba.otter.canal.client.impl.SimpleCanalConnector;
import com.alibaba.otter.canal.protocol.FlatMessage;
import com.alibaba.otter.canal.protocol.Message;
import com.alibaba.otter.canal.protocol.exception.CanalClientException;
import com.google.common.collect.Lists;
/**
* RocketMQ的连接
*
* <pre>
* 注意点:
* 1. 相比于canal {@linkplain SimpleCanalConnector}, 这里get和ack操作不能有并发, 必须是一个线程执行get后,内存里执行完毕ack后再取下一个get
* </pre>
*
* @since 1.1.1
*/
public class RocketMQCanalConnector implements CanalMQConnector {
private static final Logger logger = LoggerFactory.getLogger(RocketMQCanalConnector.class);
private String nameServer;
private String topic;
private String groupName;
private volatile boolean connected = false;
private DefaultMQPushConsumer rocketMQConsumer;
private BlockingQueue<ConsumerBatchMessage> messageBlockingQueue;
private long batchProcessTimeout = 60 * 1000;
private boolean flatMessage;
private volatile ConsumerBatchMessage lastGetBatchMessage = null;
public RocketMQCanalConnector(String nameServer, String topic, String groupName, boolean flatMessage){
this.nameServer = nameServer;
this.topic = topic;
this.groupName = groupName;
this.flatMessage = flatMessage;
this.messageBlockingQueue = new LinkedBlockingQueue<>(1024);
}
public void connect() throws CanalClientException {
rocketMQConsumer = new DefaultMQPushConsumer(groupName);
if (!StringUtils.isBlank(nameServer)) {
rocketMQConsumer.setNamesrvAddr(nameServer);
}
}
public void disconnect() throws CanalClientException {
rocketMQConsumer.shutdown();
connected = false;
}
public boolean checkValid() throws CanalClientException {
return connected;
}
public synchronized void subscribe(String filter) throws CanalClientException {
if (connected) {
return;
}
try {
if (rocketMQConsumer == null) {
this.connect();
}
rocketMQConsumer.subscribe(this.topic, "*");
rocketMQConsumer.registerMessageListener(new MessageListenerOrderly() {
@Override
public ConsumeOrderlyStatus consumeMessage(List<MessageExt> messageExts, ConsumeOrderlyContext context) {
context.setAutoCommit(true);
boolean isSuccess = process(messageExts);
if (isSuccess) {
return ConsumeOrderlyStatus.SUCCESS;
} else {
return ConsumeOrderlyStatus.SUSPEND_CURRENT_QUEUE_A_MOMENT;
}
}
});
rocketMQConsumer.start();
} catch (MQClientException ex) {
connected = false;
logger.error("Start RocketMQ consumer error", ex);
}
connected = true;
}
private boolean process(List<MessageExt> messageExts) {
logger.info("Get Message:{}", messageExts);
List messageList = Lists.newArrayList();
for (MessageExt messageExt : messageExts) {
byte[] data = messageExt.getBody();
if (data != null) {
try {
if (!flatMessage) {
Message message = CanalMessageDeserializer.deserializer(data);
messageList.add(message);
} else {
FlatMessage flatMessage = JSON.parseObject(data, FlatMessage.class);
messageList.add(flatMessage);
}
} catch (Exception ex) {
logger.error("Add message error", ex);
throw new CanalClientException(ex);
}
} else {
logger.warn("Received message data is null");
}
}
ConsumerBatchMessage batchMessage;
if (!flatMessage) {
batchMessage = new ConsumerBatchMessage<Message>(messageList);
} else {
batchMessage = new ConsumerBatchMessage<FlatMessage>(messageList);
}
try {
messageBlockingQueue.put(batchMessage);
} catch (InterruptedException e) {
logger.error("Put message to queue error", e);
throw new RuntimeException(e);
}
boolean isCompleted;
try {
isCompleted = batchMessage.waitFinish(batchProcessTimeout);
} catch (InterruptedException e) {
logger.error("Interrupted when waiting messages to be finished.", e);
throw new RuntimeException(e);
}
boolean isSuccess = batchMessage.isSuccess();
return isCompleted && isSuccess;
}
public void subscribe() throws CanalClientException {
this.subscribe(null);
}
public void unsubscribe() throws CanalClientException {
this.rocketMQConsumer.unsubscribe(this.topic);
}
@Override
public List<Message> getList(Long timeout, TimeUnit unit) throws CanalClientException {
List<Message> messages = getListWithoutAck(timeout, unit);
if (messages != null && !messages.isEmpty()) {
ack();
}
return messages;
}
@Override
public List<Message> getListWithoutAck(Long timeout, TimeUnit unit) throws CanalClientException {
try {
if (this.lastGetBatchMessage != null) {
throw new CanalClientException("mq get/ack not support concurrent & async ack");
}
ConsumerBatchMessage batchMessage = messageBlockingQueue.poll(timeout, unit);
if (batchMessage != null) {
this.lastGetBatchMessage = batchMessage;
return batchMessage.getData();
}
} catch (InterruptedException ex) {
logger.warn("Get message timeout", ex);
throw new CanalClientException("Failed to fetch the data after: " + timeout);
}
return Lists.newArrayList();
}
@Override
public List<FlatMessage> getFlatList(Long timeout, TimeUnit unit) throws CanalClientException {
List<FlatMessage> messages = getFlatListWithoutAck(timeout, unit);
if (messages != null && !messages.isEmpty()) {
ack();
}
return messages;
}
@Override
public List<FlatMessage> getFlatListWithoutAck(Long timeout, TimeUnit unit) throws CanalClientException {
try {
if (this.lastGetBatchMessage != null) {
throw new CanalClientException("mq get/ack not support concurrent & async ack");
}
ConsumerBatchMessage batchMessage = messageBlockingQueue.poll(timeout, unit);
if (batchMessage != null) {
this.lastGetBatchMessage = batchMessage;
return batchMessage.getData();
}
} catch (InterruptedException ex) {
logger.warn("Get message timeout", ex);
throw new CanalClientException("Failed to fetch the data after: " + timeout);
}
return Lists.newArrayList();
}
@Override
public void ack() throws CanalClientException {
try {
this.lastGetBatchMessage.ack();
} catch (Throwable e) {
this.lastGetBatchMessage.fail();
} finally {
this.lastGetBatchMessage = null;
}
}
@Override
public void rollback() throws CanalClientException {
try {
this.lastGetBatchMessage.fail();
} finally {
this.lastGetBatchMessage = null;
}
}
public Message get(int batchSize) throws CanalClientException {
throw new CanalClientException("mq not support this method");
}
@Override
public Message get(int batchSize, Long timeout, TimeUnit unit) throws CanalClientException {
throw new CanalClientException("mq not support this method");
}
@Override
public Message getWithoutAck(int batchSize) throws CanalClientException {
throw new CanalClientException("mq not support this method");
}
@Override
public Message getWithoutAck(int batchSize, Long timeout, TimeUnit unit) throws CanalClientException {
throw new CanalClientException("mq not support this method");
}
@Override
public void ack(long batchId) throws CanalClientException {
throw new CanalClientException("mq not support this method");
}
@Override
public void rollback(long batchId) throws CanalClientException {
throw new CanalClientException("mq not support this method");
}
}

View File

@@ -1,24 +0,0 @@
package com.alibaba.otter.canal.client.rocketmq;
/**
* RocketMQ connector provider.
*/
public class RocketMQCanalConnectors {
/**
* Create RocketMQ connector
*
* @param nameServers name servers for RocketMQ
* @param topic
* @param groupId
* @return {@link RocketMQCanalConnector}
*/
public static RocketMQCanalConnector newRocketMQConnector(String nameServers, String topic, String groupId) {
return new RocketMQCanalConnector(nameServers, topic, groupId, false);
}
public static RocketMQCanalConnector newRocketMQConnector(String nameServers, String topic, String groupId,
boolean flatMessage) {
return new RocketMQCanalConnector(nameServers, topic, groupId, flatMessage);
}
}

View File

@@ -1,9 +0,0 @@
package com.alibaba.otter.canal.client.running.rocketmq;
public class AbstractRocektMQTest {
public static String topic = "example";
public static String groupId = "group";
public static String nameServers = "localhost:9876";
}

View File

@@ -1,138 +0,0 @@
package com.alibaba.otter.canal.client.running.rocketmq;
import java.util.List;
import java.util.concurrent.TimeUnit;
import org.apache.kafka.common.errors.WakeupException;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
import org.springframework.util.Assert;
import com.alibaba.otter.canal.client.rocketmq.RocketMQCanalConnector;
import com.alibaba.otter.canal.client.rocketmq.RocketMQCanalConnectors;
import com.alibaba.otter.canal.client.running.kafka.AbstractKafkaTest;
import com.alibaba.otter.canal.protocol.Message;
/**
* Kafka client example
*
* @author machengyuan @ 2018-6-12
* @version 1.0.0
*/
public class CanalRocketMQClientExample extends AbstractRocektMQTest {
protected final static Logger logger = LoggerFactory.getLogger(CanalRocketMQClientExample.class);
private RocketMQCanalConnector connector;
private static volatile boolean running = false;
private Thread thread = null;
private Thread.UncaughtExceptionHandler handler = new Thread.UncaughtExceptionHandler() {
public void uncaughtException(Thread t, Throwable e) {
logger.error("parse events has an error", e);
}
};
public CanalRocketMQClientExample(String nameServers, String topic, String groupId){
connector = RocketMQCanalConnectors.newRocketMQConnector(nameServers, topic, groupId);
}
public static void main(String[] args) {
try {
final CanalRocketMQClientExample rocketMQClientExample = new CanalRocketMQClientExample(nameServers,
topic,
groupId);
logger.info("## Start the rocketmq consumer: {}-{}", AbstractKafkaTest.topic, AbstractKafkaTest.groupId);
rocketMQClientExample.start();
logger.info("## The canal rocketmq consumer is running now ......");
Runtime.getRuntime().addShutdownHook(new Thread() {
public void run() {
try {
logger.info("## Stop the rocketmq consumer");
rocketMQClientExample.stop();
} catch (Throwable e) {
logger.warn("## Something goes wrong when stopping rocketmq consumer:", e);
} finally {
logger.info("## Rocketmq consumer is down.");
}
}
});
while (running)
;
} catch (Throwable e) {
logger.error("## Something going wrong when starting up the rocketmq consumer:", e);
System.exit(0);
}
}
public void start() {
Assert.notNull(connector, "connector is null");
thread = new Thread(new Runnable() {
public void run() {
process();
}
});
thread.setUncaughtExceptionHandler(handler);
thread.start();
running = true;
}
public void stop() {
if (!running) {
return;
}
running = false;
if (thread != null) {
try {
thread.join();
} catch (InterruptedException e) {
// ignore
}
}
}
private void process() {
while (!running)
;
while (running) {
try {
connector.connect();
connector.subscribe();
while (running) {
List<Message> messages = connector.getListWithoutAck(100L, TimeUnit.MILLISECONDS); // 获取message
for (Message message : messages) {
long batchId = message.getId();
int size = message.getEntries().size();
if (batchId == -1 || size == 0) {
// try {
// Thread.sleep(1000);
// } catch (InterruptedException e) {
// }
} else {
// printSummary(message, batchId, size);
// printEntry(message.getEntries());
logger.info(message.toString());
}
}
connector.ack(); // 提交确认
}
} catch (Exception e) {
logger.error(e.getMessage(), e);
}
}
try {
connector.unsubscribe();
} catch (WakeupException e) {
// No-op. Continue process
}
// connector.stopRunning();
}
}

View File

@@ -3,7 +3,7 @@
<parent>
<groupId>com.alibaba.otter</groupId>
<artifactId>canal</artifactId>
<version>1.1.1</version>
<version>1.0.26-SNAPSHOT</version>
<relativePath>../pom.xml</relativePath>
</parent>
<artifactId>canal.common</artifactId>
@@ -83,10 +83,6 @@
<groupId>org.springframework</groupId>
<artifactId>spring-orm</artifactId>
</dependency>
<dependency>
<groupId>com.google.code.findbugs</groupId>
<artifactId>jsr305</artifactId>
</dependency>
<dependency>
<groupId>org.springframework</groupId>
<artifactId>spring-test</artifactId>

View File

@@ -5,7 +5,7 @@ import java.util.concurrent.TimeoutException;
import java.util.concurrent.locks.AbstractQueuedSynchronizer;
/**
* 实现一个互斥实现,基于Cocurrent中的AbstractQueuedSynchronizer实现了自己的sync <br/>
* 实现一个互斥实现,基于Cocurrent中的{@linkplain AbstractQueuedSynchronizer}实现了自己的sync <br/>
* 应用场景:系统初始化/授权控制,没权限时阻塞等待。有权限时所有线程都可以快速通过
*
* <pre>

View File

@@ -3,7 +3,7 @@
<parent>
<groupId>com.alibaba.otter</groupId>
<artifactId>canal</artifactId>
<version>1.1.1</version>
<version>1.0.26-SNAPSHOT</version>
<relativePath>../pom.xml</relativePath>
</parent>
<groupId>com.alibaba.otter</groupId>
@@ -11,11 +11,6 @@
<packaging>jar</packaging>
<name>canal dbsync module for otter ${project.version}</name>
<dependencies>
<dependency>
<groupId>com.alibaba.otter</groupId>
<artifactId>canal.parse.driver</artifactId>
<version>${project.version}</version>
</dependency>
<!-- log -->
<dependency>
<groupId>ch.qos.logback</groupId>

View File

@@ -168,9 +168,7 @@ public final class FileLogFetcher extends LogFetcher {
* @see com.taobao.tddl.dbsync.binlog.LogFetcher#close()
*/
public void close() throws IOException {
if (fin != null) {
fin.close();
}
if (fin != null) fin.close();
fin = null;
}

View File

@@ -3,9 +3,7 @@ package com.taobao.tddl.dbsync.binlog;
import java.util.HashMap;
import java.util.Map;
import com.alibaba.otter.canal.parse.driver.mysql.packets.GTIDSet;
import com.taobao.tddl.dbsync.binlog.event.FormatDescriptionLogEvent;
import com.taobao.tddl.dbsync.binlog.event.GtidLogEvent;
import com.taobao.tddl.dbsync.binlog.event.TableMapLogEvent;
/**
@@ -22,10 +20,6 @@ public final class LogContext {
private LogPosition logPosition;
private GTIDSet gtidSet;
private GtidLogEvent gtidLogEvent; // save current gtid log event
public LogContext(){
this.formatDescription = FormatDescriptionLogEvent.FORMAT_DESCRIPTION_EVENT_5_x;
}
@@ -66,20 +60,4 @@ public final class LogContext {
formatDescription = FormatDescriptionLogEvent.FORMAT_DESCRIPTION_EVENT_5_x;
mapOfTable.clear();
}
public GTIDSet getGtidSet() {
return gtidSet;
}
public void setGtidSet(GTIDSet gtidSet) {
this.gtidSet = gtidSet;
}
public GtidLogEvent getGtidLogEvent() {
return gtidLogEvent;
}
public void setGtidLogEvent(GtidLogEvent gtidLogEvent) {
this.gtidLogEvent = gtidLogEvent;
}
}

View File

@@ -6,7 +6,6 @@ import java.util.BitSet;
import org.apache.commons.logging.Log;
import org.apache.commons.logging.LogFactory;
import com.alibaba.otter.canal.parse.driver.mysql.packets.GTIDSet;
import com.taobao.tddl.dbsync.binlog.event.AppendBlockLogEvent;
import com.taobao.tddl.dbsync.binlog.event.BeginLoadQueryLogEvent;
import com.taobao.tddl.dbsync.binlog.event.CreateFileLogEvent;
@@ -47,7 +46,7 @@ import com.taobao.tddl.dbsync.binlog.event.mariadb.StartEncryptionLogEvent;
/**
* Implements a binary-log decoder.
*
*
* <pre>
* LogDecoder decoder = new LogDecoder();
* decoder.handle(...);
@@ -62,7 +61,7 @@ import com.taobao.tddl.dbsync.binlog.event.mariadb.StartEncryptionLogEvent;
* while (event != null);
* // no more events in buffer.
* </pre>
*
*
* @author <a href="mailto:changyuan.lh@taobao.com">Changyuan.lh</a>
* @version 1.0
*/
@@ -89,7 +88,7 @@ public final class LogDecoder {
/**
* Decoding an event from binary-log buffer.
*
*
* @return <code>UknownLogEvent</code> if event type is unknown or skipped,
* <code>null</code> if buffer is not including a full event.
*/
@@ -141,7 +140,7 @@ public final class LogDecoder {
/**
* Deserialize an event from buffer.
*
*
* @return <code>UknownLogEvent</code> if event type is unknown or skipped.
*/
public static LogEvent decode(LogBuffer buffer, LogHeader header, LogContext context) throws IOException {
@@ -160,21 +159,18 @@ public final class LogDecoder {
// remove checksum bytes
buffer.limit(header.getEventLen() - LogEvent.BINLOG_CHECKSUM_LEN);
}
GTIDSet gtidSet = context.getGtidSet();
GtidLogEvent gtidLogEvent = context.getGtidLogEvent();
switch (header.getType()) {
case LogEvent.QUERY_EVENT: {
QueryLogEvent event = new QueryLogEvent(header, buffer, descriptionEvent);
/* updating position in context */
logPosition.position = header.getLogPos();
header.putGtid(context.getGtidSet(), gtidLogEvent);
return event;
}
case LogEvent.XID_EVENT: {
XidLogEvent event = new XidLogEvent(header, buffer, descriptionEvent);
/* updating position in context */
logPosition.position = header.getLogPos();
header.putGtid(context.getGtidSet(), gtidLogEvent);
return event;
}
case LogEvent.TABLE_MAP_EVENT: {
@@ -189,7 +185,6 @@ public final class LogDecoder {
/* updating position in context */
logPosition.position = header.getLogPos();
event.fillTable(context);
header.putGtid(context.getGtidSet(), gtidLogEvent);
return event;
}
case LogEvent.UPDATE_ROWS_EVENT_V1: {
@@ -197,7 +192,6 @@ public final class LogDecoder {
/* updating position in context */
logPosition.position = header.getLogPos();
event.fillTable(context);
header.putGtid(context.getGtidSet(), gtidLogEvent);
return event;
}
case LogEvent.DELETE_ROWS_EVENT_V1: {
@@ -205,7 +199,6 @@ public final class LogDecoder {
/* updating position in context */
logPosition.position = header.getLogPos();
event.fillTable(context);
header.putGtid(context.getGtidSet(), gtidLogEvent);
return event;
}
case LogEvent.ROTATE_EVENT: {
@@ -275,14 +268,12 @@ public final class LogDecoder {
RandLogEvent event = new RandLogEvent(header, buffer, descriptionEvent);
/* updating position in context */
logPosition.position = header.getLogPos();
header.putGtid(context.getGtidSet(), gtidLogEvent);
return event;
}
case LogEvent.USER_VAR_EVENT: {
UserVarLogEvent event = new UserVarLogEvent(header, buffer, descriptionEvent);
/* updating position in context */
logPosition.position = header.getLogPos();
header.putGtid(context.getGtidSet(), gtidLogEvent);
return event;
}
case LogEvent.FORMAT_DESCRIPTION_EVENT: {
@@ -345,7 +336,6 @@ public final class LogDecoder {
RowsQueryLogEvent event = new RowsQueryLogEvent(header, buffer, descriptionEvent);
/* updating position in context */
logPosition.position = header.getLogPos();
header.putGtid(context.getGtidSet(), gtidLogEvent);
return event;
}
case LogEvent.WRITE_ROWS_EVENT: {
@@ -353,7 +343,6 @@ public final class LogDecoder {
/* updating position in context */
logPosition.position = header.getLogPos();
event.fillTable(context);
header.putGtid(context.getGtidSet(), gtidLogEvent);
return event;
}
case LogEvent.UPDATE_ROWS_EVENT: {
@@ -361,7 +350,6 @@ public final class LogDecoder {
/* updating position in context */
logPosition.position = header.getLogPos();
event.fillTable(context);
header.putGtid(context.getGtidSet(), gtidLogEvent);
return event;
}
case LogEvent.DELETE_ROWS_EVENT: {
@@ -369,7 +357,6 @@ public final class LogDecoder {
/* updating position in context */
logPosition.position = header.getLogPos();
event.fillTable(context);
header.putGtid(context.getGtidSet(), gtidLogEvent);
return event;
}
case LogEvent.GTID_LOG_EVENT:
@@ -377,13 +364,6 @@ public final class LogDecoder {
GtidLogEvent event = new GtidLogEvent(header, buffer, descriptionEvent);
/* updating position in context */
logPosition.position = header.getLogPos();
if (gtidSet != null) {
gtidSet.update(event.getGtidStr());
// update latest gtid
header.putGtid(gtidSet, event);
}
// update current gtid event to context
context.setGtidLogEvent(event);
return event;
}
case LogEvent.PREVIOUS_GTIDS_LOG_EVENT: {
@@ -414,7 +394,6 @@ public final class LogDecoder {
AnnotateRowsEvent event = new AnnotateRowsEvent(header, buffer, descriptionEvent);
/* updating position in context */
logPosition.position = header.getLogPos();
header.putGtid(context.getGtidSet(), gtidLogEvent);
return event;
}
case LogEvent.BINLOG_CHECKPOINT_EVENT: {

View File

@@ -14,10 +14,10 @@ import com.taobao.tddl.dbsync.binlog.LogEvent;
public class GtidLogEvent extends LogEvent {
// / Length of the commit_flag in event encoding
public static final int ENCODED_FLAG_LENGTH = 1;
public static final int ENCODED_FLAG_LENGTH = 1;
// / Length of SID in event encoding
public static final int ENCODED_SID_LENGTH = 16;
public static final int LOGICAL_TIMESTAMP_TYPE_CODE = 2;
public static final int ENCODED_SID_LENGTH = 16;
public static final int LOGICAL_TIMESTAMP_TYPE_CODE = 2;
private boolean commitFlag;
private UUID sid;
@@ -50,6 +50,8 @@ public class GtidLogEvent extends LogEvent {
sequenceNumber = buffer.getLong64();
}
// ignore gtid info read
// sid.copy_from((uchar *)ptr_buffer);
// ptr_buffer+= ENCODED_SID_LENGTH;
@@ -80,11 +82,4 @@ public class GtidLogEvent extends LogEvent {
public Long getSequenceNumber() {
return sequenceNumber;
}
public String getGtidStr() {
StringBuilder sb = new StringBuilder();
sb.append(sid.toString()).append(":");
sb.append(gno);
return sb.toString();
}
}

View File

@@ -1,12 +1,8 @@
package com.taobao.tddl.dbsync.binlog.event;
import com.alibaba.otter.canal.parse.driver.mysql.packets.GTIDSet;
import com.taobao.tddl.dbsync.binlog.LogBuffer;
import com.taobao.tddl.dbsync.binlog.LogEvent;
import java.util.HashMap;
import java.util.Map;
/**
* The Common-Header, documented in the table @ref Table_common_header "below",
* always has the same form and length within one version of MySQL. Each event
@@ -124,13 +120,6 @@ public final class LogHeader {
*/
protected String logFileName;
protected Map<String, String> gtidMap = new HashMap<>();
private static final String CURRENT_GTID_STRING = "curt_gtid";
private static final String CURRENT_GTID_SN = "curt_gtid_sn";
private static final String CURRENT_GTID_LAST_COMMIT = "curt_gtid_lct";
private static final String GTID_SET_STRING = "gtid_str";
/* for Start_event_v3 */
public LogHeader(final int type){
this.type = type;
@@ -299,31 +288,4 @@ public final class LogHeader {
crc = buffer.getUint32(eventLen - LogEvent.BINLOG_CHECKSUM_LEN);
}
}
public String getGtidSetStr() {
return gtidMap.get(GTID_SET_STRING);
}
public String getCurrentGtid() {
return gtidMap.get(CURRENT_GTID_STRING);
}
public String getCurrentGtidSn() {
return gtidMap.get(CURRENT_GTID_SN);
}
public String getCurrentGtidLastCommit() {
return gtidMap.get(CURRENT_GTID_LAST_COMMIT);
}
public void putGtid(GTIDSet gtidSet, GtidLogEvent event) {
if (gtidSet != null) {
gtidMap.put(GTID_SET_STRING, gtidSet.toString());
if (event != null) {
gtidMap.put(CURRENT_GTID_STRING, event.getGtidStr());
gtidMap.put(CURRENT_GTID_SN, String.valueOf(event.getSequenceNumber()));
gtidMap.put(CURRENT_GTID_LAST_COMMIT, String.valueOf(event.getLastCommitted()));
}
}
}
}

View File

@@ -367,7 +367,7 @@ public final class RowsLogBuffer {
// 转化为unsign long
switch (len) {
case 1:
value = buffer.getUint8();
value = buffer.getInt8();
break;
case 2:
value = buffer.getBeUint16();
@@ -859,7 +859,7 @@ public final class RowsLogBuffer {
// 转化为unsign long
switch (len) {
case 1:
value = buffer.getUint8();
value = buffer.getInt8();
break;
case 2:
value = buffer.getUint16();
@@ -1082,10 +1082,10 @@ public final class RowsLogBuffer {
buffer.fillBytes(binary, 0, len);
/* Warning unsupport cloumn type */
// logger.warn(String.format("!! Unsupport column type MYSQL_TYPE_GEOMETRY: meta=%d (%04X), len = %d",
// meta,
// meta,
// len));
logger.warn(String.format("!! Unsupport column type MYSQL_TYPE_GEOMETRY: meta=%d (%04X), len = %d",
meta,
meta,
len));
javaType = Types.BINARY;
value = binary;
length = len;

View File

@@ -15,7 +15,9 @@ public class FetcherPerformanceTest {
DirectLogFetcher fetcher = new DirectLogFetcher();
try {
Class.forName("com.mysql.jdbc.Driver");
Connection connection = DriverManager.getConnection("jdbc:mysql://127.0.0.1:3306", "root", "hello");
Connection connection = DriverManager.getConnection("jdbc:mysql://127.0.0.1:3306",
"root",
"hello");
Statement statement = connection.createStatement();
statement.execute("SET @master_binlog_checksum='@@global.binlog_checksum'");
statement.execute("SET @mariadb_slave_capability='" + LogEvent.MARIA_SLAVE_CAPABILITY_MINE + "'");

View File

@@ -8,15 +8,6 @@ import java.sql.Statement;
import org.junit.Assert;
import org.junit.Test;
import com.taobao.tddl.dbsync.binlog.event.DeleteRowsLogEvent;
import com.taobao.tddl.dbsync.binlog.event.QueryLogEvent;
import com.taobao.tddl.dbsync.binlog.event.RotateLogEvent;
import com.taobao.tddl.dbsync.binlog.event.RowsQueryLogEvent;
import com.taobao.tddl.dbsync.binlog.event.UpdateRowsLogEvent;
import com.taobao.tddl.dbsync.binlog.event.WriteRowsLogEvent;
import com.taobao.tddl.dbsync.binlog.event.XidLogEvent;
import com.taobao.tddl.dbsync.binlog.event.mariadb.AnnotateRowsEvent;
public class DirectLogFetcherTest extends BaseLogFetcherTest {
@Test
@@ -24,49 +15,54 @@ public class DirectLogFetcherTest extends BaseLogFetcherTest {
DirectLogFetcher fecther = new DirectLogFetcher();
try {
Class.forName("com.mysql.jdbc.Driver");
Connection connection = DriverManager.getConnection("jdbc:mysql://127.0.0.1:3306", "root", "hello");
Connection connection = DriverManager.getConnection("jdbc:mysql://100.81.154.142:3306", "root", "hello");
Statement statement = connection.createStatement();
statement.execute("SET @master_binlog_checksum='@@global.binlog_checksum'");
statement.execute("SET @mariadb_slave_capability='" + LogEvent.MARIA_SLAVE_CAPABILITY_MINE + "'");
fecther.open(connection, "mysql-bin.000007", 89797036L, 2);
fecther.open(connection, "mysql-bin.000006", 120L, 2);
LogDecoder decoder = new LogDecoder(LogEvent.UNKNOWN_EVENT, LogEvent.ENUM_END_EVENT);
LogDecoder decoder = new LogDecoder(LogEvent.UNKNOWN_EVENT, LogEvent.UNKNOWN_EVENT);
LogContext context = new LogContext();
while (fecther.fetch()) {
LogEvent event = decoder.decode(fecther, context);
int eventType = event.getHeader().getType();
switch (eventType) {
case LogEvent.ROTATE_EVENT:
binlogFileName = ((RotateLogEvent) event).getFilename();
break;
case LogEvent.WRITE_ROWS_EVENT_V1:
case LogEvent.WRITE_ROWS_EVENT:
parseRowsEvent((WriteRowsLogEvent) event);
break;
case LogEvent.UPDATE_ROWS_EVENT_V1:
case LogEvent.UPDATE_ROWS_EVENT:
parseRowsEvent((UpdateRowsLogEvent) event);
break;
case LogEvent.DELETE_ROWS_EVENT_V1:
case LogEvent.DELETE_ROWS_EVENT:
parseRowsEvent((DeleteRowsLogEvent) event);
break;
case LogEvent.QUERY_EVENT:
parseQueryEvent((QueryLogEvent) event);
break;
case LogEvent.ROWS_QUERY_LOG_EVENT:
parseRowsQueryEvent((RowsQueryLogEvent) event);
break;
case LogEvent.ANNOTATE_ROWS_EVENT:
parseAnnotateRowsEvent((AnnotateRowsEvent) event);
break;
case LogEvent.XID_EVENT:
parseXidEvent((XidLogEvent) event);
break;
default:
break;
}
decoder.decode(fecther, context);
continue;
// if (event == null) {
// continue;
// }
//
// int eventType = event.getHeader().getType();
// switch (eventType) {
// case LogEvent.ROTATE_EVENT:
// binlogFileName = ((RotateLogEvent) event).getFilename();
// break;
// case LogEvent.WRITE_ROWS_EVENT_V1:
// case LogEvent.WRITE_ROWS_EVENT:
// parseRowsEvent((WriteRowsLogEvent) event);
// break;
// case LogEvent.UPDATE_ROWS_EVENT_V1:
// case LogEvent.UPDATE_ROWS_EVENT:
// parseRowsEvent((UpdateRowsLogEvent) event);
// break;
// case LogEvent.DELETE_ROWS_EVENT_V1:
// case LogEvent.DELETE_ROWS_EVENT:
// parseRowsEvent((DeleteRowsLogEvent) event);
// break;
// case LogEvent.QUERY_EVENT:
// parseQueryEvent((QueryLogEvent) event);
// break;
// case LogEvent.ROWS_QUERY_LOG_EVENT:
// parseRowsQueryEvent((RowsQueryLogEvent) event);
// break;
// case LogEvent.ANNOTATE_ROWS_EVENT:
// parseAnnotateRowsEvent((AnnotateRowsEvent) event);
// break;
// case LogEvent.XID_EVENT:
// parseXidEvent((XidLogEvent) event);
// break;
// default:
// break;
// }
}
} catch (Exception e) {
e.printStackTrace();

View File

@@ -3,7 +3,7 @@
<parent>
<groupId>com.alibaba.otter</groupId>
<artifactId>canal</artifactId>
<version>1.1.1</version>
<version>1.0.26-SNAPSHOT</version>
<relativePath>../pom.xml</relativePath>
</parent>
<groupId>com.alibaba.otter</groupId>
@@ -40,7 +40,6 @@
<exclude>**/canal.properties</exclude>
<exclude>**/spring/**</exclude>
<exclude>**/example/**</exclude>
<exclude>**/mq.yml</exclude>
</excludes>
</configuration>
</plugin>

View File

@@ -15,7 +15,6 @@ public class CanalConstants {
public static final String CANAL_ID = ROOT + "." + "id";
public static final String CANAL_IP = ROOT + "." + "ip";
public static final String CANAL_PORT = ROOT + "." + "port";
public static final String CANAL_METRICS_PULL_PORT = ROOT + "." + "metrics.pull.port";
public static final String CANAL_ZKSERVERS = ROOT + "." + "zkServers";
public static final String CANAL_WITHOUT_NETTY = ROOT + "." + "withoutNetty";
@@ -23,7 +22,6 @@ public class CanalConstants {
public static final String CANAL_AUTO_SCAN = ROOT + "." + "auto.scan";
public static final String CANAL_AUTO_SCAN_INTERVAL = ROOT + "." + "auto.scan.interval";
public static final String CANAL_CONF_DIR = ROOT + "." + "conf.dir";
public static final String CANAL_SERVER_MODE = ROOT + "." + "serverMode";
public static final String CANAL_DESTINATION_SPLIT = ",";
public static final String GLOBAL_NAME = "global";

View File

@@ -98,14 +98,6 @@ public class CanalController {
port = Integer.valueOf(getProperty(properties, CanalConstants.CANAL_PORT));
embededCanalServer = CanalServerWithEmbedded.instance();
embededCanalServer.setCanalInstanceGenerator(instanceGenerator);// 设置自定义的instanceGenerator
try {
int metricsPort = Integer.valueOf(getProperty(properties, CanalConstants.CANAL_METRICS_PULL_PORT));
embededCanalServer.setMetricsPort(metricsPort);
} catch (NumberFormatException e) {
logger.info("No valid metrics server port found, use default 11112.");
embededCanalServer.setMetricsPort(11112);
}
String canalWithoutNetty = getProperty(properties, CanalConstants.CANAL_WITHOUT_NETTY);
if (canalWithoutNetty == null || "false".equals(canalWithoutNetty)) {
canalServer = CanalServerWithNetty.instance();
@@ -387,7 +379,7 @@ public class CanalController {
return config;
}
public static String getProperty(Properties properties, String key) {
private String getProperty(Properties properties, String key) {
key = StringUtils.trim(key);
String value = System.getProperty(key);

View File

@@ -7,14 +7,9 @@ import org.apache.commons.lang.StringUtils;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
import com.alibaba.otter.canal.kafka.CanalKafkaProducer;
import com.alibaba.otter.canal.rocketmq.CanalRocketMQProducer;
import com.alibaba.otter.canal.server.CanalMQStarter;
import com.alibaba.otter.canal.spi.CanalMQProducer;
/**
* canal独立版本启动的入口类
*
*
* @author jianghang 2012-11-6 下午05:20:49
* @version 1.0.0
*/
@@ -38,19 +33,6 @@ public class CanalLauncher {
properties.load(new FileInputStream(conf));
}
CanalMQProducer canalMQProducer = null;
String serverMode = CanalController.getProperty(properties, CanalConstants.CANAL_SERVER_MODE);
if (serverMode.equalsIgnoreCase("kafka")) {
canalMQProducer = new CanalKafkaProducer();
} else if (serverMode.equalsIgnoreCase("rocketmq")) {
canalMQProducer = new CanalRocketMQProducer();
}
if (canalMQProducer != null) {
// disable netty
System.setProperty(CanalConstants.CANAL_WITHOUT_NETTY, "true");
}
logger.info("## start the canal server.");
final CanalController controller = new CanalController(properties);
controller.start();
@@ -69,13 +51,6 @@ public class CanalLauncher {
}
});
if (canalMQProducer != null) {
CanalMQStarter canalServerStarter = new CanalMQStarter(canalMQProducer);
if (canalServerStarter != null) {
canalServerStarter.init();
}
}
} catch (Throwable e) {
logger.error("## Something goes wrong when starting up the canal Server:", e);
System.exit(0);

View File

@@ -4,13 +4,10 @@
canal.id= 1
canal.ip=
canal.port=11111
canal.metrics.pull.port=11112
canal.zkServers=
# flush data to zk
canal.zookeeper.flush.period = 1000
canal.withoutNetty = false
# tcp, kafka, RocketMQ
canal.serverMode = tcp
# flush meta cursor/parse position to file
canal.file.data.dir = ${canal.conf.dir}
canal.file.flush.period = 1000
@@ -20,7 +17,6 @@ canal.instance.memory.buffer.size = 16384
canal.instance.memory.buffer.memunit = 1024
## meory store gets mode used MEMSIZE or ITEMSIZE
canal.instance.memory.batch.mode = MEMSIZE
canal.instance.memory.rawEntry = true
## detecing config
canal.instance.detecting.enable = false
@@ -69,14 +65,6 @@ canal.instance.tsdb.dir=${canal.file.data.dir:../conf}/${canal.instance.destinat
canal.instance.tsdb.url=jdbc:h2:${canal.instance.tsdb.dir}/h2;CACHE_SIZE=1000;MODE=MYSQL;
canal.instance.tsdb.dbUsername=canal
canal.instance.tsdb.dbPassword=canal
# dump snapshot interval, default 24 hour
canal.instance.tsdb.snapshot.interval=24
# purge snapshot expire , default 360 hour(15 days)
canal.instance.tsdb.snapshot.expire=360
# rds oss binlog account
canal.instance.rds.accesskey =
canal.instance.rds.secretkey =
#################################################
######### destinations #############
@@ -94,6 +82,7 @@ canal.instance.tsdb.spring.xml=classpath:spring/tsdb/h2-tsdb.xml
canal.instance.global.mode = spring
canal.instance.global.lazy = false
#canal.instance.global.manager.address = 127.0.0.1:1099
#canal.instance.global.spring.xml = classpath:spring/local-instance.xml
#canal.instance.global.spring.xml = classpath:spring/memory-instance.xml
canal.instance.global.spring.xml = classpath:spring/file-instance.xml
#canal.instance.global.spring.xml = classpath:spring/default-instance.xml

View File

@@ -12,11 +12,6 @@ canal.instance.master.position=
canal.instance.master.timestamp=
canal.instance.master.gtid=
# rds oss binlog
canal.instance.rds.accesskey=
canal.instance.rds.secretkey=
canal.instance.rds.instanceId=
# table meta tsdb info
canal.instance.tsdb.enable=true
#canal.instance.tsdb.url=jdbc:mysql://127.0.0.1:3306/canal_tsdb
@@ -25,18 +20,14 @@ canal.instance.tsdb.enable=true
#canal.instance.standby.address =
#canal.instance.standby.journal.name =
#canal.instance.standby.position =
#canal.instance.standby.position =
#canal.instance.standby.timestamp =
#canal.instance.standby.gtid=
# username/password
canal.instance.dbUsername=canal
canal.instance.dbPassword=cZozNf1mzW6EQLGO2q9u99619xbZLO0fbua3EX08r4BWNXb8lAt1aHrTEOBttd6UY8Vnuc0easlVXZDdLtt8BQ==
canal.instance.pwdPublicKey=MFwwDQYJKoZIhvcNAQEBBQADSwAwSAJBALK4BUxdDltRRE5/zXpVEVPUgunvscYFtEip3pmLlhrWpacX7y7GCMo2/JM6LeHmiiNdH1FWgGCpUfircSwlWKUCAwEAAQ==
canal.instance.connectionCharset = UTF-8
canal.instance.defaultDatabaseName =test
# enable druid Decrypt database password
canal.instance.enableDruid=true
canal.instance.dbPassword=canal
canal.instance.connectionCharset=UTF-8
# table regex
canal.instance.filter.regex=.*\\..*

View File

@@ -0,0 +1,32 @@
#################################################
# rds openapi binlog
canal.instance.rds.open.url=https://rds.aliyuncs.com/
canal.instance.rds.open.accesskey=
canal.instance.rds.open.secretkey=
canal.instance.rds.instanceId=
canal.instance.rds.startTime=
canal.instance.rds.endTime=
# local binlog dir
canal.instance.parser.directory=${canal.file.data.dir:../conf}/${canal.instance.destination:}/binlog
# position info
canal.instance.master.address=127.0.0.1:3306
# table meta tsdb info
canal.instance.tsdb.enable=true
canal.instance.tsdb.dir=${canal.file.data.dir:../conf}/${canal.instance.destination:}
canal.instance.tsdb.url=jdbc:h2:${canal.instance.tsdb.dir}/h2;CACHE_SIZE=1000;MODE=MYSQL;
#canal.instance.tsdb.url=jdbc:mysql://127.0.0.1:3306/canal_tsdb
#canal.instance.tsdb.dbUsername=canal
#canal.instance.tsdb.dbPassword=canal
# username/password
canal.instance.dbUsername=canal
canal.instance.dbPassword=canal
canal.instance.defaultDatabaseName=test
canal.instance.connectionCharset=UTF-8
# table regex
canal.instance.filter.regex=test\\..*
# table black regex
canal.instance.filter.black.regex=
#################################################

View File

@@ -6,7 +6,7 @@
</pattern>
</encoder>
</appender>
<appender name="CANAL-ROOT" class="ch.qos.logback.classic.sift.SiftingAppender">
<discriminator>
<Key>destination</Key>
@@ -16,7 +16,7 @@
<appender name="FILE-${destination}" class="ch.qos.logback.core.rolling.RollingFileAppender">
<File>../logs/${destination}/${destination}.log</File>
<rollingPolicy
class="ch.qos.logback.core.rolling.TimeBasedRollingPolicy">
class="ch.qos.logback.core.rolling.TimeBasedRollingPolicy">
<!-- rollover daily -->
<fileNamePattern>../logs/${destination}/%d{yyyy-MM-dd}/${destination}-%d{yyyy-MM-dd}-%i.log.gz</fileNamePattern>
<timeBasedFileNamingAndTriggeringPolicy class="ch.qos.logback.core.rolling.SizeAndTimeBasedFNATP">
@@ -33,7 +33,7 @@
</appender>
</sift>
</appender>
<appender name="CANAL-META" class="ch.qos.logback.classic.sift.SiftingAppender">
<discriminator>
<Key>destination</Key>
@@ -43,7 +43,7 @@
<appender name="META-FILE-${destination}" class="ch.qos.logback.core.rolling.RollingFileAppender">
<File>../logs/${destination}/meta.log</File>
<rollingPolicy
class="ch.qos.logback.core.rolling.TimeBasedRollingPolicy">
class="ch.qos.logback.core.rolling.TimeBasedRollingPolicy">
<!-- rollover daily -->
<fileNamePattern>../logs/${destination}/%d{yyyy-MM-dd}/meta-%d{yyyy-MM-dd}-%i.log.gz</fileNamePattern>
<timeBasedFileNamingAndTriggeringPolicy class="ch.qos.logback.core.rolling.SizeAndTimeBasedFNATP">
@@ -60,24 +60,24 @@
</appender>
</sift>
</appender>
<logger name="com.alibaba.otter.canal.instance" additivity="false">
<level value="INFO" />
<appender-ref ref="CANAL-ROOT" />
</logger>
<logger name="com.alibaba.otter.canal.deployer" additivity="false">
<level value="INFO" />
<appender-ref ref="CANAL-ROOT" />
</logger>
<logger name="com.alibaba.otter.canal.meta.FileMixedMetaManager" additivity="false">
<level value="INFO" />
<appender-ref ref="CANAL-META" />
</logger>
<logger name="com.alibaba.otter.canal.instance" additivity="false">
<level value="INFO" />
<appender-ref ref="CANAL-ROOT" />
</logger>
<logger name="com.alibaba.otter.canal.deployer" additivity="false">
<level value="INFO" />
<appender-ref ref="CANAL-ROOT" />
</logger>
<logger name="com.alibaba.otter.canal.meta.FileMixedMetaManager" additivity="false">
<level value="INFO" />
<appender-ref ref="CANAL-META" />
</logger>
<logger name="com.alibaba.otter.canal.kafka" additivity="false">
<level value="INFO" />
<appender-ref ref="CANAL-ROOT" />
</logger>
<root level="WARN">
<appender-ref ref="STDOUT"/>
<appender-ref ref="CANAL-ROOT" />

File diff suppressed because it is too large Load Diff

View File

@@ -1,23 +0,0 @@
servers: localhost:9876 #for rocketmq: means the nameserver
retries: 0
batchSize: 16384
lingerMs: 1
bufferMemory: 33554432
# Canal的batch size, 默认50K, 由于kafka最大消息体限制请勿超过1M(900K以下)
canalBatchSize: 50
# Canal get数据的超时时间, 单位: 毫秒, 0为不限超时
canalGetTimeout: 100
flatMessage: true
canalDestinations:
- canalDestination: example
topic: example
partition: 1
# #对应topic分区数量
# partitionsNum: 3
# partitionHash:
# #库名.表名: 唯一主键
# mytest.person: id

View File

@@ -1,39 +0,0 @@
<?xml version="1.0" encoding="UTF-8"?>
<beans xmlns="http://www.springframework.org/schema/beans"
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xmlns:tx="http://www.springframework.org/schema/tx"
xmlns:aop="http://www.springframework.org/schema/aop" xmlns:lang="http://www.springframework.org/schema/lang"
xmlns:context="http://www.springframework.org/schema/context"
xsi:schemaLocation="http://www.springframework.org/schema/beans http://www.springframework.org/schema/beans/spring-beans-2.0.xsd
http://www.springframework.org/schema/aop http://www.springframework.org/schema/aop/spring-aop-2.0.xsd
http://www.springframework.org/schema/lang http://www.springframework.org/schema/lang/spring-lang-2.0.xsd
http://www.springframework.org/schema/tx http://www.springframework.org/schema/tx/spring-tx-2.0.xsd
http://www.springframework.org/schema/context http://www.springframework.org/schema/context/spring-context-2.5.xsd"
default-autowire="byName">
<!-- properties -->
<bean class="com.alibaba.otter.canal.instance.spring.support.PropertyPlaceholderConfigurer" lazy-init="false">
<property name="ignoreResourceNotFound" value="true" />
<property name="systemPropertiesModeName" value="SYSTEM_PROPERTIES_MODE_OVERRIDE"/><!-- 允许system覆盖 -->
<property name="locationNames">
<list>
<value>classpath:canal.properties</value>
<value>classpath:${canal.instance.destination:}/instance.properties</value>
</list>
</property>
</bean>
<bean id="socketAddressEditor" class="com.alibaba.otter.canal.instance.spring.support.SocketAddressEditor" />
<bean class="org.springframework.beans.factory.config.CustomEditorConfigurer">
<property name="propertyEditorRegistrars">
<list>
<ref bean="socketAddressEditor" />
</list>
</property>
</bean>
<bean id="baseEventParser" class="com.alibaba.otter.canal.parse.inbound.mysql.rds.RdsBinlogEventParserProxy" abstract="true">
<property name="accesskey" value="${canal.instance.rds.accesskey:}" />
<property name="secretkey" value="${canal.instance.rds.secretkey:}" />
<property name="instanceId" value="${canal.instance.rds.instanceId:}" />
</bean>
</beans>

View File

@@ -10,7 +10,26 @@
http://www.springframework.org/schema/context http://www.springframework.org/schema/context/spring-context-2.5.xsd"
default-autowire="byName">
<import resource="classpath:spring/base-instance.xml" />
<!-- properties -->
<bean class="com.alibaba.otter.canal.instance.spring.support.PropertyPlaceholderConfigurer" lazy-init="false">
<property name="ignoreResourceNotFound" value="true" />
<property name="systemPropertiesModeName" value="SYSTEM_PROPERTIES_MODE_OVERRIDE"/><!-- 允许system覆盖 -->
<property name="locationNames">
<list>
<value>classpath:canal.properties</value>
<value>classpath:${canal.instance.destination:}/instance.properties</value>
</list>
</property>
</bean>
<bean id="socketAddressEditor" class="com.alibaba.otter.canal.instance.spring.support.SocketAddressEditor" />
<bean class="org.springframework.beans.factory.config.CustomEditorConfigurer">
<property name="propertyEditorRegistrars">
<list>
<ref bean="socketAddressEditor" />
</list>
</property>
</bean>
<bean id="instance" class="com.alibaba.otter.canal.instance.spring.CanalInstanceWithSpring">
<property name="destination" value="${canal.instance.destination}" />
@@ -58,7 +77,6 @@
<property name="bufferMemUnit" value="${canal.instance.memory.buffer.memunit:1024}" />
<property name="batchMode" value="${canal.instance.memory.batch.mode:MEMSIZE}" />
<property name="ddlIsolation" value="${canal.instance.get.ddl.isolation:false}" />
<property name="raw" value="${canal.instance.memory.rawEntry:true}" />
</bean>
<bean id="eventSink" class="com.alibaba.otter.canal.sink.entry.EntryEventSink">
@@ -66,7 +84,7 @@
<property name="filterTransactionEntry" value="${canal.instance.filter.transaction.entry:false}"/>
</bean>
<bean id="eventParser" parent="baseEventParser" >
<bean id="eventParser" class="com.alibaba.otter.canal.parse.inbound.mysql.MysqlEventParser">
<property name="destination" value="${canal.instance.destination}" />
<property name="slaveId" value="${canal.instance.mysql.slaveId:0}" />
<!-- 心跳配置 -->
@@ -127,22 +145,18 @@
<!-- 解析数据库信息 -->
<property name="masterInfo">
<bean class="com.alibaba.otter.canal.parse.support.AuthenticationInfo" init-method="initPwd">
<bean class="com.alibaba.otter.canal.parse.support.AuthenticationInfo">
<property name="address" value="${canal.instance.master.address}" />
<property name="username" value="${canal.instance.dbUsername:retl}" />
<property name="password" value="${canal.instance.dbPassword:retl}" />
<property name="pwdPublicKey" value="${canal.instance.pwdPublicKey:retl}" />
<property name="enableDruid" value="${canal.instance.enableDruid:false}" />
<property name="defaultDatabaseName" value="${canal.instance.defaultDatabaseName:test}" />
</bean>
</property>
<property name="standbyInfo">
<bean class="com.alibaba.otter.canal.parse.support.AuthenticationInfo" init-method="initPwd">
<bean class="com.alibaba.otter.canal.parse.support.AuthenticationInfo">
<property name="address" value="${canal.instance.standby.address}" />
<property name="username" value="${canal.instance.dbUsername:retl}" />
<property name="password" value="${canal.instance.dbPassword:retl}" />
<property name="pwdPublicKey" value="${canal.instance.pwdPublicKey:retl}" />
<property name="enableDruid" value="${canal.instance.enableDruid:false}" />
<property name="defaultDatabaseName" value="${canal.instance.defaultDatabaseName:test}" />
</bean>
</property>
@@ -176,8 +190,6 @@
<!--表结构相关-->
<property name="enableTsdb" value="${canal.instance.tsdb.enable:true}"/>
<property name="tsdbSpringXml" value="${canal.instance.tsdb.spring.xml:}"/>
<property name="tsdbSnapshotInterval" value="${canal.instance.tsdb.snapshot.interval:24}" />
<property name="tsdbSnapshotExpire" value="${canal.instance.tsdb.snapshot.expire:360}" />
<!--是否启用GTID模式-->
<property name="isGTIDMode" value="${canal.instance.gtidon:false}"/>

View File

@@ -10,7 +10,26 @@
http://www.springframework.org/schema/context http://www.springframework.org/schema/context/spring-context-2.5.xsd"
default-autowire="byName">
<import resource="classpath:spring/base-instance.xml" />
<!-- properties -->
<bean class="com.alibaba.otter.canal.instance.spring.support.PropertyPlaceholderConfigurer" lazy-init="false">
<property name="ignoreResourceNotFound" value="true" />
<property name="systemPropertiesModeName" value="SYSTEM_PROPERTIES_MODE_OVERRIDE"/><!-- 允许system覆盖 -->
<property name="locationNames">
<list>
<value>classpath:canal.properties</value>
<value>classpath:${canal.instance.destination:}/instance.properties</value>
</list>
</property>
</bean>
<bean id="socketAddressEditor" class="com.alibaba.otter.canal.instance.spring.support.SocketAddressEditor" />
<bean class="org.springframework.beans.factory.config.CustomEditorConfigurer">
<property name="propertyEditorRegistrars">
<list>
<ref bean="socketAddressEditor" />
</list>
</property>
</bean>
<bean id="instance" class="com.alibaba.otter.canal.instance.spring.CanalInstanceWithSpring">
<property name="destination" value="${canal.instance.destination}" />
@@ -44,7 +63,6 @@
<property name="bufferMemUnit" value="${canal.instance.memory.buffer.memunit:1024}" />
<property name="batchMode" value="${canal.instance.memory.batch.mode:MEMSIZE}" />
<property name="ddlIsolation" value="${canal.instance.get.ddl.isolation:false}" />
<property name="raw" value="${canal.instance.memory.rawEntry:true}" />
</bean>
<bean id="eventSink" class="com.alibaba.otter.canal.sink.entry.EntryEventSink">
@@ -52,7 +70,7 @@
<property name="filterTransactionEntry" value="${canal.instance.filter.transaction.entry:false}"/>
</bean>
<bean id="eventParser" parent="baseEventParser">
<bean id="eventParser" class="com.alibaba.otter.canal.parse.inbound.mysql.MysqlEventParser">
<property name="destination" value="${canal.instance.destination}" />
<property name="slaveId" value="${canal.instance.mysql.slaveId:0}" />
<!-- 心跳配置 -->
@@ -112,26 +130,22 @@
<!-- 解析数据库信息 -->
<property name="masterInfo">
<bean class="com.alibaba.otter.canal.parse.support.AuthenticationInfo" init-method="initPwd">
<bean class="com.alibaba.otter.canal.parse.support.AuthenticationInfo">
<property name="address" value="${canal.instance.master.address}" />
<property name="username" value="${canal.instance.dbUsername:retl}" />
<property name="password" value="${canal.instance.dbPassword:retl}" />
<property name="pwdPublicKey" value="${canal.instance.pwdPublicKey:retl}" />
<property name="enableDruid" value="${canal.instance.enableDruid:false}" />
<property name="defaultDatabaseName" value="${canal.instance.defaultDatabaseName:test}" />
</bean>
</property>
<property name="standbyInfo">
<bean class="com.alibaba.otter.canal.parse.support.AuthenticationInfo" init-method="initPwd">
<bean class="com.alibaba.otter.canal.parse.support.AuthenticationInfo">
<property name="address" value="${canal.instance.standby.address}" />
<property name="username" value="${canal.instance.dbUsername:retl}" />
<property name="password" value="${canal.instance.dbPassword:retl}" />
<property name="pwdPublicKey" value="${canal.instance.pwdPublicKey:retl}" />
<property name="enableDruid" value="${canal.instance.enableDruid:false}" />
<property name="defaultDatabaseName" value="${canal.instance.defaultDatabaseName:test}" />
</bean>
</property>
<!-- 解析起始位点 -->
<property name="masterPosition">
<bean class="com.alibaba.otter.canal.protocol.position.EntryPosition">
@@ -161,8 +175,6 @@
<!--表结构相关-->
<property name="enableTsdb" value="${canal.instance.tsdb.enable:true}"/>
<property name="tsdbSpringXml" value="${canal.instance.tsdb.spring.xml:}"/>
<property name="tsdbSnapshotInterval" value="${canal.instance.tsdb.snapshot.interval:24}" />
<property name="tsdbSnapshotExpire" value="${canal.instance.tsdb.snapshot.expire:360}" />
<!--是否启用GTID模式-->
<property name="isGTIDMode" value="${canal.instance.gtidon:false}"/>

View File

@@ -10,7 +10,26 @@
http://www.springframework.org/schema/context http://www.springframework.org/schema/context/spring-context-2.5.xsd"
default-autowire="byName">
<import resource="classpath:spring/base-instance.xml" />
<!-- properties -->
<bean class="com.alibaba.otter.canal.instance.spring.support.PropertyPlaceholderConfigurer" lazy-init="false">
<property name="ignoreResourceNotFound" value="true" />
<property name="systemPropertiesModeName" value="SYSTEM_PROPERTIES_MODE_OVERRIDE"/><!-- 允许system覆盖 -->
<property name="locationNames">
<list>
<value>classpath:canal.properties</value>
<value>classpath:${canal.instance.destination:}/instance.properties</value>
</list>
</property>
</bean>
<bean id="socketAddressEditor" class="com.alibaba.otter.canal.instance.spring.support.SocketAddressEditor" />
<bean class="org.springframework.beans.factory.config.CustomEditorConfigurer">
<property name="propertyEditorRegistrars">
<list>
<ref bean="socketAddressEditor" />
</list>
</property>
</bean>
<bean id="instance" class="com.alibaba.otter.canal.instance.spring.CanalInstanceWithSpring">
<property name="destination" value="${canal.instance.destination}" />
@@ -41,7 +60,6 @@
<property name="bufferMemUnit" value="${canal.instance.memory.buffer.memunit:1024}" />
<property name="batchMode" value="${canal.instance.memory.batch.mode:MEMSIZE}" />
<property name="ddlIsolation" value="${canal.instance.get.ddl.isolation:false}" />
<property name="raw" value="${canal.instance.memory.rawEntry:true}" />
</bean>
<bean id="eventSink" class="com.alibaba.otter.canal.sink.entry.EntryEventSink">
@@ -58,7 +76,7 @@
</property>
</bean>
<bean id="eventParser1" parent="baseEventParser">
<bean id="eventParser1" class="com.alibaba.otter.canal.parse.inbound.mysql.MysqlEventParser">
<property name="destination" value="${canal.instance.destination}" />
<property name="slaveId" value="${canal.instance.mysql.slaveId:0}" />
<!-- 心跳配置 -->
@@ -109,22 +127,18 @@
<!-- 解析数据库信息 -->
<property name="masterInfo">
<bean class="com.alibaba.otter.canal.parse.support.AuthenticationInfo" init-method="initPwd">
<bean class="com.alibaba.otter.canal.parse.support.AuthenticationInfo">
<property name="address" value="${canal.instance.master1.address}" />
<property name="username" value="${canal.instance.dbUsername:retl}" />
<property name="password" value="${canal.instance.dbPassword:retl}" />
<property name="pwdPublicKey" value="${canal.instance.pwdPublicKey:retl}" />
<property name="enableDruid" value="${canal.instance.enableDruid:false}" />
<property name="defaultDatabaseName" value="${canal.instance.defaultDatabaseName:test}" />
</bean>
</property>
<property name="standbyInfo">
<bean class="com.alibaba.otter.canal.parse.support.AuthenticationInfo" init-method="initPwd">
<bean class="com.alibaba.otter.canal.parse.support.AuthenticationInfo">
<property name="address" value="${canal.instance.standby1.address}" />
<property name="username" value="${canal.instance.dbUsername:retl}" />
<property name="password" value="${canal.instance.dbPassword:retl}" />
<property name="pwdPublicKey" value="${canal.instance.pwdPublicKey:retl}" />
<property name="enableDruid" value="${canal.instance.enableDruid:false}" />
<property name="defaultDatabaseName" value="${canal.instance.defaultDatabaseName:test}" />
</bean>
</property>
@@ -160,7 +174,7 @@
<property name="parallelBufferSize" value="${canal.instance.parser.parallelBufferSize:256}" />
</bean>
<bean id="eventParser2" parent="baseEventParser">
<bean id="eventParser2" class="com.alibaba.otter.canal.parse.inbound.mysql.MysqlEventParser">
<property name="destination" value="${canal.instance.destination}" />
<property name="slaveId" value="${canal.instance.mysql.slaveId:0}" />
<!-- 心跳配置 -->
@@ -211,22 +225,18 @@
<!-- 解析数据库信息 -->
<property name="masterInfo">
<bean class="com.alibaba.otter.canal.parse.support.AuthenticationInfo" init-method="initPwd">
<bean class="com.alibaba.otter.canal.parse.support.AuthenticationInfo">
<property name="address" value="${canal.instance.master2.address}" />
<property name="username" value="${canal.instance.dbUsername:retl}" />
<property name="password" value="${canal.instance.dbPassword:retl}" />
<property name="pwdPublicKey" value="${canal.instance.pwdPublicKey:retl}" />
<property name="enableDruid" value="${canal.instance.enableDruid:false}" />
<property name="defaultDatabaseName" value="${canal.instance.defaultDatabaseName:test}" />
</bean>
</property>
<property name="standbyInfo">
<bean class="com.alibaba.otter.canal.parse.support.AuthenticationInfo" init-method="initPwd">
<bean class="com.alibaba.otter.canal.parse.support.AuthenticationInfo">
<property name="address" value="${canal.instance.standby2.address}" />
<property name="username" value="${canal.instance.dbUsername:retl}" />
<property name="password" value="${canal.instance.dbPassword:retl}" />
<property name="pwdPublicKey" value="${canal.instance.pwdPublicKey:retl}" />
<property name="enableDruid" value="${canal.instance.enableDruid:false}" />
<property name="defaultDatabaseName" value="${canal.instance.defaultDatabaseName:test}" />
</bean>
</property>

View File

@@ -0,0 +1,147 @@
<?xml version="1.0" encoding="UTF-8"?>
<beans xmlns="http://www.springframework.org/schema/beans"
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xmlns:tx="http://www.springframework.org/schema/tx"
xmlns:aop="http://www.springframework.org/schema/aop" xmlns:lang="http://www.springframework.org/schema/lang"
xmlns:context="http://www.springframework.org/schema/context"
xsi:schemaLocation="http://www.springframework.org/schema/beans http://www.springframework.org/schema/beans/spring-beans-2.0.xsd
http://www.springframework.org/schema/aop http://www.springframework.org/schema/aop/spring-aop-2.0.xsd
http://www.springframework.org/schema/lang http://www.springframework.org/schema/lang/spring-lang-2.0.xsd
http://www.springframework.org/schema/tx http://www.springframework.org/schema/tx/spring-tx-2.0.xsd
http://www.springframework.org/schema/context http://www.springframework.org/schema/context/spring-context-2.5.xsd"
default-autowire="byName">
<!-- properties -->
<bean class="com.alibaba.otter.canal.instance.spring.support.PropertyPlaceholderConfigurer" lazy-init="false">
<property name="ignoreResourceNotFound" value="true" />
<property name="systemPropertiesModeName" value="SYSTEM_PROPERTIES_MODE_OVERRIDE"/><!-- 允许system覆盖 -->
<property name="locationNames">
<list>
<value>classpath:canal.properties</value>
<value>classpath:${canal.instance.destination:}/instance.properties</value>
</list>
</property>
</bean>
<bean id="socketAddressEditor" class="com.alibaba.otter.canal.instance.spring.support.SocketAddressEditor" />
<bean class="org.springframework.beans.factory.config.CustomEditorConfigurer">
<property name="propertyEditorRegistrars">
<list>
<ref bean="socketAddressEditor" />
</list>
</property>
</bean>
<bean id="instance" class="com.alibaba.otter.canal.instance.spring.CanalInstanceWithSpring">
<property name="destination" value="${canal.instance.destination}" />
<property name="eventParser">
<ref local="eventParser" />
</property>
<property name="eventSink">
<ref local="eventSink" />
</property>
<property name="eventStore">
<ref local="eventStore" />
</property>
<property name="metaManager">
<ref local="metaManager" />
</property>
<property name="alarmHandler">
<ref local="alarmHandler" />
</property>
</bean>
<!-- 报警处理类 -->
<bean id="alarmHandler" class="com.alibaba.otter.canal.common.alarm.LogAlarmHandler" />
<bean id="metaManager" class="com.alibaba.otter.canal.meta.FileMixedMetaManager">
<property name="dataDir" value="${canal.file.data.dir:../conf}" />
<property name="period" value="${canal.file.flush.period:1000}" />
</bean>
<bean id="eventStore" class="com.alibaba.otter.canal.store.memory.MemoryEventStoreWithBuffer">
<property name="bufferSize" value="${canal.instance.memory.buffer.size:16384}" />
<property name="bufferMemUnit" value="${canal.instance.memory.buffer.memunit:1024}" />
<property name="batchMode" value="${canal.instance.memory.batch.mode:MEMSIZE}" />
<property name="ddlIsolation" value="${canal.instance.get.ddl.isolation:false}" />
</bean>
<bean id="eventSink" class="com.alibaba.otter.canal.sink.entry.EntryEventSink">
<property name="eventStore" ref="eventStore" />
<property name="filterTransactionEntry" value="${canal.instance.filter.transaction.entry:false}"/>
</bean>
<bean id="eventParser" class="com.alibaba.otter.canal.parse.inbound.mysql.rds.RdsLocalBinlogEventParser">
<property name="destination" value="${canal.instance.destination}" />
<property name="alarmHandler" ref="alarmHandler" />
<!-- 解析过滤处理 -->
<property name="eventFilter">
<bean class="com.alibaba.otter.canal.filter.aviater.AviaterRegexFilter" >
<constructor-arg index="0" value="${canal.instance.filter.regex:.*\..*}" />
</bean>
</property>
<property name="eventBlackFilter">
<bean class="com.alibaba.otter.canal.filter.aviater.AviaterRegexFilter" >
<constructor-arg index="0" value="${canal.instance.filter.black.regex:}" />
<constructor-arg index="1" value="false" />
</bean>
</property>
<!-- 最大事务解析大小,超过该大小后事务将被切分为多个事务投递 -->
<property name="transactionSize" value="${canal.instance.transaction.size:1024}" />
<!-- 解析编码 -->
<property name="connectionCharset" value="${canal.instance.connectionCharset:UTF-8}" />
<!-- 解析位点记录 -->
<property name="logPositionManager">
<bean class="com.alibaba.otter.canal.parse.index.FailbackLogPositionManager">
<constructor-arg>
<bean class="com.alibaba.otter.canal.parse.index.MemoryLogPositionManager" />
</constructor-arg>
<constructor-arg>
<bean class="com.alibaba.otter.canal.parse.index.MetaLogPositionManager">
<constructor-arg ref="metaManager"/>
</bean>
</constructor-arg>
</bean>
</property>
<!-- 解析数据库信息 -->
<property name="masterInfo">
<bean class="com.alibaba.otter.canal.parse.support.AuthenticationInfo">
<property name="address" value="${canal.instance.master.address}" />
<property name="username" value="${canal.instance.dbUsername:retl}" />
<property name="password" value="${canal.instance.dbPassword:retl}" />
<property name="defaultDatabaseName" value="${canal.instance.defaultDatabaseName:test}" />
</bean>
</property>
<property name="filterQueryDml" value="${canal.instance.filter.query.dml:false}" />
<property name="filterQueryDcl" value="${canal.instance.filter.query.dcl:false}" />
<property name="filterQueryDdl" value="${canal.instance.filter.query.ddl:false}" />
<property name="useDruidDdlFilter" value="${canal.instance.filter.druid.ddl:true}" />
<property name="filterRows" value="${canal.instance.filter.rows:false}" />
<property name="filterTableError" value="${canal.instance.filter.table.error:false}" />
<property name="needWait" value="${canal.instance.parser.needWait:false}"/>
<property name="directory" value="${canal.instance.parser.directory:}"/>
<!-- rds相关 -->
<property name="url" value="${canal.instance.rds.open.url:}"/>
<property name="accesskey" value="${canal.instance.rds.open.accesskey:}"/>
<property name="secretkey" value="${canal.instance.rds.open.secretkey:}"/>
<property name="instanceId" value="${canal.instance.rds.instanceId:}"/>
<property name="startTime" value="${canal.instance.rds.startTime:}"/>
<property name="endTime" value="${canal.instance.rds.endTime:}"/>
<!--表结构相关-->
<property name="enableTsdb" value="${canal.instance.tsdb.enable:true}"/>
<property name="tsdbSpringXml" value="${canal.instance.tsdb.spring.xml:}"/>
<!-- parallel parser -->
<property name="parallel" value="${canal.instance.parser.parallel:true}" />
<property name="parallelThreadSize" value="${canal.instance.parser.parallelThreadSize}" />
<property name="parallelBufferSize" value="${canal.instance.parser.parallelBufferSize:256}" />
</bean>
</beans>

View File

@@ -10,7 +10,26 @@
http://www.springframework.org/schema/context http://www.springframework.org/schema/context/spring-context-2.5.xsd"
default-autowire="byName">
<import resource="classpath:spring/base-instance.xml" />
<!-- properties -->
<bean class="com.alibaba.otter.canal.instance.spring.support.PropertyPlaceholderConfigurer" lazy-init="false">
<property name="ignoreResourceNotFound" value="true" />
<property name="systemPropertiesModeName" value="SYSTEM_PROPERTIES_MODE_OVERRIDE"/><!-- 允许system覆盖 -->
<property name="locationNames">
<list>
<value>classpath:canal.properties</value>
<value>classpath:${canal.instance.destination:}/instance.properties</value>
</list>
</property>
</bean>
<bean id="socketAddressEditor" class="com.alibaba.otter.canal.instance.spring.support.SocketAddressEditor" />
<bean class="org.springframework.beans.factory.config.CustomEditorConfigurer">
<property name="propertyEditorRegistrars">
<list>
<ref bean="socketAddressEditor" />
</list>
</property>
</bean>
<bean id="instance" class="com.alibaba.otter.canal.instance.spring.CanalInstanceWithSpring">
<property name="destination" value="${canal.instance.destination}" />
@@ -41,7 +60,6 @@
<property name="bufferMemUnit" value="${canal.instance.memory.buffer.memunit:1024}" />
<property name="batchMode" value="${canal.instance.memory.batch.mode:MEMSIZE}" />
<property name="ddlIsolation" value="${canal.instance.get.ddl.isolation:false}" />
<property name="raw" value="${canal.instance.memory.rawEntry:true}" />
</bean>
<bean id="eventSink" class="com.alibaba.otter.canal.sink.entry.EntryEventSink">
@@ -49,7 +67,7 @@
<property name="filterTransactionEntry" value="${canal.instance.filter.transaction.entry:false}"/>
</bean>
<bean id="eventParser" parent="baseEventParser">
<bean id="eventParser" class="com.alibaba.otter.canal.parse.inbound.mysql.MysqlEventParser">
<property name="destination" value="${canal.instance.destination}" />
<property name="slaveId" value="${canal.instance.mysql.slaveId:0}" />
<!-- 心跳配置 -->
@@ -97,25 +115,21 @@
<!-- failover切换时回退的时间 -->
<property name="fallbackIntervalInSeconds" value="${canal.instance.fallbackIntervalInSeconds:60}" />
<!-- 解析数据库信息 -->
<property name="masterInfo">
<bean class="com.alibaba.otter.canal.parse.support.AuthenticationInfo" init-method="initPwd">
<bean class="com.alibaba.otter.canal.parse.support.AuthenticationInfo">
<property name="address" value="${canal.instance.master.address}" />
<property name="username" value="${canal.instance.dbUsername:retl}" />
<property name="password" value="${canal.instance.dbPassword:retl}" />
<property name="pwdPublicKey" value="${canal.instance.pwdPublicKey:retl}" />
<property name="enableDruid" value="${canal.instance.enableDruid:false}" />
<property name="defaultDatabaseName" value="${canal.instance.defaultDatabaseName:test}" />
</bean>
</property>
<property name="standbyInfo">
<bean class="com.alibaba.otter.canal.parse.support.AuthenticationInfo" init-method="initPwd">
<bean class="com.alibaba.otter.canal.parse.support.AuthenticationInfo">
<property name="address" value="${canal.instance.standby.address}" />
<property name="username" value="${canal.instance.dbUsername:retl}" />
<property name="password" value="${canal.instance.dbPassword:retl}" />
<property name="pwdPublicKey" value="${canal.instance.pwdPublicKey:retl}" />
<property name="enableDruid" value="${canal.instance.enableDruid:false}" />
<property name="defaultDatabaseName" value="${canal.instance.defaultDatabaseName:test}" />
</bean>
</property>
@@ -149,8 +163,6 @@
<!--表结构相关-->
<property name="enableTsdb" value="${canal.instance.tsdb.enable:false}"/>
<property name="tsdbSpringXml" value="${canal.instance.tsdb.spring.xml:}"/>
<property name="tsdbSnapshotInterval" value="${canal.instance.tsdb.snapshot.interval:24}" />
<property name="tsdbSnapshotExpire" value="${canal.instance.tsdb.snapshot.expire:360}" />
<!--是否启用GTID模式-->
<property name="isGTIDMode" value="${canal.instance.gtidon:false}"/>

View File

@@ -36,10 +36,10 @@
</delete>
<delete id="deleteByTimestamp" parameterClass="java.util.Map">
<delete id="deleteByGmtModified" parameterClass="java.util.Map">
<![CDATA[
delete from meta_history
where destination=#destination# and binlog_timestamp < #timestamp#
where gmt_modified < timestamp(#timestamp#)
]]>
</delete>
</sqlMap>

View File

@@ -42,10 +42,10 @@
where destination=#destination#
</delete>
<delete id="deleteByTimestamp" parameterClass="java.util.Map">
<delete id="deleteByGmtModified" parameterClass="java.util.Map">
<![CDATA[
delete from meta_snapshot
where destination=#destination# and binlog_timestamp < #timestamp# and binlog_timestamp > 0
where gmt_modified < timestamp(#timestamp#)
]]>
</delete>
</sqlMap>

View File

@@ -26,10 +26,10 @@ CREATE TABLE IF NOT EXISTS `meta_history` (
`binlog_master_id` varchar(64) DEFAULT NULL COMMENT 'binlog节点id',
`binlog_timestamp` bigint(20) DEFAULT NULL COMMENT 'binlog应用的时间戳',
`use_schema` varchar(1024) DEFAULT NULL COMMENT '执行sql时对应的schema',
`sql_schema` varchar(1024) DEFAULT NULL COMMENT '对应的schema',
`sql_table` varchar(1024) DEFAULT NULL COMMENT '对应的table',
`sql_text` longtext DEFAULT NULL COMMENT '执行的sql',
`sql_type` varchar(256) DEFAULT NULL COMMENT 'sql类型',
`schema` varchar(1024) DEFAULT NULL COMMENT '对应的schema',
`table` varchar(1024) DEFAULT NULL COMMENT '对应的table',
`sql` longtext DEFAULT NULL COMMENT '执行的sql',
`type` varchar(256) DEFAULT NULL COMMENT 'sql类型',
`extra` text DEFAULT NULL COMMENT '额外的扩展信息',
PRIMARY KEY (`id`),
UNIQUE KEY binlog_file_offest(`destination`,`binlog_master_id`,`binlog_file`,`binlog_offest`),

View File

@@ -1,10 +1,36 @@
FROM canal/osbase:v1
FROM centos:centos6.7
MAINTAINER agapple (jianghang115@gmail.com)
# install system
RUN \
/bin/cp -f /usr/share/zoneinfo/Asia/Shanghai /etc/localtime && \
echo 'root:Hello1234' | chpasswd && \
groupadd -r admin && useradd -g admin admin && \
yum install -y man && \
yum install -y dstat && \
yum install -y unzip && \
yum install -y nc && \
yum install -y openssh-server && \
yum install -y tar && \
yum install -y which && \
yum install -y wget && \
yum install -y perl && \
yum install -y file && \
ssh-keygen -q -N "" -t dsa -f /etc/ssh/ssh_host_dsa_key && \
ssh-keygen -q -N "" -t rsa -f /etc/ssh/ssh_host_rsa_key && \
sed -ri 's/session required pam_loginuid.so/#session required pam_loginuid.so/g' /etc/pam.d/sshd && \
sed -i -e 's/^#Port 22$/Port 2222/' /etc/ssh/sshd_config && \
mkdir -p /root/.ssh && chown root.root /root && chmod 700 /root/.ssh && \
yum install -y cronie && \
sed -i '/session required pam_loginuid.so/d' /etc/pam.d/crond && \
yum clean all && \
true
# install canal
COPY image/ /tmp/docker/
COPY canal.deployer-*.tar.gz /home/admin/
COPY jdk-8-linux-x64.rpm /tmp/
RUN \
cp -R /tmp/docker/alidata /alidata && \
@@ -13,6 +39,14 @@ RUN \
cp -R /tmp/docker/admin/* /home/admin/ && \
/bin/cp -f alidata/bin/lark-wait /usr/bin/lark-wait && \
touch /var/lib/rpm/* && \
yum -y install /tmp/jdk-8-linux-x64.rpm && \
/bin/rm -f /tmp/jdk-8-linux-x64.rpm && \
echo "export JAVA_HOME=/usr/java/latest" >> /etc/profile && \
echo "export PATH=\$JAVA_HOME/bin:\$PATH" >> /etc/profile && \
/bin/mv /home/admin/bin/clean_log /etc/cron.d && \
mkdir -p /home/admin/canal-server && \
tar -xzvf /home/admin/canal.deployer-*.tar.gz -C /home/admin/canal-server && \
/bin/rm -f /home/admin/canal.deployer-*.tar.gz && \
@@ -24,10 +58,10 @@ RUN \
yum clean all && \
true
# 2222 sys , 8000 debug , 11111 canal , 11112 metrics
EXPOSE 2222 11111 8000 11112
# 2222 sys , 8080 web , 8000 debug , 11111 canal
EXPOSE 2222 11111 8000 8080
WORKDIR /home/admin
ENTRYPOINT [ "/alidata/bin/main.sh" ]
CMD [ "/home/admin/app.sh" ]
CMD [ "/home/admin/app.sh" ]

View File

@@ -1,41 +0,0 @@
FROM centos:centos6.10
MAINTAINER agapple (jianghang115@gmail.com)
env DOWNLOAD_LINK="http://download.oracle.com/otn-pub/java/jdk/8u181-b13/96a7b8442fe848ef90c96a2fad6ed6d1/jdk-8u181-linux-x64.rpm"
# install system
RUN \
/bin/cp -f /usr/share/zoneinfo/Asia/Shanghai /etc/localtime && \
echo 'root:Hello1234' | chpasswd && \
groupadd -r admin && useradd -g admin admin && \
yum install -y man && \
yum install -y dstat && \
yum install -y unzip && \
yum install -y nc && \
yum install -y openssh-server && \
yum install -y tar && \
yum install -y which && \
yum install -y wget && \
yum install -y perl && \
yum install -y file && \
ssh-keygen -q -N "" -t dsa -f /etc/ssh/ssh_host_dsa_key && \
ssh-keygen -q -N "" -t rsa -f /etc/ssh/ssh_host_rsa_key && \
sed -ri 's/session required pam_loginuid.so/#session required pam_loginuid.so/g' /etc/pam.d/sshd && \
sed -i -e 's/^#Port 22$/Port 2222/' /etc/ssh/sshd_config && \
mkdir -p /root/.ssh && chown root.root /root && chmod 700 /root/.ssh && \
yum install -y cronie && \
sed -i '/session required pam_loginuid.so/d' /etc/pam.d/crond && \
true
RUN \
touch /var/lib/rpm/* && \
wget --no-cookies --no-check-certificate --header "Cookie: gpw_e24=xxx; oraclelicense=accept-securebackup-cookie" "$DOWNLOAD_LINK" -O /tmp/jdk-8-linux-x64.rpm && \
yum -y install /tmp/jdk-8-linux-x64.rpm && \
/bin/rm -f /tmp/jdk-8-linux-x64.rpm && \
echo "export JAVA_HOME=/usr/java/latest" >> /etc/profile && \
echo "export PATH=\$JAVA_HOME/bin:\$PATH" >> /etc/profile && \
yum clean all && \
true
CMD ["/bin/bash"]

View File

@@ -14,16 +14,17 @@ case "`uname`" in
esac
BASE=${bin_abs_path}
if [ "$1" == "base" ] ; then
docker build --no-cache -t canal/osbase $BASE/base
else
rm -rf $BASE/canal.*.tar.gz ;
cd $BASE/../ && mvn clean package -Dmaven.test.skip -Denv=release && cd $current_path ;
if [ "$1" == "kafka" ] ; then
cp $BASE/../target/canal-kafka-*.tar.gz $BASE/
docker build --no-cache -t canal/canal-server $BASE/
else
cp $BASE/../target/canal.deployer-*.tar.gz $BASE/
docker build --no-cache -t canal/canal-server $BASE/
fi
if [ ! -f $BASE/jdk*.rpm ] ; then
DOWNLOAD_LINK="http://download.oracle.com/otn-pub/java/jdk/8u181-b13/96a7b8442fe848ef90c96a2fad6ed6d1/jdk-8u181-linux-x64.tar.gz"
wget --no-cookies --no-check-certificate --header "Cookie: gpw_e24=xxx; oraclelicense=accept-securebackup-cookie" "$DOWNLOAD_LINK" -O $BASE/jdk-8-linux-x64.rpm
fi
cd $BASE/../ && mvn clean package -Dmaven.test.skip -Denv=release && cd $current_path ;
if [ "$1" == "kafka" ] ; then
cp $BASE/../target/canal-kafka-*.tar.gz $BASE/
docker build --no-cache -t canal/canal-server $BASE/
else
cp $BASE/../target/canal.deployer-*.tar.gz $BASE/
docker build --no-cache -t canal/canal-server $BASE/
fi

View File

@@ -86,11 +86,7 @@ function start_canal() {
echo "multi destination:$destination is not support"
exit 1;
else
if [ "$destination" != "" ] && [ "$destination" != "example" ] ; then
if [ -d /home/admin/canal-server/conf/example ]; then
mv /home/admin/canal-server/conf/example /home/admin/canal-server/conf/$destination
fi
fi
mv /home/admin/canal-server/conf/example /home/admin/canal-server/conf/$destination
fi
su admin -c 'cd /home/admin/canal-server/bin/ && sh restart.sh 1>>/tmp/start.log 2>&1'
sleep 5

View File

@@ -1,11 +1,6 @@
#!/bin/sh
metrics_port=`perl -le 'print $ENV{"canal.metrics.pull.port"}'`
if [ "$metrics_port" == "" ]; then
metrics_port="11112"
fi
CHECK_URL="http://127.0.0.1:$metrics_port/metrics"
CHECK_POINT="canal"
CHECK_URL="http://127.0.0.1:8080/metrics"
CHECK_POINT="success"
CHECK_COUNT=`curl -s --connect-timeout 7 --max-time 7 $CHECK_URL | grep -c $CHECK_POINT`
if [ $CHECK_COUNT -eq 0 ]; then
echo "[FAILED]"

Some files were not shown because too many files have changed in this diff Show More