chenjl/canal
1
0
Fork 0
mirror of https://github.com/alibaba/canal.git synced 2026-11-04 00:30:15 +08:00
Code Issues Packages Projects Releases Wiki Activity
1,710 Commits 13 Branches 38 Tags
c0de9795951c3f01d8416b09dc8c6cd9aff3531f
Commit Graph

1710 Commits

This Branch
This Branch
All Branches
Author SHA1 Message Date
jiaowc
c0de979595 修复同步mysql数据到es7和es8 使用es别名报错 Not found the mapping info of index 问题 (#5560) 2026-02-12 15:21:39 +08:00
usedssl
fcadf564ba fix(canal/client-adapter): 修复adapter插件全量同步sqlservice错误(#5564) (#5566) 2026-02-12 15:13:37 +08:00
jianghang.loujh
54c84b1535 merge code for table metacache performance 2026-01-28 12:07:33 +08:00
jianghang.loujh
4ded2869c9 merge code 2026-01-28 12:06:35 +08:00
sunxien
f07a075dce refactor(canal/server): 优化程序代码 (#5545) 
- canal server logback.xml 配置增加
   CanalServerWithEmbedded 单独日志配置，流量非常大的情况下，getMessages/ack等日志量非常大
   日志刷盘太快也影响性能，占磁盘空间。用户可以根据需要单独调整日志级别。默认是INFO

- TableMetaCache 锁的范围太大，针对这块做了代码逻辑优化。以下场景影响性能：
   - 当用户关闭tsdb.enable=false, 分库分表数量非常多回查次数很多；
   - 每张表有四五百列甚至更多，里面的集合容器提前初始化，减少扩容的次数；

- DatabaseTableMeta 标记一个内存不安全的地方，以后看情况优化掉
	使用MemoryTableMeta在内存区记录表元数据，当分库分表数量非常多，表结构非常复杂非常吃内存，容易OOM
 2026-01-28 11:52:25 +08:00
sunxien
dcea8afc93 refactor(canal): 升级JVM参数 (#5544) 
- 重点升级JVM参数

- 重构SessionHandler代码, 处理错误请求会响应两次包；
 2026-01-28 11:20:19 +08:00
jianghang.loujh
c0619c4217 merge code 2025-12-25 11:54:25 +08:00
dingxiaobo
8a4199a7b0 Fix 2 oss binlog issue (#5536) 
* 1. Fix compressed OSS binlog data

2. Fix first second data loss caused by dumping from OSS binlog

* Fix CI failed test cases
 2025-12-25 11:52:24 +08:00
sunxien
79338be02d refactor(canal/adapter): improve perf (#5548) 
- String.format is lower than StringBuilder. Benchmark like below:

code snippet:

String str = String.format("%s-%s-%s", 0, 1, 10);

Benchmark                         Mode     Cnt         Score    Error  Units
StringBenchmark.append           thrpt          46431458.255           ops/s
StringBenchmark.format           thrpt            985724.313           ops/s
StringBenchmark.append            avgt                ≈ 10⁻⁸            s/op
StringBenchmark.format            avgt                ≈ 10⁻⁶            s/op
StringBenchmark.append          sample  364232        ≈ 10⁻⁷            s/op
StringBenchmark.append:p0.00    sample                ≈ 10⁻⁸            s/op
StringBenchmark.append:p0.50    sample                ≈ 10⁻⁷            s/op
StringBenchmark.append:p0.90    sample                ≈ 10⁻⁷            s/op
StringBenchmark.append:p0.95    sample                ≈ 10⁻⁷            s/op
StringBenchmark.append:p0.99    sample                ≈ 10⁻⁷            s/op
StringBenchmark.append:p0.999   sample                ≈ 10⁻⁷            s/op
StringBenchmark.append:p0.9999  sample                ≈ 10⁻⁵            s/op
StringBenchmark.append:p1.00    sample                 0.001            s/op
StringBenchmark.format          sample  336220        ≈ 10⁻⁶            s/op
StringBenchmark.format:p0.00    sample                ≈ 10⁻⁶            s/op
StringBenchmark.format:p0.50    sample                ≈ 10⁻⁶            s/op
StringBenchmark.format:p0.90    sample                ≈ 10⁻⁶            s/op
StringBenchmark.format:p0.95    sample                ≈ 10⁻⁶            s/op
StringBenchmark.format:p0.99    sample                ≈ 10⁻⁶            s/op
StringBenchmark.format:p0.999   sample                ≈ 10⁻⁵            s/op
StringBenchmark.format:p0.9999  sample                ≈ 10⁻⁴            s/op
StringBenchmark.format:p1.00    sample                 0.001            s/op
StringBenchmark.append              ss                ≈ 10⁻⁶            s/op
StringBenchmark.format              ss                ≈ 10⁻⁵            s/op
 2025-12-25 11:49:06 +08:00
jianghang.loujh
d18bde71c6 merge and support polardb-x recycle_bin binlog format 2025-12-25 11:47:36 +08:00
sunxien
5a5e60d371 feat(canal/adapter): adapter支持prometheus暴露jvm指标 (#5549) 
- adapter 默认提供prometheus暴露JVM指标

- 指定 maven-antrun-plugin 未指定版本号，默认是 3.0.0，但是配置中使用 <tasks> 标签，编译报错。
   - 降级版本号，显式指定 1.8 版本
   - 或者把配置中的 <tasks> 标签改成 <target>
 2025-12-25 11:41:08 +08:00
jianghang.loujh
af9375440d fixed empty config 2025-08-06 10:18:37 +08:00
jianghang.loujh
9f2fc7409f merge issue #5496 , 修复mysql rds oss订阅本地binlog过期需要再次订阅oss 2025-08-05 16:49:47 +08:00
jianghang.loujh
556d998db1 adjust preferIPv4Stack=false, support ipv6 2025-08-05 16:42:00 +08:00
jianghang.loujh
25501db3ac migrate mvn deploy from OSSRH to maven central 2025-08-05 14:51:14 +08:00
jianghang.loujh
8523a51623 fixed JSON datatype string parse 2025-08-05 14:26:57 +08:00
dingxiaobo
16831825b8 fix JSON datatype string value truncated by '\0' (#5482) 2025-08-05 14:22:00 +08:00
jianghang.loujh
1e37fe2dee fixed polardbx panda index 2025-08-05 14:14:01 +08:00
jianghang.loujh
5607444bad merge from github and rebase 2025-07-28 14:38:10 +08:00
jianghang.loujh
3a7223e460 merge from github 2025-07-28 14:37:47 +08:00
Canlin Guo
553438e227 CI: add a workflow for CI (#5483) 
* Create SECURITY.md

* CI: Create maven.yml

* Update maven.yml

* Update maven.yml

* fix bugs of fastsql schema test
 2025-07-28 14:33:52 +08:00
Canlin Guo
a3ea3d32d9 Create SECURITY.md (#5472) 2025-06-27 19:05:52 +08:00
Sunxien®
50a570290a fix(canal/client-adapter): 修复adapter插件重试错误缺陷 (#5427) 
* fix(canal/prometheus): 修复延迟指标的缺陷

-【缺陷描述】
   Admin控制台修改canal.properties配置后，Server自动进行重启，在Grafan上观测到同步实例的PUT/GET/ACK延迟居高不下。

-【原因定位】
   源库心跳正常触发，MemoryStoreWithBuffer正常推进位点，profiling 正常统计，但是延迟指标依然是越来越来越高。Prometheus调用collect接口采集到的exec time时间始终是固定不变的。经debug排查到StoreCollector采集器内存hold的StoreMetricsHolder与CanalInstance实例中的引用已经不同啦，CanalIntance重启时已经被重建过一份新的实例，但是StoreMetricsHolder却没有保存到内存Hold中。原因是Map.putIfAbsent调用引起。

-【修复效果】
- 修复后，重启Server心跳正常推进，延迟瞬间降下来。梳理其他Collector代码都是调用Map.put，只有这里使用putIfAbsent可能是粗心导致吧。修复效果

* fix(canal/client-adapter): 修复adapter插件重试错误缺陷

-【缺陷描述】
  adapter运行期间，针对server或者Instance进行过一次重启或者断网测试，发现adapter通过client拉取数据报错以后持续打印错误日志，永远不得恢复。虽然adapter processor 针对错误有重试逻辑，但是client的tcp连接已经发生中断，且无法恢复。

-【问题定位】
  adapter processor 针对错误的处理比较粗，需要细分是从上游get message报错，还是下游sync data报错？
  如果是 get message报错，特别是tcp 断开，需要跳出重试，重新尝试重连。如果是下游 sync data 报错，再考虑重试写入。下游插件开发者自行保证连接重连机制、写入幂等。框架只提供重试。

- 说明：顺带更新一下 .gitignore 文件
 2025-05-16 15:12:42 +08:00
Sunxien®
6b7ddbbe92 fix(canal/prometheus): 修复延迟指标的缺陷 (#5426) 
-【缺陷描述】
   Admin控制台修改canal.properties配置后，Server自动进行重启，在Grafan上观测到同步实例的PUT/GET/ACK延迟居高不下。

-【原因定位】
   源库心跳正常触发，MemoryStoreWithBuffer正常推进位点，profiling 正常统计，但是延迟指标依然是越来越来越高。Prometheus调用collect接口采集到的exec time时间始终是固定不变的。经debug排查到StoreCollector采集器内存hold的StoreMetricsHolder与CanalInstance实例中的引用已经不同啦，CanalIntance重启时已经被重建过一份新的实例，但是StoreMetricsHolder却没有保存到内存Hold中。原因是Map.putIfAbsent调用引起。

-【修复效果】
- 修复后，重启Server心跳正常推进，延迟瞬间降下来。梳理其他Collector代码都是调用Map.put，只有这里使用putIfAbsent可能是粗心导致吧。修复效果
 2025-05-16 15:09:22 +08:00
kkz-01
2b53164253 解决8.0.30+使用cahcing_sha2_password时Fastauth报错问题 (#5406) 
Co-authored-by: zhangkai <zhangkai02@xgd.com>
 2025-05-16 15:06:36 +08:00
Sunxien®
e101287c25 fix(canal/startup.sh): 修复生产环境ZK启用sasl安全认证 (#5438) 
* fix(canal/startup.sh): 修复生产环境ZK启用sasl安全认证

- 如果生产环境zk启用sasl安全认证，canal-server/canal-adapter启动无法连接zk进行读写操作
- 默认支持用户将 jaas.conf 放置在 ./conf/ 目录下，启动脚本增加-Djava.security.auth.login.config 启动参数即可

* fix(canal/parent): upgrade h2 version

- runtime error: Value too long for column "CHARACTER VARYING" #5086

- Ref: https://github.com/alibaba/canal/issues/5086
 2025-05-16 15:04:02 +08:00
elegance code
b153e61f0d fix issue #5401 (#5402) 2025-05-16 15:03:01 +08:00
hygugb
9bd4357f37 fix: tsdb存储元数据时支持大对象（大于64M） (#5393) 2025-05-16 15:00:19 +08:00
jianghang.loujh
dad209feb8 fixed issue #5453 , skipping Query_log_event by unknown status vars 2025-05-16 14:57:54 +08:00
jianghang.loujh
e21e98aa83 fixed CanalAdmin instanceLog and polardbx columnar test 2025-03-19 15:51:27 +08:00
nakeiven
c998b3f100 Feature/adapter postgresql 20250124 (#5390) 
* adapter tcp模式mysql数据全量同步至postgresql时 报错
fix issue 2146

* fix issue 1980

* feat 增加 当dbType 为 postgresql时，返回双引号(避免表名为postgresql的关键字时，sql查询报错)

---------

Co-authored-by: guodong.zhang <zgd8988@126.com>
 2025-01-24 17:14:41 +08:00
jianghang.loujh
c15129b0ac [maven-release-plugin] prepare for next development iteration 2025-01-16 16:07:08 +08:00
jianghang.loujh
0741ccda1e [maven-release-plugin] prepare release canal-1.1.8 canal-1.1.8 2025-01-16 16:07:00 +08:00
jianghang.loujh
02fffc9e74 compatiable polardb-x 2025-01-13 22:07:12 +08:00
jianghang.loujh
5ca4c39313 compatiable percona & polardb-x 2025-01-13 22:02:09 +08:00
jianghang.loujh
3162d7093d fixed polardb-x variables(flashback & gpp) 2025-01-13 21:35:33 +08:00
jianghang.loujh
f350fbdc0a fixed mysql 8.0.30+ , support new variable 2025-01-08 10:10:33 +08:00
jianghang.loujh
2d0dcf0e5d fixed issue #5301, fixed caching_sha2_password full auth & client SHA2 2024-12-24 17:51:31 +08:00
jianghang.loujh
515c05c873 fixed issue #5301, fixed caching_sha2_password full auth 2024-12-24 16:39:21 +08:00
jianghang.loujh
7ee551cbf7 fixed issue #5301, fixed caching_sha2_password full auth 2024-12-24 16:33:41 +08:00
jianghang.loujh
923675aa49 fixed issue #5334 , adjust canal_manager default schema 2024-12-17 16:06:57 +08:00
jianghang.loujh
ebb541001c fixed issue #5337 , remove docker privileged 2024-12-17 15:52:01 +08:00
Chongzhi Deng
44ed76ebb8 fix typo (#5299) 
Signed-off-by: BruceAko <chongzhi@hust.edu.cn>
 2024-12-17 15:32:03 +08:00
jianghang.loujh
fe836f7515 fixed issue #5336 #5332 , etl bugfix 2024-12-17 15:29:51 +08:00
quanllong
b30544c750 bugfix: 兼容MariaDB GTID 为空的情况 (#5352) 2024-12-17 15:09:41 +08:00
lol
66483ef83f Feature/helm charts (#5285) 
* canal-admin canal-server k8s helm chart

* 拼写错误
 2024-10-14 16:48:05 +08:00
jianghang.loujh
b391188baf code format & fine-tune 2024-10-14 16:44:59 +08:00
lol
2a7a507987 fixed health.sh failed (#5281) 
fixed health.sh failed
 2024-10-14 16:44:15 +08:00
lanicc
f53749bfc5 fixed issue #5289 (#5290) 
Co-authored-by: lanicc <liaonan@souche.com>
 2024-10-14 16:18:21 +08:00
jianghang.loujh
80ca4366db fixed #5270 , add CanalMQStarter stop latch 2024-09-13 18:04:56 +08:00