chenjl/canal
1
0
Fork 0
mirror of https://github.com/alibaba/canal.git synced 2026-11-04 00:30:15 +08:00
Code Issues Packages Projects Releases Wiki Activity
1,707 Commits 13 Branches 38 Tags
4ded2869c96e76a55d41a92dfe6c435d44e1a43d
Commit Graph

397 Commits

Author SHA1 Message Date
jianghang.loujh
4ded2869c9 merge code 2026-01-28 12:06:35 +08:00
sunxien
dcea8afc93 refactor(canal): 升级JVM参数 (#5544) 
- 重点升级JVM参数

- 重构SessionHandler代码, 处理错误请求会响应两次包；
 2026-01-28 11:20:19 +08:00
jianghang.loujh
d18bde71c6 merge and support polardb-x recycle_bin binlog format 2025-12-25 11:47:36 +08:00
sunxien
5a5e60d371 feat(canal/adapter): adapter支持prometheus暴露jvm指标 (#5549) 
- adapter 默认提供prometheus暴露JVM指标

- 指定 maven-antrun-plugin 未指定版本号，默认是 3.0.0，但是配置中使用 <tasks> 标签，编译报错。
   - 降级版本号，显式指定 1.8 版本
   - 或者把配置中的 <tasks> 标签改成 <target>
 2025-12-25 11:41:08 +08:00
jianghang.loujh
556d998db1 adjust preferIPv4Stack=false, support ipv6 2025-08-05 16:42:00 +08:00
jianghang.loujh
25501db3ac migrate mvn deploy from OSSRH to maven central 2025-08-05 14:51:14 +08:00
Sunxien®
50a570290a fix(canal/client-adapter): 修复adapter插件重试错误缺陷 (#5427) 
* fix(canal/prometheus): 修复延迟指标的缺陷

-【缺陷描述】
   Admin控制台修改canal.properties配置后，Server自动进行重启，在Grafan上观测到同步实例的PUT/GET/ACK延迟居高不下。

-【原因定位】
   源库心跳正常触发，MemoryStoreWithBuffer正常推进位点，profiling 正常统计，但是延迟指标依然是越来越来越高。Prometheus调用collect接口采集到的exec time时间始终是固定不变的。经debug排查到StoreCollector采集器内存hold的StoreMetricsHolder与CanalInstance实例中的引用已经不同啦，CanalIntance重启时已经被重建过一份新的实例，但是StoreMetricsHolder却没有保存到内存Hold中。原因是Map.putIfAbsent调用引起。

-【修复效果】
- 修复后，重启Server心跳正常推进，延迟瞬间降下来。梳理其他Collector代码都是调用Map.put，只有这里使用putIfAbsent可能是粗心导致吧。修复效果

* fix(canal/client-adapter): 修复adapter插件重试错误缺陷

-【缺陷描述】
  adapter运行期间，针对server或者Instance进行过一次重启或者断网测试，发现adapter通过client拉取数据报错以后持续打印错误日志，永远不得恢复。虽然adapter processor 针对错误有重试逻辑，但是client的tcp连接已经发生中断，且无法恢复。

-【问题定位】
  adapter processor 针对错误的处理比较粗，需要细分是从上游get message报错，还是下游sync data报错？
  如果是 get message报错，特别是tcp 断开，需要跳出重试，重新尝试重连。如果是下游 sync data 报错，再考虑重试写入。下游插件开发者自行保证连接重连机制、写入幂等。框架只提供重试。

- 说明：顺带更新一下 .gitignore 文件
 2025-05-16 15:12:42 +08:00
Sunxien®
e101287c25 fix(canal/startup.sh): 修复生产环境ZK启用sasl安全认证 (#5438) 
* fix(canal/startup.sh): 修复生产环境ZK启用sasl安全认证

- 如果生产环境zk启用sasl安全认证，canal-server/canal-adapter启动无法连接zk进行读写操作
- 默认支持用户将 jaas.conf 放置在 ./conf/ 目录下，启动脚本增加-Djava.security.auth.login.config 启动参数即可

* fix(canal/parent): upgrade h2 version

- runtime error: Value too long for column "CHARACTER VARYING" #5086

- Ref: https://github.com/alibaba/canal/issues/5086
 2025-05-16 15:04:02 +08:00
jianghang.loujh
e21e98aa83 fixed CanalAdmin instanceLog and polardbx columnar test 2025-03-19 15:51:27 +08:00
nakeiven
c998b3f100 Feature/adapter postgresql 20250124 (#5390) 
* adapter tcp模式mysql数据全量同步至postgresql时 报错
fix issue 2146

* fix issue 1980

* feat 增加 当dbType 为 postgresql时，返回双引号(避免表名为postgresql的关键字时，sql查询报错)

---------

Co-authored-by: guodong.zhang <zgd8988@126.com>
 2025-01-24 17:14:41 +08:00
jianghang.loujh
c15129b0ac [maven-release-plugin] prepare for next development iteration 2025-01-16 16:07:08 +08:00
jianghang.loujh
0741ccda1e [maven-release-plugin] prepare release canal-1.1.8 2025-01-16 16:07:00 +08:00
Chongzhi Deng
44ed76ebb8 fix typo (#5299) 
Signed-off-by: BruceAko <chongzhi@hust.edu.cn>
 2024-12-17 15:32:03 +08:00
jianghang.loujh
fe836f7515 fixed issue #5336 #5332 , etl bugfix 2024-12-17 15:29:51 +08:00
JZ
fb3d2c6811 add compressionType,enableChunking for pulsar client , upgrade dependencies for security (#5191) 
* add support for listener name which is described in https://pulsar.apache.org/docs/next/concepts-multiple-advertised-listeners/

* upgrade for CVE-2023-20883  CVE-2022-1471  CVE-2023-20860

* add   enableChunking

* downgrade  pulsar client version

* fix

* 增加压缩算法
 2024-07-16 14:59:29 +08:00
jianghang.loujh
3439c48948 fixed pull #5175 , reformat 2024-07-16 14:57:53 +08:00
Yazid Lee
9373429015 fix: resource loading failure caused by incorrect parameter order in StringUtils.startsWithIgnoreCase (#5173) 2024-06-26 15:53:24 +08:00
lwd-coding
9493ff733d fix: 同个instance配置多个group避免共享CanalMsgConsumer (#5175) 2024-06-26 15:49:48 +08:00
Tass
31530dc9af fix(adapter): add args into adapter/restart.sh (#5149) 
Signed-off-by: Tass <tasszz2k@gmail.com>
 2024-06-26 15:34:30 +08:00
jianghang.loujh
7bb619cf12 fixed #5154,avoid spring RCE 2024-06-06 15:29:17 +08:00
jianghang.loujh
aa05d8e2a0 reformat 2024-05-22 09:46:13 +08:00
ziyou
bbea802914 feat(es8): es8 支持证书认证 (#5106) 2024-04-28 09:54:53 +08:00
jianghang.loujh
059f21208d fixed issue #5066 , es7.x npe 2024-03-18 14:19:45 +08:00
jianghang.loujh
8ca900377a merge & reformat code 2024-03-08 11:58:46 +08:00
jackzlz
6dffcc8573 yml解析支持占位符 (#5045) 2024-03-08 11:56:30 +08:00
wbxlll
711971599f fix issue client-adapter.rdb模块在转换时间时丢失精度 #5052 (#5053) 
* fix the issue that client-adapter.rdb module loses precision when converting microsencond time

* fix the issue that client-adapter.rdb module loses precision when converting to java.sql.Time or java.sql.Timestamp

* fix the issue that client-adapter.rdb module loses precision when converting to java.sql.Time or java.sql.Timestamp
 2024-03-08 11:50:53 +08:00
jianghang.loujh
b9ed5caf05 format 2024-02-04 13:35:10 +08:00
jianghang.loujh
348d218120 fixed issue #5029 , add deserializer message 2024-02-04 13:28:27 +08:00
zoemak
e2a64daad8 fixed issue #5037 ,使用client-adapter.rdb模块进行ETL同步，源库为MySQL，数据库名有短横线，读取源数据时报SQL异常 (#5044) 2024-02-04 13:07:51 +08:00
jianghang.loujh
93dac2d335 format clickhouse adapter code 2023-12-04 09:46:23 +08:00
xander-os
9baeff55e4 enhance clickhouse support (#4957) 2023-12-04 09:34:43 +08:00
jianghang.loujh
b322201889 [maven-release-plugin] prepare for next development iteration 2023-10-09 14:27:37 +08:00
jianghang.loujh
ea8949298f [maven-release-plugin] prepare release canal-1.1.7 2023-10-09 14:27:33 +08:00
dack-su
ba924b9f1d update org.postgresql:postgresql 42.3.3 to 42.5.1 (#4898) 2023-10-09 13:46:20 +08:00
jianghang.loujh
6440b744f2 fixed arm64 stack size too small, upgrade all to 1m 2023-10-08 18:55:06 +08:00
cooper lee
cda330e49e Fix Configuration property name '-id' is not valid on canal 1.1.7 alpha 3 #4879 (#4887) 
* Fix Configuration property name '-id' is not valid

* Fix Configuration property name '-id' is not valid
读取原配置文件
 2023-09-21 10:53:47 +08:00
weirhp
d623bea71a 解决rdb下有多个配制，会导致配置相互覆盖的问题 (#4560) 
恩baseProperties这个变量是共享的，不能修改这个变量的属性值。要不后续的配置都会改变。关联issue为4355
 2023-09-04 17:23:26 +08:00
jianghang.loujh
2786e06ccf format es8 2023-09-01 19:58:23 +08:00
ymz
63dfbf65df es8x adapter (#4640) 2023-09-01 19:56:03 +08:00
jianghang.loujh
d90066e01b upgrade spring cloud & spring boot version 2023-09-01 19:51:49 +08:00
jianghang.loujh
681d03fbe7 upgrade dependency version for CVE 2023-09-01 13:59:05 +08:00
jianghang.loujh
7241af84f7 fixed issue #4742 , format 2023-08-22 17:52:38 +08:00
zard
159457cccd very very major：Adapter rdb bug fix (#4742) 
* 异常点描述1

* 问题描述

* bug fix

rdb yml文件 编码格式问题，导致所有outerAdapter初始化失败；
kafka消息空跑，数据全都丢失 问题 fix
 2023-08-22 17:50:12 +08:00
jianghang.loujh
d8a70d0c2f fixed issue #4822 , fix connection close 2023-08-22 17:32:42 +08:00
elizwy
f067f761c7 fix issue 4806, make Common.init success (#4830) 
Co-authored-by: eli.zwy <wenyang.zhang@ipinyou.com>
 2023-08-22 17:21:50 +08:00
rewerma
cd798b71e9 add proxy for SPI (#4520) 
* 测试类报错

* add proxy for SPI
 2022-12-06 17:35:30 +08:00
jianghang.loujh
4c7cdbc7f3 optimize code 2022-11-16 10:27:14 +08:00
杰锅不是锅
279d503cf4 fix_ETL同步mysql关键字报错 (#4346) 
Co-authored-by: foleyang <foleyang@cogobuy.com>
 2022-11-16 10:24:56 +08:00
Jonathan Leitschuh
272b1d5d11 vuln-fix: Use HTTPS instead of HTTP to resolve dependencies (#4437) 
This fixes a security vulnerability in this project where the `build.gradle`
files were configuring Gradle to resolve dependencies over HTTP instead of
HTTPS.

Weakness: CWE-829: Inclusion of Functionality from Untrusted Control Sphere
Severity: High
CVSSS: 8.1
Detection: OpenRewrite

Reported-by: Jonathan Leitschuh <Jonathan.Leitschuh@gmail.com>
Signed-off-by: Jonathan Leitschuh <Jonathan.Leitschuh@gmail.com>

Bug-tracker: https://github.com/JLLeitschuh/security-research/issues/9


Co-authored-by: Moderne <team@moderne.io>

Co-authored-by: Moderne <team@moderne.io>
 2022-11-16 09:59:27 +08:00
jianghang.loujh
ef7e21e4b2 support jdk11 2022-09-01 18:27:19 +08:00