Compare commits

...

376 Commits

Author SHA1 Message Date
jianghang.loujh
87be50e876 fix(server/admin): 修复 Canal Admin 认证绕过漏洞 (#2076021)
在 ClientAuthenticationHandler.messageReceived() 的认证失败分支补充
ctx.getChannel().close() 与 return,防止 fall-through 到成功 ACK 分支
导致认证 handler 被移除、后续命令被当作已认证执行。

新增 ClientAuthenticationHandlerTest 覆盖错误密码、seed 为空、
认证成功三条路径。
2026-07-30 15:38:12 +08:00
jianghang.loujh
e20e442446 fix(client-adapter/phoenix): 修复 Phoenix ETL params 参数 SQL 注入漏洞
将 SYS_TIME 默认条件和 etlCondition 中的用户输入改为 PreparedStatement
占位符绑定,避免直接拼接 SQL 导致注入。
2026-07-15 14:59:08 +08:00
jianghang.loujh
08d736b33f fixed docker build 2026-06-04 10:29:28 +08:00
Guimu
cf628769d6 fix: preserve commas inside regex grouping when splitting filter patterns (#5588)
AviaterRegexFilter#init splits the configured pattern on every comma to
separate multiple regular expressions. That blindly breaks regex
quantifiers such as \d{1,2} or non-capturing alternations like
(a,b|c) because the comma inside the grouping construct is treated as
a top-level separator.

Replace the StringUtils.split call with a parse-aware splitter that
tracks the depth of {}, [], () and respects backslash escapes, so only
top-level commas separate patterns. Apply the change to both copies of
AviaterRegexFilter (filter module and connector/core module).

Add AviaterFilterTest#test_regex_with_quantifier_comma covering: a
single-pattern \d{1,2} quantifier, a multi-pattern list combined with
\d{1,2}, and an open-ended {n,} quantifier.

Fixes #5442
2026-05-29 10:39:04 +08:00
jianghang.loujh
cf97b2ae31 fix: mitigate SnakeYAML deserialization RCE and enforce default password change
1. Use SafeConstructor in ApplicationConfigMonitor to block arbitrary
   object instantiation via YAML type tags (e.g. !!javax.script.ScriptEngineManager).
2. Detect default admin password (admin/123456) on login and enforce
   password change before allowing any other API operations (code 50015).

🤖 Generated with [Qoder][https://qoder.com]
2026-04-29 18:07:02 +08:00
jianghang.loujh
ac6242bae9 compatiable polardb-x hippo&spc index status 2026-04-23 14:22:53 +08:00
Guimu
c24307c893 fixed issue #5391: skip unknown optional metadata types in TableMapLogEvent (#5577)
Replace IllegalArgumentException with buffer.forward(len) to gracefully
skip unrecognized optional metadata field types. This fixes crashes when
parsing binlog events from PolarDB or future MySQL versions that include
extended metadata types (e.g. type 72).

Fixes #5391

Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-04-23 14:10:58 +08:00
Tsugikuni Michikatsu
a97d841737 fix: remove duplicate string in startup.bat CMD_STR (#5573) 2026-04-23 14:09:47 +08:00
Guimu
8a5dc38a58 fix: ensure super.stop() is called exactly once in CanalRabbitMQProducer (#5578)
Previously super.stop() was called both inside the try block and after
the catch blocks. On success it was invoked twice; on IOException it was
never reached. Move super.stop() to a finally block so it executes
exactly once regardless of whether close() succeeds or fails.

Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-04-23 14:09:05 +08:00
Guimu
5110cfd49d fix: close HttpClient after binlog download to prevent resource leak (#5579)
BinlogDownloadQueue.download() creates a CloseableHttpClient for each
download but never closes it. Over time this exhausts connection pool
resources. Wrap the HTTP execution in a try-finally block to ensure the
client is always closed.

Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-04-23 14:07:09 +08:00
zhenggc1
da7ab64d55 remove “\“” from wrong wiki url (#5581)
remove “\“” from wrong wiki url
2026-04-23 14:01:40 +08:00
jianghang.loujh
09382c6b91 fixed dockerfile security 2026-04-02 14:44:03 +08:00
jianghang.loujh
69a06dba09 fixed issue #5517: H2 database field type error causing 'Value too long for column' exception
- Changed meta_snapshot.data from longtext to CLOB in H2 schema
- Changed meta_history.sql_text from longtext to CLOB in H2 schema
- H2 database doesn't recognize MySQL's longtext type and converts it to VARCHAR with length limit
- CLOB type supports up to 2GB text data, solving the issue when storing large table structures (>1MB DDL)
2026-02-12 17:42:44 +08:00
jianghang.loujh
1a265c8960 fixed issue #5474 , scaled_float -> BigDecimal 2026-02-12 17:33:39 +08:00
jianghang.loujh
24877fdf03 merge code and format 2026-02-12 17:32:33 +08:00
jiaowc
c0de979595 修复同步mysql数据到es7和es8 使用es别名报错 Not found the mapping info of index 问题 (#5560) 2026-02-12 15:21:39 +08:00
usedssl
fcadf564ba fix(canal/client-adapter): 修复adapter插件全量同步sqlservice错误(#5564) (#5566) 2026-02-12 15:13:37 +08:00
jianghang.loujh
54c84b1535 merge code for table metacache performance 2026-01-28 12:07:33 +08:00
jianghang.loujh
4ded2869c9 merge code 2026-01-28 12:06:35 +08:00
sunxien
f07a075dce refactor(canal/server): 优化程序代码 (#5545)
- canal server logback.xml 配置增加
   CanalServerWithEmbedded 单独日志配置,流量非常大的情况下,getMessages/ack等日志量非常大
   日志刷盘太快也影响性能,占磁盘空间。用户可以根据需要单独调整日志级别。默认是INFO

- TableMetaCache 锁的范围太大,针对这块做了代码逻辑优化。以下场景影响性能:
   - 当用户关闭tsdb.enable=false, 分库分表数量非常多回查次数很多;
   - 每张表有四五百列甚至更多,里面的集合容器提前初始化,减少扩容的次数;

- DatabaseTableMeta 标记一个内存不安全的地方,以后看情况优化掉
	使用MemoryTableMeta在内存区记录表元数据,当分库分表数量非常多,表结构非常复杂非常吃内存,容易OOM
2026-01-28 11:52:25 +08:00
sunxien
dcea8afc93 refactor(canal): 升级JVM参数 (#5544)
- 重点升级JVM参数

- 重构SessionHandler代码, 处理错误请求会响应两次包;
2026-01-28 11:20:19 +08:00
jianghang.loujh
c0619c4217 merge code 2025-12-25 11:54:25 +08:00
dingxiaobo
8a4199a7b0 Fix 2 oss binlog issue (#5536)
* 1. Fix compressed OSS binlog data

2. Fix first second data loss caused by dumping from OSS binlog

* Fix CI failed test cases
2025-12-25 11:52:24 +08:00
sunxien
79338be02d refactor(canal/adapter): improve perf (#5548)
- String.format is lower than StringBuilder. Benchmark like below:

code snippet:

String str = String.format("%s-%s-%s", 0, 1, 10);

Benchmark                         Mode     Cnt         Score    Error  Units
StringBenchmark.append           thrpt          46431458.255           ops/s
StringBenchmark.format           thrpt            985724.313           ops/s
StringBenchmark.append            avgt                ≈ 10⁻⁸            s/op
StringBenchmark.format            avgt                ≈ 10⁻⁶            s/op
StringBenchmark.append          sample  364232        ≈ 10⁻⁷            s/op
StringBenchmark.append:p0.00    sample                ≈ 10⁻⁸            s/op
StringBenchmark.append:p0.50    sample                ≈ 10⁻⁷            s/op
StringBenchmark.append:p0.90    sample                ≈ 10⁻⁷            s/op
StringBenchmark.append:p0.95    sample                ≈ 10⁻⁷            s/op
StringBenchmark.append:p0.99    sample                ≈ 10⁻⁷            s/op
StringBenchmark.append:p0.999   sample                ≈ 10⁻⁷            s/op
StringBenchmark.append:p0.9999  sample                ≈ 10⁻⁵            s/op
StringBenchmark.append:p1.00    sample                 0.001            s/op
StringBenchmark.format          sample  336220        ≈ 10⁻⁶            s/op
StringBenchmark.format:p0.00    sample                ≈ 10⁻⁶            s/op
StringBenchmark.format:p0.50    sample                ≈ 10⁻⁶            s/op
StringBenchmark.format:p0.90    sample                ≈ 10⁻⁶            s/op
StringBenchmark.format:p0.95    sample                ≈ 10⁻⁶            s/op
StringBenchmark.format:p0.99    sample                ≈ 10⁻⁶            s/op
StringBenchmark.format:p0.999   sample                ≈ 10⁻⁵            s/op
StringBenchmark.format:p0.9999  sample                ≈ 10⁻⁴            s/op
StringBenchmark.format:p1.00    sample                 0.001            s/op
StringBenchmark.append              ss                ≈ 10⁻⁶            s/op
StringBenchmark.format              ss                ≈ 10⁻⁵            s/op
2025-12-25 11:49:06 +08:00
jianghang.loujh
d18bde71c6 merge and support polardb-x recycle_bin binlog format 2025-12-25 11:47:36 +08:00
sunxien
5a5e60d371 feat(canal/adapter): adapter支持prometheus暴露jvm指标 (#5549)
- adapter 默认提供prometheus暴露JVM指标

- 指定 maven-antrun-plugin 未指定版本号,默认是 3.0.0,但是配置中使用 <tasks> 标签,编译报错。
   - 降级版本号,显式指定 1.8 版本
   - 或者把配置中的 <tasks> 标签改成 <target>
2025-12-25 11:41:08 +08:00
jianghang.loujh
af9375440d fixed empty config 2025-08-06 10:18:37 +08:00
jianghang.loujh
9f2fc7409f merge issue #5496 , 修复mysql rds oss订阅本地binlog过期需要再次订阅oss 2025-08-05 16:49:47 +08:00
jianghang.loujh
556d998db1 adjust preferIPv4Stack=false, support ipv6 2025-08-05 16:42:00 +08:00
jianghang.loujh
25501db3ac migrate mvn deploy from OSSRH to maven central 2025-08-05 14:51:14 +08:00
jianghang.loujh
8523a51623 fixed JSON datatype string parse 2025-08-05 14:26:57 +08:00
dingxiaobo
16831825b8 fix JSON datatype string value truncated by '\0' (#5482) 2025-08-05 14:22:00 +08:00
jianghang.loujh
1e37fe2dee fixed polardbx panda index 2025-08-05 14:14:01 +08:00
jianghang.loujh
5607444bad merge from github and rebase 2025-07-28 14:38:10 +08:00
jianghang.loujh
3a7223e460 merge from github 2025-07-28 14:37:47 +08:00
Canlin Guo
553438e227 CI: add a workflow for CI (#5483)
* Create SECURITY.md

* CI: Create maven.yml

* Update maven.yml

* Update maven.yml

* fix bugs of fastsql schema test
2025-07-28 14:33:52 +08:00
Canlin Guo
a3ea3d32d9 Create SECURITY.md (#5472) 2025-06-27 19:05:52 +08:00
Sunxien®
50a570290a fix(canal/client-adapter): 修复adapter插件重试错误缺陷 (#5427)
* fix(canal/prometheus): 修复延迟指标的缺陷

-【缺陷描述】
   Admin控制台修改canal.properties配置后,Server自动进行重启,在Grafan上观测到同步实例的PUT/GET/ACK延迟居高不下。

-【原因定位】
   源库心跳正常触发,MemoryStoreWithBuffer正常推进位点,profiling 正常统计,但是延迟指标依然是越来越来越高。Prometheus调用collect接口采集到的exec time时间始终是固定不变的。经debug排查到StoreCollector采集器内存hold的StoreMetricsHolder与CanalInstance实例中的引用已经不同啦,CanalIntance重启时已经被重建过一份新的实例,但是StoreMetricsHolder却没有保存到内存Hold中。原因是Map.putIfAbsent调用引起。

-【修复效果】
- 修复后,重启Server心跳正常推进,延迟瞬间降下来。梳理其他Collector代码都是调用Map.put,只有这里使用putIfAbsent可能是粗心导致吧。修复效果

* fix(canal/client-adapter): 修复adapter插件重试错误缺陷

-【缺陷描述】
  adapter运行期间,针对server或者Instance进行过一次重启或者断网测试,发现adapter通过client拉取数据报错以后持续打印错误日志,永远不得恢复。虽然adapter processor 针对错误有重试逻辑,但是client的tcp连接已经发生中断,且无法恢复。

-【问题定位】
  adapter processor 针对错误的处理比较粗,需要细分是从上游get message报错,还是下游sync data报错?
  如果是 get message报错,特别是tcp 断开,需要跳出重试,重新尝试重连。如果是下游 sync data 报错,再考虑重试写入。下游插件开发者自行保证连接重连机制、写入幂等。框架只提供重试。

- 说明:顺带更新一下 .gitignore 文件
2025-05-16 15:12:42 +08:00
Sunxien®
6b7ddbbe92 fix(canal/prometheus): 修复延迟指标的缺陷 (#5426)
-【缺陷描述】
   Admin控制台修改canal.properties配置后,Server自动进行重启,在Grafan上观测到同步实例的PUT/GET/ACK延迟居高不下。

-【原因定位】
   源库心跳正常触发,MemoryStoreWithBuffer正常推进位点,profiling 正常统计,但是延迟指标依然是越来越来越高。Prometheus调用collect接口采集到的exec time时间始终是固定不变的。经debug排查到StoreCollector采集器内存hold的StoreMetricsHolder与CanalInstance实例中的引用已经不同啦,CanalIntance重启时已经被重建过一份新的实例,但是StoreMetricsHolder却没有保存到内存Hold中。原因是Map.putIfAbsent调用引起。

-【修复效果】
- 修复后,重启Server心跳正常推进,延迟瞬间降下来。梳理其他Collector代码都是调用Map.put,只有这里使用putIfAbsent可能是粗心导致吧。修复效果
2025-05-16 15:09:22 +08:00
kkz-01
2b53164253 解决8.0.30+使用cahcing_sha2_password时Fastauth报错问题 (#5406)
Co-authored-by: zhangkai <zhangkai02@xgd.com>
2025-05-16 15:06:36 +08:00
Sunxien®
e101287c25 fix(canal/startup.sh): 修复生产环境ZK启用sasl安全认证 (#5438)
* fix(canal/startup.sh): 修复生产环境ZK启用sasl安全认证

- 如果生产环境zk启用sasl安全认证,canal-server/canal-adapter启动无法连接zk进行读写操作
- 默认支持用户将 jaas.conf 放置在 ./conf/ 目录下,启动脚本增加-Djava.security.auth.login.config 启动参数即可

* fix(canal/parent): upgrade h2 version

- runtime error: Value too long for column "CHARACTER VARYING" #5086

- Ref: https://github.com/alibaba/canal/issues/5086
2025-05-16 15:04:02 +08:00
elegance code
b153e61f0d fix issue #5401 (#5402) 2025-05-16 15:03:01 +08:00
hygugb
9bd4357f37 fix: tsdb存储元数据时支持大对象(大于64M) (#5393) 2025-05-16 15:00:19 +08:00
jianghang.loujh
dad209feb8 fixed issue #5453 , skipping Query_log_event by unknown status vars 2025-05-16 14:57:54 +08:00
jianghang.loujh
e21e98aa83 fixed CanalAdmin instanceLog and polardbx columnar test 2025-03-19 15:51:27 +08:00
nakeiven
c998b3f100 Feature/adapter postgresql 20250124 (#5390)
* adapter tcp模式mysql数据全量同步至postgresql时 报错
fix issue 2146

* fix issue 1980

* feat 增加 当dbType 为 postgresql时,返回双引号(避免表名为postgresql的关键字时,sql查询报错)

---------

Co-authored-by: guodong.zhang <zgd8988@126.com>
2025-01-24 17:14:41 +08:00
jianghang.loujh
c15129b0ac [maven-release-plugin] prepare for next development iteration 2025-01-16 16:07:08 +08:00
jianghang.loujh
0741ccda1e [maven-release-plugin] prepare release canal-1.1.8 2025-01-16 16:07:00 +08:00
jianghang.loujh
02fffc9e74 compatiable polardb-x 2025-01-13 22:07:12 +08:00
jianghang.loujh
5ca4c39313 compatiable percona & polardb-x 2025-01-13 22:02:09 +08:00
jianghang.loujh
3162d7093d fixed polardb-x variables(flashback & gpp) 2025-01-13 21:35:33 +08:00
jianghang.loujh
f350fbdc0a fixed mysql 8.0.30+ , support new variable 2025-01-08 10:10:33 +08:00
jianghang.loujh
2d0dcf0e5d fixed issue #5301, fixed caching_sha2_password full auth & client SHA2 2024-12-24 17:51:31 +08:00
jianghang.loujh
515c05c873 fixed issue #5301, fixed caching_sha2_password full auth 2024-12-24 16:39:21 +08:00
jianghang.loujh
7ee551cbf7 fixed issue #5301, fixed caching_sha2_password full auth 2024-12-24 16:33:41 +08:00
jianghang.loujh
923675aa49 fixed issue #5334 , adjust canal_manager default schema 2024-12-17 16:06:57 +08:00
jianghang.loujh
ebb541001c fixed issue #5337 , remove docker privileged 2024-12-17 15:52:01 +08:00
Chongzhi Deng
44ed76ebb8 fix typo (#5299)
Signed-off-by: BruceAko <chongzhi@hust.edu.cn>
2024-12-17 15:32:03 +08:00
jianghang.loujh
fe836f7515 fixed issue #5336 #5332 , etl bugfix 2024-12-17 15:29:51 +08:00
quanllong
b30544c750 bugfix: 兼容MariaDB GTID 为空的情况 (#5352) 2024-12-17 15:09:41 +08:00
lol
66483ef83f Feature/helm charts (#5285)
* canal-admin canal-server k8s helm chart

* 拼写错误
2024-10-14 16:48:05 +08:00
jianghang.loujh
b391188baf code format & fine-tune 2024-10-14 16:44:59 +08:00
lol
2a7a507987 fixed health.sh failed (#5281)
fixed health.sh failed
2024-10-14 16:44:15 +08:00
lanicc
f53749bfc5 fixed issue #5289 (#5290)
Co-authored-by: lanicc <liaonan@souche.com>
2024-10-14 16:18:21 +08:00
jianghang.loujh
80ca4366db fixed #5270 , add CanalMQStarter stop latch 2024-09-13 18:04:56 +08:00
lwd-coding
be0d945d7d 当zk出现session time out可能会导致instance出现多个消费者,从而出现乱序 (#5270) 2024-09-13 17:02:19 +08:00
jianghang.loujh
5a968b3715 fixed typo 2024-09-11 15:05:36 +08:00
jianghang.loujh
1e5b8a2021 fixed schema blackFilter & mysql 8.4 2024-08-29 13:07:26 +08:00
hbzheng8
ff82fd6532 直接close socket,解决TLS half-close policy报错问题 (#5240) 2024-08-29 12:47:50 +08:00
Yifan Xia
b060f46ccf Update AbstractRequest.java (#5208) 2024-08-29 12:47:09 +08:00
TK337
7e31b40531 support mysql 8.4 (#5231) 2024-08-29 12:43:06 +08:00
dingxiaobo
b2b3223d1d fix mysql 5.1 negotiate NPE. (#5255) 2024-08-29 12:42:31 +08:00
jianghang.loujh
fea4d8210a fixed pull #5191 , reformat 2024-07-16 15:02:10 +08:00
JZ
fb3d2c6811 add compressionType,enableChunking for pulsar client , upgrade dependencies for security (#5191)
* add support for listener name which is described in https://pulsar.apache.org/docs/next/concepts-multiple-advertised-listeners/

* upgrade for CVE-2023-20883  CVE-2022-1471  CVE-2023-20860

* add   enableChunking

* downgrade  pulsar client version

* fix

* 增加压缩算法
2024-07-16 14:59:29 +08:00
jianghang.loujh
3439c48948 fixed pull #5175 , reformat 2024-07-16 14:57:53 +08:00
iamssx
2b8a309ec7 fixed Rocketmq can not batch send messages when producer set namespace (#5202) 2024-07-16 14:51:56 +08:00
Yazid Lee
9373429015 fix: resource loading failure caused by incorrect parameter order in StringUtils.startsWithIgnoreCase (#5173) 2024-06-26 15:53:24 +08:00
lwd-coding
9493ff733d fix: 同个instance配置多个group避免共享CanalMsgConsumer (#5175) 2024-06-26 15:49:48 +08:00
Tass
31530dc9af fix(adapter): add args into adapter/restart.sh (#5149)
Signed-off-by: Tass <tasszz2k@gmail.com>
2024-06-26 15:34:30 +08:00
jianghang.loujh
b0f3ff8dcc fixed pull #5147, reformat & add more configs 2024-06-26 15:33:14 +08:00
dingxiaobo
5c61f2fb56 Feat, support SSL(TLS), improve SHA2 password. (#5147)
* feat #4489, mysql SSL

* feat #4489, mysql SHA2 and SSL

* merge master, fix conflict.

* improve SHA2 auth plugin, fix SSL status print.

* SSL key store, use url rather than path, keep same as JDBC driver.
2024-06-26 15:06:16 +08:00
jianghang.loujh
7bb619cf12 fixed #5154,avoid spring RCE 2024-06-06 15:29:17 +08:00
lulu2panpan
d66a15fe35 migrate sql parser from druid to polardbx-parser (#5097)
* migrate parser from druid to polardbx-parser

* migrate parser from druid to polardbx-parser(with version 5.4.19)

* fix bug : full ddl sqls should not assign to extra field but data field

* fix bug : full ddl sqls should not assign to extra field but data field

---------

Co-authored-by: agapple <jianghang.loujh@alibaba-inc.com>
2024-05-25 22:05:06 +08:00
jianghang.loujh
aa05d8e2a0 reformat 2024-05-22 09:46:13 +08:00
leoloel
5ff6449620 fix issue #5127 escape key of json data (#5130) 2024-05-22 09:41:50 +08:00
ziyou
bbea802914 feat(es8): es8 支持证书认证 (#5106) 2024-04-28 09:54:53 +08:00
JZ
e2219c88dd add support for listener name which is described in https://pulsar.apache.org/docs/next/concepts-multiple-advertised-listeners/ (#5118) 2024-04-28 09:53:39 +08:00
jianghang.loujh
a92e61f5a9 fixed issue #5073 , optimize docker build 2024-04-15 18:52:42 +08:00
jianghang.loujh
eaacc01b77 fixed issue #5094 , support function unique index 2024-04-15 15:34:50 +08:00
jianghang.loujh
caa46f3672 merge pull #5089 2024-04-03 10:45:29 +08:00
jianghang.loujh
c0ff058fe1 remove unnecessary bean copy 2024-03-18 15:58:53 +08:00
jianghang.loujh
059f21208d fixed issue #5066 , es7.x npe 2024-03-18 14:19:45 +08:00
jianghang.loujh
8ca900377a merge & reformat code 2024-03-08 11:58:46 +08:00
jackzlz
6dffcc8573 yml解析支持占位符 (#5045) 2024-03-08 11:56:30 +08:00
wbxlll
711971599f fix issue client-adapter.rdb模块在转换时间时丢失精度 #5052 (#5053)
* fix the issue that client-adapter.rdb module loses precision when converting microsencond time

* fix the issue that client-adapter.rdb module loses precision when converting to java.sql.Time or java.sql.Timestamp

* fix the issue that client-adapter.rdb module loses precision when converting to java.sql.Time or java.sql.Timestamp
2024-03-08 11:50:53 +08:00
Devin Zhang
87c7a95523 startup.sh支持JDK17启动 (#5060) 2024-03-08 11:48:16 +08:00
ChengDaqi2023
e255baff11 update com.rabbitmq:amqp-client 5.5.0 to 5.18.0 (#5050) 2024-02-26 19:25:48 +08:00
jianghang.loujh
d457cd835e fixed issue #4819 , optmize code 2024-02-05 17:01:33 +08:00
jianghang.loujh
cf9c64629e optimize split IPAndPort 2024-02-05 16:49:05 +08:00
jianghang.loujh
307547746c optimize test & connected mistake 2024-02-04 13:59:29 +08:00
jianghang.loujh
c7d5ed4ba1 fixed issue #5036 , connected mistake 2024-02-04 13:58:52 +08:00
jianghang.loujh
b9ed5caf05 format 2024-02-04 13:35:10 +08:00
jianghang.loujh
348d218120 fixed issue #5029 , add deserializer message 2024-02-04 13:28:27 +08:00
zoemak
e2a64daad8 fixed issue #5037 ,使用client-adapter.rdb模块进行ETL同步,源库为MySQL,数据库名有短横线,读取源数据时报SQL异常 (#5044) 2024-02-04 13:07:51 +08:00
jianghang.loujh
1413cc9196 fixed issue #4899 , druid SchemaRepository support quota 2024-01-15 15:14:11 +08:00
jianghang.loujh
04c11ec8eb fixed issue #5017 , json partial update fixed & format 2024-01-02 16:42:01 +08:00
Wang Minan
54145ced5e fix parser failed when binlog_row_value_options=partial_json (#5018)
* 修改json部分更新的问题

* 保证after-image的情况下位点回退完整
2024-01-02 10:44:40 +08:00
agapple
fb35dd4254 Update README.md 2023-12-29 12:12:22 +08:00
jianghang.loujh
e233bec66a fixed issue #5012 , support mariadb 11 for Q_CHARACTER_SET_COLLATIONS 2023-12-29 12:08:28 +08:00
jianghang.loujh
67e0928d80 fixed security bugfix #4941 , process auth failed 2023-12-22 10:29:38 +08:00
jianghang.loujh
93dac2d335 format clickhouse adapter code 2023-12-04 09:46:23 +08:00
xander-os
9baeff55e4 enhance clickhouse support (#4957) 2023-12-04 09:34:43 +08:00
Devin Zhang
ca79258e4f support rabbitmq ssl (#4965) 2023-12-04 09:28:26 +08:00
jianghang.loujh
0e4c5c40d7 format code 2023-11-09 15:35:37 +08:00
e345
c65477d97a loadVersionComment() add logger.warn info (#4943) 2023-11-09 15:31:33 +08:00
mask
fc3cfc5c0c Fixed bug where cp -f actually failed, (#4942)
because the symlink already existed.
2023-11-09 15:31:04 +08:00
jianghang.loujh
b4074206ca fixed issue #4940 , support percona 8.0.33 2023-11-08 16:37:24 +08:00
jianghang.loujh
98677c5bc8 fixed issue #4941 , remove default passwd 2023-11-08 10:58:43 +08:00
PeanutVvl
2c7e918f30 将 Logback 升级到 1.2.9 及以上版本 (#4934)
Co-authored-by: wangsc <Asdf1234>
2023-11-07 14:42:56 +08:00
jianghang.loujh
163f719971 fixed issue #4923 , support fastJSON Feature.LargeObject 2023-11-03 12:39:53 +08:00
jianghang.loujh
55a5e7592b fixed issue #4930 , check embeddedServerstate 2023-11-03 11:04:11 +08:00
jianghang.loujh
259008a7cd update config 2023-11-03 10:39:17 +08:00
marun
26ba84e27e Update KafkaCanalConnector.java (#4927)
update fastjson to fastjson2
2023-11-03 10:36:44 +08:00
jianghang.loujh
c8ef9552ec fixed #4907 , reset tsdb build context 2023-10-13 14:05:15 +08:00
jianghang.loujh
bff394042a fixed testcase 2023-10-09 14:31:46 +08:00
jianghang.loujh
b322201889 [maven-release-plugin] prepare for next development iteration 2023-10-09 14:27:37 +08:00
jianghang.loujh
ea8949298f [maven-release-plugin] prepare release canal-1.1.7 2023-10-09 14:27:33 +08:00
dack-su
ba924b9f1d update org.postgresql:postgresql 42.3.3 to 42.5.1 (#4898) 2023-10-09 13:46:20 +08:00
jianghang.loujh
7e0061fe93 build.sh add platform 2023-10-08 19:31:10 +08:00
jianghang.loujh
6440b744f2 fixed arm64 stack size too small, upgrade all to 1m 2023-10-08 18:55:06 +08:00
jianghang.loujh
066b022bb3 fixed arm64 , support mysql 8.0 2023-10-08 17:51:56 +08:00
jianghang.loujh
738f41049f fixed arm64 stack size too small, upgrade to 1m 2023-10-08 14:39:59 +08:00
jianghang.loujh
0f6b9d156a fixed concurrent set properties NPE 2023-10-07 21:54:34 +08:00
jianghang.loujh
ed754b5203 fixed issue #4884, osbase/osadmin image support amd64/arm64 2023-10-07 21:54:00 +08:00
小鹏
11f89163c1 fixed issue #4813, support metric.port = -1 (#4891) 2023-10-07 13:15:07 +08:00
cooper lee
cda330e49e Fix Configuration property name '-id' is not valid on canal 1.1.7 alpha 3 #4879 (#4887)
* Fix Configuration property name '-id' is not valid

* Fix Configuration property name '-id' is not valid
读取原配置文件
2023-09-21 10:53:47 +08:00
jianghang.loujh
f9f810226d fixed issue #4869 , support admin tsdb config 2023-09-11 10:57:43 +08:00
jianghang.loujh
d4a09704f2 optimize gtid 2023-09-07 09:48:58 +08:00
jianghang.loujh
34e9440383 fixed issue #4864 , connector.disconnect optimize 2023-09-04 19:34:06 +08:00
weirhp
d623bea71a 解决rdb下有多个配制,会导致配置相互覆盖的问题 (#4560)
恩baseProperties这个变量是共享的,不能修改这个变量的属性值。要不后续的配置都会改变。关联issue为4355
2023-09-04 17:23:26 +08:00
jianghang.loujh
5ea9259f0d upgrade kafka-client 2.4.0 2023-09-04 14:27:05 +08:00
jianghang.loujh
a216b55010 fixed password length 6-digit limit 2023-09-02 11:12:09 +08:00
jianghang.loujh
2786e06ccf format es8 2023-09-01 19:58:23 +08:00
ymz
63dfbf65df es8x adapter (#4640) 2023-09-01 19:56:03 +08:00
jianghang.loujh
d90066e01b upgrade spring cloud & spring boot version 2023-09-01 19:51:49 +08:00
jianghang.loujh
69cfbabb10 upgrade spring cloud & spring boot version 2023-09-01 17:20:26 +08:00
jianghang.loujh
ab30f3bd8d fixed issue #3350 ,upgrade netty 2023-09-01 14:12:35 +08:00
jianghang.loujh
681d03fbe7 upgrade dependency version for CVE 2023-09-01 13:59:05 +08:00
jianghang.loujh
f13cc4ee9f fixes issue #4388 , support mariadb 10.x log_bin_compress 2023-09-01 13:42:49 +08:00
jianghang.loujh
252ca0279f fixes issue #4388 , support mysql 8.0.20+ TransactionPayloadLogEvent parse 2023-08-25 16:57:22 +08:00
jianghang.loujh
7241af84f7 fixed issue #4742 , format 2023-08-22 17:52:38 +08:00
zard
159457cccd very very major:Adapter rdb bug fix (#4742)
* 异常点描述1

* 问题描述

* bug fix

rdb yml文件 编码格式问题,导致所有outerAdapter初始化失败;
kafka消息空跑,数据全都丢失 问题 fix
2023-08-22 17:50:12 +08:00
jianghang.loujh
d8a70d0c2f fixed issue #4822 , fix connection close 2023-08-22 17:32:42 +08:00
CVEDetect
ec247114fd Update pom.xml (#4591) 2023-08-22 17:25:13 +08:00
elizwy
f067f761c7 fix issue 4806, make Common.init success (#4830)
Co-authored-by: eli.zwy <wenyang.zhang@ipinyou.com>
2023-08-22 17:21:50 +08:00
jianghang.loujh
d4c8780f26 fixed issue #4786 , revert compact BigDecimal optimize 2023-08-22 17:14:01 +08:00
jianghang.loujh
894ef6e610 fixed issue #4659 , gtid NPE 2023-07-29 09:57:52 +08:00
jianghang.loujh
cc75d3dda4 fixed issue #4706 , upgrade fastjson 2.0.31 2023-07-29 09:12:52 +08:00
jianghang.loujh
d8169cbf71 fixed issue #4813, support metric.port = -1 2023-07-28 13:38:47 +08:00
jianghang.loujh
fccc6bc816 fixed #4665 2023-07-28 13:25:05 +08:00
jianghang.loujh
97aa441bca fixed PR format 2023-07-28 13:20:56 +08:00
高原
b8c2a757a0 增加对mysql的caching_sha2_password认证插件fullauth流程支持 (#4767) 2023-07-18 16:33:40 +08:00
TENET
c8de48bbd5 fix typo (#4720) 2023-07-18 16:21:44 +08:00
jianghang.loujh
08239ef23f Merge branch 'master' of github.com:alibaba/canal 2023-07-18 16:17:10 +08:00
施昱平
04a2b6f941 修复从远程下载rds-binlog文件到本地时响应头获取不到Content-Length导致的NPE的问题 (#4760) 2023-07-18 16:16:50 +08:00
jianghang.loujh
5f667c3574 Merge branch 'master' of github.com:alibaba/canal 2023-07-18 16:11:24 +08:00
toywei
bfb8cf4190 Update Dockerfile (#4775)
Set the Author field of the generated images. This instruction has been deprecated in favor of LABEL.
2023-07-18 16:10:07 +08:00
jianghang.loujh
18c38ee057 upgrade fastjson & druid 2023-04-23 09:54:01 +08:00
SofiaChenBot
82b3aed62b Update README.md (#4688) 2023-04-20 11:28:33 +08:00
lulu2panpan
a04a1912a8 add support for polardbx multi binlog streams (#4660)
* AbstractCanalInstance的subscribeChange方法,增加对AbstractEventParser的类型判断,以支持用户的自定义CanalEventParser

* MysqlEventParser对Gtid模式的处理需要考虑兼容性,支持系统平滑的从非gtid模式升级为gtid模式

* add support for polardbx multi binlog streams #4657

* support auto generate instance names #4657
2023-03-23 13:46:10 +08:00
Smile-cmd
5d7d19fdb1 增加rabbitmq配置,以持久化方式投递消息 (#4644)
Co-authored-by: wangheng <760261296@qq.com>
2023-03-16 18:42:37 +08:00
fuzhong-jiye
9edbe7e827 支持 mysql_clear_password 明文密码模式, 在一些开发、测试环境可能遇上。 (#4628)
Co-authored-by: zhongfu <zhongfu@tesla.com>
2023-03-16 18:39:11 +08:00
starrywu
b52c5d149b Update LogEventConvert.java (#4653)
fixed issue #4652,  解决mysql中类型为MEDIUMTEXT的字段的javaType在不同场景的解析不一致的问题
2023-03-16 17:47:21 +08:00
Frank Zhao
aada80ba38 doc: update readme (#4589)
Add contribution leaderboard badge into README file.

Signed-off-by: frank-zsy <syzhao1988@126.com>
2023-02-07 17:10:39 +08:00
jianghang.loujh
a10698e4a8 fixed issue #4491 , upgrade druid 1.2.15 2023-01-04 13:40:39 +08:00
jianghang.loujh
9d0a37241f fixed desc table case insensitive 2022-12-26 14:40:58 +08:00
dengyangguang
41a2a87a91 fix issue #4267 old和data长度不一样而产生的下标越界异常 (#4504) 2022-12-23 16:13:42 +08:00
shicongyang
62c4333790 canal中的flatMessage中添加一个参数,记录gtid #4509 (#4521) 2022-12-23 16:04:46 +08:00
rewerma
ea8fb5e5b8 修复client中的kafka consumer消费回滚的bug (#4532) 2022-12-14 19:45:25 +08:00
rewerma
cd798b71e9 add proxy for SPI (#4520)
* 测试类报错

* add proxy for SPI
2022-12-06 17:35:30 +08:00
rewerma
b937ee613d Revert 4516 feature/macy (#4519)
* 修复测试类报错 (#4516)

* optimize YAML config loader (#4332)

* fix bug BASE TABLE as table name (#4217)

* fix issues#4328 (#4329)

* docs: add nodejs canal client support (#4260)

Co-authored-by: zhangxunwei <zhangxunwei@dxy.cn>

* fix destination not encoded (#4279)

* 修复Canal指定时间戳启动失效,总是从最新的点位开始同步问题 (#4348)

* 支持用户自定义的CanalAlarmHandler

* RowsLogEvent增加对TableMapLogEvent判空检查,防止NPE异常

* 修复Canal指定时间戳启动失效,总是从最新的点位开始同步问题, issue: #4347

Co-authored-by: 云时 <mingya.wmy@alibaba-inc.com>

* fixed 4334 , support jdk8/jdk11

* fixed issue #4266 , typo

* fixed issue #4243 , support auto register for cluster = null

* fixed issue #4225 , support mysql version >= 8.0.26 heartbeat v2

* fixed issue #4308 , support query_log_event for maraiadb 10.10.1

* ignore compression event

* support jdk11

* support druid 1.2.12

* fixed mariadb 10.x

* sync canal-template.properties

* update fastjson & druid version (#4406)

* 修复升级2.0.4导致兼容的问题

* update druid & fastjson version

* update fastjson version

* update druid & fastjson version

* meta.dat文件数据丢失 (#4397)

* update fastjson & druid version (#4438)

* 局部变量线程安全,优先使用StringBuilder替换StringBuffer (#4472)

Co-authored-by: 夏亮 <xialiang@newerabc.com>

* 1. CanalController stop 需要同时将 embededCanalServer.stop (#4477)

2. ServerRunningMonitor 线程池未正常回收,线程池管理与 start/stop保持一致

* vuln-fix: Use HTTPS instead of HTTP to resolve dependencies (#4437)

This fixes a security vulnerability in this project where the `build.gradle`
files were configuring Gradle to resolve dependencies over HTTP instead of
HTTPS.

Weakness: CWE-829: Inclusion of Functionality from Untrusted Control Sphere
Severity: High
CVSSS: 8.1
Detection: OpenRewrite

Reported-by: Jonathan Leitschuh <Jonathan.Leitschuh@gmail.com>
Signed-off-by: Jonathan Leitschuh <Jonathan.Leitschuh@gmail.com>

Bug-tracker: https://github.com/JLLeitschuh/security-research/issues/9


Co-authored-by: Moderne <team@moderne.io>

Co-authored-by: Moderne <team@moderne.io>

* fix_ETL同步mysql关键字报错 (#4346)

Co-authored-by: foleyang <foleyang@cogobuy.com>

* optimize code

* use compact BigDecimal

* performance optimize , 1. cache string names 2. cache Charset

* performance optimize ,cache integer/long valueof

* 测试类报错

Co-authored-by: gongchangyou <gongchangyou@gmail.com>
Co-authored-by: ChanaLii <316529035@qq.com>
Co-authored-by: zhangxunwei <zhangxunweia@gmail.com>
Co-authored-by: zhangxunwei <zhangxunwei@dxy.cn>
Co-authored-by: tianpeidong <39491687+tianpeidong@users.noreply.github.com>
Co-authored-by: dataccs <dataccs@163.com>
Co-authored-by: 云时 <mingya.wmy@alibaba-inc.com>
Co-authored-by: jianghang.loujh <jianghang.loujh@alibaba-inc.com>
Co-authored-by: 温绍锦 <shaojin.wensj@alibaba-inc.com>
Co-authored-by: noaso <noasoso@gmail.com>
Co-authored-by: HumanPassenger <40585855+HumanPassenger@users.noreply.github.com>
Co-authored-by: 夏亮 <xialiang@newerabc.com>
Co-authored-by: 华仔 <591327356@qq.com>
Co-authored-by: Jonathan Leitschuh <jonathan.leitschuh@gmail.com>
Co-authored-by: Moderne <team@moderne.io>
Co-authored-by: 杰锅不是锅 <1105568074@qq.com>
Co-authored-by: foleyang <foleyang@cogobuy.com>

* Revert "修复测试类报错 (#4516)"

This reverts commit d899981354.

Co-authored-by: gongchangyou <gongchangyou@gmail.com>
Co-authored-by: ChanaLii <316529035@qq.com>
Co-authored-by: zhangxunwei <zhangxunweia@gmail.com>
Co-authored-by: zhangxunwei <zhangxunwei@dxy.cn>
Co-authored-by: tianpeidong <39491687+tianpeidong@users.noreply.github.com>
Co-authored-by: dataccs <dataccs@163.com>
Co-authored-by: 云时 <mingya.wmy@alibaba-inc.com>
Co-authored-by: jianghang.loujh <jianghang.loujh@alibaba-inc.com>
Co-authored-by: 温绍锦 <shaojin.wensj@alibaba-inc.com>
Co-authored-by: noaso <noasoso@gmail.com>
Co-authored-by: HumanPassenger <40585855+HumanPassenger@users.noreply.github.com>
Co-authored-by: 夏亮 <xialiang@newerabc.com>
Co-authored-by: 华仔 <591327356@qq.com>
Co-authored-by: Jonathan Leitschuh <jonathan.leitschuh@gmail.com>
Co-authored-by: Moderne <team@moderne.io>
Co-authored-by: 杰锅不是锅 <1105568074@qq.com>
Co-authored-by: foleyang <foleyang@cogobuy.com>
2022-12-06 16:43:25 +08:00
rewerma
9c6f6f5cdf 测试类报错 (#4518) 2022-12-06 16:40:11 +08:00
jianghang.loujh
869e831b04 performance optimize ,cache integer/long valueof 2022-11-16 19:41:41 +08:00
jianghang.loujh
f7a4979b15 performance optimize , 1. cache string names 2. cache Charset 2022-11-16 17:33:02 +08:00
jianghang.loujh
afef45500e use compact BigDecimal 2022-11-16 16:14:11 +08:00
jianghang.loujh
4c7cdbc7f3 optimize code 2022-11-16 10:27:14 +08:00
杰锅不是锅
279d503cf4 fix_ETL同步mysql关键字报错 (#4346)
Co-authored-by: foleyang <foleyang@cogobuy.com>
2022-11-16 10:24:56 +08:00
Jonathan Leitschuh
272b1d5d11 vuln-fix: Use HTTPS instead of HTTP to resolve dependencies (#4437)
This fixes a security vulnerability in this project where the `build.gradle`
files were configuring Gradle to resolve dependencies over HTTP instead of
HTTPS.

Weakness: CWE-829: Inclusion of Functionality from Untrusted Control Sphere
Severity: High
CVSSS: 8.1
Detection: OpenRewrite

Reported-by: Jonathan Leitschuh <Jonathan.Leitschuh@gmail.com>
Signed-off-by: Jonathan Leitschuh <Jonathan.Leitschuh@gmail.com>

Bug-tracker: https://github.com/JLLeitschuh/security-research/issues/9


Co-authored-by: Moderne <team@moderne.io>

Co-authored-by: Moderne <team@moderne.io>
2022-11-16 09:59:27 +08:00
华仔
fa0c08c170 1. CanalController stop 需要同时将 embededCanalServer.stop (#4477)
2. ServerRunningMonitor 线程池未正常回收,线程池管理与 start/stop保持一致
2022-11-16 09:57:59 +08:00
HumanPassenger
10ff148aba 局部变量线程安全,优先使用StringBuilder替换StringBuffer (#4472)
Co-authored-by: 夏亮 <xialiang@newerabc.com>
2022-11-16 09:55:18 +08:00
温绍锦
571fa0d6fb update fastjson & druid version (#4438) 2022-11-16 09:49:55 +08:00
noaso
aa29f12d96 meta.dat文件数据丢失 (#4397) 2022-11-16 09:44:11 +08:00
温绍锦
3bceb07c44 update fastjson & druid version (#4406)
* 修复升级2.0.4导致兼容的问题

* update druid & fastjson version

* update fastjson version

* update druid & fastjson version
2022-09-15 14:57:59 +08:00
jianghang.loujh
c16d6e7729 sync canal-template.properties 2022-09-13 23:33:39 +08:00
jianghang.loujh
fb6981356f fixed mariadb 10.x 2022-09-13 22:28:49 +08:00
jianghang.loujh
c055737e24 support druid 1.2.12 2022-09-13 15:03:30 +08:00
jianghang.loujh
ef7e21e4b2 support jdk11 2022-09-01 18:27:19 +08:00
jianghang.loujh
ce916ee407 ignore compression event 2022-09-01 18:27:08 +08:00
jianghang.loujh
92758b8c87 fixed issue #4308 , support query_log_event for maraiadb 10.10.1 2022-09-01 17:35:29 +08:00
jianghang.loujh
7585d9c98b fixed issue #4225 , support mysql version >= 8.0.26 heartbeat v2 2022-09-01 16:14:55 +08:00
jianghang.loujh
9d37bb87b7 fixed issue #4243 , support auto register for cluster = null 2022-09-01 13:31:01 +08:00
jianghang.loujh
2faecec749 fixed issue #4266 , typo 2022-09-01 13:02:02 +08:00
jianghang.loujh
0ef7b28c03 fixed 4334 , support jdk8/jdk11 2022-09-01 11:15:39 +08:00
dataccs
41bf3e2289 修复Canal指定时间戳启动失效,总是从最新的点位开始同步问题 (#4348)
* 支持用户自定义的CanalAlarmHandler

* RowsLogEvent增加对TableMapLogEvent判空检查,防止NPE异常

* 修复Canal指定时间戳启动失效,总是从最新的点位开始同步问题, issue: #4347

Co-authored-by: 云时 <mingya.wmy@alibaba-inc.com>
2022-08-24 14:21:47 +08:00
tianpeidong
4d4199c801 fix destination not encoded (#4279) 2022-08-11 11:31:28 +08:00
zhangxunwei
b390e74a18 docs: add nodejs canal client support (#4260)
Co-authored-by: zhangxunwei <zhangxunwei@dxy.cn>
2022-08-11 10:48:54 +08:00
ChanaLii
72c0e24cfd fix issues#4328 (#4329) 2022-08-11 10:44:01 +08:00
gongchangyou
3dc86330a8 fix bug BASE TABLE as table name (#4217) 2022-08-11 10:41:01 +08:00
rewerma
eee73dc4b1 optimize YAML config loader (#4332) 2022-08-01 20:15:14 +08:00
jianghang.loujh
a96db39420 add polardb-x parser test 2022-06-13 09:39:29 +08:00
温绍锦
bac1d543f1 update fastjson & druid version (#4224)
* 修复升级2.0.4导致兼容的问题

* update druid & fastjson version
2022-06-13 09:31:46 +08:00
jianghang.loujh
753fad6639 fixed issue #4216 , support polardb-x show tables 2022-06-10 13:36:39 +08:00
jianghang.loujh
bc4dceea1a fixed issue #4216 , support polardb-x show databases 2022-06-10 11:37:52 +08:00
温绍锦
7aaee7b439 修复升级2.0.4导致兼容的问题 (#4202) 2022-05-30 10:29:38 +08:00
温绍锦
e19bf1ecff Fastjson2 compatible fix (#4189)
* update fastjson2 version, 2.0.2 -> 2.0.3

* update fastjson & druid version

* bug fix for fastjson2 compatible
2022-05-25 10:22:26 +08:00
jianghang.loujh
a99dc0cbe8 fixed testcase 2022-05-24 11:18:40 +08:00
jianghang.loujh
7b8ff04d20 [maven-release-plugin] prepare for next development iteration 2022-05-24 10:26:12 +08:00
jianghang.loujh
7ee246ca5a [maven-release-plugin] prepare release canal-1.1.6 2022-05-24 10:26:06 +08:00
jianghang.loujh
54c645f315 fixed test failed 2022-05-24 10:12:26 +08:00
jianghang.loujh
6c81ab6fdc fixed compiler error 2022-05-24 10:08:02 +08:00
jianghang.loujh
99aefaa118 support mysql 8.0 transaction_payload_event 2022-05-23 23:23:52 +08:00
jianghang.loujh
e2248b3d35 update phoenix pom.xml 2022-05-23 19:30:19 +08:00
jianghang.loujh
eef1ee85fb fixed issue #4014, support java8 & java11 G1 GC 2022-05-23 18:23:31 +08:00
温绍锦
7dfec19c68 update druid & fastjson2 version (#4183)
* update fastjson2 version, 2.0.2 -> 2.0.3

* update fastjson & druid version
2022-05-23 10:03:41 +08:00
jianghang.loujh
05a52f1bd3 upgrade log4j version 2022-05-20 16:58:20 +08:00
jianghang.loujh
fb0f274ba8 fixed logger 2022-05-20 16:09:53 +08:00
jianghang.loujh
d71a208123 fixed issue #4178 , support -00:00:00.12 2022-05-20 10:37:53 +08:00
jianghang.loujh
656a1d23c2 instance.properties demo 2022-05-19 19:56:50 +08:00
jianghang.loujh
a574b91fdf fixed #3976 , support LogEvent.MYSQL_TYPE_TYPED_ARRAY 2022-05-19 19:53:54 +08:00
jianghang.loujh
e41e903984 fixed issue #3538 , compatiable mysql8.0 filed meta 2022-05-17 17:17:32 +08:00
jianghang.loujh
4a22d72ef4 fixed issue #4018, fixed CanalInstanceWithManager thread-safe 2022-05-16 19:46:34 +08:00
jianghang.loujh
b7eff723a3 fixed issue #4027, CanalServerWithEmbedded.get support newList 2022-05-16 19:41:46 +08:00
jianghang.loujh
9b9cfe4037 fixed issue #4152, TableMetaTSDB support canal.instance.filter.query.ddl=true 2022-05-16 17:25:43 +08:00
jianghang.loujh
ee51f338e0 fixed outAdapterkey autoGen 2022-05-13 15:13:14 +08:00
zhuchao941
4315f203a2 repair issue #3401 (#3923)
* 解决OutAdapter单例导致的问题,如 1.多个es7 outAdapter的多线程问题(共享同一个es bulk request),2.多个es7 outAdapter共用一份es配置问题

* 1. 优化adapter和监听器之间的代码结构
2. 监听器只处理匹配的配置文件
3. bugfix-es adapter配置文件回调时漏写esVersion
理论上一个adapter实例对应一个config配置文件

Co-authored-by: agapple <jianghang.loujh@alibaba-inc.com>
2022-05-13 15:02:06 +08:00
jianghang.loujh
b12945d4a9 format yml 2022-05-13 14:39:57 +08:00
Treeyh
885f05fb6a 支持定义rocketmq client日志路径,及修复一些bug (#3229)
* 修复接入阿里云RocketMq no route info for this topic 问题

* 支持deployer同步rocketmq时定义rocketmq client日志路径

* 支持deployer同步rocketmq时定义rocketmq client日志路径

* 回退guava版本号,解决adapter的java.lang.NoSuchMethodError: com.google.common.util.concurrent.MoreExecutors.sameThreadExecutor()异常

* 解决adapter启动报com.alibaba.druid.pool.DruidDataSource cannot be cast to com.alibaba.druid.pool.DruidDataSource异常。

* 支持rocketmq client 配置日志路径

Co-authored-by: hai.yu <hai.yu@snowballtech.com>
Co-authored-by: agapple <jianghang.loujh@alibaba-inc.com>
2022-05-13 14:39:29 +08:00
ChengTie(鐵手)
591126ee6b update (#3326) 2022-05-13 14:37:17 +08:00
He Wang
af432bf10a fix data truncated for column issue for type year (#3866) 2022-05-13 14:33:06 +08:00
jumao
b532e90ff4 ESAdapter Support the same instance cross -library query (#4161)
* ESAdapter Support the same instance cross -library query

* fix null point ex
2022-05-13 14:31:09 +08:00
xiaoyu.zhang
e48bc35f35 通过索引别名获取mapping,返回数据为真实索引,而不是别名,导致mappings.get(index).get(type)报空指针异常 (#4122) 2022-05-13 14:28:53 +08:00
He Wang
11a5f2a0fa add druid stat filter (#3985) 2022-05-13 14:24:37 +08:00
He Wang
c775478a56 set backtick by DbType (#3984)
Co-authored-by: agapple <jianghang.loujh@alibaba-inc.com>
2022-05-13 14:23:40 +08:00
He Wang
bd1f91cd9c fix sync order in RdbMirrorDbSyncService (#3928) 2022-05-13 14:19:40 +08:00
jianghang.loujh
67e85d201c remove client adapter Docker 2022-05-13 12:46:12 +08:00
wanglei
6965726dfe fix bug #2023 (#3871)
* fix bug #2023

* 配置canal-adapter,docker生成脚本
2022-05-13 12:39:43 +08:00
willmao
c0001afe8f use hostname if register name not set (#3968) 2022-05-13 12:24:33 +08:00
xiayingfeng
977031a666 Improve MAVEN build Performance (#4055) 2022-05-13 12:23:33 +08:00
zhangshaoning
bd3224eea1 Upgrade reference log4j package version (#4017) 2022-05-13 12:22:45 +08:00
jianghang.loujh
33733aef25 fixed compatible and code format 2022-05-13 11:59:57 +08:00
zhangjukai
25be0259a4 Some tweaks for pulsarmq-connector (#4060)
* 1. Add the com.alibaba.otter.canal.connector.core.spi.CanalMsgConsumer file to the META-INF/canal directory under the pulsarmq-connector project
2. Introduce pulsar-client-admin for Canal to automatically create multi-partition topics
3. Add the judgment that roleToken is null
4. The disconnect method in CanalPulsarMQConsumer removes this.pulsarMQConsumer.unsubscribe();, this code will cause data loss during stop
5. When getting Pulsar messages, they are all processed as flat messages, because CanalMessageSerializerUtil.deserializer(data) will deserialize exceptions
6. Use groupId as subscriptName, without the pulsarmq.subscriptName parameter, the entire adapter will be the same subscriber name using pulsarmq.subscriptName

* 逻辑优化

* 代码优化

* 针对pulsar的调整

* 恢复代码

* 恢复代码

* 处理Producer失效的情况
2022-05-13 11:41:50 +08:00
liteng2430
8114d77a20 Add verification: the user can set a password with a length of less than 6 digits when updating the password, but the password length must be greater than or equal to 6 digits when logging in to canal admin, so the length verification function when updating the login password is added (#4103) 2022-05-13 11:34:36 +08:00
ZhangJian He
571e7f4e25 fix ojdbc6 wrong corrinator (#4132) 2022-05-13 11:33:42 +08:00
温绍锦
ead91433d7 fastjson update 1.x to 2.0.2 (#4160) 2022-05-13 11:32:47 +08:00
xiaoma20082008
bac016f89e <feat>: support visibility columns (#3976) 2022-05-13 11:30:19 +08:00
willmao
8804647c30 avoid print password when logging (#3969) 2022-05-13 11:28:05 +08:00
mclubing
2623d08b8c rdb配置不支持yaml后缀文件 (#3964)
增加rdb配置支持yaml后缀文件
2022-05-13 11:26:04 +08:00
wuare
622fed84ce fixed logger name (#3887) 2022-05-13 11:24:00 +08:00
He Wang
9f5e8fd1c4 rollback guava version in adapter module (#3880) 2022-05-13 11:23:30 +08:00
Junwang Zhao
88d472ef1e move protocol test to protocol test directory (#3864)
1. move the protocol test to right directory
2. fix wrong string format params count

Signed-off-by: 元组 <zhaojunwang.zjw@alibaba-inc.com>
2022-05-13 11:16:06 +08:00
Aiden.Ao
2987d08161 fix issue #3144 (#3712) 2022-05-13 11:09:09 +08:00
tuple
b54bea5e33 polish the code thus make it more stable (#3855)
Signed-off-by: 元组 <zhaojunwang.zjw@alibaba-inc.com>
2021-10-11 14:46:22 +08:00
jianghang.loujh
c42ba95eed fixed pulsarmq package 2021-10-09 18:06:34 +08:00
jianghang.loujh
b48c434ffd fixed phoenix version 2021-10-09 17:52:13 +08:00
jianghang.loujh
2c892fce9b fixed issue #3484 , fixed bat script 2021-10-09 17:39:22 +08:00
jianghang.loujh
3bb49e19d2 fixed issue #3576, stop exporter 2021-10-09 17:29:23 +08:00
jianghang.loujh
f26b84ffc2 fixed issue #3829 , kafka connector support k8s env 2021-10-09 16:31:59 +08:00
364102729
0a2254eeaf add tablestore adapter (#3754)
* add tablestore adapter

* add tablestore adapter

* fix bug add add log when etl fail

* fix bug,tinyint(1)以及rowupdatechange

Co-authored-by: agapple <jianghang115@gmail.com>
2021-10-09 16:06:13 +08:00
jianghang.loujh
40d840375f fixed MR 2021-10-09 15:59:12 +08:00
Chad2li
5b6fd8094a 增加Apache Pulsar MQ支持 (#3791)
* done pulsarmq

* del pulsar consumer

* pulsar partition

* done pulsar consumer&connector

* update pulsar-client version to 2.8.1

Co-authored-by: agapple <jianghang115@gmail.com>
2021-10-09 15:51:45 +08:00
mikoto
4ce401d219 fix: 多destination的场景下只能生效一个 (#3772) 2021-10-09 15:38:26 +08:00
Jonathan Schneider
de95a6a128 refactor: Use Java standard library instead of Guava (#3708)
Co-authored-by: Moderne <team@moderne.io>

Co-authored-by: Moderne <team@moderne.io>
2021-10-09 15:33:37 +08:00
happyrita
9e11c3c018 定时保存snapshot到DB的逻辑中,schema名称获取错误,在有配置canal.instance.filter.regex时,由于对应的table无法被匹配中,导致compareTableMetaDbAndMemory方法不被执行 (#3693)
* 避免重复订阅zk变更状态

* Update DatabaseTableMeta.java

获取schema的名称

* Update CanalController.java
2021-10-09 15:32:20 +08:00
arthasking123
d67bea8f3d repair issue #3678,#3680 (#3684)
* repair issue #3678,#3680

* repair issue #3678,#3680
2021-10-09 15:30:32 +08:00
kkjinping
997957d6e0 新增配置 canal.mq.enableDynamicQueuePartition,获取topic对应的队列的数量为分区的数量,以支持动态队列的场景(如阿里云的rocketmq,无法人工设置队列数量,可能会动态伸缩) (#3670) 2021-10-09 15:26:23 +08:00
tanpenggood
ea20076a32 fix: rowChange typo (#3593) 2021-10-09 15:23:36 +08:00
dinosaur
2550d8eb65 Connection is already closed fix (#3587) 2021-10-09 14:54:05 +08:00
nick
31a0fdc807 add note (#3550) 2021-10-09 14:53:28 +08:00
lanicc
d79431c7e5 fix typo (#3532) 2021-10-09 14:53:00 +08:00
ruanjl
911711248c 修复rds高可用主从切换和oss相关问题 (#3480)
* 修复rds oos 拉取的binlog 的消费逻辑以及本地消费到直连消费的逻辑衔接

* 去除无用代码
2021-10-09 14:51:41 +08:00
spark
63407dc570 canal adapter phoenix (#3290)
* canal adapter phoenix

* canal adapter phoenix

* canal adapter phoenix

* canal adapter phoenix

* canal adapter phoenix

* canal adapter phoenix

* canal adapter phoenix

Co-authored-by: lihuav2017 <lihua@able-elec.com>
2021-10-09 14:37:11 +08:00
flowell
3c20fd09ad fix issue #3839: 修复alter table character set类型DDL语句解析BUG (#3840) 2021-10-09 14:28:56 +08:00
ChenZhangg
eef3bd92ba Improve MAVEN build Performance (#3851)
Co-authored-by: Chen Zhang <340355960@qq.com>
2021-10-09 14:26:26 +08:00
agapple
474063af3b fixed issue #3494 , polardb-x 2.0 cdc support implicit_id/varaibles 2021-04-28 13:42:55 +08:00
agapple
5cc85ff2de [maven-release-plugin] prepare for next development iteration 2021-04-19 15:48:20 +08:00
agapple
64862503ba [maven-release-plugin] prepare release canal-1.1.5 2021-04-19 15:48:09 +08:00
agapple
f9e0dc7daa fixed testcase 2021-04-19 15:15:10 +08:00
agapple
0c399509a9 upgrade druid 1.2.6 2021-04-19 12:57:21 +08:00
agapple
a09cd9adb5 fixed issue #3110 , json escapse 2021-04-19 11:28:45 +08:00
agapple
1369418c6a fixed code format & merge 2021-04-19 10:55:43 +08:00
995586041
75174a63af 增加rabbitmq消息持久化deliveryMode配置 (#3300) 2021-04-19 10:29:23 +08:00
zoemak
17f4ee13e7 自动注册时可自定义node server名称,canal.properties#canal.admin.register.name,为空则使用canal.register.ip (#3459) 2021-04-19 10:18:01 +08:00
Fei Yang
f0417502f8 fix: unit_present typo (#3461)
https://github.com/alibaba/canal/issues/3434
2021-04-19 10:16:17 +08:00
renyansongno1
a7720b51e4 支持rocketmq的消息按照tag去发送 (#3438)
* 支持rocketmq的消息按照tag去发送
并对应修改了配置文件

* 将tag的设计转移到Rocketmq的单独配置项
2021-04-19 10:14:27 +08:00
suman-lee
21904ef1dc 基于明确的I/U/D是否过滤 (#3452)
* dmls filter

* dmls filters

* 基于明确的I/U/D是否过滤

* 基于明确的I/U/D是否过滤
2021-04-19 10:10:54 +08:00
agapple
94425223bd fixed issue #3454 , concurrent env bugfix 2021-04-17 10:57:25 +08:00
agapple
7857df9701 add sql parser testcase 2021-04-17 10:49:17 +08:00
agapple
64ea4ff222 fixed issue #3267 , fixed rockemq send null message 2021-04-17 10:25:49 +08:00
agapple
85df89f777 fixed issue #3377 , escapse schema name 2021-04-16 23:11:41 +08:00
AlphaKitty
a0f8e182ec 修复当host中存在冒号(比如https:)抛UnknownHostException问题 (#3223)
* 修复当host中存在冒号(比如https:)抛UnknownHostException问题

* 更优雅地解析url
2021-04-16 22:11:21 +08:00
温绍锦
33721d0394 update druid version (#3146)
* replace fastsql with druid

* update druid version
2021-04-16 22:09:40 +08:00
Treeyh
91bddfeed6 修复接入阿里云RocketMq no route info for this topic 问题 (#3220)
Co-authored-by: hai.yu <hai.yu@snowballtech.com>
2021-04-16 22:08:28 +08:00
16075140
25feab4af8 support rabbitmq port by config (#3236) 2021-04-16 22:07:12 +08:00
binbin
f66baeabf1 修复最近位置被remove掉,导致重复消息的问题 (#3276) 2021-04-16 22:03:51 +08:00
agapple
b1dc3821ea fixed config url auth 2021-04-16 21:52:23 +08:00
Jump-Boy
f5f79104eb fix centos 6.10 yum bug and update jdk8.rpm download url (#3398) 2021-04-16 21:41:55 +08:00
ltamber
e8d138d3bd fix: uuidset combine fix (#3432) 2021-04-16 21:37:55 +08:00
zoemak
32d9e58915 升级rocketmq到4.8.0,并且支持环境变量优先配置 (#3450) 2021-04-16 21:35:11 +08:00
许道杰
58448e559d fix: issue #2649 client-adapter启动时抛LinkageError异常、#3073 client-adapter通过启动参数传入canal.tcp.zookeeper.hosts为空值时抛异常 (#3074) 2021-04-16 21:25:14 +08:00
许道杰
fede3b3d1d fix #3126 将hbase-client依赖修改为hbase-shaded-client (#3127) 2021-04-16 21:24:44 +08:00
许道杰
5d8ec31bc4 fix #3119 (#3121) 2021-04-16 21:22:31 +08:00
WangCong
82b18ebe62 feat unsigned bug (#3104) 2021-04-16 21:17:43 +08:00
hupeng
66408782b0 process()方法中,在retry循环中处理成功应该break (#3080)
process()方法中,在retry循环中处理成功应该break
2021-04-16 21:14:43 +08:00
许道杰
846c8870ae fix: issue #3076 (#3077) 2021-04-16 21:12:12 +08:00
ljcoder
8cfb07f07e 数据库名称如何存在中划线"-"情况下,SQL执行异常 (#3228)
Co-authored-by: liguowang <liguowang@aerozhonghaun.com>
2020-12-31 11:33:01 +08:00
文哥
d741425895 升级guava 至 22.0 (#3184)
* fix issue:#561 增加一个外部参数控制,允许位点不存在时自动重置到当前位点

* Add MariaDB GTID support

* remove client pom bad ns

* upgrade guava version to 22.0
2020-11-20 13:47:26 +08:00
文哥
20add85ff6 统一Canal 依赖的Spring 版本,升级ibatis 至mybatis (#3182)
* fix issue:#561 增加一个外部参数控制,允许位点不存在时自动重置到当前位点

* Add MariaDB GTID support

* 1. fix maven central repository url access error.(Update to https)
2. upgrade spring version to 5.0.5.RELEASE
3. upgrade ibatis to mybatis

* fix unit test run error
2020-11-20 09:24:56 +08:00
agapple
a5c09c521e Update README.md 2020-11-02 12:54:54 +08:00
agapple
c94276b1ac Update README.md 2020-11-02 12:53:35 +08:00
agapple
272d575c78 Update README.md 2020-11-02 12:44:59 +08:00
温绍锦
693f8d501a replace fastsql with druid (#3132) 2020-10-31 13:19:03 +08:00
温绍锦
bb1b6b9862 Update README.md 2020-10-23 14:41:31 +08:00
温绍锦
8ed82caaef Update README.md 2020-09-29 16:18:23 +08:00
文哥
de1dd70609 1. Add MariaDB gtid support 2. fix issue:#561 增加一个外部参数控制,允许位点不存在时自动重置到当前位点 (#3025)
* fix issue:#561 增加一个外部参数控制,允许位点不存在时自动重置到当前位点

* Add MariaDB GTID support
2020-09-24 17:56:40 +08:00
zavakid
09429c241f = inner class MessageEventFactory to be static 2020-09-23 11:37:17 +08:00
zavakid
9a332940d7 = remove unused import 2020-09-23 11:34:45 +08:00
zavakid
362974b1c2 = code cleanup: remove unnessary boxing 2020-09-18 00:54:23 +08:00
zavakid
435db30b7b = performance improve
- using bulk operation instead of iteration if possible
  - terminate loop after condition is met
2020-09-18 00:51:53 +08:00
zavakid
e7de8289cd = code cleanup : modified inner class to 'static' if possible 2020-09-18 00:49:48 +08:00
zavakid
5065fea963 = fix BooleanMutex's inner Sync to static class and add test case 2020-09-18 00:44:25 +08:00
zavakid
6989eb3d2b + fix logging call placeholders number 2020-09-17 23:26:02 +08:00
zavakid
df24596092 = code cleanup
- for loop replaceable with enhanced 'for' loop
  - using diamond in generic type
  - collapse try catch exception
  - using try-with-resource
  - using  List#sort instead of Collections#sort
  - replaced lambda with method reference
  - using native compare method to compare numbers
  - using switch instead of if for in ESSyncUtil
2020-09-17 23:22:25 +08:00
zavakid
43c23ffb97 = remove unused improt 2020-09-17 22:40:06 +08:00
zavakid
e7658879bc = cold cleanup
* using lambda if possible (we are already building in Java8)
  * using diamond when new generic instance
  * remove used improt
2020-09-17 17:46:20 +08:00
zavakid
df60eda679 = code cleanup 2020-09-16 14:18:05 +08:00
liguangyao
b81e9dfeb8 支持不同topic指定专属partitionNum (#2479) 2020-08-22 13:11:53 +08:00
agapple
b79af46054 merge code and fixed issue #2755 #2756 #2735 #2714 #2711 #2710 #2616 #2602 #2585 #2565 #2935 #2434 #2623 #2418 #2387 2020-08-22 13:07:24 +08:00
calvinhkf
90bf9e6de5 align h2 version to 1.4.196 (#2663) 2020-08-20 21:19:22 +08:00
zhaohua
182163c3bd MemoryTableMeta ddl 处理注释2 (#2694)
* MemoryTableMeta ddl 处理注释

* MemoryTableMeta ddl 处理注释2
2020-08-20 21:19:08 +08:00
dengliming
e4726972de refactor: delete duplicate code (#2748) 2020-08-20 21:18:51 +08:00
dengliming
985892cf7e Remove deprecated sudo setting. (#2749) 2020-08-20 21:18:37 +08:00
chank
746d51e621 Change maven repository to HTTPS (#2763)
* change maven repository to https

* change maven repository to https
2020-08-20 21:18:25 +08:00
caoyixiong
cdd077b758 FIX NPE When ddl event (#2801) 2020-08-20 21:18:12 +08:00
DinoZhang
72ae7ba8d7 meta admin (#2836) 2020-08-20 21:18:00 +08:00
Tocker
b82c8bca83 Fix:Alibaba Coding Guidelines-Object的equals方法容易抛空指针异常,应使用常量或确定有值的对象来调用equals。 (#2919) 2020-08-20 21:17:44 +08:00
Tocker
576406a4ce Fix:Alibaba Coding Guidelines-所有的包装类对象之间值的比较,全部使用equals方法比较 (#2918) 2020-08-20 21:17:32 +08:00
Ymangz
b5b08e92e3 Unbuntu 20.0 默认sh为dash执行时会报语法异常 (#2939) 2020-08-20 21:17:11 +08:00
wanshao
9f6021cf36 fix(dml): revert wrong fix 2020-03-13 17:41:53 +08:00
bucketli
0e3d719a39 refact(canal):duplicate dependence 2020-03-09 14:15:40 +08:00
agapple
afbf125231 fixed format 2020-03-05 15:26:59 +08:00
agapple
b2b2a82554 client-adapter 写入kudu (#2396)
* client-adapter 提交写入kudu适配器

* kudu etl 处理数据缓存提交问题

* Update KuduTemplate.java

* 1.写入kudu前获取元数据方式更改
2.解决etl批量导入kudu数据少量丢失问题

* 1.修改文件头注释
2.调整kudu更新事件update为upsert
2020-03-05 15:22:01 +08:00
wanshao
30a1b067a8 fix(project): no need to process tinyint(1) specially #631 2020-02-28 14:34:30 +08:00
wanshao
beaca165d0 Merge branch 'master' of github.com:alibaba/canal 2020-02-28 14:24:33 +08:00
wanshao
c21c18458b fix(project): fix isDdl judge #2599 2020-02-28 14:21:12 +08:00
agapple
5e761e14fa format 2020-02-28 13:15:27 +08:00
agapple
18418ebd98 merge pull/2120 2020-02-28 12:29:41 +08:00
844392610
92f04f86cb 添加目标表不区分大小写的支持,并可开关(默认为区分大小写) (#2329)
* 添加目标表不区分大小写的支持,并可开关(默认为区分大小写)

* 添加目标表不区分大小写的支持,并可开关(默认为区分大小写)-format
2020-02-28 12:22:17 +08:00
littleneko
0d5505aa2c fix_json_type_opaque_parse_err (#2318)
对于`JSONB_TYPE_OPAQUE`类型的解析计算`q_max_bytes`时,多算了`1`字节
2020-02-28 12:15:09 +08:00
zhshy11
bd56438eca sql语句增加引号,兼容mysql关键字 (#2358) 2020-02-28 11:47:38 +08:00
agapple
b9a157657f merge /pull/2373 2020-02-28 11:46:28 +08:00
agapple
523e8b85e6 format 2020-02-28 11:40:14 +08:00
agapple
ab74b2c82c fixed mq spi package 2020-02-28 11:32:30 +08:00
rewerma
265a8e8594 模块化mq connector (#2562)
* add connector module, add kafka producer connector module for server

* add rocketMQ producer connector module for server

* add rabbitMQ producer connector module for server

* optimize code, add annotation

* add canal message consumer connector for canal-adapter module
2020-02-28 11:00:08 +08:00
jianqingdu
4fc7e5f7d9 fix bug: wrong quit command implementation (#2554) 2020-02-28 10:52:35 +08:00
Li Qiang
9cc8caa834 feature(canal):support start with timestamp position(empty journal_file,set position=0),#2594 (#2595) 2020-02-28 10:50:38 +08:00
Li Qiang
a71375dad5 fix(canal):fix endless local binlog mode with exception. (#2597) 2020-02-28 10:48:54 +08:00
Rg
70d8a379f2 add Rust canal client support (#2518) 2020-01-15 20:12:28 +08:00
rewerma
b6333f018f Revert "修复kafka消费回滚的bug (#2521)" (#2526)
This reverts commit 31acfc0101.
2020-01-02 17:04:57 +08:00
rewerma
31acfc0101 修复kafka消费回滚的bug (#2521) 2020-01-02 16:15:47 +08:00
rewerma
6d9ea57186 修复子线程阻塞造成不能中止的问题 (#2508) 2019-12-26 19:29:48 +08:00
Li Qiang
c4c0255073 Update README.md 2019-11-04 10:20:00 +08:00
agapple
1fa9020599 format 2019-10-15 21:38:48 +08:00
agapple
dd7554374f format 2019-10-15 21:35:57 +08:00
shicongyang
65b5325ca6 建议增加一个属性,针对根据某一字段hash时,hash算法可以不根据database算出 #2248 (#2294) 2019-10-15 21:32:42 +08:00
agapple
b64a3e301a fixed issue #2286 , ArithmeticException zero 2019-10-10 15:37:17 +08:00
agapple
31562919bb rename canal-admin project to admin 2019-10-09 14:50:08 +08:00
819 changed files with 31915 additions and 13499 deletions

46
.github/workflows/maven.yml vendored Normal file
View File

@@ -0,0 +1,46 @@
# This workflow will build a Java project with Maven, and cache/restore any dependencies to improve the workflow execution time
# For more information see: https://docs.github.com/en/actions/automating-builds-and-tests/building-and-testing-java-with-maven
# This workflow uses actions that are not certified by GitHub.
# They are provided by a third-party and are governed by
# separate terms of service, privacy policy, and support
# documentation.
name: Java CI with Maven
on:
push:
branches: [ "master" ]
pull_request:
branches: [ "master" ]
permissions:
contents: read
jobs:
test:
runs-on: ${{ matrix.os }}
permissions:
contents: read
strategy:
matrix:
os: [ ubuntu-latest ]
java: [ 8, 11, 17, 21 ]
fail-fast: false
max-parallel: 16
name: Test JDK ${{ matrix.java }}, ${{ matrix.os }}
steps:
- uses: actions/checkout@11bd71901bbe5b1630ceea73d27597364c9af683 # v4.2.2
- name: Set up JDK
uses: actions/setup-java@c5195efecf7bdfc987ee8bae7a71cb8b11521c00 # v4.7.1
with:
distribution: 'temurin'
java-version: ${{ matrix.java }}
cache: 'maven'
- name: Build with Maven if test jdk8
if: ${{ matrix.java == '8' || matrix.java == '11'}}
run: ./mvnw -Pgen-javadoc clean package -B
- name: Build with Maven if test jdk17
if: ${{ matrix.java == '17' || matrix.java == '21' }}
run: ./mvnw -Penable-for-jdk17+,gen-code-cov clean package -B

3
.gitignore vendored
View File

@@ -7,6 +7,7 @@ test-output/
.settings/
tmp
temp
logs/
*.log
antx.properties
otter.properties
@@ -18,6 +19,8 @@ jtester.properties
*.rpm
client-adapter/example/
*.dat
.qwen/
.vscode/
conf/
canal-admin/canal-admin-ui/dist/
canal-admin/canal-admin-ui/node/

View File

@@ -14,7 +14,7 @@ cache:
install: true
script:
- travis_wait 30 ./mvnw clean install -DskipTests=false -Dcheckstyle.skip=true -Dmaven.javadoc.skip=true
- travis_wait 30 ./mvnw -T 1C clean install -DskipTests=false -Dcheckstyle.skip=true -Dmaven.javadoc.skip=true
after_success:
- bash <(curl -s https://codecov.io/bash)

View File

@@ -4,10 +4,12 @@
![license](https://img.shields.io/github/license/alibaba/canal.svg)
[![average time to resolve an issue](http://isitmaintained.com/badge/resolution/alibaba/canal.svg)](http://isitmaintained.com/project/alibaba/canal "average time to resolve an issue")
[![percentage of issues still open](http://isitmaintained.com/badge/open/alibaba/canal.svg)](http://isitmaintained.com/project/alibaba/canal "percentage of issues still open")
[![Leaderboard](https://img.shields.io/badge/Canal-%E6%9F%A5%E7%9C%8B%E8%B4%A1%E7%8C%AE%E6%8E%92%E8%A1%8C%E6%A6%9C-orange)](https://opensource.alibaba.com/contribution_leaderboard/details?projectValue=canal)
## 简介
![](https://github.com/bucketli/simpletools/blob/master/main_func_v2.png?raw=true)
![](https://img-blog.csdnimg.cn/20191104101735947.png)
**canal [kə'næl]**,译意为水道/管道/沟渠,主要用途是基于 MySQL 数据库增量日志解析,提供增量数据订阅和消费
@@ -55,7 +57,7 @@
- [Introduction](https://github.com/alibaba/canal/wiki/Introduction)
- [QuickStart](https://github.com/alibaba/canal/wiki/QuickStart)
- [Docker QuickStart](https://github.com/alibaba/canal/wiki/Docker-QuickStart)
- [Canal Kafka/RocketMQ QuickStart](https://github.com/alibaba/canal/wiki/Canal-Kafka-RocketMQ-QuickStart")
- [Canal Kafka/RocketMQ QuickStart](https://github.com/alibaba/canal/wiki/Canal-Kafka-RocketMQ-QuickStart)
- [Aliyun RDS for MySQL QuickStart](https://github.com/alibaba/canal/wiki/aliyun-RDS-QuickStart)
- [Prometheus QuickStart](https://github.com/alibaba/canal/wiki/Prometheus-QuickStart)
- Canal Admin
@@ -84,15 +86,26 @@ canal 特别设计了 client-server 模式,交互协议使用 protobuf 3.0 , c
- canal go客户端: [https://github.com/CanalClient/canal-go](https://github.com/CanalClient/canal-go)
- canal php客户端: [https://github.com/xingwenge/canal-php](https://github.com/xingwenge/canal-php)
- canal Python客户端:[https://github.com/haozi3156666/canal-python](https://github.com/haozi3156666/canal-python)
- canal Rust客户端:[https://github.com/laohanlinux/canal-rs](https://github.com/laohanlinux/canal-rs)
- canal Nodejs客户端:[https://github.com/marmot-z/canal-nodejs](https://github.com/marmot-z/canal-nodejs)
canal 作为 MySQL binlog 增量获取和解析工具,可将变更记录投递到 MQ 系统中,比如 Kafka/RocketMQ,可以借助于 MQ 的多语言能力
- 参考文档: [Canal Kafka/RocketMQ QuickStart](https://github.com/alibaba/canal/wiki/Canal-Kafka-RocketMQ-QuickStart)
## 相关开源
## 基于canal开发的工具
- canal2sql(基于binlog生成SQL) : [https://github.com/zhuchao941/canal2sql]
## 相关开源&产品
- [canal 消费端开源项目: Otter](http://github.com/alibaba/otter)
- [阿里巴巴去 Oracle 数据迁移同步工具: yugong](http://github.com/alibaba/yugong)
- [阿里巴巴离线同步开源项目 DataX](https://github.com/alibaba/datax)
- [阿里巴巴数据库连接池开源项目 Druid](https://github.com/alibaba/druid)
- [阿里巴巴实时数据同步工具 DTS](https://www.aliyun.com/product/dts)
## 问题反馈
- 报告 issue: [github issues](https://github.com/alibaba/canal/issues)
本项目的Issues会被同步沉淀至[阿里云开发者社区](https://developer.aliyun.com/ask)

118
SECURITY.md Normal file
View File

@@ -0,0 +1,118 @@
# 漏洞奖励计划
## 报告
如果您认为自己在本程序中发现了任何安全(技术)漏洞,欢迎您通过 https://security.alibaba.com 向我们提交漏洞报告。
如果您报告任何安全漏洞,请注意您可能包含以下信息(合格报告):
* git程序URL地址,运行的环境
* 包含必要屏幕截图的详细说明
* 重现漏洞的步骤以及修复漏洞的建议。
* 其他有用信息
## 处理
ASRC(Alibaba Security Response Center阿里安全响应中心)将尽快审核并回复您的提交内容,并在我们努力修复您提交的漏洞时随时通知您。如有必要,我们可能会与您联系以获取更多信息。
## 条款和条件
1. 仅接受技术漏洞并对其进行评级
2. 出于安全原因,上报者同意与ASRC合作完成他/她提交的漏洞,不向任何第三方透露任何漏洞信息
3. 如果不止一个人报告相同的安全漏洞,奖励将给予完成合格报告的第一个人
4. 为了保护程序的用户,请在修复之前不要直接提交git的issue,也不要在社区讨论任何漏洞信息
5. 所有奖励和声誉积分将提供给仅向ASRC提交其安全漏洞的上报者
6. 安全漏洞奖励的解释权利归ASRC所有
## 收集范围
我们的主要收集漏洞类别是:
* 服务器端请求伪造(SSRF)
* SQL注入
* 拒绝服务攻击
* 远程执行代码(RCE)
* XML外部实体攻击(XXE)
* 访问控制问题(不安全的直接对象参考问题等)
* 敏感目录遍历问题
* 本地文件读取(LFD)
* 敏感信息泄露(密钥,Cookie,Session等)
## 奖励
* 可直接导致严重问题的每个漏洞奖励7000元人民币
* 存在限制及需要一定特殊环境下才能利用的问题将给予700-5600元人民币不等的奖励,比如需要用户主动点击才会触发的问题或需要admin权限
* 只有在指定环境下才可以运行的利用将有可能被收纳但不给予奖励,或直接被忽略,比如只在fastjson+linux特定版本才会出现的问题
## 不在收集范围的报告
* 影响过时浏览器或平台用户的漏洞
* Self-XSS
* 会话固定
* 内容欺骗
* 缺少cookie标记
* 混合内容警告
* SSL / TLS问题
* Clickjacking
* 基于Flash的漏洞
* 反射文件下载攻击(RFD)
* 物理或社会工程攻击
* 未验证自动化工具或扫描仪的结果
* 登录/注销/未认证/低影响CSRF
* 需要MITM或物理访问用户设备的攻击
* 与网络协议或行业标准相关的问题
* 不能用于直接攻击的错误信息泄露
* 缺少与安全相关的HTTP标头等
# Vulnerability Reward Program
## Reporting
If you believe you have found any security (technical) vulnerability in the Program, you are welcomed to submit a vulnerability report to us at https://security.alibaba.com
In case of reporting any security vulnerability, please be noted that you may including following information (Qualified Reporting):
* The git program URL and running version
* A detailed description with necessary screenshots
* Steps to reappearance the vulnerability and your advice to fix it
* Other useful information
## Processing
ASRC (Alibaba Security Response Center) will review and respond as quickly as possible to your submission, and keep you informed as we work to fix the vulnerability you submitted. We may contact you for further information if necessary.
## Terms and Conditions
1. ONLY technical vulnerabilities will be accepted and rated.
2. With regarding to security reasons, reporters agree to cooperate with ASRC exclusively on the vulnerability he/she submitted and not disclose any information of vulnerability to any third-parties.
3. In the case that more than one person report the same security vulnerability, the reward will be given to the first person who accomplish a Qualified Reporting.
4. To protect users of the program, please do not directly submit issue on github or discuss anything with the community
5. All Rewards and Reputation Credits are given to the reporters who submit his/her security vulnerabilities ONLY to ASRC.
6. All rights for the security vulnerability rewards are reserved by ASRC.
## Scope of Collecting
The main categories of vulnerabilities that we are sincerely looking for are:
* Server-Side Request Forgery (SSRF)
* SQL Injection
* Denial of Service Attack
* Remote Code Execution (RCE)
* XML External Entity Attacks (XXE)
* Access Control Issues (Insecure Direct Object Reference issues, etc.)
* Directory Traversal Issues
* Local File Disclosure (LFD)
* Sensitive Information Leakage (Key, Cookie, Session etc.)
## Reward
* $1,000 for one valid report
* $100-$800 for Vuls which is limited. For example, Vuls that need user interactions or administrator authority
* Vuls which only work on the special version will be accepted but no reward, or directly rejected. For example, Vul runs only on a special linux version
## Ineligible Reports
* Vulnerabilities affecting users of outdated browsers or platforms
* "Self" XSS
* Session fixation
* Content Spoofing
* Missing cookie flags
* Mixed content warnings
* SSL/TLS best practices
* Clickjacking/UI redressing
* Flash-based vulnerabilities
* Reflected file download attacks (RFD)
* Physical or social engineering attacks
* Unverified Results of automated tools or scanners
* Login/logout/unauthenticated/low-impact CSRF
* Attacks requiring MITM or physical access to a user's device
* Issues related to networking protocols or industry standards
* Error information disclosure that cannot be used to make a direct attack
* Missing security-related HTTP headers which do not lead directly to a vulnerability

BIN
admin/admin-ui/node/node Executable file

Binary file not shown.

View File

@@ -1,14 +1,14 @@
<?xml version="1.0" encoding="UTF-8"?>
<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
<parent>
<artifactId>canal-admin</artifactId>
<artifactId>canal.admin</artifactId>
<groupId>com.alibaba.otter</groupId>
<version>1.1.5-SNAPSHOT</version>
<version>1.1.9-SNAPSHOT</version>
</parent>
<modelVersion>4.0.0</modelVersion>
<artifactId>canal-admin-ui</artifactId>
<artifactId>canal.admin-ui</artifactId>
<name>canal admin-ui module for otter ${project.version}</name>
<properties>
<project.build.sourceEncoding>UTF-8</project.build.sourceEncoding>
<project.reporting.outputEncoding>UTF-8</project.reporting.outputEncoding>

View File

Before

Width:  |  Height:  |  Size: 6.2 KiB

After

Width:  |  Height:  |  Size: 6.2 KiB

View File

Before

Width:  |  Height:  |  Size: 66 KiB

After

Width:  |  Height:  |  Size: 66 KiB

View File

Before

Width:  |  Height:  |  Size: 21 KiB

After

Width:  |  Height:  |  Size: 21 KiB

View File

Before

Width:  |  Height:  |  Size: 96 KiB

After

Width:  |  Height:  |  Size: 96 KiB

View File

Before

Width:  |  Height:  |  Size: 4.7 KiB

After

Width:  |  Height:  |  Size: 4.7 KiB

View File

Before

Width:  |  Height:  |  Size: 2.3 KiB

After

Width:  |  Height:  |  Size: 2.3 KiB

View File

Before

Width:  |  Height:  |  Size: 497 B

After

Width:  |  Height:  |  Size: 497 B

View File

Before

Width:  |  Height:  |  Size: 1.3 KiB

After

Width:  |  Height:  |  Size: 1.3 KiB

View File

Before

Width:  |  Height:  |  Size: 944 B

After

Width:  |  Height:  |  Size: 944 B

View File

Before

Width:  |  Height:  |  Size: 2.4 KiB

After

Width:  |  Height:  |  Size: 2.4 KiB

View File

Before

Width:  |  Height:  |  Size: 285 B

After

Width:  |  Height:  |  Size: 285 B

View File

Before

Width:  |  Height:  |  Size: 821 B

After

Width:  |  Height:  |  Size: 821 B

View File

Before

Width:  |  Height:  |  Size: 623 B

After

Width:  |  Height:  |  Size: 623 B

View File

Before

Width:  |  Height:  |  Size: 597 B

After

Width:  |  Height:  |  Size: 597 B

View File

Before

Width:  |  Height:  |  Size: 1.8 KiB

After

Width:  |  Height:  |  Size: 1.8 KiB

View File

Before

Width:  |  Height:  |  Size: 440 B

After

Width:  |  Height:  |  Size: 440 B

View File

@@ -59,7 +59,7 @@ export default {
canalInstanceDetail(this.$route.query.id).then(response => {
const data = response.data
this.form.id = data.id
this.form.name = data.name + '/instance.propertios'
this.form.name = data.name + '/instance.properties'
this.form.content = data.content
this.form.clusterServerId = data.clusterServerId
})

View File

@@ -66,8 +66,8 @@ export default {
}
}
const validatePassword = (rule, value, callback) => {
if (value.length < 6) {
callback(new Error('The password can not be less than 6 digits'))
if (value.length < 5) {
callback(new Error('The password can not be less than 5 digits'))
} else {
callback()
}

Some files were not shown because too many files have changed in this diff Show More