fix(server/admin): 修复 Canal Admin 认证绕过漏洞 (#2076021)

在 ClientAuthenticationHandler.messageReceived() 的认证失败分支补充
ctx.getChannel().close() 与 return，防止 fall-through 到成功 ACK 分支
导致认证 handler 被移除、后续命令被当作已认证执行。

新增 ClientAuthenticationHandlerTest 覆盖错误密码、seed 为空、
认证成功三条路径。

.github/workflows/maven.yml

@@ -0,0 +1,46 @@
+# This workflow will build a Java project with Maven, and cache/restore any dependencies to improve the workflow execution time
+# For more information see: https://docs.github.com/en/actions/automating-builds-and-tests/building-and-testing-java-with-maven
+
+# This workflow uses actions that are not certified by GitHub.
+# They are provided by a third-party and are governed by
+# separate terms of service, privacy policy, and support
+# documentation.
+
+name: Java CI with Maven
+
+on:
+  push:
+    branches: [ "master" ]
+  pull_request:
+    branches: [ "master" ]
+
+permissions:
+  contents: read
+
+jobs:
+  test:
+    runs-on: ${{ matrix.os }}
+    permissions:
+      contents: read
+    strategy:
+      matrix:
+        os: [ ubuntu-latest ]
+        java: [ 8, 11, 17, 21 ]
+      fail-fast: false
+      max-parallel: 16
+    name: Test JDK ${{ matrix.java }}, ${{ matrix.os }}
+
+    steps:
+      - uses: actions/checkout@11bd71901bbe5b1630ceea73d27597364c9af683 # v4.2.2
+      - name: Set up JDK
+        uses: actions/setup-java@c5195efecf7bdfc987ee8bae7a71cb8b11521c00 # v4.7.1
+        with:
+          distribution: 'temurin'
+          java-version: ${{ matrix.java }}
+          cache: 'maven'
+      - name: Build with Maven if test jdk8
+        if: ${{ matrix.java == '8' || matrix.java == '11'}}
+        run: ./mvnw -Pgen-javadoc clean package -B
+      - name: Build with Maven if test jdk17
+        if: ${{ matrix.java == '17' || matrix.java == '21' }}
+        run: ./mvnw -Penable-for-jdk17+,gen-code-cov clean package -B

.gitignore

@@ -7,6 +7,7 @@ test-output/
 .settings/
 tmp
 temp
+logs/
 *.log
 antx.properties
 otter.properties
@@ -18,6 +19,8 @@ jtester.properties
 *.rpm
 client-adapter/example/
 *.dat
+.qwen/
+.vscode/
 conf/
 canal-admin/canal-admin-ui/dist/
 canal-admin/canal-admin-ui/node/

README.md

@@ -57,7 +57,7 @@
 - [Introduction](https://github.com/alibaba/canal/wiki/Introduction)
 - [QuickStart](https://github.com/alibaba/canal/wiki/QuickStart)
   - [Docker QuickStart](https://github.com/alibaba/canal/wiki/Docker-QuickStart)
-  - [Canal Kafka/RocketMQ QuickStart](https://github.com/alibaba/canal/wiki/Canal-Kafka-RocketMQ-QuickStart")
+  - [Canal Kafka/RocketMQ QuickStart](https://github.com/alibaba/canal/wiki/Canal-Kafka-RocketMQ-QuickStart)
   - [Aliyun RDS for MySQL QuickStart](https://github.com/alibaba/canal/wiki/aliyun-RDS-QuickStart)
   - [Prometheus QuickStart](https://github.com/alibaba/canal/wiki/Prometheus-QuickStart)
 - Canal Admin

SECURITY.md

@@ -0,0 +1,118 @@
+# 漏洞奖励计划 
+## 报告
+如果您认为自己在本程序中发现了任何安全（技术）漏洞，欢迎您通过 https://security.alibaba.com 向我们提交漏洞报告。
+如果您报告任何安全漏洞，请注意您可能包含以下信息（合格报告）：
+* git程序URL地址，运行的环境
+* 包含必要屏幕截图的详细说明
+* 重现漏洞的步骤以及修复漏洞的建议。
+* 其他有用信息
+
+## 处理
+ASRC（Alibaba Security Response Center阿里安全响应中心）将尽快审核并回复您的提交内容，并在我们努力修复您提交的漏洞时随时通知您。如有必要，我们可能会与您联系以获取更多信息。
+
+
+## 条款和条件
+1. 仅接受技术漏洞并对其进行评级
+2. 出于安全原因，上报者同意与ASRC合作完成他/她提交的漏洞，不向任何第三方透露任何漏洞信息
+3. 如果不止一个人报告相同的安全漏洞，奖励将给予完成合格报告的第一个人
+4. 为了保护程序的用户，请在修复之前不要直接提交git的issue，也不要在社区讨论任何漏洞信息
+5. 所有奖励和声誉积分将提供给仅向ASRC提交其安全漏洞的上报者
+6. 安全漏洞奖励的解释权利归ASRC所有
+
+## 收集范围
+我们的主要收集漏洞类别是：
+* 服务器端请求伪造（SSRF）
+* SQL注入
+* 拒绝服务攻击
+* 远程执行代码（RCE）
+* XML外部实体攻击（XXE）
+* 访问控制问题（不安全的直接对象参考问题等）
+* 敏感目录遍历问题
+* 本地文件读取（LFD）
+* 敏感信息泄露（密钥，Cookie，Session等）
+
+## 奖励
+* 可直接导致严重问题的每个漏洞奖励7000元人民币
+* 存在限制及需要一定特殊环境下才能利用的问题将给予700-5600元人民币不等的奖励，比如需要用户主动点击才会触发的问题或需要admin权限
+* 只有在指定环境下才可以运行的利用将有可能被收纳但不给予奖励，或直接被忽略，比如只在fastjson+linux特定版本才会出现的问题
+
+## 不在收集范围的报告
+* 影响过时浏览器或平台用户的漏洞
+* Self-XSS
+* 会话固定
+* 内容欺骗
+* 缺少cookie标记
+* 混合内容警告
+* SSL / TLS问题
+* Clickjacking 
+* 基于Flash的漏洞
+* 反射文件下载攻击（RFD）
+* 物理或社会工程攻击
+* 未验证自动化工具或扫描仪的结果
+* 登录/注销/未认证/低影响CSRF
+* 需要MITM或物理访问用户设备的攻击
+* 与网络协议或行业标准相关的问题
+* 不能用于直接攻击的错误信息泄露
+* 缺少与安全相关的HTTP标头等
+
+
+
+
+
+# Vulnerability Reward Program
+## Reporting
+If you believe you have found any security (technical) vulnerability in the Program, you are welcomed to submit a vulnerability report to us at https://security.alibaba.com 
+In case of reporting any security vulnerability, please be noted that you may including following information (Qualified Reporting):
+* The git program URL and running version 
+* A detailed description with necessary screenshots
+* Steps to reappearance the vulnerability and your advice to fix it
+* Other useful information
+
+
+## Processing
+ASRC (Alibaba Security Response Center) will review and respond as quickly as possible to your submission, and keep you informed as we work to fix the vulnerability you submitted. We may contact you for further information if necessary.
+
+
+## Terms and Conditions
+1. ONLY technical vulnerabilities will be accepted and rated.
+2. With regarding to security reasons, reporters agree to cooperate with ASRC exclusively on the vulnerability he/she submitted and not disclose any information of vulnerability to any third-parties.
+3. In the case that more than one person report the same security vulnerability, the reward will be given to the first person who accomplish a Qualified Reporting.
+4. To protect users of the program, please do not directly submit issue on github or discuss anything with the community 
+5. All Rewards and Reputation Credits are given to the reporters who submit his/her security vulnerabilities ONLY to ASRC.
+6. All rights for the security vulnerability rewards are reserved by ASRC.
+
+## Scope of Collecting
+The main categories of vulnerabilities that we are sincerely looking for are:
+* Server-Side Request Forgery (SSRF)
+* SQL Injection
+* Denial of Service Attack
+* Remote Code Execution (RCE)
+* XML External Entity Attacks (XXE)
+* Access Control Issues (Insecure Direct Object Reference issues, etc.)
+* Directory Traversal Issues
+* Local File Disclosure (LFD)
+* Sensitive Information Leakage (Key, Cookie, Session etc.)
+
+## Reward
+* $1,000 for one valid report
+* $100-$800 for Vuls which is limited. For example, Vuls that need user interactions or administrator authority
+* Vuls which only work on the special version will be accepted but no reward, or directly rejected. For example, Vul runs only on a special linux version
+
+## Ineligible Reports
+* Vulnerabilities affecting users of outdated browsers or platforms
+* "Self" XSS
+* Session fixation
+* Content Spoofing
+* Missing cookie flags
+* Mixed content warnings
+* SSL/TLS best practices
+* Clickjacking/UI redressing
+* Flash-based vulnerabilities
+* Reflected file download attacks (RFD)
+* Physical or social engineering attacks
+* Unverified Results of automated tools or scanners
+* Login/logout/unauthenticated/low-impact CSRF
+* Attacks requiring MITM or physical access to a user's device
+* Issues related to networking protocols or industry standards
+* Error information disclosure that cannot be used to make a direct attack
+* Missing security-related HTTP headers which do not lead directly to a vulnerability

admin/admin-ui/pom.xml

@@ -3,7 +3,7 @@
   <parent>
     <artifactId>canal.admin</artifactId>
     <groupId>com.alibaba.otter</groupId>
-    <version>1.1.8</version>
+    <version>1.1.9-SNAPSHOT</version>
   </parent>
   <modelVersion>4.0.0</modelVersion>
 

admin/admin-web/pom.xml

@@ -3,7 +3,7 @@
     <parent>
         <artifactId>canal.admin</artifactId>
         <groupId>com.alibaba.otter</groupId>
-        <version>1.1.8</version>
+        <version>1.1.9-SNAPSHOT</version>
     </parent>
     <modelVersion>4.0.0</modelVersion>
     <artifactId>canal.admin-web</artifactId>

admin/admin-web/src/main/bin/startup.bat

@@ -11,7 +11,7 @@ set CLASSPATH=%conf_dir%
 set CLASSPATH=%conf_dir%\..\lib\*;%CLASSPATH%
 
 set JAVA_MEM_OPTS= -Xms128m -Xmx512m
-set JAVA_OPTS_EXT= -Djava.awt.headless=true -Djava.net.preferIPv4Stack=true -Dapplication.codeset=UTF-8 -Dfile.encoding=UTF-8
+set JAVA_OPTS_EXT= -Djava.awt.headless=true -Djava.net.preferIPv4Stack=false -Dapplication.codeset=UTF-8 -Dfile.encoding=UTF-8
 set ADAPTER_OPTS= -DappName=canal-admin
 
 set JAVA_OPTS= %JAVA_MEM_OPTS% %JAVA_OPTS_EXT% %ADAPTER_OPTS%
@@ -19,4 +19,4 @@ set JAVA_OPTS= %JAVA_MEM_OPTS% %JAVA_OPTS_EXT% %ADAPTER_OPTS%
 set CMD_STR= java %JAVA_OPTS% -classpath "%CLASSPATH%" com.alibaba.otter.canal.admin.CanalAdminApplication
 echo start cmd : %CMD_STR%
 
-java %JAVA_OPTS% -classpath "%CLASSPATH%" com.alibaba.otter.canal.admin.CanalAdminApplication
+java %JAVA_OPTS% -classpath "%CLASSPATH%" com.alibaba.otter.canal.admin.CanalAdminApplication

admin/admin-web/src/main/bin/startup.sh

@@ -79,7 +79,7 @@ else
 	JAVA_OPTS="-server -Xms1024m -Xmx1024m -XX:NewSize=256m -XX:MaxNewSize=256m -XX:MaxPermSize=128m $JAVA_OPTS"
 fi
 
-JAVA_OPTS=" $JAVA_OPTS -Djava.awt.headless=true -Djava.net.preferIPv4Stack=true -Dfile.encoding=UTF-8"
+JAVA_OPTS=" $JAVA_OPTS -Djava.awt.headless=true -Djava.net.preferIPv4Stack=false -Dfile.encoding=UTF-8"
 CANAL_OPTS="-DappName=canal-admin"
 
 for i in $base/lib/*;

admin/admin-web/src/main/java/com/alibaba/otter/canal/admin/config/WebConfig.java

@@ -98,6 +98,26 @@ public class WebConfig implements WebMvcConfigurer {
                     return false;
                 }
 
+                // 检查是否使用默认密码，如果是则仅允许改密和登出操作
+                User validUser = (User) httpServletRequest.getAttribute("user");
+                if (validUser != null && validUser.isDefaultPassword()) {
+                    String uri = httpServletRequest.getRequestURI();
+                    String method = httpServletRequest.getMethod();
+                    boolean isPasswordChange = "PUT".equals(method) && uri.matches("/api/[^/]+/user");
+                    boolean isLogout = uri.contains("/user/logout");
+                    if (!isPasswordChange && !isLogout) {
+                        BaseModel baseModel = BaseModel.getInstance(null);
+                        baseModel.setCode(50015);
+                        baseModel.setMessage("Default password must be changed before proceeding");
+                        ObjectMapper mapper = new ObjectMapper();
+                        String json = mapper.writeValueAsString(baseModel);
+                        httpServletResponse.setContentType("application/json;charset=UTF-8");
+                        PrintWriter out = httpServletResponse.getWriter();
+                        out.print(json);
+                        return false;
+                    }
+                }
+
                 return true;
             }
         })

admin/admin-web/src/main/java/com/alibaba/otter/canal/admin/controller/UserController.java

@@ -56,6 +56,9 @@ public class UserController {
             String token = UUID.randomUUID().toString();
             loginUsers.put(token, loginUser);
             tokenResp.put("token", token);
+            if (loginUser.isDefaultPassword()) {
+                tokenResp.put("needPasswordChange", "true");
+            }
             return BaseModel.getInstance(tokenResp);
         } else {
             BaseModel<Map<String, String>> model = BaseModel.getInstance(null);
@@ -98,6 +101,7 @@ public class UserController {
                                     HttpServletRequest httpServletRequest) {
         userService.update(user);
         String token = (String) httpServletRequest.getAttribute("token");
+        user.setDefaultPassword(false);
         loginUsers.put(token, user);
         return BaseModel.getInstance("success");
     }

admin/admin-web/src/main/java/com/alibaba/otter/canal/admin/model/User.java

@@ -44,6 +44,9 @@ public class User extends Model {
     @Transient
     private String oldPassword;
 
+    @Transient
+    private boolean defaultPassword;
+
     public Long getId() {
         return id;
     }
@@ -115,4 +118,12 @@ public class User extends Model {
     public void setOldPassword(String oldPassword) {
         this.oldPassword = oldPassword;
     }
+
+    public boolean isDefaultPassword() {
+        return defaultPassword;
+    }
+
+    public void setDefaultPassword(boolean defaultPassword) {
+        this.defaultPassword = defaultPassword;
+    }
 }

admin/admin-web/src/main/java/com/alibaba/otter/canal/admin/service/impl/UserServiceImpl.java

@@ -3,6 +3,8 @@ package com.alibaba.otter.canal.admin.service.impl;
 import java.security.NoSuchAlgorithmException;
 
 import org.apache.commons.lang.StringUtils;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
 import org.springframework.stereotype.Service;
 
 import com.alibaba.otter.canal.admin.common.exception.ServiceException;
@@ -19,7 +21,11 @@ import com.alibaba.otter.canal.protocol.SecurityUtil;
 @Service
 public class UserServiceImpl implements UserService {
 
-    private static byte[] seeds = "canal is best!".getBytes();
+    private static final Logger  logger                = LoggerFactory.getLogger(UserServiceImpl.class);
+
+    private static final String  DEFAULT_PASSWORD_HASH = "6BB4837EB74329105EE4568DDA7DC67ED2CA2AD9";
+
+    private static byte[]        seeds                 = "canal is best!".getBytes();
 
     private static final Integer PASSWORD_LENGTH = 5;
 
@@ -40,6 +46,11 @@ public class UserServiceImpl implements UserService {
             throw new ServiceException("user:" + user.getName() + " auth failed!");
         }
 
+        if (DEFAULT_PASSWORD_HASH.equalsIgnoreCase(user.getPassword())) {
+            user.setDefaultPassword(true);
+            logger.warn("Default admin password detected for user: {}. Please change immediately.", username);
+        }
+
         user.setPassword("");
         return user;
     }

admin/pom.xml

@@ -3,13 +3,13 @@
     <parent>
         <groupId>com.alibaba.otter</groupId>
         <artifactId>canal</artifactId>
-        <version>1.1.8</version>
+        <version>1.1.9-SNAPSHOT</version>
         <relativePath>../pom.xml</relativePath>
     </parent>
     <modelVersion>4.0.0</modelVersion>
     <groupId>com.alibaba.otter</groupId>
     <artifactId>canal.admin</artifactId>
-    <version>1.1.8</version>
+    <version>1.1.9-SNAPSHOT</version>
     <packaging>pom</packaging>
     <name>canal admin module for otter ${project.version}</name>
     <modules>

client-adapter/clickhouse/pom.xml

@@ -3,7 +3,7 @@
     <parent>
         <artifactId>canal.client-adapter</artifactId>
         <groupId>com.alibaba.otter</groupId>
-        <version>1.1.8</version>
+        <version>1.1.9-SNAPSHOT</version>
         <relativePath>../pom.xml</relativePath>
     </parent>
     <modelVersion>4.0.0</modelVersion>
@@ -58,7 +58,9 @@
                 </executions>
             </plugin>
             <plugin>
+                <groupId>org.apache.maven.plugins</groupId>
                 <artifactId>maven-antrun-plugin</artifactId>
+                <version>1.8</version>
                 <executions>
                     <execution>
                         <phase>package</phase>
@@ -79,4 +81,4 @@
             </plugin>
         </plugins>
     </build>
-</project>
+</project>

client-adapter/common/pom.xml

@@ -3,7 +3,7 @@
     <parent>
         <artifactId>canal.client-adapter</artifactId>
         <groupId>com.alibaba.otter</groupId>
-        <version>1.1.8</version>
+        <version>1.1.9-SNAPSHOT</version>
         <relativePath>../pom.xml</relativePath>
     </parent>
     <modelVersion>4.0.0</modelVersion>

client-adapter/common/src/main/java/com/alibaba/otter/canal/client/adapter/support/Util.java

@@ -16,6 +16,7 @@ import java.util.function.Function;
 
 import javax.sql.DataSource;
 
+import com.alibaba.druid.pool.DruidDataSource;
 import org.apache.commons.lang.StringUtils;
 import org.joda.time.DateTime;
 import org.joda.time.DateTimeZone;
@@ -38,7 +39,17 @@ public class Util {
     public static Object sqlRS(DataSource ds, String sql, Function<ResultSet, Object> fun) {
         try (Connection conn = ds.getConnection();
                 Statement stmt = conn.createStatement(ResultSet.TYPE_FORWARD_ONLY, ResultSet.CONCUR_READ_ONLY)) {
-            stmt.setFetchSize(Integer.MIN_VALUE);
+
+            DruidDataSource druidDataSource = (DruidDataSource) ds;
+            if ("postgresql".equals(druidDataSource.getDbType())) {
+                conn.setAutoCommit(false);
+                stmt.setFetchSize(1000);
+            } else if ("sqlserver".equals(druidDataSource.getDbType())) {
+                stmt.setFetchSize(100);
+            } else {
+                stmt.setFetchSize(Integer.MIN_VALUE);
+            }
+
             try (ResultSet rs = stmt.executeQuery(sql)) {
                 return fun.apply(rs);
             }
@@ -52,7 +63,13 @@ public class Util {
         try (Connection conn = ds.getConnection()) {
             try (PreparedStatement pstmt = conn
                 .prepareStatement(sql, ResultSet.TYPE_FORWARD_ONLY, ResultSet.CONCUR_READ_ONLY)) {
-                pstmt.setFetchSize(Integer.MIN_VALUE);
+                DruidDataSource druidDataSource = (DruidDataSource) ds;
+                if ("postgresql".equals(druidDataSource.getDbType())) {
+                    conn.setAutoCommit(false);
+                    pstmt.setFetchSize(1000);
+                } else {
+                    pstmt.setFetchSize(Integer.MIN_VALUE);
+                }
                 if (values != null) {
                     for (int i = 0; i < values.size(); i++) {
                         pstmt.setObject(i + 1, values.get(i));

client-adapter/es6x/pom.xml

@@ -3,7 +3,7 @@
     <parent>
         <artifactId>canal.client-adapter</artifactId>
         <groupId>com.alibaba.otter</groupId>
-        <version>1.1.8</version>
+        <version>1.1.9-SNAPSHOT</version>
         <relativePath>../pom.xml</relativePath>
     </parent>
     <modelVersion>4.0.0</modelVersion>
@@ -74,7 +74,9 @@
                 </executions>
             </plugin>
             <plugin>
+                <groupId>org.apache.maven.plugins</groupId>
                 <artifactId>maven-antrun-plugin</artifactId>
+                <version>1.8</version>
                 <executions>
                     <execution>
                         <phase>package</phase>

client-adapter/es7x/pom.xml

@@ -3,7 +3,7 @@
     <parent>
         <artifactId>canal.client-adapter</artifactId>
         <groupId>com.alibaba.otter</groupId>
-        <version>1.1.8</version>
+        <version>1.1.9-SNAPSHOT</version>
         <relativePath>../pom.xml</relativePath>
     </parent>
     <modelVersion>4.0.0</modelVersion>
@@ -74,7 +74,9 @@
                 </executions>
             </plugin>
             <plugin>
+                <groupId>org.apache.maven.plugins</groupId>
                 <artifactId>maven-antrun-plugin</artifactId>
+                <version>1.8</version>
                 <executions>
                     <execution>
                         <phase>package</phase>

client-adapter/es7x/src/main/java/com/alibaba/otter/canal/client/adapter/es7x/support/ES7xTemplate.java

@@ -39,13 +39,10 @@ public class ES7xTemplate implements ESTemplate {
         .getLogger(ESTemplate.class);
 
     private static final int                                  MAX_BATCH_SIZE = 1000;
-
-    private ESConnection                                      esConnection;
-
-    private ESBulkRequest                                     esBulkRequest;
-
     // es 字段类型本地缓存
     private static ConcurrentMap<String, Map<String, String>> esFieldTypes   = new ConcurrentHashMap<>();
+    private ESConnection                                      esConnection;
+    private ESBulkRequest                                     esBulkRequest;
 
     public ES7xTemplate(ESConnection esConnection){
         this.esConnection = esConnection;
@@ -65,8 +62,9 @@ public class ES7xTemplate implements ESTemplate {
         if (mapping.getId() != null) {
             String parentVal = (String) esFieldData.remove("$parent_routing");
             if (mapping.isUpsert()) {
-                ESUpdateRequest updateRequest = esConnection.new ES7xUpdateRequest(mapping.getIndex(),
-                    pkVal.toString()).setDoc(esFieldData).setDocAsUpsert(true);
+                ESUpdateRequest updateRequest = esConnection.new ES7xUpdateRequest(mapping.getIndex(), pkVal.toString())
+                    .setDoc(esFieldData)
+                    .setDocAsUpsert(true);
                 if (StringUtils.isNotEmpty(parentVal)) {
                     updateRequest.setRouting(parentVal);
                 }
@@ -283,7 +281,8 @@ public class ES7xTemplate implements ESTemplate {
     }
 
     @Override
-    public Object getESDataFromDmlData(ESMapping mapping, Map<String, Object> dmlData, Map<String, Object> esFieldData) {
+    public Object getESDataFromDmlData(ESMapping mapping, Map<String, Object> dmlData,
+                                       Map<String, Object> esFieldData) {
         SchemaItem schemaItem = mapping.getSchemaItem();
         String idFieldName = mapping.getId() == null ? mapping.getPk() : mapping.getId();
         Object resultIdVal = null;
@@ -307,8 +306,8 @@ public class ES7xTemplate implements ESTemplate {
     }
 
     @Override
-    public Object getESDataFromDmlData(ESMapping mapping,String owner, Map<String, Object> dmlData, Map<String, Object> dmlOld,
-                                       Map<String, Object> esFieldData) {
+    public Object getESDataFromDmlData(ESMapping mapping, String owner, Map<String, Object> dmlData,
+                                       Map<String, Object> dmlOld, Map<String, Object> esFieldData) {
         SchemaItem schemaItem = mapping.getSchemaItem();
         String idFieldName = mapping.getId() == null ? mapping.getPk() : mapping.getId();
         Object resultIdVal = null;
@@ -329,7 +328,7 @@ public class ES7xTemplate implements ESTemplate {
 
             if (dmlOld.containsKey(columnName) && !mapping.getSkips().contains(fieldItem.getFieldName())) {
                 esFieldData.put(Util.cleanColumn(fieldItem.getFieldName()),
-                        getValFromData(mapping, dmlData, fieldItem.getFieldName(), columnName));
+                    getValFromData(mapping, dmlData, fieldItem.getFieldName(), columnName));
             }
         }
 

client-adapter/es7x/src/main/java/com/alibaba/otter/canal/client/adapter/es7x/support/ESConnection.java

@@ -55,16 +55,9 @@ import com.alibaba.otter.canal.client.adapter.es.core.support.ESBulkRequest;
 public class ESConnection {
 
     private static final Logger logger = LoggerFactory.getLogger(ESConnection.class);
-
-    public enum ESClientMode {
-        TRANSPORT, REST
-    }
-
     private ESClientMode        mode;
-
     @SuppressWarnings("deprecation")
     private TransportClient     transportClient;
-
     private RestHighLevelClient restHighLevelClient;
 
     public ESConnection(String[] hosts, Map<String, String> properties, ESClientMode mode) throws UnknownHostException{
@@ -86,9 +79,10 @@ public class ESConnection {
             if (StringUtils.isNotEmpty(nameAndPwd) && nameAndPwd.contains(":")) {
                 String[] nameAndPwdArr = nameAndPwd.split(":");
                 final CredentialsProvider credentialsProvider = new BasicCredentialsProvider();
-                credentialsProvider.setCredentials(AuthScope.ANY, new UsernamePasswordCredentials(nameAndPwdArr[0],
-                    nameAndPwdArr[1]));
-                restClientBuilder.setHttpClientConfigCallback(httpClientBuilder -> httpClientBuilder.setDefaultCredentialsProvider(credentialsProvider));
+                credentialsProvider.setCredentials(AuthScope.ANY,
+                    new UsernamePasswordCredentials(nameAndPwdArr[0], nameAndPwdArr[1]));
+                restClientBuilder.setHttpClientConfigCallback(
+                    httpClientBuilder -> httpClientBuilder.setDefaultCredentialsProvider(credentialsProvider));
             }
             restHighLevelClient = new RestHighLevelClient(restClientBuilder);
         }
@@ -138,11 +132,94 @@ public class ESConnection {
                 logger.error(e.getMessage(), e);
                 return null;
             }
+
+            // 通过别名查询mapping返回的是真实索引名称，mappings.get(index)返回null，为兼容别名情况修改如下：
             mappingMetaData = mappings.get(index);
+            if (mappingMetaData == null && !mappings.isEmpty()) {
+                // 如果通过索引名找不到，可能是别名，取第一个映射
+                mappingMetaData = mappings.values().iterator().next();
+            }
+
+            // 如果还是null，说明索引不存在或没有mapping
+            if (mappingMetaData == null) {
+                throw new IllegalArgumentException("Not found the mapping info of index: " + index);
+            }
         }
         return mappingMetaData;
     }
 
+    // ------ get/set ------
+    public ESClientMode getMode() {
+        return mode;
+    }
+
+    public void setMode(ESClientMode mode) {
+        this.mode = mode;
+    }
+
+    public TransportClient getTransportClient() {
+        return transportClient;
+    }
+
+    public void setTransportClient(TransportClient transportClient) {
+        this.transportClient = transportClient;
+    }
+
+    public RestHighLevelClient getRestHighLevelClient() {
+        return restHighLevelClient;
+    }
+
+    public void setRestHighLevelClient(RestHighLevelClient restHighLevelClient) {
+        this.restHighLevelClient = restHighLevelClient;
+    }
+
+    private HttpHost createHttpHost(String uriStr) {
+        URI uri = URI.create(uriStr);
+        if (!org.springframework.util.StringUtils.hasLength(uri.getUserInfo())) {
+            return HttpHost.create(uri.toString());
+        }
+        try {
+            return HttpHost.create(new URI(uri
+                .getScheme(), null, uri.getHost(), uri.getPort(), uri.getPath(), uri.getQuery(), uri.getFragment())
+                    .toString());
+        } catch (URISyntaxException ex) {
+            throw new IllegalStateException(ex);
+        }
+    }
+
+    public enum ESClientMode {
+                              TRANSPORT, REST
+    }
+
+    public static class ES7xBulkResponse implements ESBulkRequest.ESBulkResponse {
+
+        private BulkResponse bulkResponse;
+
+        public ES7xBulkResponse(BulkResponse bulkResponse){
+            this.bulkResponse = bulkResponse;
+        }
+
+        @Override
+        public boolean hasFailures() {
+            return bulkResponse.hasFailures();
+        }
+
+        @Override
+        public void processFailBulkResponse(String errorMsg) {
+            for (BulkItemResponse itemResponse : bulkResponse.getItems()) {
+                if (!itemResponse.isFailed()) {
+                    continue;
+                }
+
+                if (itemResponse.getFailure().getStatus() == RestStatus.NOT_FOUND) {
+                    logger.error(itemResponse.getFailureMessage());
+                } else {
+                    throw new RuntimeException(errorMsg + itemResponse.getFailureMessage());
+                }
+            }
+        }
+    }
+
     public class ES7xIndexRequest implements ESBulkRequest.ESIndexRequest {
 
         private IndexRequestBuilder indexRequestBuilder;
@@ -443,71 +520,4 @@ public class ESConnection {
             this.bulkRequest = bulkRequest;
         }
     }
-
-    public static class ES7xBulkResponse implements ESBulkRequest.ESBulkResponse {
-
-        private BulkResponse bulkResponse;
-
-        public ES7xBulkResponse(BulkResponse bulkResponse){
-            this.bulkResponse = bulkResponse;
-        }
-
-        @Override
-        public boolean hasFailures() {
-            return bulkResponse.hasFailures();
-        }
-
-        @Override
-        public void processFailBulkResponse(String errorMsg) {
-            for (BulkItemResponse itemResponse : bulkResponse.getItems()) {
-                if (!itemResponse.isFailed()) {
-                    continue;
-                }
-
-                if (itemResponse.getFailure().getStatus() == RestStatus.NOT_FOUND) {
-                    logger.error(itemResponse.getFailureMessage());
-                } else {
-                    throw new RuntimeException(errorMsg + itemResponse.getFailureMessage());
-                }
-            }
-        }
-    }
-
-    // ------ get/set ------
-    public ESClientMode getMode() {
-        return mode;
-    }
-
-    public void setMode(ESClientMode mode) {
-        this.mode = mode;
-    }
-
-    public TransportClient getTransportClient() {
-        return transportClient;
-    }
-
-    public void setTransportClient(TransportClient transportClient) {
-        this.transportClient = transportClient;
-    }
-
-    public RestHighLevelClient getRestHighLevelClient() {
-        return restHighLevelClient;
-    }
-
-    public void setRestHighLevelClient(RestHighLevelClient restHighLevelClient) {
-        this.restHighLevelClient = restHighLevelClient;
-    }
-
-    private HttpHost createHttpHost(String uriStr) {
-        URI uri = URI.create(uriStr);
-        if (!org.springframework.util.StringUtils.hasLength(uri.getUserInfo())) {
-            return HttpHost.create(uri.toString());
-        }
-        try {
-            return HttpHost.create(new URI(uri.getScheme(), null, uri.getHost(), uri.getPort(), uri.getPath(),
-                                           uri.getQuery(), uri.getFragment()).toString());
-        } catch (URISyntaxException ex) {
-            throw new IllegalStateException(ex);
-        }
-    }
 }

client-adapter/es8x/pom.xml

@@ -3,7 +3,7 @@
     <parent>
         <artifactId>canal.client-adapter</artifactId>
         <groupId>com.alibaba.otter</groupId>
-        <version>1.1.8</version>
+        <version>1.1.9-SNAPSHOT</version>
         <relativePath>../pom.xml</relativePath>
     </parent>
     <modelVersion>4.0.0</modelVersion>
@@ -69,7 +69,9 @@
                 </executions>
             </plugin>
             <plugin>
+                <groupId>org.apache.maven.plugins</groupId>
                 <artifactId>maven-antrun-plugin</artifactId>
+                <version>1.8</version>
                 <executions>
                     <execution>
                         <phase>package</phase>

client-adapter/es8x/src/main/java/com/alibaba/otter/canal/client/adapter/es8x/ES8xAdapter.java

@@ -1,5 +1,14 @@
 package com.alibaba.otter.canal.client.adapter.es8x;
 
+import java.util.LinkedHashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.Properties;
+
+import javax.sql.DataSource;
+
+import org.elasticsearch.action.search.SearchResponse;
+
 import com.alibaba.otter.canal.client.adapter.es.core.ESAdapter;
 import com.alibaba.otter.canal.client.adapter.es.core.config.ESSyncConfig;
 import com.alibaba.otter.canal.client.adapter.es8x.etl.ESEtlService;
@@ -9,13 +18,6 @@ import com.alibaba.otter.canal.client.adapter.support.DatasourceConfig;
 import com.alibaba.otter.canal.client.adapter.support.EtlResult;
 import com.alibaba.otter.canal.client.adapter.support.OuterAdapterConfig;
 import com.alibaba.otter.canal.client.adapter.support.SPI;
-import org.elasticsearch.action.search.SearchResponse;
-
-import javax.sql.DataSource;
-import java.util.LinkedHashMap;
-import java.util.List;
-import java.util.Map;
-import java.util.Properties;
 
 /**
  * ES 8.x 外部适配器

client-adapter/es8x/src/main/java/com/alibaba/otter/canal/client/adapter/es8x/support/ES8xTemplate.java

@@ -39,13 +39,10 @@ public class ES8xTemplate implements ESTemplate {
         .getLogger(ESTemplate.class);
 
     private static final int                                  MAX_BATCH_SIZE = 1000;
-
-    private ESConnection                                      esConnection;
-
-    private ESBulkRequest                                     esBulkRequest;
-
     // es 字段类型本地缓存
     private static ConcurrentMap<String, Map<String, String>> esFieldTypes   = new ConcurrentHashMap<>();
+    private ESConnection                                      esConnection;
+    private ESBulkRequest                                     esBulkRequest;
 
     public ES8xTemplate(ESConnection esConnection){
         this.esConnection = esConnection;

client-adapter/es8x/src/main/java/com/alibaba/otter/canal/client/adapter/es8x/support/ESConnection.java

@@ -142,11 +142,73 @@ public class ESConnection {
             logger.error(e.getMessage(), e);
             return null;
         }
+
+        // 通过别名查询mapping返回的是真实索引名称，mappings.get(index)返回null，为兼容别名情况修改如下：
         mappingMetaData = mappings.get(index);
+        if (mappingMetaData == null && !mappings.isEmpty()) {
+            // 如果通过索引名找不到，可能是别名，取第一个映射
+            mappingMetaData = mappings.values().iterator().next();
+        }
+
+        // 如果还是null，说明索引不存在或没有mapping
+        if (mappingMetaData == null) {
+            throw new IllegalArgumentException("Not found the mapping info of index: " + index);
+        }
 
         return mappingMetaData;
     }
 
+    public RestHighLevelClient getRestHighLevelClient() {
+        return restHighLevelClient;
+    }
+
+    public void setRestHighLevelClient(RestHighLevelClient restHighLevelClient) {
+        this.restHighLevelClient = restHighLevelClient;
+    }
+
+    private HttpHost createHttpHost(String uriStr) {
+        URI uri = URI.create(uriStr);
+        if (!org.springframework.util.StringUtils.hasLength(uri.getUserInfo())) {
+            return HttpHost.create(uri.toString());
+        }
+        try {
+            return HttpHost.create(new URI(uri
+                .getScheme(), null, uri.getHost(), uri.getPort(), uri.getPath(), uri.getQuery(), uri.getFragment())
+                    .toString());
+        } catch (URISyntaxException ex) {
+            throw new IllegalStateException(ex);
+        }
+    }
+
+    public static class ES8xBulkResponse implements ESBulkRequest.ESBulkResponse {
+
+        private BulkResponse bulkResponse;
+
+        public ES8xBulkResponse(BulkResponse bulkResponse){
+            this.bulkResponse = bulkResponse;
+        }
+
+        @Override
+        public boolean hasFailures() {
+            return bulkResponse.hasFailures();
+        }
+
+        @Override
+        public void processFailBulkResponse(String errorMsg) {
+            for (BulkItemResponse itemResponse : bulkResponse.getItems()) {
+                if (!itemResponse.isFailed()) {
+                    continue;
+                }
+
+                if (itemResponse.getFailure().getStatus() == RestStatus.NOT_FOUND) {
+                    logger.error(itemResponse.getFailureMessage());
+                } else {
+                    throw new RuntimeException(errorMsg + itemResponse.getFailureMessage());
+                }
+            }
+        }
+    }
+
     public class ES8xIndexRequest implements ESBulkRequest.ESIndexRequest {
 
         private IndexRequestBuilder indexRequestBuilder;
@@ -395,55 +457,4 @@ public class ESConnection {
             this.bulkRequest = bulkRequest;
         }
     }
-
-    public static class ES8xBulkResponse implements ESBulkRequest.ESBulkResponse {
-
-        private BulkResponse bulkResponse;
-
-        public ES8xBulkResponse(BulkResponse bulkResponse){
-            this.bulkResponse = bulkResponse;
-        }
-
-        @Override
-        public boolean hasFailures() {
-            return bulkResponse.hasFailures();
-        }
-
-        @Override
-        public void processFailBulkResponse(String errorMsg) {
-            for (BulkItemResponse itemResponse : bulkResponse.getItems()) {
-                if (!itemResponse.isFailed()) {
-                    continue;
-                }
-
-                if (itemResponse.getFailure().getStatus() == RestStatus.NOT_FOUND) {
-                    logger.error(itemResponse.getFailureMessage());
-                } else {
-                    throw new RuntimeException(errorMsg + itemResponse.getFailureMessage());
-                }
-            }
-        }
-    }
-
-    public RestHighLevelClient getRestHighLevelClient() {
-        return restHighLevelClient;
-    }
-
-    public void setRestHighLevelClient(RestHighLevelClient restHighLevelClient) {
-        this.restHighLevelClient = restHighLevelClient;
-    }
-
-    private HttpHost createHttpHost(String uriStr) {
-        URI uri = URI.create(uriStr);
-        if (!org.springframework.util.StringUtils.hasLength(uri.getUserInfo())) {
-            return HttpHost.create(uri.toString());
-        }
-        try {
-            return HttpHost.create(new URI(uri
-                .getScheme(), null, uri.getHost(), uri.getPort(), uri.getPath(), uri.getQuery(), uri.getFragment())
-                    .toString());
-        } catch (URISyntaxException ex) {
-            throw new IllegalStateException(ex);
-        }
-    }
 }

client-adapter/escore/pom.xml

@@ -3,7 +3,7 @@
     <parent>
         <artifactId>canal.client-adapter</artifactId>
         <groupId>com.alibaba.otter</groupId>
-        <version>1.1.8</version>
+        <version>1.1.9-SNAPSHOT</version>
         <relativePath>../pom.xml</relativePath>
     </parent>
     <modelVersion>4.0.0</modelVersion>

client-adapter/escore/src/main/java/com/alibaba/otter/canal/client/adapter/es/core/support/ESSyncUtil.java

@@ -2,6 +2,7 @@ package com.alibaba.otter.canal.client.adapter.es.core.support;
 
 import java.io.IOException;
 import java.io.InputStream;
+import java.math.BigDecimal;
 import java.nio.charset.StandardCharsets;
 import java.sql.Blob;
 import java.sql.SQLException;
@@ -101,13 +102,22 @@ public class ESSyncUtil {
                 break;
             case "float":
             case "half_float":
-            case "scaled_float":
                 if (val instanceof Number) {
                     res = ((Number) val).floatValue();
                 } else {
                     res = Float.parseFloat(val.toString());
                 }
                 break;
+            case "scaled_float":
+                // scaled_float 需要保持高精度，使用 BigDecimal 进行处理
+                if (val instanceof BigDecimal) {
+                    res = val; // 保持 BigDecimal 精度
+                } else if (val instanceof Number) {
+                    res = new BigDecimal(val.toString());
+                } else {
+                    res = new BigDecimal(val.toString());
+                }
+                break;
             case "boolean":
                 if (val instanceof Boolean) {
                     res = val;

client-adapter/escore/src/test/java/com/alibaba/otter/canal/client/adapter/es/core/support/ESSyncUtilTest.java

@@ -0,0 +1,76 @@
+package com.alibaba.otter.canal.client.adapter.es.core.support;
+
+import java.math.BigDecimal;
+
+import org.junit.Assert;
+import org.junit.Test;
+
+/**
+ * ESSyncUtil 测试类
+ * 
+ * @author jianghang
+ */
+public class ESSyncUtilTest {
+
+    @Test
+    public void testScaledFloatPrecision() {
+        // 测试高精度 decimal 转换为 scaled_float
+        BigDecimal originalValue = new BigDecimal("1234567890.12345678901234567890");
+
+        // 转换为 scaled_float
+        Object result = ESSyncUtil.typeConvert(originalValue, "scaled_float");
+
+        // 验证结果仍然是 BigDecimal，保持精度
+        Assert.assertTrue("scaled_float 转换结果应该是 BigDecimal 类型", result instanceof BigDecimal);
+        Assert.assertEquals("scaled_float 转换应该保持原始精度", originalValue, result);
+
+        // 验证 toString 表示
+        Assert.assertEquals("scaled_float 精度应该保持完整", "1234567890.12345678901234567890", result.toString());
+    }
+
+    @Test
+    public void testScaledFloatFromString() {
+        // 测试字符串转换为 scaled_float
+        String originalString = "9876543210.98765432109876543210";
+
+        // 转换为 scaled_float
+        Object result = ESSyncUtil.typeConvert(originalString, "scaled_float");
+
+        // 验证结果是 BigDecimal
+        Assert.assertTrue("scaled_float 字符串转换结果应该是 BigDecimal 类型", result instanceof BigDecimal);
+
+        // 验证值正确
+        BigDecimal expected = new BigDecimal(originalString);
+        Assert.assertEquals("scaled_float 字符串转换应该正确解析", expected, result);
+    }
+
+    @Test
+    public void testScaledFloatFromDouble() {
+        // 测试 double 转换为 scaled_float（注意：double 本身有精度限制）
+        Double originalDouble = 1234567890.1234567890;
+
+        // 转换为 scaled_float
+        Object result = ESSyncUtil.typeConvert(originalDouble, "scaled_float");
+
+        // 验证结果是 BigDecimal
+        Assert.assertTrue("scaled_float double 转换结果应该是 BigDecimal 类型", result instanceof BigDecimal);
+
+        // 验证值正确（由于 double 精度限制，这里验证转换过程正确）
+        BigDecimal expected = new BigDecimal(originalDouble.toString());
+        Assert.assertEquals("scaled_float double 转换应该正确处理", expected, result);
+    }
+
+    @Test
+    public void testOtherFloatTypes() {
+        // 测试其他浮点类型仍然正常工作
+        BigDecimal value = new BigDecimal("123.45");
+
+        // 测试 float 类型
+        Object floatResult = ESSyncUtil.typeConvert(value, "float");
+        Assert.assertTrue("float 转换结果应该是 Float 类型", floatResult instanceof Float);
+
+        // 测试 half_float 类型
+        Object halfFloatResult = ESSyncUtil.typeConvert(value, "half_float");
+        Assert.assertTrue("half_float 转换结果应该是 Float 类型", halfFloatResult instanceof Float);
+    }
+}

client-adapter/hbase/pom.xml

@@ -3,7 +3,7 @@
     <parent>
         <artifactId>canal.client-adapter</artifactId>
         <groupId>com.alibaba.otter</groupId>
-        <version>1.1.8</version>
+        <version>1.1.9-SNAPSHOT</version>
         <relativePath>../pom.xml</relativePath>
     </parent>
     <modelVersion>4.0.0</modelVersion>
@@ -60,7 +60,9 @@
                 </executions>
             </plugin>
             <plugin>
+                <groupId>org.apache.maven.plugins</groupId>
                 <artifactId>maven-antrun-plugin</artifactId>
+                <version>1.8</version>
                 <executions>
                     <execution>
                         <phase>package</phase>

client-adapter/kudu/pom.xml

@@ -3,7 +3,7 @@
     <parent>
         <artifactId>canal.client-adapter</artifactId>
         <groupId>com.alibaba.otter</groupId>
-        <version>1.1.8</version>
+        <version>1.1.9-SNAPSHOT</version>
         <relativePath>../pom.xml</relativePath>
     </parent>
     <modelVersion>4.0.0</modelVersion>
@@ -57,7 +57,9 @@
                 </executions>
             </plugin>
             <plugin>
+                <groupId>org.apache.maven.plugins</groupId>
                 <artifactId>maven-antrun-plugin</artifactId>
+                <version>1.8</version>
                 <executions>
                     <execution>
                         <phase>package</phase>

client-adapter/launcher/pom.xml

@@ -3,7 +3,7 @@
     <parent>
         <artifactId>canal.client-adapter</artifactId>
         <groupId>com.alibaba.otter</groupId>
-        <version>1.1.8</version>
+        <version>1.1.9-SNAPSHOT</version>
         <relativePath>../pom.xml</relativePath>
     </parent>
     <modelVersion>4.0.0</modelVersion>
@@ -68,7 +68,6 @@
             <groupId>com.microsoft.sqlserver</groupId>
             <artifactId>mssql-jdbc</artifactId>
         </dependency>
-
         <dependency>
             <groupId>org.apache.hbase</groupId>
             <artifactId>hbase-shaded-client</artifactId>
@@ -79,6 +78,18 @@
             <artifactId>clickhouse-jdbc</artifactId>
         </dependency>
 
+        <!-- prometheus hotspot exporter -->
+        <dependency>
+            <groupId>io.prometheus</groupId>
+            <artifactId>simpleclient_hotspot</artifactId>
+            <version>0.4.0</version>
+        </dependency>
+        <dependency>
+            <groupId>io.prometheus</groupId>
+            <artifactId>simpleclient_httpserver</artifactId>
+            <version>0.4.0</version>
+        </dependency>
+
         <!-- outer adapter jar with dependencies-->
         <dependency>
             <groupId>com.alibaba.otter</groupId>

client-adapter/launcher/src/main/bin/startup.bat

@@ -11,7 +11,7 @@ set CLASSPATH=%conf_dir%
 set CLASSPATH=%conf_dir%\..\lib\*;%CLASSPATH%
 
 set JAVA_MEM_OPTS= -Xms128m -Xmx512m -XX:PermSize=128m
-set JAVA_OPTS_EXT= -Djava.awt.headless=true -Djava.net.preferIPv4Stack=true -Dapplication.codeset=UTF-8 -Dfile.encoding=UTF-8
+set JAVA_OPTS_EXT= -Djava.awt.headless=true -Djava.net.preferIPv4Stack=false -Dapplication.codeset=UTF-8 -Dfile.encoding=UTF-8
 set ADAPTER_OPTS= -DappName=canal-adapter
 
 set JAVA_OPTS= %JAVA_MEM_OPTS% %JAVA_OPTS_EXT% %ADAPTER_OPTS%
@@ -19,4 +19,4 @@ set JAVA_OPTS= %JAVA_MEM_OPTS% %JAVA_OPTS_EXT% %ADAPTER_OPTS%
 set CMD_STR= java %JAVA_OPTS% -classpath "%CLASSPATH%" com.alibaba.otter.canal.adapter.launcher.CanalAdapterApplication
 echo start cmd : %CMD_STR%
 
-java %JAVA_OPTS% -classpath "%CLASSPATH%" com.alibaba.otter.canal.adapter.launcher.CanalAdapterApplication
+java %JAVA_OPTS% -classpath "%CLASSPATH%" com.alibaba.otter.canal.adapter.launcher.CanalAdapterApplication

client-adapter/launcher/src/main/bin/startup.sh

@@ -9,7 +9,10 @@ case "`uname`" in
 		bin_abs_path=`cd $(dirname $0); pwd`
 		;;
 esac
+
 base=${bin_abs_path}/..
+jaas_conf=$base/conf/jaas.conf
+
 export LANG=en_US.UTF-8
 export BASE=$base
 
@@ -56,31 +59,44 @@ in
   exit;;
 esac
 
-JavaVersion=`$JAVA -version 2>&1 |awk 'NR==1{ gsub(/"/,""); print $3 }' | awk  -F '.' '{print $1}'`
-str=`file -L $JAVA | grep 64-bit`
+JAVA_VERSION=`$JAVA -version 2>&1 |awk 'NR==1{ gsub(/"/,""); print $3 }' | awk  -F '.' '{print $1}'`
 
-JAVA_OPTS="$JAVA_OPTS -Xss1m -XX:+AggressiveOpts -XX:-UseBiasedLocking -XX:-OmitStackTraceInFastThrow -XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=$base/logs"
-if [ $JavaVersion -ge 11 ] ; then
+JAVA_OPTS="$JAVA_OPTS -Xss1m -XX:+AggressiveOpts -XX:-UseBiasedLocking -XX:-OmitStackTraceInFastThrow"
+JAVA_OPTS="$JAVA_OPTS -XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=$base/logs"
+
+if [ $JAVA_VERSION -ge 11 ] ; then
   #JAVA_OPTS="$JAVA_OPTS -Xlog:gc*:$base_log/gc.log:time "
-  JAVA_OPTS="$JAVA_OPTS"
+  JAVA_OPTS="$JAVA_OPTS -Xlog:gc*:file=$base/logs/adapter/gc.log::filecount=5,filesize=32M"
 else
-  #JAVA_OPTS="$JAVA_OPTS -Xloggc:$base/logs/canal/gc.log -XX:+PrintGCDetails -XX:+PrintGCDateStamps -XX:+PrintGCApplicationStoppedTime"
-  JAVA_OPTS="$JAVA_OPTS -XX:+UseFastAccessorMethods -XX:+PrintAdaptiveSizePolicy -XX:+PrintTenuringDistribution"
+  JAVA_OPTS="$JAVA_OPTS -Xloggc:$base/logs/canal/gc.log -XX:+UseGCLogFileRotation -XX:NumberOfGCLogFiles=5 -XX:GCLogFileSize=32M"
+  JAVA_OPTS="$JAVA_OPTS -XX:+PrintGCDetails -XX:+PrintGCDateStamps -XX:-PrintHeapAtGC -XX:+PrintGCApplicationStoppedTime"
+  JAVA_OPTS="$JAVA_OPTS -XX:+UseFastAccessorMethods -XX:-PrintReferenceGC -XX:+PrintAdaptiveSizePolicy -XX:+PrintTenuringDistribution"
+  JAVA_OPTS="$JAVA_OPTS -XX:+PrintSafepointStatistics -XX:PrintSafepointStatisticsCount=1 -XX:+SafepointTimeout -XX:SafepointTimeoutDelay=2000"
+  JAVA_OPTS="$JAVA_OPTS -XX:+UseCountedLoopSafepoints -XX:+DisableExplicitGC -XX:+ExplicitGCInvokesConcurrent -XX:+ExplicitGCInvokesConcurrentAndUnloadsClasses"
 fi
 
-if [ -n "$str" ]; then
-  if [ $JavaVersion -ge 11 ] ; then
+OS_ARCH=`file -L $JAVA | grep 64-bit`
+if [ -n "$OS_ARCH" ]; then
+  if [ $JAVA_VERSION -ge 11 ] ; then
     # For G1
     JAVA_OPTS="-server -Xms2g -Xmx3g -XX:+UseG1GC -XX:MaxGCPauseMillis=250 -XX:+UseGCOverheadLimit -XX:+ExplicitGCInvokesConcurrent $JAVA_OPTS"
   else
 	  JAVA_OPTS="-server -Xms2g -Xmx3g -Xmn1g -XX:SurvivorRatio=2 -XX:PermSize=96m -XX:MaxPermSize=256m -XX:MaxTenuringThreshold=15 -XX:+DisableExplicitGC $JAVA_OPTS"
+	  ## JDK 1.8 2c4g [-XX:ConcGCThreads=1 -XX:ParallelGCThreads=2], 4c8g [-XX:ConcGCThreads=2 -XX:ParallelGCThreads=4]
+    #JAVA_OPTS="$JAVA_OPTS -XX:InitialRAMPercentage=45.0 -XX:MinRAMPercentage=45.0 -XX:MaxRAMPercentage=45.0"
+    #JAVA_OPTS="$JAVA_OPTS -XX:-UseParallelGC -XX:+UseConcMarkSweepGC -XX:ConcGCThreads=2 -XX:ParallelGCThreads=4"
+    #JAVA_OPTS="$JAVA_OPTS -XX:+UseCMSInitiatingOccupancyOnly -XX:CMSInitiatingOccupancyFraction=70"
+    #JAVA_OPTS="-server -XX:NewRatio=1 -XX:SurvivorRatio=2 -XX:MetaspaceSize=96m -XX:MaxMetaspaceSize=128m "
 	fi
 else
 	JAVA_OPTS="-server -Xms1024m -Xmx1024m -XX:NewSize=256m -XX:MaxNewSize=256m -XX:MaxPermSize=128m $JAVA_OPTS"
 fi
 
-JAVA_OPTS=" $JAVA_OPTS -Djava.awt.headless=true -Djava.net.preferIPv4Stack=true -Dfile.encoding=UTF-8"
+JAVA_OPTS=" $JAVA_OPTS -Djava.awt.headless=true -Djava.net.preferIPv4Stack=false -Dfile.encoding=UTF-8"
 ADAPTER_OPTS="-DappName=canal-adapter"
+if [ -f "$jaas_conf" ]; then
+  ADAPTER_OPTS="$ADAPTER_OPTS -Djava.security.auth.login.config=$jaas_conf"
+fi
 
 for i in $base/lib/*;
     do CLASSPATH=$i:"$CLASSPATH";

client-adapter/launcher/src/main/java/com/alibaba/otter/canal/adapter/launcher/CanalAdapterApplication.java

@@ -13,6 +13,7 @@ import org.springframework.boot.autoconfigure.jdbc.DataSourceAutoConfiguration;
  */
 @SpringBootApplication(exclude= { DataSourceAutoConfiguration.class})
 public class CanalAdapterApplication {
+
     public static void main(String[] args) {
         // 支持rocketmq client 配置日志路径
         System.setProperty("rocketmq.client.logUseSlf4j","true");

client-adapter/launcher/src/main/java/com/alibaba/otter/canal/adapter/launcher/loader/AdapterProcessor.java

@@ -8,6 +8,8 @@ import java.util.concurrent.Future;
 import java.util.concurrent.TimeUnit;
 import java.util.concurrent.TimeoutException;
 
+import com.alibaba.otter.canal.protocol.exception.CanalClientException;
+import org.apache.commons.lang.exception.ExceptionUtils;
 import org.slf4j.Logger;
 import org.slf4j.LoggerFactory;
 
@@ -186,7 +188,7 @@ public class AdapterProcessor {
                 logger.info("=============> Start to connect destination: {} <=============", this.canalDestination);
                 canalMsgConsumer.connect();
                 logger.info("=============> Subscribe destination: {} succeed <=============", this.canalDestination);
-                while (running) {
+                out: while (running) {
                     try {
                         syncSwitch.get(canalDestination, 1L, TimeUnit.MINUTES);
                     } catch (TimeoutException e) {
@@ -216,6 +218,17 @@ public class AdapterProcessor {
                             }
                             break;
                         } catch (Exception e) {
+                            Throwable th = e.getCause();
+                            // Handle source error when getting message
+                            if (th instanceof CanalClientException) {
+                                String message = ExceptionUtils.getRootCauseMessage(th);
+                                if (message.contains("end of stream when reading header")
+                                    || message.contains("Connection reset by peer") || message.contains("Broken pipe")) {
+                                    logger.error("Sync failed, reconnect to canal instance. Error: {}", message);
+                                    break out;
+                                }
+                            }
+                            // Handle sink error
                             if (i != retry - 1) {
                                 canalMsgConsumer.rollback(); // 处理失败, 回滚数据
                                 logger.error(e.getMessage() + " Error sync and rollback, execute times: " + (i + 1));

client-adapter/launcher/src/main/java/com/alibaba/otter/canal/adapter/launcher/monitor/ApplicationConfigMonitor.java

@@ -18,7 +18,9 @@ import org.slf4j.Logger;
 import org.slf4j.LoggerFactory;
 import org.springframework.cloud.context.refresh.ContextRefresher;
 import org.springframework.stereotype.Component;
+import org.yaml.snakeyaml.LoaderOptions;
 import org.yaml.snakeyaml.Yaml;
+import org.yaml.snakeyaml.constructor.SafeConstructor;
 
 import com.alibaba.otter.canal.adapter.launcher.loader.CanalAdapterService;
 import com.alibaba.otter.canal.client.adapter.support.Util;
@@ -70,7 +72,7 @@ public class ApplicationConfigMonitor {
             super.onFileChange(file);
             try {
                 // 检查yml格式
-                new Yaml().loadAs(new InputStreamReader(new FileInputStream(file), StandardCharsets.UTF_8), Map.class);
+                new Yaml(new SafeConstructor(new LoaderOptions())).loadAs(new InputStreamReader(new FileInputStream(file), StandardCharsets.UTF_8), Map.class);
 
                 canalAdapterService.destroy();
 

client-adapter/launcher/src/main/java/com/alibaba/otter/canal/adapter/launcher/prometheus/CanalAdapterExports.java

@@ -0,0 +1,33 @@
+package com.alibaba.otter.canal.adapter.launcher.prometheus;
+
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+/**
+ * @author sunxien
+ * @date 2025/5/19
+ * @since 1.0.0-SNAPSHOT
+ */
+public class CanalAdapterExports {
+
+    private static final Logger logger = LoggerFactory.getLogger(CanalAdapterExports.class);
+
+    private CanalAdapterExports() {
+    }
+
+    private static class SingletonHolder {
+        private static final CanalAdapterExports SINGLETON = new CanalAdapterExports();
+    }
+
+    public static CanalAdapterExports instance() {
+        return SingletonHolder.SINGLETON;
+    }
+
+    public void initialize() {
+        // TODO
+    }
+
+    public void terminate() {
+        // TODO
+    }
+}

client-adapter/launcher/src/main/java/com/alibaba/otter/canal/adapter/launcher/prometheus/PrometheusService.java

@@ -0,0 +1,100 @@
+package com.alibaba.otter.canal.adapter.launcher.prometheus;
+
+import java.io.IOException;
+
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+import org.springframework.beans.factory.DisposableBean;
+import org.springframework.beans.factory.InitializingBean;
+import org.springframework.stereotype.Component;
+
+import io.prometheus.client.exporter.HTTPServer;
+import io.prometheus.client.hotspot.DefaultExports;
+
+/**
+ * @author sunxien
+ * @date 2025/5/19
+ * @since 1.0.0-SNAPSHOT
+ */
+@Component
+public class PrometheusService implements InitializingBean, DisposableBean {
+
+    private static final Logger logger = LoggerFactory.getLogger(PrometheusService.class);
+
+    private final CanalAdapterExports adapterExports;
+    private volatile boolean running = false;
+
+    /**
+     * <pre>
+     * Canal Server admin port: 11110
+     * Canal Server tcp port: 11111
+     * Canal Server metric port: 11112
+     * Canal Adapter metric port: 11113
+     * </pre>
+     */
+    private int port = 11113;
+    private HTTPServer httpServer;
+
+    private PrometheusService() {
+        this.adapterExports = CanalAdapterExports.instance();
+    }
+
+    private static class SingletonHolder {
+        private static final PrometheusService SINGLETON = new PrometheusService();
+    }
+
+    public static PrometheusService getInstance() {
+        return SingletonHolder.SINGLETON;
+    }
+
+    @Override
+    public void afterPropertiesSet() throws Exception {
+        PrometheusService.getInstance().initialize();
+    }
+
+    @Override
+    public void destroy() throws Exception {
+        PrometheusService.getInstance().terminate();
+    }
+
+    public void initialize() {
+        try {
+            logger.info("Starting prometheus HTTPServer on port {}....", port);
+            // TODO 2.Https?
+            httpServer = new HTTPServer(port);
+            logger.info("Start prometheus HTTPServer on port {} success", port);
+        } catch (IOException e) {
+            logger.error("Unable to start prometheus HTTPServer on port {}.", port, e);
+            return;
+        }
+        try {
+            // JVM exports
+            DefaultExports.initialize();
+            // adapterExports.initialize();
+            this.running = true;
+        } catch (Throwable t) {
+            logger.error("Unable to initialize adapter exports. (Register the default Hotspot collectors)", t);
+        }
+    }
+
+    public void terminate() {
+        try {
+            this.running = false;
+            adapterExports.terminate();
+            if (httpServer != null) {
+                httpServer.stop();
+                logger.info("Stop prometheus HTTPServer on port {} success", port);
+            }
+        } catch (Throwable t) {
+            logger.error("Something happened while terminating prometheus HTTPServer.", t);
+        }
+    }
+
+    public boolean isRunning() {
+        return running;
+    }
+
+    public void setServerPort(int port) {
+        this.port = port;
+    }
+}

client-adapter/logger/pom.xml

@@ -3,7 +3,7 @@
     <parent>
         <artifactId>canal.client-adapter</artifactId>
         <groupId>com.alibaba.otter</groupId>
-        <version>1.1.8</version>
+        <version>1.1.9-SNAPSHOT</version>
         <relativePath>../pom.xml</relativePath>
     </parent>
     <modelVersion>4.0.0</modelVersion>

client-adapter/phoenix/pom.xml

@@ -3,7 +3,7 @@
     <parent>
         <artifactId>canal.client-adapter</artifactId>
         <groupId>com.alibaba.otter</groupId>
-        <version>1.1.8</version>
+        <version>1.1.9-SNAPSHOT</version>
         <relativePath>../pom.xml</relativePath>
     </parent>
     <modelVersion>4.0.0</modelVersion>
@@ -100,7 +100,9 @@
                 </executions>
             </plugin>
             <plugin>
+                <groupId>org.apache.maven.plugins</groupId>
                 <artifactId>maven-antrun-plugin</artifactId>
+                <version>1.8</version>
                 <executions>
                     <execution>
                         <phase>package</phase>

client-adapter/phoenix/src/main/java/com/alibaba/otter/canal/client/adapter/phoenix/service/PhoenixEtlService.java

@@ -172,11 +172,12 @@ public class PhoenixEtlService {
                     "SELECT * FROM " + dbMapping.getDatabase() + "." + dbMapping.getTable());
 
             // 拼接条件
-            appendCondition(params, dbMapping, srcDS, sql);
+            List<Object> values = new ArrayList<>();
+            appendCondition(params, dbMapping, srcDS, sql, values);
 
             // 获取总数
             String countSql = "SELECT COUNT(1) FROM ( " + sql + ") _CNT ";
-            long cnt = (Long) Util.sqlRS(srcDS, countSql, rs -> {
+            long cnt = (Long) Util.sqlRS(srcDS, countSql, values, rs -> {
                 Long count = null;
                 try {
                     if (rs.next()) {
@@ -206,14 +207,14 @@ public class PhoenixEtlService {
                         sqlFinal = sql + " LIMIT " + offset + "," + cnt;
                     }
                     executor
-                            .execute(() -> executeSqlImport(srcDS, targetDSConnection, sqlFinal, dbMapping, successCount, errMsg, debug));
+                            .execute(() -> executeSqlImport(srcDS, targetDSConnection, sqlFinal, values, dbMapping, successCount, errMsg, debug));
                 }
 
                 executor.shutdown();
                 //noinspection StatementWithEmptyBody
                 while (!executor.awaitTermination(3, TimeUnit.SECONDS)) ;
             } else {
-                executeSqlImport(srcDS, targetDSConnection, sql.toString(), dbMapping, successCount, errMsg, debug);
+                executeSqlImport(srcDS, targetDSConnection, sql.toString(), values, dbMapping, successCount, errMsg, debug);
             }
 
             logger.info(
@@ -235,7 +236,7 @@ public class PhoenixEtlService {
     }
 
     private static void appendCondition(List<String> params, DbMapping dbMapping, DataSource ds,
-                                        StringBuilder sql) {
+                                        StringBuilder sql, List<Object> values) {
         if (params != null && params.size() == 1 && dbMapping.getEtlCondition() == null) {
             AtomicBoolean stExists = new AtomicBoolean(false);
             // 验证是否有SYS_TIME字段
@@ -256,13 +257,27 @@ public class PhoenixEtlService {
                 return null;
             });
             if (stExists.get()) {
-                sql.append(" WHERE SYS_TIME >= '").append(params.get(0)).append("' ");
+                sql.append(" WHERE SYS_TIME >= ? ");
+                values.add(params.get(0));
             }
         } else if (dbMapping.getEtlCondition() != null && params != null && params.size() > 0) {
             String etlCondition = dbMapping.getEtlCondition();
             int size = params.size();
             for (int i = 0; i < size; i++) {
-                etlCondition = etlCondition.replace("{" + i + "}", params.get(i));
+                String placeholder = "{" + i + "}";
+                int idx = etlCondition.indexOf(placeholder);
+                if (idx >= 0) {
+                    int start = idx;
+                    int end = idx + placeholder.length();
+                    // 兼容配置中 '{i}' 这种被单引号包裹的写法，避免生成 '?' 语法错误
+                    if (start > 0 && etlCondition.charAt(start - 1) == '\''
+                        && end < etlCondition.length() && etlCondition.charAt(end) == '\'') {
+                        start--;
+                        end++;
+                    }
+                    etlCondition = etlCondition.substring(0, start) + "?" + etlCondition.substring(end);
+                    values.add(params.get(i));
+                }
             }
 
             sql.append(" ").append(etlCondition);
@@ -272,8 +287,8 @@ public class PhoenixEtlService {
     /**
      * 执行导入
      */
-    private static boolean executeSqlImport(DataSource srcDS, Connection targetDSConnection, String sql, DbMapping dbMapping,
-                                            AtomicLong successCount, List<String> errMsg, boolean debug) {
+    private static boolean executeSqlImport(DataSource srcDS, Connection targetDSConnection, String sql, List<Object> values,
+                                            DbMapping dbMapping, AtomicLong successCount, List<String> errMsg, boolean debug) {
         try {
             Map<String, String> columnsMap = new LinkedHashMap<>();
             Map<String, Integer> columnType = new LinkedHashMap<>();
@@ -300,7 +315,7 @@ public class PhoenixEtlService {
                     return false;
                 }
             });
-            Util.sqlRS(srcDS, sql, rs -> {
+            Util.sqlRS(srcDS, sql, values, rs -> {
                 int idx = 1;
 
                 try {
@@ -337,13 +352,13 @@ public class PhoenixEtlService {
                             pstmt.clearParameters();
 
                             // 删除数据
-                            Map<String, Object> values = new LinkedHashMap<>();
+                            Map<String, Object> deleteValues = new LinkedHashMap<>();
                             StringBuilder deleteSql = new StringBuilder(
                                     "DELETE FROM " + SyncUtil.getDbTableName(dbMapping) + " WHERE ");
-                            appendCondition(dbMapping, deleteSql, values, rs);
+                            appendCondition(dbMapping, deleteSql, deleteValues, rs);
                             try (PreparedStatement pstmt2 = connTarget.prepareStatement(deleteSql.toString())) {
                                 int k = 1;
-                                for (Object val : values.values()) {
+                                for (Object val : deleteValues.values()) {
                                     pstmt2.setObject(k++, val);
                                 }
                                 pstmt2.execute();

client-adapter/pom.xml

@@ -4,12 +4,12 @@
     <parent>
         <groupId>com.alibaba.otter</groupId>
         <artifactId>canal</artifactId>
-        <version>1.1.8</version>
+        <version>1.1.9-SNAPSHOT</version>
         <relativePath>../pom.xml</relativePath>
     </parent>
     <groupId>com.alibaba.otter</groupId>
     <artifactId>canal.client-adapter</artifactId>
-    <version>1.1.8</version>
+    <version>1.1.9-SNAPSHOT</version>
     <packaging>pom</packaging>
     <name>canal client adapter module for otter ${project.version}</name>
 

client-adapter/rdb/pom.xml

@@ -3,7 +3,7 @@
     <parent>
         <artifactId>canal.client-adapter</artifactId>
         <groupId>com.alibaba.otter</groupId>
-        <version>1.1.8</version>
+        <version>1.1.9-SNAPSHOT</version>
         <relativePath>../pom.xml</relativePath>
     </parent>
     <modelVersion>4.0.0</modelVersion>
@@ -68,7 +68,9 @@
                 </executions>
             </plugin>
             <plugin>
+                <groupId>org.apache.maven.plugins</groupId>
                 <artifactId>maven-antrun-plugin</artifactId>
+                <version>1.8</version>
                 <executions>
                     <execution>
                         <phase>package</phase>

client-adapter/rdb/src/main/java/com/alibaba/otter/canal/client/adapter/rdb/service/RdbSyncService.java

@@ -294,7 +294,7 @@ public class RdbSyncService {
             batchExecutor.execute(insertSql.toString(), values);
         } catch (SQLException e) {
             if (skipDupException
-                && (e.getMessage().contains("Duplicate entry") || e.getMessage().startsWith("ORA-00001:"))) {
+                && (e.getMessage().contains("Duplicate entry") || e.getMessage().contains("duplicate key") || e.getMessage().startsWith("ORA-00001:"))) {
                 // ignore
                 // TODO 增加更多关系数据库的主键冲突的错误码
             } else {

client-adapter/rdb/src/main/java/com/alibaba/otter/canal/client/adapter/rdb/support/SyncUtil.java

@@ -313,6 +313,9 @@ public class SyncUtil {
             case mariadb:
             case oceanbase:
                 return "`";
+        //  当dbType 为 postgresql时，返回双引号(避免表名为postgresql的关键字时，sql查询报错)
+            case postgresql:
+                return "\"";
             default:
                 return "";
         }

client-adapter/tablestore/pom.xml

@@ -3,7 +3,7 @@
     <parent>
         <artifactId>canal.client-adapter</artifactId>
         <groupId>com.alibaba.otter</groupId>
-        <version>1.1.8</version>
+        <version>1.1.9-SNAPSHOT</version>
         <relativePath>../pom.xml</relativePath>
     </parent>
     <modelVersion>4.0.0</modelVersion>
@@ -66,7 +66,9 @@
                 </executions>
             </plugin>
             <plugin>
+                <groupId>org.apache.maven.plugins</groupId>
                 <artifactId>maven-antrun-plugin</artifactId>
+                <version>1.8</version>
                 <executions>
                     <execution>
                         <phase>package</phase>

client/pom.xml

@@ -3,7 +3,7 @@
 	<parent>
 		<groupId>com.alibaba.otter</groupId>
 		<artifactId>canal</artifactId>
-		<version>1.1.8</version>
+		<version>1.1.9-SNAPSHOT</version>
 		<relativePath>../pom.xml</relativePath>
 	</parent>
 	<groupId>com.alibaba.otter</groupId>

common/pom.xml

@@ -3,7 +3,7 @@
 	<parent>
 		<groupId>com.alibaba.otter</groupId>
 		<artifactId>canal</artifactId>
-		<version>1.1.8</version>
+		<version>1.1.9-SNAPSHOT</version>
 		<relativePath>../pom.xml</relativePath>
 	</parent>
 	<artifactId>canal.common</artifactId>

common/src/main/java/com/alibaba/otter/canal/common/utils/FileUtils.java

@@ -1,5 +1,6 @@
 package com.alibaba.otter.canal.common.utils;
 
+import java.io.File;
 import java.io.IOException;
 import java.io.RandomAccessFile;
 import java.nio.charset.StandardCharsets;
@@ -88,8 +89,43 @@ public class FileUtils {
         return res.toString();
     }
 
-    public static void main(String[] args) {
-        String res = readFileFromOffset("test2.txt", 2, "UTF-8");
-        System.out.println(res);
+    /**
+     * 校验自定义的文件名，是否在允许的基目录范围内，如何合法就返回全路径，否则就直接报错
+     *
+     * @param baseDir
+     * @param destination
+     * @return
+     */
+    public static String validateFileName(String baseDir, String destination) {
+        try {
+            // 验证 destination 是否在允许的基目录范围内
+            String basePath = new File(baseDir).getCanonicalPath();
+            String fullPath = new File(basePath, destination).getCanonicalPath();
+
+            // 检查 fullPath 是否以 basePath 开头
+            if (!fullPath.startsWith(basePath + File.separator)) {
+                throw new IllegalArgumentException("Invalid destination path");
+            }
+
+            return fullPath;
+        } catch (IOException e) {
+            throw new RuntimeException("Failed to read file", e);
+        }
+    }
+
+    public static void main(String[] args) throws IOException {
+        String fullPath = validateFileName("/tmp/", "1.txt");
+        System.out.println(fullPath);
+        System.out.println(org.apache.commons.io.FileUtils.readLines(new File(fullPath)));
+
+        fullPath = validateFileName("/tmp/", "test");
+        fullPath = validateFileName(fullPath,"1.txt");
+        System.out.println(fullPath);
+        System.out.println(org.apache.commons.io.FileUtils.readLines(new File(fullPath)));
+
+
+        fullPath = validateFileName("/tmp/", "../etc/hosts");
+        System.out.println(fullPath);
+        System.out.println(org.apache.commons.io.FileUtils.readLines(new File(fullPath)));
     }
 }

connector/core/pom.xml

@@ -3,7 +3,7 @@
     <parent>
         <artifactId>canal.connector</artifactId>
         <groupId>com.alibaba.otter</groupId>
-        <version>1.1.8</version>
+        <version>1.1.9-SNAPSHOT</version>
     </parent>
     <modelVersion>4.0.0</modelVersion>
     <groupId>com.alibaba.otter</groupId>

connector/core/src/main/java/com/alibaba/otter/canal/connector/core/filter/AviaterRegexFilter.java

@@ -1,7 +1,6 @@
 package com.alibaba.otter.canal.connector.core.filter;
 
 import java.util.ArrayList;
-import java.util.Arrays;
 import java.util.Comparator;
 import java.util.HashMap;
 import java.util.List;
@@ -19,7 +18,6 @@ import com.googlecode.aviator.Expression;
  */
 public class AviaterRegexFilter {
 
-    private static final String             SPLIT             = ",";
     private static final String             PATTERN_SPLIT     = "|";
     private static final String             FILTER_EXPRESSION = "regex(pattern,target)";
     private static final RegexFunction      regexFunction     = new RegexFunction();
@@ -43,8 +41,9 @@ public class AviaterRegexFilter {
         if (StringUtils.isEmpty(pattern)) {
             list = new ArrayList<>();
         } else {
-            String[] ss = StringUtils.split(pattern, SPLIT);
-            list = Arrays.asList(ss);
+            // fix issue #5442: only split on commas outside regex grouping constructs,
+            // so quantifiers like \d{1,2} are kept intact.
+            list = splitPattern(pattern);
         }
 
         // 对pattern按照从长到短的排序
@@ -56,6 +55,60 @@ public class AviaterRegexFilter {
         this.pattern = StringUtils.join(list, PATTERN_SPLIT);
     }
 
+    /**
+     * 按顶层逗号拆分多个正则表达式，跳过位于 {}, [], () 内的逗号，避免破坏类似 \d{1,2} 的量词写法。
+     */
+    private static List<String> splitPattern(String pattern) {
+        List<String> result = new ArrayList<>();
+        int braceDepth = 0;
+        int bracketDepth = 0;
+        int parenDepth = 0;
+        int start = 0;
+        int len = pattern.length();
+        for (int i = 0; i < len; i++) {
+            char c = pattern.charAt(i);
+            // 反斜杠转义：跳过下一个字符，避免把 \{ \[ \( 等误判为分组开始
+            if (c == '\\' && i + 1 < len) {
+                i++;
+                continue;
+            }
+            switch (c) {
+                case '{':
+                    braceDepth++;
+                    break;
+                case '}':
+                    if (braceDepth > 0) braceDepth--;
+                    break;
+                case '[':
+                    bracketDepth++;
+                    break;
+                case ']':
+                    if (bracketDepth > 0) bracketDepth--;
+                    break;
+                case '(':
+                    parenDepth++;
+                    break;
+                case ')':
+                    if (parenDepth > 0) parenDepth--;
+                    break;
+                case ',':
+                    if (braceDepth == 0 && bracketDepth == 0 && parenDepth == 0) {
+                        if (i > start) {
+                            result.add(pattern.substring(start, i));
+                        }
+                        start = i + 1;
+                    }
+                    break;
+                default:
+                    break;
+            }
+        }
+        if (start < len) {
+            result.add(pattern.substring(start));
+        }
+        return result;
+    }
+
     public boolean filter(String filtered) {
         if (StringUtils.isEmpty(pattern)) {
             return defaultEmptyValue;

connector/kafka-connector/pom.xml

@@ -3,7 +3,7 @@
     <parent>
         <artifactId>canal.connector</artifactId>
         <groupId>com.alibaba.otter</groupId>
-        <version>1.1.8</version>
+        <version>1.1.9-SNAPSHOT</version>
         <relativePath>../pom.xml</relativePath>
     </parent>
 

connector/pom.xml

@@ -4,13 +4,13 @@
     <parent>
         <groupId>com.alibaba.otter</groupId>
         <artifactId>canal</artifactId>
-        <version>1.1.8</version>
+        <version>1.1.9-SNAPSHOT</version>
         <relativePath>../pom.xml</relativePath>
     </parent>
 
     <groupId>com.alibaba.otter</groupId>
     <artifactId>canal.connector</artifactId>
-    <version>1.1.8</version>
+    <version>1.1.9-SNAPSHOT</version>
     <packaging>pom</packaging>
     <name>canal connector module for otter ${project.version}</name>
 

connector/pulsarmq-connector/pom.xml

@@ -3,7 +3,7 @@
     <parent>
         <artifactId>canal.connector</artifactId>
         <groupId>com.alibaba.otter</groupId>
-        <version>1.1.8</version>
+        <version>1.1.9-SNAPSHOT</version>
         <relativePath>../pom.xml</relativePath>
     </parent>
     <modelVersion>4.0.0</modelVersion>

connector/rabbitmq-connector/pom.xml

@@ -3,7 +3,7 @@
     <parent>
         <artifactId>canal.connector</artifactId>
         <groupId>com.alibaba.otter</groupId>
-        <version>1.1.8</version>
+        <version>1.1.9-SNAPSHOT</version>
         <relativePath>../pom.xml</relativePath>
     </parent>
 

connector/rabbitmq-connector/src/main/java/com/alibaba/otter/canal/connector/rabbitmq/producer/CanalRabbitMQProducer.java

@@ -210,13 +210,12 @@ public class CanalRabbitMQProducer extends AbstractMQProducer implements CanalMQ
         try {
             this.channel.close();
             this.connect.close();
-            super.stop();
         } catch (AlreadyClosedException ex) {
             logger.error("Connection is already closed", ex);
         } catch (IOException | TimeoutException ex) {
             throw new CanalException("Stop RabbitMQ producer error", ex);
+        } finally {
+            super.stop();
         }
-
-        super.stop();
     }
 }

connector/rocketmq-connector/pom.xml

@@ -3,7 +3,7 @@
     <parent>
         <artifactId>canal.connector</artifactId>
         <groupId>com.alibaba.otter</groupId>
-        <version>1.1.8</version>
+        <version>1.1.9-SNAPSHOT</version>
         <relativePath>../pom.xml</relativePath>
     </parent>
 

connector/rocketmq-connector/src/main/java/com/alibaba/otter/canal/connector/rocketmq/producer/CanalRocketMQProducer.java

@@ -372,6 +372,10 @@ public class CanalRocketMQProducer extends AbstractMQProducer implements CanalMQ
             topicName = this.defaultMQProducer.withNamespace(topicName);
             DefaultMQProducerImpl innerProducer = this.defaultMQProducer.getDefaultMQProducerImpl();
             TopicPublishInfo topicInfo = innerProducer.getTopicPublishInfoTable().get(topicName);
+            if (topicInfo == null) {
+                innerProducer.getMqClientFactory().updateTopicRouteInfoFromNameServer(topicName);
+            }
+            topicInfo = innerProducer.getTopicPublishInfoTable().get(topicName);
             if (topicInfo == null) {
                 return null;
             } else {

connector/tcp-connector/pom.xml

@@ -3,7 +3,7 @@
     <parent>
         <artifactId>canal.connector</artifactId>
         <groupId>com.alibaba.otter</groupId>
-        <version>1.1.8</version>
+        <version>1.1.9-SNAPSHOT</version>
         <relativePath>../pom.xml</relativePath>
     </parent>
 

dbsync/pom.xml

@@ -3,7 +3,7 @@
 	<parent>
 		<groupId>com.alibaba.otter</groupId>
 		<artifactId>canal</artifactId>
-		<version>1.1.8</version>
+		<version>1.1.9-SNAPSHOT</version>
 		<relativePath>../pom.xml</relativePath>
 	</parent>
 	<groupId>com.alibaba.otter</groupId>

dbsync/src/main/java/com/taobao/tddl/dbsync/binlog/JsonConversion.java

@@ -158,7 +158,7 @@ public class JsonConversion {
                     throw new IllegalArgumentException("illegal json data");
                 }
                 return new Json_Value(Json_enum_type.STRING,
-                    buffer.rewind().forward((int) n).getFixString((int) str_len, charset));
+                    buffer.rewind().forward((int) n).getFixLengthString((int) str_len, charset));
             case JSONB_TYPE_OPAQUE:
                 /*
                  * There should always be at least one byte, which tells the field type of the
@@ -257,7 +257,7 @@ public class JsonConversion {
             // entry, always two
             // bytes.
             long key_length = m_data.getUint16();
-            return m_data.rewind().forward((int) key_offset).getFixString((int) key_length, charset);
+            return m_data.rewind().forward((int) key_offset).getFixLengthString((int) key_length, charset);
         }
 
         public Json_Value element(int i, Charset charset) {
@@ -400,7 +400,7 @@ public class JsonConversion {
                                    }
                                    buf.append('"').append(text).append('"');
                                } else {
-                                   text = m_data.getFixString((int) m_length, charset);
+                                   text = m_data.getFixLengthString((int) m_length, charset);
                                    buf.append('"').append(escapse(text)).append('"');
                                }
 

dbsync/src/main/java/com/taobao/tddl/dbsync/binlog/LogBuffer.java

@@ -188,8 +188,9 @@ public class LogBuffer {
      * <tt>newLimit</tt> do not hold
      */
     public final LogBuffer limit(int newLimit) {
-        if (origin + newLimit > buffer.length || newLimit < 0) throw new IllegalArgumentException("capacity excceed: "
-                                                                                                  + (origin + newLimit));
+        if (origin + newLimit > buffer.length || newLimit < 0) {
+            throw new IllegalArgumentException("capacity excceed: " + (origin + newLimit));
+        }
 
         limit = newLimit;
         return this;
@@ -1155,6 +1156,23 @@ public class LogBuffer {
         return string;
     }
 
+    /**
+     * Return next fix length string from buffer.
+     * see json_binary.cc read_variable_length
+     */
+    public final String getFixLengthString(final int len, Charset charset) {
+        if (position + len > origin + limit) {
+            throw new IllegalArgumentException("limit excceed: " + (position + len - origin));
+        }
+
+        final int from = position;
+        byte[] buf = buffer;
+
+        String string = new String(buf, from, len, charset);
+        position += len;
+        return string;
+    }
+
     public final String getFixName(final int len, Charset charset) {
         if (position + len > origin + limit) {
             throw new IllegalArgumentException("limit excceed: " + (position + len - origin));

dbsync/src/main/java/com/taobao/tddl/dbsync/binlog/LogEvent.java

@@ -187,6 +187,15 @@ public abstract class LogEvent {
     /* New MySQL/Sun events are to be added right above this comment */
     public static final int    MYSQL_EVENTS_END                         = 49;
 
+    // PolarDB-X 2.5.0
+    // CONSENSUS EVENTS START
+    public static final int    CONSENSUS_LOG_EVENT                      = 101;
+    public static final int    PREVIOUS_CONSENSUS_INDEX_LOG_EVENT       = 102;
+    public static final int    CONSENSUS_CLUSTER_INFO_EVENT             = 103;
+    public static final int    CONSENSUS_EMPTY_EVENT                    = 104;
+    public static final int    GCN_LOG_EVENT                            = 105;
+    // CONSENSUS EVENTS END
+
     public static final int    MARIA_EVENTS_BEGIN                       = 160;
     /* New Maria event numbers start from here */
     public static final int    ANNOTATE_ROWS_EVENT                      = 160;
@@ -229,7 +238,7 @@ public abstract class LogEvent {
     public static final int    DELETE_ROWS_COMPRESSED_EVENT             = 171;
 
     /** end marker */
-    public static final int    ENUM_END_EVENT                           = 171;
+    public static final int    ENUM_END_EVENT                           = 172;
 
     /**
      * 1 byte length, 1 byte format Length is total length in bytes, including 2
@@ -404,6 +413,16 @@ public abstract class LogEvent {
                 return "Xa_prepare";
             case TRANSACTION_PAYLOAD_EVENT :
                 return "transaction_payload";
+            case CONSENSUS_LOG_EVENT:
+                return "Consensus_log";
+            case PREVIOUS_CONSENSUS_INDEX_LOG_EVENT:
+                return "Previous_consensus_index";
+            case CONSENSUS_CLUSTER_INFO_EVENT:
+                return "Consensus_cluster_info";
+            case CONSENSUS_EMPTY_EVENT:
+                return "Consensus_empty";
+            case GCN_LOG_EVENT:
+                return "Gcn_log";
             default:
                 return "Unknown type:" + type;
         }

dbsync/src/main/java/com/taobao/tddl/dbsync/binlog/event/QueryLogEvent.java

@@ -3,6 +3,7 @@ package com.taobao.tddl.dbsync.binlog.event;
 import java.io.IOException;
 import java.math.BigInteger;
 import java.nio.charset.Charset;
+import java.nio.charset.StandardCharsets;
 
 import com.taobao.tddl.dbsync.binlog.CharsetConversion;
 import com.taobao.tddl.dbsync.binlog.LogBuffer;
@@ -281,82 +282,30 @@ public class QueryLogEvent extends LogEvent {
      * packet (i.e. a query) sent from client to master; First, an auxiliary
      * log_event status vars estimation:
      */
-    public static final int MAX_SIZE_LOG_EVENT_STATUS = (1 + 4 /* type, flags2 */
-                                                         + 1 + 8 /*
-                                                                  * type,
-                                                                  * sql_mode
-                                                                  */
-                                                         + 1 + 1 + 255/*
-                                                                       * type,
-                                                                       * length
-                                                                       * ,
-                                                                       * catalog
-                                                                       */
-                                                         + 1 + 4 /*
-                                                                  * type,
-                                                                  * auto_increment
-                                                                  */
-                                                         + 1 + 6 /*
-                                                                  * type,
-                                                                  * charset
-                                                                  */
-                                                         + 1 + 1 + 255 /*
-                                                                        * type,
-                                                                        * length
-                                                                        * ,
-                                                                        * time_zone
-                                                                        */
-                                                         + 1 + 2 /*
-                                                                  * type,
-                                                                  * lc_time_names_number
-                                                                  */
-                                                         + 1 + 2 /*
-                                                                  * type,
-                                                                  * charset_database_number
-                                                                  */
-                                                         + 1 + 8 /*
-                                                                  * type,
-                                                                  * table_map_for_update
-                                                                  */
-                                                         + 1 + 4 /*
-                                                                  * type,
-                                                                  * master_data_written
-                                                                  */
-                                                         /*
-                                                          * type, db_1, db_2,
-                                                          * ...
-                                                          */
-                                                         /* type, microseconds */
-                                                         /*
-                                                          * MariaDb type,
-                                                          * sec_part of NOW()
-                                                          */
-                                                         + 1 + (MAX_DBS_IN_EVENT_MTS * (1 + NAME_LEN)) + 3 /*
-                                                                                                            * type
-                                                                                                            * ,
-                                                                                                            * microseconds
-                                                                                                            */+ 1 + 32
-                                                         * 3 + 1 + 60/*
-                                                                      * type ,
-                                                                      * user_len
-                                                                      * , user ,
-                                                                      * host_len
-                                                                      * , host
-                                                                      */)
-                                                        + 1 + 1 /*
-                                                                 * type,
-                                                                 * explicit_def
-                                                                 * ..ts
-                                                                 */+ 1 + 8 /*
-                                                                            * type,
-                                                                            * xid
-                                                                            * of
-                                                                            * DDL
-                                                                            */+ 1 + 2 /*
-                                                                                       * type
-                                                                                       * ,
-                                                                                       * default_collation_for_utf8mb4_number
-                                                                                       */+ 1 /* sql_require_primary_key */;
+    public static final int MAX_SIZE_LOG_EVENT_STATUS = (1 + 4 /* type, flags2 */ + 1 + 8 /* type,sql_mode */ + 1 + 1 + 255 /* type,length, catalog */
+                                                         + 1 + 4 /* type,auto_increment */
+                                                         + 1 + 6 /* type, charset */
+                                                         + 1 + 1 + 255 /* type, length, time_zone */
+                                                         + 1 + 2 /* type, lc_time_names_number */
+                                                         + 1 + 2 /* type, charset_database_number */
+                                                         + 1 + 8 /* type, table_map_for_update */
+                                                         + 1 + 4 /* type,master_data_written */
+                                                         + 1 + 1 + 32 * 3 /* type, user_len, user */
+                                                         + 1 + 255 /* host_len, host */
+                                                         + 1 + 1 + (MAX_DBS_IN_EVENT_MTS * (1 + NAME_LEN)) /* type, db_1, db_2, ... */
+                                                         + 1 + 3 /* type, microseconds */
+                                                         + 1 + 1 /* type, explicit_def..ts*/
+                                                         + 1 + 8 /* type, xid of DDL */
+                                                         + 1 + 2 /* type, default_collation_for_utf8mb4_number */
+                                                         + 1 + 1 /* sql_require_primary_key */
+                                                         + 1 + 1 /* type, default_table_encryption */
+                                                         + 1 + 1 /* opt_flashback_area */
+                                                         + 1 + 1 /* type, opt_index_format_gpp_enabled */
+                                                         + 1 + 1 /* type, opt_index_format_hippo_enabled */
+                                                         + 1 + 1 /* type, opt_index_format_spc_enabled */
+                                                         + 1 + 1 /* type, opt_index_format_panda_enabled */
+                                                         + 1 + 1 + NAME_LEN /* type, length, recycle_bin_table_name */);
+
     /**
      * Fixed data part:
      * <ul>
@@ -670,6 +619,104 @@ public class QueryLogEvent extends LogEvent {
 
     public static final int Q_LIZARD_PREPARE_GCN              = 201;
 
+    /**
+     * Replicate recycle_bin_table_name.
+     */
+    /**
+     * Replicate opt_index_format_spc_enabled.
+     *
+     * @since PolarDB-X
+     */
+    public static final int Q_OPT_INDEX_FORMAT_SPC_ENABLED    = 252;
+
+    /**
+     * Replicate opt_index_format_hippo_enabled.
+     *
+     * @since PolarDB-X
+     */
+    public static final int Q_OPT_INDEX_FORMAT_HIPPO_ENABLED  = 253;
+
+    /*
+      Replicate recycle_bin_table_name.
+
+      @since PolarDB-X
+    */
+    public static final int Q_OPT_RECYCLE_BIN_TABLE_NAME      = 254;
+
+    /*
+      Replicate opt_index_format_panda_enabled.
+      Private codes are allocated from 255 downward (deployment order).
+
+      @since PolarDB-X
+    */
+    public static final int Q_OPT_INDEX_FORMAT_PANDA_ENABLED  = 255;
+
+    private static final String findCodeName(final int code) {
+        switch (code) {
+            case Q_FLAGS2_CODE:
+                return "Q_FLAGS2_CODE";
+            case Q_SQL_MODE_CODE:
+                return "Q_SQL_MODE_CODE";
+            case Q_CATALOG_CODE:
+                return "Q_CATALOG_CODE";
+            case Q_AUTO_INCREMENT:
+                return "Q_AUTO_INCREMENT";
+            case Q_CHARSET_CODE:
+                return "Q_CHARSET_CODE";
+            case Q_TIME_ZONE_CODE:
+                return "Q_TIME_ZONE_CODE";
+            case Q_CATALOG_NZ_CODE:
+                return "Q_CATALOG_NZ_CODE";
+            case Q_LC_TIME_NAMES_CODE:
+                return "Q_LC_TIME_NAMES_CODE";
+            case Q_CHARSET_DATABASE_CODE:
+                return "Q_CHARSET_DATABASE_CODE";
+            case Q_TABLE_MAP_FOR_UPDATE_CODE:
+                return "Q_TABLE_MAP_FOR_UPDATE_CODE";
+            case Q_MASTER_DATA_WRITTEN_CODE:
+                return "Q_MASTER_DATA_WRITTEN_CODE";
+            case Q_INVOKER:
+                return "Q_INVOKER";
+            case Q_MICROSECONDS:
+                return "Q_MICROSECONDS";
+            case Q_UPDATED_DB_NAMES:
+                return "Q_UPDATED_DB_NAMES";
+            case Q_EXPLICIT_DEFAULTS_FOR_TIMESTAMP:
+                return "Q_EXPLICIT_DEFAULTS_FOR_TIMESTAMP";
+            case Q_DDL_LOGGED_WITH_XID:
+                return "Q_DDL_LOGGED_WITH_XID";
+            case Q_DEFAULT_COLLATION_FOR_UTF8MB4:
+                return "Q_DEFAULT_COLLATION_FOR_UTF8MB4";
+            case Q_SQL_REQUIRE_PRIMARY_KEY:
+                return "Q_SQL_REQUIRE_PRIMARY_KEY";
+            case Q_DEFAULT_TABLE_ENCRYPTION:
+                return "Q_DEFAULT_TABLE_ENCRYPTION";
+            case Q_OPT_FLASHBACK_AREA:
+                // or Q_DDL_SKIP_REWRITE
+                return "Q_OPT_FLASHBACK_AREA";
+            case Q_OPT_INDEX_FORMAT_GPP_ENABLED:
+                return "Q_OPT_INDEX_FORMAT_GPP_ENABLED";
+            case Q_OPT_INDEX_FORMAT_SPC_ENABLED:
+                return "Q_OPT_INDEX_FORMAT_SPC_ENABLED";
+            case Q_OPT_INDEX_FORMAT_HIPPO_ENABLED:
+                return "Q_OPT_INDEX_FORMAT_HIPPO_ENABLED";
+            case Q_OPT_RECYCLE_BIN_TABLE_NAME:
+                return "Q_OPT_RECYCLE_BIN_TABLE_NAME";
+            case Q_OPT_INDEX_FORMAT_PANDA_ENABLED:
+                return "Q_OPT_INDEX_FORMAT_PANDA_ENABLED";
+            case Q_HRNOW:
+                // or Q_WSREP_SKIP_READONLY_CHECKS
+                return "Q_HRNOW";
+            case Q_XID:
+                return "Q_XID";
+            case Q_GTID_FLAGS3:
+                return "Q_GTID_FLAGS3";
+            case Q_CHARACTER_SET_COLLATIONS :
+                return "Q_CHARACTER_SET_COLLATIONS";
+        }
+        return "CODE#" + code;
+    }
+
     private final void unpackVariables(LogBuffer buffer, final int end) throws IOException {
         int code = -1;
         try {
@@ -773,7 +820,7 @@ public class QueryLogEvent extends LogEvent {
                             // percona
                             // *start++ = thd->variables.binlog_ddl_skip_rewrite;
                             buffer.forward(1);
-                        }else {
+                        } else {
                             // PolarDB-X
                             // *start++ = thd->variables.opt_flashback_area;
                             buffer.forward(1);
@@ -828,6 +875,30 @@ public class QueryLogEvent extends LogEvent {
                         // prepareGCN = buffer.getLong64();
                         buffer.forward(8);
                         break;
+                    case Q_OPT_INDEX_FORMAT_SPC_ENABLED:
+                        // *start++ = thd->variables.opt_index_format_spc_enabled;
+                        buffer.forward(1);
+                        break;
+                    case Q_OPT_INDEX_FORMAT_HIPPO_ENABLED:
+                        // *start++ = thd->variables.opt_index_format_hippo_enabled;
+                        buffer.forward(1);
+                        break;
+                    case Q_OPT_RECYCLE_BIN_TABLE_NAME:
+                        int recycle_bin_table_name_len = buffer.getUint8();
+                        String recycle_bin_table_name = null;
+                        if (recycle_bin_table_name_len > 0) {
+                            if (charset != null) {
+                                recycle_bin_table_name = buffer.getFixLengthString(recycle_bin_table_name_len, charset);
+                            } else {
+                                recycle_bin_table_name = buffer.getFixLengthString(recycle_bin_table_name_len,
+                                    StandardCharsets.ISO_8859_1);
+                            }
+                        }
+                        break;
+                    case Q_OPT_INDEX_FORMAT_PANDA_ENABLED:
+                        // *start++ = thd->variables.opt_index_format_panda_enabled;
+                        buffer.forward(1);
+                        break;
                     default:
                         /*
                          * That's why you must write status vars in growing
@@ -835,7 +906,7 @@ public class QueryLogEvent extends LogEvent {
                          */
                         logger.error("Query_log_event has unknown status vars (first has code: " + code
                                      + "), skipping the rest of them");
-                        break; // Break loop
+                        return; // Break loop
                 }
             }
         } catch (RuntimeException e) {
@@ -843,64 +914,6 @@ public class QueryLogEvent extends LogEvent {
         }
     }
 
-    private static final String findCodeName(final int code) {
-        switch (code) {
-            case Q_FLAGS2_CODE:
-                return "Q_FLAGS2_CODE";
-            case Q_SQL_MODE_CODE:
-                return "Q_SQL_MODE_CODE";
-            case Q_CATALOG_CODE:
-                return "Q_CATALOG_CODE";
-            case Q_AUTO_INCREMENT:
-                return "Q_AUTO_INCREMENT";
-            case Q_CHARSET_CODE:
-                return "Q_CHARSET_CODE";
-            case Q_TIME_ZONE_CODE:
-                return "Q_TIME_ZONE_CODE";
-            case Q_CATALOG_NZ_CODE:
-                return "Q_CATALOG_NZ_CODE";
-            case Q_LC_TIME_NAMES_CODE:
-                return "Q_LC_TIME_NAMES_CODE";
-            case Q_CHARSET_DATABASE_CODE:
-                return "Q_CHARSET_DATABASE_CODE";
-            case Q_TABLE_MAP_FOR_UPDATE_CODE:
-                return "Q_TABLE_MAP_FOR_UPDATE_CODE";
-            case Q_MASTER_DATA_WRITTEN_CODE:
-                return "Q_MASTER_DATA_WRITTEN_CODE";
-            case Q_INVOKER:
-                return "Q_INVOKER";
-            case Q_MICROSECONDS:
-                return "Q_MICROSECONDS";
-            case Q_UPDATED_DB_NAMES:
-                return "Q_UPDATED_DB_NAMES";
-            case Q_EXPLICIT_DEFAULTS_FOR_TIMESTAMP:
-                return "Q_EXPLICIT_DEFAULTS_FOR_TIMESTAMP";
-            case Q_DDL_LOGGED_WITH_XID:
-                return "Q_DDL_LOGGED_WITH_XID";
-            case Q_DEFAULT_COLLATION_FOR_UTF8MB4:
-                return "Q_DEFAULT_COLLATION_FOR_UTF8MB4";
-            case Q_SQL_REQUIRE_PRIMARY_KEY:
-                return "Q_SQL_REQUIRE_PRIMARY_KEY";
-            case Q_DEFAULT_TABLE_ENCRYPTION:
-                return "Q_DEFAULT_TABLE_ENCRYPTION";
-            case Q_OPT_FLASHBACK_AREA:
-                // or Q_DDL_SKIP_REWRITE
-                return "Q_DDL_SKIP_REWRITE";
-            case Q_OPT_INDEX_FORMAT_GPP_ENABLED:
-                return "Q_DDL_SKIP_REWRITE";
-            case Q_HRNOW:
-                // or Q_WSREP_SKIP_READONLY_CHECKS
-                return "Q_HRNOW";
-            case Q_XID:
-                return "Q_XID";
-            case Q_GTID_FLAGS3:
-                return "Q_GTID_FLAGS3";
-            case Q_CHARACTER_SET_COLLATIONS :
-                return "Q_CHARACTER_SET_COLLATIONS";
-        }
-        return "CODE#" + code;
-    }
-
     public final String getUser() {
         return user;
     }

dbsync/src/main/java/com/taobao/tddl/dbsync/binlog/event/TableMapLogEvent.java

@@ -501,7 +501,10 @@ public final class TableMapLogEvent extends LogEvent {
                         parse_column_visibility(buffer, len);
                         break;
                     default:
-                        throw new IllegalArgumentException("unknow type : " + type);
+                        // Skip unknown optional metadata types for forward compatibility
+                        // (e.g. PolarDB, MariaDB, or future MySQL versions may add new types)
+                        buffer.forward(len);
+                        break;
                 }
             }
 

dbsync/src/test/java/com/taobao/tddl/dbsync/binlog/event/TableMapLogEventTest.java

@@ -0,0 +1,59 @@
+package com.taobao.tddl.dbsync.binlog.event;
+
+import com.taobao.tddl.dbsync.binlog.LogBuffer;
+import org.junit.Assert;
+import org.junit.Test;
+
+/**
+ * Tests for {@link TableMapLogEvent} optional metadata parsing.
+ */
+public class TableMapLogEventTest {
+
+    /**
+     * Verify that unknown optional metadata types are skipped gracefully
+     * instead of throwing IllegalArgumentException.
+     *
+     * @see <a href="https://github.com/alibaba/canal/issues/5391">Issue #5391</a>
+     */
+    @Test
+    public void testUnknownOptionalMetadataTypeIsSkipped() {
+        // LogBuffer.forward() is the mechanism used to skip unknown metadata.
+        // Verify it correctly advances the position without reading the data.
+        byte[] data = new byte[]{0x01, 0x02, 0x03, 0x04, 0x05};
+        LogBuffer buffer = new LogBuffer(data, 0, data.length);
+
+        int posBefore = buffer.position();
+        buffer.forward(3);
+        int posAfter = buffer.position();
+
+        Assert.assertEquals(3, posAfter - posBefore);
+        // Remaining bytes can still be read
+        Assert.assertEquals(0x04, buffer.getUint8());
+        Assert.assertEquals(0x05, buffer.getUint8());
+    }
+
+    /**
+     * Verify that forward() with the exact remaining length consumes the buffer completely.
+     */
+    @Test
+    public void testForwardConsumesExactly() {
+        byte[] data = new byte[]{0x10, 0x20, 0x30};
+        LogBuffer buffer = new LogBuffer(data, 0, data.length);
+
+        buffer.forward(3);
+        Assert.assertFalse(buffer.hasRemaining());
+    }
+
+    /**
+     * Verify that forward(0) is a no-op.
+     */
+    @Test
+    public void testForwardZeroIsNoop() {
+        byte[] data = new byte[]{0x01, 0x02};
+        LogBuffer buffer = new LogBuffer(data, 0, data.length);
+
+        int posBefore = buffer.position();
+        buffer.forward(0);
+        Assert.assertEquals(posBefore, buffer.position());
+    }
+}

deployer/pom.xml

@@ -3,7 +3,7 @@
     <parent>
         <groupId>com.alibaba.otter</groupId>
         <artifactId>canal</artifactId>
-        <version>1.1.8</version>
+        <version>1.1.9-SNAPSHOT</version>
         <relativePath>../pom.xml</relativePath>
     </parent>
     <groupId>com.alibaba.otter</groupId>

deployer/src/main/bin/startup.bat

@@ -15,13 +15,13 @@ set CLASSPATH=%conf_dir%
 set CLASSPATH=%conf_dir%\..\lib\*;%CLASSPATH%
 
 set JAVA_MEM_OPTS= -Xms128m -Xmx512m -XX:PermSize=128m
-set JAVA_OPTS_EXT= -Djava.awt.headless=true -Djava.net.preferIPv4Stack=true -Dapplication.codeset=UTF-8 -Dfile.encoding=UTF-8
+set JAVA_OPTS_EXT= -Djava.awt.headless=true -Djava.net.preferIPv4Stack=false -Dapplication.codeset=UTF-8 -Dfile.encoding=UTF-8
 set JAVA_DEBUG_OPT= -server -Xdebug -Xnoagent -Djava.compiler=NONE -Xrunjdwp:transport=dt_socket,address=9099,server=y,suspend=n
 set CANAL_OPTS= -DappName=otter-canal -Dlogback.configurationFile="%logback_configurationFile%" -Dcanal.conf="%canal_conf%"
 
 set JAVA_OPTS= %JAVA_MEM_OPTS% %JAVA_OPTS_EXT% %JAVA_DEBUG_OPT% %CANAL_OPTS%
 
-set CMD_STR= java %JAVA_OPTS% -classpath "%CLASSPATH%" java %JAVA_OPTS% -classpath "%CLASSPATH%" com.alibaba.otter.canal.deployer.CanalLauncher
+set CMD_STR= java %JAVA_OPTS% -classpath "%CLASSPATH%" com.alibaba.otter.canal.deployer.CanalLauncher
 echo start cmd : %CMD_STR%
 
 java %JAVA_OPTS% -classpath "%CLASSPATH%" com.alibaba.otter.canal.deployer.CanalLauncher

deployer/src/main/bin/startup.sh

@@ -9,10 +9,14 @@ case "`uname`" in
 		bin_abs_path=`cd $(dirname $0); pwd`
 		;;
 esac
+
 base=${bin_abs_path}/..
+
+jaas_conf=$base/conf/jaas.conf
 canal_conf=$base/conf/canal.properties
 canal_local_conf=$base/conf/canal_local.properties
 logback_configurationFile=$base/conf/logback.xml
+
 export LANG=en_US.UTF-8
 export BASE=$base
 
@@ -79,31 +83,45 @@ in
 	exit;;
 esac
 
-JavaVersion=`$JAVA -version 2>&1 |awk 'NR==1{ gsub(/"/,""); print $3 }' | awk  -F '.' '{print $1}'`
-str=`file -L $JAVA | grep 64-bit`
-JAVA_OPTS="$JAVA_OPTS -Xss1m -XX:-UseBiasedLocking -XX:-OmitStackTraceInFastThrow -XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=$base/logs"
+JAVA_VERSION=`$JAVA -version 2>&1 |awk 'NR==1{ gsub(/"/,""); print $3 }' | awk  -F '.' '{print $1}'`
 
-if [ $JavaVersion -ge 11 ] ; then
+JAVA_OPTS="$JAVA_OPTS -Xss1m -XX:+AggressiveOpts -XX:-UseBiasedLocking -XX:-OmitStackTraceInFastThrow"
+JAVA_OPTS="$JAVA_OPTS -XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=$base/logs"
+
+if [ $JAVA_VERSION -ge 11 ] ; then
   #JAVA_OPTS="$JAVA_OPTS -Xlog:gc*:$base_log/gc.log:time "
-  JAVA_OPTS="$JAVA_OPTS"
+  JAVA_OPTS="$JAVA_OPTS -Xlog:gc*:file=$base/logs/adapter/gc.log::filecount=5,filesize=32M"
 else
-  #JAVA_OPTS="$JAVA_OPTS -Xloggc:$base/logs/canal/gc.log -XX:+PrintGCDetails -XX:+PrintGCDateStamps -XX:+PrintGCApplicationStoppedTime"
-  JAVA_OPTS="$JAVA_OPTS -XX:+AggressiveOpts -XX:+UseFastAccessorMethods -XX:+PrintAdaptiveSizePolicy -XX:+PrintTenuringDistribution"
+  JAVA_OPTS="$JAVA_OPTS -Xloggc:$base/logs/canal/gc.log -XX:+UseGCLogFileRotation -XX:NumberOfGCLogFiles=5 -XX:GCLogFileSize=32M"
+  JAVA_OPTS="$JAVA_OPTS -XX:+PrintGCDetails -XX:+PrintGCDateStamps -XX:-PrintHeapAtGC -XX:+PrintGCApplicationStoppedTime"
+  JAVA_OPTS="$JAVA_OPTS -XX:+UseFastAccessorMethods -XX:-PrintReferenceGC -XX:+PrintAdaptiveSizePolicy -XX:+PrintTenuringDistribution"
+  JAVA_OPTS="$JAVA_OPTS -XX:+PrintSafepointStatistics -XX:PrintSafepointStatisticsCount=1 -XX:+SafepointTimeout -XX:SafepointTimeoutDelay=2000"
+  JAVA_OPTS="$JAVA_OPTS -XX:+UseCountedLoopSafepoints -XX:+DisableExplicitGC -XX:+ExplicitGCInvokesConcurrent -XX:+ExplicitGCInvokesConcurrentAndUnloadsClasses"
 fi
 
-if [ -n "$str" ]; then
-  if [ $JavaVersion -ge 11 ] ; then
+OS_ARCH=`file -L $JAVA | grep 64-bit`
+
+if [ -n "$OS_ARCH" ]; then
+  if [ $JAVA_VERSION -ge 11 ] ; then
     # For G1
     JAVA_OPTS="-server -Xms2g -Xmx3g -XX:+UseG1GC -XX:MaxGCPauseMillis=250 -XX:+UseGCOverheadLimit -XX:+ExplicitGCInvokesConcurrent $JAVA_OPTS"
   else
 	  JAVA_OPTS="-server -Xms2g -Xmx3g -Xmn1g -XX:SurvivorRatio=2 -XX:PermSize=96m -XX:MaxPermSize=256m -XX:MaxTenuringThreshold=15 -XX:+DisableExplicitGC $JAVA_OPTS"
+	  ## JDK 1.8 2c4g [-XX:ConcGCThreads=1 -XX:ParallelGCThreads=2], 4c8g [-XX:ConcGCThreads=2 -XX:ParallelGCThreads=4]
+	  #JAVA_OPTS="$JAVA_OPTS -XX:InitialRAMPercentage=45.0 -XX:MinRAMPercentage=45.0 -XX:MaxRAMPercentage=45.0"
+	  #JAVA_OPTS="$JAVA_OPTS -XX:-UseParallelGC -XX:+UseConcMarkSweepGC -XX:ConcGCThreads=2 -XX:ParallelGCThreads=4"
+	  #JAVA_OPTS="$JAVA_OPTS -XX:+UseCMSInitiatingOccupancyOnly -XX:CMSInitiatingOccupancyFraction=70"
+	  #JAVA_OPTS="-server -XX:NewRatio=1 -XX:SurvivorRatio=2 -XX:MetaspaceSize=96m -XX:MaxMetaspaceSize=128m "
 	fi
 else
 	JAVA_OPTS="-server -Xms1024m -Xmx1024m -XX:NewSize=256m -XX:MaxNewSize=256m -XX:MaxPermSize=128m $JAVA_OPTS"
 fi
 
-JAVA_OPTS=" $JAVA_OPTS -Djava.awt.headless=true -Djava.net.preferIPv4Stack=true -Dfile.encoding=UTF-8"
+JAVA_OPTS=" $JAVA_OPTS -Djava.awt.headless=true -Djava.net.preferIPv4Stack=false -Dfile.encoding=UTF-8"
 CANAL_OPTS="-DappName=otter-canal -Dlogback.configurationFile=$logback_configurationFile -Dcanal.conf=$canal_conf"
+if [ -f "$jaas_conf" ]; then
+  CANAL_OPTS="$CANAL_OPTS -Djava.security.auth.login.config=$jaas_conf"
+fi
 
 if [ -e $canal_conf -a -e $logback_configurationFile ]
 then 

deployer/src/main/java/com/alibaba/otter/canal/deployer/admin/CanalAdminController.java

@@ -1,6 +1,7 @@
 package com.alibaba.otter.canal.deployer.admin;
 
 import java.io.File;
+import java.io.IOException;
 import java.security.NoSuchAlgorithmException;
 import java.util.ArrayList;
 import java.util.Collection;
@@ -203,7 +204,10 @@ public class CanalAdminController implements CanalAdmin {
 
     @Override
     public String listInstanceLog(String destination) {
-        Collection<File> files = org.apache.commons.io.FileUtils.listFiles(new File("../logs/" + destination + "/"),
+        // 校验destination
+        String desPath = FileUtils.validateFileName("../logs", destination);
+
+        Collection<File> files = org.apache.commons.io.FileUtils.listFiles(new File(desPath),
             TrueFileFilter.TRUE,
             TrueFileFilter.TRUE);
         List<String> names = files.stream().map(File::getName).collect(Collectors.toList());
@@ -215,7 +219,12 @@ public class CanalAdminController implements CanalAdmin {
         if (StringUtils.isEmpty(fileName)) {
             fileName = destination + ".log";
         }
-        return FileUtils.readFileFromOffset("../logs/" + destination + "/" + fileName, lines, "UTF-8");
+
+        // 分别校验destination和fileName目录
+        String desPath = FileUtils.validateFileName("../logs", destination);
+        String fullPath = FileUtils.validateFileName(desPath , fileName);
+
+        return FileUtils.readFileFromOffset(fullPath, lines, "UTF-8");
     }
 
     private InstanceAction getInstanceAction(String destination) {

deployer/src/main/resources/logback.xml

@@ -86,6 +86,10 @@
 		<level value="INFO" />
 		<appender-ref ref="CANAL-ROOT" />
 	</logger>
+	<logger name="com.alibaba.otter.canal.server.embedded" additivity="false">
+		<level value="INFO" />
+		<appender-ref ref="CANAL-ROOT" />
+	</logger>
 	<logger name="com.alibaba.otter.canal.meta.FileMixedMetaManager" additivity="false">
 		<level value="INFO" />
 		<appender-ref ref="CANAL-META" />

docker/base/amd64/Dockerfile

@@ -23,7 +23,6 @@ RUN \
 
 RUN \
     /usr/bin/ln -fs /usr/share/zoneinfo/Asia/Shanghai /etc/localtime && \
-    echo 'root:Hello1234' | chpasswd && \
     groupadd -r admin && useradd -g admin admin && \
     yum install -y man && \
     yum install -y dstat && \
@@ -39,6 +38,8 @@ RUN \
     ssh-keygen -q -N "" -t rsa -f /etc/ssh/ssh_host_rsa_key && \
     sed -ri 's/session    required     pam_loginuid.so/#session    required     pam_loginuid.so/g' /etc/pam.d/sshd && \
     sed -i -e 's/^#Port 22$/Port 2222/' /etc/ssh/sshd_config && \
+    sed -i 's/^#\?PermitRootLogin.*/PermitRootLogin prohibit-password/' /etc/ssh/sshd_config && \
+    sed -i 's/^#\?PasswordAuthentication.*/PasswordAuthentication no/' /etc/ssh/sshd_config && \
     mkdir -p /root/.ssh && chown root.root /root && chmod 700 /root/.ssh && \
     yum install -y cronie && \
     sed -i '/session required pam_loginuid.so/d' /etc/pam.d/crond && \

docker/base/arm64/Dockerfile

@@ -23,7 +23,6 @@ RUN \
 
 RUN \
     /usr/bin/ln -fs /usr/share/zoneinfo/Asia/Shanghai /etc/localtime && \
-    echo 'root:Hello1234' | chpasswd && \
     groupadd -r admin && useradd -g admin admin && \
     yum install -y man && \
     yum install -y dstat && \
@@ -39,6 +38,8 @@ RUN \
     ssh-keygen -q -N "" -t rsa -f /etc/ssh/ssh_host_rsa_key && \
     sed -ri 's/session    required     pam_loginuid.so/#session    required     pam_loginuid.so/g' /etc/pam.d/sshd && \
     sed -i -e 's/^#Port 22$/Port 2222/' /etc/ssh/sshd_config && \
+    sed -i 's/^#\?PermitRootLogin.*/PermitRootLogin prohibit-password/' /etc/ssh/sshd_config && \
+    sed -i 's/^#\?PasswordAuthentication.*/PasswordAuthentication no/' /etc/ssh/sshd_config && \
     mkdir -p /root/.ssh && chown root.root /root && chmod 700 /root/.ssh && \
     yum install -y cronie && \
     sed -i '/session required pam_loginuid.so/d' /etc/pam.d/crond && \

docker/image/app_admin.sh

@@ -13,6 +13,7 @@ MYSQL_USER_PASSWORD=`perl -le 'print $ENV{"spring.datasource.password"}'`
 MYSQL_USER=`perl -le 'print $ENV{"spring.datasource.username"}'`
 MYSQL_DATABASE=`perl -le 'print $ENV{"spring.datasource.database"}'`
 MYSQL_ADDRESS=`perl -le 'print $ENV{"spring.datasource.address"}'`
+MYSQL_ROOT_PASSWORD=`perl -le 'print $ENV{"spring.datasource.root.password"}'`
 
 if [ -z "${MYSQL_USER_PASSWORD}" ]; then
     MYSQL_USER_PASSWORD="canal"
@@ -91,7 +92,9 @@ function checkStart() {
 function start_mysql() {
     echo "start mysql ..."
     # start mysql
-    MYSQL_ROOT_PASSWORD=Hello1234
+    if [ -z "${MYSQL_ROOT_PASSWORD}" ]; then
+        MYSQL_ROOT_PASSWORD=$(cat /dev/urandom | tr -dc 'A-Za-z0-9!@#$%^&*' | head -c 16)
+    fi
     # connect local mysql
     if [ -z "$(ls -A /var/lib/mysql)" ]; then
         TEMP_FILE='/tmp/init.sql'

driver/pom.xml

@@ -3,7 +3,7 @@
 	<parent>
 		<groupId>com.alibaba.otter</groupId>
 		<artifactId>canal</artifactId>
-		<version>1.1.8</version>
+		<version>1.1.9-SNAPSHOT</version>
 		<relativePath>../pom.xml</relativePath>
 	</parent>
 	<groupId>com.alibaba.otter</groupId>

driver/src/main/java/com/alibaba/otter/canal/parse/driver/mysql/MysqlConnector.java

@@ -306,7 +306,7 @@ public class MysqlConnector {
                         byte[] scramble = authData;
                         try {
                             encryptedPassword = MySQLPasswordEncrypter.scrambleCachingSha2(getPassword().getBytes(),
-                                    scramble);
+                                scramble);
                         } catch (DigestException e) {
                             throw new RuntimeException("can't encrypt password that will be sent to MySQL server.", e);
                         }
@@ -317,6 +317,9 @@ public class MysqlConnector {
                             // fixed issue https://github.com/alibaba/canal/pull/4767, support mysql 8.0.30+
                             header = cachingSha2PasswordFullAuth(channel, header, getPassword().getBytes(), scramble);
                             body = PacketManager.readBytes(channel, header.getPacketBodyLength(), timeout);
+                        } else {
+                            header = PacketManager.readHeader(channel, 4);
+                            body = PacketManager.readBytes(channel, header.getPacketBodyLength(), timeout);
                         }
                     }
                 } else {

driver/src/main/java/com/alibaba/otter/canal/parse/driver/mysql/packets/MariaGtid.java

@@ -72,6 +72,9 @@ public class MariaGtid {
 
     @Override
     public String toString() {
-        return String.format("%s-%s-%s", domainId, serverId, sequence);
+        StringBuilder sb = new StringBuilder(16);
+        sb.append(domainId).append("-");
+        sb.append(serverId).append("-");
+        return sb.append(sequence).toString();
     }
 }

example/pom.xml

@@ -3,7 +3,7 @@
 	<parent>
 		<groupId>com.alibaba.otter</groupId>
 		<artifactId>canal</artifactId>
-		<version>1.1.8</version>
+		<version>1.1.9-SNAPSHOT</version>
 		<relativePath>../pom.xml</relativePath>
 	</parent>
 	<groupId>com.alibaba.otter</groupId>

example/src/main/bin/startup.bat

@@ -14,7 +14,7 @@ set CLASSPATH=%conf_dir%
 set CLASSPATH=%conf_dir%\..\lib\*;%CLASSPATH%
 
 set JAVA_MEM_OPTS= -Xms128m -Xmx512m -XX:PermSize=128m
-set JAVA_OPTS_EXT= -Djava.awt.headless=true -Djava.net.preferIPv4Stack=true -Dapplication.codeset=UTF-8 -Dfile.encoding=UTF-8
+set JAVA_OPTS_EXT= -Djava.awt.headless=true -Djava.net.preferIPv4Stack=false -Dapplication.codeset=UTF-8 -Dfile.encoding=UTF-8
 set JAVA_DEBUG_OPT= -server -Xdebug -Xnoagent -Djava.compiler=NONE -Xrunjdwp:transport=dt_socket,address=9199,server=y,suspend=n
 set CANAL_OPTS= -DappName=otter-canal-example -Dlogback.configurationFile="%logback_configurationFile%"
 

example/src/main/bin/startup.sh

@@ -81,7 +81,7 @@ else
 	JAVA_OPTS="-server -Xms1024m -Xmx1024m -XX:NewSize=256m -XX:MaxNewSize=256m -XX:MaxPermSize=128m $JAVA_OPTS"
 fi
 
-JAVA_OPTS=" $JAVA_OPTS -Djava.awt.headless=true -Djava.net.preferIPv4Stack=true -Dfile.encoding=UTF-8"
+JAVA_OPTS=" $JAVA_OPTS -Djava.awt.headless=true -Djava.net.preferIPv4Stack=false -Dfile.encoding=UTF-8"
 CANAL_OPTS="-DappName=otter-canal-example -Dlogback.configurationFile=$logback_configurationFile"
 
 if [ -e $logback_configurationFile ]

filter/pom.xml

@@ -3,7 +3,7 @@
 	<parent>
 		<groupId>com.alibaba.otter</groupId>
 		<artifactId>canal</artifactId>
-		<version>1.1.8</version>
+		<version>1.1.9-SNAPSHOT</version>
 		<relativePath>../pom.xml</relativePath>
 	</parent>
 	<groupId>com.alibaba.otter</groupId>

filter/src/main/java/com/alibaba/otter/canal/filter/aviater/AviaterRegexFilter.java

@@ -1,7 +1,6 @@
 package com.alibaba.otter.canal.filter.aviater;
 
 import java.util.ArrayList;
-import java.util.Arrays;
 import java.util.Comparator;
 import java.util.HashMap;
 import java.util.List;
@@ -21,7 +20,6 @@ import com.googlecode.aviator.Expression;
  */
 public class AviaterRegexFilter implements CanalEventFilter<String> {
 
-    private static final String             SPLIT             = ",";
     private static final String             PATTERN_SPLIT     = "|";
     private static final String             FILTER_EXPRESSION = "regex(pattern,target)";
     private static final RegexFunction      regexFunction     = new RegexFunction();
@@ -45,8 +43,9 @@ public class AviaterRegexFilter implements CanalEventFilter<String> {
         if (StringUtils.isEmpty(pattern)) {
             list = new ArrayList<>();
         } else {
-            String[] ss = StringUtils.split(pattern, SPLIT);
-            list = Arrays.asList(ss);
+            // fix issue #5442: only split on commas outside regex grouping constructs,
+            // so quantifiers like \d{1,2} are kept intact.
+            list = splitPattern(pattern);
         }
 
         // 对pattern按照从长到短的排序
@@ -58,6 +57,60 @@ public class AviaterRegexFilter implements CanalEventFilter<String> {
         this.pattern = StringUtils.join(list, PATTERN_SPLIT);
     }
 
+    /**
+     * 按顶层逗号拆分多个正则表达式，跳过位于 {}, [], () 内的逗号，避免破坏类似 \d{1,2} 的量词写法。
+     */
+    private static List<String> splitPattern(String pattern) {
+        List<String> result = new ArrayList<>();
+        int braceDepth = 0;
+        int bracketDepth = 0;
+        int parenDepth = 0;
+        int start = 0;
+        int len = pattern.length();
+        for (int i = 0; i < len; i++) {
+            char c = pattern.charAt(i);
+            // 反斜杠转义：跳过下一个字符，避免把 \{ \[ \( 等误判为分组开始
+            if (c == '\\' && i + 1 < len) {
+                i++;
+                continue;
+            }
+            switch (c) {
+                case '{':
+                    braceDepth++;
+                    break;
+                case '}':
+                    if (braceDepth > 0) braceDepth--;
+                    break;
+                case '[':
+                    bracketDepth++;
+                    break;
+                case ']':
+                    if (bracketDepth > 0) bracketDepth--;
+                    break;
+                case '(':
+                    parenDepth++;
+                    break;
+                case ')':
+                    if (parenDepth > 0) parenDepth--;
+                    break;
+                case ',':
+                    if (braceDepth == 0 && bracketDepth == 0 && parenDepth == 0) {
+                        if (i > start) {
+                            result.add(pattern.substring(start, i));
+                        }
+                        start = i + 1;
+                    }
+                    break;
+                default:
+                    break;
+            }
+        }
+        if (start < len) {
+            result.add(pattern.substring(start));
+        }
+        return result;
+    }
+
     public boolean filter(String filtered) throws CanalFilterException {
         if (StringUtils.isEmpty(pattern)) {
             return defaultEmptyValue;

filter/src/test/java/com/alibaba/otter/canal/filter/AviaterFilterTest.java

@@ -77,6 +77,28 @@ public class AviaterFilterTest {
         Assert.assertEquals(true, result);
     }
 
+    @Test
+    public void test_regex_with_quantifier_comma() {
+        // fix issue #5442: comma inside {n,m} quantifier should not split the pattern
+        AviaterRegexFilter filter = new AviaterRegexFilter("test\\.order_\\d{1,2}");
+        Assert.assertTrue(filter.filter("test.order_5"));
+        Assert.assertTrue(filter.filter("test.order_42"));
+        Assert.assertFalse(filter.filter("test.order_123"));
+        Assert.assertFalse(filter.filter("test.order_"));
+
+        // multiple patterns combined with quantifier should still split correctly
+        AviaterRegexFilter filter2 = new AviaterRegexFilter("test\\.order_\\d{1,2},test\\.user_\\d+");
+        Assert.assertTrue(filter2.filter("test.order_7"));
+        Assert.assertTrue(filter2.filter("test.user_99"));
+        Assert.assertFalse(filter2.filter("test.order_123"));
+
+        // open-ended quantifier {n,}
+        AviaterRegexFilter filter3 = new AviaterRegexFilter("test\\.t_\\d{2,}");
+        Assert.assertTrue(filter3.filter("test.t_12"));
+        Assert.assertTrue(filter3.filter("test.t_123"));
+        Assert.assertFalse(filter3.filter("test.t_1"));
+    }
+
     @Test
     public void testDisordered() {
         AviaterRegexFilter filter = new AviaterRegexFilter("u\\..*,uvw\\..*,uv\\..*,a\\.x,a\\.xyz,a\\.xy,abc\\.x,abc\\.xyz,abc\\.xy,ab\\.x,ab\\.xyz,ab\\.xy");

instance/core/pom.xml

@@ -3,7 +3,7 @@
 	<parent>
 		<groupId>com.alibaba.otter</groupId>
 		<artifactId>canal</artifactId>
-		<version>1.1.8</version>
+		<version>1.1.9-SNAPSHOT</version>
 		<relativePath>../../pom.xml</relativePath>
 	</parent>
 	<artifactId>canal.instance.core</artifactId>

instance/manager/pom.xml

@@ -3,7 +3,7 @@
 	<parent>
 		<groupId>com.alibaba.otter</groupId>
 		<artifactId>canal</artifactId>
-		<version>1.1.8</version>
+		<version>1.1.9-SNAPSHOT</version>
 		<relativePath>../../pom.xml</relativePath>
 	</parent>
 	<groupId>com.alibaba.otter</groupId>

instance/pom.xml

@@ -3,7 +3,7 @@
 	<parent>
 		<groupId>com.alibaba.otter</groupId>
 		<artifactId>canal</artifactId>
-		<version>1.1.8</version>
+		<version>1.1.9-SNAPSHOT</version>
 		<relativePath>../pom.xml</relativePath>
 	</parent>
 	<groupId>com.alibaba.otter</groupId>

instance/spring/pom.xml

@@ -3,7 +3,7 @@
 	<parent>
 		<groupId>com.alibaba.otter</groupId>
 		<artifactId>canal</artifactId>
-		<version>1.1.8</version>
+		<version>1.1.9-SNAPSHOT</version>
 		<relativePath>../../pom.xml</relativePath>
 	</parent>
 	<groupId>com.alibaba.otter</groupId>

instance/spring/src/main/java/com/alibaba/otter/canal/instance/spring/support/PropertyPlaceholderConfigurer.java

@@ -4,6 +4,7 @@ import java.util.ArrayList;
 import java.util.List;
 import java.util.Properties;
 
+import org.apache.commons.lang.StringUtils;
 import org.springframework.beans.factory.InitializingBean;
 import org.springframework.context.ResourceLoaderAware;
 import org.springframework.core.io.Resource;
@@ -126,20 +127,20 @@ public class PropertyPlaceholderConfigurer extends org.springframework.beans.fac
         }
 
         // 以threadlocal的为准覆盖file properties
-        if (propVal == null) {
+        if (StringUtils.isBlank(propVal)) {
             Properties localProperties = propertiesLocal.get();
             propVal = resolvePlaceholder(dp.placeholder, localProperties);
         }
 
-        if (propVal == null) {
+        if (StringUtils.isBlank(propVal)) {
             propVal = resolvePlaceholder(dp.placeholder, props);
         }
 
-        if (propVal == null && systemPropertiesMode == SYSTEM_PROPERTIES_MODE_FALLBACK) {
+        if (StringUtils.isBlank(propVal) && systemPropertiesMode == SYSTEM_PROPERTIES_MODE_FALLBACK) {
             propVal = resolveSystemProperty(dp.placeholder);
         }
 
-        if (propVal == null) {
+        if (StringUtils.isBlank(propVal)) {
             propVal = dp.defaultValue;
         }
 

meta/pom.xml

@@ -3,7 +3,7 @@
 	<parent>
 		<groupId>com.alibaba.otter</groupId>
 		<artifactId>canal</artifactId>
-		<version>1.1.8</version>
+		<version>1.1.9-SNAPSHOT</version>
 		<relativePath>../pom.xml</relativePath>
 	</parent>
 	<groupId>com.alibaba.otter</groupId>

parse/pom.xml

@@ -3,7 +3,7 @@
 	<parent>
 		<groupId>com.alibaba.otter</groupId>
 		<artifactId>canal</artifactId>
-		<version>1.1.8</version>
+		<version>1.1.9-SNAPSHOT</version>
 		<relativePath>../pom.xml</relativePath>
 	</parent>
 	<artifactId>canal.parse</artifactId>

parse/src/main/java/com/alibaba/otter/canal/parse/exception/ServerLogPurgedException.java

@@ -0,0 +1 @@
+package com.alibaba.otter.canal.parse.exception;

parse/src/main/java/com/alibaba/otter/canal/parse/inbound/mysql/LocalBinLogConnection.java

@@ -18,11 +18,7 @@ import com.alibaba.otter.canal.parse.inbound.ErosaConnection;
 import com.alibaba.otter.canal.parse.inbound.MultiStageCoprocessor;
 import com.alibaba.otter.canal.parse.inbound.SinkFunction;
 import com.alibaba.otter.canal.parse.inbound.mysql.local.BinLogFileQueue;
-import com.taobao.tddl.dbsync.binlog.FileLogFetcher;
-import com.taobao.tddl.dbsync.binlog.LogContext;
-import com.taobao.tddl.dbsync.binlog.LogDecoder;
-import com.taobao.tddl.dbsync.binlog.LogEvent;
-import com.taobao.tddl.dbsync.binlog.LogPosition;
+import com.taobao.tddl.dbsync.binlog.*;
 import com.taobao.tddl.dbsync.binlog.event.QueryLogEvent;
 
 /**
@@ -32,27 +28,29 @@ import com.taobao.tddl.dbsync.binlog.event.QueryLogEvent;
  */
 public class LocalBinLogConnection implements ErosaConnection {
 
-    private static final Logger logger     = LoggerFactory.getLogger(LocalBinLogConnection.class);
-    private BinLogFileQueue     binlogs    = null;
+    private static final Logger logger                       = LoggerFactory.getLogger(LocalBinLogConnection.class);
+    /** rdsOosMode 主从信息 */
+    private final Set<Long>     rdsOssMasterSlaveInfo        = new HashSet<>(4);
+    private BinLogFileQueue     binlogs                      = null;
     private boolean             needWait;
     private String              directory;
-    private int                 bufferSize = 16 * 1024;
-    private boolean             running    = false;
+    private int                 bufferSize                   = 16 * 1024;
+    private boolean             running                      = false;
     private long                serverId;
-
     /** rdsOosMode binlog 的 serverId 是两个 */
-    private boolean             isRdsOssMode = false;
-
-    /** rdsOosMode 主从信息 */
-    private final Set<Long> rdsOssMasterSlaveInfo = new HashSet<>(4);
-
-    private boolean firstUpdateRdsOssMasterSlave = true;
+    private boolean             isRdsOssMode                 = false;
+    private boolean             firstUpdateRdsOssMasterSlave = true;
 
     private FileParserListener  parserListener;
 
     public LocalBinLogConnection(){
     }
 
+    public LocalBinLogConnection(String directory, boolean needWait){
+        this.needWait = needWait;
+        this.directory = directory;
+    }
+
     public boolean isRdsOssMode() {
         return isRdsOssMode;
     }
@@ -61,11 +59,6 @@ public class LocalBinLogConnection implements ErosaConnection {
         isRdsOssMode = rdsOssMode;
     }
 
-    public LocalBinLogConnection(String directory, boolean needWait){
-        this.needWait = needWait;
-        this.directory = directory;
-    }
-
     @Override
     public void connect() throws IOException {
         if (this.binlogs == null) {
@@ -117,8 +110,8 @@ public class LocalBinLogConnection implements ErosaConnection {
                     List<LogEvent> iterateEvents = decoder.processIterateDecode(event, context);
                     if (!iterateEvents.isEmpty()) {
                         // 处理compress event
-                        for(LogEvent itEvent : iterateEvents) {
-                            if (!func.sink(event)) {
+                        for (LogEvent itEvent : iterateEvents) {
+                            if (!func.sink(itEvent)) {
                                 needContinue = false;
                                 break;
                             }
@@ -158,9 +151,10 @@ public class LocalBinLogConnection implements ErosaConnection {
     }
 
     /**
-     * 1. 非 rdsOos 模式下需要要校验 serverId 是否一致 防止解析其他实例的 binlog
-     * 2. rdsOos 高可用模式下解析 binlog 会有两个 serverId,分别对应着主从节点 binlog解析出来的 serverId
-     * 主从的关系可能会变, 但是 serverId一直都会是这两个 serverId
+     * 1. 非 rdsOos 模式下需要要校验 serverId 是否一致 防止解析其他实例的 binlog <br/>
+     * 2. rdsOos 高可用模式下解析 binlog
+     * 会有两个 serverId,分别对应着主从节点 binlog解析出来的 serverId 主从的关系可能会变, 但是 serverId一直都会是这两个
+     * serverId
      *
      * @param event
      */
@@ -210,7 +204,8 @@ public class LocalBinLogConnection implements ErosaConnection {
                     if (event != null) {
                         checkServerId(event);
 
-                        if (event.getWhen() > timestampSeconds) {
+                        // 闭区间，防止丢开始那一秒的数据
+                        if (event.getWhen() >= timestampSeconds) {
                             break;
                         }
 

parse/src/main/java/com/alibaba/otter/canal/parse/inbound/mysql/dbsync/DirectLogFetcher.java

@@ -5,10 +5,12 @@ import java.io.InterruptedIOException;
 import java.net.SocketTimeoutException;
 import java.nio.channels.ClosedByInterruptException;
 
+import org.apache.commons.lang.StringUtils;
 import org.slf4j.Logger;
 import org.slf4j.LoggerFactory;
 
 import com.alibaba.otter.canal.parse.driver.mysql.socket.SocketChannel;
+import com.alibaba.otter.canal.parse.exception.ServerLogPurgedException;
 import com.taobao.tddl.dbsync.binlog.LogFetcher;
 
 /**
@@ -99,6 +101,13 @@ public class DirectLogFetcher extends LogFetcher {
                     final int errno = getInt16();
                     String sqlstate = forward(1).getFixString(SQLSTATE_LENGTH);
                     String errmsg = getFixString(limit - position);
+                    if (StringUtils.containsIgnoreCase(errmsg, "not find first log file name")
+                        || StringUtils.containsIgnoreCase(errmsg, "purged binary logs")) {
+                        // 开始 dump 后，server 位点过期，DUMP 和 DUMP_GTID 两种错误信息
+                        throw new ServerLogPurgedException(
+                            " errno = " + errno + ", sqlstate = " + sqlstate + " errmsg = " + errmsg);
+                    }
+
                     throw new IOException("Received error packet:" + " errno = " + errno + ", sqlstate = " + sqlstate
                                           + " errmsg = " + errmsg);
                 } else if (mark == 254) {

parse/src/main/java/com/alibaba/otter/canal/parse/inbound/mysql/dbsync/TableMetaCache.java

@@ -5,6 +5,7 @@ import java.util.ArrayList;
 import java.util.HashMap;
 import java.util.List;
 import java.util.Map;
+import java.util.concurrent.ConcurrentMap;
 
 import org.apache.commons.lang.StringUtils;
 
@@ -43,69 +44,35 @@ public class TableMetaCache {
     private boolean                         isOnTSDB       = false;
 
     private TableMetaTSDB                   tableMetaTSDB;
+
     // 第一层tableId,第二层schema.table,解决tableId重复，对应多张表
-    private LoadingCache<String, TableMeta> tableMetaDB;
+    private LoadingCache<String, TableMeta> tableMetaCache;
 
     public TableMetaCache(MysqlConnection con, TableMetaTSDB tableMetaTSDB){
         this.connection = con;
         this.tableMetaTSDB = tableMetaTSDB;
         // 如果持久存储的表结构为空，从db里面获取下
         if (tableMetaTSDB == null) {
-            this.tableMetaDB = CacheBuilder.newBuilder().build(new CacheLoader<String, TableMeta>() {
-
-                @Override
-                public TableMeta load(String name) throws Exception {
-                    try {
-                        return getTableMetaByDB(name);
-                    } catch (Throwable e) {
-                        // 尝试做一次retry操作
-                        try {
-                            connection.reconnect();
-                            return getTableMetaByDB(name);
-                        } catch (IOException e1) {
-                            throw new CanalParseException("fetch failed by table meta:" + name, e1);
-                        }
-                    }
-                }
-
-            });
+            this.tableMetaCache = createLocalCache();
         } else {
             isOnTSDB = true;
         }
-
         try {
             ResultSetPacket packet = connection.query("show global variables  like 'rds\\_%'");
-            if (packet.getFieldValues().size() > 0) {
-                isOnRDS = true;
-            }
+            // if (packet.getFieldValues().size() > 0) {
+            isOnRDS = packet.getFieldValues().size() > 0;
+            // }
         } catch (IOException e) {
         }
-
         try {
             ResultSetPacket packet = connection.query("show global variables  like 'polarx\\_%'");
-            if (packet.getFieldValues().size() > 0) {
-                isOnPolarX = true;
-            }
+            // if (packet.getFieldValues().size() > 0) {
+            isOnPolarX = packet.getFieldValues().size() > 0;
+            // }
         } catch (IOException e) {
         }
     }
 
-    private synchronized TableMeta getTableMetaByDB(String fullname) throws IOException {
-        try {
-            ResultSetPacket packet = connection.query("show create table " + fullname);
-            String[] names = StringUtils.split(fullname, "`.`");
-            String schema = names[0];
-            String table = names[1].substring(0, names[1].length());
-            return new TableMeta(schema, table, parseTableMeta(schema, table, packet));
-        } catch (Throwable e) { // fallback to desc table
-            ResultSetPacket packet = connection.query("desc " + fullname);
-            String[] names = StringUtils.split(fullname, "`.`");
-            String schema = names[0];
-            String table = names[1].substring(0, names[1].length());
-            return new TableMeta(schema, table, parseTableMetaByDesc(packet));
-        }
-    }
-
     public static List<FieldMeta> parseTableMeta(String schema, String table, ResultSetPacket packet) {
         if (packet.getFieldValues().size() > 1) {
             String createDDL = packet.getFieldValues().get(1);
@@ -122,86 +89,139 @@ public class TableMetaCache {
      * 处理desc table的结果
      */
     public static List<FieldMeta> parseTableMetaByDesc(ResultSetPacket packet) {
-        Map<String, Integer> nameMaps = new HashMap<>(6, 1f);
+        List<FieldPacket> fieldPackets = packet.getFieldDescriptors();
+        int size = fieldPackets.size();
+        Map<String, Integer> nameMaps = new HashMap<>(size);
         int index = 0;
-        for (FieldPacket fieldPacket : packet.getFieldDescriptors()) {
+        for (FieldPacket fieldPacket : fieldPackets) {
             nameMaps.put(StringUtils.lowerCase(fieldPacket.getName()), index++);
         }
 
-        int size = packet.getFieldDescriptors().size();
-        int count = packet.getFieldValues().size() / packet.getFieldDescriptors().size();
-        List<FieldMeta> result = new ArrayList<>();
+        List<String> fieldValues = packet.getFieldValues();
+        int count = fieldValues.size() / size;
+        List<FieldMeta> result = new ArrayList<>(count);
         for (int i = 0; i < count; i++) {
             FieldMeta meta = new FieldMeta();
             // 做一个优化，使用String.intern()，共享String对象，减少内存使用
-            meta.setColumnName(packet.getFieldValues().get(nameMaps.get(COLUMN_NAME) + i * size).intern());
-            meta.setColumnType(packet.getFieldValues().get(nameMaps.get(COLUMN_TYPE) + i * size));
-            meta.setNullable(StringUtils.equalsIgnoreCase(packet.getFieldValues().get(nameMaps.get(IS_NULLABLE) + i
-                                                                                      * size),
-                "YES"));
-            meta.setKey("PRI".equalsIgnoreCase(packet.getFieldValues().get(nameMaps.get(COLUMN_KEY) + i * size)));
-            meta.setUnique("UNI".equalsIgnoreCase(packet.getFieldValues().get(nameMaps.get(COLUMN_KEY) + i * size)));
+            meta.setColumnName(fieldValues.get(nameMaps.get(COLUMN_NAME) + i * size).intern());
+            meta.setColumnType(fieldValues.get(nameMaps.get(COLUMN_TYPE) + i * size));
+            meta.setNullable(
+                StringUtils.equalsIgnoreCase(fieldValues.get(nameMaps.get(IS_NULLABLE) + i * size), "YES"));
+            meta.setKey("PRI".equalsIgnoreCase(fieldValues.get(nameMaps.get(COLUMN_KEY) + i * size)));
+            meta.setUnique("UNI".equalsIgnoreCase(fieldValues.get(nameMaps.get(COLUMN_KEY) + i * size)));
             // 特殊处理引号
-            meta.setDefaultValue(DruidDdlParser.unescapeQuotaName(packet.getFieldValues()
-                .get(nameMaps.get(COLUMN_DEFAULT) + i * size)));
-            meta.setExtra(packet.getFieldValues().get(nameMaps.get(EXTRA) + i * size));
-
+            meta.setDefaultValue(
+                DruidDdlParser.unescapeQuotaName(fieldValues.get(nameMaps.get(COLUMN_DEFAULT) + i * size)));
+            meta.setExtra(fieldValues.get(nameMaps.get(EXTRA) + i * size));
             result.add(meta);
         }
-
         return result;
     }
 
+    private LoadingCache<String, TableMeta> createLocalCache() {
+        return CacheBuilder.newBuilder().build(new CacheLoader<String, TableMeta>() {
+
+            @Override
+            public TableMeta load(String name) throws Exception {
+                try {
+                    return getTableMetaByDB(name);
+                } catch (Throwable e) {
+                    // 尝试做一次retry操作
+                    try {
+                        connection.reconnect();
+                        return getTableMetaByDB(name);
+                    } catch (IOException e1) {
+                        throw new CanalParseException("fetch table meta failed. table: " + name, e1);
+                    }
+                }
+            }
+        });
+    }
+
+    private TableMeta getTableMetaByDB(String fullname) throws IOException {
+        boolean showCreateTable = true;
+        ResultSetPacket packet = null;
+        synchronized (this) {
+            try {
+                packet = connection.query("show create table " + fullname);
+            } catch (Exception ex) {
+                showCreateTable = false;
+                packet = connection.query("desc " + fullname);
+            }
+        }
+        String[] names = StringUtils.split(fullname, "`.`");
+        String schema = names[0];
+        String table = names[1].substring(0, names[1].length());
+        List<FieldMeta> fieldMetas = showCreateTable ? parseTableMeta(schema, table, packet) : parseTableMetaByDesc(
+            packet);
+        return new TableMeta(schema, table, fieldMetas);
+    }
+
+    // This method is unused
     public TableMeta getTableMeta(String schema, String table) {
         return getTableMeta(schema, table, true);
     }
 
+    // This method is unused
     public TableMeta getTableMeta(String schema, String table, boolean useCache) {
+        String fullName = getFullName(schema, table);
         if (!useCache) {
-            tableMetaDB.invalidate(getFullName(schema, table));
+            // tableMetaCache.invalidate(getFullName(schema, table));
+            tableMetaCache.invalidate(fullName);
         }
-
-        return tableMetaDB.getUnchecked(getFullName(schema, table));
+        // return tableMetaCache.getUnchecked(getFullName(schema, table));
+        return tableMetaCache.getUnchecked(fullName);
     }
 
+    // This method is unused
     public TableMeta getTableMeta(String schema, String table, EntryPosition position) {
         return getTableMeta(schema, table, true, position);
     }
 
-    public synchronized TableMeta getTableMeta(String schema, String table, boolean useCache, EntryPosition position) {
+    public TableMeta getTableMeta(String schema, String table, boolean useCache, EntryPosition position) {
         TableMeta tableMeta = null;
         if (tableMetaTSDB != null) {
             tableMeta = tableMetaTSDB.find(schema, table);
             if (tableMeta == null) {
                 // 因为条件变化，可能第一次的tableMeta没取到，需要从db获取一次，并记录到snapshot中
                 String fullName = getFullName(schema, table);
-                ResultSetPacket packet = null;
-                String createDDL = null;
                 try {
-                    try {
-                        packet = connection.query("show create table " + fullName);
-                    } catch (Exception e) {
-                        // 尝试做一次retry操作
-                        connection.reconnect();
-                        packet = connection.query("show create table " + fullName);
+                    synchronized (this) {
+                        tableMeta = tableMetaTSDB.find(schema, table);
+                        if (tableMeta != null) {
+                            return tableMeta;
+                        }
+
+                        ResultSetPacket packet = null;
+                        try {
+                            packet = connection.query("show create table " + fullName);
+                        } catch (Exception e) {
+                            // 尝试做一次retry操作
+                            connection.reconnect();
+                            packet = connection.query("show create table " + fullName);
+                        }
+
+                        String createDDL = null;
+                        if (packet.getFieldValues().size() > 0) {
+                            createDDL = packet.getFieldValues().get(1);
+                        }
+                        // 强制覆盖掉内存值
+                        tableMetaTSDB.apply(position, schema, createDDL, "first");
+                        tableMeta = tableMetaTSDB.find(schema, table);
                     }
-                    if (packet.getFieldValues().size() > 0) {
-                        createDDL = packet.getFieldValues().get(1);
-                    }
-                    // 强制覆盖掉内存值
-                    tableMetaTSDB.apply(position, schema, createDDL, "first");
-                    tableMeta = tableMetaTSDB.find(schema, table);
                 } catch (IOException e) {
                     throw new CanalParseException("fetch failed by table meta:" + fullName, e);
                 }
             }
             return tableMeta;
         } else {
+            String fullName = getFullName(schema, table);
             if (!useCache) {
-                tableMetaDB.invalidate(getFullName(schema, table));
+                // tableMetaCache.invalidate(getFullName(schema, table));
+                tableMetaCache.invalidate(fullName);
             }
-
-            return tableMetaDB.getUnchecked(getFullName(schema, table));
+            // return tableMetaCache.getUnchecked(getFullName(schema, table));
+            return tableMetaCache.getUnchecked(fullName);
         }
     }
 
@@ -209,7 +229,7 @@ public class TableMetaCache {
         if (tableMetaTSDB != null) {
             // tsdb不需要做,会基于ddl sql自动清理
         } else {
-            tableMetaDB.invalidate(getFullName(schema, table));
+            tableMetaCache.invalidate(getFullName(schema, table));
         }
     }
 
@@ -217,10 +237,13 @@ public class TableMetaCache {
         if (tableMetaTSDB != null) {
             // tsdb不需要做,会基于ddl sql自动清理
         } else {
-            for (String name : tableMetaDB.asMap().keySet()) {
-                if (StringUtils.startsWithIgnoreCase(name, schema + ".")) {
-                    // removeNames.add(name);
-                    tableMetaDB.invalidate(name);
+            ConcurrentMap<String, TableMeta> map = tableMetaCache.asMap();
+            if (!map.isEmpty()) {
+                for (String name : map.keySet()) {
+                    if (StringUtils.startsWithIgnoreCase(name, schema + ".")) {
+                        // removeNames.add(name);
+                        tableMetaCache.invalidate(name);
+                    }
                 }
             }
         }
@@ -230,7 +253,7 @@ public class TableMetaCache {
         if (tableMetaTSDB != null) {
             // tsdb不需要做,会基于ddl sql自动清理
         } else {
-            tableMetaDB.invalidateAll();
+            tableMetaCache.invalidateAll();
         }
     }
 
@@ -252,13 +275,13 @@ public class TableMetaCache {
     }
 
     private String getFullName(String schema, String table) {
-        StringBuilder builder = new StringBuilder();
+        StringBuilder builder = new StringBuilder(64);
         return builder.append('`')
             .append(schema)
             .append('`')
             .append('.')
             .append('`')
-            .append(StringUtils.replace(table,"`","``"))
+            .append(StringUtils.replace(table, "`", "``"))
             .append('`')
             .toString();
     }
@@ -286,5 +309,4 @@ public class TableMetaCache {
     public void setOnPolarX(boolean isOnPolarX) {
         this.isOnPolarX = isOnPolarX;
     }
-
 }

parse/src/main/java/com/alibaba/otter/canal/parse/inbound/mysql/rds/BinlogDownloadQueue.java

@@ -189,19 +189,23 @@ public class BinlogDownloadQueue {
             httpClient = HttpClientBuilder.create().setMaxConnPerRoute(50).setMaxConnTotal(100).build();
         }
 
-        HttpGet httpGet = new HttpGet(downloadLink);
-        RequestConfig requestConfig = RequestConfig.custom()
-            .setConnectTimeout(TIMEOUT)
-            .setConnectionRequestTimeout(TIMEOUT)
-            .setSocketTimeout(TIMEOUT)
-            .build();
-        httpGet.setConfig(requestConfig);
-        HttpResponse response = httpClient.execute(httpGet);
-        int statusCode = response.getStatusLine().getStatusCode();
-        if (statusCode != HttpResponseStatus.OK.code()) {
-            throw new RuntimeException("download failed , url:" + downloadLink + " , statusCode:" + statusCode);
+        try {
+            HttpGet httpGet = new HttpGet(downloadLink);
+            RequestConfig requestConfig = RequestConfig.custom()
+                .setConnectTimeout(TIMEOUT)
+                .setConnectionRequestTimeout(TIMEOUT)
+                .setSocketTimeout(TIMEOUT)
+                .build();
+            httpGet.setConfig(requestConfig);
+            HttpResponse response = httpClient.execute(httpGet);
+            int statusCode = response.getStatusLine().getStatusCode();
+            if (statusCode != HttpResponseStatus.OK.code()) {
+                throw new RuntimeException("download failed , url:" + downloadLink + " , statusCode:" + statusCode);
+            }
+            saveFile(new File(destDir), "mysql-bin." + fileName, response);
+        } finally {
+            httpClient.close();
         }
-        saveFile(new File(destDir), "mysql-bin." + fileName, response);
     }
 
     private static void saveFile(File parentFile, String fileName, HttpResponse response) throws IOException {

parse/src/main/java/com/alibaba/otter/canal/parse/inbound/mysql/rds/RdsBinlogEventParserProxy.java

@@ -6,6 +6,7 @@ import java.util.concurrent.Executors;
 import org.apache.commons.lang.StringUtils;
 
 import com.alibaba.otter.canal.parse.exception.PositionNotFoundException;
+import com.alibaba.otter.canal.parse.exception.ServerLogPurgedException;
 import com.alibaba.otter.canal.parse.inbound.ParserExceptionHandler;
 import com.alibaba.otter.canal.parse.inbound.mysql.MysqlEventParser;
 
@@ -92,7 +93,7 @@ public class RdsBinlogEventParserProxy extends MysqlEventParser {
     }
 
     private void handleMysqlParserException(Throwable throwable) {
-        if (throwable instanceof PositionNotFoundException) {
+        if (throwable instanceof PositionNotFoundException || throwable instanceof ServerLogPurgedException) {
             logger.info("remove rds not found position, try download rds binlog!");
             executorService.execute(() -> {
                 try {

parse/src/main/java/com/alibaba/otter/canal/parse/inbound/mysql/tsdb/DatabaseTableMeta.java

@@ -13,9 +13,7 @@ import java.util.concurrent.atomic.AtomicBoolean;
 import java.util.concurrent.locks.ReadWriteLock;
 import java.util.concurrent.locks.ReentrantReadWriteLock;
 import java.util.regex.Pattern;
-import com.alibaba.otter.canal.parse.driver.mysql.packets.server.FieldPacket;
-import com.alibaba.polardbx.druid.sql.repository.Schema;
-import org.apache.commons.beanutils.BeanUtils;
+
 import org.apache.commons.lang.ObjectUtils;
 import org.apache.commons.lang.StringUtils;
 import org.slf4j.Logger;
@@ -24,6 +22,7 @@ import org.slf4j.MDC;
 
 import com.alibaba.fastjson2.JSON;
 import com.alibaba.fastjson2.JSONObject;
+import com.alibaba.fastjson2.JSONWriter;
 import com.alibaba.otter.canal.filter.CanalEventFilter;
 import com.alibaba.otter.canal.parse.driver.mysql.packets.server.FieldPacket;
 import com.alibaba.otter.canal.parse.driver.mysql.packets.server.ResultSetPacket;
@@ -39,6 +38,7 @@ import com.alibaba.otter.canal.parse.inbound.mysql.tsdb.dao.MetaHistoryDO;
 import com.alibaba.otter.canal.parse.inbound.mysql.tsdb.dao.MetaSnapshotDAO;
 import com.alibaba.otter.canal.parse.inbound.mysql.tsdb.dao.MetaSnapshotDO;
 import com.alibaba.otter.canal.protocol.position.EntryPosition;
+import com.alibaba.polardbx.druid.sql.repository.Schema;
 
 /**
  * 基于db远程管理 see internal class: CanalTableMeta , ConsoleTableMetaTSDB
@@ -51,7 +51,8 @@ public class DatabaseTableMeta implements TableMetaTSDB {
     public static final EntryPosition       INIT_POSITION       = new EntryPosition("0", 0L, -2L, -1L);
     private static Logger                   logger              = LoggerFactory.getLogger(DatabaseTableMeta.class);
     private static Pattern                  pattern             = Pattern.compile("Duplicate entry '.*' for key '*'");
-    private static Pattern                  h2Pattern           = Pattern.compile("Unique index or primary key violation");
+    private static Pattern                  h2Pattern           = Pattern
+        .compile("Unique index or primary key violation");
     private static ScheduledExecutorService scheduler           = Executors.newSingleThreadScheduledExecutor(r -> {
                                                                     Thread thread = new Thread(r,
                                                                         "[scheduler-table-meta-snapshot]");
@@ -62,7 +63,7 @@ public class DatabaseTableMeta implements TableMetaTSDB {
     private AtomicBoolean                   initialized         = new AtomicBoolean(false);
     private String                          destination;
     private MemoryTableMeta                 memoryTableMeta;
-    private volatile MysqlConnection        connection;                                                                    // 查询meta信息的链接
+    private volatile MysqlConnection        connection;                                                               // 查询meta信息的链接
     private CanalEventFilter                filter;
     private CanalEventFilter                blackFilter;
     private Map<String, List<String>>       fieldFilterMap      = new HashMap<>();
@@ -79,6 +80,143 @@ public class DatabaseTableMeta implements TableMetaTSDB {
 
     }
 
+    public static boolean compareTableMeta(TableMeta source, TableMeta target) {
+        if (!StringUtils.equalsIgnoreCase(source.getSchema(), target.getSchema())) {
+            return false;
+        }
+
+        if (!StringUtils.equalsIgnoreCase(source.getTable(), target.getTable())) {
+            return false;
+        }
+
+        List<FieldMeta> sourceFields = source.getFields();
+        List<FieldMeta> targetFields = target.getFields();
+        if (sourceFields.size() != targetFields.size()) {
+            return false;
+        }
+
+        /**
+         * MySQL DDL的一些默认行为:
+         *
+         * <pre>
+         * 1. Timestamp类型的列在第一次添加时，未指定默认值会默认为CURRENT_TIMESTAMP ON UPDATE CURRENT_TIMESTAMP
+         * 2. Timestamp类型的列在第二次时，必须指定默认值
+         * 3. BLOB和TEXT类型不存在NULL、NOT NULL属性
+         * 4. 部分数据类型是synonyms，实际show create table时会转成对应类型
+         * 5. 非BLOB和TEXT类型在默认未指定NULL、NOT NULL时，默认default null
+         * 6. 在列变更时，不仅变更列名数据类型，同时各个约束中列名也会变更，同时如果约束中包含key length，则变更后的数据类型不应违背key length的约束（有长度的应大于key length；BLOB、TEXT应有key length；可以在存在key length情况下变更为无key length的数据类型，约束中key length忽略；等等）
+         * 7. 字符集每列（char类、eumn、set）默认保存，指定使用指定的，未指定使用表默认的，不受修改表默认字符集而改变，同表默认时，字符集显示省略
+         * 8. 新建表默认innodb引擎，latin1字符集
+         * 9. BLOB、TEXT会根据给定长度自动转换为对应的TINY、MEDIUM，LONG类型，长度和字符集也有关
+         * 10. unique约束在没有指定索引名是非幂等的，会自动以约束索引第一个列名称命名，同时以_2,_3这种形式添加后缀
+         * </pre>
+         */
+
+        for (int i = 0; i < sourceFields.size(); i++) {
+            FieldMeta sourceField = sourceFields.get(i);
+            FieldMeta targetField = targetFields.get(i);
+            if (!StringUtils.equalsIgnoreCase(sourceField.getColumnName(), targetField.getColumnName())) {
+                return false;
+            }
+
+            // if (!StringUtils.equalsIgnoreCase(sourceField.getColumnType(),
+            // targetField.getColumnType())) {
+            // return false;
+            // }
+
+            // https://github.com/alibaba/canal/issues/1100
+            // 支持一下 int vs int(10)
+            if ((sourceField.isUnsigned() && !targetField.isUnsigned())
+                || (!sourceField.isUnsigned() && targetField.isUnsigned())) {
+                return false;
+            }
+
+            String sourceColumnType = StringUtils.removeEndIgnoreCase(sourceField.getColumnType(), "zerofill").trim();
+            String targetColumnType = StringUtils.removeEndIgnoreCase(targetField.getColumnType(), "zerofill").trim();
+
+            String sign = sourceField.isUnsigned() ? "unsigned" : "signed";
+            sourceColumnType = StringUtils.removeEndIgnoreCase(sourceColumnType, sign).trim();
+            targetColumnType = StringUtils.removeEndIgnoreCase(targetColumnType, sign).trim();
+
+            boolean columnTypeCompare = false;
+            columnTypeCompare |= StringUtils.containsIgnoreCase(sourceColumnType, targetColumnType);
+            columnTypeCompare |= StringUtils.containsIgnoreCase(targetColumnType, sourceColumnType);
+            if (!columnTypeCompare) {
+                // 去掉精度参数再对比一次
+                sourceColumnType = synonymsType(StringUtils.substringBefore(sourceColumnType, "(")).trim();
+                targetColumnType = synonymsType(StringUtils.substringBefore(targetColumnType, "(")).trim();
+                columnTypeCompare |= StringUtils.containsIgnoreCase(sourceColumnType, targetColumnType);
+                columnTypeCompare |= StringUtils.containsIgnoreCase(targetColumnType, sourceColumnType);
+                if (!columnTypeCompare) {
+                    return false;
+                }
+            }
+
+            // if (!StringUtils.equalsIgnoreCase(sourceField.getDefaultValue(),
+            // targetField.getDefaultValue())) {
+            // return false;
+            // }
+
+            // BLOB, TEXT, GEOMETRY or JSON默认都是nullable，可以忽略比较，但比较了也是对齐
+            if (StringUtils.containsIgnoreCase(sourceColumnType, "timestamp")
+                || StringUtils.containsIgnoreCase(targetColumnType, "timestamp")) {
+                // timestamp可能会加default current_timestamp默认值,忽略对比nullable
+            } else {
+                if (sourceField.isNullable() != targetField.isNullable()) {
+                    return false;
+                }
+            }
+
+            // mysql会有一种处理,针对show create只有uk没有pk时，会在desc默认将uk当做pk
+            boolean isSourcePkOrUk = sourceField.isKey() || sourceField.isUnique();
+            boolean isTargetPkOrUk = targetField.isKey() || targetField.isUnique();
+            if (isSourcePkOrUk != isTargetPkOrUk) {
+                return false;
+            }
+        }
+
+        return true;
+    }
+
+    /**
+     * <pre>
+     * synonyms处理
+     * 1. BOOL/BOOLEAN => TINYINT
+     * 2. DEC/NUMERIC/FIXED => DECIMAL
+     * 3. INTEGER => INT
+     * </pre>
+     *
+     * @param originType
+     * @return
+     */
+    private static String synonymsType(String originType) {
+        if (StringUtils.equalsIgnoreCase(originType, "bool") || StringUtils.equalsIgnoreCase(originType, "boolean")) {
+            return "tinyint";
+        } else
+            if (StringUtils.equalsIgnoreCase(originType, "dec") || StringUtils.equalsIgnoreCase(originType, "numeric")
+                || StringUtils.equalsIgnoreCase(originType, "fixed")) {
+                    return "decimal";
+                } else if (StringUtils.equalsIgnoreCase(originType, "integer")) {
+                    return "int";
+                } else if (StringUtils.equalsIgnoreCase(originType, "real")
+                           || StringUtils.equalsIgnoreCase(originType, "double precision")) {
+                               return "double";
+                           }
+
+        // BLOB、TEXT会根据给定长度自动转换为对应的TINY、MEDIUM，LONG类型，长度和字符集也有关，统一按照blob对比
+        if (StringUtils.equalsIgnoreCase(originType, "tinyblob")
+            || StringUtils.equalsIgnoreCase(originType, "mediumblob")
+            || StringUtils.equalsIgnoreCase(originType, "longblob")) {
+            return "blob";
+        } else if (StringUtils.equalsIgnoreCase(originType, "tinytext")
+                   || StringUtils.equalsIgnoreCase(originType, "mediumtext")
+                   || StringUtils.equalsIgnoreCase(originType, "longtext")) {
+                       return "text";
+                   }
+
+        return originType;
+    }
+
     @Override
     public boolean init(final String destination) {
         if (initialized.compareAndSet(false, true)) {
@@ -120,8 +258,9 @@ public class DatabaseTableMeta implements TableMetaTSDB {
             try {
                 connection.disconnect();
             } catch (IOException e) {
-                logger.error("ERROR # disconnect meta connection for address:{}", connection.getConnector()
-                    .getAddress(), e);
+                logger.error("ERROR # disconnect meta connection for address:{}",
+                    connection.getConnector().getAddress(),
+                    e);
             }
         }
 
@@ -161,6 +300,7 @@ public class DatabaseTableMeta implements TableMetaTSDB {
     @Override
     public boolean rollback(EntryPosition position) {
         // 每次rollback需要重新构建一次memory data
+        // FIXME The global MemoryTableMeta may cause OOM when it store too many tables
         this.memoryTableMeta = new MemoryTableMeta();
         boolean flag = false;
         EntryPosition snapshotPosition = buildMemFromSnapshot(position);
@@ -233,7 +373,7 @@ public class DatabaseTableMeta implements TableMetaTSDB {
 
                 StringBuilder sql = new StringBuilder();
                 for (String table : tables) {
-                    sql.append("show create table `" + schema + "`.`" + StringUtils.replace(table,"`","``") + "`;");
+                    sql.append("show create table `" + schema + "`.`" + StringUtils.replace(table, "`", "``") + "`;");
                 }
 
                 List<ResultSetPacket> packets = connection.queryMulti(sql.toString());
@@ -369,7 +509,7 @@ public class DatabaseTableMeta implements TableMetaTSDB {
             snapshotDO.setBinlogOffest(position.getPosition());
             snapshotDO.setBinlogMasterId(String.valueOf(position.getServerId()));
             snapshotDO.setBinlogTimestamp(position.getTimestamp());
-            snapshotDO.setData(JSON.toJSONString(schemaDdls));
+            snapshotDO.setData(JSON.toJSONString(schemaDdls, JSONWriter.Feature.LargeObject));
             try {
                 metaSnapshotDAO.insert(snapshotDO);
             } catch (Throwable e) {
@@ -465,9 +605,8 @@ public class DatabaseTableMeta implements TableMetaTSDB {
 
     private boolean applyHistoryOnMemory(EntryPosition position, EntryPosition rollbackPosition) {
         try {
-            List<MetaHistoryDO> metaHistoryDOList = metaHistoryDAO.findByTimestamp(destination,
-                position.getTimestamp(),
-                rollbackPosition.getTimestamp());
+            List<MetaHistoryDO> metaHistoryDOList = metaHistoryDAO
+                .findByTimestamp(destination, position.getTimestamp(), rollbackPosition.getTimestamp());
             if (metaHistoryDOList == null) {
                 return true;
             }
@@ -489,8 +628,8 @@ public class DatabaseTableMeta implements TableMetaTSDB {
                     continue;
                 } else if (rollbackPosition.getServerId().equals(snapshotPosition.getServerId())
                            && snapshotPosition.compareTo(rollbackPosition) > 0) {
-                    continue;
-                }
+                               continue;
+                           }
 
                 // 记录到内存
                 if (!memoryTableMeta.apply(snapshotPosition, useSchema, sqlData, null)) {
@@ -514,156 +653,18 @@ public class DatabaseTableMeta implements TableMetaTSDB {
 
     private String getFullName(String schema, String table) {
         StringBuilder builder = new StringBuilder();
-        return builder.append(structureSchema(schema)).append('.').append('`').append(StringUtils.replace(table,"`","``")).append('`').toString();
-    }
-
-    public static boolean compareTableMeta(TableMeta source, TableMeta target) {
-        if (!StringUtils.equalsIgnoreCase(source.getSchema(), target.getSchema())) {
-            return false;
-        }
-
-        if (!StringUtils.equalsIgnoreCase(source.getTable(), target.getTable())) {
-            return false;
-        }
-
-        List<FieldMeta> sourceFields = source.getFields();
-        List<FieldMeta> targetFields = target.getFields();
-        if (sourceFields.size() != targetFields.size()) {
-            return false;
-        }
-
-        /**
-         * MySQL DDL的一些默认行为:
-         * 
-         * <pre>
-         * 1. Timestamp类型的列在第一次添加时，未指定默认值会默认为CURRENT_TIMESTAMP ON UPDATE CURRENT_TIMESTAMP
-         * 2. Timestamp类型的列在第二次时，必须指定默认值
-         * 3. BLOB和TEXT类型不存在NULL、NOT NULL属性
-         * 4. 部分数据类型是synonyms，实际show create table时会转成对应类型
-         * 5. 非BLOB和TEXT类型在默认未指定NULL、NOT NULL时，默认default null
-         * 6. 在列变更时，不仅变更列名数据类型，同时各个约束中列名也会变更，同时如果约束中包含key length，则变更后的数据类型不应违背key length的约束（有长度的应大于key length；BLOB、TEXT应有key length；可以在存在key length情况下变更为无key length的数据类型，约束中key length忽略；等等）
-         * 7. 字符集每列（char类、eumn、set）默认保存，指定使用指定的，未指定使用表默认的，不受修改表默认字符集而改变，同表默认时，字符集显示省略
-         * 8. 新建表默认innodb引擎，latin1字符集
-         * 9. BLOB、TEXT会根据给定长度自动转换为对应的TINY、MEDIUM，LONG类型，长度和字符集也有关
-         * 10. unique约束在没有指定索引名是非幂等的，会自动以约束索引第一个列名称命名，同时以_2,_3这种形式添加后缀
-         * </pre>
-         */
-
-        for (int i = 0; i < sourceFields.size(); i++) {
-            FieldMeta sourceField = sourceFields.get(i);
-            FieldMeta targetField = targetFields.get(i);
-            if (!StringUtils.equalsIgnoreCase(sourceField.getColumnName(), targetField.getColumnName())) {
-                return false;
-            }
-
-            // if (!StringUtils.equalsIgnoreCase(sourceField.getColumnType(),
-            // targetField.getColumnType())) {
-            // return false;
-            // }
-
-            // https://github.com/alibaba/canal/issues/1100
-            // 支持一下 int vs int(10)
-            if ((sourceField.isUnsigned() && !targetField.isUnsigned())
-                || (!sourceField.isUnsigned() && targetField.isUnsigned())) {
-                return false;
-            }
-
-            String sourceColumnType = StringUtils.removeEndIgnoreCase(sourceField.getColumnType(), "zerofill").trim();
-            String targetColumnType = StringUtils.removeEndIgnoreCase(targetField.getColumnType(), "zerofill").trim();
-
-            String sign = sourceField.isUnsigned() ? "unsigned" : "signed";
-            sourceColumnType = StringUtils.removeEndIgnoreCase(sourceColumnType, sign).trim();
-            targetColumnType = StringUtils.removeEndIgnoreCase(targetColumnType, sign).trim();
-
-            boolean columnTypeCompare = false;
-            columnTypeCompare |= StringUtils.containsIgnoreCase(sourceColumnType, targetColumnType);
-            columnTypeCompare |= StringUtils.containsIgnoreCase(targetColumnType, sourceColumnType);
-            if (!columnTypeCompare) {
-                // 去掉精度参数再对比一次
-                sourceColumnType = synonymsType(StringUtils.substringBefore(sourceColumnType, "(")).trim();
-                targetColumnType = synonymsType(StringUtils.substringBefore(targetColumnType, "(")).trim();
-                columnTypeCompare |= StringUtils.containsIgnoreCase(sourceColumnType, targetColumnType);
-                columnTypeCompare |= StringUtils.containsIgnoreCase(targetColumnType, sourceColumnType);
-                if (!columnTypeCompare) {
-                    return false;
-                }
-            }
-
-            // if (!StringUtils.equalsIgnoreCase(sourceField.getDefaultValue(),
-            // targetField.getDefaultValue())) {
-            // return false;
-            // }
-
-            // BLOB, TEXT, GEOMETRY or JSON默认都是nullable，可以忽略比较，但比较了也是对齐
-            if (StringUtils.containsIgnoreCase(sourceColumnType, "timestamp")
-                || StringUtils.containsIgnoreCase(targetColumnType, "timestamp")) {
-                // timestamp可能会加default current_timestamp默认值,忽略对比nullable
-            } else {
-                if (sourceField.isNullable() != targetField.isNullable()) {
-                    return false;
-                }
-            }
-
-            // mysql会有一种处理,针对show create只有uk没有pk时，会在desc默认将uk当做pk
-            boolean isSourcePkOrUk = sourceField.isKey() || sourceField.isUnique();
-            boolean isTargetPkOrUk = targetField.isKey() || targetField.isUnique();
-            if (isSourcePkOrUk != isTargetPkOrUk) {
-                return false;
-            }
-        }
-
-        return true;
-    }
-
-    /**
-     * <pre>
-     * synonyms处理 
-     * 1. BOOL/BOOLEAN => TINYINT 
-     * 2. DEC/NUMERIC/FIXED => DECIMAL
-     * 3. INTEGER => INT
-     * 
-     * 
-     * </pre>
-     * 
-     * @param originType
-     * @return
-     */
-    private static String synonymsType(String originType) {
-        if (StringUtils.equalsIgnoreCase(originType, "bool") || StringUtils.equalsIgnoreCase(originType, "boolean")) {
-            return "tinyint";
-        } else if (StringUtils.equalsIgnoreCase(originType, "dec")
-                   || StringUtils.equalsIgnoreCase(originType, "numeric")
-                   || StringUtils.equalsIgnoreCase(originType, "fixed")) {
-            return "decimal";
-        } else if (StringUtils.equalsIgnoreCase(originType, "integer")) {
-            return "int";
-        } else if (StringUtils.equalsIgnoreCase(originType, "real")
-                   || StringUtils.equalsIgnoreCase(originType, "double precision")) {
-            return "double";
-        }
-
-        // BLOB、TEXT会根据给定长度自动转换为对应的TINY、MEDIUM，LONG类型，长度和字符集也有关，统一按照blob对比
-        if (StringUtils.equalsIgnoreCase(originType, "tinyblob")
-            || StringUtils.equalsIgnoreCase(originType, "mediumblob")
-            || StringUtils.equalsIgnoreCase(originType, "longblob")) {
-            return "blob";
-        } else if (StringUtils.equalsIgnoreCase(originType, "tinytext")
-                   || StringUtils.equalsIgnoreCase(originType, "mediumtext")
-                   || StringUtils.equalsIgnoreCase(originType, "longtext")) {
-            return "text";
-        }
-
-        return originType;
+        return builder.append(structureSchema(schema))
+            .append('.')
+            .append('`')
+            .append(StringUtils.replace(table, "`", "``"))
+            .append('`')
+            .toString();
     }
 
     private int snapshotExpire(int expireTimestamp) {
         return metaSnapshotDAO.deleteByTimestamp(destination, expireTimestamp);
     }
 
-    public void setConnection(MysqlConnection connection) {
-        this.connection = connection;
-    }
-
     public void setFilter(CanalEventFilter filter) {
         this.filter = filter;
     }
@@ -688,22 +689,22 @@ public class DatabaseTableMeta implements TableMetaTSDB {
         this.blackFilter = blackFilter;
     }
 
-    public void setFieldFilterMap(Map<String, List<String>> fieldFilterMap) {
-        this.fieldFilterMap = fieldFilterMap;
-    }
-
-    public void setFieldBlackFilterMap(Map<String, List<String>> fieldBlackFilterMap) {
-        this.fieldBlackFilterMap = fieldBlackFilterMap;
-    }
-
     public Map<String, List<String>> getFieldFilterMap() {
         return fieldFilterMap;
     }
 
+    public void setFieldFilterMap(Map<String, List<String>> fieldFilterMap) {
+        this.fieldFilterMap = fieldFilterMap;
+    }
+
     public Map<String, List<String>> getFieldBlackFilterMap() {
         return fieldBlackFilterMap;
     }
 
+    public void setFieldBlackFilterMap(Map<String, List<String>> fieldBlackFilterMap) {
+        this.fieldBlackFilterMap = fieldBlackFilterMap;
+    }
+
     public int getSnapshotInterval() {
         return snapshotInterval;
     }
@@ -724,6 +725,10 @@ public class DatabaseTableMeta implements TableMetaTSDB {
         return connection;
     }
 
+    public void setConnection(MysqlConnection connection) {
+        this.connection = connection;
+    }
+
     public boolean isUkDuplicateException(Throwable t) {
         if (pattern.matcher(t.getMessage()).find() || h2Pattern.matcher(t.getMessage()).find()) {
             // 违反外键约束时也抛出这种异常，所以这里还要判断包含字符串Duplicate entry

parse/src/main/resources/ddl/h2/meta_history.sql

@@ -10,7 +10,7 @@ CREATE TABLE IF NOT EXISTS `meta_history` (
   `use_schema` varchar(1024) DEFAULT NULL COMMENT '执行sql时对应的schema',
   `sql_schema` varchar(1024) DEFAULT NULL COMMENT '对应的schema',
   `sql_table` varchar(1024) DEFAULT NULL COMMENT '对应的table',
-  `sql_text` longtext DEFAULT NULL COMMENT '执行的sql',
+  `sql_text` CLOB DEFAULT NULL COMMENT '执行的sql',
   `sql_type` varchar(256) DEFAULT NULL COMMENT 'sql类型',
   `extra` text DEFAULT NULL COMMENT '额外的扩展信息',
   PRIMARY KEY (`id`),

parse/src/main/resources/ddl/h2/meta_snapshot.sql

@@ -7,7 +7,7 @@ CREATE TABLE IF NOT EXISTS `meta_snapshot` (
   `binlog_offest` bigint(20) DEFAULT NULL COMMENT 'binlog偏移量',
   `binlog_master_id` varchar(64) DEFAULT NULL COMMENT 'binlog节点id',
   `binlog_timestamp` bigint(20) DEFAULT NULL COMMENT 'binlog应用的时间戳',
-  `data` longtext DEFAULT NULL COMMENT '表结构数据',
+  `data` CLOB DEFAULT NULL COMMENT '表结构数据',
   `extra` text DEFAULT NULL COMMENT '额外的扩展信息',
   PRIMARY KEY (`id`),
   UNIQUE KEY meta_snapshot_binlog_file_offest(`destination`,`binlog_master_id`,`binlog_file`,`binlog_offest`),

parse/src/test/java/com/alibaba/otter/canal/parse/inbound/mysql/tsdb/FastsqlSchemaTest.java

@@ -20,10 +20,7 @@ public class FastsqlSchemaTest {
         String sql1 = "CREATE TABLE `table_x1` ( `id` bigint(20) NOT NULL AUTO_INCREMENT, "
                       + "`key1` longtext NOT NULL COMMENT 'key1', `value1` longtext NOT NULL COMMENT 'value1', PRIMARY KEY (`id`) )"
                       + "ENGINE=InnoDB DEFAULT CHARSET=utf8mb4";
-        String sql2 = " CREATE TABLE IF NOT EXISTS `table_x1` ( `id` bigint(20) NOT NULL AUTO_INCREMENT,"
-                      + "`key1` longtext NOT NULL COMMENT 'key1',PRIMARY KEY (`id`)) ENGINE=InnoDB DEFAULT CHARSET=utf8mb4";
         repository.console(sql1);
-        repository.console(sql2);
         repository.setDefaultSchema("test");
         SchemaObject table = repository.findTable("table_x1");
         System.out.println(table.getStatement().toString());
@@ -48,7 +45,7 @@ public class FastsqlSchemaTest {
         SchemaRepository repository = new SchemaRepository(JdbcConstants.MYSQL);
         String sql = " CREATE TABLE `articles` ( `article_id` bigint NOT NULL AUTO_INCREMENT,"
                      + " `tags` json DEFAULT NULL, PRIMARY KEY (`article_id`),"
-                     + " KEY `articles_tags` ((cast(json_extract(`tags`,_utf8mb4'$[*]') as char(40) array)))"
+                     + " KEY `articles_tags` ((cast(json_extract(`tags`,_utf8mb4'$[*]') as char(40))))"
                      + ") ENGINE=InnoDB AUTO_INCREMENT=1054 DEFAULT CHARSET=utf8mb4 COLLATE=utf8mb4_unicode_ci";
         repository.console(sql);
         repository.setDefaultSchema("test");
@@ -62,7 +59,7 @@ public class FastsqlSchemaTest {
         SchemaRepository repository = new SchemaRepository(JdbcConstants.MYSQL);
         String sql = " CREATE TABLE `proposal_order_info` (`id` bigint(20) unsigned NOT NULL AUTO_INCREMENT,"
                      + "`created_at` timestamp NULL DEFAULT NULL, " + "PRIMARY KEY (`id`) , "
-                     + "KEY `idx_create_time` (`created_at`) /*!80000 INVISIBLE */"
+                     + "KEY `idx_create_time` (`created_at`)"
                      + ") ENGINE=InnoDB DEFAULT CHARSET=utf8mb4 BLOCK_FORMAT=ENCRYPTED";
         repository.console(sql);
         repository.setDefaultSchema("test");
@@ -76,7 +73,7 @@ public class FastsqlSchemaTest {
         SchemaRepository repository = new SchemaRepository(JdbcConstants.MYSQL);
         String sql = " create table example_vc_tbl( c1 int not null auto_increment primary key,"
                      + "c2 varchar(70), vc1 int as (length(c2)) virtual,"
-                     + "DIM_SUM varchar(128) AS (MD5(UPPER(CONCAT(c2, c1)))) PERSISTENT)";
+                     + "DIM_SUM varchar(128) AS (MD5(UPPER(CONCAT(c2, c1)))) STORED)";
         repository.console(sql);
         repository.setDefaultSchema("test");
         SchemaObject table = repository.findTable("example_vc_tbl");
@@ -117,7 +114,7 @@ public class FastsqlSchemaTest {
                       + " name varchar(32) \n" + " )\n" + " partition by range(id) (\n"
                       + " partition p1 values less than (10),\n" + " partition px values less than MAXVALUE\n"
                       + " );";
-        String sql2 = "alter table test add partition ( partition 2 VALUES LESS THAN (738552) ENGINE = InnoDB, PARTITION pmax VALUES LESS THAN MAXVALUE ENGINE = InnoDB)";
+        String sql2 = "alter table test add partition ( partition p2 VALUES LESS THAN (738552) ENGINE = InnoDB, PARTITION pmax VALUES LESS THAN MAXVALUE ENGINE = InnoDB)";
         repository.console(sql1);
         repository.console(sql2);
         repository.setDefaultSchema("test");
@@ -136,7 +133,7 @@ public class FastsqlSchemaTest {
                       + "avg_frequency decimal(12,4) DEFAULT NULL,\n" + "hist_size tinyint(3) unsigned DEFAULT NULL,\n"
                       + "hist_type enum('SINGLE_PREC_HB','DOUBLE_PREC_HB') COLLATE utf8_bin DEFAULT NULL,\n"
                       + "histogram varbinary(255) DEFAULT NULL,\n" + "PRIMARY KEY (db_name,table_name,column_name)\n"
-                      + ") ENGINE=Aria DEFAULT CHARSET=utf8 COLLATE=utf8_bin PAGE_CHECKSUM=1 TRANSACTIONAL=0";
+                      + ") ENGINE=Aria DEFAULT CHARSET=utf8 COLLATE=utf8_bin";
         repository.console(sql1);
         repository.setDefaultSchema("test");
         SchemaObject table = repository.findTable("test");

parse/src/test/java/com/alibaba/otter/canal/parse/inbound/mysql/tsdb/MemoryTableMeta_DDL_Test.java

@@ -131,4 +131,26 @@ public class MemoryTableMeta_DDL_Test {
             Assert.assertTrue(field.isUnique());
         }
     }
+
+    @Test
+    public void test_polardbx_columnar_index () throws Throwable {
+        MemoryTableMeta memoryTableMeta = new MemoryTableMeta();
+        URL url = Thread.currentThread().getContextClassLoader().getResource("dummy.txt");
+        File dummyFile = new File(url.getFile());
+        File create = new File(dummyFile.getParent() + "/ddl", "ddl_test4.sql");
+        String sql = StringUtils.join(IOUtils.readLines(new FileInputStream(create)), "\n");
+        memoryTableMeta.apply(null, "test", sql, null);
+
+        List<String> tableNames = new ArrayList<>();
+        for (Schema schema : memoryTableMeta.getRepository().getSchemas()) {
+            tableNames.addAll(schema.showTables());
+        }
+
+        for (String table : tableNames) {
+            TableMeta sourceMeta = memoryTableMeta.find("test", table);
+            TableMeta.FieldMeta field = sourceMeta.getFieldMetaByName("address");
+            System.out.println(sourceMeta.toString());
+            Assert.assertTrue(field.isUnique());
+        }
+    }
 }