optimizer pom dependency

* Update SimpleAdminConnectors.java

add more information about which server execute failed

* Update CanalStarter.java

make canal.admin.port canal.admin.user canal.admin.passwd can be override by system env

README.md

@@ -5,11 +5,13 @@
 [![average time to resolve an issue](http://isitmaintained.com/badge/resolution/alibaba/canal.svg)](http://isitmaintained.com/project/alibaba/canal "average time to resolve an issue")
 [![percentage of issues still open](http://isitmaintained.com/badge/open/alibaba/canal.svg)](http://isitmaintained.com/project/alibaba/canal "percentage of issues still open")
 
-## 背景
+## 简介
 
-早期，阿里巴巴 B2B 公司因为存在杭州和美国双机房部署，存在跨机房同步的业务需求 ，主要是基于trigger的方式获取增量变更。从 2010 年开始，公司开始逐步尝试数据库日志解析，获取增量变更进行同步，由此衍生出了增量订阅和消费业务，从此开启一段新纪元。
+![](https://github.com/bucketli/simpletools/blob/master/main_func_v2.png?raw=true)
 
-当前的 canal 支持源端 MySQL 版本包括 5.1.x , 5.5.x , 5.6.x , 5.7.x , 8.0.x
+**canal [kə'næl]**，译意为水道/管道/沟渠，主要用途是基于 MySQL 数据库增量日志解析，提供增量数据订阅和消费
+
+早期阿里巴巴因为杭州和美国双机房部署，存在跨机房同步的业务需求，实现方式主要是基于业务 trigger 获取增量变更。从 2010 年开始，业务逐步尝试数据库日志解析获取增量变更进行同步，由此衍生出了大量的数据库增量订阅和消费业务。
 
 基于日志增量订阅和消费的业务包括
 - 数据库镜像
@@ -18,12 +20,7 @@
 - 业务 cache 刷新
 - 带业务逻辑的增量数据处理
 
-## 项目介绍
-
-#### 名称：canal [kə'næl]
-#### 译意： 水道/管道/沟渠
-#### 产品定位： 基于数据库增量日志解析，提供增量数据订阅和消费
-#### 关键词： MySQL binlog parser / real-time / queue&topic / index build
+当前的 canal 支持源端 MySQL 版本包括 5.1.x , 5.5.x , 5.6.x , 5.7.x , 8.0.x
 
 ## 工作原理
 
@@ -35,7 +32,6 @@
 - MySQL slave 重放 relay log 中事件，将数据变更反映它自己的数据
 
 #### canal 工作原理
-![](http://dl.iteye.com/upload/attachment/0080/3107/c87b67ba-394c-3086-9577-9db05be04c95.jpg)
 
 - canal 模拟 MySQL slave 的交互协议，伪装自己为 MySQL slave ，向 MySQL master 发送dump 协议
 - MySQL master 收到 dump 请求，开始推送 binary log 给 slave (即 canal )
@@ -43,7 +39,7 @@
 
 ## 重要版本更新说明
 
-canal 1.1.x 版本（[release_note](https://github.com/alibaba/canal/releases)）,性能与功能层面有较大的突破,重要提升包括:
+1. canal 1.1.x 版本（[release_note](https://github.com/alibaba/canal/releases)）,性能与功能层面有较大的突破,重要提升包括:
 
 - 整体性能测试&优化,提升了150%. #726 参考: [Performance](https://github.com/alibaba/canal/wiki/Performance)
 - 原生支持prometheus监控 #765 [Prometheus QuickStart](https://github.com/alibaba/canal/wiki/Prometheus-QuickStart)
@@ -51,6 +47,8 @@ canal 1.1.x 版本（[release_note](https://github.com/alibaba/canal/releases)
 - 原生支持aliyun rds的binlog订阅 (解决自动主备切换/oss binlog离线解析) 参考: [Aliyun RDS QuickStart](https://github.com/alibaba/canal/wiki/aliyun-RDS-QuickStart)
 - 原生支持docker镜像 #801 参考: [Docker QuickStart](https://github.com/alibaba/canal/wiki/Docker-QuickStart)
 
+2.  canal 1.1.4版本，迎来最重要的WebUI能力，引入canal-admin工程，支持面向WebUI的canal动态管理能力，支持配置、任务、日志等在线白屏运维能力，具体文档：[Canal Admin Guide](https://github.com/alibaba/canal/wiki/Canal-Admin-Guide)
+
 ## 文档
 
 - [Home](https://github.com/alibaba/canal/wiki/Home)
@@ -60,6 +58,11 @@ canal 1.1.x 版本（[release_note](https://github.com/alibaba/canal/releases)
   - [Canal Kafka/RocketMQ QuickStart](https://github.com/alibaba/canal/wiki/Canal-Kafka-RocketMQ-QuickStart")
   - [Aliyun RDS for MySQL QuickStart](https://github.com/alibaba/canal/wiki/aliyun-RDS-QuickStart)
   - [Prometheus QuickStart](https://github.com/alibaba/canal/wiki/Prometheus-QuickStart)
+- Canal Admin
+  - [Canal Admin QuickStart](https://github.com/alibaba/canal/wiki/Canal-Admin-QuickStart)
+  - [Canal Admin Guide](https://github.com/alibaba/canal/wiki/Canal-Admin-Guide)
+  - [Canal Admin ServerGuide](https://github.com/alibaba/canal/wiki/Canal-Admin-ServerGuide)
+  - [Canal Admin Docker](https://github.com/alibaba/canal/wiki/Canal-Admin-Docker)
 - [AdminGuide](https://github.com/alibaba/canal/wiki/AdminGuide)
 - [ClientExample](https://github.com/alibaba/canal/wiki/ClientExample)
 - [ClientAPI](https://github.com/alibaba/canal/wiki/ClientAPI)
@@ -92,9 +95,4 @@ canal 作为 MySQL binlog 增量获取和解析工具，可将变更记录投递
 - [阿里巴巴去 Oracle 数据迁移同步工具: yugong](http://github.com/alibaba/yugong)
 
 ## 问题反馈
-- 邮件: jianghang115@gmail.com
 - 报告 issue: [github issues](https://github.com/alibaba/canal/issues)
-
-## 其他资料
-
-  - [ ADC 阿里技术嘉年华分享 ppt ( google docs)](https://docs.google.com/presentation/d/1MkszUPYRDkfVPz9IqOT1LLT5d9tuwde_WC8GZvjaDRg/edit?usp=sharing) 

canal-admin/canal-admin-server/pom.xml

@@ -3,7 +3,7 @@
     <parent>
         <artifactId>canal-admin</artifactId>
         <groupId>com.alibaba.otter</groupId>
-        <version>1.1.4</version>
+        <version>1.1.5-SNAPSHOT</version>
     </parent>
     <modelVersion>4.0.0</modelVersion>
     <artifactId>canal-admin-server</artifactId>

canal-admin/canal-admin-server/src/main/java/com/alibaba/otter/canal/admin/connector/SimpleAdminConnectors.java

@@ -25,6 +25,7 @@ public class SimpleAdminConnectors {
             connector.connect();
             return function.apply(connector);
         } catch (Exception e) {
+            logger.error("connect to ip:{},port:{},user:{},password:{}, failed", ip, port, user,passwd);
             logger.error(e.getMessage());
         } finally {
             connector.disconnect();

canal-admin/canal-admin-server/src/main/java/com/alibaba/otter/canal/admin/model/CanalCluster.java

@@ -1,6 +1,7 @@
 package com.alibaba.otter.canal.admin.model;
 
 import io.ebean.Finder;
+import io.ebean.annotation.WhenModified;
 
 import javax.persistence.Entity;
 import javax.persistence.Id;
@@ -34,6 +35,7 @@ public class CanalCluster extends Model {
     private Long   id;
     private String name;
     private String zkHosts;
+    @WhenModified
     private Date   modifiedTime;
 
     public Long getId() {

canal-admin/canal-admin-server/src/main/java/com/alibaba/otter/canal/admin/model/CanalConfig.java

@@ -1,6 +1,7 @@
 package com.alibaba.otter.canal.admin.model;
 
 import io.ebean.Finder;
+import io.ebean.annotation.WhenModified;
 
 import java.util.Date;
 
@@ -37,6 +38,7 @@ public class CanalConfig extends Model {
     private String content;
     private String contentMd5;
     private String status;
+    @WhenModified
     private Date   modifiedTime;
 
     public void init() {

canal-admin/canal-admin-server/src/main/java/com/alibaba/otter/canal/admin/model/CanalInstanceConfig.java

@@ -5,6 +5,7 @@ import java.util.Date;
 import javax.persistence.*;
 
 import io.ebean.Finder;
+import io.ebean.annotation.WhenModified;
 
 /**
  * Canal实例配置信息实体类
@@ -44,6 +45,7 @@ public class CanalInstanceConfig extends Model {
     private String       content;
     private String       contentMd5;
     private String       status;         // 1: 正常 0: 停止
+    @WhenModified
     private Date         modifiedTime;
 
     @Transient

canal-admin/canal-admin-server/src/main/java/com/alibaba/otter/canal/admin/model/Model.java

@@ -1,16 +1,13 @@
 package com.alibaba.otter.canal.admin.model;
 
-import io.ebean.Ebean;
-import io.ebean.EbeanServer;
-
-import java.lang.reflect.Field;
-
-import javax.persistence.Id;
 import javax.persistence.MappedSuperclass;
 import javax.persistence.OptimisticLockException;
 
 import org.apache.commons.beanutils.PropertyUtils;
 
+import io.ebean.Ebean;
+import io.ebean.EbeanServer;
+
 /**
  * EBean Model扩展类
  *
@@ -35,21 +32,10 @@ public abstract class Model extends io.ebean.Model {
 
     public void saveOrUpdate() {
         try {
-            Field idField = null;
-            // find id field
-            Field[] fields = this.getClass().getDeclaredFields();
-            for (Field field : fields) {
-                Id idAnn = field.getAnnotation(Id.class);
-                if (idAnn != null) {
-                    idField = field;
-                    break;
-                }
-            }
-            if (idField == null) {
-                return;
-            }
-            Object idVal = PropertyUtils.getProperty(this, idField.getName());
-            if (idVal == null) {
+            EbeanServer ebeanServer = Ebean.getDefaultServer();
+            Object id = ebeanServer.getBeanId(this);
+            if (id == null) {
+                init();
                 this.save();
             } else {
                 this.update();
@@ -63,7 +49,13 @@ public abstract class Model extends io.ebean.Model {
         try {
             EbeanServer ebeanServer = Ebean.getDefaultServer();
             Object id = ebeanServer.getBeanId(this);
+            if (id == null) {
+                return;
+            }
             Object model = ebeanServer.createQuery(this.getClass()).where().idEq(id).findOne();
+            if (model == null) {
+                return;
+            }
             for (String propertyName : propertiesNames) {
                 if (propertyName.startsWith("nn:")) { // not null
                     propertyName = propertyName.substring(3);

canal-admin/canal-admin-server/src/main/java/com/alibaba/otter/canal/admin/model/NodeServer.java

@@ -1,6 +1,7 @@
 package com.alibaba.otter.canal.admin.model;
 
 import io.ebean.Finder;
+import io.ebean.annotation.WhenModified;
 
 import java.util.Date;
 
@@ -42,6 +43,7 @@ public class NodeServer extends Model {
     private Integer      metricPort;
     private Integer      tcpPort;
     private String       status;
+    @WhenModified
     private Date         modifiedTime;
 
     public void init() {

canal-admin/canal-admin-server/src/main/java/com/alibaba/otter/canal/admin/model/User.java

@@ -1,6 +1,7 @@
 package com.alibaba.otter.canal.admin.model;
 
 import io.ebean.Finder;
+import io.ebean.annotation.WhenCreated;
 
 import javax.persistence.Entity;
 import javax.persistence.Id;
@@ -36,6 +37,7 @@ public class User extends Model {
     private String introduction;
     private String avatar;
     private String name;
+    @WhenCreated
     private Date   creationDate;
 
     @Transient

canal-admin/canal-admin-server/src/main/java/com/alibaba/otter/canal/admin/service/impl/CanalInstanceServiceImpl.java

@@ -109,13 +109,15 @@ public class CanalInstanceServiceImpl implements CanalInstanceService {
             }));
         }
 
-        futures.forEach(f -> {
+        for (Future<Void> f :futures){
             try {
                 f.get(3, TimeUnit.SECONDS);
-            } catch (TimeoutException | InterruptedException | ExecutionException e) {
+            } catch (InterruptedException | ExecutionException e) {
                 // ignore
+            } catch (TimeoutException e){
+                break;
             }
-        });
+        }
         return pager;
     }
 

canal-admin/canal-admin-server/src/main/java/com/alibaba/otter/canal/admin/service/impl/NodeServerServiceImpl.java

@@ -154,13 +154,15 @@ public class NodeServerServiceImpl implements NodeServerService {
                 return !status;
             }));
         }
-        futures.forEach(f -> {
+        for (Future<Boolean> f :futures){
             try {
                 f.get(3, TimeUnit.SECONDS);
-            } catch (TimeoutException | InterruptedException | ExecutionException e) {
+            } catch (InterruptedException | ExecutionException e) {
                 // ignore
+            } catch (TimeoutException e){
+                break;
             }
-        });
+        }
 
         return pager;
     }

canal-admin/canal-admin-ui/pom.xml

@@ -3,7 +3,7 @@
   <parent>
     <artifactId>canal-admin</artifactId>
     <groupId>com.alibaba.otter</groupId>
-    <version>1.1.4</version>
+    <version>1.1.5-SNAPSHOT</version>
   </parent>
   <modelVersion>4.0.0</modelVersion>
 

canal-admin/pom.xml

@@ -4,7 +4,7 @@
     <modelVersion>4.0.0</modelVersion>
     <groupId>com.alibaba.otter</groupId>
     <artifactId>canal-admin</artifactId>
-    <version>1.1.4</version>
+    <version>1.1.5-SNAPSHOT</version>
 
     <packaging>pom</packaging>
     <modules>
@@ -34,7 +34,7 @@
             <dependency>
                 <groupId>mysql</groupId>
                 <artifactId>mysql-connector-java</artifactId>
-                <version>5.1.40</version>
+                <version>8.0.17</version>
             </dependency>
             <dependency>
                 <groupId>com.github.ben-manes.caffeine</groupId>

client-adapter/common/pom.xml

@@ -3,7 +3,8 @@
     <parent>
         <artifactId>canal.client-adapter</artifactId>
         <groupId>com.alibaba.otter</groupId>
-        <version>1.1.4</version>
+        <version>1.1.5-SNAPSHOT</version>
+        <relativePath>../pom.xml</relativePath>
     </parent>
     <modelVersion>4.0.0</modelVersion>
     <groupId>com.alibaba.otter</groupId>
@@ -14,27 +15,80 @@
         <dependency>
             <groupId>com.alibaba.otter</groupId>
             <artifactId>canal.protocol</artifactId>
-            <version>1.1.4</version>
-        </dependency>
-        <dependency>
-            <groupId>joda-time</groupId>
-            <artifactId>joda-time</artifactId>
-            <version>2.9.4</version>
-        </dependency>
-        <dependency>
-            <groupId>com.alibaba</groupId>
-            <artifactId>druid</artifactId>
-            <version>1.1.9</version>
+            <version>${project.version}</version>
         </dependency>
         <dependency>
             <groupId>org.springframework</groupId>
             <artifactId>spring-context</artifactId>
             <version>5.0.5.RELEASE</version>
         </dependency>
+        <dependency>
+            <groupId>org.springframework</groupId>
+            <artifactId>spring-core</artifactId>
+            <version>5.0.5.RELEASE</version>
+        </dependency>
+        <dependency>
+            <groupId>org.springframework</groupId>
+            <artifactId>spring-aop</artifactId>
+            <version>5.0.5.RELEASE</version>
+        </dependency>
+        <dependency>
+            <groupId>org.springframework</groupId>
+            <artifactId>spring-context</artifactId>
+            <version>5.0.5.RELEASE</version>
+        </dependency>
+        <dependency>
+            <groupId>org.springframework</groupId>
+            <artifactId>spring-jdbc</artifactId>
+            <version>5.0.5.RELEASE</version>
+        </dependency>
+        <dependency>
+            <groupId>org.springframework</groupId>
+            <artifactId>spring-orm</artifactId>
+            <version>5.0.5.RELEASE</version>
+        </dependency>
+        <dependency>
+            <groupId>org.springframework</groupId>
+            <artifactId>spring-test</artifactId>
+            <version>5.0.5.RELEASE</version>
+            <scope>test</scope>
+        </dependency>
+        <dependency>
+            <groupId>joda-time</groupId>
+            <artifactId>joda-time</artifactId>
+        </dependency>
+        <dependency>
+            <groupId>com.alibaba</groupId>
+            <artifactId>druid</artifactId>
+        </dependency>        
         <dependency>
             <groupId>org.yaml</groupId>
             <artifactId>snakeyaml</artifactId>
-            <version>1.19</version>
+        </dependency>
+        <dependency>
+            <groupId>org.powermock</groupId>
+            <artifactId>powermock-api-mockito</artifactId>
+            <scope>test</scope>
+        </dependency>
+        <dependency>
+            <groupId>org.powermock</groupId>
+            <artifactId>powermock-module-junit4</artifactId>
+            <scope>test</scope>
+        </dependency>
+        <dependency>
+            <groupId>org.mockito</groupId>
+            <artifactId>mockito-all</artifactId>
+            <scope>test</scope>
+        </dependency>
+        <dependency>
+            <groupId>junit</groupId>
+            <artifactId>junit</artifactId>
+            <scope>test</scope>
+        </dependency>
+        <dependency>
+            <groupId>com.diffblue</groupId>
+            <artifactId>deeptestutils</artifactId>
+            <scope>test</scope>
         </dependency>
     </dependencies>
 

client-adapter/common/src/main/java/com/alibaba/otter/canal/client/adapter/support/CanalClientConfig.java

@@ -33,6 +33,13 @@ public class CanalClientConfig {
     // aliyun ak/sk
     private String             accessKey;
     private String             secretKey;
+    // rabbitmq 账号密码
+    private String             username;
+    private String             password;
+    // rabbitmq vhost
+    private String             vhost         = "/";
+
+    private Long               resourceOwnerId;
     // 是否启用消息轨迹
     private boolean            enableMessageTrace;
     // 在使用阿里云商业化mq服务时，如果想使用云上消息轨迹功能，请设置此配置为true
@@ -132,6 +139,38 @@ public class CanalClientConfig {
         this.secretKey = secretKey;
     }
 
+    public String getUsername() {
+        return username;
+    }
+
+    public void setUsername(String username) {
+        this.username = username;
+    }
+
+    public String getPassword() {
+        return password;
+    }
+
+    public void setPassword(String password) {
+        this.password = password;
+    }
+
+    public String getVhost() {
+        return vhost;
+    }
+
+    public void setVhost(String vhost) {
+        this.vhost = vhost;
+    }
+
+    public Long getResourceOwnerId() {
+        return resourceOwnerId;
+    }
+
+    public void setResourceOwnerId(Long resourceOwnerId) {
+        this.resourceOwnerId = resourceOwnerId;
+    }
+
     public List<CanalAdapter> getCanalAdapters() {
         return canalAdapters;
     }
@@ -176,7 +215,7 @@ public class CanalClientConfig {
 
         private String      instance; // 实例名
 
-        private List<Group> groups;  // 适配器分组列表
+        private List<Group> groups;   // 适配器分组列表
 
         public String getInstance() {
             return instance;

client-adapter/common/src/test/java/com/alibaba/otter/canal/client/adapter/support/JdbcTypeUtilTest.java

@@ -0,0 +1,200 @@
+package com.alibaba.otter.canal.client.adapter.support;
+
+import static org.mockito.AdditionalMatchers.or;
+import static org.mockito.Matchers.isA;
+import static org.mockito.Matchers.isNull;
+import static org.powermock.api.mockito.PowerMockito.mockStatic;
+
+import com.alibaba.otter.canal.client.adapter.support.JdbcTypeUtil;
+import com.alibaba.otter.canal.client.adapter.support.Util;
+import org.junit.Assert;
+import org.junit.Rule;
+import org.junit.Test;
+import org.junit.rules.ExpectedException;
+import org.junit.rules.Timeout;
+import org.junit.runner.RunWith;
+import org.powermock.api.mockito.PowerMockito;
+import org.powermock.core.classloader.annotations.PrepareForTest;
+import org.powermock.modules.junit4.PowerMockRunner;
+
+import java.lang.reflect.Array;
+
+@RunWith(PowerMockRunner.class)
+public class JdbcTypeUtilTest {
+
+  @Rule public final ExpectedException thrown = ExpectedException.none();
+
+  @Rule public final Timeout globalTimeout = new Timeout(10000);
+
+  /* testedClasses: JdbcTypeUtil */
+  // Test written by Diffblue Cover.
+  @Test
+  public void typeConvertInputNotNullNotNullNotNullNegativeNotNullOutput3() {
+
+    // Arrange
+    final String tableName = "foo";
+    final String columnName = "foo";
+    final String value = "foo";
+    final int sqlType = -4;
+    final String mysqlType = "foo";
+
+    // Act
+    final Object actual =
+        JdbcTypeUtil.typeConvert(tableName, columnName, value, sqlType, mysqlType);
+
+    // Assert result
+    Assert.assertArrayEquals(new byte[] {(byte)102, (byte)111, (byte)111}, ((byte[])actual));
+  }
+
+  // Test written by Diffblue Cover.
+  @Test
+  public void typeConvertInputNotNullNotNullNotNullNegativeNotNullOutputPositive() {
+
+    // Arrange
+    final String tableName = "foo";
+    final String columnName = "foo";
+    final String value = "1234";
+    final int sqlType = -5;
+    final String mysqlType = "foo";
+
+    // Act
+    final Object actual =
+        JdbcTypeUtil.typeConvert(tableName, columnName, value, sqlType, mysqlType);
+
+    // Assert result
+    Assert.assertEquals(1234L, actual);
+  }
+
+  // Test written by Diffblue Cover.
+  @Test
+  public void typeConvertInputNotNullNotNullNotNullNegativeNotNullOutputPositive2() {
+
+    // Arrange
+    final String tableName = "?????????";
+    final String columnName = "?";
+    final String value = "2";
+    final int sqlType = -5;
+    final String mysqlType = "bigint";
+
+    // Act
+    final Object actual =
+        JdbcTypeUtil.typeConvert(tableName, columnName, value, sqlType, mysqlType);
+
+    // Assert result
+    Assert.assertEquals(2L, actual);
+  }
+
+  // Test written by Diffblue Cover.
+  @Test
+  public void typeConvertInputNotNullNotNullNotNullPositiveNotNullOutputFalse() {
+
+    // Arrange
+    final String tableName = "?????????";
+    final String columnName = "?";
+    final String value = "0";
+    final int sqlType = 16;
+    final String mysqlType = "bigint\u046bunsigned";
+
+    // Act
+    final Object actual =
+        JdbcTypeUtil.typeConvert(tableName, columnName, value, sqlType, mysqlType);
+
+    // Assert result
+    Assert.assertFalse((boolean)actual);
+  }
+
+  // Test written by Diffblue Cover.
+  @Test
+  public void typeConvertInputNotNullNotNullNotNullPositiveNotNullOutputNotNull() {
+
+    // Arrange
+    final String tableName = "1";
+    final String columnName = "Bar";
+    final String value = "1";
+    final int sqlType = 12;
+    final String mysqlType = "2";
+
+    // Act
+    final Object actual =
+        JdbcTypeUtil.typeConvert(tableName, columnName, value, sqlType, mysqlType);
+
+    // Assert result
+    Assert.assertEquals("1", actual);
+  }
+
+  // Test written by Diffblue Cover.
+  @Test
+  public void typeConvertInputNotNullNotNullNotNullPositiveNotNullOutputPositive() {
+
+    // Arrange
+    final String tableName = "foo";
+    final String columnName = "foo";
+    final String value = "3";
+    final int sqlType = 4;
+    final String mysqlType = "foo";
+
+    // Act
+    final Object actual =
+        JdbcTypeUtil.typeConvert(tableName, columnName, value, sqlType, mysqlType);
+
+    // Assert result
+    Assert.assertEquals(3, actual);
+  }
+
+  // Test written by Diffblue Cover.
+  @Test
+  public void typeConvertInputNotNullNotNullNotNullPositiveNotNullOutputTrue() {
+
+    // Arrange
+    final String tableName = "?????????";
+    final String columnName = "?";
+    final String value = "5";
+    final int sqlType = 16;
+    final String mysqlType = "bigintsunsigned";
+
+    // Act
+    final Object actual =
+        JdbcTypeUtil.typeConvert(tableName, columnName, value, sqlType, mysqlType);
+
+    // Assert result
+    Assert.assertTrue((boolean)actual);
+  }
+
+  // Test written by Diffblue Cover.
+  @Test
+  public void typeConvertInputNotNullNotNullNotNullPositiveNotNullOutputZero2() {
+
+    // Arrange
+    final String tableName = "?????????";
+    final String columnName = "?";
+    final String value = "0";
+    final int sqlType = 7;
+    final String mysqlType = "bigint\u046bunsigned";
+
+    // Act
+    final Object actual =
+        JdbcTypeUtil.typeConvert(tableName, columnName, value, sqlType, mysqlType);
+
+    // Assert result
+    Assert.assertEquals(0.0f, (float)actual, 0.0f);
+  }
+
+  // Test written by Diffblue Cover.
+  @Test
+  public void typeConvertInputNotNullNotNullNotNullPositiveNotNullOutputZero3() {
+
+    // Arrange
+    final String tableName = "?????????";
+    final String columnName = "?";
+    final String value = "0";
+    final int sqlType = 5;
+    final String mysqlType = "bigintiunsigned";
+
+    // Act
+    final Object actual =
+        JdbcTypeUtil.typeConvert(tableName, columnName, value, sqlType, mysqlType);
+
+    // Assert result
+    Assert.assertEquals((short)0, actual);
+  }
+}

client-adapter/common/src/test/java/com/alibaba/otter/canal/client/adapter/support/UtilTest.java

@@ -0,0 +1,130 @@
+package com.alibaba.otter.canal.client.adapter.support;
+
+import static org.mockito.AdditionalMatchers.or;
+import static org.mockito.Matchers.isA;
+import static org.mockito.Matchers.isNull;
+import static org.powermock.api.mockito.PowerMockito.mockStatic;
+
+import com.alibaba.otter.canal.client.adapter.support.Util;
+import com.diffblue.deeptestutils.mock.DTUMemberMatcher;
+import org.apache.commons.lang.StringUtils;
+import org.junit.Assert;
+import org.junit.Rule;
+import org.junit.Test;
+import org.junit.rules.ExpectedException;
+import org.junit.rules.Timeout;
+import org.junit.runner.RunWith;
+import org.powermock.api.mockito.PowerMockito;
+import org.powermock.core.classloader.annotations.PowerMockIgnore;
+import org.powermock.core.classloader.annotations.PrepareForTest;
+import org.powermock.modules.junit4.PowerMockRunner;
+
+import java.lang.reflect.Method;
+import java.util.Date;
+
+@RunWith(PowerMockRunner.class)
+@PowerMockIgnore({"javax.management.*"})
+public class UtilTest {
+
+  @Rule public final ExpectedException thrown = ExpectedException.none();
+
+  @Rule public final Timeout globalTimeout = new Timeout(10000);
+
+  /* testedClasses: Util */
+  // Test written by Diffblue Cover.
+  @Test
+  public void cleanColumnInputNotNullOutputNotNull() {
+
+    // Arrange
+    final String column = "1234";
+
+    // Act
+    final String actual = Util.cleanColumn(column);
+
+    // Assert result
+    Assert.assertEquals("1234", actual);
+  }
+
+  // Test written by Diffblue Cover.
+  @Test
+  public void cleanColumnInputNullOutputNull() {
+
+    // Arrange
+    final String column = null;
+
+    // Act
+    final String actual = Util.cleanColumn(column);
+
+    // Assert result
+    Assert.assertNull(actual);
+  }
+
+  // Test written by Diffblue Cover.
+  @PrepareForTest(StringUtils.class)
+  @Test
+  public void parseDate2InputNotNullOutputNull() throws Exception {
+
+    // Setup mocks
+    PowerMockito.mockStatic(StringUtils.class);
+
+    // Arrange
+    final String datetimeStr = "1a 2b 3c";
+    final Method isEmptyMethod =
+        DTUMemberMatcher.method(StringUtils.class, "isEmpty", String.class);
+    PowerMockito.doReturn(true)
+        .when(StringUtils.class, isEmptyMethod)
+        .withArguments(or(isA(String.class), isNull(String.class)));
+
+    // Act
+    final Date actual = Util.parseDate2(datetimeStr);
+
+    // Assert result
+    Assert.assertNull(actual);
+  }
+
+  // Test written by Diffblue Cover.
+  @PrepareForTest(StringUtils.class)
+  @Test
+  public void parseDate2InputNotNullOutputNull2() throws Exception {
+
+    // Setup mocks
+    PowerMockito.mockStatic(StringUtils.class);
+
+    // Arrange
+    final String datetimeStr = "1a 2b 3c";
+    final Method isEmptyMethod =
+        DTUMemberMatcher.method(StringUtils.class, "isEmpty", String.class);
+    PowerMockito.doReturn(false)
+        .when(StringUtils.class, isEmptyMethod)
+        .withArguments(or(isA(String.class), isNull(String.class)));
+
+    // Act
+    final Date actual = Util.parseDate2(datetimeStr);
+
+    // Assert result
+    Assert.assertNull(actual);
+  }
+
+  // Test written by Diffblue Cover.
+  @PrepareForTest(StringUtils.class)
+  @Test
+  public void parseDateInputNotNullOutputNull() throws Exception {
+
+    // Setup mocks
+    PowerMockito.mockStatic(StringUtils.class);
+
+    // Arrange
+    final String datetimeStr = "a/b/c";
+    final Method isEmptyMethod =
+        DTUMemberMatcher.method(StringUtils.class, "isEmpty", String.class);
+    PowerMockito.doReturn(true)
+        .when(StringUtils.class, isEmptyMethod)
+        .withArguments(or(isA(String.class), isNull(String.class)));
+
+    // Act
+    final Date actual = Util.parseDate(datetimeStr);
+
+    // Assert result
+    Assert.assertNull(actual);
+  }
+}

client-adapter/elasticsearch/src/main/java/com/alibaba/otter/canal/client/adapter/es/ESAdapter.java

@@ -1,250 +0,0 @@
-package com.alibaba.otter.canal.client.adapter.es;
-
-import java.util.LinkedHashMap;
-import java.util.List;
-import java.util.Map;
-import java.util.Properties;
-import java.util.concurrent.ConcurrentHashMap;
-import java.util.regex.Matcher;
-import java.util.regex.Pattern;
-
-import javax.sql.DataSource;
-
-import org.apache.commons.lang.StringUtils;
-import org.elasticsearch.action.search.SearchResponse;
-
-import com.alibaba.druid.pool.DruidDataSource;
-import com.alibaba.otter.canal.client.adapter.OuterAdapter;
-import com.alibaba.otter.canal.client.adapter.es.config.ESSyncConfig;
-import com.alibaba.otter.canal.client.adapter.es.config.ESSyncConfig.ESMapping;
-import com.alibaba.otter.canal.client.adapter.es.config.ESSyncConfigLoader;
-import com.alibaba.otter.canal.client.adapter.es.config.SchemaItem;
-import com.alibaba.otter.canal.client.adapter.es.config.SqlParser;
-import com.alibaba.otter.canal.client.adapter.es.monitor.ESConfigMonitor;
-import com.alibaba.otter.canal.client.adapter.es.service.ESEtlService;
-import com.alibaba.otter.canal.client.adapter.es.service.ESSyncService;
-import com.alibaba.otter.canal.client.adapter.es.support.ESConnection;
-import com.alibaba.otter.canal.client.adapter.es.support.ESTemplate;
-import com.alibaba.otter.canal.client.adapter.support.*;
-
-/**
- * ES外部适配器
- *
- * @author rewerma 2018-10-20
- * @version 1.0.0
- */
-@SPI("es")
-public class ESAdapter implements OuterAdapter {
-
-    private Map<String, ESSyncConfig>              esSyncConfig        = new ConcurrentHashMap<>(); // 文件名对应配置
-    private Map<String, Map<String, ESSyncConfig>> dbTableEsSyncConfig = new ConcurrentHashMap<>(); // schema-table对应配置
-
-    private ESConnection                           esConnection;
-
-    private ESSyncService                          esSyncService;
-
-    private ESConfigMonitor                        esConfigMonitor;
-
-    private Properties                             envProperties;
-
-    public ESSyncService getEsSyncService() {
-        return esSyncService;
-    }
-
-    public Map<String, ESSyncConfig> getEsSyncConfig() {
-        return esSyncConfig;
-    }
-
-    public Map<String, Map<String, ESSyncConfig>> getDbTableEsSyncConfig() {
-        return dbTableEsSyncConfig;
-    }
-
-    public ESConnection getEsConnection() {
-        return esConnection;
-    }
-
-    public void setEsConnection(ESConnection esConnection) {
-        this.esConnection = esConnection;
-    }
-
-    @Override
-    public void init(OuterAdapterConfig configuration, Properties envProperties) {
-        try {
-            this.envProperties = envProperties;
-            Map<String, ESSyncConfig> esSyncConfigTmp = ESSyncConfigLoader.load(envProperties);
-            // 过滤不匹配的key的配置
-            esSyncConfigTmp.forEach((key, config) -> {
-                if ((config.getOuterAdapterKey() == null && configuration.getKey() == null)
-                    || (config.getOuterAdapterKey() != null
-                        && config.getOuterAdapterKey().equalsIgnoreCase(configuration.getKey()))) {
-                    esSyncConfig.put(key, config);
-                }
-            });
-
-            for (Map.Entry<String, ESSyncConfig> entry : esSyncConfig.entrySet()) {
-                String configName = entry.getKey();
-                ESSyncConfig config = entry.getValue();
-                SchemaItem schemaItem = SqlParser.parse(config.getEsMapping().getSql());
-                config.getEsMapping().setSchemaItem(schemaItem);
-
-                DruidDataSource dataSource = DatasourceConfig.DATA_SOURCES.get(config.getDataSourceKey());
-                if (dataSource == null || dataSource.getUrl() == null) {
-                    throw new RuntimeException("No data source found: " + config.getDataSourceKey());
-                }
-                Pattern pattern = Pattern.compile(".*:(.*)://.*/(.*)\\?.*$");
-                Matcher matcher = pattern.matcher(dataSource.getUrl());
-                if (!matcher.find()) {
-                    throw new RuntimeException("Not found the schema of jdbc-url: " + config.getDataSourceKey());
-                }
-                String schema = matcher.group(2);
-
-                schemaItem.getAliasTableItems().values().forEach(tableItem -> {
-                    Map<String, ESSyncConfig> esSyncConfigMap;
-                    if (envProperties != null
-                        && !"tcp".equalsIgnoreCase(envProperties.getProperty("canal.conf.mode"))) {
-                        esSyncConfigMap = dbTableEsSyncConfig
-                            .computeIfAbsent(StringUtils.trimToEmpty(config.getDestination()) + "-"
-                                             + StringUtils.trimToEmpty(config.getGroupId()) + "_" + schema + "-"
-                                             + tableItem.getTableName(),
-                                k -> new ConcurrentHashMap<>());
-                    } else {
-                        esSyncConfigMap = dbTableEsSyncConfig
-                            .computeIfAbsent(StringUtils.trimToEmpty(config.getDestination()) + "_" + schema + "-"
-                                             + tableItem.getTableName(),
-                                k -> new ConcurrentHashMap<>());
-                    }
-
-                    esSyncConfigMap.put(configName, config);
-                });
-            }
-
-            Map<String, String> properties = configuration.getProperties();
-
-            String[] hostArray = configuration.getHosts().split(",");
-            String mode = properties.get("mode");
-            if ("rest".equalsIgnoreCase(mode) || "http".equalsIgnoreCase(mode)) {
-                esConnection = new ESConnection(hostArray, properties, ESConnection.ESClientMode.REST);
-            } else {
-                esConnection = new ESConnection(hostArray, properties, ESConnection.ESClientMode.TRANSPORT);
-            }
-
-            ESTemplate esTemplate = new ESTemplate(esConnection);
-            esSyncService = new ESSyncService(esTemplate);
-
-            esConfigMonitor = new ESConfigMonitor();
-            esConfigMonitor.init(this, envProperties);
-        } catch (Throwable e) {
-            throw new RuntimeException(e);
-        }
-    }
-
-    @Override
-    public void sync(List<Dml> dmls) {
-        if (dmls == null || dmls.isEmpty()) {
-            return;
-        }
-        for (Dml dml : dmls) {
-            if (!dml.getIsDdl()) {
-                sync(dml);
-            }
-        }
-        esSyncService.commit(); // 批次统一提交
-
-    }
-
-    private void sync(Dml dml) {
-        String database = dml.getDatabase();
-        String table = dml.getTable();
-        Map<String, ESSyncConfig> configMap;
-        if (envProperties != null && !"tcp".equalsIgnoreCase(envProperties.getProperty("canal.conf.mode"))) {
-            configMap = dbTableEsSyncConfig
-                .get(StringUtils.trimToEmpty(dml.getDestination()) + "-" + StringUtils.trimToEmpty(dml.getGroupId())
-                     + "_" + database + "-" + table);
-        } else {
-            configMap = dbTableEsSyncConfig
-                .get(StringUtils.trimToEmpty(dml.getDestination()) + "_" + database + "-" + table);
-        }
-
-        if (configMap != null && !configMap.values().isEmpty()) {
-            esSyncService.sync(configMap.values(), dml);
-        }
-    }
-
-    @Override
-    public EtlResult etl(String task, List<String> params) {
-        EtlResult etlResult = new EtlResult();
-        ESSyncConfig config = esSyncConfig.get(task);
-        if (config != null) {
-            DataSource dataSource = DatasourceConfig.DATA_SOURCES.get(config.getDataSourceKey());
-            ESEtlService esEtlService = new ESEtlService(esConnection, config);
-            if (dataSource != null) {
-                return esEtlService.importData(params);
-            } else {
-                etlResult.setSucceeded(false);
-                etlResult.setErrorMessage("DataSource not found");
-                return etlResult;
-            }
-        } else {
-            StringBuilder resultMsg = new StringBuilder();
-            boolean resSuccess = true;
-            for (ESSyncConfig configTmp : esSyncConfig.values()) {
-                // 取所有的destination为task的配置
-                if (configTmp.getDestination().equals(task)) {
-                    ESEtlService esEtlService = new ESEtlService(esConnection, configTmp);
-                    EtlResult etlRes = esEtlService.importData(params);
-                    if (!etlRes.getSucceeded()) {
-                        resSuccess = false;
-                        resultMsg.append(etlRes.getErrorMessage()).append("\n");
-                    } else {
-                        resultMsg.append(etlRes.getResultMessage()).append("\n");
-                    }
-                }
-            }
-            if (resultMsg.length() > 0) {
-                etlResult.setSucceeded(resSuccess);
-                if (resSuccess) {
-                    etlResult.setResultMessage(resultMsg.toString());
-                } else {
-                    etlResult.setErrorMessage(resultMsg.toString());
-                }
-                return etlResult;
-            }
-        }
-        etlResult.setSucceeded(false);
-        etlResult.setErrorMessage("Task not found");
-        return etlResult;
-    }
-
-    @Override
-    public Map<String, Object> count(String task) {
-        ESSyncConfig config = esSyncConfig.get(task);
-        ESMapping mapping = config.getEsMapping();
-        SearchResponse response = this.esConnection.new ESSearchRequest(mapping.get_index(), mapping.get_type()).size(0)
-            .getResponse();
-
-        long rowCount = response.getHits().getTotalHits();
-        Map<String, Object> res = new LinkedHashMap<>();
-        res.put("esIndex", mapping.get_index());
-        res.put("count", rowCount);
-        return res;
-    }
-
-    @Override
-    public void destroy() {
-        if (esConfigMonitor != null) {
-            esConfigMonitor.destroy();
-        }
-        if (esConnection != null) {
-            esConnection.close();
-        }
-    }
-
-    @Override
-    public String getDestination(String task) {
-        ESSyncConfig config = esSyncConfig.get(task);
-        if (config != null) {
-            return config.getDestination();
-        }
-        return null;
-    }
-}

client-adapter/elasticsearch/src/main/resources/META-INF/canal/com.alibaba.otter.canal.client.adapter.OuterAdapter

@@ -1 +0,0 @@
-es=com.alibaba.otter.canal.client.adapter.es.ESAdapter

client-adapter/es6x/pom.xml

@@ -1,15 +1,18 @@
 <?xml version="1.0" encoding="UTF-8"?>
-<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
+<project xmlns="http://maven.apache.org/POM/4.0.0"
+         xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
+         xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
     <parent>
         <artifactId>canal.client-adapter</artifactId>
         <groupId>com.alibaba.otter</groupId>
-        <version>1.1.4</version>
+        <version>1.1.5-SNAPSHOT</version>
+        <relativePath>../pom.xml</relativePath>
     </parent>
     <modelVersion>4.0.0</modelVersion>
     <groupId>com.alibaba.otter</groupId>
-    <artifactId>client-adapter.elasticsearch</artifactId>
+    <artifactId>client-adapter.es6x</artifactId>
     <packaging>jar</packaging>
-    <name>canal client adapter elasticsearch module for otter ${project.version}</name>
+    <name>canal client adapter es v6x module for otter ${project.version}</name>
 
     <dependencies>
         <dependency>
@@ -19,9 +22,9 @@
             <scope>provided</scope>
         </dependency>
         <dependency>
-            <groupId>com.alibaba.fastsql</groupId>
-            <artifactId>fastsql</artifactId>
-            <version>2.0.0_preview_855</version>
+            <groupId>com.alibaba.otter</groupId>
+            <artifactId>client-adapter.escore</artifactId>
+            <version>${project.version}</version>
         </dependency>
         <dependency>
             <groupId>org.elasticsearch</groupId>
@@ -47,13 +50,6 @@
         <dependency>
             <groupId>junit</groupId>
             <artifactId>junit</artifactId>
-            <version>4.12</version>
-            <scope>test</scope>
-        </dependency>
-        <dependency>
-            <groupId>mysql</groupId>
-            <artifactId>mysql-connector-java</artifactId>
-            <version>5.1.40</version>
             <scope>test</scope>
         </dependency>
     </dependencies>
@@ -89,8 +85,8 @@
                         </goals>
                         <configuration>
                             <tasks>
-                                <copy todir="${project.basedir}/../launcher/target/classes/es" overwrite="true">
-                                    <fileset dir="${project.basedir}/target/classes/es" erroronmissingdir="true">
+                                <copy todir="${project.basedir}/../launcher/target/classes/es6" overwrite="true">
+                                    <fileset dir="${project.basedir}/target/classes/es6" erroronmissingdir="true">
                                         <include name="*.yml" />
                                     </fileset>
                                 </copy>

client-adapter/es6x/src/main/java/com/alibaba/otter/canal/client/adapter/es6x/ES6xAdapter.java

@@ -0,0 +1,123 @@
+package com.alibaba.otter.canal.client.adapter.es6x;
+
+import java.util.LinkedHashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.Properties;
+
+import com.alibaba.otter.canal.client.adapter.es.core.ESAdapter;
+import com.alibaba.otter.canal.client.adapter.es.core.config.ESSyncConfig;
+import com.alibaba.otter.canal.client.adapter.es6x.etl.ESEtlService;
+import com.alibaba.otter.canal.client.adapter.es6x.support.ES6xTemplate;
+import com.alibaba.otter.canal.client.adapter.es6x.support.ESConnection;
+import com.alibaba.otter.canal.client.adapter.support.DatasourceConfig;
+import com.alibaba.otter.canal.client.adapter.support.EtlResult;
+import com.alibaba.otter.canal.client.adapter.support.OuterAdapterConfig;
+import com.alibaba.otter.canal.client.adapter.support.SPI;
+import org.elasticsearch.action.search.SearchResponse;
+
+import javax.sql.DataSource;
+
+/**
+ * ES 6.x 外部适配器
+ *
+ * @author rewerma 2019-09-23
+ * @version 1.0.0
+ */
+@SPI("es6")
+public class ES6xAdapter extends ESAdapter {
+
+    private ESConnection esConnection;
+
+    public ESConnection getEsConnection() {
+        return esConnection;
+    }
+
+    @Override
+    public void init(OuterAdapterConfig configuration, Properties envProperties) {
+        try {
+            Map<String, String> properties = configuration.getProperties();
+
+            String[] hostArray = configuration.getHosts().split(",");
+            String mode = properties.get("mode");
+            if ("rest".equalsIgnoreCase(mode) || "http".equalsIgnoreCase(mode)) {
+                esConnection = new ESConnection(hostArray, properties, ESConnection.ESClientMode.REST);
+            } else {
+                esConnection = new ESConnection(hostArray, properties, ESConnection.ESClientMode.TRANSPORT);
+            }
+            this.esTemplate = new ES6xTemplate(esConnection);
+
+            envProperties.put("es.version", "es6");
+            super.init(configuration, envProperties);
+        } catch (Throwable e) {
+            throw new RuntimeException(e);
+        }
+    }
+
+    @Override
+    public Map<String, Object> count(String task) {
+        ESSyncConfig config = esSyncConfig.get(task);
+        ESSyncConfig.ESMapping mapping = config.getEsMapping();
+        SearchResponse response = this.esConnection.new ESSearchRequest(mapping.get_index(), mapping.get_type()).size(0)
+            .getResponse();
+
+        long rowCount = response.getHits().getTotalHits();
+        Map<String, Object> res = new LinkedHashMap<>();
+        res.put("esIndex", mapping.get_index());
+        res.put("count", rowCount);
+        return res;
+    }
+
+    @Override
+    public EtlResult etl(String task, List<String> params) {
+        EtlResult etlResult = new EtlResult();
+        ESSyncConfig config = esSyncConfig.get(task);
+        if (config != null) {
+            DataSource dataSource = DatasourceConfig.DATA_SOURCES.get(config.getDataSourceKey());
+            ESEtlService esEtlService = new ESEtlService(esConnection, config);
+            if (dataSource != null) {
+                return esEtlService.importData(params);
+            } else {
+                etlResult.setSucceeded(false);
+                etlResult.setErrorMessage("DataSource not found");
+                return etlResult;
+            }
+        } else {
+            StringBuilder resultMsg = new StringBuilder();
+            boolean resSuccess = true;
+            for (ESSyncConfig configTmp : esSyncConfig.values()) {
+                // 取所有的destination为task的配置
+                if (configTmp.getDestination().equals(task)) {
+                    ESEtlService esEtlService = new ESEtlService(esConnection, configTmp);
+                    EtlResult etlRes = esEtlService.importData(params);
+                    if (!etlRes.getSucceeded()) {
+                        resSuccess = false;
+                        resultMsg.append(etlRes.getErrorMessage()).append("\n");
+                    } else {
+                        resultMsg.append(etlRes.getResultMessage()).append("\n");
+                    }
+                }
+            }
+            if (resultMsg.length() > 0) {
+                etlResult.setSucceeded(resSuccess);
+                if (resSuccess) {
+                    etlResult.setResultMessage(resultMsg.toString());
+                } else {
+                    etlResult.setErrorMessage(resultMsg.toString());
+                }
+                return etlResult;
+            }
+        }
+        etlResult.setSucceeded(false);
+        etlResult.setErrorMessage("Task not found");
+        return etlResult;
+    }
+
+    @Override
+    public void destroy() {
+        super.destroy();
+        if (esConnection != null) {
+            esConnection.close();
+        }
+    }
+}

client-adapter/es6x/src/main/java/com/alibaba/otter/canal/client/adapter/es6x/etl/ESEtlService.java

@@ -1,4 +1,4 @@
-package com.alibaba.otter.canal.client.adapter.es.service;
+package com.alibaba.otter.canal.client.adapter.es6x.etl;
 
 import java.sql.SQLException;
 import java.util.HashMap;
@@ -10,22 +10,22 @@ import java.util.concurrent.atomic.AtomicLong;
 import javax.sql.DataSource;
 
 import org.apache.commons.lang.StringUtils;
-import org.elasticsearch.action.bulk.BulkItemResponse;
 import org.elasticsearch.action.bulk.BulkResponse;
 import org.elasticsearch.action.search.SearchResponse;
 import org.elasticsearch.index.query.QueryBuilders;
-import org.elasticsearch.rest.RestStatus;
 import org.elasticsearch.search.SearchHit;
 
-import com.alibaba.otter.canal.client.adapter.es.config.ESSyncConfig;
-import com.alibaba.otter.canal.client.adapter.es.config.ESSyncConfig.ESMapping;
-import com.alibaba.otter.canal.client.adapter.es.config.SchemaItem.FieldItem;
-import com.alibaba.otter.canal.client.adapter.es.support.ESConnection;
-import com.alibaba.otter.canal.client.adapter.es.support.ESConnection.ESBulkRequest;
-import com.alibaba.otter.canal.client.adapter.es.support.ESConnection.ESIndexRequest;
-import com.alibaba.otter.canal.client.adapter.es.support.ESConnection.ESSearchRequest;
-import com.alibaba.otter.canal.client.adapter.es.support.ESConnection.ESUpdateRequest;
-import com.alibaba.otter.canal.client.adapter.es.support.ESTemplate;
+import com.alibaba.otter.canal.client.adapter.es.core.config.ESSyncConfig;
+import com.alibaba.otter.canal.client.adapter.es.core.config.ESSyncConfig.ESMapping;
+import com.alibaba.otter.canal.client.adapter.es.core.config.SchemaItem.FieldItem;
+import com.alibaba.otter.canal.client.adapter.es.core.support.ESBulkRequest;
+import com.alibaba.otter.canal.client.adapter.es.core.support.ESBulkRequest.ESBulkResponse;
+import com.alibaba.otter.canal.client.adapter.es.core.support.ESBulkRequest.ESIndexRequest;
+import com.alibaba.otter.canal.client.adapter.es.core.support.ESBulkRequest.ESUpdateRequest;
+import com.alibaba.otter.canal.client.adapter.es.core.support.ESTemplate;
+import com.alibaba.otter.canal.client.adapter.es6x.support.ES6xTemplate;
+import com.alibaba.otter.canal.client.adapter.es6x.support.ESConnection;
+import com.alibaba.otter.canal.client.adapter.es6x.support.ESConnection.ESSearchRequest;
 import com.alibaba.otter.canal.client.adapter.support.AbstractEtlService;
 import com.alibaba.otter.canal.client.adapter.support.AdapterConfig;
 import com.alibaba.otter.canal.client.adapter.support.EtlResult;
@@ -46,7 +46,7 @@ public class ESEtlService extends AbstractEtlService {
     public ESEtlService(ESConnection esConnection, ESSyncConfig config){
         super("ES", config);
         this.esConnection = esConnection;
-        this.esTemplate = new ESTemplate(esConnection);
+        this.esTemplate = new ES6xTemplate(esConnection);
         this.config = config;
     }
 
@@ -57,21 +57,6 @@ public class ESEtlService extends AbstractEtlService {
         return importData(sql, params);
     }
 
-    private void processFailBulkResponse(BulkResponse bulkResponse) {
-        for (BulkItemResponse response : bulkResponse.getItems()) {
-            if (!response.isFailed()) {
-                continue;
-            }
-
-            if (response.getFailure().getStatus() == RestStatus.NOT_FOUND) {
-                logger.warn(response.getFailureMessage());
-            } else {
-                logger.error("全量导入数据有误 {}", response.getFailureMessage());
-                throw new RuntimeException("全量数据 etl 异常: " + response.getFailureMessage());
-            }
-        }
-    }
-
     protected boolean executeSqlImport(DataSource ds, String sql, List<Object> values,
                                        AdapterConfig.AdapterMapping adapterMapping, AtomicLong impCount,
                                        List<String> errMsg) {
@@ -80,7 +65,7 @@ public class ESEtlService extends AbstractEtlService {
             Util.sqlRS(ds, sql, values, rs -> {
                 int count = 0;
                 try {
-                    ESBulkRequest esBulkRequest = this.esConnection.new ESBulkRequest();
+                    ESBulkRequest esBulkRequest = this.esConnection.new ES6xBulkRequest();
 
                     long batchBegin = System.currentTimeMillis();
                     while (rs.next()) {
@@ -133,7 +118,7 @@ public class ESEtlService extends AbstractEtlService {
                         if (idVal != null) {
                             String parentVal = (String) esFieldData.remove("$parent_routing");
                             if (mapping.isUpsert()) {
-                                ESUpdateRequest esUpdateRequest = this.esConnection.new ESUpdateRequest(
+                                ESUpdateRequest esUpdateRequest = this.esConnection.new ES6xUpdateRequest(
                                     mapping.get_index(),
                                     mapping.get_type(),
                                     idVal.toString()).setDoc(esFieldData).setDocAsUpsert(true);
@@ -144,7 +129,7 @@ public class ESEtlService extends AbstractEtlService {
 
                                 esBulkRequest.add(esUpdateRequest);
                             } else {
-                                ESIndexRequest esIndexRequest = this.esConnection.new ESIndexRequest(mapping
+                                ESIndexRequest esIndexRequest = this.esConnection.new ES6xIndexRequest(mapping
                                     .get_index(), mapping.get_type(), idVal.toString()).setSource(esFieldData);
                                 if (StringUtils.isNotEmpty(parentVal)) {
                                     esIndexRequest.setRouting(parentVal);
@@ -158,7 +143,7 @@ public class ESEtlService extends AbstractEtlService {
                                     .size(10000);
                             SearchResponse response = esSearchRequest.getResponse();
                             for (SearchHit hit : response.getHits()) {
-                                ESUpdateRequest esUpdateRequest = this.esConnection.new ESUpdateRequest(mapping
+                                ESUpdateRequest esUpdateRequest = this.esConnection.new ES6xUpdateRequest(mapping
                                     .get_index(), mapping.get_type(), hit.getId()).setDoc(esFieldData);
                                 esBulkRequest.add(esUpdateRequest);
                             }
@@ -167,9 +152,9 @@ public class ESEtlService extends AbstractEtlService {
                         if (esBulkRequest.numberOfActions() % mapping.getCommitBatch() == 0
                             && esBulkRequest.numberOfActions() > 0) {
                             long esBatchBegin = System.currentTimeMillis();
-                            BulkResponse rp = esBulkRequest.bulk();
+                            ESBulkResponse rp = esBulkRequest.bulk();
                             if (rp.hasFailures()) {
-                                this.processFailBulkResponse(rp);
+                                rp.processFailBulkResponse("全量数据 etl 异常 ");
                             }
 
                             if (logger.isTraceEnabled()) {
@@ -188,9 +173,9 @@ public class ESEtlService extends AbstractEtlService {
 
                     if (esBulkRequest.numberOfActions() > 0) {
                         long esBatchBegin = System.currentTimeMillis();
-                        BulkResponse rp = esBulkRequest.bulk();
+                        ESBulkResponse rp = esBulkRequest.bulk();
                         if (rp.hasFailures()) {
-                            this.processFailBulkResponse(rp);
+                            rp.processFailBulkResponse("全量数据 etl 异常 ");
                         }
                         if (logger.isTraceEnabled()) {
                             logger.trace("全量数据批量导入最后批次耗时: {}, es执行时间: {}, 批次大小: {}, index; {}",

client-adapter/es6x/src/main/java/com/alibaba/otter/canal/client/adapter/es6x/support/ES6xTemplate.java

@@ -0,0 +1,468 @@
+package com.alibaba.otter.canal.client.adapter.es6x.support;
+
+import java.sql.ResultSet;
+import java.sql.SQLException;
+import java.util.*;
+import java.util.concurrent.ConcurrentHashMap;
+import java.util.concurrent.ConcurrentMap;
+
+import javax.sql.DataSource;
+
+import org.apache.commons.lang.StringUtils;
+import org.elasticsearch.action.search.SearchResponse;
+import org.elasticsearch.cluster.metadata.MappingMetaData;
+import org.elasticsearch.index.query.BoolQueryBuilder;
+import org.elasticsearch.index.query.QueryBuilders;
+import org.elasticsearch.search.SearchHit;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import com.alibaba.otter.canal.client.adapter.es.core.config.ESSyncConfig;
+import com.alibaba.otter.canal.client.adapter.es.core.config.ESSyncConfig.ESMapping;
+import com.alibaba.otter.canal.client.adapter.es.core.config.SchemaItem;
+import com.alibaba.otter.canal.client.adapter.es.core.config.SchemaItem.FieldItem;
+import com.alibaba.otter.canal.client.adapter.es.core.config.SchemaItem.ColumnItem;
+import com.alibaba.otter.canal.client.adapter.es.core.support.ESBulkRequest;
+import com.alibaba.otter.canal.client.adapter.es.core.support.ESBulkRequest.ESBulkResponse;
+import com.alibaba.otter.canal.client.adapter.es.core.support.ESBulkRequest.ESDeleteRequest;
+import com.alibaba.otter.canal.client.adapter.es.core.support.ESBulkRequest.ESIndexRequest;
+import com.alibaba.otter.canal.client.adapter.es.core.support.ESBulkRequest.ESUpdateRequest;
+import com.alibaba.otter.canal.client.adapter.es.core.support.ESSyncUtil;
+import com.alibaba.otter.canal.client.adapter.es.core.support.ESTemplate;
+import com.alibaba.otter.canal.client.adapter.es6x.support.ESConnection.ESSearchRequest;
+import com.alibaba.otter.canal.client.adapter.support.DatasourceConfig;
+import com.alibaba.otter.canal.client.adapter.support.Util;
+
+public class ES6xTemplate implements ESTemplate {
+
+    private static final Logger                               logger         = LoggerFactory
+        .getLogger(ESTemplate.class);
+
+    private static final int                                  MAX_BATCH_SIZE = 1000;
+
+    private ESConnection                                      esConnection;
+
+    private ESBulkRequest                                     esBulkRequest;
+
+    // es 字段类型本地缓存
+    private static ConcurrentMap<String, Map<String, String>> esFieldTypes   = new ConcurrentHashMap<>();
+
+    public ES6xTemplate(ESConnection esConnection){
+        this.esConnection = esConnection;
+        this.esBulkRequest = this.esConnection.new ES6xBulkRequest();
+    }
+
+    public ESBulkRequest getBulk() {
+        return esBulkRequest;
+    }
+
+    public void resetBulkRequestBuilder() {
+        this.esBulkRequest.resetBulk();
+    }
+
+    @Override
+    public void insert(ESSyncConfig.ESMapping mapping, Object pkVal, Map<String, Object> esFieldData) {
+        if (mapping.get_id() != null) {
+            String parentVal = (String) esFieldData.remove("$parent_routing");
+            if (mapping.isUpsert()) {
+                ESUpdateRequest updateRequest = esConnection.new ES6xUpdateRequest(mapping.get_index(),
+                    mapping.get_type(),
+                    pkVal.toString()).setDoc(esFieldData).setDocAsUpsert(true);
+                if (StringUtils.isNotEmpty(parentVal)) {
+                    updateRequest.setRouting(parentVal);
+                }
+                getBulk().add(updateRequest);
+            } else {
+                ESIndexRequest indexRequest = esConnection.new ES6xIndexRequest(mapping.get_index(),
+                    mapping.get_type(),
+                    pkVal.toString()).setSource(esFieldData);
+                if (StringUtils.isNotEmpty(parentVal)) {
+                    indexRequest.setRouting(parentVal);
+                }
+                getBulk().add(indexRequest);
+            }
+            commitBulk();
+        } else {
+            ESSearchRequest esSearchRequest = this.esConnection.new ESSearchRequest(mapping.get_index(),
+                mapping.get_type()).setQuery(QueryBuilders.termQuery(mapping.getPk(), pkVal)).size(10000);
+            SearchResponse response = esSearchRequest.getResponse();
+
+            for (SearchHit hit : response.getHits()) {
+                ESUpdateRequest esUpdateRequest = this.esConnection.new ES6xUpdateRequest(mapping.get_index(),
+                    mapping.get_type(),
+                    hit.getId()).setDoc(esFieldData);
+                getBulk().add(esUpdateRequest);
+                commitBulk();
+            }
+        }
+    }
+
+    @Override
+    public void update(ESSyncConfig.ESMapping mapping, Object pkVal, Map<String, Object> esFieldData) {
+        Map<String, Object> esFieldDataTmp = new LinkedHashMap<>(esFieldData.size());
+        esFieldData.forEach((k, v) -> esFieldDataTmp.put(Util.cleanColumn(k), v));
+        append4Update(mapping, pkVal, esFieldDataTmp);
+        commitBulk();
+    }
+
+    @Override
+    public void updateByQuery(ESSyncConfig config, Map<String, Object> paramsTmp, Map<String, Object> esFieldData) {
+        if (paramsTmp.isEmpty()) {
+            return;
+        }
+        ESMapping mapping = config.getEsMapping();
+        BoolQueryBuilder queryBuilder = QueryBuilders.boolQuery();
+        paramsTmp.forEach((fieldName, value) -> queryBuilder.must(QueryBuilders.termsQuery(fieldName, value)));
+
+        // 查询sql批量更新
+        DataSource ds = DatasourceConfig.DATA_SOURCES.get(config.getDataSourceKey());
+        StringBuilder sql = new StringBuilder("SELECT * FROM (" + mapping.getSql() + ") _v WHERE ");
+        List<Object> values = new ArrayList<>();
+        paramsTmp.forEach((fieldName, value) -> {
+            sql.append("_v.").append(fieldName).append("=? AND ");
+            values.add(value);
+        });
+        // TODO 直接外部包裹sql会导致全表扫描性能低, 待优化拼接内部where条件
+        int len = sql.length();
+        sql.delete(len - 4, len);
+        Integer syncCount = (Integer) Util.sqlRS(ds, sql.toString(), values, rs -> {
+            int count = 0;
+            try {
+                while (rs.next()) {
+                    Object idVal = getIdValFromRS(mapping, rs);
+                    append4Update(mapping, idVal, esFieldData);
+                    commitBulk();
+                    count++;
+                }
+            } catch (Exception e) {
+                throw new RuntimeException(e);
+            }
+            return count;
+        });
+        if (logger.isTraceEnabled()) {
+            logger.trace("Update ES by query affected {} records", syncCount);
+        }
+    }
+
+    @Override
+    public void delete(ESSyncConfig.ESMapping mapping, Object pkVal, Map<String, Object> esFieldData) {
+        if (mapping.get_id() != null) {
+            ESDeleteRequest esDeleteRequest = this.esConnection.new ES6xDeleteRequest(mapping.get_index(),
+                mapping.get_type(),
+                pkVal.toString());
+            getBulk().add(esDeleteRequest);
+            commitBulk();
+        } else {
+            ESSearchRequest esSearchRequest = this.esConnection.new ESSearchRequest(mapping.get_index(),
+                mapping.get_type()).setQuery(QueryBuilders.termQuery(mapping.getPk(), pkVal)).size(10000);
+            SearchResponse response = esSearchRequest.getResponse();
+            for (SearchHit hit : response.getHits()) {
+                ESUpdateRequest esUpdateRequest = this.esConnection.new ES6xUpdateRequest(mapping.get_index(),
+                    mapping.get_type(),
+                    hit.getId()).setDoc(esFieldData);
+                getBulk().add(esUpdateRequest);
+                commitBulk();
+            }
+        }
+    }
+
+    @Override
+    public void commit() {
+        if (getBulk().numberOfActions() > 0) {
+            ESBulkResponse response = getBulk().bulk();
+            if (response.hasFailures()) {
+                response.processFailBulkResponse("ES sync commit error ");
+            }
+            resetBulkRequestBuilder();
+        }
+    }
+
+    @Override
+    public Object getValFromRS(ESSyncConfig.ESMapping mapping, ResultSet resultSet, String fieldName,
+                               String columnName) throws SQLException {
+        fieldName = Util.cleanColumn(fieldName);
+        columnName = Util.cleanColumn(columnName);
+        String esType = getEsType(mapping, fieldName);
+
+        Object value = resultSet.getObject(columnName);
+        if (value instanceof Boolean) {
+            if (!"boolean".equals(esType)) {
+                value = resultSet.getByte(columnName);
+            }
+        }
+
+        // 如果是对象类型
+        if (mapping.getObjFields().containsKey(fieldName)) {
+            return ESSyncUtil.convertToEsObj(value, mapping.getObjFields().get(fieldName));
+        } else {
+            return ESSyncUtil.typeConvert(value, esType);
+        }
+    }
+
+    @Override
+    public Object getESDataFromRS(ESSyncConfig.ESMapping mapping, ResultSet resultSet,
+                                  Map<String, Object> esFieldData) throws SQLException {
+        SchemaItem schemaItem = mapping.getSchemaItem();
+        String idFieldName = mapping.get_id() == null ? mapping.getPk() : mapping.get_id();
+        Object resultIdVal = null;
+        for (FieldItem fieldItem : schemaItem.getSelectFields().values()) {
+            Object value = getValFromRS(mapping, resultSet, fieldItem.getFieldName(), fieldItem.getFieldName());
+
+            if (fieldItem.getFieldName().equals(idFieldName)) {
+                resultIdVal = value;
+            }
+
+            if (!fieldItem.getFieldName().equals(mapping.get_id())
+                    && !mapping.getSkips().contains(fieldItem.getFieldName())) {
+                esFieldData.put(Util.cleanColumn(fieldItem.getFieldName()), value);
+            }
+        }
+
+        // 添加父子文档关联信息
+        putRelationDataFromRS(mapping, schemaItem, resultSet, esFieldData);
+
+        return resultIdVal;
+    }
+
+    @Override
+    public Object getIdValFromRS(ESSyncConfig.ESMapping mapping, ResultSet resultSet) throws SQLException {
+        SchemaItem schemaItem = mapping.getSchemaItem();
+        String idFieldName = mapping.get_id() == null ? mapping.getPk() : mapping.get_id();
+        Object resultIdVal = null;
+        for (FieldItem fieldItem : schemaItem.getSelectFields().values()) {
+            Object value = getValFromRS(mapping, resultSet, fieldItem.getFieldName(), fieldItem.getFieldName());
+
+            if (fieldItem.getFieldName().equals(idFieldName)) {
+                resultIdVal = value;
+                break;
+            }
+        }
+        return resultIdVal;
+    }
+
+    @Override
+    public Object getESDataFromRS(ESSyncConfig.ESMapping mapping, ResultSet resultSet, Map<String, Object> dmlOld,
+                                  Map<String, Object> esFieldData) throws SQLException {
+        SchemaItem schemaItem = mapping.getSchemaItem();
+        String idFieldName = mapping.get_id() == null ? mapping.getPk() : mapping.get_id();
+        Object resultIdVal = null;
+        for (FieldItem fieldItem : schemaItem.getSelectFields().values()) {
+            if (fieldItem.getFieldName().equals(idFieldName)) {
+                resultIdVal = getValFromRS(mapping, resultSet, fieldItem.getFieldName(), fieldItem.getFieldName());
+            }
+
+            for (ColumnItem columnItem : fieldItem.getColumnItems()) {
+                if (dmlOld.containsKey(columnItem.getColumnName())
+                        && !mapping.getSkips().contains(fieldItem.getFieldName())) {
+                    esFieldData.put(Util.cleanColumn(fieldItem.getFieldName()),
+                            getValFromRS(mapping, resultSet, fieldItem.getFieldName(), fieldItem.getFieldName()));
+                    break;
+                }
+            }
+        }
+
+        // 添加父子文档关联信息
+        putRelationDataFromRS(mapping, schemaItem, resultSet, esFieldData);
+
+        return resultIdVal;
+    }
+
+    @Override
+    public Object getValFromData(ESSyncConfig.ESMapping mapping, Map<String, Object> dmlData, String fieldName,
+                                 String columnName) {
+        String esType = getEsType(mapping, fieldName);
+        Object value = dmlData.get(columnName);
+        if (value instanceof Byte) {
+            if ("boolean".equals(esType)) {
+                value = ((Byte) value).intValue() != 0;
+            }
+        }
+
+        // 如果是对象类型
+        if (mapping.getObjFields().containsKey(fieldName)) {
+            return ESSyncUtil.convertToEsObj(value, mapping.getObjFields().get(fieldName));
+        } else {
+            return ESSyncUtil.typeConvert(value, esType);
+        }
+    }
+
+    @Override
+    public Object getESDataFromDmlData(ESSyncConfig.ESMapping mapping, Map<String, Object> dmlData,
+                                       Map<String, Object> esFieldData) {
+        SchemaItem schemaItem = mapping.getSchemaItem();
+        String idFieldName = mapping.get_id() == null ? mapping.getPk() : mapping.get_id();
+        Object resultIdVal = null;
+        for (FieldItem fieldItem : schemaItem.getSelectFields().values()) {
+            String columnName = fieldItem.getColumnItems().iterator().next().getColumnName();
+            Object value = getValFromData(mapping, dmlData, fieldItem.getFieldName(), columnName);
+
+            if (fieldItem.getFieldName().equals(idFieldName)) {
+                resultIdVal = value;
+            }
+
+            if (!fieldItem.getFieldName().equals(mapping.get_id())
+                    && !mapping.getSkips().contains(fieldItem.getFieldName())) {
+                esFieldData.put(Util.cleanColumn(fieldItem.getFieldName()), value);
+            }
+        }
+
+        // 添加父子文档关联信息
+        putRelationData(mapping, schemaItem, dmlData, esFieldData);
+        return resultIdVal;
+    }
+
+    @Override
+    public Object getESDataFromDmlData(ESSyncConfig.ESMapping mapping, Map<String, Object> dmlData,
+                                       Map<String, Object> dmlOld, Map<String, Object> esFieldData) {
+        SchemaItem schemaItem = mapping.getSchemaItem();
+        String idFieldName = mapping.get_id() == null ? mapping.getPk() : mapping.get_id();
+        Object resultIdVal = null;
+        for (FieldItem fieldItem : schemaItem.getSelectFields().values()) {
+            String columnName = fieldItem.getColumnItems().iterator().next().getColumnName();
+
+            if (fieldItem.getFieldName().equals(idFieldName)) {
+                resultIdVal = getValFromData(mapping, dmlData, fieldItem.getFieldName(), columnName);
+            }
+
+            if (dmlOld.containsKey(columnName) && !mapping.getSkips().contains(fieldItem.getFieldName())) {
+                esFieldData.put(Util.cleanColumn(fieldItem.getFieldName()),
+                        getValFromData(mapping, dmlData, fieldItem.getFieldName(), columnName));
+            }
+        }
+
+        // 添加父子文档关联信息
+        putRelationData(mapping, schemaItem, dmlOld, esFieldData);
+        return resultIdVal;
+    }
+
+    /**
+     * 如果大于批量数则提交批次
+     */
+    private void commitBulk() {
+        if (getBulk().numberOfActions() >= MAX_BATCH_SIZE) {
+            commit();
+        }
+    }
+
+    private void append4Update(ESMapping mapping, Object pkVal, Map<String, Object> esFieldData) {
+        if (mapping.get_id() != null) {
+            String parentVal = (String) esFieldData.remove("$parent_routing");
+            if (mapping.isUpsert()) {
+                ESUpdateRequest esUpdateRequest = this.esConnection.new ES6xUpdateRequest(mapping.get_index(),
+                    mapping.get_type(),
+                    pkVal.toString()).setDoc(esFieldData).setDocAsUpsert(true);
+                if (StringUtils.isNotEmpty(parentVal)) {
+                    esUpdateRequest.setRouting(parentVal);
+                }
+                getBulk().add(esUpdateRequest);
+            } else {
+                ESUpdateRequest esUpdateRequest = this.esConnection.new ES6xUpdateRequest(mapping.get_index(),
+                    mapping.get_type(),
+                    pkVal.toString()).setDoc(esFieldData);
+                if (StringUtils.isNotEmpty(parentVal)) {
+                    esUpdateRequest.setRouting(parentVal);
+                }
+                getBulk().add(esUpdateRequest);
+            }
+        } else {
+            ESSearchRequest esSearchRequest = this.esConnection.new ESSearchRequest(mapping.get_index(),
+                mapping.get_type()).setQuery(QueryBuilders.termQuery(mapping.getPk(), pkVal)).size(10000);
+            SearchResponse response = esSearchRequest.getResponse();
+            for (SearchHit hit : response.getHits()) {
+                ESUpdateRequest esUpdateRequest = this.esConnection.new ES6xUpdateRequest(mapping.get_index(),
+                    mapping.get_type(),
+                    hit.getId()).setDoc(esFieldData);
+                getBulk().add(esUpdateRequest);
+            }
+        }
+    }
+
+    /**
+     * 获取es mapping中的属性类型
+     *
+     * @param mapping mapping配置
+     * @param fieldName 属性名
+     * @return 类型
+     */
+    @SuppressWarnings("unchecked")
+    private String getEsType(ESMapping mapping, String fieldName) {
+        String key = mapping.get_index() + "-" + mapping.get_type();
+        Map<String, String> fieldType = esFieldTypes.get(key);
+        if (fieldType != null) {
+            return fieldType.get(fieldName);
+        } else {
+            MappingMetaData mappingMetaData = esConnection.getMapping(mapping.get_index(), mapping.get_type());
+
+            if (mappingMetaData == null) {
+                throw new IllegalArgumentException("Not found the mapping info of index: " + mapping.get_index());
+            }
+
+            fieldType = new LinkedHashMap<>();
+
+            Map<String, Object> sourceMap = mappingMetaData.getSourceAsMap();
+            Map<String, Object> esMapping = (Map<String, Object>) sourceMap.get("properties");
+            for (Map.Entry<String, Object> entry : esMapping.entrySet()) {
+                Map<String, Object> value = (Map<String, Object>) entry.getValue();
+                if (value.containsKey("properties")) {
+                    fieldType.put(entry.getKey(), "object");
+                } else {
+                    fieldType.put(entry.getKey(), (String) value.get("type"));
+                }
+            }
+            esFieldTypes.put(key, fieldType);
+
+            return fieldType.get(fieldName);
+        }
+    }
+
+    private void putRelationDataFromRS(ESMapping mapping, SchemaItem schemaItem, ResultSet resultSet,
+                                       Map<String, Object> esFieldData) {
+        // 添加父子文档关联信息
+        if (!mapping.getRelations().isEmpty()) {
+            mapping.getRelations().forEach((relationField, relationMapping) -> {
+                Map<String, Object> relations = new HashMap<>();
+                relations.put("name", relationMapping.getName());
+                if (StringUtils.isNotEmpty(relationMapping.getParent())) {
+                    FieldItem parentFieldItem = schemaItem.getSelectFields().get(relationMapping.getParent());
+                    Object parentVal;
+                    try {
+                        parentVal = getValFromRS(mapping,
+                                resultSet,
+                                parentFieldItem.getFieldName(),
+                                parentFieldItem.getFieldName());
+                    } catch (SQLException e) {
+                        throw new RuntimeException(e);
+                    }
+                    if (parentVal != null) {
+                        relations.put("parent", parentVal.toString());
+                        esFieldData.put("$parent_routing", parentVal.toString());
+
+                    }
+                }
+                esFieldData.put(relationField, relations);
+            });
+        }
+    }
+
+    private void putRelationData(ESMapping mapping, SchemaItem schemaItem, Map<String, Object> dmlData,
+                                 Map<String, Object> esFieldData) {
+        // 添加父子文档关联信息
+        if (!mapping.getRelations().isEmpty()) {
+            mapping.getRelations().forEach((relationField, relationMapping) -> {
+                Map<String, Object> relations = new HashMap<>();
+                relations.put("name", relationMapping.getName());
+                if (StringUtils.isNotEmpty(relationMapping.getParent())) {
+                    FieldItem parentFieldItem = schemaItem.getSelectFields().get(relationMapping.getParent());
+                    String columnName = parentFieldItem.getColumnItems().iterator().next().getColumnName();
+                    Object parentVal = getValFromData(mapping, dmlData, parentFieldItem.getFieldName(), columnName);
+                    if (parentVal != null) {
+                        relations.put("parent", parentVal.toString());
+                        esFieldData.put("$parent_routing", parentVal.toString());
+
+                    }
+                }
+                esFieldData.put(relationField, relations);
+            });
+        }
+    }
+}

client-adapter/es6x/src/main/java/com/alibaba/otter/canal/client/adapter/es6x/support/ESConnection.java

@@ -1,10 +1,11 @@
-package com.alibaba.otter.canal.client.adapter.es.support;
+package com.alibaba.otter.canal.client.adapter.es6x.support;
 
 import java.io.IOException;
 import java.net.InetAddress;
 import java.net.UnknownHostException;
 import java.util.Map;
 
+import com.alibaba.otter.canal.client.adapter.es.core.support.ESBulkRequest;
 import org.apache.commons.lang.StringUtils;
 import org.apache.http.HttpHost;
 import org.apache.http.auth.AuthScope;
@@ -13,6 +14,7 @@ import org.apache.http.client.CredentialsProvider;
 import org.apache.http.impl.client.BasicCredentialsProvider;
 import org.elasticsearch.action.admin.indices.mapping.get.GetMappingsRequest;
 import org.elasticsearch.action.admin.indices.mapping.get.GetMappingsResponse;
+import org.elasticsearch.action.bulk.BulkItemResponse;
 import org.elasticsearch.action.bulk.BulkRequest;
 import org.elasticsearch.action.bulk.BulkRequestBuilder;
 import org.elasticsearch.action.bulk.BulkResponse;
@@ -32,6 +34,7 @@ import org.elasticsearch.common.collect.ImmutableOpenMap;
 import org.elasticsearch.common.settings.Settings;
 import org.elasticsearch.common.transport.TransportAddress;
 import org.elasticsearch.index.query.QueryBuilder;
+import org.elasticsearch.rest.RestStatus;
 import org.elasticsearch.search.builder.SearchSourceBuilder;
 import org.elasticsearch.transport.client.PreBuiltTransportClient;
 import org.slf4j.Logger;
@@ -61,7 +64,7 @@ public class ESConnection {
         this.mode = mode;
         if (mode == ESClientMode.TRANSPORT) {
             Settings.Builder settingBuilder = Settings.builder();
-            properties.forEach(settingBuilder::put);
+            settingBuilder.put("cluster.name", properties.get("cluster.name"));
             Settings settings = settingBuilder.build();
             transportClient = new PreBuiltTransportClient(settings);
             for (String host : hosts) {
@@ -152,13 +155,13 @@ public class ESConnection {
         return mappingMetaData;
     }
 
-    public class ESIndexRequest {
+    public class ES6xIndexRequest implements ESBulkRequest.ESIndexRequest {
 
         private IndexRequestBuilder indexRequestBuilder;
 
         private IndexRequest        indexRequest;
 
-        public ESIndexRequest(String index, String type, String id){
+        public ES6xIndexRequest(String index, String type, String id){
             if (mode == ESClientMode.TRANSPORT) {
                 indexRequestBuilder = transportClient.prepareIndex(index, type, id);
             } else {
@@ -166,7 +169,7 @@ public class ESConnection {
             }
         }
 
-        public ESIndexRequest setSource(Map<String, ?> source) {
+        public ES6xIndexRequest setSource(Map<String, ?> source) {
             if (mode == ESClientMode.TRANSPORT) {
                 indexRequestBuilder.setSource(source);
             } else {
@@ -175,7 +178,7 @@ public class ESConnection {
             return this;
         }
 
-        public ESIndexRequest setRouting(String routing) {
+        public ES6xIndexRequest setRouting(String routing) {
             if (mode == ESClientMode.TRANSPORT) {
                 indexRequestBuilder.setRouting(routing);
             } else {
@@ -201,13 +204,13 @@ public class ESConnection {
         }
     }
 
-    public class ESUpdateRequest {
+    public class ES6xUpdateRequest implements ESBulkRequest.ESUpdateRequest {
 
         private UpdateRequestBuilder updateRequestBuilder;
 
         private UpdateRequest        updateRequest;
 
-        public ESUpdateRequest(String index, String type, String id){
+        public ES6xUpdateRequest(String index, String type, String id){
             if (mode == ESClientMode.TRANSPORT) {
                 updateRequestBuilder = transportClient.prepareUpdate(index, type, id);
             } else {
@@ -215,7 +218,7 @@ public class ESConnection {
             }
         }
 
-        public ESUpdateRequest setDoc(Map source) {
+        public ES6xUpdateRequest setDoc(Map source) {
             if (mode == ESClientMode.TRANSPORT) {
                 updateRequestBuilder.setDoc(source);
             } else {
@@ -224,7 +227,7 @@ public class ESConnection {
             return this;
         }
 
-        public ESUpdateRequest setDocAsUpsert(boolean shouldUpsertDoc) {
+        public ES6xUpdateRequest setDocAsUpsert(boolean shouldUpsertDoc) {
             if (mode == ESClientMode.TRANSPORT) {
                 updateRequestBuilder.setDocAsUpsert(shouldUpsertDoc);
             } else {
@@ -233,7 +236,7 @@ public class ESConnection {
             return this;
         }
 
-        public ESUpdateRequest setRouting(String routing) {
+        public ES6xUpdateRequest setRouting(String routing) {
             if (mode == ESClientMode.TRANSPORT) {
                 updateRequestBuilder.setRouting(routing);
             } else {
@@ -259,13 +262,13 @@ public class ESConnection {
         }
     }
 
-    public class ESDeleteRequest {
+    public class ES6xDeleteRequest implements ESBulkRequest.ESDeleteRequest {
 
         private DeleteRequestBuilder deleteRequestBuilder;
 
         private DeleteRequest        deleteRequest;
 
-        public ESDeleteRequest(String index, String type, String id){
+        public ES6xDeleteRequest(String index, String type, String id){
             if (mode == ESClientMode.TRANSPORT) {
                 deleteRequestBuilder = transportClient.prepareDelete(index, type, id);
             } else {
@@ -355,13 +358,13 @@ public class ESConnection {
         }
     }
 
-    public class ESBulkRequest {
+    public class ES6xBulkRequest implements ESBulkRequest {
 
         private BulkRequestBuilder bulkRequestBuilder;
 
         private BulkRequest        bulkRequest;
 
-        public ESBulkRequest(){
+        public ES6xBulkRequest(){
             if (mode == ESClientMode.TRANSPORT) {
                 bulkRequestBuilder = transportClient.prepareBulk();
             } else {
@@ -377,29 +380,32 @@ public class ESConnection {
             }
         }
 
-        public ESBulkRequest add(ESIndexRequest esIndexRequest) {
+        public ES6xBulkRequest add(ESIndexRequest esIndexRequest) {
+            ES6xIndexRequest eir = (ES6xIndexRequest) esIndexRequest;
             if (mode == ESClientMode.TRANSPORT) {
-                bulkRequestBuilder.add(esIndexRequest.indexRequestBuilder);
+                bulkRequestBuilder.add(eir.indexRequestBuilder);
             } else {
-                bulkRequest.add(esIndexRequest.indexRequest);
+                bulkRequest.add(eir.indexRequest);
             }
             return this;
         }
 
-        public ESBulkRequest add(ESUpdateRequest esUpdateRequest) {
+        public ES6xBulkRequest add(ESUpdateRequest esUpdateRequest) {
+            ES6xUpdateRequest eur = (ES6xUpdateRequest) esUpdateRequest;
             if (mode == ESClientMode.TRANSPORT) {
-                bulkRequestBuilder.add(esUpdateRequest.updateRequestBuilder);
+                bulkRequestBuilder.add(eur.updateRequestBuilder);
             } else {
-                bulkRequest.add(esUpdateRequest.updateRequest);
+                bulkRequest.add(eur.updateRequest);
             }
             return this;
         }
 
-        public ESBulkRequest add(ESDeleteRequest esDeleteRequest) {
+        public ES6xBulkRequest add(ESDeleteRequest esDeleteRequest) {
+            ES6xDeleteRequest edr = (ES6xDeleteRequest) esDeleteRequest;
             if (mode == ESClientMode.TRANSPORT) {
-                bulkRequestBuilder.add(esDeleteRequest.deleteRequestBuilder);
+                bulkRequestBuilder.add(edr.deleteRequestBuilder);
             } else {
-                bulkRequest.add(esDeleteRequest.deleteRequest);
+                bulkRequest.add(edr.deleteRequest);
             }
             return this;
         }
@@ -412,12 +418,14 @@ public class ESConnection {
             }
         }
 
-        public BulkResponse bulk() {
+        public ESBulkResponse bulk() {
             if (mode == ESClientMode.TRANSPORT) {
-                return bulkRequestBuilder.execute().actionGet();
+                BulkResponse responses = bulkRequestBuilder.execute().actionGet();
+                return new ES6xBulkResponse(responses);
             } else {
                 try {
-                    return restHighLevelClient.bulk(bulkRequest);
+                    BulkResponse responses = restHighLevelClient.bulk(bulkRequest);
+                    return new ES6xBulkResponse(responses);
                 } catch (IOException e) {
                     throw new RuntimeException(e);
                 }
@@ -441,6 +449,35 @@ public class ESConnection {
         }
     }
 
+    public static class ES6xBulkResponse implements ESBulkRequest.ESBulkResponse {
+
+        private BulkResponse bulkResponse;
+
+        public ES6xBulkResponse(BulkResponse bulkResponse){
+            this.bulkResponse = bulkResponse;
+        }
+
+        @Override
+        public boolean hasFailures() {
+            return bulkResponse.hasFailures();
+        }
+
+        @Override
+        public void processFailBulkResponse(String errorMsg) {
+            for (BulkItemResponse itemResponse : bulkResponse.getItems()) {
+                if (!itemResponse.isFailed()) {
+                    continue;
+                }
+
+                if (itemResponse.getFailure().getStatus() == RestStatus.NOT_FOUND) {
+                    logger.error(itemResponse.getFailureMessage());
+                } else {
+                    throw new RuntimeException(errorMsg + itemResponse.getFailureMessage());
+                }
+            }
+        }
+    }
+
     // ------ get/set ------
     public ESClientMode getMode() {
         return mode;

client-adapter/es6x/src/main/java/org/elasticsearch/client/RequestConvertersExt.java

@@ -1,11 +1,11 @@
 package org.elasticsearch.client;
 
+import java.io.IOException;
+
 import org.apache.http.client.methods.HttpGet;
 import org.elasticsearch.action.admin.indices.mapping.get.GetMappingsRequest;
 import org.elasticsearch.common.Strings;
 
-import java.io.IOException;
-
 /**
  * RequestConverters扩展
  *

client-adapter/es6x/src/main/resources/META-INF/canal/com.alibaba.otter.canal.client.adapter.OuterAdapter

@@ -0,0 +1 @@
+es6=com.alibaba.otter.canal.client.adapter.es6x.ES6xAdapter

client-adapter/es6x/src/test/java/com/alibaba/otter/canal/client/adapter/es6x/test/ConfigLoadTest.java

@@ -1,4 +1,4 @@
-package com.alibaba.otter.canal.client.adapter.es.test;
+package com.alibaba.otter.canal.client.adapter.es6x.test;
 
 import java.util.Map;
 
@@ -7,9 +7,10 @@ import org.junit.Before;
 import org.junit.Ignore;
 import org.junit.Test;
 
-import com.alibaba.otter.canal.client.adapter.es.config.ESSyncConfig;
-import com.alibaba.otter.canal.client.adapter.es.config.ESSyncConfigLoader;
+import com.alibaba.otter.canal.client.adapter.es.core.config.ESSyncConfig;
+import com.alibaba.otter.canal.client.adapter.es.core.config.ESSyncConfigLoader;
 import com.alibaba.otter.canal.client.adapter.support.DatasourceConfig;
+
 @Ignore
 public class ConfigLoadTest {
 

client-adapter/es6x/src/test/java/com/alibaba/otter/canal/client/adapter/es6x/test/ESTest.java

@@ -1,4 +1,4 @@
-package com.alibaba.otter.canal.client.adapter.es.test;
+package com.alibaba.otter.canal.client.adapter.es6x.test;
 
 import java.net.InetAddress;
 import java.net.UnknownHostException;
@@ -20,6 +20,7 @@ import org.junit.After;
 import org.junit.Before;
 import org.junit.Ignore;
 import org.junit.Test;
+
 @Ignore
 public class ESTest {
 

client-adapter/es6x/src/test/java/com/alibaba/otter/canal/client/adapter/es6x/test/SqlParseTest.java

@@ -1,4 +1,4 @@
-package com.alibaba.otter.canal.client.adapter.es.test;
+package com.alibaba.otter.canal.client.adapter.es6x.test;
 
 import java.util.List;
 import java.util.Map;
@@ -6,10 +6,10 @@ import java.util.Map;
 import org.junit.Assert;
 import org.junit.Test;
 
-import com.alibaba.otter.canal.client.adapter.es.config.SchemaItem;
-import com.alibaba.otter.canal.client.adapter.es.config.SchemaItem.FieldItem;
-import com.alibaba.otter.canal.client.adapter.es.config.SchemaItem.TableItem;
-import com.alibaba.otter.canal.client.adapter.es.config.SqlParser;
+import com.alibaba.otter.canal.client.adapter.es.core.config.SchemaItem;
+import com.alibaba.otter.canal.client.adapter.es.core.config.SchemaItem.FieldItem;
+import com.alibaba.otter.canal.client.adapter.es.core.config.SchemaItem.TableItem;
+import com.alibaba.otter.canal.client.adapter.es.core.config.SqlParser;
 
 public class SqlParseTest {
 

client-adapter/es6x/src/test/java/com/alibaba/otter/canal/client/adapter/es6x/test/TestConstant.java

@@ -1,4 +1,4 @@
-package com.alibaba.otter.canal.client.adapter.es.test;
+package com.alibaba.otter.canal.client.adapter.es6x.test;
 
 import java.sql.SQLException;
 

client-adapter/es6x/src/test/java/com/alibaba/otter/canal/client/adapter/es6x/test/sync/Common.java

@@ -1,4 +1,4 @@
-package com.alibaba.otter.canal.client.adapter.es.test.sync;
+package com.alibaba.otter.canal.client.adapter.es6x.test.sync;
 
 import java.sql.Connection;
 import java.sql.SQLException;
@@ -8,14 +8,14 @@ import java.util.Map;
 
 import javax.sql.DataSource;
 
-import com.alibaba.otter.canal.client.adapter.es.ESAdapter;
-import com.alibaba.otter.canal.client.adapter.es.test.TestConstant;
+import com.alibaba.otter.canal.client.adapter.es6x.ES6xAdapter;
+import com.alibaba.otter.canal.client.adapter.es6x.test.TestConstant;
 import com.alibaba.otter.canal.client.adapter.support.DatasourceConfig;
 import com.alibaba.otter.canal.client.adapter.support.OuterAdapterConfig;
 
 public class Common {
 
-    public static ESAdapter init() {
+    public static ES6xAdapter init() {
         DatasourceConfig.DATA_SOURCES.put("defaultDS", TestConstant.dataSource);
 
         OuterAdapterConfig outerAdapterConfig = new OuterAdapterConfig();
@@ -25,7 +25,7 @@ public class Common {
         properties.put("cluster.name", TestConstant.clusterName);
         outerAdapterConfig.setProperties(properties);
 
-        ESAdapter esAdapter = new ESAdapter();
+        ES6xAdapter esAdapter = new ES6xAdapter();
         esAdapter.init(outerAdapterConfig, null);
         return esAdapter;
     }

client-adapter/es6x/src/test/java/com/alibaba/otter/canal/client/adapter/es6x/test/sync/LabelSyncJoinSub2Test.java

@@ -1,10 +1,6 @@
-package com.alibaba.otter.canal.client.adapter.es.test.sync;
+package com.alibaba.otter.canal.client.adapter.es6x.test.sync;
 
-import java.util.ArrayList;
-import java.util.Date;
-import java.util.LinkedHashMap;
-import java.util.List;
-import java.util.Map;
+import java.util.*;
 
 import javax.sql.DataSource;
 
@@ -14,15 +10,15 @@ import org.junit.Before;
 import org.junit.Ignore;
 import org.junit.Test;
 
-import com.alibaba.otter.canal.client.adapter.es.ESAdapter;
-import com.alibaba.otter.canal.client.adapter.es.config.ESSyncConfig;
+import com.alibaba.otter.canal.client.adapter.es.core.config.ESSyncConfig;
+import com.alibaba.otter.canal.client.adapter.es6x.ES6xAdapter;
 import com.alibaba.otter.canal.client.adapter.support.DatasourceConfig;
 import com.alibaba.otter.canal.client.adapter.support.Dml;
 
 @Ignore
 public class LabelSyncJoinSub2Test {
 
-    private ESAdapter esAdapter;
+    private ES6xAdapter esAdapter;
 
     @Before
     public void init() {

client-adapter/es6x/src/test/java/com/alibaba/otter/canal/client/adapter/es6x/test/sync/LabelSyncJoinSubTest.java

@@ -1,10 +1,6 @@
-package com.alibaba.otter.canal.client.adapter.es.test.sync;
+package com.alibaba.otter.canal.client.adapter.es6x.test.sync;
 
-import java.util.ArrayList;
-import java.util.Date;
-import java.util.LinkedHashMap;
-import java.util.List;
-import java.util.Map;
+import java.util.*;
 
 import javax.sql.DataSource;
 
@@ -14,15 +10,15 @@ import org.junit.Before;
 import org.junit.Ignore;
 import org.junit.Test;
 
-import com.alibaba.otter.canal.client.adapter.es.ESAdapter;
-import com.alibaba.otter.canal.client.adapter.es.config.ESSyncConfig;
+import com.alibaba.otter.canal.client.adapter.es.core.config.ESSyncConfig;
+import com.alibaba.otter.canal.client.adapter.es6x.ES6xAdapter;
 import com.alibaba.otter.canal.client.adapter.support.DatasourceConfig;
 import com.alibaba.otter.canal.client.adapter.support.Dml;
 
 @Ignore
 public class LabelSyncJoinSubTest {
 
-    private ESAdapter esAdapter;
+    private ES6xAdapter esAdapter;
 
     @Before
     public void init() {

client-adapter/es6x/src/test/java/com/alibaba/otter/canal/client/adapter/es6x/test/sync/RoleSyncJoinOne2Test.java

@@ -1,10 +1,6 @@
-package com.alibaba.otter.canal.client.adapter.es.test.sync;
+package com.alibaba.otter.canal.client.adapter.es6x.test.sync;
 
-import java.util.ArrayList;
-import java.util.Date;
-import java.util.LinkedHashMap;
-import java.util.List;
-import java.util.Map;
+import java.util.*;
 
 import javax.sql.DataSource;
 
@@ -14,15 +10,15 @@ import org.junit.Before;
 import org.junit.Ignore;
 import org.junit.Test;
 
-import com.alibaba.otter.canal.client.adapter.es.ESAdapter;
-import com.alibaba.otter.canal.client.adapter.es.config.ESSyncConfig;
+import com.alibaba.otter.canal.client.adapter.es.core.config.ESSyncConfig;
+import com.alibaba.otter.canal.client.adapter.es6x.ES6xAdapter;
 import com.alibaba.otter.canal.client.adapter.support.DatasourceConfig;
 import com.alibaba.otter.canal.client.adapter.support.Dml;
 
 @Ignore
 public class RoleSyncJoinOne2Test {
 
-    private ESAdapter esAdapter;
+    private ES6xAdapter esAdapter;
 
     @Before
     public void init() {

client-adapter/es6x/src/test/java/com/alibaba/otter/canal/client/adapter/es6x/test/sync/RoleSyncJoinOneTest.java

@@ -1,10 +1,6 @@
-package com.alibaba.otter.canal.client.adapter.es.test.sync;
+package com.alibaba.otter.canal.client.adapter.es6x.test.sync;
 
-import java.util.ArrayList;
-import java.util.Date;
-import java.util.LinkedHashMap;
-import java.util.List;
-import java.util.Map;
+import java.util.*;
 
 import javax.sql.DataSource;
 
@@ -14,14 +10,15 @@ import org.junit.Before;
 import org.junit.Ignore;
 import org.junit.Test;
 
-import com.alibaba.otter.canal.client.adapter.es.ESAdapter;
-import com.alibaba.otter.canal.client.adapter.es.config.ESSyncConfig;
+import com.alibaba.otter.canal.client.adapter.es.core.config.ESSyncConfig;
+import com.alibaba.otter.canal.client.adapter.es6x.ES6xAdapter;
 import com.alibaba.otter.canal.client.adapter.support.DatasourceConfig;
 import com.alibaba.otter.canal.client.adapter.support.Dml;
+
 @Ignore
 public class RoleSyncJoinOneTest {
 
-    private ESAdapter esAdapter;
+    private ES6xAdapter esAdapter;
 
     @Before
     public void init() {
@@ -58,7 +55,10 @@ public class RoleSyncJoinOneTest {
 
         esAdapter.getEsSyncService().sync(esSyncConfigs.values(), dml);
 
-        GetResponse response = esAdapter.getEsConnection().getTransportClient().prepareGet("mytest_user", "_doc", "1").get();
+        GetResponse response = esAdapter.getEsConnection()
+            .getTransportClient()
+            .prepareGet("mytest_user", "_doc", "1")
+            .get();
         Assert.assertEquals("admin", response.getSource().get("_role_name"));
     }
 
@@ -95,7 +95,10 @@ public class RoleSyncJoinOneTest {
 
         esAdapter.getEsSyncService().sync(esSyncConfigs.values(), dml);
 
-        GetResponse response = esAdapter.getEsConnection().getTransportClient().prepareGet("mytest_user", "_doc", "1").get();
+        GetResponse response = esAdapter.getEsConnection()
+            .getTransportClient()
+            .prepareGet("mytest_user", "_doc", "1")
+            .get();
         Assert.assertEquals("admin2", response.getSource().get("_role_name"));
     }
 
@@ -133,7 +136,10 @@ public class RoleSyncJoinOneTest {
 
         esAdapter.getEsSyncService().sync(esSyncConfigs.values(), dml);
 
-        GetResponse response = esAdapter.getEsConnection().getTransportClient().prepareGet("mytest_user", "_doc", "1").get();
+        GetResponse response = esAdapter.getEsConnection()
+            .getTransportClient()
+            .prepareGet("mytest_user", "_doc", "1")
+            .get();
         Assert.assertEquals("operator", response.getSource().get("_role_name"));
 
         Common.sqlExe(ds, "update user set role_id=1 where id=1");
@@ -158,7 +164,10 @@ public class RoleSyncJoinOneTest {
 
         esAdapter.getEsSyncService().sync(esSyncConfigs.values(), dml2);
 
-        GetResponse response2 = esAdapter.getEsConnection().getTransportClient().prepareGet("mytest_user", "_doc", "1").get();
+        GetResponse response2 = esAdapter.getEsConnection()
+            .getTransportClient()
+            .prepareGet("mytest_user", "_doc", "1")
+            .get();
         Assert.assertEquals("admin2", response2.getSource().get("_role_name"));
     }
 
@@ -191,7 +200,9 @@ public class RoleSyncJoinOneTest {
         esAdapter.getEsSyncService().sync(esSyncConfigs.values(), dml);
 
         GetResponse response = esAdapter.getEsConnection()
-            .getTransportClient().prepareGet("mytest_user", "_doc", "1").get();
+            .getTransportClient()
+            .prepareGet("mytest_user", "_doc", "1")
+            .get();
         Assert.assertNull(response.getSource().get("_role_name"));
     }
 }

client-adapter/es6x/src/test/java/com/alibaba/otter/canal/client/adapter/es6x/test/sync/UserSyncJoinOneTest.java

@@ -1,10 +1,6 @@
-package com.alibaba.otter.canal.client.adapter.es.test.sync;
+package com.alibaba.otter.canal.client.adapter.es6x.test.sync;
 
-import java.util.ArrayList;
-import java.util.Date;
-import java.util.LinkedHashMap;
-import java.util.List;
-import java.util.Map;
+import java.util.*;
 
 import javax.sql.DataSource;
 
@@ -14,15 +10,15 @@ import org.junit.Before;
 import org.junit.Ignore;
 import org.junit.Test;
 
-import com.alibaba.otter.canal.client.adapter.es.ESAdapter;
-import com.alibaba.otter.canal.client.adapter.es.config.ESSyncConfig;
+import com.alibaba.otter.canal.client.adapter.es.core.config.ESSyncConfig;
+import com.alibaba.otter.canal.client.adapter.es6x.ES6xAdapter;
 import com.alibaba.otter.canal.client.adapter.support.DatasourceConfig;
 import com.alibaba.otter.canal.client.adapter.support.Dml;
 
 @Ignore
 public class UserSyncJoinOneTest {
 
-    private ESAdapter esAdapter;
+    private ES6xAdapter esAdapter;
 
     @Before
     public void init() {

client-adapter/es6x/src/test/java/com/alibaba/otter/canal/client/adapter/es6x/test/sync/UserSyncSingleTest.java

@@ -1,10 +1,6 @@
-package com.alibaba.otter.canal.client.adapter.es.test.sync;
+package com.alibaba.otter.canal.client.adapter.es6x.test.sync;
 
-import java.util.ArrayList;
-import java.util.Date;
-import java.util.LinkedHashMap;
-import java.util.List;
-import java.util.Map;
+import java.util.*;
 
 import org.elasticsearch.action.get.GetResponse;
 import org.junit.Assert;
@@ -12,14 +8,14 @@ import org.junit.Before;
 import org.junit.Ignore;
 import org.junit.Test;
 
-import com.alibaba.otter.canal.client.adapter.es.ESAdapter;
-import com.alibaba.otter.canal.client.adapter.es.config.ESSyncConfig;
+import com.alibaba.otter.canal.client.adapter.es.core.config.ESSyncConfig;
+import com.alibaba.otter.canal.client.adapter.es6x.ES6xAdapter;
 import com.alibaba.otter.canal.client.adapter.support.Dml;
 
 @Ignore
 public class UserSyncSingleTest {
 
-    private ESAdapter esAdapter;
+    private ES6xAdapter esAdapter;
 
     @Before
     public void init() {

client-adapter/es7x/pom.xml

@@ -0,0 +1,100 @@
+<?xml version="1.0" encoding="UTF-8"?>
+<project xmlns="http://maven.apache.org/POM/4.0.0"
+         xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
+         xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
+    <parent>
+        <artifactId>canal.client-adapter</artifactId>
+        <groupId>com.alibaba.otter</groupId>
+        <version>1.1.5-SNAPSHOT</version>
+        <relativePath>../pom.xml</relativePath>
+    </parent>
+    <modelVersion>4.0.0</modelVersion>
+    <groupId>com.alibaba.otter</groupId>
+    <artifactId>client-adapter.es7x</artifactId>
+    <packaging>jar</packaging>
+    <name>canal client adapter es v7x module for otter ${project.version}</name>
+
+    <dependencies>
+        <dependency>
+            <groupId>com.alibaba.otter</groupId>
+            <artifactId>client-adapter.common</artifactId>
+            <version>${project.version}</version>
+            <scope>provided</scope>
+        </dependency>
+        <dependency>
+            <groupId>com.alibaba.otter</groupId>
+            <artifactId>client-adapter.escore</artifactId>
+            <version>${project.version}</version>
+        </dependency>
+        <dependency>
+            <groupId>org.elasticsearch</groupId>
+            <artifactId>elasticsearch</artifactId>
+            <version>7.3.0</version>
+        </dependency>
+        <dependency>
+            <groupId>org.elasticsearch.client</groupId>
+            <artifactId>transport</artifactId>
+            <version>7.3.0</version>
+        </dependency>
+        <dependency>
+            <groupId>org.elasticsearch.client</groupId>
+            <artifactId>elasticsearch-rest-client</artifactId>
+            <version>7.3.0</version>
+        </dependency>
+        <dependency>
+            <groupId>org.elasticsearch.client</groupId>
+            <artifactId>elasticsearch-rest-high-level-client</artifactId>
+            <version>7.3.0</version>
+        </dependency>
+
+        <dependency>
+            <groupId>junit</groupId>
+            <artifactId>junit</artifactId>
+            <scope>test</scope>
+        </dependency>
+    </dependencies>
+
+    <build>
+        <plugins>
+            <plugin>
+                <groupId>org.apache.maven.plugins</groupId>
+                <artifactId>maven-assembly-plugin</artifactId>
+                <version>2.4</version>
+                <configuration>
+                    <descriptorRefs>
+                        <descriptorRef>jar-with-dependencies</descriptorRef>
+                    </descriptorRefs>
+                </configuration>
+                <executions>
+                    <execution>
+                        <id>make-assembly</id>
+                        <phase>package</phase>
+                        <goals>
+                            <goal>single</goal>
+                        </goals>
+                    </execution>
+                </executions>
+            </plugin>
+            <plugin>
+                <artifactId>maven-antrun-plugin</artifactId>
+                <executions>
+                    <execution>
+                        <phase>package</phase>
+                        <goals>
+                            <goal>run</goal>
+                        </goals>
+                        <configuration>
+                            <tasks>
+                                <copy todir="${project.basedir}/../launcher/target/classes/es7" overwrite="true">
+                                    <fileset dir="${project.basedir}/target/classes/es7" erroronmissingdir="true">
+                                        <include name="*.yml" />
+                                    </fileset>
+                                </copy>
+                            </tasks>
+                        </configuration>
+                    </execution>
+                </executions>
+            </plugin>
+        </plugins>
+    </build>
+</project>

client-adapter/es7x/src/main/java/com/alibaba/otter/canal/client/adapter/es7x/ES7xAdapter.java

@@ -0,0 +1,123 @@
+package com.alibaba.otter.canal.client.adapter.es7x;
+
+import java.util.LinkedHashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.Properties;
+
+import com.alibaba.otter.canal.client.adapter.es7x.etl.ESEtlService;
+import com.alibaba.otter.canal.client.adapter.support.DatasourceConfig;
+import org.elasticsearch.action.search.SearchResponse;
+
+import com.alibaba.otter.canal.client.adapter.es.core.ESAdapter;
+import com.alibaba.otter.canal.client.adapter.es.core.config.ESSyncConfig;
+import com.alibaba.otter.canal.client.adapter.es7x.support.ES7xTemplate;
+import com.alibaba.otter.canal.client.adapter.es7x.support.ESConnection;
+import com.alibaba.otter.canal.client.adapter.support.EtlResult;
+import com.alibaba.otter.canal.client.adapter.support.OuterAdapterConfig;
+import com.alibaba.otter.canal.client.adapter.support.SPI;
+
+import javax.sql.DataSource;
+
+/**
+ * ES 7.x 外部适配器
+ *
+ * @author rewerma 2019-09-23
+ * @version 1.0.0
+ */
+@SPI("es7")
+public class ES7xAdapter extends ESAdapter {
+
+    private ESConnection esConnection;
+
+    public ESConnection getEsConnection() {
+        return esConnection;
+    }
+
+    @Override
+    public void init(OuterAdapterConfig configuration, Properties envProperties) {
+        try {
+            Map<String, String> properties = configuration.getProperties();
+
+            String[] hostArray = configuration.getHosts().split(",");
+            String mode = properties.get("mode");
+            if ("rest".equalsIgnoreCase(mode) || "http".equalsIgnoreCase(mode)) {
+                esConnection = new ESConnection(hostArray, properties, ESConnection.ESClientMode.REST);
+            } else {
+                esConnection = new ESConnection(hostArray, properties, ESConnection.ESClientMode.TRANSPORT);
+            }
+            this.esTemplate = new ES7xTemplate(esConnection);
+
+            envProperties.put("es.version", "es7");
+            super.init(configuration, envProperties);
+        } catch (Throwable e) {
+            throw new RuntimeException(e);
+        }
+    }
+
+    @Override
+    public Map<String, Object> count(String task) {
+        ESSyncConfig config = esSyncConfig.get(task);
+        ESSyncConfig.ESMapping mapping = config.getEsMapping();
+        SearchResponse response = this.esConnection.new ESSearchRequest(mapping.get_index()).size(0).getResponse();
+
+        long rowCount = response.getHits().getTotalHits().value;
+        Map<String, Object> res = new LinkedHashMap<>();
+        res.put("esIndex", mapping.get_index());
+        res.put("count", rowCount);
+        return res;
+    }
+
+    @Override
+    public EtlResult etl(String task, List<String> params) {
+        EtlResult etlResult = new EtlResult();
+        ESSyncConfig config = esSyncConfig.get(task);
+        if (config != null) {
+            DataSource dataSource = DatasourceConfig.DATA_SOURCES.get(config.getDataSourceKey());
+            ESEtlService esEtlService = new ESEtlService(esConnection, config);
+            if (dataSource != null) {
+                return esEtlService.importData(params);
+            } else {
+                etlResult.setSucceeded(false);
+                etlResult.setErrorMessage("DataSource not found");
+                return etlResult;
+            }
+        } else {
+            StringBuilder resultMsg = new StringBuilder();
+            boolean resSuccess = true;
+            for (ESSyncConfig configTmp : esSyncConfig.values()) {
+                // 取所有的destination为task的配置
+                if (configTmp.getDestination().equals(task)) {
+                    ESEtlService esEtlService = new ESEtlService(esConnection, configTmp);
+                    EtlResult etlRes = esEtlService.importData(params);
+                    if (!etlRes.getSucceeded()) {
+                        resSuccess = false;
+                        resultMsg.append(etlRes.getErrorMessage()).append("\n");
+                    } else {
+                        resultMsg.append(etlRes.getResultMessage()).append("\n");
+                    }
+                }
+            }
+            if (resultMsg.length() > 0) {
+                etlResult.setSucceeded(resSuccess);
+                if (resSuccess) {
+                    etlResult.setResultMessage(resultMsg.toString());
+                } else {
+                    etlResult.setErrorMessage(resultMsg.toString());
+                }
+                return etlResult;
+            }
+        }
+        etlResult.setSucceeded(false);
+        etlResult.setErrorMessage("Task not found");
+        return etlResult;
+    }
+
+    @Override
+    public void destroy() {
+        super.destroy();
+        if (esConnection != null) {
+            esConnection.close();
+        }
+    }
+}

client-adapter/es7x/src/main/java/com/alibaba/otter/canal/client/adapter/es7x/etl/ESEtlService.java

@@ -0,0 +1,202 @@
+package com.alibaba.otter.canal.client.adapter.es7x.etl;
+
+import java.sql.SQLException;
+import java.util.HashMap;
+import java.util.LinkedHashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.concurrent.atomic.AtomicLong;
+
+import javax.sql.DataSource;
+
+import org.apache.commons.lang.StringUtils;
+import org.elasticsearch.action.search.SearchResponse;
+import org.elasticsearch.index.query.QueryBuilders;
+import org.elasticsearch.search.SearchHit;
+
+import com.alibaba.otter.canal.client.adapter.es.core.config.ESSyncConfig;
+import com.alibaba.otter.canal.client.adapter.es.core.config.ESSyncConfig.ESMapping;
+import com.alibaba.otter.canal.client.adapter.es.core.config.SchemaItem.FieldItem;
+import com.alibaba.otter.canal.client.adapter.es.core.support.ESBulkRequest;
+import com.alibaba.otter.canal.client.adapter.es.core.support.ESBulkRequest.ESBulkResponse;
+import com.alibaba.otter.canal.client.adapter.es.core.support.ESBulkRequest.ESIndexRequest;
+import com.alibaba.otter.canal.client.adapter.es.core.support.ESBulkRequest.ESUpdateRequest;
+import com.alibaba.otter.canal.client.adapter.es.core.support.ESTemplate;
+import com.alibaba.otter.canal.client.adapter.es7x.support.ES7xTemplate;
+import com.alibaba.otter.canal.client.adapter.es7x.support.ESConnection;
+import com.alibaba.otter.canal.client.adapter.es7x.support.ESConnection.ESSearchRequest;
+import com.alibaba.otter.canal.client.adapter.support.AbstractEtlService;
+import com.alibaba.otter.canal.client.adapter.support.AdapterConfig;
+import com.alibaba.otter.canal.client.adapter.support.EtlResult;
+import com.alibaba.otter.canal.client.adapter.support.Util;
+
+/**
+ * ES ETL Service
+ *
+ * @author rewerma 2018-11-01
+ * @version 1.0.0
+ */
+public class ESEtlService extends AbstractEtlService {
+
+    private ESConnection esConnection;
+    private ESTemplate   esTemplate;
+    private ESSyncConfig config;
+
+    public ESEtlService(ESConnection esConnection, ESSyncConfig config){
+        super("ES", config);
+        this.esConnection = esConnection;
+        this.esTemplate = new ES7xTemplate(esConnection);
+        this.config = config;
+    }
+
+    public EtlResult importData(List<String> params) {
+        ESMapping mapping = config.getEsMapping();
+        logger.info("start etl to import data to index: {}", mapping.get_index());
+        String sql = mapping.getSql();
+        return importData(sql, params);
+    }
+
+    protected boolean executeSqlImport(DataSource ds, String sql, List<Object> values,
+                                       AdapterConfig.AdapterMapping adapterMapping, AtomicLong impCount,
+                                       List<String> errMsg) {
+        try {
+            ESMapping mapping = (ESMapping) adapterMapping;
+            Util.sqlRS(ds, sql, values, rs -> {
+                int count = 0;
+                try {
+                    ESBulkRequest esBulkRequest = this.esConnection.new ES7xBulkRequest();
+
+                    long batchBegin = System.currentTimeMillis();
+                    while (rs.next()) {
+                        Map<String, Object> esFieldData = new LinkedHashMap<>();
+                        Object idVal = null;
+                        for (FieldItem fieldItem : mapping.getSchemaItem().getSelectFields().values()) {
+
+                            String fieldName = fieldItem.getFieldName();
+                            if (mapping.getSkips().contains(fieldName)) {
+                                continue;
+                            }
+
+                            // 如果是主键字段则不插入
+                            if (fieldItem.getFieldName().equals(mapping.get_id())) {
+                                idVal = esTemplate.getValFromRS(mapping, rs, fieldName, fieldName);
+                            } else {
+                                Object val = esTemplate.getValFromRS(mapping, rs, fieldName, fieldName);
+                                esFieldData.put(Util.cleanColumn(fieldName), val);
+                            }
+
+                        }
+
+                        if (!mapping.getRelations().isEmpty()) {
+                            mapping.getRelations().forEach((relationField, relationMapping) -> {
+                                Map<String, Object> relations = new HashMap<>();
+                                relations.put("name", relationMapping.getName());
+                                if (StringUtils.isNotEmpty(relationMapping.getParent())) {
+                                    FieldItem parentFieldItem = mapping.getSchemaItem()
+                                        .getSelectFields()
+                                        .get(relationMapping.getParent());
+                                    Object parentVal;
+                                    try {
+                                        parentVal = esTemplate.getValFromRS(mapping,
+                                            rs,
+                                            parentFieldItem.getFieldName(),
+                                            parentFieldItem.getFieldName());
+                                    } catch (SQLException e) {
+                                        throw new RuntimeException(e);
+                                    }
+                                    if (parentVal != null) {
+                                        relations.put("parent", parentVal.toString());
+                                        esFieldData.put("$parent_routing", parentVal.toString());
+
+                                    }
+                                }
+                                esFieldData.put(Util.cleanColumn(relationField), relations);
+                            });
+                        }
+
+                        if (idVal != null) {
+                            String parentVal = (String) esFieldData.remove("$parent_routing");
+                            if (mapping.isUpsert()) {
+                                ESUpdateRequest esUpdateRequest = this.esConnection.new ES7xUpdateRequest(
+                                    mapping.get_index(),
+                                    idVal.toString()).setDoc(esFieldData).setDocAsUpsert(true);
+
+                                if (StringUtils.isNotEmpty(parentVal)) {
+                                    esUpdateRequest.setRouting(parentVal);
+                                }
+
+                                esBulkRequest.add(esUpdateRequest);
+                            } else {
+                                ESIndexRequest esIndexRequest = this.esConnection.new ES7xIndexRequest(
+                                    mapping.get_index(),
+                                    idVal.toString()).setSource(esFieldData);
+                                if (StringUtils.isNotEmpty(parentVal)) {
+                                    esIndexRequest.setRouting(parentVal);
+                                }
+                                esBulkRequest.add(esIndexRequest);
+                            }
+                        } else {
+                            idVal = esFieldData.get(mapping.getPk());
+                            ESSearchRequest esSearchRequest = this.esConnection.new ESSearchRequest(mapping.get_index())
+                                .setQuery(QueryBuilders.termQuery(mapping.getPk(), idVal))
+                                .size(10000);
+                            SearchResponse response = esSearchRequest.getResponse();
+                            for (SearchHit hit : response.getHits()) {
+                                ESUpdateRequest esUpdateRequest = this.esConnection.new ES7xUpdateRequest(
+                                    mapping.get_index(),
+                                    hit.getId()).setDoc(esFieldData);
+                                esBulkRequest.add(esUpdateRequest);
+                            }
+                        }
+
+                        if (esBulkRequest.numberOfActions() % mapping.getCommitBatch() == 0
+                            && esBulkRequest.numberOfActions() > 0) {
+                            long esBatchBegin = System.currentTimeMillis();
+                            ESBulkResponse rp = esBulkRequest.bulk();
+                            if (rp.hasFailures()) {
+                                rp.processFailBulkResponse("全量数据 etl 异常 ");
+                            }
+
+                            if (logger.isTraceEnabled()) {
+                                logger.trace("全量数据批量导入批次耗时: {}, es执行时间: {}, 批次大小: {}, index; {}",
+                                    (System.currentTimeMillis() - batchBegin),
+                                    (System.currentTimeMillis() - esBatchBegin),
+                                    esBulkRequest.numberOfActions(),
+                                    mapping.get_index());
+                            }
+                            batchBegin = System.currentTimeMillis();
+                            esBulkRequest.resetBulk();
+                        }
+                        count++;
+                        impCount.incrementAndGet();
+                    }
+
+                    if (esBulkRequest.numberOfActions() > 0) {
+                        long esBatchBegin = System.currentTimeMillis();
+                        ESBulkResponse rp = esBulkRequest.bulk();
+                        if (rp.hasFailures()) {
+                            rp.processFailBulkResponse("全量数据 etl 异常 ");
+                        }
+                        if (logger.isTraceEnabled()) {
+                            logger.trace("全量数据批量导入最后批次耗时: {}, es执行时间: {}, 批次大小: {}, index; {}",
+                                (System.currentTimeMillis() - batchBegin),
+                                (System.currentTimeMillis() - esBatchBegin),
+                                esBulkRequest.numberOfActions(),
+                                mapping.get_index());
+                        }
+                    }
+                } catch (Exception e) {
+                    logger.error(e.getMessage(), e);
+                    errMsg.add(mapping.get_index() + " etl failed! ==>" + e.getMessage());
+                    throw new RuntimeException(e);
+                }
+                return count;
+            });
+
+            return true;
+        } catch (Exception e) {
+            logger.error(e.getMessage(), e);
+            return false;
+        }
+    }
+}

client-adapter/es7x/src/main/java/com/alibaba/otter/canal/client/adapter/es7x/support/ES7xTemplate.java

@@ -1,4 +1,4 @@
-package com.alibaba.otter.canal.client.adapter.es.support;
+package com.alibaba.otter.canal.client.adapter.es7x.support;
 
 import java.sql.ResultSet;
 import java.sql.SQLException;
@@ -9,45 +9,47 @@ import java.util.concurrent.ConcurrentMap;
 import javax.sql.DataSource;
 
 import org.apache.commons.lang.StringUtils;
-import org.elasticsearch.action.bulk.BulkItemResponse;
-import org.elasticsearch.action.bulk.BulkResponse;
 import org.elasticsearch.action.search.SearchResponse;
 import org.elasticsearch.cluster.metadata.MappingMetaData;
 import org.elasticsearch.index.query.BoolQueryBuilder;
 import org.elasticsearch.index.query.QueryBuilders;
-import org.elasticsearch.rest.RestStatus;
 import org.elasticsearch.search.SearchHit;
 import org.slf4j.Logger;
 import org.slf4j.LoggerFactory;
 
-import com.alibaba.otter.canal.client.adapter.es.config.ESSyncConfig;
-import com.alibaba.otter.canal.client.adapter.es.config.ESSyncConfig.ESMapping;
-import com.alibaba.otter.canal.client.adapter.es.config.SchemaItem;
-import com.alibaba.otter.canal.client.adapter.es.config.SchemaItem.ColumnItem;
-import com.alibaba.otter.canal.client.adapter.es.config.SchemaItem.FieldItem;
-import com.alibaba.otter.canal.client.adapter.es.support.ESConnection.*;
+import com.alibaba.otter.canal.client.adapter.es.core.config.ESSyncConfig;
+import com.alibaba.otter.canal.client.adapter.es.core.config.ESSyncConfig.ESMapping;
+import com.alibaba.otter.canal.client.adapter.es.core.config.SchemaItem;
+import com.alibaba.otter.canal.client.adapter.es.core.config.SchemaItem.ColumnItem;
+import com.alibaba.otter.canal.client.adapter.es.core.config.SchemaItem.FieldItem;
+import com.alibaba.otter.canal.client.adapter.es.core.support.ESBulkRequest;
+import com.alibaba.otter.canal.client.adapter.es.core.support.ESBulkRequest.ESBulkResponse;
+import com.alibaba.otter.canal.client.adapter.es.core.support.ESBulkRequest.ESDeleteRequest;
+import com.alibaba.otter.canal.client.adapter.es.core.support.ESBulkRequest.ESIndexRequest;
+import com.alibaba.otter.canal.client.adapter.es.core.support.ESBulkRequest.ESUpdateRequest;
+import com.alibaba.otter.canal.client.adapter.es.core.support.ESSyncUtil;
+import com.alibaba.otter.canal.client.adapter.es.core.support.ESTemplate;
+import com.alibaba.otter.canal.client.adapter.es7x.support.ESConnection.ESSearchRequest;
 import com.alibaba.otter.canal.client.adapter.support.DatasourceConfig;
 import com.alibaba.otter.canal.client.adapter.support.Util;
 
-/**
- * ES 操作模板
- *
- * @author rewerma 2018-11-01
- * @version 1.0.0
- */
-public class ESTemplate {
+public class ES7xTemplate implements ESTemplate {
 
-    private static final Logger logger = LoggerFactory.getLogger(ESTemplate.class);
+    private static final Logger                               logger         = LoggerFactory
+        .getLogger(ESTemplate.class);
 
-    private static final int MAX_BATCH_SIZE = 1000;
+    private static final int                                  MAX_BATCH_SIZE = 1000;
 
-    private ESConnection esConnection;
+    private ESConnection                                      esConnection;
 
-    private ESBulkRequest esBulkRequest;
+    private ESBulkRequest                                     esBulkRequest;
 
-    public ESTemplate(ESConnection esConnection) {
+    // es 字段类型本地缓存
+    private static ConcurrentMap<String, Map<String, String>> esFieldTypes   = new ConcurrentHashMap<>();
+
+    public ES7xTemplate(ESConnection esConnection){
         this.esConnection = esConnection;
-        this.esBulkRequest = this.esConnection.new ESBulkRequest();
+        this.esBulkRequest = this.esConnection.new ES7xBulkRequest();
     }
 
     public ESBulkRequest getBulk() {
@@ -58,28 +60,20 @@ public class ESTemplate {
         this.esBulkRequest.resetBulk();
     }
 
-    /**
-     * 插入数据
-     *
-     * @param mapping     配置对象
-     * @param pkVal       主键值
-     * @param esFieldData 数据Map
-     */
+    @Override
     public void insert(ESMapping mapping, Object pkVal, Map<String, Object> esFieldData) {
         if (mapping.get_id() != null) {
             String parentVal = (String) esFieldData.remove("$parent_routing");
             if (mapping.isUpsert()) {
-                ESUpdateRequest updateRequest = esConnection.new ESUpdateRequest(mapping.get_index(),
-                        mapping.get_type(),
-                        pkVal.toString()).setDoc(esFieldData).setDocAsUpsert(true);
+                ESUpdateRequest updateRequest = esConnection.new ES7xUpdateRequest(mapping.get_index(),
+                    pkVal.toString()).setDoc(esFieldData).setDocAsUpsert(true);
                 if (StringUtils.isNotEmpty(parentVal)) {
                     updateRequest.setRouting(parentVal);
                 }
                 getBulk().add(updateRequest);
             } else {
-                ESIndexRequest indexRequest = esConnection.new ESIndexRequest(mapping.get_index(),
-                        mapping.get_type(),
-                        pkVal.toString()).setSource(esFieldData);
+                ESIndexRequest indexRequest = esConnection.new ES7xIndexRequest(mapping.get_index(), pkVal.toString())
+                    .setSource(esFieldData);
                 if (StringUtils.isNotEmpty(parentVal)) {
                     indexRequest.setRouting(parentVal);
                 }
@@ -87,28 +81,21 @@ public class ESTemplate {
             }
             commitBulk();
         } else {
-            ESSearchRequest esSearchRequest = this.esConnection.new ESSearchRequest(mapping.get_index(),
-                    mapping.get_type()).setQuery(QueryBuilders.termQuery(mapping.getPk(), pkVal)).size(10000);
+            ESSearchRequest esSearchRequest = this.esConnection.new ESSearchRequest(mapping.get_index())
+                .setQuery(QueryBuilders.termQuery(mapping.getPk(), pkVal))
+                .size(10000);
             SearchResponse response = esSearchRequest.getResponse();
 
             for (SearchHit hit : response.getHits()) {
-                ESUpdateRequest esUpdateRequest = this.esConnection.new ESUpdateRequest(mapping.get_index(),
-                        mapping.get_type(),
-                        hit.getId()).setDoc(esFieldData);
+                ESUpdateRequest esUpdateRequest = this.esConnection.new ES7xUpdateRequest(mapping.get_index(),
+                    hit.getId()).setDoc(esFieldData);
                 getBulk().add(esUpdateRequest);
                 commitBulk();
             }
         }
-
     }
 
-    /**
-     * 根据主键更新数据
-     *
-     * @param mapping     配置对象
-     * @param pkVal       主键值
-     * @param esFieldData 数据Map
-     */
+    @Override
     public void update(ESMapping mapping, Object pkVal, Map<String, Object> esFieldData) {
         Map<String, Object> esFieldDataTmp = new LinkedHashMap<>(esFieldData.size());
         esFieldData.forEach((k, v) -> esFieldDataTmp.put(Util.cleanColumn(k), v));
@@ -116,13 +103,7 @@ public class ESTemplate {
         commitBulk();
     }
 
-    /**
-     * update by query
-     *
-     * @param config      配置对象
-     * @param paramsTmp   sql查询条件
-     * @param esFieldData 数据Map
-     */
+    @Override
     public void updateByQuery(ESSyncConfig config, Map<String, Object> paramsTmp, Map<String, Object> esFieldData) {
         if (paramsTmp.isEmpty()) {
             return;
@@ -161,100 +142,39 @@ public class ESTemplate {
         }
     }
 
-    /**
-     * 通过主键删除数据
-     *
-     * @param mapping     配置对象
-     * @param pkVal       主键值
-     * @param esFieldData 数据Map
-     */
+    @Override
     public void delete(ESMapping mapping, Object pkVal, Map<String, Object> esFieldData) {
         if (mapping.get_id() != null) {
-            ESDeleteRequest esDeleteRequest = this.esConnection.new ESDeleteRequest(mapping.get_index(),
-                    mapping.get_type(),
-                    pkVal.toString());
+            ESDeleteRequest esDeleteRequest = this.esConnection.new ES7xDeleteRequest(mapping.get_index(),
+                pkVal.toString());
             getBulk().add(esDeleteRequest);
             commitBulk();
         } else {
-            ESSearchRequest esSearchRequest = this.esConnection.new ESSearchRequest(mapping.get_index(),
-                    mapping.get_type()).setQuery(QueryBuilders.termQuery(mapping.getPk(), pkVal)).size(10000);
+            ESSearchRequest esSearchRequest = this.esConnection.new ESSearchRequest(mapping.get_index())
+                .setQuery(QueryBuilders.termQuery(mapping.getPk(), pkVal))
+                .size(10000);
             SearchResponse response = esSearchRequest.getResponse();
             for (SearchHit hit : response.getHits()) {
-                ESUpdateRequest esUpdateRequest = this.esConnection.new ESUpdateRequest(mapping.get_index(),
-                        mapping.get_type(),
-                        hit.getId()).setDoc(esFieldData);
+                ESUpdateRequest esUpdateRequest = this.esConnection.new ES7xUpdateRequest(mapping.get_index(),
+                    hit.getId()).setDoc(esFieldData);
                 getBulk().add(esUpdateRequest);
                 commitBulk();
             }
         }
-
     }
 
-    /**
-     * 提交批次
-     */
+    @Override
     public void commit() {
         if (getBulk().numberOfActions() > 0) {
-            BulkResponse response = getBulk().bulk();
+            ESBulkResponse response = getBulk().bulk();
             if (response.hasFailures()) {
-                for (BulkItemResponse itemResponse : response.getItems()) {
-                    if (!itemResponse.isFailed()) {
-                        continue;
-                    }
-
-                    if (itemResponse.getFailure().getStatus() == RestStatus.NOT_FOUND) {
-                        logger.error(itemResponse.getFailureMessage());
-                    } else {
-                        throw new RuntimeException("ES sync commit error" + itemResponse.getFailureMessage());
-                    }
-                }
+                response.processFailBulkResponse("ES sync commit error ");
             }
             resetBulkRequestBuilder();
         }
     }
 
-    /**
-     * 如果大于批量数则提交批次
-     */
-    private void commitBulk() {
-        if (getBulk().numberOfActions() >= MAX_BATCH_SIZE) {
-            commit();
-        }
-    }
-
-    private void append4Update(ESMapping mapping, Object pkVal, Map<String, Object> esFieldData) {
-        if (mapping.get_id() != null) {
-            String parentVal = (String) esFieldData.remove("$parent_routing");
-            if (mapping.isUpsert()) {
-                ESUpdateRequest esUpdateRequest = this.esConnection.new ESUpdateRequest(mapping.get_index(),
-                        mapping.get_type(),
-                        pkVal.toString()).setDoc(esFieldData).setDocAsUpsert(true);
-                if (StringUtils.isNotEmpty(parentVal)) {
-                    esUpdateRequest.setRouting(parentVal);
-                }
-                getBulk().add(esUpdateRequest);
-            } else {
-                ESUpdateRequest esUpdateRequest = this.esConnection.new ESUpdateRequest(mapping.get_index(),
-                        mapping.get_type(),
-                        pkVal.toString()).setDoc(esFieldData);
-                if (StringUtils.isNotEmpty(parentVal)) {
-                    esUpdateRequest.setRouting(parentVal);
-                }
-                getBulk().add(esUpdateRequest);
-            }
-        } else {
-            ESSearchRequest esSearchRequest = this.esConnection.new ESSearchRequest(mapping.get_index(),
-                    mapping.get_type()).setQuery(QueryBuilders.termQuery(mapping.getPk(), pkVal)).size(10000);
-            SearchResponse response = esSearchRequest.getResponse();
-            for (SearchHit hit : response.getHits()) {
-                ESUpdateRequest esUpdateRequest = this.esConnection.new ESUpdateRequest(mapping.get_index(),
-                        mapping.get_type(),
-                        hit.getId()).setDoc(esFieldData);
-                getBulk().add(esUpdateRequest);
-            }
-        }
-    }
-
+    @Override
     public Object getValFromRS(ESMapping mapping, ResultSet resultSet, String fieldName,
                                String columnName) throws SQLException {
         fieldName = Util.cleanColumn(fieldName);
@@ -276,6 +196,7 @@ public class ESTemplate {
         }
     }
 
+    @Override
     public Object getESDataFromRS(ESMapping mapping, ResultSet resultSet,
                                   Map<String, Object> esFieldData) throws SQLException {
         SchemaItem schemaItem = mapping.getSchemaItem();
@@ -289,7 +210,7 @@ public class ESTemplate {
             }
 
             if (!fieldItem.getFieldName().equals(mapping.get_id())
-                    && !mapping.getSkips().contains(fieldItem.getFieldName())) {
+                && !mapping.getSkips().contains(fieldItem.getFieldName())) {
                 esFieldData.put(Util.cleanColumn(fieldItem.getFieldName()), value);
             }
         }
@@ -300,6 +221,7 @@ public class ESTemplate {
         return resultIdVal;
     }
 
+    @Override
     public Object getIdValFromRS(ESMapping mapping, ResultSet resultSet) throws SQLException {
         SchemaItem schemaItem = mapping.getSchemaItem();
         String idFieldName = mapping.get_id() == null ? mapping.getPk() : mapping.get_id();
@@ -315,6 +237,7 @@ public class ESTemplate {
         return resultIdVal;
     }
 
+    @Override
     public Object getESDataFromRS(ESMapping mapping, ResultSet resultSet, Map<String, Object> dmlOld,
                                   Map<String, Object> esFieldData) throws SQLException {
         SchemaItem schemaItem = mapping.getSchemaItem();
@@ -327,9 +250,9 @@ public class ESTemplate {
 
             for (ColumnItem columnItem : fieldItem.getColumnItems()) {
                 if (dmlOld.containsKey(columnItem.getColumnName())
-                        && !mapping.getSkips().contains(fieldItem.getFieldName())) {
+                    && !mapping.getSkips().contains(fieldItem.getFieldName())) {
                     esFieldData.put(Util.cleanColumn(fieldItem.getFieldName()),
-                            getValFromRS(mapping, resultSet, fieldItem.getFieldName(), fieldItem.getFieldName()));
+                        getValFromRS(mapping, resultSet, fieldItem.getFieldName(), fieldItem.getFieldName()));
                     break;
                 }
             }
@@ -341,6 +264,7 @@ public class ESTemplate {
         return resultIdVal;
     }
 
+    @Override
     public Object getValFromData(ESMapping mapping, Map<String, Object> dmlData, String fieldName, String columnName) {
         String esType = getEsType(mapping, fieldName);
         Object value = dmlData.get(columnName);
@@ -358,14 +282,7 @@ public class ESTemplate {
         }
     }
 
-    /**
-     * 将dml的data转换为es的data
-     *
-     * @param mapping     配置mapping
-     * @param dmlData     dml data
-     * @param esFieldData es data
-     * @return 返回 id 值
-     */
+    @Override
     public Object getESDataFromDmlData(ESMapping mapping, Map<String, Object> dmlData,
                                        Map<String, Object> esFieldData) {
         SchemaItem schemaItem = mapping.getSchemaItem();
@@ -380,7 +297,7 @@ public class ESTemplate {
             }
 
             if (!fieldItem.getFieldName().equals(mapping.get_id())
-                    && !mapping.getSkips().contains(fieldItem.getFieldName())) {
+                && !mapping.getSkips().contains(fieldItem.getFieldName())) {
                 esFieldData.put(Util.cleanColumn(fieldItem.getFieldName()), value);
             }
         }
@@ -390,14 +307,7 @@ public class ESTemplate {
         return resultIdVal;
     }
 
-    /**
-     * 将dml的data, old转换为es的data
-     *
-     * @param mapping     配置mapping
-     * @param dmlData     dml data
-     * @param esFieldData es data
-     * @return 返回 id 值
-     */
+    @Override
     public Object getESDataFromDmlData(ESMapping mapping, Map<String, Object> dmlData, Map<String, Object> dmlOld,
                                        Map<String, Object> esFieldData) {
         SchemaItem schemaItem = mapping.getSchemaItem();
@@ -421,6 +331,84 @@ public class ESTemplate {
         return resultIdVal;
     }
 
+    /**
+     * 如果大于批量数则提交批次
+     */
+    private void commitBulk() {
+        if (getBulk().numberOfActions() >= MAX_BATCH_SIZE) {
+            commit();
+        }
+    }
+
+    private void append4Update(ESMapping mapping, Object pkVal, Map<String, Object> esFieldData) {
+        if (mapping.get_id() != null) {
+            String parentVal = (String) esFieldData.remove("$parent_routing");
+            if (mapping.isUpsert()) {
+                ESUpdateRequest esUpdateRequest = this.esConnection.new ES7xUpdateRequest(mapping.get_index(),
+                    pkVal.toString()).setDoc(esFieldData).setDocAsUpsert(true);
+                if (StringUtils.isNotEmpty(parentVal)) {
+                    esUpdateRequest.setRouting(parentVal);
+                }
+                getBulk().add(esUpdateRequest);
+            } else {
+                ESUpdateRequest esUpdateRequest = this.esConnection.new ES7xUpdateRequest(mapping.get_index(),
+                    pkVal.toString()).setDoc(esFieldData);
+                if (StringUtils.isNotEmpty(parentVal)) {
+                    esUpdateRequest.setRouting(parentVal);
+                }
+                getBulk().add(esUpdateRequest);
+            }
+        } else {
+            ESSearchRequest esSearchRequest = this.esConnection.new ESSearchRequest(mapping.get_index())
+                .setQuery(QueryBuilders.termQuery(mapping.getPk(), pkVal))
+                .size(10000);
+            SearchResponse response = esSearchRequest.getResponse();
+            for (SearchHit hit : response.getHits()) {
+                ESUpdateRequest esUpdateRequest = this.esConnection.new ES7xUpdateRequest(mapping.get_index(),
+                    hit.getId()).setDoc(esFieldData);
+                getBulk().add(esUpdateRequest);
+            }
+        }
+    }
+
+    /**
+     * 获取es mapping中的属性类型
+     *
+     * @param mapping mapping配置
+     * @param fieldName 属性名
+     * @return 类型
+     */
+    @SuppressWarnings("unchecked")
+    private String getEsType(ESMapping mapping, String fieldName) {
+        String key = mapping.get_index() + "-" + mapping.get_type();
+        Map<String, String> fieldType = esFieldTypes.get(key);
+        if (fieldType != null) {
+            return fieldType.get(fieldName);
+        } else {
+            MappingMetaData mappingMetaData = esConnection.getMapping(mapping.get_index());
+
+            if (mappingMetaData == null) {
+                throw new IllegalArgumentException("Not found the mapping info of index: " + mapping.get_index());
+            }
+
+            fieldType = new LinkedHashMap<>();
+
+            Map<String, Object> sourceMap = mappingMetaData.getSourceAsMap();
+            Map<String, Object> esMapping = (Map<String, Object>) sourceMap.get("properties");
+            for (Map.Entry<String, Object> entry : esMapping.entrySet()) {
+                Map<String, Object> value = (Map<String, Object>) entry.getValue();
+                if (value.containsKey("properties")) {
+                    fieldType.put(entry.getKey(), "object");
+                } else {
+                    fieldType.put(entry.getKey(), (String) value.get("type"));
+                }
+            }
+            esFieldTypes.put(key, fieldType);
+
+            return fieldType.get(fieldName);
+        }
+    }
+
     private void putRelationDataFromRS(ESMapping mapping, SchemaItem schemaItem, ResultSet resultSet,
                                        Map<String, Object> esFieldData) {
         // 添加父子文档关联信息
@@ -433,9 +421,9 @@ public class ESTemplate {
                     Object parentVal;
                     try {
                         parentVal = getValFromRS(mapping,
-                                resultSet,
-                                parentFieldItem.getFieldName(),
-                                parentFieldItem.getFieldName());
+                            resultSet,
+                            parentFieldItem.getFieldName(),
+                            parentFieldItem.getFieldName());
                     } catch (SQLException e) {
                         throw new RuntimeException(e);
                     }
@@ -471,48 +459,4 @@ public class ESTemplate {
             });
         }
     }
-
-    /**
-     * es 字段类型本地缓存
-     */
-    private static ConcurrentMap<String, Map<String, String>> esFieldTypes = new ConcurrentHashMap<>();
-
-    /**
-     * 获取es mapping中的属性类型
-     *
-     * @param mapping   mapping配置
-     * @param fieldName 属性名
-     * @return 类型
-     */
-    @SuppressWarnings("unchecked")
-    private String getEsType(ESMapping mapping, String fieldName) {
-        String key = mapping.get_index() + "-" + mapping.get_type();
-        Map<String, String> fieldType = esFieldTypes.get(key);
-        if (fieldType != null) {
-            return fieldType.get(fieldName);
-        } else {
-            MappingMetaData mappingMetaData = esConnection.getMapping(mapping.get_index(), mapping.get_type());
-
-            if (mappingMetaData == null) {
-                throw new IllegalArgumentException("Not found the mapping info of index: " + mapping.get_index());
-            }
-
-            fieldType = new LinkedHashMap<>();
-
-            Map<String, Object> sourceMap = mappingMetaData.getSourceAsMap();
-            Map<String, Object> esMapping = (Map<String, Object>) sourceMap.get("properties");
-            for (Map.Entry<String, Object> entry : esMapping.entrySet()) {
-                Map<String, Object> value = (Map<String, Object>) entry.getValue();
-                if (value.containsKey("properties")) {
-                    fieldType.put(entry.getKey(), "object");
-                } else {
-                    fieldType.put(entry.getKey(), (String) value.get("type"));
-                }
-            }
-            esFieldTypes.put(key, fieldType);
-
-            return fieldType.get(fieldName);
-        }
-    }
-
 }

client-adapter/es7x/src/main/java/com/alibaba/otter/canal/client/adapter/es7x/support/ESConnection.java

@@ -0,0 +1,504 @@
+package com.alibaba.otter.canal.client.adapter.es7x.support;
+
+import java.io.IOException;
+import java.net.InetAddress;
+import java.net.UnknownHostException;
+import java.util.Map;
+
+import org.apache.commons.lang.StringUtils;
+import org.apache.http.HttpHost;
+import org.apache.http.auth.AuthScope;
+import org.apache.http.auth.UsernamePasswordCredentials;
+import org.apache.http.client.CredentialsProvider;
+import org.apache.http.impl.client.BasicCredentialsProvider;
+import org.elasticsearch.action.bulk.BulkItemResponse;
+import org.elasticsearch.action.bulk.BulkRequest;
+import org.elasticsearch.action.bulk.BulkRequestBuilder;
+import org.elasticsearch.action.bulk.BulkResponse;
+import org.elasticsearch.action.delete.DeleteRequest;
+import org.elasticsearch.action.delete.DeleteRequestBuilder;
+import org.elasticsearch.action.index.IndexRequest;
+import org.elasticsearch.action.index.IndexRequestBuilder;
+import org.elasticsearch.action.search.SearchRequest;
+import org.elasticsearch.action.search.SearchRequestBuilder;
+import org.elasticsearch.action.search.SearchResponse;
+import org.elasticsearch.action.update.UpdateRequest;
+import org.elasticsearch.action.update.UpdateRequestBuilder;
+import org.elasticsearch.client.RequestOptions;
+import org.elasticsearch.client.RestClient;
+import org.elasticsearch.client.RestClientBuilder;
+import org.elasticsearch.client.RestHighLevelClient;
+import org.elasticsearch.client.indices.GetMappingsRequest;
+import org.elasticsearch.client.indices.GetMappingsResponse;
+import org.elasticsearch.client.transport.TransportClient;
+import org.elasticsearch.cluster.metadata.MappingMetaData;
+import org.elasticsearch.common.settings.Settings;
+import org.elasticsearch.common.transport.TransportAddress;
+import org.elasticsearch.index.query.QueryBuilder;
+import org.elasticsearch.rest.RestStatus;
+import org.elasticsearch.search.builder.SearchSourceBuilder;
+import org.elasticsearch.transport.client.PreBuiltTransportClient;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import com.alibaba.otter.canal.client.adapter.es.core.support.ESBulkRequest;
+
+/**
+ * ES 连接器, Transport Rest 两种方式
+ *
+ * @author rewerma 2019-08-01
+ * @version 1.0.0
+ */
+public class ESConnection {
+
+    private static final Logger logger = LoggerFactory.getLogger(ESConnection.class);
+
+    public enum ESClientMode {
+                              TRANSPORT, REST
+    }
+
+    private ESClientMode        mode;
+
+    private TransportClient     transportClient;
+
+    private RestHighLevelClient restHighLevelClient;
+
+    public ESConnection(String[] hosts, Map<String, String> properties, ESClientMode mode) throws UnknownHostException{
+        this.mode = mode;
+        if (mode == ESClientMode.TRANSPORT) {
+            Settings.Builder settingBuilder = Settings.builder();
+            settingBuilder.put("cluster.name", properties.get("cluster.name"));
+            Settings settings = settingBuilder.build();
+            transportClient = new PreBuiltTransportClient(settings);
+            for (String host : hosts) {
+                int i = host.indexOf(":");
+                transportClient.addTransportAddress(new TransportAddress(InetAddress.getByName(host.substring(0, i)),
+                    Integer.parseInt(host.substring(i + 1))));
+            }
+        } else {
+            HttpHost[] httpHosts = new HttpHost[hosts.length];
+            for (int i = 0; i < hosts.length; i++) {
+                String host = hosts[i];
+                int j = host.indexOf(":");
+                HttpHost httpHost = new HttpHost(InetAddress.getByName(host.substring(0, j)),
+                    Integer.parseInt(host.substring(j + 1)));
+                httpHosts[i] = httpHost;
+            }
+            RestClientBuilder restClientBuilder = RestClient.builder(httpHosts);
+            String nameAndPwd = properties.get("security.auth");
+            if (StringUtils.isNotEmpty(nameAndPwd) && nameAndPwd.contains(":")) {
+                String[] nameAndPwdArr = nameAndPwd.split(":");
+                final CredentialsProvider credentialsProvider = new BasicCredentialsProvider();
+                credentialsProvider.setCredentials(AuthScope.ANY,
+                    new UsernamePasswordCredentials(nameAndPwdArr[0], nameAndPwdArr[1]));
+                restClientBuilder.setHttpClientConfigCallback(
+                    httpClientBuilder -> httpClientBuilder.setDefaultCredentialsProvider(credentialsProvider));
+            }
+            restHighLevelClient = new RestHighLevelClient(restClientBuilder);
+        }
+    }
+
+    public void close() {
+        if (mode == ESClientMode.TRANSPORT) {
+            transportClient.close();
+        } else {
+            try {
+                restHighLevelClient.close();
+            } catch (IOException e) {
+                throw new RuntimeException(e);
+            }
+        }
+    }
+
+    public MappingMetaData getMapping(String index) {
+        MappingMetaData mappingMetaData = null;
+        if (mode == ESClientMode.TRANSPORT) {
+            try {
+                mappingMetaData = transportClient.admin()
+                    .cluster()
+                    .prepareState()
+                    .execute()
+                    .actionGet()
+                    .getState()
+                    .getMetaData()
+                    .getIndices()
+                    .get(index)
+                    .mapping();
+            } catch (NullPointerException e) {
+                throw new IllegalArgumentException("Not found the mapping info of index: " + index);
+            }
+        } else {
+            Map<String, MappingMetaData> mappings;
+            try {
+                GetMappingsRequest request = new GetMappingsRequest();
+                request.indices(index);
+                GetMappingsResponse response = restHighLevelClient.indices()
+                    .getMapping(request, RequestOptions.DEFAULT);
+
+                mappings = response.mappings();
+            } catch (NullPointerException e) {
+                throw new IllegalArgumentException("Not found the mapping info of index: " + index);
+            } catch (IOException e) {
+                logger.error(e.getMessage(), e);
+                return null;
+            }
+            mappingMetaData = mappings.get(index);
+        }
+        return mappingMetaData;
+    }
+
+    public class ES7xIndexRequest implements ESBulkRequest.ESIndexRequest {
+
+        private IndexRequestBuilder indexRequestBuilder;
+
+        private IndexRequest        indexRequest;
+
+        public ES7xIndexRequest(String index, String id){
+            if (mode == ESClientMode.TRANSPORT) {
+                indexRequestBuilder = transportClient.prepareIndex();
+                indexRequestBuilder.setIndex(index);
+                indexRequestBuilder.setId(id);
+            } else {
+                indexRequest = new IndexRequest(index);
+                indexRequest.id(id);
+            }
+        }
+
+        public ES7xIndexRequest setSource(Map<String, ?> source) {
+            if (mode == ESClientMode.TRANSPORT) {
+                indexRequestBuilder.setSource(source);
+            } else {
+                indexRequest.source(source);
+            }
+            return this;
+        }
+
+        public ES7xIndexRequest setRouting(String routing) {
+            if (mode == ESClientMode.TRANSPORT) {
+                indexRequestBuilder.setRouting(routing);
+            } else {
+                indexRequest.routing(routing);
+            }
+            return this;
+        }
+
+        public IndexRequestBuilder getIndexRequestBuilder() {
+            return indexRequestBuilder;
+        }
+
+        public void setIndexRequestBuilder(IndexRequestBuilder indexRequestBuilder) {
+            this.indexRequestBuilder = indexRequestBuilder;
+        }
+
+        public IndexRequest getIndexRequest() {
+            return indexRequest;
+        }
+
+        public void setIndexRequest(IndexRequest indexRequest) {
+            this.indexRequest = indexRequest;
+        }
+    }
+
+    public class ES7xUpdateRequest implements ESBulkRequest.ESUpdateRequest {
+
+        private UpdateRequestBuilder updateRequestBuilder;
+
+        private UpdateRequest        updateRequest;
+
+        public ES7xUpdateRequest(String index, String id){
+            if (mode == ESClientMode.TRANSPORT) {
+                updateRequestBuilder = transportClient.prepareUpdate();
+                updateRequestBuilder.setIndex(index);
+                updateRequestBuilder.setId(id);
+            } else {
+                updateRequest = new UpdateRequest(index, id);
+            }
+        }
+
+        public ES7xUpdateRequest setDoc(Map source) {
+            if (mode == ESClientMode.TRANSPORT) {
+                updateRequestBuilder.setDoc(source);
+            } else {
+                updateRequest.doc(source);
+            }
+            return this;
+        }
+
+        public ES7xUpdateRequest setDocAsUpsert(boolean shouldUpsertDoc) {
+            if (mode == ESClientMode.TRANSPORT) {
+                updateRequestBuilder.setDocAsUpsert(shouldUpsertDoc);
+            } else {
+                updateRequest.docAsUpsert(shouldUpsertDoc);
+            }
+            return this;
+        }
+
+        public ES7xUpdateRequest setRouting(String routing) {
+            if (mode == ESClientMode.TRANSPORT) {
+                updateRequestBuilder.setRouting(routing);
+            } else {
+                updateRequest.routing(routing);
+            }
+            return this;
+        }
+
+        public UpdateRequestBuilder getUpdateRequestBuilder() {
+            return updateRequestBuilder;
+        }
+
+        public void setUpdateRequestBuilder(UpdateRequestBuilder updateRequestBuilder) {
+            this.updateRequestBuilder = updateRequestBuilder;
+        }
+
+        public UpdateRequest getUpdateRequest() {
+            return updateRequest;
+        }
+
+        public void setUpdateRequest(UpdateRequest updateRequest) {
+            this.updateRequest = updateRequest;
+        }
+    }
+
+    public class ES7xDeleteRequest implements ESBulkRequest.ESDeleteRequest {
+
+        private DeleteRequestBuilder deleteRequestBuilder;
+
+        private DeleteRequest        deleteRequest;
+
+        public ES7xDeleteRequest(String index, String id){
+            if (mode == ESClientMode.TRANSPORT) {
+                deleteRequestBuilder = transportClient.prepareDelete();
+                deleteRequestBuilder.setIndex(index);
+                deleteRequestBuilder.setId(id);
+            } else {
+                deleteRequest = new DeleteRequest(index, id);
+            }
+        }
+
+        public DeleteRequestBuilder getDeleteRequestBuilder() {
+            return deleteRequestBuilder;
+        }
+
+        public void setDeleteRequestBuilder(DeleteRequestBuilder deleteRequestBuilder) {
+            this.deleteRequestBuilder = deleteRequestBuilder;
+        }
+
+        public DeleteRequest getDeleteRequest() {
+            return deleteRequest;
+        }
+
+        public void setDeleteRequest(DeleteRequest deleteRequest) {
+            this.deleteRequest = deleteRequest;
+        }
+    }
+
+    public class ESSearchRequest {
+
+        private SearchRequestBuilder searchRequestBuilder;
+
+        private SearchRequest        searchRequest;
+
+        private SearchSourceBuilder  sourceBuilder;
+
+        public ESSearchRequest(String index){
+            if (mode == ESClientMode.TRANSPORT) {
+                searchRequestBuilder = transportClient.prepareSearch(index);
+            } else {
+                searchRequest = new SearchRequest(index);
+                sourceBuilder = new SearchSourceBuilder();
+            }
+        }
+
+        public ESSearchRequest setQuery(QueryBuilder queryBuilder) {
+            if (mode == ESClientMode.TRANSPORT) {
+                searchRequestBuilder.setQuery(queryBuilder);
+            } else {
+                sourceBuilder.query(queryBuilder);
+            }
+            return this;
+        }
+
+        public ESSearchRequest size(int size) {
+            if (mode == ESClientMode.TRANSPORT) {
+                searchRequestBuilder.setSize(size);
+            } else {
+                sourceBuilder.size(size);
+            }
+            return this;
+        }
+
+        public SearchResponse getResponse() {
+            if (mode == ESClientMode.TRANSPORT) {
+                return searchRequestBuilder.get();
+            } else {
+                searchRequest.source(sourceBuilder);
+                try {
+                    return restHighLevelClient.search(searchRequest, RequestOptions.DEFAULT);
+                } catch (IOException e) {
+                    throw new RuntimeException(e);
+                }
+            }
+        }
+
+        public SearchRequestBuilder getSearchRequestBuilder() {
+            return searchRequestBuilder;
+        }
+
+        public void setSearchRequestBuilder(SearchRequestBuilder searchRequestBuilder) {
+            this.searchRequestBuilder = searchRequestBuilder;
+        }
+
+        public SearchRequest getSearchRequest() {
+            return searchRequest;
+        }
+
+        public void setSearchRequest(SearchRequest searchRequest) {
+            this.searchRequest = searchRequest;
+        }
+    }
+
+    public class ES7xBulkRequest implements ESBulkRequest {
+
+        private BulkRequestBuilder bulkRequestBuilder;
+
+        private BulkRequest        bulkRequest;
+
+        public ES7xBulkRequest(){
+            if (mode == ESClientMode.TRANSPORT) {
+                bulkRequestBuilder = transportClient.prepareBulk();
+            } else {
+                bulkRequest = new BulkRequest();
+            }
+        }
+
+        public void resetBulk() {
+            if (mode == ESClientMode.TRANSPORT) {
+                bulkRequestBuilder = transportClient.prepareBulk();
+            } else {
+                bulkRequest = new BulkRequest();
+            }
+        }
+
+        public ES7xBulkRequest add(ESIndexRequest esIndexRequest) {
+            ES7xIndexRequest eir = (ES7xIndexRequest) esIndexRequest;
+            if (mode == ESClientMode.TRANSPORT) {
+                bulkRequestBuilder.add(eir.indexRequestBuilder);
+            } else {
+                bulkRequest.add(eir.indexRequest);
+            }
+            return this;
+        }
+
+        public ES7xBulkRequest add(ESUpdateRequest esUpdateRequest) {
+            ES7xUpdateRequest eur = (ES7xUpdateRequest) esUpdateRequest;
+            if (mode == ESClientMode.TRANSPORT) {
+                bulkRequestBuilder.add(eur.updateRequestBuilder);
+            } else {
+                bulkRequest.add(eur.updateRequest);
+            }
+            return this;
+        }
+
+        public ES7xBulkRequest add(ESDeleteRequest esDeleteRequest) {
+            ES7xDeleteRequest edr = (ES7xDeleteRequest) esDeleteRequest;
+            if (mode == ESClientMode.TRANSPORT) {
+                bulkRequestBuilder.add(edr.deleteRequestBuilder);
+            } else {
+                bulkRequest.add(edr.deleteRequest);
+            }
+            return this;
+        }
+
+        public int numberOfActions() {
+            if (mode == ESClientMode.TRANSPORT) {
+                return bulkRequestBuilder.numberOfActions();
+            } else {
+                return bulkRequest.numberOfActions();
+            }
+        }
+
+        public ESBulkResponse bulk() {
+            if (mode == ESClientMode.TRANSPORT) {
+                BulkResponse responses = bulkRequestBuilder.execute().actionGet();
+                return new ES7xBulkResponse(responses);
+            } else {
+                try {
+                    BulkResponse responses = restHighLevelClient.bulk(bulkRequest, RequestOptions.DEFAULT);
+                    return new ES7xBulkResponse(responses);
+                } catch (IOException e) {
+                    throw new RuntimeException(e);
+                }
+            }
+        }
+
+        public BulkRequestBuilder getBulkRequestBuilder() {
+            return bulkRequestBuilder;
+        }
+
+        public void setBulkRequestBuilder(BulkRequestBuilder bulkRequestBuilder) {
+            this.bulkRequestBuilder = bulkRequestBuilder;
+        }
+
+        public BulkRequest getBulkRequest() {
+            return bulkRequest;
+        }
+
+        public void setBulkRequest(BulkRequest bulkRequest) {
+            this.bulkRequest = bulkRequest;
+        }
+    }
+
+    public static class ES7xBulkResponse implements ESBulkRequest.ESBulkResponse {
+
+        private BulkResponse bulkResponse;
+
+        public ES7xBulkResponse(BulkResponse bulkResponse){
+            this.bulkResponse = bulkResponse;
+        }
+
+        @Override
+        public boolean hasFailures() {
+            return bulkResponse.hasFailures();
+        }
+
+        @Override
+        public void processFailBulkResponse(String errorMsg) {
+            for (BulkItemResponse itemResponse : bulkResponse.getItems()) {
+                if (!itemResponse.isFailed()) {
+                    continue;
+                }
+
+                if (itemResponse.getFailure().getStatus() == RestStatus.NOT_FOUND) {
+                    logger.error(itemResponse.getFailureMessage());
+                } else {
+                    throw new RuntimeException(errorMsg + itemResponse.getFailureMessage());
+                }
+            }
+        }
+    }
+
+    // ------ get/set ------
+    public ESClientMode getMode() {
+        return mode;
+    }
+
+    public void setMode(ESClientMode mode) {
+        this.mode = mode;
+    }
+
+    public TransportClient getTransportClient() {
+        return transportClient;
+    }
+
+    public void setTransportClient(TransportClient transportClient) {
+        this.transportClient = transportClient;
+    }
+
+    public RestHighLevelClient getRestHighLevelClient() {
+        return restHighLevelClient;
+    }
+
+    public void setRestHighLevelClient(RestHighLevelClient restHighLevelClient) {
+        this.restHighLevelClient = restHighLevelClient;
+    }
+}

client-adapter/es7x/src/main/resources/META-INF/canal/com.alibaba.otter.canal.client.adapter.OuterAdapter

@@ -0,0 +1 @@
+es7=com.alibaba.otter.canal.client.adapter.es7x.ES7xAdapter

client-adapter/es7x/src/main/resources/es7/biz_order.yml

@@ -0,0 +1,20 @@
+dataSourceKey: defaultDS
+destination: example
+groupId: g1
+esMapping:
+  _index: customer
+  _id: _id
+  relations:
+    customer_order:
+      name: order
+      parent: customer_id
+  sql: "select concat('oid_', t.id) as _id,
+        t.customer_id,
+        t.id as order_id,
+        t.serial_code as order_serial,
+        t.c_time as order_time
+        from biz_order t"
+  skips:
+    - customer_id
+  etlCondition: "where t.c_time>={}"
+  commitBatch: 3000

client-adapter/es7x/src/main/resources/es7/customer.yml

@@ -0,0 +1,46 @@
+dataSourceKey: defaultDS
+destination: example
+groupId: g1
+esMapping:
+  _index: customer
+  _id: id
+  relations:
+    customer_order:
+      name: customer
+  sql: "select t.id, t.name, t.email from customer t"
+  etlCondition: "where t.c_time>={}"
+  commitBatch: 3000
+
+
+#{
+#  "mappings":{
+#    "_doc":{
+#      "properties":{
+#        "id": {
+#          "type": "long"
+#        },
+#        "name": {
+#          "type": "text"
+#        },
+#        "email": {
+#          "type": "text"
+#        },
+#        "order_id": {
+#          "type": "long"
+#        },
+#        "order_serial": {
+#          "type": "text"
+#        },
+#        "order_time": {
+#          "type": "date"
+#        },
+#        "customer_order":{
+#          "type":"join",
+#          "relations":{
+#            "customer":"order"
+#          }
+#        }
+#      }
+#    }
+#  }
+#}

client-adapter/es7x/src/main/resources/es7/mytest_user.yml

@@ -0,0 +1,15 @@
+dataSourceKey: defaultDS
+destination: example
+groupId: g1
+esMapping:
+  _index: mytest_user
+  _id: _id
+#  upsert: true
+#  pk: id
+  sql: "select a.id as _id, a.name, a.role_id, b.role_name,
+        a.c_time from user a
+        left join role b on b.id=a.role_id"
+#  objFields:
+#    _labels: array:;
+  etlCondition: "where a.c_time>={}"
+  commitBatch: 3000

client-adapter/es7x/src/test/java/com/alibaba/otter/canal/client/adapter/es7x/test/ES7xTest.java

@@ -0,0 +1,119 @@
+package com.alibaba.otter.canal.client.adapter.es7x.test;
+
+import java.net.InetAddress;
+import java.net.UnknownHostException;
+import java.util.LinkedHashMap;
+import java.util.Map;
+
+import org.elasticsearch.action.bulk.BulkItemResponse;
+import org.elasticsearch.action.bulk.BulkRequestBuilder;
+import org.elasticsearch.action.bulk.BulkResponse;
+import org.elasticsearch.action.search.SearchResponse;
+import org.elasticsearch.client.transport.TransportClient;
+import org.elasticsearch.common.settings.Settings;
+import org.elasticsearch.common.transport.TransportAddress;
+import org.elasticsearch.index.query.QueryBuilders;
+import org.elasticsearch.rest.RestStatus;
+import org.elasticsearch.search.SearchHit;
+import org.elasticsearch.transport.client.PreBuiltTransportClient;
+import org.junit.After;
+import org.junit.Before;
+import org.junit.Ignore;
+import org.junit.Test;
+
+@Ignore
+public class ES7xTest {
+
+    private TransportClient transportClient;
+
+    @Before
+    public void init() throws UnknownHostException {
+        Settings.Builder settingBuilder = Settings.builder();
+        settingBuilder.put("cluster.name", TestConstant.clusterName);
+        Settings settings = settingBuilder.build();
+        transportClient = new PreBuiltTransportClient(settings);
+        String[] hostArray = TestConstant.esHosts.split(",");
+        for (String host : hostArray) {
+            int i = host.indexOf(":");
+            transportClient.addTransportAddress(new TransportAddress(InetAddress.getByName(host.substring(0, i)),
+                Integer.parseInt(host.substring(i + 1))));
+        }
+    }
+
+    @Test
+    public void test01() {
+        SearchResponse response = transportClient.prepareSearch("test")
+            .setQuery(QueryBuilders.termQuery("_id", "1"))
+            .setSize(10000)
+            .get();
+        for (SearchHit hit : response.getHits()) {
+            System.out.println(hit.getSourceAsMap().get("data").getClass());
+        }
+    }
+
+    @Test
+    public void test02() {
+        Map<String, Object> esFieldData = new LinkedHashMap<>();
+        esFieldData.put("userId", 2L);
+        esFieldData.put("eventId", 4L);
+        esFieldData.put("eventName", "网络异常");
+        esFieldData.put("description", "第四个事件信息");
+
+        Map<String, Object> relations = new LinkedHashMap<>();
+        esFieldData.put("user_event", relations);
+        relations.put("name", "event");
+        relations.put("parent", "2");
+
+        BulkRequestBuilder bulkRequestBuilder = transportClient.prepareBulk();
+        bulkRequestBuilder
+            .add(transportClient.prepareIndex("test", "osm", "2_4").setRouting("2").setSource(esFieldData));
+        commit(bulkRequestBuilder);
+    }
+
+    @Test
+    public void test03() {
+        Map<String, Object> esFieldData = new LinkedHashMap<>();
+        esFieldData.put("userId", 2L);
+        esFieldData.put("eventName", "网络异常1");
+
+        Map<String, Object> relations = new LinkedHashMap<>();
+        esFieldData.put("user_event", relations);
+        relations.put("name", "event");
+        relations.put("parent", "2");
+
+        BulkRequestBuilder bulkRequestBuilder = transportClient.prepareBulk();
+        bulkRequestBuilder.add(transportClient.prepareUpdate("test", "osm", "2_4").setRouting("2").setDoc(esFieldData));
+        commit(bulkRequestBuilder);
+    }
+
+    @Test
+    public void test04() {
+        BulkRequestBuilder bulkRequestBuilder = transportClient.prepareBulk();
+        bulkRequestBuilder.add(transportClient.prepareDelete("test", "osm", "2_4"));
+        commit(bulkRequestBuilder);
+    }
+
+    private void commit(BulkRequestBuilder bulkRequestBuilder) {
+        if (bulkRequestBuilder.numberOfActions() > 0) {
+            BulkResponse response = bulkRequestBuilder.execute().actionGet();
+            if (response.hasFailures()) {
+                for (BulkItemResponse itemResponse : response.getItems()) {
+                    if (!itemResponse.isFailed()) {
+                        continue;
+                    }
+
+                    if (itemResponse.getFailure().getStatus() == RestStatus.NOT_FOUND) {
+                        System.out.println(itemResponse.getFailureMessage());
+                    } else {
+                        System.out.println("ES bulk commit error" + itemResponse.getFailureMessage());
+                    }
+                }
+            }
+        }
+    }
+
+    @After
+    public void after() {
+        transportClient.close();
+    }
+}

client-adapter/es7x/src/test/java/com/alibaba/otter/canal/client/adapter/es7x/test/ESConnectionTest.java

@@ -0,0 +1,45 @@
+package com.alibaba.otter.canal.client.adapter.es7x.test;
+
+import java.net.UnknownHostException;
+import java.util.HashMap;
+import java.util.Map;
+
+import org.elasticsearch.cluster.metadata.MappingMetaData;
+import org.junit.Before;
+import org.junit.Ignore;
+import org.junit.Test;
+
+import com.alibaba.otter.canal.client.adapter.es7x.support.ESConnection;
+import org.springframework.util.Assert;
+
+@Ignore
+public class ESConnectionTest {
+
+    ESConnection esConnection;
+
+    @Before
+    public void init() throws UnknownHostException {
+        String[] hosts = new String[] { "127.0.0.1:9200" };
+        Map<String, String> properties = new HashMap<>();
+        properties.put("cluster.name", "elasticsearch");
+        esConnection = new ESConnection(hosts, properties, ESConnection.ESClientMode.REST);
+    }
+
+    @Test
+    public void test01() {
+        MappingMetaData mappingMetaData = esConnection.getMapping("mytest_user");
+
+        Map<String, Object> sourceMap = mappingMetaData.getSourceAsMap();
+        Map<String, Object> esMapping = (Map<String, Object>) sourceMap.get("properties");
+        for (Map.Entry<String, Object> entry : esMapping.entrySet()) {
+            Map<String, Object> value = (Map<String, Object>) entry.getValue();
+            if (value.containsKey("properties")) {
+                System.out.println(entry.getKey() + " object");
+            } else {
+                System.out.println(entry.getKey() + " " + value.get("type"));
+                Assert.notNull(entry.getKey(), "null column name");
+                Assert.notNull(value.get("type"), "null column type");
+            }
+        }
+    }
+}

client-adapter/es7x/src/test/java/com/alibaba/otter/canal/client/adapter/es7x/test/TestConstant.java

@@ -0,0 +1,40 @@
+package com.alibaba.otter.canal.client.adapter.es7x.test;
+
+import java.sql.SQLException;
+
+import com.alibaba.druid.pool.DruidDataSource;
+
+public class TestConstant {
+
+    public final static String    jdbcUrl      = "jdbc:mysql://127.0.0.1:3306/mytest?useUnicode=true";
+    public final static String    jdbcUser     = "root";
+    public final static String    jdbcPassword = "121212";
+
+    public final static String    esHosts      = "127.0.0.1:9300";
+    public final static String    clusterName  = "elasticsearch";
+
+    public final static DruidDataSource dataSource;
+
+    static {
+        dataSource = new DruidDataSource();
+        dataSource.setDriverClassName("com.mysql.jdbc.Driver");
+        dataSource.setUrl(jdbcUrl);
+        dataSource.setUsername(jdbcUser);
+        dataSource.setPassword(jdbcPassword);
+        dataSource.setInitialSize(1);
+        dataSource.setMinIdle(1);
+        dataSource.setMaxActive(1);
+        dataSource.setMaxWait(60000);
+        dataSource.setTimeBetweenEvictionRunsMillis(60000);
+        dataSource.setMinEvictableIdleTimeMillis(300000);
+        dataSource.setPoolPreparedStatements(false);
+        dataSource.setMaxPoolPreparedStatementPerConnectionSize(20);
+        dataSource.setValidationQuery("select 1");
+        try {
+            dataSource.init();
+        } catch (SQLException e) {
+            e.printStackTrace();
+        }
+    }
+
+}

client-adapter/escore/pom.xml

@@ -0,0 +1,45 @@
+<?xml version="1.0" encoding="UTF-8"?>
+<project xmlns="http://maven.apache.org/POM/4.0.0"
+         xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
+         xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
+    <parent>
+        <artifactId>canal.client-adapter</artifactId>
+        <groupId>com.alibaba.otter</groupId>
+        <version>1.1.5-SNAPSHOT</version>
+        <relativePath>../pom.xml</relativePath>
+    </parent>
+    <modelVersion>4.0.0</modelVersion>
+    <groupId>com.alibaba.otter</groupId>
+    <artifactId>client-adapter.escore</artifactId>
+    <packaging>jar</packaging>
+    <name>canal client adapter escore module for otter ${project.version}</name>
+
+    <dependencies>
+        <dependency>
+            <groupId>com.alibaba.otter</groupId>
+            <artifactId>client-adapter.common</artifactId>
+            <version>${project.version}</version>
+            <scope>provided</scope>
+        </dependency>
+        <dependency>
+            <groupId>com.alibaba.fastsql</groupId>
+            <artifactId>fastsql</artifactId>
+        </dependency>
+
+        <dependency>
+            <groupId>junit</groupId>
+            <artifactId>junit</artifactId>
+            <scope>test</scope>
+        </dependency>
+        <dependency>
+            <groupId>mysql</groupId>
+            <artifactId>mysql-connector-java</artifactId>
+            <scope>test</scope>
+        </dependency>
+        <dependency>
+            <groupId>com.h2database</groupId>
+            <artifactId>h2</artifactId>
+            <scope>test</scope>
+        </dependency>
+    </dependencies>
+</project>

client-adapter/escore/src/main/java/com/alibaba/otter/canal/client/adapter/es/core/ESAdapter.java

@@ -0,0 +1,177 @@
+package com.alibaba.otter.canal.client.adapter.es.core;
+
+import java.util.LinkedHashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.Properties;
+import java.util.concurrent.ConcurrentHashMap;
+import java.util.regex.Matcher;
+import java.util.regex.Pattern;
+
+import com.alibaba.otter.canal.client.adapter.es.core.support.ESTemplate;
+import org.apache.commons.lang.StringUtils;
+
+import com.alibaba.druid.pool.DruidDataSource;
+import com.alibaba.otter.canal.client.adapter.OuterAdapter;
+import com.alibaba.otter.canal.client.adapter.es.core.config.ESSyncConfig;
+import com.alibaba.otter.canal.client.adapter.es.core.config.ESSyncConfigLoader;
+import com.alibaba.otter.canal.client.adapter.es.core.config.SchemaItem;
+import com.alibaba.otter.canal.client.adapter.es.core.config.SqlParser;
+import com.alibaba.otter.canal.client.adapter.es.core.monitor.ESConfigMonitor;
+import com.alibaba.otter.canal.client.adapter.es.core.service.ESSyncService;
+import com.alibaba.otter.canal.client.adapter.support.DatasourceConfig;
+import com.alibaba.otter.canal.client.adapter.support.Dml;
+import com.alibaba.otter.canal.client.adapter.support.EtlResult;
+import com.alibaba.otter.canal.client.adapter.support.OuterAdapterConfig;
+
+/**
+ * ES外部适配器
+ *
+ * @author rewerma 2018-10-20
+ * @version 1.0.0
+ */
+public abstract class ESAdapter implements OuterAdapter {
+
+    protected Map<String, ESSyncConfig>              esSyncConfig        = new ConcurrentHashMap<>(); // 文件名对应配置
+    protected Map<String, Map<String, ESSyncConfig>> dbTableEsSyncConfig = new ConcurrentHashMap<>(); // schema-table对应配置
+
+    protected ESTemplate                             esTemplate;
+
+    protected ESSyncService                          esSyncService;
+
+    protected ESConfigMonitor                        esConfigMonitor;
+
+    protected Properties                             envProperties;
+
+    public ESSyncService getEsSyncService() {
+        return esSyncService;
+    }
+
+    public Map<String, ESSyncConfig> getEsSyncConfig() {
+        return esSyncConfig;
+    }
+
+    public Map<String, Map<String, ESSyncConfig>> getDbTableEsSyncConfig() {
+        return dbTableEsSyncConfig;
+    }
+
+    @Override
+    public void init(OuterAdapterConfig configuration, Properties envProperties) {
+        try {
+            this.envProperties = envProperties;
+            Map<String, ESSyncConfig> esSyncConfigTmp = ESSyncConfigLoader.load(envProperties);
+            // 过滤不匹配的key的配置
+            esSyncConfigTmp.forEach((key, config) -> {
+                if ((config.getOuterAdapterKey() == null && configuration.getKey() == null)
+                    || (config.getOuterAdapterKey() != null
+                        && config.getOuterAdapterKey().equalsIgnoreCase(configuration.getKey()))) {
+                    esSyncConfig.put(key, config);
+                }
+            });
+
+            for (Map.Entry<String, ESSyncConfig> entry : esSyncConfig.entrySet()) {
+                String configName = entry.getKey();
+                ESSyncConfig config = entry.getValue();
+
+                addSyncConfigToCache(configName, config);
+            }
+
+            esSyncService = new ESSyncService(esTemplate);
+
+            esConfigMonitor = new ESConfigMonitor();
+            esConfigMonitor.init(this, envProperties);
+        } catch (Throwable e) {
+            throw new RuntimeException(e);
+        }
+    }
+
+    @Override
+    public void sync(List<Dml> dmls) {
+        if (dmls == null || dmls.isEmpty()) {
+            return;
+        }
+        for (Dml dml : dmls) {
+            if (!dml.getIsDdl()) {
+                sync(dml);
+            }
+        }
+        esSyncService.commit(); // 批次统一提交
+
+    }
+
+    private void sync(Dml dml) {
+        String database = dml.getDatabase();
+        String table = dml.getTable();
+        Map<String, ESSyncConfig> configMap;
+        if (envProperties != null && !"tcp".equalsIgnoreCase(envProperties.getProperty("canal.conf.mode"))) {
+            configMap = dbTableEsSyncConfig
+                .get(StringUtils.trimToEmpty(dml.getDestination()) + "-" + StringUtils.trimToEmpty(dml.getGroupId())
+                     + "_" + database + "-" + table);
+        } else {
+            configMap = dbTableEsSyncConfig
+                .get(StringUtils.trimToEmpty(dml.getDestination()) + "_" + database + "-" + table);
+        }
+
+        if (configMap != null && !configMap.values().isEmpty()) {
+            esSyncService.sync(configMap.values(), dml);
+        }
+    }
+
+    @Override
+    public abstract EtlResult etl(String task, List<String> params);
+
+    @Override
+    public abstract Map<String, Object> count(String task);
+
+    @Override
+    public void destroy() {
+        if (esConfigMonitor != null) {
+            esConfigMonitor.destroy();
+        }
+    }
+
+    @Override
+    public String getDestination(String task) {
+        ESSyncConfig config = esSyncConfig.get(task);
+        if (config != null) {
+            return config.getDestination();
+        }
+        return null;
+    }
+
+    public void addSyncConfigToCache(String configName, ESSyncConfig config) {
+        Properties envProperties = this.envProperties;
+        SchemaItem schemaItem = SqlParser.parse(config.getEsMapping().getSql());
+        config.getEsMapping().setSchemaItem(schemaItem);
+
+        DruidDataSource dataSource = DatasourceConfig.DATA_SOURCES.get(config.getDataSourceKey());
+        if (dataSource == null || dataSource.getUrl() == null) {
+            throw new RuntimeException("No data source found: " + config.getDataSourceKey());
+        }
+        Pattern pattern = Pattern.compile(".*:(.*)://.*/(.*)\\?.*$");
+        Matcher matcher = pattern.matcher(dataSource.getUrl());
+        if (!matcher.find()) {
+            throw new RuntimeException("Not found the schema of jdbc-url: " + config.getDataSourceKey());
+        }
+        String schema = matcher.group(2);
+
+        schemaItem.getAliasTableItems().values().forEach(tableItem -> {
+            Map<String, ESSyncConfig> esSyncConfigMap;
+            if (envProperties != null
+                    && !"tcp".equalsIgnoreCase(envProperties.getProperty("canal.conf.mode"))) {
+                esSyncConfigMap = dbTableEsSyncConfig
+                        .computeIfAbsent(StringUtils.trimToEmpty(config.getDestination()) + "-"
+                                        + StringUtils.trimToEmpty(config.getGroupId()) + "_" + schema + "-"
+                                        + tableItem.getTableName(),
+                                k -> new ConcurrentHashMap<>());
+            } else {
+                esSyncConfigMap = dbTableEsSyncConfig
+                        .computeIfAbsent(StringUtils.trimToEmpty(config.getDestination()) + "_" + schema + "-"
+                                        + tableItem.getTableName(),
+                                k -> new ConcurrentHashMap<>());
+            }
+
+            esSyncConfigMap.put(configName, config);
+        });
+    }
+}

client-adapter/escore/src/main/java/com/alibaba/otter/canal/client/adapter/es/core/config/ESSyncConfig.java

@@ -1,12 +1,12 @@
-package com.alibaba.otter.canal.client.adapter.es.config;
-
-import com.alibaba.otter.canal.client.adapter.support.AdapterConfig;
+package com.alibaba.otter.canal.client.adapter.es.core.config;
 
 import java.util.ArrayList;
 import java.util.LinkedHashMap;
 import java.util.List;
 import java.util.Map;
 
+import com.alibaba.otter.canal.client.adapter.support.AdapterConfig;
+
 /**
  * ES 映射配置
  *
@@ -15,21 +15,23 @@ import java.util.Map;
  */
 public class ESSyncConfig implements AdapterConfig {
 
-    private String    dataSourceKey;   // 数据源key
+    private String    dataSourceKey;    // 数据源key
 
-    private String    outerAdapterKey; // adapter key
+    private String    outerAdapterKey;  // adapter key
 
-    private String    groupId;         // group id
+    private String    groupId;          // group id
 
-    private String    destination;     // canal destination
+    private String    destination;      // canal destination
 
     private ESMapping esMapping;
 
+    private String    esVersion = "es6";
+
     public void validate() {
         if (esMapping._index == null) {
             throw new NullPointerException("esMapping._index");
         }
-        if (esMapping._type == null) {
+        if ("es6".equals(esVersion) && esMapping._type == null) {
             throw new NullPointerException("esMapping._type");
         }
         if (esMapping._id == null && esMapping.getPk() == null) {
@@ -80,11 +82,19 @@ public class ESSyncConfig implements AdapterConfig {
         this.esMapping = esMapping;
     }
 
-    public ESMapping getMapping(){
+    public ESMapping getMapping() {
         return esMapping;
     }
 
-    public static class ESMapping implements AdapterMapping{
+    public String getEsVersion() {
+        return esVersion;
+    }
+
+    public void setEsVersion(String esVersion) {
+        this.esVersion = esVersion;
+    }
+
+    public static class ESMapping implements AdapterMapping {
 
         private String                       _index;
         private String                       _type;

client-adapter/escore/src/main/java/com/alibaba/otter/canal/client/adapter/es/core/config/ESSyncConfigLoader.java

@@ -1,4 +1,4 @@
-package com.alibaba.otter.canal.client.adapter.es.config;
+package com.alibaba.otter.canal.client.adapter.es.core.config;
 
 import java.util.LinkedHashMap;
 import java.util.Map;
@@ -25,12 +25,14 @@ public class ESSyncConfigLoader {
 
         Map<String, ESSyncConfig> esSyncConfig = new LinkedHashMap<>();
 
-        Map<String, String> configContentMap = MappingConfigsLoader.loadConfigs("es");
+        String esv = envProperties.getProperty("es.version");
+        Map<String, String> configContentMap = MappingConfigsLoader.loadConfigs(esv);
         configContentMap.forEach((fileName, content) -> {
             ESSyncConfig config = YmlConfigBinder.bindYmlToObj(null, content, ESSyncConfig.class, null, envProperties);
             if (config == null) {
                 return;
             }
+            config.setEsVersion(esv);
             try {
                 config.validate();
             } catch (Exception e) {

client-adapter/escore/src/main/java/com/alibaba/otter/canal/client/adapter/es/core/config/SchemaItem.java

@@ -1,11 +1,10 @@
-package com.alibaba.otter.canal.client.adapter.es.config;
+package com.alibaba.otter.canal.client.adapter.es.core.config;
 
 import java.util.ArrayList;
 import java.util.LinkedHashMap;
 import java.util.List;
 import java.util.Map;
 
-import com.alibaba.otter.canal.client.adapter.es.config.ESSyncConfig.ESMapping;
 
 /**
  * ES 映射配置视图
@@ -87,10 +86,13 @@ public class SchemaItem {
                         .forEach((fieldName, fieldItem) -> fieldItem.getColumnItems().forEach(columnItem -> {
                             // TableItem tableItem = getAliasTableItems().get(columnItem.getOwner());
                             // if (!tableItem.isSubQuery()) {
-                            List<FieldItem> fieldItems = columnFields.computeIfAbsent(
-                                columnItem.getOwner() + "." + columnItem.getColumnName(),
-                                k -> new ArrayList<>());
-                            fieldItems.add(fieldItem);
+                            //当数据列并非原始列时，columnName是空的，例如concat('px',id)
+                            if(columnItem.getColumnName() != null) {
+                                List<FieldItem> fieldItems = columnFields.computeIfAbsent(
+                                        columnItem.getOwner() + "." + columnItem.getColumnName(),
+                                        k -> new ArrayList<>());
+                                fieldItems.add(fieldItem);
+                            }
                             // } else {
                             // tableItem.getSubQueryFields().forEach(subQueryField -> {
                             // List<FieldItem> fieldItems = columnFields.computeIfAbsent(
@@ -133,7 +135,7 @@ public class SchemaItem {
         }
     }
 
-    public FieldItem getIdFieldItem(ESMapping mapping) {
+    public FieldItem getIdFieldItem(ESSyncConfig.ESMapping mapping) {
         if (mapping.get_id() != null) {
             return getSelectFields().get(mapping.get_id());
         } else {

client-adapter/escore/src/main/java/com/alibaba/otter/canal/client/adapter/es/core/config/SqlParser.java

@@ -1,4 +1,4 @@
-package com.alibaba.otter.canal.client.adapter.es.config;
+package com.alibaba.otter.canal.client.adapter.es.core.config;
 
 import static com.alibaba.fastsql.sql.ast.expr.SQLBinaryOperator.BooleanAnd;
 import static com.alibaba.fastsql.sql.ast.expr.SQLBinaryOperator.Equality;
@@ -10,19 +10,15 @@ import java.util.stream.Collectors;
 import com.alibaba.fastsql.sql.SQLUtils;
 import com.alibaba.fastsql.sql.ast.SQLExpr;
 import com.alibaba.fastsql.sql.ast.expr.*;
-import com.alibaba.fastsql.sql.ast.statement.SQLExprTableSource;
-import com.alibaba.fastsql.sql.ast.statement.SQLJoinTableSource;
-import com.alibaba.fastsql.sql.ast.statement.SQLSelectStatement;
-import com.alibaba.fastsql.sql.ast.statement.SQLSubqueryTableSource;
-import com.alibaba.fastsql.sql.ast.statement.SQLTableSource;
+import com.alibaba.fastsql.sql.ast.statement.*;
 import com.alibaba.fastsql.sql.dialect.mysql.ast.statement.MySqlSelectQueryBlock;
 import com.alibaba.fastsql.sql.dialect.mysql.parser.MySqlStatementParser;
 import com.alibaba.fastsql.sql.parser.ParserException;
 import com.alibaba.fastsql.sql.parser.SQLStatementParser;
-import com.alibaba.otter.canal.client.adapter.es.config.SchemaItem.ColumnItem;
-import com.alibaba.otter.canal.client.adapter.es.config.SchemaItem.FieldItem;
-import com.alibaba.otter.canal.client.adapter.es.config.SchemaItem.RelationFieldsPair;
-import com.alibaba.otter.canal.client.adapter.es.config.SchemaItem.TableItem;
+import com.alibaba.otter.canal.client.adapter.es.core.config.SchemaItem.ColumnItem;
+import com.alibaba.otter.canal.client.adapter.es.core.config.SchemaItem.FieldItem;
+import com.alibaba.otter.canal.client.adapter.es.core.config.SchemaItem.RelationFieldsPair;
+import com.alibaba.otter.canal.client.adapter.es.core.config.SchemaItem.TableItem;
 
 /**
  * ES同步指定sql格式解析
@@ -125,8 +121,8 @@ public class SqlParser {
         } else if (expr instanceof SQLCaseExpr) {
             SQLCaseExpr sqlCaseExpr = (SQLCaseExpr) expr;
             fieldItem.setMethod(true);
-            sqlCaseExpr.getItems().forEach(item-> visitColumn(item.getConditionExpr(), fieldItem));
-        }else if(expr instanceof SQLCharExpr) {
+            sqlCaseExpr.getItems().forEach(item -> visitColumn(item.getConditionExpr(), fieldItem));
+        } else if (expr instanceof SQLCharExpr) {
             SQLCharExpr sqlCharExpr = (SQLCharExpr) expr;
             String owner = null;
             String columnName = null;
@@ -232,4 +228,49 @@ public class SqlParser {
             throw new UnsupportedOperationException("Unsupported for complex of on-condition");
         }
     }
+
+    public static MySqlSelectQueryBlock parseSQLSelectQueryBlock(String sql) {
+        if (sql == null || "".equals(sql)) {
+            return null;
+        }
+        SQLStatementParser parser = new MySqlStatementParser(sql);
+        SQLSelectStatement statement = (SQLSelectStatement) parser.parseStatement();
+        return (MySqlSelectQueryBlock) statement.getSelect().getQuery();
+    }
+
+    public static String parse4SQLSelectItem(MySqlSelectQueryBlock sqlSelectQueryBlock) {
+        List<SQLSelectItem> selectItems = sqlSelectQueryBlock.getSelectList();
+        StringBuilder subSql = new StringBuilder();
+        int i = 0;
+        for (SQLSelectItem sqlSelectItem : selectItems) {
+            if (i != 0) {
+                subSql.append(",");
+            } else {
+                i++;
+            }
+            subSql.append(SQLUtils.toMySqlString(sqlSelectItem));
+        }
+        return subSql.toString();
+    }
+
+    public static String parse4FromTableSource(MySqlSelectQueryBlock sqlSelectQueryBlock) {
+        SQLTableSource sqlTableSource = sqlSelectQueryBlock.getFrom();
+        return SQLUtils.toMySqlString(sqlTableSource);
+    }
+
+    public static String parse4WhereItem(MySqlSelectQueryBlock sqlSelectQueryBlock) {
+        SQLExpr sqlExpr = sqlSelectQueryBlock.getWhere();
+        if (sqlExpr != null) {
+            return SQLUtils.toMySqlString(sqlExpr);
+        }
+        return null;
+    }
+
+    public static String parse4GroupBy(MySqlSelectQueryBlock sqlSelectQueryBlock) {
+        SQLSelectGroupByClause expr = sqlSelectQueryBlock.getGroupBy();
+        if (expr != null) {
+            return SQLUtils.toMySqlString(expr);
+        }
+        return null;
+    }
 }

client-adapter/escore/src/main/java/com/alibaba/otter/canal/client/adapter/es/core/monitor/ESConfigMonitor.java

@@ -1,4 +1,4 @@
-package com.alibaba.otter.canal.client.adapter.es.monitor;
+package com.alibaba.otter.canal.client.adapter.es.core.monitor;
 
 import java.io.File;
 import java.util.HashMap;
@@ -7,6 +7,7 @@ import java.util.Properties;
 import java.util.regex.Matcher;
 import java.util.regex.Pattern;
 
+import com.alibaba.otter.canal.client.adapter.es.core.ESAdapter;
 import org.apache.commons.io.filefilter.FileFilterUtils;
 import org.apache.commons.io.monitor.FileAlterationListenerAdaptor;
 import org.apache.commons.io.monitor.FileAlterationMonitor;
@@ -16,19 +17,18 @@ import org.slf4j.LoggerFactory;
 
 import com.alibaba.druid.pool.DruidDataSource;
 import com.alibaba.otter.canal.client.adapter.config.YmlConfigBinder;
-import com.alibaba.otter.canal.client.adapter.es.ESAdapter;
-import com.alibaba.otter.canal.client.adapter.es.config.ESSyncConfig;
-import com.alibaba.otter.canal.client.adapter.es.config.SchemaItem;
-import com.alibaba.otter.canal.client.adapter.es.config.SqlParser;
+import com.alibaba.otter.canal.client.adapter.es.core.config.ESSyncConfig;
+import com.alibaba.otter.canal.client.adapter.es.core.config.SchemaItem;
+import com.alibaba.otter.canal.client.adapter.es.core.config.SqlParser;
 import com.alibaba.otter.canal.client.adapter.support.DatasourceConfig;
 import com.alibaba.otter.canal.client.adapter.support.MappingConfigsLoader;
 import com.alibaba.otter.canal.client.adapter.support.Util;
 
 public class ESConfigMonitor {
 
-    private static final Logger   logger      = LoggerFactory.getLogger(ESConfigMonitor.class);
+    private static final Logger   logger = LoggerFactory.getLogger(ESConfigMonitor.class);
 
-    private static final String   adapterName = "es";
+    private String                adapterName;
 
     private ESAdapter             esAdapter;
 
@@ -39,6 +39,7 @@ public class ESConfigMonitor {
     public void init(ESAdapter esAdapter, Properties envProperties) {
         this.esAdapter = esAdapter;
         this.envProperties = envProperties;
+        this.adapterName = envProperties.getProperty("es.version");
         File confDir = Util.getConfDirPath(adapterName);
         try {
             FileAlterationObserver observer = new FileAlterationObserver(confDir,
@@ -130,26 +131,7 @@ public class ESConfigMonitor {
         private void addConfigToCache(File file, ESSyncConfig config) {
             esAdapter.getEsSyncConfig().put(file.getName(), config);
 
-            SchemaItem schemaItem = SqlParser.parse(config.getEsMapping().getSql());
-            config.getEsMapping().setSchemaItem(schemaItem);
-
-            DruidDataSource dataSource = DatasourceConfig.DATA_SOURCES.get(config.getDataSourceKey());
-            if (dataSource == null || dataSource.getUrl() == null) {
-                throw new RuntimeException("No data source found: " + config.getDataSourceKey());
-            }
-            Pattern pattern = Pattern.compile(".*:(.*)://.*/(.*)\\?.*$");
-            Matcher matcher = pattern.matcher(dataSource.getUrl());
-            if (!matcher.find()) {
-                throw new RuntimeException("Not found the schema of jdbc-url: " + config.getDataSourceKey());
-            }
-            String schema = matcher.group(2);
-
-            schemaItem.getAliasTableItems().values().forEach(tableItem -> {
-                Map<String, ESSyncConfig> esSyncConfigMap = esAdapter.getDbTableEsSyncConfig()
-                    .computeIfAbsent(schema + "-" + tableItem.getTableName(), k -> new HashMap<>());
-                esSyncConfigMap.put(file.getName(), config);
-            });
-
+            esAdapter.addSyncConfigToCache(file.getName(), config);
         }
 
         private void deleteConfigFromCache(File file) {

client-adapter/escore/src/main/java/com/alibaba/otter/canal/client/adapter/es/core/service/ESSyncService.java

@@ -1,22 +1,24 @@
-package com.alibaba.otter.canal.client.adapter.es.service;
+package com.alibaba.otter.canal.client.adapter.es.core.service;
 
 import java.util.*;
 
 import javax.sql.DataSource;
 
+import com.alibaba.fastsql.sql.dialect.mysql.ast.statement.MySqlSelectQueryBlock;
+import com.alibaba.otter.canal.client.adapter.es.core.config.SqlParser;
 import org.slf4j.Logger;
 import org.slf4j.LoggerFactory;
 
 import com.alibaba.fastjson.JSON;
 import com.alibaba.fastjson.serializer.SerializerFeature;
-import com.alibaba.otter.canal.client.adapter.es.config.ESSyncConfig;
-import com.alibaba.otter.canal.client.adapter.es.config.ESSyncConfig.ESMapping;
-import com.alibaba.otter.canal.client.adapter.es.config.SchemaItem;
-import com.alibaba.otter.canal.client.adapter.es.config.SchemaItem.ColumnItem;
-import com.alibaba.otter.canal.client.adapter.es.config.SchemaItem.FieldItem;
-import com.alibaba.otter.canal.client.adapter.es.config.SchemaItem.TableItem;
-import com.alibaba.otter.canal.client.adapter.es.support.ESSyncUtil;
-import com.alibaba.otter.canal.client.adapter.es.support.ESTemplate;
+import com.alibaba.otter.canal.client.adapter.es.core.config.ESSyncConfig;
+import com.alibaba.otter.canal.client.adapter.es.core.config.ESSyncConfig.ESMapping;
+import com.alibaba.otter.canal.client.adapter.es.core.config.SchemaItem;
+import com.alibaba.otter.canal.client.adapter.es.core.config.SchemaItem.ColumnItem;
+import com.alibaba.otter.canal.client.adapter.es.core.config.SchemaItem.FieldItem;
+import com.alibaba.otter.canal.client.adapter.es.core.config.SchemaItem.TableItem;
+import com.alibaba.otter.canal.client.adapter.es.core.support.ESSyncUtil;
+import com.alibaba.otter.canal.client.adapter.es.core.support.ESTemplate;
 import com.alibaba.otter.canal.client.adapter.support.DatasourceConfig;
 import com.alibaba.otter.canal.client.adapter.support.Dml;
 import com.alibaba.otter.canal.client.adapter.support.Util;
@@ -581,16 +583,35 @@ public class ESSyncService {
     private void subTableSimpleFieldOperation(ESSyncConfig config, Dml dml, Map<String, Object> data,
                                               Map<String, Object> old, TableItem tableItem) {
         ESMapping mapping = config.getEsMapping();
-        StringBuilder sql = new StringBuilder(
-            "SELECT * FROM (" + tableItem.getSubQuerySql() + ") " + tableItem.getAlias() + " WHERE ");
+
+        MySqlSelectQueryBlock queryBlock = SqlParser.parseSQLSelectQueryBlock(tableItem.getSubQuerySql());
+        StringBuilder sql = new StringBuilder();
+        sql.append("SELECT ")
+            .append(SqlParser.parse4SQLSelectItem(queryBlock))
+            .append(" FROM ")
+            .append(SqlParser.parse4FromTableSource(queryBlock));
+
+        String whereSql = SqlParser.parse4WhereItem(queryBlock);
+        if (whereSql != null) {
+            sql.append(" WHERE ").append(whereSql);
+        } else {
+            sql.append(" WHERE 1=1 ");
+        }
+
+        List<Object> values = new ArrayList<>();
 
         for (FieldItem fkFieldItem : tableItem.getRelationTableFields().keySet()) {
             String columnName = fkFieldItem.getColumn().getColumnName();
             Object value = esTemplate.getValFromData(mapping, data, fkFieldItem.getFieldName(), columnName);
-            ESSyncUtil.appendCondition(sql, value, tableItem.getAlias(), columnName);
+            sql.append(" AND ").append(columnName).append("=? ");
+            values.add(value);
         }
-        int len = sql.length();
-        sql.delete(len - 5, len);
+
+        String groupSql = SqlParser.parse4GroupBy(queryBlock);
+        if (groupSql != null) {
+            sql.append(groupSql);
+        }
+
         DataSource ds = DatasourceConfig.DATA_SOURCES.get(config.getDataSourceKey());
         if (logger.isTraceEnabled()) {
             logger.trace("Join table update es index by query sql, destination:{}, table: {}, index: {}, sql: {}",
@@ -599,7 +620,7 @@ public class ESSyncService {
                 mapping.get_index(),
                 sql.toString().replace("\n", " "));
         }
-        Util.sqlRS(ds, sql.toString(), rs -> {
+        Util.sqlRS(ds, sql.toString(), values, rs -> {
             try {
                 while (rs.next()) {
                     Map<String, Object> esFieldData = new LinkedHashMap<>();
@@ -674,14 +695,14 @@ public class ESSyncService {
     private void wholeSqlOperation(ESSyncConfig config, Dml dml, Map<String, Object> data, Map<String, Object> old,
                                    TableItem tableItem) {
         ESMapping mapping = config.getEsMapping();
-        //防止最后出现groupby 导致sql解析异常
+        // 防止最后出现groupby 导致sql解析异常
         String[] sqlSplit = mapping.getSql().split("GROUP\\ BY(?!(.*)ON)");
         String sqlNoWhere = sqlSplit[0];
 
         String sqlGroupBy = "";
 
-        if(sqlSplit.length > 1){
-            sqlGroupBy =  "GROUP BY "+ sqlSplit[1];
+        if (sqlSplit.length > 1) {
+            sqlGroupBy = "GROUP BY " + sqlSplit[1];
         }
 
         StringBuilder sql = new StringBuilder(sqlNoWhere + " WHERE ");

client-adapter/escore/src/main/java/com/alibaba/otter/canal/client/adapter/es/core/support/ESBulkRequest.java

@@ -0,0 +1,43 @@
+package com.alibaba.otter.canal.client.adapter.es.core.support;
+
+import java.util.Map;
+
+public interface ESBulkRequest {
+
+    void resetBulk();
+
+    ESBulkRequest add(ESIndexRequest esIndexRequest);
+
+    ESBulkRequest add(ESUpdateRequest esUpdateRequest);
+
+    ESBulkRequest add(ESDeleteRequest esDeleteRequest);
+
+    int numberOfActions();
+
+    ESBulkResponse bulk();
+
+    interface ESIndexRequest {
+
+        ESIndexRequest setSource(Map<String, ?> source);
+
+        ESIndexRequest setRouting(String routing);
+    }
+
+    interface ESUpdateRequest {
+
+        ESUpdateRequest setDoc(Map source);
+
+        ESUpdateRequest setDocAsUpsert(boolean shouldUpsertDoc);
+
+        ESUpdateRequest setRouting(String routing);
+    }
+
+    interface ESDeleteRequest {
+    }
+
+    interface ESBulkResponse {
+        boolean hasFailures();
+
+        void processFailBulkResponse(String errorMsg);
+    }
+}

client-adapter/escore/src/main/java/com/alibaba/otter/canal/client/adapter/es/core/support/ESSyncUtil.java

@@ -1,17 +1,11 @@
-package com.alibaba.otter.canal.client.adapter.es.support;
+package com.alibaba.otter.canal.client.adapter.es.core.support;
 
 import java.io.IOException;
 import java.io.InputStream;
 import java.nio.charset.StandardCharsets;
 import java.sql.Blob;
 import java.sql.SQLException;
-import java.util.ArrayList;
-import java.util.Arrays;
-import java.util.Date;
-import java.util.HashMap;
-import java.util.LinkedHashSet;
-import java.util.Map;
-import java.util.Set;
+import java.util.*;
 
 import org.apache.commons.codec.binary.Base64;
 import org.joda.time.DateTime;
@@ -19,10 +13,10 @@ import org.slf4j.Logger;
 import org.slf4j.LoggerFactory;
 
 import com.alibaba.fastjson.JSON;
-import com.alibaba.otter.canal.client.adapter.es.config.ESSyncConfig.ESMapping;
-import com.alibaba.otter.canal.client.adapter.es.config.SchemaItem;
-import com.alibaba.otter.canal.client.adapter.es.config.SchemaItem.ColumnItem;
-import com.alibaba.otter.canal.client.adapter.es.config.SchemaItem.TableItem;
+import com.alibaba.otter.canal.client.adapter.es.core.config.ESSyncConfig.ESMapping;
+import com.alibaba.otter.canal.client.adapter.es.core.config.SchemaItem;
+import com.alibaba.otter.canal.client.adapter.es.core.config.SchemaItem.ColumnItem;
+import com.alibaba.otter.canal.client.adapter.es.core.config.SchemaItem.TableItem;
 import com.alibaba.otter.canal.client.adapter.support.Util;
 
 /**

client-adapter/escore/src/main/java/com/alibaba/otter/canal/client/adapter/es/core/support/ESTemplate.java

@@ -0,0 +1,69 @@
+package com.alibaba.otter.canal.client.adapter.es.core.support;
+
+import java.sql.ResultSet;
+import java.sql.SQLException;
+import java.util.Map;
+
+import com.alibaba.otter.canal.client.adapter.es.core.config.ESSyncConfig;
+import com.alibaba.otter.canal.client.adapter.es.core.config.ESSyncConfig.ESMapping;
+
+public interface ESTemplate {
+
+    /**
+     * 插入数据
+     *
+     * @param mapping 配置对象
+     * @param pkVal 主键值
+     * @param esFieldData 数据Map
+     */
+    void insert(ESMapping mapping, Object pkVal, Map<String, Object> esFieldData);
+
+    /**
+     * 根据主键更新数据
+     *
+     * @param mapping 配置对象
+     * @param pkVal 主键值
+     * @param esFieldData 数据Map
+     */
+    void update(ESMapping mapping, Object pkVal, Map<String, Object> esFieldData);
+
+    /**
+     * update by query
+     *
+     * @param config 配置对象
+     * @param paramsTmp sql查询条件
+     * @param esFieldData 数据Map
+     */
+    void updateByQuery(ESSyncConfig config, Map<String, Object> paramsTmp, Map<String, Object> esFieldData);
+
+    /**
+     * 通过主键删除数据
+     *
+     * @param mapping 配置对象
+     * @param pkVal 主键值
+     * @param esFieldData 数据Map
+     */
+    void delete(ESMapping mapping, Object pkVal, Map<String, Object> esFieldData);
+
+    /**
+     * 提交批次
+     */
+    void commit();
+
+    Object getValFromRS(ESMapping mapping, ResultSet resultSet, String fieldName,
+                        String columnName) throws SQLException;
+
+    Object getESDataFromRS(ESMapping mapping, ResultSet resultSet, Map<String, Object> esFieldData) throws SQLException;
+
+    Object getIdValFromRS(ESMapping mapping, ResultSet resultSet) throws SQLException;
+
+    Object getESDataFromRS(ESMapping mapping, ResultSet resultSet, Map<String, Object> dmlOld,
+                           Map<String, Object> esFieldData) throws SQLException;
+
+    Object getValFromData(ESMapping mapping, Map<String, Object> dmlData, String fieldName, String columnName);
+
+    Object getESDataFromDmlData(ESMapping mapping, Map<String, Object> dmlData, Map<String, Object> esFieldData);
+
+    Object getESDataFromDmlData(ESMapping mapping, Map<String, Object> dmlData, Map<String, Object> dmlOld,
+                                Map<String, Object> esFieldData);
+}

client-adapter/hbase/pom.xml

@@ -3,7 +3,8 @@
     <parent>
         <artifactId>canal.client-adapter</artifactId>
         <groupId>com.alibaba.otter</groupId>
-        <version>1.1.4</version>
+        <version>1.1.5-SNAPSHOT</version>
+        <relativePath>../pom.xml</relativePath>
     </parent>
     <modelVersion>4.0.0</modelVersion>
     <groupId>com.alibaba.otter</groupId>
@@ -35,7 +36,6 @@
         <dependency>
             <groupId>com.google.guava</groupId>
             <artifactId>guava</artifactId>
-            <version>12.0.1</version>
         </dependency>
     </dependencies>
 

client-adapter/launcher/pom.xml

@@ -3,7 +3,8 @@
     <parent>
         <artifactId>canal.client-adapter</artifactId>
         <groupId>com.alibaba.otter</groupId>
-        <version>1.1.4</version>
+        <version>1.1.5-SNAPSHOT</version>
+        <relativePath>../pom.xml</relativePath>
     </parent>
     <modelVersion>4.0.0</modelVersion>
     <groupId>com.alibaba.otter</groupId>
@@ -37,7 +38,6 @@
         <dependency>
             <groupId>org.yaml</groupId>
             <artifactId>snakeyaml</artifactId>
-            <version>1.19</version>
         </dependency>
         <dependency>
             <groupId>org.springframework.boot</groupId>
@@ -51,46 +51,46 @@
         <dependency>
             <groupId>org.springframework.cloud</groupId>
             <artifactId>spring-cloud-context</artifactId>
-            <version>2.0.0.RELEASE</version>
         </dependency>
         <dependency>
             <groupId>org.apache.curator</groupId>
             <artifactId>curator-recipes</artifactId>
-            <version>2.10.0</version>
         </dependency>
         <!-- 单独引入rocketmq依赖 -->
         <dependency>
             <groupId>org.apache.rocketmq</groupId>
             <artifactId>rocketmq-client</artifactId>
-	    <version>4.5.1</version>
         </dependency>
         <!-- 单独引入kafka依赖 -->
         <dependency>
             <groupId>org.apache.kafka</groupId>
             <artifactId>kafka-clients</artifactId>
-            <version>1.1.1</version>
         </dependency>
 
+        <dependency>
+            <groupId>com.rabbitmq</groupId>
+            <artifactId>amqp-client</artifactId>
+        </dependency>
+        <dependency>
+            <groupId>com.alibaba.mq-amqp</groupId>
+            <artifactId>mq-amqp-client</artifactId>
+        </dependency>
         <!-- jdbc -->
         <dependency>
             <groupId>mysql</groupId>
             <artifactId>mysql-connector-java</artifactId>
-            <version>5.1.40</version>
         </dependency>
         <dependency>
             <groupId>org.postgresql</groupId>
             <artifactId>postgresql</artifactId>
-            <version>42.1.4</version>
         </dependency>
         <dependency>
             <groupId>com.oracle</groupId>
             <artifactId>ojdbc6</artifactId>
-            <version>11.2.0.3</version>
         </dependency>
         <dependency>
             <groupId>com.microsoft.sqlserver</groupId>
             <artifactId>mssql-jdbc</artifactId>
-            <version>7.0.0.jre8</version>
         </dependency>
 
         <!-- outer adapter jar with dependencies-->
@@ -122,7 +122,20 @@
         </dependency>
         <dependency>
             <groupId>com.alibaba.otter</groupId>
-            <artifactId>client-adapter.elasticsearch</artifactId>
+            <artifactId>client-adapter.es6x</artifactId>
+            <version>${project.version}</version>
+            <exclusions>
+                <exclusion>
+                    <artifactId>*</artifactId>
+                    <groupId>*</groupId>
+                </exclusion>
+            </exclusions>
+            <classifier>jar-with-dependencies</classifier>
+            <scope>provided</scope>
+        </dependency>
+        <dependency>
+            <groupId>com.alibaba.otter</groupId>
+            <artifactId>client-adapter.es7x</artifactId>
             <version>${project.version}</version>
             <exclusions>
                 <exclusion>

client-adapter/launcher/src/main/assembly/dev.xml

@@ -30,8 +30,15 @@
             </includes>
         </fileSet>
         <fileSet>
-            <directory>../elasticsearch/src/main/resources/es</directory>
-            <outputDirectory>/conf/es</outputDirectory>
+            <directory>../es6x/src/main/resources/es6</directory>
+            <outputDirectory>/conf/es6</outputDirectory>
+            <includes>
+                <include>**/*</include>
+            </includes>
+        </fileSet>
+        <fileSet>
+            <directory>../es7x/src/main/resources/es7</directory>
+            <outputDirectory>/conf/es7</outputDirectory>
             <includes>
                 <include>**/*</include>
             </includes>

client-adapter/launcher/src/main/assembly/release.xml

@@ -31,8 +31,15 @@
             </includes>
         </fileSet>
         <fileSet>
-            <directory>../elasticsearch/src/main/resources/es</directory>
-            <outputDirectory>/conf/es</outputDirectory>
+            <directory>../es6x/src/main/resources/es6</directory>
+            <outputDirectory>/conf/es6</outputDirectory>
+            <includes>
+                <include>**/*</include>
+            </includes>
+        </fileSet>
+        <fileSet>
+            <directory>../es7x/src/main/resources/es7</directory>
+            <outputDirectory>/conf/es7</outputDirectory>
             <includes>
                 <include>**/*</include>
             </includes>
@@ -71,4 +78,4 @@
             </excludes>
         </dependencySet>
     </dependencySets>
-</assembly>
+</assembly>

client-adapter/launcher/src/main/java/com/alibaba/otter/canal/adapter/launcher/common/SyncSwitch.java

@@ -64,6 +64,27 @@ public class SyncSwitch {
         }
     }
 
+    public synchronized void refresh() {
+        for (String destination : adapterCanalConfig.DESTINATIONS) {
+            BooleanMutex booleanMutex;
+            if (mode == Mode.DISTRIBUTED) {
+                CuratorFramework curator = curatorClient.getCurator();
+                booleanMutex = DISTRIBUTED_LOCK.get(destination);
+                if (booleanMutex == null) {
+                    BooleanMutex mutex = new BooleanMutex(true);
+                    initMutex(curator, destination, mutex);
+                    DISTRIBUTED_LOCK.put(destination, mutex);
+                    startListen(destination, mutex);
+                }
+            } else {
+                booleanMutex = LOCAL_LOCK.get(destination);
+                if (booleanMutex == null) {
+                    LOCAL_LOCK.put(destination, new BooleanMutex(true));
+                }
+            }
+        }
+    }
+
     @SuppressWarnings("resource")
     private synchronized void startListen(String destination, BooleanMutex mutex) {
         try {

client-adapter/launcher/src/main/java/com/alibaba/otter/canal/adapter/launcher/config/AdapterConfigHolder.java

@@ -0,0 +1,41 @@
+package com.alibaba.otter.canal.adapter.launcher.config;
+
+import com.alibaba.otter.canal.adapter.launcher.monitor.remote.ConfigItem;
+
+import java.util.Map;
+import java.util.concurrent.ConcurrentHashMap;
+
+/**
+ * @author L.J.R @ 2019-09-03
+ * @version 1.1.4
+ */
+public class AdapterConfigHolder {
+
+    private volatile long adapterConfigTimestamp = 0;
+
+    private final Map<String, ConfigItem> adapterConfigs = new ConcurrentHashMap<>();
+
+    private static AdapterConfigHolder adapterConfigHolder;
+
+    private AdapterConfigHolder() {
+    }
+
+    public synchronized static AdapterConfigHolder getInstance() {
+        if (adapterConfigHolder == null) {
+            adapterConfigHolder = new AdapterConfigHolder();
+        }
+        return adapterConfigHolder;
+    }
+
+    public void setAdapterConfigTimestamp(long configTimestamp) {
+        this.adapterConfigTimestamp = configTimestamp;
+    }
+
+    public long getAdapterConfigTimestamp() {
+        return adapterConfigTimestamp;
+    }
+
+    public Map<String, ConfigItem> getAdapterConfigs() {
+        return this.adapterConfigs;
+    }
+}

client-adapter/launcher/src/main/java/com/alibaba/otter/canal/adapter/launcher/loader/CanalAdapterLoader.java

@@ -7,6 +7,7 @@ import java.util.HashMap;
 import java.util.List;
 import java.util.Map;
 import java.util.Properties;
+import java.util.concurrent.CopyOnWriteArrayList;
 import java.util.concurrent.ExecutorService;
 import java.util.concurrent.Executors;
 import java.util.concurrent.TimeUnit;
@@ -63,10 +64,10 @@ public class CanalAdapterLoader {
         if ("tcp".equalsIgnoreCase(canalClientConfig.getMode())) {
             // 初始化canal-client的适配器
             for (CanalClientConfig.CanalAdapter canalAdapter : canalClientConfig.getCanalAdapters()) {
-                List<List<OuterAdapter>> canalOuterAdapterGroups = new ArrayList<>();
+                List<List<OuterAdapter>> canalOuterAdapterGroups = new CopyOnWriteArrayList<>();
 
                 for (CanalClientConfig.Group connectorGroup : canalAdapter.getGroups()) {
-                    List<OuterAdapter> canalOutConnectors = new ArrayList<>();
+                    List<OuterAdapter> canalOutConnectors = new CopyOnWriteArrayList<>();
                     for (OuterAdapterConfig c : connectorGroup.getOuterAdapters()) {
                         loadAdapter(c, canalOutConnectors);
                     }
@@ -94,8 +95,8 @@ public class CanalAdapterLoader {
             // 初始化canal-client-kafka的适配器
             for (CanalClientConfig.CanalAdapter canalAdapter : canalClientConfig.getCanalAdapters()) {
                 for (CanalClientConfig.Group group : canalAdapter.getGroups()) {
-                    List<List<OuterAdapter>> canalOuterAdapterGroups = new ArrayList<>();
-                    List<OuterAdapter> canalOuterAdapters = new ArrayList<>();
+                    List<List<OuterAdapter>> canalOuterAdapterGroups = new CopyOnWriteArrayList<>();
+                    List<OuterAdapter> canalOuterAdapters = new CopyOnWriteArrayList<>();
                     for (OuterAdapterConfig config : group.getOuterAdapters()) {
                         loadAdapter(config, canalOuterAdapters);
                     }
@@ -117,8 +118,8 @@ public class CanalAdapterLoader {
             // 初始化canal-client-rocketMQ的适配器
             for (CanalClientConfig.CanalAdapter canalAdapter : canalClientConfig.getCanalAdapters()) {
                 for (CanalClientConfig.Group group : canalAdapter.getGroups()) {
-                    List<List<OuterAdapter>> canalOuterAdapterGroups = new ArrayList<>();
-                    List<OuterAdapter> canalOuterAdapters = new ArrayList<>();
+                    List<List<OuterAdapter>> canalOuterAdapterGroups = new CopyOnWriteArrayList<>();
+                    List<OuterAdapter> canalOuterAdapters = new CopyOnWriteArrayList<>();
                     for (OuterAdapterConfig config : group.getOuterAdapters()) {
                         loadAdapter(config, canalOuterAdapters);
                     }
@@ -142,6 +143,29 @@ public class CanalAdapterLoader {
                         canalAdapter.getInstance() + "-" + group.getGroupId());
                 }
             }
+        } else if ("rabbitMQ".equalsIgnoreCase(canalClientConfig.getMode())) {
+            // 初始化canal-client-rabbitMQ的适配器
+            for (CanalClientConfig.CanalAdapter canalAdapter : canalClientConfig.getCanalAdapters()) {
+                for (CanalClientConfig.Group group : canalAdapter.getGroups()) {
+                    List<List<OuterAdapter>> canalOuterAdapterGroups = new CopyOnWriteArrayList<>();
+                    List<OuterAdapter> canalOuterAdapters = new CopyOnWriteArrayList<>();
+                    for (OuterAdapterConfig config : group.getOuterAdapters()) {
+                        loadAdapter(config, canalOuterAdapters);
+                    }
+                    canalOuterAdapterGroups.add(canalOuterAdapters);
+                    CanalAdapterRabbitMQWorker rabbitMQWork = new CanalAdapterRabbitMQWorker(canalClientConfig,
+                        canalOuterAdapterGroups,
+                        canalAdapter.getInstance(),
+                        group.getGroupId(),
+                        canalClientConfig.getFlatMessage());
+                    canalMQWorker.put(canalAdapter.getInstance() + "-rabbitmq-" + group.getGroupId(), rabbitMQWork);
+                    rabbitMQWork.start();
+
+                    logger.info("Start adapter for canal-client mq topic: {} succeed",
+                        canalAdapter.getInstance() + "-" + group.getGroupId());
+                }
+            }
+            // CanalAdapterRabbitMQWork
         }
     }
 
@@ -161,7 +185,7 @@ public class CanalAdapterLoader {
                     if (propertySource instanceof EnumerablePropertySource) {
                         String[] names = ((EnumerablePropertySource<?>) propertySource).getPropertyNames();
                         for (String name : names) {
-                            Object val = propertySource.getProperty(name);
+                            Object val = env.getProperty(name);
                             if (val != null) {
                                 evnProperties.put(name, val);
                             }

client-adapter/launcher/src/main/java/com/alibaba/otter/canal/adapter/launcher/loader/CanalAdapterRabbitMQWorker.java

@@ -0,0 +1,92 @@
+package com.alibaba.otter.canal.adapter.launcher.loader;
+
+import com.alibaba.otter.canal.client.adapter.OuterAdapter;
+import com.alibaba.otter.canal.client.adapter.support.CanalClientConfig;
+import com.alibaba.otter.canal.client.adapter.support.Util;
+import com.alibaba.otter.canal.client.rabbitmq.RabbitMQCanalConnector;
+import org.apache.kafka.common.errors.WakeupException;
+
+import java.util.List;
+import java.util.concurrent.ExecutorService;
+
+public class CanalAdapterRabbitMQWorker extends AbstractCanalAdapterWorker {
+
+    private RabbitMQCanalConnector connector;
+    private String                 topic;
+    private boolean                flatMessage;
+
+    public CanalAdapterRabbitMQWorker(CanalClientConfig canalClientConfig, List<List<OuterAdapter>> canalOuterAdapters,
+                                      String topic, String groupId, boolean flatMessage){
+        super(canalOuterAdapters);
+        this.canalClientConfig = canalClientConfig;
+        this.topic = topic;
+        this.flatMessage = flatMessage;
+        this.canalDestination = topic;
+        this.groupId = groupId;
+        this.connector = new RabbitMQCanalConnector(canalClientConfig.getMqServers(),
+            canalClientConfig.getVhost(),
+            topic,
+            canalClientConfig.getAccessKey(),
+            canalClientConfig.getSecretKey(),
+            canalClientConfig.getUsername(),
+            canalClientConfig.getPassword(),
+            canalClientConfig.getResourceOwnerId(),
+            flatMessage);
+    }
+
+    @Override
+    protected void process() {
+        while (!running) {
+            try {
+                Thread.sleep(1000);
+            } catch (InterruptedException ignored) {
+            }
+        }
+
+        ExecutorService workerExecutor = Util.newSingleThreadExecutor(5000L);
+        int retry = canalClientConfig.getRetries() == null
+                    || canalClientConfig.getRetries() == 0 ? 1 : canalClientConfig.getRetries();
+        long timeout = canalClientConfig.getTimeout() == null ? 30000 : canalClientConfig.getTimeout(); // 默认超时30秒
+        while (running) {
+            try {
+                syncSwitch.get(canalDestination);
+                logger.info("=============> Start to connect topic: {} <=============", this.topic);
+                connector.connect();
+                logger.info("=============> Start to subscribe topic: {}<=============", this.topic);
+                connector.subscribe();
+                logger.info("=============> Subscribe topic: {} succeed<=============", this.topic);
+                while (running) {
+                    boolean status = syncSwitch.status(canalDestination);
+                    if (!status) {
+                        connector.disconnect();
+                        break;
+                    }
+                    if (retry == -1) {
+                        retry = Integer.MAX_VALUE;
+                    }
+                    for (int i = 0; i < retry; i++) {
+                        if (!running) {
+                            break;
+                        }
+                        if (mqWriteOutData(retry, timeout, i, flatMessage, connector, workerExecutor)) {
+                            break;
+                        }
+                    }
+                }
+            } catch (Exception e) {
+                logger.error(e.getMessage(), e);
+            }
+        }
+
+        workerExecutor.shutdown();
+
+        try {
+            connector.unsubscribe();
+        } catch (WakeupException e) {
+            // No-op. Continue process
+        }
+        connector.disconnect();
+        logger.info("=============> Disconnect topic: {} <=============", this.topic);
+
+    }
+}

client-adapter/launcher/src/main/java/com/alibaba/otter/canal/adapter/launcher/loader/CanalAdapterService.java

@@ -53,6 +53,8 @@ public class CanalAdapterService {
             return;
         }
         try {
+            syncSwitch.refresh();
+            logger.info("## syncSwitch refreshed.");
             logger.info("## start the canal client adapters.");
             adapterLoader = new CanalAdapterLoader(adapterCanalConfig);
             adapterLoader.init();

client-adapter/launcher/src/main/java/com/alibaba/otter/canal/adapter/launcher/monitor/remote/DbRemoteConfigLoader.java

@@ -20,10 +20,10 @@ import org.slf4j.Logger;
 import org.slf4j.LoggerFactory;
 
 import com.alibaba.druid.pool.DruidDataSource;
+import com.alibaba.otter.canal.adapter.launcher.config.AdapterConfigHolder;
 import com.alibaba.otter.canal.common.utils.CommonUtils;
 import com.alibaba.otter.canal.common.utils.NamedThreadFactory;
 import com.google.common.base.Joiner;
-import com.google.common.collect.MapMaker;
 
 /**
  * 基于数据库的远程配置装载器
@@ -33,17 +33,16 @@ import com.google.common.collect.MapMaker;
  */
 public class DbRemoteConfigLoader implements RemoteConfigLoader {
 
-    private static final Logger      logger                 = LoggerFactory.getLogger(DbRemoteConfigLoader.class);
+    private static final Logger      logger                    = LoggerFactory.getLogger(DbRemoteConfigLoader.class);
 
     private DruidDataSource          dataSource;
 
-    private volatile long            currentConfigTimestamp = 0;
-    private Map<String, ConfigItem>  remoteAdapterConfigs   = new MapMaker().makeMap();
+    private AdapterConfigHolder      remoteAdapterConfigHolder = AdapterConfigHolder.getInstance();
 
-    private ScheduledExecutorService executor               = Executors.newScheduledThreadPool(2,
+    private ScheduledExecutorService executor                  = Executors.newScheduledThreadPool(2,
         new NamedThreadFactory("remote-adapter-config-scan"));
 
-    private RemoteAdapterMonitor     remoteAdapterMonitor   = new RemoteAdapterMonitorImpl();
+    private RemoteAdapterMonitor     remoteAdapterMonitor      = new RemoteAdapterMonitorImpl();
 
     public DbRemoteConfigLoader(String driverName, String jdbcUrl, String jdbcUsername, String jdbcPassword){
         dataSource = new DruidDataSource();
@@ -76,8 +75,8 @@ public class DbRemoteConfigLoader implements RemoteConfigLoader {
             // 加载远程adapter配置
             ConfigItem configItem = getRemoteAdapterConfig();
             if (configItem != null) {
-                if (configItem.getModifiedTime() != currentConfigTimestamp) {
-                    currentConfigTimestamp = configItem.getModifiedTime();
+                if (configItem.getModifiedTime() != remoteAdapterConfigHolder.getAdapterConfigTimestamp()) {
+                    remoteAdapterConfigHolder.setAdapterConfigTimestamp(configItem.getModifiedTime());
                     overrideLocalCanalConfig(configItem.getContent());
                     logger.info("## Loaded remote adapter config: application.yml");
                 }
@@ -166,7 +165,7 @@ public class DbRemoteConfigLoader implements RemoteConfigLoader {
             List<Long> changedIds = new ArrayList<>();
 
             for (ConfigItem remoteConfigStat : remoteConfigStatus.values()) {
-                ConfigItem currentConfig = remoteAdapterConfigs
+                ConfigItem currentConfig = remoteAdapterConfigHolder.getAdapterConfigs()
                     .get(remoteConfigStat.getCategory() + "/" + remoteConfigStat.getName());
                 if (currentConfig == null) {
                     // 新增
@@ -192,8 +191,8 @@ public class DbRemoteConfigLoader implements RemoteConfigLoader {
                         configItemNew.setContent(rs.getString("content"));
                         configItemNew.setModifiedTime(rs.getTimestamp("modified_time").getTime());
 
-                        remoteAdapterConfigs.put(configItemNew.getCategory() + "/" + configItemNew.getName(),
-                            configItemNew);
+                        remoteAdapterConfigHolder.getAdapterConfigs()
+                            .put(configItemNew.getCategory() + "/" + configItemNew.getName(), configItemNew);
                         remoteAdapterMonitor.onModify(configItemNew);
                     }
 
@@ -203,10 +202,11 @@ public class DbRemoteConfigLoader implements RemoteConfigLoader {
             }
         }
 
-        for (ConfigItem configItem : remoteAdapterConfigs.values()) {
+        for (ConfigItem configItem : remoteAdapterConfigHolder.getAdapterConfigs().values()) {
             if (!remoteConfigStatus.containsKey(configItem.getCategory() + "/" + configItem.getName())) {
                 // 删除
-                remoteAdapterConfigs.remove(configItem.getCategory() + "/" + configItem.getName());
+                remoteAdapterConfigHolder.getAdapterConfigs()
+                    .remove(configItem.getCategory() + "/" + configItem.getName());
                 remoteAdapterMonitor.onDelete(configItem.getCategory() + "/" + configItem.getName());
             }
         }

client-adapter/launcher/src/main/resources/application.yml

@@ -18,6 +18,9 @@ canal.conf:
   timeout:
   accessKey:
   secretKey:
+  username:
+  password:
+  vhost:
 #  srcDataSources:
 #    defaultDS:
 #      url: jdbc:mysql://127.0.0.1:3306/mytest?useUnicode=true

client-adapter/logger/pom.xml

@@ -3,7 +3,8 @@
     <parent>
         <artifactId>canal.client-adapter</artifactId>
         <groupId>com.alibaba.otter</groupId>
-        <version>1.1.4</version>
+        <version>1.1.5-SNAPSHOT</version>
+        <relativePath>../pom.xml</relativePath>
     </parent>
     <modelVersion>4.0.0</modelVersion>
     <groupId>com.alibaba.otter</groupId>

client-adapter/pom.xml

@@ -1,9 +1,15 @@
 <?xml version="1.0" encoding="UTF-8"?>
 <project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
     <modelVersion>4.0.0</modelVersion>
+    <parent>
+        <groupId>com.alibaba.otter</groupId>
+        <artifactId>canal</artifactId>
+        <version>1.1.5-SNAPSHOT</version>
+        <relativePath>../pom.xml</relativePath>
+    </parent>
     <groupId>com.alibaba.otter</groupId>
     <artifactId>canal.client-adapter</artifactId>
-    <version>1.1.4</version>
+    <version>1.1.5-SNAPSHOT</version>
     <packaging>pom</packaging>
     <name>canal client adapter module for otter ${project.version}</name>
 
@@ -20,9 +26,11 @@
         <module>common</module>
         <module>logger</module>
         <module>hbase</module>
-        <module>elasticsearch</module>
         <module>launcher</module>
         <module>rdb</module>
+        <module>es6x</module>
+        <module>es7x</module>
+        <module>escore</module>
     </modules>
 
     <licenses>
@@ -93,6 +101,162 @@
         </repository>
     </repositories>
 
+    <dependencyManagement>
+        <dependencies>
+            <dependency>
+                <groupId>org.springframework.cloud</groupId>
+                <artifactId>spring-cloud-context</artifactId>
+                <version>2.0.0.RELEASE</version>
+            </dependency>
+            <dependency>
+                <groupId>org.springframework</groupId>
+                <artifactId>spring-context</artifactId>
+                <version>5.0.5.RELEASE</version>
+            </dependency>
+            <dependency>
+                <groupId>org.springframework</groupId>
+                <artifactId>spring-core</artifactId>
+                <version>5.0.5.RELEASE</version>
+            </dependency>
+            <dependency>
+                <groupId>org.springframework</groupId>
+                <artifactId>spring-aop</artifactId>
+                <version>5.0.5.RELEASE</version>
+            </dependency>
+            <dependency>
+                <groupId>org.springframework</groupId>
+                <artifactId>spring-context</artifactId>
+                <version>5.0.5.RELEASE</version>
+            </dependency>
+            <dependency>
+                <groupId>org.springframework</groupId>
+                <artifactId>spring-jdbc</artifactId>
+                <version>5.0.5.RELEASE</version>
+            </dependency>
+            <dependency>
+                <groupId>org.springframework</groupId>
+                <artifactId>spring-orm</artifactId>
+                <version>5.0.5.RELEASE</version>
+            </dependency>
+            <dependency>
+                <groupId>org.springframework</groupId>
+                <artifactId>spring-test</artifactId>
+                <version>5.0.5.RELEASE</version>
+                <scope>test</scope>
+            </dependency>
+            <dependency>
+                <groupId>com.h2database</groupId>
+                <artifactId>h2</artifactId>
+                <version>1.4.192</version>
+            </dependency>
+            <!-- 单独引入rocketmq依赖 -->
+            <dependency>
+                <groupId>org.apache.rocketmq</groupId>
+                <artifactId>rocketmq-client</artifactId>
+            <version>4.5.1</version>
+            </dependency>
+            <!-- 单独引入kafka依赖 -->
+            <dependency>
+                <groupId>org.apache.kafka</groupId>
+                <artifactId>kafka-clients</artifactId>
+                <version>1.1.1</version>
+            </dependency>
+
+            <dependency>
+                <groupId>com.rabbitmq</groupId>
+                <artifactId>amqp-client</artifactId>
+                <version>5.5.0</version>
+            </dependency>
+            <dependency>
+                <groupId>com.alibaba.mq-amqp</groupId>
+                <artifactId>mq-amqp-client</artifactId>
+                <version>1.0.3</version>
+            </dependency>
+            <!-- jdbc -->
+            <dependency>
+                <groupId>mysql</groupId>
+                <artifactId>mysql-connector-java</artifactId>
+                <version>5.1.47</version>
+            </dependency>
+            <dependency>
+                <groupId>org.postgresql</groupId>
+                <artifactId>postgresql</artifactId>
+                <version>42.1.4</version>
+            </dependency>
+            <dependency>
+                <groupId>com.oracle</groupId>
+                <artifactId>ojdbc6</artifactId>
+                <version>11.2.0.3</version>
+            </dependency>
+            <dependency>
+                <groupId>com.microsoft.sqlserver</groupId>
+                <artifactId>mssql-jdbc</artifactId>
+                <version>7.0.0.jre8</version>
+            </dependency>
+
+            <dependency>
+                <groupId>org.apache.hbase</groupId>
+                <artifactId>hbase-client</artifactId>
+                <version>1.1.2</version>
+                <exclusions>
+                    <exclusion>
+                        <groupId>org.slf4j</groupId>
+                        <artifactId>slf4j-log4j12</artifactId>
+                    </exclusion>
+                    <exclusion>
+                        <groupId>jdk.tools</groupId>
+                        <artifactId>jdk.tools</artifactId>
+                    </exclusion>
+                </exclusions>
+            </dependency>
+            <dependency>
+                <groupId>org.apache.curator</groupId>
+                <artifactId>curator-recipes</artifactId>
+                <version>2.10.0</version>
+            </dependency>
+            <dependency>
+                <groupId>joda-time</groupId>
+                <artifactId>joda-time</artifactId>
+                <version>2.9.4</version>
+            </dependency>
+            <dependency>
+                <groupId>org.yaml</groupId>
+                <artifactId>snakeyaml</artifactId>
+                <version>1.19</version>
+            </dependency>
+            <dependency>
+                <groupId>org.powermock</groupId>
+                <artifactId>powermock-api-mockito</artifactId>
+                <version>1.6.5</version>
+                <scope>test</scope>
+            </dependency>
+            <dependency>
+                <groupId>org.powermock</groupId>
+                <artifactId>powermock-module-junit4</artifactId>
+                <version>1.6.5</version>
+                <scope>test</scope>
+            </dependency>
+            <dependency>
+                <groupId>org.mockito</groupId>
+                <artifactId>mockito-all</artifactId>
+                <version>1.10.19</version>
+                <scope>test</scope>
+            </dependency>
+            <dependency>
+                <groupId>junit</groupId>
+                <artifactId>junit</artifactId>
+                <version>4.8.2</version>
+                <scope>test</scope>
+            </dependency>
+            <dependency>
+                <groupId>com.diffblue</groupId>
+                <artifactId>deeptestutils</artifactId>
+                <version>1.9.0</version>
+                <scope>test</scope>
+            </dependency>
+        </dependencies>
+    </dependencyManagement>
+
     <build>
         <plugins>
             <plugin>
@@ -120,4 +284,4 @@
             <url>https://oss.sonatype.org/service/local/staging/deploy/maven2/</url>
         </repository>
     </distributionManagement>
-</project>
+</project>

client-adapter/rdb/pom.xml

@@ -3,7 +3,8 @@
     <parent>
         <artifactId>canal.client-adapter</artifactId>
         <groupId>com.alibaba.otter</groupId>
-        <version>1.1.4</version>
+        <version>1.1.5-SNAPSHOT</version>
+        <relativePath>../pom.xml</relativePath>
     </parent>
     <modelVersion>4.0.0</modelVersion>
     <groupId>com.alibaba.otter</groupId>
@@ -21,25 +22,21 @@
         <dependency>
             <groupId>mysql</groupId>
             <artifactId>mysql-connector-java</artifactId>
-            <version>5.1.47</version>
             <scope>test</scope>
         </dependency>
         <dependency>
             <groupId>org.postgresql</groupId>
             <artifactId>postgresql</artifactId>
-            <version>42.1.4</version>
             <scope>test</scope>
         </dependency>
         <dependency>
             <groupId>com.oracle</groupId>
             <artifactId>ojdbc6</artifactId>
-            <version>11.2.0.3</version>
             <scope>test</scope>
         </dependency>
         <dependency>
             <groupId>com.microsoft.sqlserver</groupId>
             <artifactId>mssql-jdbc</artifactId>
-            <version>7.0.0.jre8</version>
             <scope>test</scope>
         </dependency>
         <dependency>

client/pom.xml

@@ -3,7 +3,7 @@
 	<parent>
 		<groupId>com.alibaba.otter</groupId>
 		<artifactId>canal</artifactId>
-		<version>1.1.4</version>
+		<version>1.1.5-SNAPSHOT</version>
 		<relativePath>../pom.xml</relativePath>
 	</parent>
 	<groupId>com.alibaba.otter</groupId>
@@ -109,6 +109,14 @@
 			<groupId>org.apache.rocketmq</groupId>
 			<artifactId>rocketmq-acl</artifactId>
 		</dependency>
+		<dependency>
+			<groupId>com.rabbitmq</groupId>
+			<artifactId>amqp-client</artifactId>
+		</dependency>
+		<dependency>
+			<groupId>com.alibaba.mq-amqp</groupId>
+			<artifactId>mq-amqp-client</artifactId>
+		</dependency>
 		<!-- 客户端要使用请单独引入kafka-clients依赖 -->
 		<dependency>
 			<groupId>org.apache.kafka</groupId>

client/src/main/java/com/alibaba/otter/canal/client/CanalConnector.java

@@ -1,5 +1,6 @@
 package com.alibaba.otter.canal.client;
 
+import java.io.IOException;
 import java.util.concurrent.TimeUnit;
 
 import com.alibaba.otter.canal.protocol.Message;

client/src/main/java/com/alibaba/otter/canal/client/ConsumerBatchMessage.java

@@ -1,4 +1,4 @@
-package com.alibaba.otter.canal.client.rocketmq;
+package com.alibaba.otter.canal.client;
 
 import java.util.List;
 import java.util.concurrent.CountDownLatch;

client/src/main/java/com/alibaba/otter/canal/client/rabbitmq/AliyunCredentialsProvider.java

@@ -0,0 +1,54 @@
+/**
+ * aliyun amqp协议 账号类
+ * 暂不支持STS授权情况
+ */
+package com.alibaba.otter.canal.client.rabbitmq;
+
+import com.alibaba.mq.amqp.utils.UserUtils;
+import com.rabbitmq.client.impl.CredentialsProvider;
+import java.security.InvalidKeyException;
+import java.security.NoSuchAlgorithmException;
+
+
+public class AliyunCredentialsProvider implements CredentialsProvider {
+
+
+    /**
+     * Access Key ID
+     */
+    private final String AliyunAccessKey;
+
+    /**
+     * Access Key Secret
+     */
+    private final String AliyunAccessSecret;
+
+    /**
+     * 资源主账号ID
+     */
+    private final long resourceOwnerId;
+
+
+    public AliyunCredentialsProvider(final String accessKey, final String accessSecret, final long resourceOwnerId) {
+        this.AliyunAccessKey = accessKey;
+        this.AliyunAccessSecret = accessSecret;
+        this.resourceOwnerId = resourceOwnerId;
+    }
+
+
+    @Override
+    public String getUsername() {
+        return UserUtils.getUserName(AliyunAccessKey, resourceOwnerId);
+    }
+
+    @Override
+    public String getPassword() {
+        try {
+            return UserUtils.getPassord(AliyunAccessSecret);
+        } catch (InvalidKeyException | NoSuchAlgorithmException ignored) {
+        }
+        return null;
+    }
+
+
+}

client/src/main/java/com/alibaba/otter/canal/client/rabbitmq/RabbitMQCanalConnector.java

@@ -0,0 +1,299 @@
+package com.alibaba.otter.canal.client.rabbitmq;
+
+import com.alibaba.fastjson.JSON;
+import com.alibaba.otter.canal.client.CanalMQConnector;
+import com.alibaba.otter.canal.client.CanalMessageDeserializer;
+import com.alibaba.otter.canal.client.ConsumerBatchMessage;
+import com.alibaba.otter.canal.protocol.FlatMessage;
+import com.alibaba.otter.canal.protocol.Message;
+import com.alibaba.otter.canal.protocol.exception.CanalClientException;
+import com.google.common.collect.Lists;
+import com.rabbitmq.client.*;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.io.IOException;
+import java.util.List;
+import java.util.concurrent.BlockingQueue;
+import java.util.concurrent.LinkedBlockingQueue;
+import java.util.concurrent.TimeUnit;
+import java.util.concurrent.TimeoutException;
+
+public class RabbitMQCanalConnector implements CanalMQConnector {
+
+    private static final Logger                 logger              = LoggerFactory
+        .getLogger(RabbitMQCanalConnector.class);
+
+    // 链接地址
+    private String                              nameServer;
+
+    // 主机名
+    private String                              vhost;
+
+    private String                              queueName;
+
+    // 一些鉴权信息
+    private String                              accessKey;
+    private String                              secretKey;
+    private Long                                resourceOwnerId;
+    private String                              username;
+    private String                              password;
+
+    private boolean                             flatMessage;
+
+    private Connection                          connect;
+    private Channel                             channel;
+
+
+    private long                                batchProcessTimeout = 60 * 1000;
+    private BlockingQueue<ConsumerBatchMessage> messageBlockingQueue;
+    private volatile ConsumerBatchMessage       lastGetBatchMessage = null;
+
+    public RabbitMQCanalConnector(String nameServer, String vhost, String queueName, String accessKey, String secretKey,
+                                  String username, String password, Long resourceOwnerId, boolean flatMessage){
+        this.nameServer = nameServer;
+        this.vhost = vhost;
+        this.queueName = queueName;
+        this.accessKey = accessKey;
+        this.secretKey = secretKey;
+        this.username = username;
+        this.password = password;
+        this.resourceOwnerId = resourceOwnerId;
+        this.flatMessage = flatMessage;
+        this.messageBlockingQueue = new LinkedBlockingQueue<>(1024);
+    }
+
+    public void connect() throws CanalClientException {
+        ConnectionFactory factory = new ConnectionFactory();
+        if (accessKey.length() > 0 && secretKey.length() > 0) {
+            factory.setCredentialsProvider(new AliyunCredentialsProvider(accessKey, secretKey, resourceOwnerId));
+        } else {
+            factory.setUsername(username);
+            factory.setPassword(password);
+        }
+        factory.setHost(nameServer);
+        factory.setAutomaticRecoveryEnabled(true);
+        factory.setNetworkRecoveryInterval(5000);
+        factory.setVirtualHost(vhost);
+        try {
+            connect = factory.newConnection();
+            channel = connect.createChannel();
+        } catch (IOException | TimeoutException e) {
+            throw new CanalClientException("Start RabbitMQ producer error", e);
+        }
+    }
+
+    @Override
+    public void disconnect() throws CanalClientException {
+        if (connect != null) {
+            try {
+                connect.close();
+            } catch (IOException e) {
+                throw new CanalClientException("stop connect error", e);
+            }
+        }
+        if (channel != null) {
+            try {
+                channel.close();
+            } catch (IOException | TimeoutException e) {
+                throw new CanalClientException("stop channel error", e);
+            }
+        }
+    }
+
+    @Override
+    public boolean checkValid() throws CanalClientException {
+        return true; // 永远true
+    }
+
+    /**
+     * RabbitMQ支持拉取 不需要订阅
+     *
+     * @param filter
+     * @throws CanalClientException
+     */
+    @Override
+    public void subscribe(String filter) throws CanalClientException {
+        // 不存在连接 则重新连接
+        if (connect == null) {
+            this.connect();
+        }
+
+        Consumer consumer = new DefaultConsumer(channel) {
+
+            @Override
+            public void handleDelivery(String consumerTag, Envelope envelope, AMQP.BasicProperties properties,
+                                       byte[] body) throws IOException {
+
+                if (body != null) {
+                    channel.basicAck(envelope.getDeliveryTag(), process(body));
+                }
+            }
+        };
+        try {
+            channel.basicConsume(queueName, false, consumer);
+        } catch (IOException e) {
+            throw new CanalClientException("error", e);
+        }
+    }
+
+    @Override
+    public void subscribe() throws CanalClientException {
+        this.subscribe(null);
+    }
+
+    @Override
+    public void unsubscribe() throws CanalClientException {
+        // 取消订阅 直接强行断开吧
+        this.disconnect();
+    }
+
+    @Override
+    public Message get(int batchSize) throws CanalClientException {
+        return null;
+    }
+
+    @Override
+    public Message get(int batchSize, Long timeout, TimeUnit unit) throws CanalClientException {
+        return null;
+    }
+
+    @Override
+    public Message getWithoutAck(int batchSize) throws CanalClientException {
+        throw new CanalClientException("mq not support this method");
+    }
+
+    @Override
+    public Message getWithoutAck(int batchSize, Long timeout, TimeUnit unit) throws CanalClientException {
+        throw new CanalClientException("mq not support this method");
+    }
+
+    @Override
+    public void ack(long batchId) throws CanalClientException {
+        throw new CanalClientException("mq not support this method");
+    }
+
+    @Override
+    public void rollback(long batchId) throws CanalClientException {
+        throw new CanalClientException("mq not support this method");
+
+    }
+
+    @Override
+    public List<Message> getList(Long timeout, TimeUnit unit) throws CanalClientException {
+        List<Message> messages = getListWithoutAck(timeout, unit);
+        if (messages != null && !messages.isEmpty()) {
+            ack();
+        }
+        return messages;
+    }
+
+    @Override
+    public List<Message> getListWithoutAck(Long timeout, TimeUnit unit) throws CanalClientException {
+        try {
+            if (this.lastGetBatchMessage != null) {
+                throw new CanalClientException("mq get/ack not support concurrent & async ack");
+            }
+
+            ConsumerBatchMessage batchMessage = messageBlockingQueue.poll(timeout, unit);
+            if (batchMessage != null) {
+                this.lastGetBatchMessage = batchMessage;
+                return batchMessage.getData();
+            }
+        } catch (InterruptedException ex) {
+            logger.warn("Get message timeout", ex);
+            throw new CanalClientException("Failed to fetch the data after: " + timeout);
+        }
+        return Lists.newArrayList();
+    }
+
+    @Override
+    public List<FlatMessage> getFlatList(Long timeout, TimeUnit unit) throws CanalClientException {
+        List<FlatMessage> messages = getFlatListWithoutAck(timeout, unit);
+        if (messages != null && !messages.isEmpty()) {
+            ack();
+        }
+        return messages;
+    }
+
+    @Override
+    public List<FlatMessage> getFlatListWithoutAck(Long timeout, TimeUnit unit) throws CanalClientException {
+        try {
+            if (this.lastGetBatchMessage != null) {
+                throw new CanalClientException("mq get/ack not support concurrent & async ack");
+            }
+
+            ConsumerBatchMessage batchMessage = messageBlockingQueue.poll(timeout, unit);
+            if (batchMessage != null) {
+                this.lastGetBatchMessage = batchMessage;
+                return batchMessage.getData();
+            }
+        } catch (InterruptedException ex) {
+            logger.warn("Get message timeout", ex);
+            throw new CanalClientException("Failed to fetch the data after: " + timeout);
+        }
+        return Lists.newArrayList();
+    }
+
+    private boolean process(byte[] messageData) {
+        if (logger.isDebugEnabled()) {
+            logger.debug("Get Message: {}", new String(messageData));
+        }
+        List messageList = Lists.newArrayList();
+        if (!flatMessage) {
+            Message message = CanalMessageDeserializer.deserializer(messageData);
+            messageList.add(message);
+        } else {
+            FlatMessage flatMessage = JSON.parseObject(messageData, FlatMessage.class);
+            messageList.add(flatMessage);
+        }
+        ConsumerBatchMessage batchMessage;
+        if (!flatMessage) {
+            batchMessage = new ConsumerBatchMessage<Message>(messageList);
+        } else {
+            batchMessage = new ConsumerBatchMessage<FlatMessage>(messageList);
+        }
+        try {
+            messageBlockingQueue.put(batchMessage);
+        } catch (InterruptedException e) {
+            logger.error("Put message to queue error", e);
+            throw new RuntimeException(e);
+        }
+        boolean isCompleted;
+        try {
+            isCompleted = batchMessage.waitFinish(batchProcessTimeout);
+        } catch (InterruptedException e) {
+            logger.error("Interrupted when waiting messages to be finished.", e);
+            throw new RuntimeException(e);
+        }
+        boolean isSuccess = batchMessage.isSuccess();
+        return isCompleted && isSuccess;
+    }
+
+    @Override
+    public void ack() throws CanalClientException {
+        try {
+            if (this.lastGetBatchMessage != null) {
+                this.lastGetBatchMessage.ack();
+            }
+        } catch (Throwable e) {
+            if (this.lastGetBatchMessage != null) {
+                this.lastGetBatchMessage.fail();
+            }
+        } finally {
+            this.lastGetBatchMessage = null;
+        }
+    }
+
+    @Override
+    public void rollback() throws CanalClientException {
+        try {
+            if (this.lastGetBatchMessage != null) {
+                this.lastGetBatchMessage.fail();
+            }
+        } finally {
+            this.lastGetBatchMessage = null;
+        }
+    }
+
+}

client/src/main/java/com/alibaba/otter/canal/client/rocketmq/RocketMQCanalConnector.java

@@ -5,6 +5,7 @@ import java.util.concurrent.BlockingQueue;
 import java.util.concurrent.LinkedBlockingQueue;
 import java.util.concurrent.TimeUnit;
 
+import com.alibaba.otter.canal.client.ConsumerBatchMessage;
 import org.apache.commons.lang.StringUtils;
 import org.apache.rocketmq.acl.common.AclClientRPCHook;
 import org.apache.rocketmq.acl.common.SessionCredentials;

common/pom.xml

@@ -3,7 +3,7 @@
 	<parent>
 		<groupId>com.alibaba.otter</groupId>
 		<artifactId>canal</artifactId>
-		<version>1.1.4</version>
+		<version>1.1.5-SNAPSHOT</version>
 		<relativePath>../pom.xml</relativePath>
 	</parent>
 	<artifactId>canal.common</artifactId>

common/src/main/java/com/alibaba/otter/canal/common/utils/ExecutorTemplate.java

@@ -0,0 +1,100 @@
+package com.alibaba.otter.canal.common.utils;
+
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.List;
+import java.util.concurrent.Callable;
+import java.util.concurrent.Future;
+import java.util.concurrent.ThreadPoolExecutor;
+
+/**
+ * 多线程执行器模板代码，otter中好多地方都写多线程，比较多的都是重复的逻辑代码，抽象一下做个模板把
+ *
+ * <pre>
+ * 示例代码：
+ * ExecutorTemplate template = new ExecutorTemplate(executor);
+ * ...
+ * try {
+ *    for ( ....) {
+ *       template.submit(new Runnable() {})
+ *    }
+ * 
+ *    List<?> result = template.waitForResult();
+ *    // do result
+ * } finally {
+ *    template.clear();
+ * }
+ * 
+ * 注意：该模板工程，不支持多业务并发调用，会出现数据混乱
+ * </pre>
+ */
+public class ExecutorTemplate {
+
+    private volatile ThreadPoolExecutor executor = null;
+    private volatile List<Future>       futures  = null;
+
+    public ExecutorTemplate(ThreadPoolExecutor executor){
+        this.futures = Collections.synchronizedList(new ArrayList<Future>());
+        this.executor = executor;
+    }
+
+    public void submit(Runnable task) {
+        Future future = executor.submit(task, null);
+        futures.add(future);
+        check(future);
+    }
+
+    public void submit(Callable<Exception> task) {
+        Future future = executor.submit(task);
+        futures.add(future);
+        check(future);
+    }
+
+    private void check(Future future) {
+        if (future.isDone()) {
+            // 立即判断一次，因为使用了CallerRun可能当场跑出结果，针对有异常时快速响应，而不是等跑完所有的才抛异常
+            try {
+                future.get();
+            } catch (Throwable e) {
+                // 取消完之后立马退出
+                cacelAllFutures();
+                throw new RuntimeException(e);
+            }
+        }
+    }
+
+    public synchronized List<?> waitForResult() {
+        List result = new ArrayList();
+        RuntimeException exception = null;
+
+        for (Future future : futures) {
+            try {
+                result.add(future.get());
+            } catch (Throwable e) {
+                exception = new RuntimeException(e);
+                // 如何一个future出现了异常，就退出
+                break;
+            }
+        }
+
+        if (exception != null) {
+            cacelAllFutures();
+            throw exception;
+        } else {
+            return result;
+        }
+    }
+
+    public void cacelAllFutures() {
+        for (Future future : futures) {
+            if (!future.isDone() && !future.isCancelled()) {
+                future.cancel(true);
+            }
+        }
+    }
+
+    public void clear() {
+        futures.clear();
+    }
+
+}

common/src/main/java/com/alibaba/otter/canal/common/utils/FileUtils.java

@@ -28,10 +28,13 @@ public class FileUtils {
                 minIndex = 0;
             }
 
-            String line;
+            if (readIndex < 0) {
+                readIndex = 0;
+            }
             rf.seek(readIndex);
             int k = 0;
             int c = -1;
+            String line = null;
             while (readIndex > start) {
                 if (k == l) {
                     break;

dbsync/pom.xml

@@ -3,7 +3,7 @@
 	<parent>
 		<groupId>com.alibaba.otter</groupId>
 		<artifactId>canal</artifactId>
-		<version>1.1.4</version>
+		<version>1.1.5-SNAPSHOT</version>
 		<relativePath>../pom.xml</relativePath>
 	</parent>
 	<groupId>com.alibaba.otter</groupId>
@@ -44,5 +44,25 @@
 			<artifactId>mysql-connector-java</artifactId>
 			<scope>test</scope>
 		</dependency>
+        <dependency>
+            <groupId>org.powermock</groupId>
+            <artifactId>powermock-api-mockito</artifactId>
+            <scope>test</scope>
+        </dependency>
+        <dependency>
+            <groupId>org.powermock</groupId>
+            <artifactId>powermock-module-junit4</artifactId>
+            <scope>test</scope>
+        </dependency>
+        <dependency>
+            <groupId>org.mockito</groupId>
+            <artifactId>mockito-all</artifactId>
+            <scope>test</scope>
+        </dependency>
+        <dependency>
+            <groupId>com.diffblue</groupId>
+            <artifactId>deeptestutils</artifactId>
+            <scope>test</scope>
+        </dependency>
 	</dependencies>
 </project>

dbsync/src/main/java/com/taobao/tddl/dbsync/binlog/event/RowsLogBuffer.java

@@ -756,7 +756,7 @@ public final class RowsLogBuffer {
                     }
 
                     int d = (int) ((intpart >> 12) % (1 << 10));
-                    if (d > 100) {
+                    if (d >= 100) {
                         builder.append(String.valueOf(d));
                     } else {
                         appendNumber2(builder, d);

dbsync/src/test/java/com/taobao/tddl/dbsync/binlog/JsonConversion_Json_ValueTest.java

@@ -0,0 +1,54 @@
+package com.taobao.tddl.dbsync.binlog;
+
+import com.taobao.tddl.dbsync.binlog.JsonConversion.Json_Value;
+import com.taobao.tddl.dbsync.binlog.JsonConversion.Json_enum_type;
+import com.taobao.tddl.dbsync.binlog.JsonConversion;
+import com.taobao.tddl.dbsync.binlog.LogBuffer;
+import org.junit.Assert;
+import org.junit.Rule;
+import org.junit.Test;
+import org.junit.rules.ExpectedException;
+import org.junit.rules.Timeout;
+
+import java.lang.reflect.Method;
+
+public class JsonConversion_Json_ValueTest {
+
+  @Rule public final ExpectedException thrown = ExpectedException.none();
+
+  @Rule public final Timeout globalTimeout = new Timeout(10000);
+
+  /* testedClasses: JsonConversion_Json_Value */
+  // Test written by Diffblue Cover.
+
+  @Test
+  public void constructorInputNullNotNullOutputVoid() {
+
+    // Arrange
+    final Json_enum_type t = null;
+    final String value = ",";
+
+    // Act, creating object to test constructor
+    final Json_Value objectUnderTest = new Json_Value(t, value);
+
+    // Assert side effects
+    Assert.assertEquals(",", objectUnderTest.m_string_value);
+  }
+
+  // Test written by Diffblue Cover.
+  @Test
+  public void parse_valueInputZeroNotNullPositiveNotNullOutputIllegalArgumentException() {
+
+    // Arrange
+    final int type = 0;
+    final LogBuffer buffer = new LogBuffer();
+    final long len = 4_294_967_297L;
+    final String charsetName = "3";
+
+    // Act
+    thrown.expect(IllegalArgumentException.class);
+    JsonConversion.parse_value(type, buffer, len, charsetName);
+
+    // Method is not expected to return due to exception thrown
+  }
+}

dbsync/src/test/java/com/taobao/tddl/dbsync/binlog/JsonDiffConversionTest.java

@@ -0,0 +1,214 @@
+package com.taobao.tddl.dbsync.binlog;
+
+import com.taobao.tddl.dbsync.binlog.JsonDiffConversion;
+import com.taobao.tddl.dbsync.binlog.LogBuffer;
+import org.junit.Assert;
+import org.junit.Rule;
+import org.junit.Test;
+import org.junit.rules.ExpectedException;
+import org.junit.rules.Timeout;
+
+import java.lang.reflect.Array;
+import java.lang.reflect.InvocationTargetException;
+
+public class JsonDiffConversionTest {
+
+  @Rule public final ExpectedException thrown = ExpectedException.none();
+
+  @Rule public final Timeout globalTimeout = new Timeout(10000);
+
+  /* testedClasses: JsonDiffConversion */
+  // Test written by Diffblue Cover.
+  @Test
+  public void print_json_diffInputNotNullPositiveNotNullZeroNotNullOutputIllegalArgumentException()
+      throws InvocationTargetException {
+
+    // Arrange
+    final LogBuffer buffer = new LogBuffer();
+    buffer.position = 28;
+    buffer.semival = 0;
+    final byte[] myByteArray = {(byte)3,  (byte)3,  (byte)67, (byte)67, (byte)67, (byte)67,
+                                (byte)67, (byte)66, (byte)67, (byte)66, (byte)67, (byte)66,
+                                (byte)67, (byte)67, (byte)67, (byte)66, (byte)67, (byte)66,
+                                (byte)67, (byte)66, (byte)67, (byte)67, (byte)67, (byte)66,
+                                (byte)67, (byte)67, (byte)67, (byte)66, (byte)2,  (byte)66};
+    buffer.buffer = myByteArray;
+    buffer.limit = -1_000_000_065;
+    buffer.origin = 1_000_000_096;
+    final long len = 71L;
+    final String columnName = "foo";
+    final int columnIndex = 0;
+    final String charsetName = "foo";
+    try {
+
+      // Act
+      thrown.expect(IllegalArgumentException.class);
+      JsonDiffConversion.print_json_diff(buffer, len, columnName, columnIndex, charsetName);
+    } catch (IllegalArgumentException ex) {
+
+      // Assert side effects
+      Assert.assertNotNull(buffer);
+      Assert.assertEquals(30, buffer.position);
+      Assert.assertEquals(0, buffer.semival);
+      Assert.assertArrayEquals(new byte[] {(byte)3,  (byte)3,  (byte)67, (byte)67, (byte)67,
+                                           (byte)67, (byte)67, (byte)66, (byte)67, (byte)66,
+                                           (byte)67, (byte)66, (byte)67, (byte)67, (byte)67,
+                                           (byte)66, (byte)67, (byte)66, (byte)67, (byte)66,
+                                           (byte)67, (byte)67, (byte)67, (byte)66, (byte)67,
+                                           (byte)67, (byte)67, (byte)66, (byte)2,  (byte)66},
+                               buffer.buffer);
+      Assert.assertEquals(-1_000_000_065, buffer.limit);
+      Assert.assertEquals(1_000_000_096, buffer.origin);
+      throw ex;
+    }
+  }
+
+  // Test written by Diffblue Cover.
+  @Test
+  public void print_json_diffInputNotNullZeroNotNullZeroNotNullOutputIllegalArgumentException()
+      throws InvocationTargetException {
+
+    // Arrange
+    final LogBuffer buffer = new LogBuffer();
+    buffer.position = 28;
+    buffer.semival = 0;
+    final byte[] myByteArray = {
+        (byte)2, (byte)2, (byte)2, (byte)2, (byte)2, (byte)2, (byte)2, (byte)2, (byte)2, (byte)2,
+        (byte)2, (byte)2, (byte)2, (byte)2, (byte)2, (byte)2, (byte)2, (byte)2, (byte)2, (byte)2,
+        (byte)2, (byte)2, (byte)2, (byte)2, (byte)2, (byte)2, (byte)2, (byte)2, (byte)3, (byte)2};
+    buffer.buffer = myByteArray;
+    buffer.limit = 31;
+    buffer.origin = 0;
+    final long len = 0L;
+    final String columnName = "foo";
+    final int columnIndex = 0;
+    final String charsetName = "foo";
+    try {
+
+      // Act
+      thrown.expect(IllegalArgumentException.class);
+      JsonDiffConversion.print_json_diff(buffer, len, columnName, columnIndex, charsetName);
+    } catch (IllegalArgumentException ex) {
+
+      // Assert side effects
+      Assert.assertNotNull(buffer);
+      Assert.assertEquals(29, buffer.position);
+      Assert.assertEquals(0, buffer.semival);
+      Assert.assertArrayEquals(new byte[] {(byte)2, (byte)2, (byte)2, (byte)2, (byte)2, (byte)2,
+                                           (byte)2, (byte)2, (byte)2, (byte)2, (byte)2, (byte)2,
+                                           (byte)2, (byte)2, (byte)2, (byte)2, (byte)2, (byte)2,
+                                           (byte)2, (byte)2, (byte)2, (byte)2, (byte)2, (byte)2,
+                                           (byte)2, (byte)2, (byte)2, (byte)2, (byte)3, (byte)2},
+                               buffer.buffer);
+      Assert.assertEquals(31, buffer.limit);
+      Assert.assertEquals(0, buffer.origin);
+      throw ex;
+    }
+  }
+
+  // Test written by Diffblue Cover.
+  @Test
+  public void print_json_diffInputNotNullZeroNotNullZeroNotNullOutputIllegalArgumentException2()
+      throws InvocationTargetException {
+
+    // Arrange
+    final LogBuffer buffer = new LogBuffer();
+    buffer.position = 15;
+    buffer.semival = 0;
+    final byte[] myByteArray = {
+        (byte)1, (byte)1, (byte)0, (byte)0, (byte)1, (byte)1, (byte)1, (byte)1, (byte)1, (byte)1,
+        (byte)0, (byte)1, (byte)1, (byte)1, (byte)1, (byte)0, (byte)1, (byte)1, (byte)0, (byte)0,
+        (byte)1, (byte)1, (byte)1, (byte)1, (byte)1, (byte)1, (byte)1, (byte)0, (byte)1, (byte)1};
+    buffer.buffer = myByteArray;
+    buffer.limit = -1_215_751_986;
+    buffer.origin = 1_215_752_002;
+    final long len = 0L;
+    final String columnName = "foo";
+    final int columnIndex = 0;
+    final String charsetName = "foo";
+    try {
+
+      // Act
+      thrown.expect(IllegalArgumentException.class);
+      JsonDiffConversion.print_json_diff(buffer, len, columnName, columnIndex, charsetName);
+    } catch (IllegalArgumentException ex) {
+
+      // Assert side effects
+      Assert.assertNotNull(buffer);
+      Assert.assertEquals(16, buffer.position);
+      Assert.assertEquals(0, buffer.semival);
+      Assert.assertArrayEquals(new byte[] {(byte)1, (byte)1, (byte)0, (byte)0, (byte)1, (byte)1,
+                                           (byte)1, (byte)1, (byte)1, (byte)1, (byte)0, (byte)1,
+                                           (byte)1, (byte)1, (byte)1, (byte)0, (byte)1, (byte)1,
+                                           (byte)0, (byte)0, (byte)1, (byte)1, (byte)1, (byte)1,
+                                           (byte)1, (byte)1, (byte)1, (byte)0, (byte)1, (byte)1},
+                               buffer.buffer);
+      Assert.assertEquals(-1_215_751_986, buffer.limit);
+      Assert.assertEquals(1_215_752_002, buffer.origin);
+      throw ex;
+    }
+  }
+
+  // Test written by Diffblue Cover.
+  @Test
+  public void print_json_diffInputNotNullZeroNotNullZeroNotNullOutputIllegalArgumentException3()
+      throws InvocationTargetException {
+
+    // Arrange
+    final LogBuffer buffer = new LogBuffer();
+    buffer.position = 27;
+    buffer.semival = 0;
+    final byte[] myByteArray = {
+        (byte)0, (byte)0, (byte)0, (byte)0, (byte)0, (byte)0, (byte)0, (byte)0, (byte)0, (byte)0,
+        (byte)0, (byte)0, (byte)0, (byte)0, (byte)0, (byte)0, (byte)0, (byte)0, (byte)0, (byte)0,
+        (byte)0, (byte)0, (byte)0, (byte)0, (byte)0, (byte)0, (byte)0, (byte)0, (byte)1, (byte)0};
+    buffer.buffer = myByteArray;
+    buffer.limit = 31;
+    buffer.origin = -1;
+    final long len = 0L;
+    final String columnName = "foo";
+    final int columnIndex = 0;
+    final String charsetName = "foo";
+    try {
+
+      // Act
+      thrown.expect(IllegalArgumentException.class);
+      JsonDiffConversion.print_json_diff(buffer, len, columnName, columnIndex, charsetName);
+    } catch (IllegalArgumentException ex) {
+
+      // Assert side effects
+      Assert.assertNotNull(buffer);
+      Assert.assertEquals(29, buffer.position);
+      Assert.assertEquals(0, buffer.semival);
+      Assert.assertArrayEquals(new byte[] {(byte)0, (byte)0, (byte)0, (byte)0, (byte)0, (byte)0,
+                                           (byte)0, (byte)0, (byte)0, (byte)0, (byte)0, (byte)0,
+                                           (byte)0, (byte)0, (byte)0, (byte)0, (byte)0, (byte)0,
+                                           (byte)0, (byte)0, (byte)0, (byte)0, (byte)0, (byte)0,
+                                           (byte)0, (byte)0, (byte)0, (byte)0, (byte)1, (byte)0},
+                               buffer.buffer);
+      Assert.assertEquals(31, buffer.limit);
+      Assert.assertEquals(-1, buffer.origin);
+      throw ex;
+    }
+  }
+
+  // Test written by Diffblue Cover.
+  @Test
+  public void print_json_diffInputNotNullZeroNotNullZeroNotNullOutputNotNull() {
+
+    // Arrange
+    final LogBuffer buffer = new LogBuffer();
+    final long len = 0L;
+    final String columnName = ",";
+    final int columnIndex = 0;
+    final String charsetName = "1a 2b 3c";
+
+    // Act
+    final StringBuilder actual =
+        JsonDiffConversion.print_json_diff(buffer, len, columnName, columnIndex, charsetName);
+
+    // Assert result
+    Assert.assertNotNull(actual);
+    Assert.assertEquals(",", actual.toString());
+  }
+}

dbsync/src/test/java/com/taobao/tddl/dbsync/binlog/LogBufferTest.java

@@ -1,312 +0,0 @@
-package com.taobao.tddl.dbsync.binlog;
-
-import java.math.BigDecimal;
-import java.math.BigInteger;
-
-import junit.framework.TestCase;
-
-public class LogBufferTest extends TestCase {
-
-    public static final int LOOP = 10000;
-
-    public void testSigned() {
-        byte[] array = { 0, 0, 0, (byte) 0xff };
-
-        LogBuffer buffer = new LogBuffer(array, 0, array.length);
-
-        System.out.println(buffer.getInt32(0));
-        System.out.println(buffer.getUint32(0));
-
-        System.out.println(buffer.getInt24(1));
-        System.out.println(buffer.getUint24(1));
-    }
-
-    public void testBigInteger() {
-        byte[] array = { (byte) 0xff, (byte) 0xff, (byte) 0xff, (byte) 0xff, (byte) 0xff, (byte) 0xff, (byte) 0xff,
-                (byte) 0xff };
-
-        LogBuffer buffer = new LogBuffer(array, 0, array.length);
-
-        long tt1 = 0;
-        long l1 = 0;
-        for (int i = 0; i < LOOP; i++) {
-            final long t1 = System.nanoTime();
-            l1 = buffer.getLong64(0);
-            tt1 += System.nanoTime() - t1;
-        }
-        System.out.print(tt1 / LOOP);
-        System.out.print("ns >> ");
-        System.out.println(l1);
-
-        long tt2 = 0;
-        BigInteger l2 = null;
-        for (int i = 0; i < LOOP; i++) {
-            final long t2 = System.nanoTime();
-            l2 = buffer.getUlong64(0);
-            tt2 += System.nanoTime() - t2;
-        }
-        System.out.print(tt2 / LOOP);
-        System.out.print("ns >> ");
-        System.out.println(l2);
-    }
-
-    /* Reads big-endian integer from no more than 4 bytes */
-    private static int convertNBytesToInt(byte[] buffer, int offset, int length) {
-        int ret = 0;
-        for (int i = offset; i < (offset + length); i++) {
-            ret = (ret << 8) | (0xff & buffer[i]);
-        }
-        return ret;
-    }
-
-    private static int convert4BytesToInt(byte[] buffer, int offset) {
-        int value;
-        value = (0xff & buffer[offset + 3]);
-        value += (0xff & buffer[offset + 2]) << 8;
-        value += (0xff & buffer[offset + 1]) << 16;
-        value += (0xff & buffer[offset]) << 24;
-        return value;
-    }
-
-    public static short convert1ByteToShort(byte[] buffer, int offset) {
-        short value;
-        value = (short) buffer[offset + 0];
-        return value;
-    }
-
-    public static short convert2bytesToShort(byte[] buffer, int offset) {
-        short value;
-        value = (short) (buffer[offset + 0] << 8);
-        value += (short) (buffer[offset + 1] & 0xff);
-        return value;
-    }
-
-    public static final BigDecimal extractDecimal(byte[] buffer, int precision, int scale) {
-        //
-        // Decimal representation in binlog seems to be as follows:
-        // 1 byte - 'precision'
-        // 1 byte - 'scale'
-        // remaining n bytes - integer such that value = n / (10^scale)
-        // Integer is represented as follows:
-        // 1st bit - sign such that set == +, unset == -
-        // every 4 bytes represent 9 digits in big-endian order, so that if
-        // you print the values of these quads as big-endian integers one after
-        // another, you get the whole number string representation in decimal.
-        // What remains is to put a sign and a decimal dot.
-        // 13 0a 80 00 00 05 1b 38 b0 60 00 means:
-        // 0x13 - precision = 19
-        // 0x0a - scale = 10
-        // 0x80 - positive
-        // 0x00000005 0x1b38b060 0x00
-        // 5 456700000 0
-        // 54567000000 / 10^{10} = 5.4567
-        //
-        // int_size below shows how long is integer part
-        //
-        // offset = offset + 2; // offset of the number part
-        //
-        int intg = precision - scale;
-        int intg0 = intg / LogBuffer.DIG_PER_INT32;
-        int frac0 = scale / LogBuffer.DIG_PER_INT32;
-        int intg0x = intg - intg0 * LogBuffer.DIG_PER_INT32;
-        int frac0x = scale - frac0 * LogBuffer.DIG_PER_INT32;
-
-        int offset = 0;
-
-        int sign = (buffer[offset] & 0x80) == 0x80 ? 1 : -1;
-
-        // how many bytes are used to represent given amount of digits?
-        int integerSize = intg0 * LogBuffer.SIZE_OF_INT32 + LogBuffer.dig2bytes[intg0x];
-        int decimalSize = frac0 * LogBuffer.SIZE_OF_INT32 + LogBuffer.dig2bytes[frac0x];
-
-        int bin_size = integerSize + decimalSize; // total bytes
-        byte[] d_copy = new byte[bin_size];
-
-        if (bin_size > buffer.length) {
-            throw new ArrayIndexOutOfBoundsException("Calculated bin_size: " + bin_size + ", available bytes: "
-                                                     + buffer.length);
-        }
-
-        // Invert first bit
-        d_copy[0] = buffer[0];
-        d_copy[0] ^= 0x80;
-        if (sign == -1) {
-            // Invert every byte
-            d_copy[0] ^= 0xFF;
-        }
-
-        for (int i = 1; i < bin_size; i++) {
-            d_copy[i] = buffer[i];
-            if (sign == -1) {
-                // Invert every byte
-                d_copy[i] ^= 0xFF;
-            }
-        }
-
-        // Integer part
-        offset = LogBuffer.dig2bytes[intg0x];
-
-        BigDecimal intPart = new BigDecimal(0);
-
-        if (offset > 0) intPart = BigDecimal.valueOf(convertNBytesToInt(d_copy, 0, offset));
-
-        while (offset < integerSize) {
-            intPart = intPart.movePointRight(LogBuffer.DIG_PER_DEC1).add(BigDecimal.valueOf(convert4BytesToInt(d_copy,
-                offset)));
-            offset += 4;
-        }
-
-        // Decimal part
-        BigDecimal fracPart = new BigDecimal(0);
-        int shift = 0;
-        for (int i = 0; i < frac0; i++) {
-            shift += LogBuffer.DIG_PER_DEC1;
-            fracPart = fracPart.add(BigDecimal.valueOf(convert4BytesToInt(d_copy, offset)).movePointLeft(shift));
-            offset += 4;
-        }
-
-        if (LogBuffer.dig2bytes[frac0x] > 0) {
-            fracPart = fracPart.add(BigDecimal.valueOf(convertNBytesToInt(d_copy, offset, LogBuffer.dig2bytes[frac0x]))
-                .movePointLeft(shift + frac0x));
-        }
-
-        return BigDecimal.valueOf(sign).multiply(intPart.add(fracPart));
-    }
-
-    public static final byte[] array1 = { (byte) 0x80, 0x00, 0x00, 0x05, 0x1b, 0x38, (byte) 0xb0, 0x60, 0x00 };
-
-    public static final byte[] array2 = { (byte) 0x7f, (byte) 0xff, (byte) 0xff, (byte) 0xfb, (byte) 0xe4, (byte) 0xc7,
-            (byte) 0x4f, (byte) 0xa0, (byte) 0xff };
-
-    public static final byte[] array3 = { -128, 0, 6, 20, 113, 56, 6, 26, -123 };
-
-    public static final byte[] array4 = { -128, 7, 0, 0, 0, 1, 0, 0, 3 };
-
-    public static final byte[] array5 = { -128, 0, 0, 0, 0, 1, 1, -122, -96, -108 };
-
-    public void testBigDecimal() throws InterruptedException {
-        do {
-            System.out.println("old extract decimal: ");
-
-            long tt1 = 0;
-            BigDecimal bd1 = null;
-            for (int i = 0; i < LOOP; i++) {
-                final long t1 = System.nanoTime();
-                bd1 = extractDecimal(array2, 19, 10);
-                tt1 += System.nanoTime() - t1;
-            }
-            System.out.print(tt1 / LOOP);
-            System.out.print("ns >> ");
-            System.out.println(bd1);
-
-            long tt2 = 0;
-            BigDecimal bd2 = null;
-            for (int i = 0; i < LOOP; i++) {
-                final long t2 = System.nanoTime();
-                bd2 = extractDecimal(array1, 19, 10);
-                tt2 += System.nanoTime() - t2;
-            }
-            System.out.print(tt2 / LOOP);
-            System.out.print("ns >> ");
-            System.out.println(bd2);
-
-            long tt3 = 0;
-            BigDecimal bd3 = null;
-            for (int i = 0; i < LOOP; i++) {
-                final long t3 = System.nanoTime();
-                bd3 = extractDecimal(array3, 18, 6);
-                tt3 += System.nanoTime() - t3;
-            }
-            System.out.print(tt3 / LOOP);
-            System.out.print("ns >> ");
-            System.out.println(bd3);
-
-            long tt4 = 0;
-            BigDecimal bd4 = null;
-            for (int i = 0; i < LOOP; i++) {
-                final long t4 = System.nanoTime();
-                bd4 = extractDecimal(array4, 18, 6);
-                tt4 += System.nanoTime() - t4;
-            }
-            System.out.print(tt4 / LOOP);
-            System.out.print("ns >> ");
-            System.out.println(bd4);
-
-            long tt5 = 0;
-            BigDecimal bd5 = null;
-            for (int i = 0; i < LOOP; i++) {
-                final long t5 = System.nanoTime();
-                bd5 = extractDecimal(array5, 18, 6);
-                tt5 += System.nanoTime() - t5;
-            }
-            System.out.print(tt5 / LOOP);
-            System.out.print("ns >> ");
-            System.out.println(bd5);
-        } while (false);
-
-        do {
-            System.out.println("new extract decimal: ");
-
-            LogBuffer buffer1 = new LogBuffer(array2, 0, array2.length);
-            LogBuffer buffer2 = new LogBuffer(array1, 0, array1.length);
-            LogBuffer buffer3 = new LogBuffer(array3, 0, array3.length);
-            LogBuffer buffer4 = new LogBuffer(array4, 0, array4.length);
-            LogBuffer buffer5 = new LogBuffer(array5, 0, array5.length);
-
-            long tt1 = 0;
-            BigDecimal bd1 = null;
-            for (int i = 0; i < LOOP; i++) {
-                final long t1 = System.nanoTime();
-                bd1 = buffer1.getDecimal(0, 19, 10);
-                tt1 += System.nanoTime() - t1;
-            }
-            System.out.print(tt1 / LOOP);
-            System.out.print("ns >> ");
-            System.out.println(bd1);
-
-            long tt2 = 0;
-            BigDecimal bd2 = null;
-            for (int i = 0; i < LOOP; i++) {
-                final long t2 = System.nanoTime();
-                bd2 = buffer2.getDecimal(0, 19, 10);
-                tt2 += System.nanoTime() - t2;
-            }
-            System.out.print(tt2 / LOOP);
-            System.out.print("ns >> ");
-            System.out.println(bd2);
-
-            long tt3 = 0;
-            BigDecimal bd3 = null;
-            for (int i = 0; i < LOOP; i++) {
-                final long t3 = System.nanoTime();
-                bd3 = buffer3.getDecimal(0, 18, 6);
-                tt3 += System.nanoTime() - t3;
-            }
-            System.out.print(tt3 / LOOP);
-            System.out.print("ns >> ");
-            System.out.println(bd3);
-
-            long tt4 = 0;
-            BigDecimal bd4 = null;
-            for (int i = 0; i < LOOP; i++) {
-                final long t4 = System.nanoTime();
-                bd4 = buffer4.getDecimal(0, 18, 6);
-                tt4 += System.nanoTime() - t4;
-            }
-            System.out.print(tt4 / LOOP);
-            System.out.print("ns >> ");
-            System.out.println(bd4);
-
-            long tt5 = 0;
-            BigDecimal bd5 = null;
-            for (int i = 0; i < LOOP; i++) {
-                final long t5 = System.nanoTime();
-                bd5 = buffer5.getDecimal(0, 18, 6);
-                tt5 += System.nanoTime() - t5;
-            }
-            System.out.print(tt5 / LOOP);
-            System.out.print("ns >> ");
-            System.out.println(bd5);
-        } while (false);
-    }
-}

dbsync/src/test/java/com/taobao/tddl/dbsync/binlog/LogEventTest.java

@@ -0,0 +1,534 @@
+package com.taobao.tddl.dbsync.binlog;
+
+import com.taobao.tddl.dbsync.binlog.LogEvent;
+import org.junit.Assert;
+import org.junit.Rule;
+import org.junit.Test;
+import org.junit.rules.ExpectedException;
+import org.junit.rules.Timeout;
+
+public class LogEventTest {
+
+  @Rule public final ExpectedException thrown = ExpectedException.none();
+
+  @Rule public final Timeout globalTimeout = new Timeout(10000);
+
+  /* testedClasses: LogEvent */
+  // Test written by Diffblue Cover.
+  @Test
+  public void getTypeNameInputPositiveOutputNotNull() {
+
+    // Arrange
+    final int type = 8;
+
+    // Act
+    final String actual = LogEvent.getTypeName(type);
+
+    // Assert result
+    Assert.assertEquals("Create_file", actual);
+  }
+
+  // Test written by Diffblue Cover.
+  @Test
+  public void getTypeNameInputPositiveOutputNotNull2() {
+
+    // Arrange
+    final int type = 12;
+
+    // Act
+    final String actual = LogEvent.getTypeName(type);
+
+    // Assert result
+    Assert.assertEquals("New_load", actual);
+  }
+
+  // Test written by Diffblue Cover.
+  @Test
+  public void getTypeNameInputPositiveOutputNotNull3() {
+
+    // Arrange
+    final int type = 39;
+
+    // Act
+    final String actual = LogEvent.getTypeName(type);
+
+    // Assert result
+    Assert.assertEquals("Update_rows_partial", actual);
+  }
+
+  // Test written by Diffblue Cover.
+  @Test
+  public void getTypeNameInputPositiveOutputNotNull4() {
+
+    // Arrange
+    final int type = 36;
+
+    // Act
+    final String actual = LogEvent.getTypeName(type);
+
+    // Assert result
+    Assert.assertEquals("Unknown", actual);
+  }
+
+  // Test written by Diffblue Cover.
+  @Test
+  public void getTypeNameInputPositiveOutputNotNull5() {
+
+    // Arrange
+    final int type = 7;
+
+    // Act
+    final String actual = LogEvent.getTypeName(type);
+
+    // Assert result
+    Assert.assertEquals("Slave", actual);
+  }
+
+  // Test written by Diffblue Cover.
+  @Test
+  public void getTypeNameInputPositiveOutputNotNull6() {
+
+    // Arrange
+    final int type = 6;
+
+    // Act
+    final String actual = LogEvent.getTypeName(type);
+
+    // Assert result
+    Assert.assertEquals("Load", actual);
+  }
+
+  // Test written by Diffblue Cover.
+  @Test
+  public void getTypeNameInputPositiveOutputNotNull7() {
+
+    // Arrange
+    final int type = 10;
+
+    // Act
+    final String actual = LogEvent.getTypeName(type);
+
+    // Assert result
+    Assert.assertEquals("Exec_load", actual);
+  }
+
+  // Test written by Diffblue Cover.
+  @Test
+  public void getTypeNameInputPositiveOutputNotNull8() {
+
+    // Arrange
+    final int type = 5;
+
+    // Act
+    final String actual = LogEvent.getTypeName(type);
+
+    // Assert result
+    Assert.assertEquals("Intvar", actual);
+  }
+
+  // Test written by Diffblue Cover.
+  @Test
+  public void getTypeNameInputPositiveOutputNotNull9() {
+
+    // Arrange
+    final int type = 11;
+
+    // Act
+    final String actual = LogEvent.getTypeName(type);
+
+    // Assert result
+    Assert.assertEquals("Delete_file", actual);
+  }
+
+  // Test written by Diffblue Cover.
+  @Test
+  public void getTypeNameInputPositiveOutputNotNull10() {
+
+    // Arrange
+    final int type = 4;
+
+    // Act
+    final String actual = LogEvent.getTypeName(type);
+
+    // Assert result
+    Assert.assertEquals("Rotate", actual);
+  }
+
+  // Test written by Diffblue Cover.
+  @Test
+  public void getTypeNameInputPositiveOutputNotNull11() {
+
+    // Arrange
+    final int type = 13;
+
+    // Act
+    final String actual = LogEvent.getTypeName(type);
+
+    // Assert result
+    Assert.assertEquals("RAND", actual);
+  }
+
+  // Test written by Diffblue Cover.
+  @Test
+  public void getTypeNameInputPositiveOutputNotNull12() {
+
+    // Arrange
+    final int type = 34;
+
+    // Act
+    final String actual = LogEvent.getTypeName(type);
+
+    // Assert result
+    Assert.assertEquals("Anonymous_Gtid", actual);
+  }
+
+  // Test written by Diffblue Cover.
+  @Test
+  public void getTypeNameInputPositiveOutputNotNull13() {
+
+    // Arrange
+    final int type = 15;
+
+    // Act
+    final String actual = LogEvent.getTypeName(type);
+
+    // Assert result
+    Assert.assertEquals("Format_desc", actual);
+  }
+
+  // Test written by Diffblue Cover.
+  @Test
+  public void getTypeNameInputPositiveOutputNotNull14() {
+
+    // Arrange
+    final int type = 32;
+
+    // Act
+    final String actual = LogEvent.getTypeName(type);
+
+    // Assert result
+    Assert.assertEquals("Delete_rows", actual);
+  }
+
+  // Test written by Diffblue Cover.
+  @Test
+  public void getTypeNameInputPositiveOutputNotNull15() {
+
+    // Arrange
+    final int type = 16;
+
+    // Act
+    final String actual = LogEvent.getTypeName(type);
+
+    // Assert result
+    Assert.assertEquals("Xid", actual);
+  }
+
+  // Test written by Diffblue Cover.
+  @Test
+  public void getTypeNameInputPositiveOutputNotNull16() {
+
+    // Arrange
+    final int type = 30;
+
+    // Act
+    final String actual = LogEvent.getTypeName(type);
+
+    // Assert result
+    Assert.assertEquals("Write_rows", actual);
+  }
+
+  // Test written by Diffblue Cover.
+  @Test
+  public void getTypeNameInputPositiveOutputNotNull17() {
+
+    // Arrange
+    final int type = 18;
+
+    // Act
+    final String actual = LogEvent.getTypeName(type);
+
+    // Assert result
+    Assert.assertEquals("Execute_load_query", actual);
+  }
+
+  // Test written by Diffblue Cover.
+  @Test
+  public void getTypeNameInputPositiveOutputNotNull18() {
+
+    // Arrange
+    final int type = 29;
+
+    // Act
+    final String actual = LogEvent.getTypeName(type);
+
+    // Assert result
+    Assert.assertEquals("Rows_query", actual);
+  }
+
+  // Test written by Diffblue Cover.
+  @Test
+  public void getTypeNameInputPositiveOutputNotNull19() {
+
+    // Arrange
+    final int type = 19;
+
+    // Act
+    final String actual = LogEvent.getTypeName(type);
+
+    // Assert result
+    Assert.assertEquals("Table_map", actual);
+  }
+
+  // Test written by Diffblue Cover.
+  @Test
+  public void getTypeNameInputPositiveOutputNotNull20() {
+
+    // Arrange
+    final int type = 28;
+
+    // Act
+    final String actual = LogEvent.getTypeName(type);
+
+    // Assert result
+    Assert.assertEquals("Ignorable", actual);
+  }
+
+  // Test written by Diffblue Cover.
+  @Test
+  public void getTypeNameInputPositiveOutputNotNull21() {
+
+    // Arrange
+    final int type = 33;
+
+    // Act
+    final String actual = LogEvent.getTypeName(type);
+
+    // Assert result
+    Assert.assertEquals("Gtid", actual);
+  }
+
+  // Test written by Diffblue Cover.
+  @Test
+  public void getTypeNameInputPositiveOutputNotNull22() {
+
+    // Arrange
+    final int type = 27;
+
+    // Act
+    final String actual = LogEvent.getTypeName(type);
+
+    // Assert result
+    Assert.assertEquals("Heartbeat", actual);
+  }
+
+  // Test written by Diffblue Cover.
+  @Test
+  public void getTypeNameInputPositiveOutputNotNull23() {
+
+    // Arrange
+    final int type = 26;
+
+    // Act
+    final String actual = LogEvent.getTypeName(type);
+
+    // Assert result
+    Assert.assertEquals("Incident", actual);
+  }
+
+  // Test written by Diffblue Cover.
+  @Test
+  public void getTypeNameInputPositiveOutputNotNull24() {
+
+    // Arrange
+    final int type = 31;
+
+    // Act
+    final String actual = LogEvent.getTypeName(type);
+
+    // Assert result
+    Assert.assertEquals("Update_rows", actual);
+  }
+
+  // Test written by Diffblue Cover.
+  @Test
+  public void getTypeNameInputPositiveOutputNotNull25() {
+
+    // Arrange
+    final int type = 25;
+
+    // Act
+    final String actual = LogEvent.getTypeName(type);
+
+    // Assert result
+    Assert.assertEquals("Delete_rows_v1", actual);
+  }
+
+  // Test written by Diffblue Cover.
+  @Test
+  public void getTypeNameInputPositiveOutputNotNull26() {
+
+    // Arrange
+    final int type = 24;
+
+    // Act
+    final String actual = LogEvent.getTypeName(type);
+
+    // Assert result
+    Assert.assertEquals("Update_rows_v1", actual);
+  }
+
+  // Test written by Diffblue Cover.
+  @Test
+  public void getTypeNameInputPositiveOutputNotNull27() {
+
+    // Arrange
+    final int type = 23;
+
+    // Act
+    final String actual = LogEvent.getTypeName(type);
+
+    // Assert result
+    Assert.assertEquals("Write_rows_v1", actual);
+  }
+
+  // Test written by Diffblue Cover.
+  @Test
+  public void getTypeNameInputPositiveOutputNotNull28() {
+
+    // Arrange
+    final int type = 35;
+
+    // Act
+    final String actual = LogEvent.getTypeName(type);
+
+    // Assert result
+    Assert.assertEquals("Previous_gtids", actual);
+  }
+
+  // Test written by Diffblue Cover.
+  @Test
+  public void getTypeNameInputPositiveOutputNotNull29() {
+
+    // Arrange
+    final int type = 22;
+
+    // Act
+    final String actual = LogEvent.getTypeName(type);
+
+    // Assert result
+    Assert.assertEquals("Delete_rows_event_old", actual);
+  }
+
+  // Test written by Diffblue Cover.
+  @Test
+  public void getTypeNameInputPositiveOutputNotNull30() {
+
+    // Arrange
+    final int type = 21;
+
+    // Act
+    final String actual = LogEvent.getTypeName(type);
+
+    // Assert result
+    Assert.assertEquals("Update_rows_event_old", actual);
+  }
+
+  // Test written by Diffblue Cover.
+  @Test
+  public void getTypeNameInputPositiveOutputNotNull31() {
+
+    // Arrange
+    final int type = 20;
+
+    // Act
+    final String actual = LogEvent.getTypeName(type);
+
+    // Assert result
+    Assert.assertEquals("Write_rows_event_old", actual);
+  }
+
+  // Test written by Diffblue Cover.
+  @Test
+  public void getTypeNameInputPositiveOutputNotNull32() {
+
+    // Arrange
+    final int type = 17;
+
+    // Act
+    final String actual = LogEvent.getTypeName(type);
+
+    // Assert result
+    Assert.assertEquals("Begin_load_query", actual);
+  }
+
+  // Test written by Diffblue Cover.
+  @Test
+  public void getTypeNameInputPositiveOutputNotNull33() {
+
+    // Arrange
+    final int type = 14;
+
+    // Act
+    final String actual = LogEvent.getTypeName(type);
+
+    // Assert result
+    Assert.assertEquals("User var", actual);
+  }
+
+  // Test written by Diffblue Cover.
+  @Test
+  public void getTypeNameInputPositiveOutputNotNull34() {
+
+    // Arrange
+    final int type = 9;
+
+    // Act
+    final String actual = LogEvent.getTypeName(type);
+
+    // Assert result
+    Assert.assertEquals("Append_block", actual);
+  }
+
+  // Test written by Diffblue Cover.
+  @Test
+  public void getTypeNameInputPositiveOutputNotNull35() {
+
+    // Arrange
+    final int type = 2;
+
+    // Act
+    final String actual = LogEvent.getTypeName(type);
+
+    // Assert result
+    Assert.assertEquals("Query", actual);
+  }
+
+  // Test written by Diffblue Cover.
+  @Test
+  public void getTypeNameInputPositiveOutputNotNull36() {
+
+    // Arrange
+    final int type = 3;
+
+    // Act
+    final String actual = LogEvent.getTypeName(type);
+
+    // Assert result
+    Assert.assertEquals("Stop", actual);
+  }
+
+  // Test written by Diffblue Cover.
+  @Test
+  public void getTypeNameInputPositiveOutputNotNull37() {
+
+    // Arrange
+    final int type = 1;
+
+    // Act
+    final String actual = LogEvent.getTypeName(type);
+
+    // Assert result
+    Assert.assertEquals("Start_v3", actual);
+  }
+}

dbsync/src/test/java/com/taobao/tddl/dbsync/binlog/event/LogHeaderTest.java

@@ -0,0 +1,231 @@
+package com.taobao.tddl.dbsync.binlog.event;
+
+import com.taobao.tddl.dbsync.binlog.event.LogHeader;
+import org.junit.Assert;
+import org.junit.Rule;
+import org.junit.Test;
+import org.junit.rules.ExpectedException;
+import org.junit.rules.Timeout;
+
+import java.util.HashMap;
+
+public class LogHeaderTest {
+
+  @Rule public final ExpectedException thrown = ExpectedException.none();
+
+  @Rule public final Timeout globalTimeout = new Timeout(10000);
+
+  /* testedClasses: LogHeader */
+  // Test written by Diffblue Cover.
+
+  @Test
+  public void constructorInputZeroOutputVoid() {
+
+    // Arrange
+    final int type = 0;
+
+    // Act, creating object to test constructor
+    final LogHeader objectUnderTest = new LogHeader(type);
+
+    // Assert side effects
+    final HashMap<String, String> hashMap = new HashMap<String, String>();
+    Assert.assertEquals(hashMap, objectUnderTest.gtidMap);
+  }
+
+  // Test written by Diffblue Cover.
+  @Test
+  public void getChecksumAlgOutputZero() {
+
+    // Arrange
+    final LogHeader objectUnderTest = new LogHeader(0);
+
+    // Act
+    final int actual = objectUnderTest.getChecksumAlg();
+
+    // Assert result
+    Assert.assertEquals(0, actual);
+  }
+
+  // Test written by Diffblue Cover.
+  @Test
+  public void getCrcOutputZero() {
+
+    // Arrange
+    final LogHeader objectUnderTest = new LogHeader(0);
+
+    // Act
+    final long actual = objectUnderTest.getCrc();
+
+    // Assert result
+    Assert.assertEquals(0L, actual);
+  }
+
+  // Test written by Diffblue Cover.
+  @Test
+  public void getCurrentGtidLastCommitOutputNull() {
+
+    // Arrange
+    final LogHeader objectUnderTest = new LogHeader(0);
+
+    // Act
+    final String actual = objectUnderTest.getCurrentGtidLastCommit();
+
+    // Assert result
+    Assert.assertNull(actual);
+  }
+
+  // Test written by Diffblue Cover.
+  @Test
+  public void getCurrentGtidOutputNull() {
+
+    // Arrange
+    final LogHeader objectUnderTest = new LogHeader(0);
+
+    // Act
+    final String actual = objectUnderTest.getCurrentGtid();
+
+    // Assert result
+    Assert.assertNull(actual);
+  }
+
+  // Test written by Diffblue Cover.
+  @Test
+  public void getCurrentGtidSnOutputNull() {
+
+    // Arrange
+    final LogHeader objectUnderTest = new LogHeader(0);
+
+    // Act
+    final String actual = objectUnderTest.getCurrentGtidSn();
+
+    // Assert result
+    Assert.assertNull(actual);
+  }
+
+  // Test written by Diffblue Cover.
+  @Test
+  public void getEventLenOutputZero() {
+
+    // Arrange
+    final LogHeader objectUnderTest = new LogHeader(0);
+
+    // Act
+    final int actual = objectUnderTest.getEventLen();
+
+    // Assert result
+    Assert.assertEquals(0, actual);
+  }
+
+  // Test written by Diffblue Cover.
+  @Test
+  public void getFlagsOutputZero() {
+
+    // Arrange
+    final LogHeader objectUnderTest = new LogHeader(0);
+
+    // Act
+    final int actual = objectUnderTest.getFlags();
+
+    // Assert result
+    Assert.assertEquals(0, actual);
+  }
+
+  // Test written by Diffblue Cover.
+  @Test
+  public void getGtidSetStrOutputNull() {
+
+    // Arrange
+    final LogHeader objectUnderTest = new LogHeader(0);
+
+    // Act
+    final String actual = objectUnderTest.getGtidSetStr();
+
+    // Assert result
+    Assert.assertNull(actual);
+  }
+
+  // Test written by Diffblue Cover.
+  @Test
+  public void getLogFileNameOutputNull() {
+
+    // Arrange
+    final LogHeader objectUnderTest = new LogHeader(0);
+
+    // Act
+    final String actual = objectUnderTest.getLogFileName();
+
+    // Assert result
+    Assert.assertNull(actual);
+  }
+
+  // Test written by Diffblue Cover.
+  @Test
+  public void getLogPosOutputZero() {
+
+    // Arrange
+    final LogHeader objectUnderTest = new LogHeader(0);
+
+    // Act
+    final long actual = objectUnderTest.getLogPos();
+
+    // Assert result
+    Assert.assertEquals(0L, actual);
+  }
+
+  // Test written by Diffblue Cover.
+  @Test
+  public void getServerIdOutputZero() {
+
+    // Arrange
+    final LogHeader objectUnderTest = new LogHeader(0);
+
+    // Act
+    final long actual = objectUnderTest.getServerId();
+
+    // Assert result
+    Assert.assertEquals(0L, actual);
+  }
+
+  // Test written by Diffblue Cover.
+  @Test
+  public void getTypeOutputZero() {
+
+    // Arrange
+    final LogHeader objectUnderTest = new LogHeader(0);
+
+    // Act
+    final int actual = objectUnderTest.getType();
+
+    // Assert result
+    Assert.assertEquals(0, actual);
+  }
+
+  // Test written by Diffblue Cover.
+  @Test
+  public void getWhenOutputZero() {
+
+    // Arrange
+    final LogHeader objectUnderTest = new LogHeader(0);
+
+    // Act
+    final long actual = objectUnderTest.getWhen();
+
+    // Assert result
+    Assert.assertEquals(0L, actual);
+  }
+
+  // Test written by Diffblue Cover.
+  @Test
+  public void setLogFileNameInputNotNullOutputVoid() {
+
+    // Arrange
+    final LogHeader objectUnderTest = new LogHeader(0);
+    final String logFileName = "3";
+
+    // Act
+    objectUnderTest.setLogFileName(logFileName);
+
+    // Assert side effects
+    Assert.assertEquals("3", objectUnderTest.getLogFileName());
+  }
+}