mirror of
https://github.com/alibaba/canal.git
synced 2026-11-04 00:30:15 +08:00
Change Contents for FQA and Naming
boxin.wbx
137
FAQ.md
137
FAQ.md
@@ -1,68 +1,69 @@
|
||||
问1：INSERT/UPDATE/DELETE被解析为Query或DDL语句？
|
||||
|
||||
答1：
|
||||
出现这类情况主要原因为收到的binlog就为Query事件，比如:
|
||||
1. binlog格式为非row模式，通过show variables like 'binlog\_format'可以查看. 针对statement/mixed模式，DML语句都会是以SQL语句存在
|
||||
2. mysql5.6+之后，在binlog为row模式下，针对DML语句通过一个开关(binlog-rows-query-log-events=true, show variables里也可以看到该变量)，记录DML的原始SQL，对应binlog事件为RowsQueryLogEvent，同时也有对应的row记录. ps. canal可以通过properties设置来过滤：canal.instance.filter.query.dml = true
|
||||
|
||||
|
||||
---
|
||||
|
||||
|
||||
问2：设置了数据表的过滤条件，但貌似没有生效？
|
||||
|
||||
答2：
|
||||
1. 首先看文档[AdminGuide](AdminGuide)，了解<span data-type="color" style="color:rgb(36, 41, 46)"><span data-type="background" style="background-color:rgb(255, 255, 255)">canal.instance.filter.regex的书写格式</span></span>
|
||||
```plain
|
||||
mysql 数据解析关注的表，Perl正则表达式.
|
||||
多个正则之间以逗号(,)分隔，转义符需要双斜杠(\\)
|
||||
常见例子：
|
||||
1. 所有表：.* or .*\\..*
|
||||
2. canal schema下所有表： canal\\..*
|
||||
3. canal下的以canal打头的表：canal\\.canal.*
|
||||
4. canal schema下的一张表：canal.test1
|
||||
5. 多个规则组合使用：canal\\..*,mysql.test1,mysql.test2 (逗号分隔)
|
||||
```
|
||||
2. 检查binlog格式，过滤条件只针对row模式的数据有效(ps. mixed/statement因为不解析sql，所以无法准确提取tableName进行过滤)
|
||||
3. 检查下CanalConnector是否调用subscribe(filter)方法；有的话，filter需要和instance.properties的canal.instance.filter.regex一致，否则subscribe的filter会覆盖instance的配置，如果subscribe的filter是.\*\\..\*，那么相当于你消费了所有的更新数据 <span data-type="color" style="color:#F5222D">【特别注意】</span>
|
||||
4. canal 1.1.3+版本之后,会在日志里记录最后使用的filter条件，可以对比使用的filter看看是否和自己期望的是一致，如果不一致检查一下第3步
|
||||
```plain
|
||||
c.a.o.canal.parse.inbound.mysql.dbsync.LogEventConvert - --> init table filter : ^.*\..*$
|
||||
c.a.o.canal.parse.inbound.mysql.dbsync.LogEventConvert - --> init table black filter :
|
||||
```
|
||||
5. 检查一下[历史的issue列表](https://github.com/alibaba/canal/issues?utf8=%E2%9C%93&q=subscribe)，很有可能你的提问别人已经遇到过并解决了，比如表达式不对，特别是双斜杠的问题
|
||||
|
||||
---
|
||||
|
||||
问3：canal是否支持aliyun rds的binlog订阅？
|
||||
|
||||
答3： 参考issue：[https://github.com/alibaba/canal/issues/727](https://github.com/alibaba/canal/issues/727)
|
||||
|
||||
|
||||
---
|
||||
|
||||
问4：canal的整体性能怎么样？
|
||||
|
||||
答4：
|
||||
* 参考issue: [https://github.com/alibaba/canal/issues/726](https://github.com/alibaba/canal/issues/726)
|
||||
* 参考性能测试文档：[https://github.com/alibaba/canal/wiki/Performance](https://github.com/alibaba/canal/wiki/Performance)
|
||||
|
||||
|
||||
---
|
||||
|
||||
|
||||
问5： 出现batchId:73 is not the firstly:72 或 clientId:1001 batchId:50560 is not exist , please check
|
||||
|
||||
答5：
|
||||
1. 第一个原因是client在ack的问题，前两个异常是client没有按照顺序ack对应的batchId
|
||||
2. 最后一个是ack的batchId在服务端被清理了ps. 服务端发生清理只有两个原因：
|
||||
* client发起过一次rollback
|
||||
* server端发生过一次instance的重启，比如scan=true时发现文件变化自动restart了
|
||||
|
||||
|
||||
---
|
||||
|
||||
|
||||
问6：Received error packet: errno = 1236, sqlstate = HY000 errmsg = Could not find first log file name in binary log index file
|
||||
|
||||
答6: 出现dump异常时，建议把errno/errmsg到google上搜索一下，这个异常是mysql标准返回的，代表对应的binlog位点在mysql里不存在，应该是被mysql server清理掉了，解决办法：重置一下canal订阅位点
|
||||
# Frequently Asked Quesionts
|
||||
|
||||
Q1: Why INSERT/UPDATE/DELETE is parsed as a Query or DDL?
|
||||
|
||||
A1: Generally, it's because the binlog is a Query event. For example:
|
||||
|
||||
1. Binlog format is configured as statement mode or mixed mode . You could check with `show variables like 'binlog_format' ` . For the statement or mixed mode, DML will exist as a SQL.
|
||||
2. For MySQL5.6+ binlog row mode, there is a bool variable for DML to control the record of raw SQL: binlog-rows-query-log-events=true (check by `show variables`). Corresponding event for binlog is RowsQueryLogEvent, also keeping row records. Via properties settings in ps.canal, users could set filters: `canal.instance.filter.query.dml=true`
|
||||
|
||||
-------------------
|
||||
|
||||
Q2: I set filters for the table but it doesn't work.
|
||||
|
||||
A2:
|
||||
|
||||
1. Please step to [AdminGuide](https://github.com/alibaba/canal/wiki/AdminGuide) for the details of canal.instance.filter.regex.
|
||||
|
||||
```
|
||||
MySQL parses the table and Perl supports regex.
|
||||
Use comma(,) for multiple regex, double backslash(\\) for escape characters.
|
||||
Examples:
|
||||
1. All the tables: .* or .*\\..*
|
||||
2. All the tables in canal scheme: canal\\..*
|
||||
3. Tables starting with canal in canal scheme: canal\\.canal.*
|
||||
4. Access a table in canal scheme: canal.test1
|
||||
5. A combination of multiple rules: canal\\..*, mysql.test1, mysql.test2 (with comma as delimeter)
|
||||
```
|
||||
|
||||
2. Check with the format of binlog. Filters will be valid only for data in row mode. (Mixed/statement doesn't parse SQL and thus cannot exactly extract tableName for filtering.)
|
||||
|
||||
3. Check whether CanalConnectors have called subscribe(filter) method. If so, filters should be consistent with the canal.instance.filter.regex in instance.properties, otherwise filters of subscribe will overwrite the settings in the instance. Please NOTE that, if the filter for subscribe is .\*/..\*, then you have consumed all the data updated.
|
||||
|
||||
4. For canal version 1.1.3+, it will log the most recent filter condition. You can see if the filter is as expected, if not, please check with Step 3
|
||||
|
||||
```
|
||||
c.a.o.canal.parse.inbound.mysql.dbsync.LogEventConvert - --> init table filter : ^.*\..*$
|
||||
c.a.o.canal.parse.inbound.mysql.dbsync.LogEventConvert - --> init table black filter :
|
||||
```
|
||||
|
||||
5. See the [Issues](https://github.com/alibaba/canal/issues?utf8=%E2%9C%93&q=subscribe). Some of the problems have already been solved, such as double backslash problems and expression incorrectness.
|
||||
|
||||
---------------
|
||||
|
||||
Q3: Does canal support binlog subscribe of aliyun rds?
|
||||
|
||||
A3: Yes. See Issue [#727](https://github.com/alibaba/canal/issues/727).
|
||||
|
||||
-----------------
|
||||
|
||||
Q4: How is the overall performance of canal?
|
||||
|
||||
A4: See Issue [#726](<https://github.com/alibaba/canal/issues/726) and [Performance Documents](https://github.com/alibaba/canal/wiki/Performance).
|
||||
|
||||
----------------
|
||||
|
||||
Q5: “batchId:73 is not the firstly:72" or "clientId:1001 batchId:50560 is not exist , please check”
|
||||
|
||||
A5:
|
||||
|
||||
1. First problem is with the client ack: the first two exceptions are caused by improper ack sequence to batchId.
|
||||
2. The second problem is that batchId of ack got cleaned by the server. There are only two reasons for server clean:
|
||||
- client has set up a rollback
|
||||
- server has launched an instance restart (e.g. scan=true causes automatic restart when file changed)
|
||||
|
||||
--------
|
||||
|
||||
Q6: "Received error packet: errno = 1236, sqlstate = HY000 errmsg = Could not find first log file name in binary log index file"
|
||||
|
||||
A6: This is a dump exception. We'd suggest google errno/errmsg, since it is by MySQL standards to prompt corresponding binlog position does not exist. It is likely to be cleaned by MySQL server. Solution: reset canal subscribe position.
|
||||
|
||||
@@ -24,4 +24,4 @@
|
||||
* [[TableMetaTSDB]]
|
||||
* <a href="http://alibaba.github.com/canal/release.html">ReleaseNotes</a>
|
||||
* <a href="https://github.com/alibaba/canal/releases">Download</a>
|
||||
* [[FAQ]]
|
||||
* [[FAQ]] / [[常见问题解答]]
|
||||
@@ -1,69 +0,0 @@
|
||||
# Frequently Asked Quesionts
|
||||
|
||||
Q1: Why INSERT/UPDATE/DELETE is parsed as a Query or DDL?
|
||||
|
||||
A1: Generally, it's because the binlog is a Query event. For example:
|
||||
|
||||
1. Binlog format is configured as statement mode or mixed mode . You could check with `show variables like 'binlog_format' ` . For the statement or mixed mode, DML will exist as a SQL.
|
||||
2. For MySQL5.6+ binlog row mode, there is a bool variable for DML to control the record of raw SQL: binlog-rows-query-log-events=true (check by `show variables`). Corresponding event for binlog is RowsQueryLogEvent, also keeping row records. Via properties settings in ps.canal, users could set filters: `canal.instance.filter.query.dml=true`
|
||||
|
||||
-------------------
|
||||
|
||||
Q2: I set filters for the table but it doesn't work.
|
||||
|
||||
A2:
|
||||
|
||||
1. Please step to [AdminGuide](https://github.com/alibaba/canal/wiki/AdminGuide) for the details of canal.instance.filter.regex.
|
||||
|
||||
```
|
||||
MySQL parses the table and Perl supports regex.
|
||||
Use comma(,) for multiple regex, double backslash(\\) for escape characters.
|
||||
Examples:
|
||||
1. All the tables: .* or .*\\..*
|
||||
2. All the tables in canal scheme: canal\\..*
|
||||
3. Tables starting with canal in canal scheme: canal\\.canal.*
|
||||
4. Access a table in canal scheme: canal.test1
|
||||
5. A combination of multiple rules: canal\\..*, mysql.test1, mysql.test2 (with comma as delimeter)
|
||||
```
|
||||
|
||||
2. Check with the format of binlog. Filters will be valid only for data in row mode. (Mixed/statement doesn't parse SQL and thus cannot exactly extract tableName for filtering.)
|
||||
|
||||
3. Check whether CanalConnectors have called subscribe(filter) method. If so, filters should be consistent with the canal.instance.filter.regex in instance.properties, otherwise filters of subscribe will overwrite the settings in the instance. Please NOTE that, if the filter for subscribe is .\*/..\*, then you have consumed all the data updated.
|
||||
|
||||
4. For canal version 1.1.3+, it will log the most recent filter condition. You can see if the filter is as expected, if not, please check with Step 3
|
||||
|
||||
```
|
||||
c.a.o.canal.parse.inbound.mysql.dbsync.LogEventConvert - --> init table filter : ^.*\..*$
|
||||
c.a.o.canal.parse.inbound.mysql.dbsync.LogEventConvert - --> init table black filter :
|
||||
```
|
||||
|
||||
5. See the [Issues](https://github.com/alibaba/canal/issues?utf8=%E2%9C%93&q=subscribe). Some of the problems have already been solved, such as double backslash problems and expression incorrectness.
|
||||
|
||||
---------------
|
||||
|
||||
Q3: Does canal support binlog subscribe of aliyun rds?
|
||||
|
||||
A3: Yes. See Issue [#727](https://github.com/alibaba/canal/issues/727).
|
||||
|
||||
-----------------
|
||||
|
||||
Q4: How is the overall performance of canal?
|
||||
|
||||
A4: See Issue [#726](<https://github.com/alibaba/canal/issues/726) and [Performance Documents](https://github.com/alibaba/canal/wiki/Performance).
|
||||
|
||||
----------------
|
||||
|
||||
Q5: “batchId:73 is not the firstly:72" or "clientId:1001 batchId:50560 is not exist , please check”
|
||||
|
||||
A5:
|
||||
|
||||
1. First problem is with the client ack: the first two exceptions are caused by improper ack sequence to batchId.
|
||||
2. The second problem is that batchId of ack got cleaned by the server. There are only two reasons for server clean:
|
||||
- client has set up a rollback
|
||||
- server has launched an instance restart (e.g. scan=true causes automatic restart when file changed)
|
||||
|
||||
--------
|
||||
|
||||
Q6: "Received error packet: errno = 1236, sqlstate = HY000 errmsg = Could not find first log file name in binary log index file"
|
||||
|
||||
A6: This is a dump exception. We'd suggest google errno/errmsg, since it is by MySQL standards to prompt corresponding binlog position does not exist. It is likely to be cleaned by MySQL server. Solution: reset canal subscribe position.
|
||||
68
常见问题解答.md
Normal file
68
常见问题解答.md
Normal file
@@ -0,0 +1,68 @@
|
||||
问1：INSERT/UPDATE/DELETE被解析为Query或DDL语句？
|
||||
|
||||
答1：
|
||||
出现这类情况主要原因为收到的binlog就为Query事件，比如:
|
||||
1. binlog格式为非row模式，通过show variables like 'binlog\_format'可以查看. 针对statement/mixed模式，DML语句都会是以SQL语句存在
|
||||
2. mysql5.6+之后，在binlog为row模式下，针对DML语句通过一个开关(binlog-rows-query-log-events=true, show variables里也可以看到该变量)，记录DML的原始SQL，对应binlog事件为RowsQueryLogEvent，同时也有对应的row记录. ps. canal可以通过properties设置来过滤：canal.instance.filter.query.dml = true
|
||||
|
||||
|
||||
---
|
||||
|
||||
|
||||
问2：设置了数据表的过滤条件，但貌似没有生效？
|
||||
|
||||
答2：
|
||||
1. 首先看文档[AdminGuide](AdminGuide)，了解<span data-type="color" style="color:rgb(36, 41, 46)"><span data-type="background" style="background-color:rgb(255, 255, 255)">canal.instance.filter.regex的书写格式</span></span>
|
||||
```plain
|
||||
mysql 数据解析关注的表，Perl正则表达式.
|
||||
多个正则之间以逗号(,)分隔，转义符需要双斜杠(\\)
|
||||
常见例子：
|
||||
1. 所有表：.* or .*\\..*
|
||||
2. canal schema下所有表： canal\\..*
|
||||
3. canal下的以canal打头的表：canal\\.canal.*
|
||||
4. canal schema下的一张表：canal.test1
|
||||
5. 多个规则组合使用：canal\\..*,mysql.test1,mysql.test2 (逗号分隔)
|
||||
```
|
||||
2. 检查binlog格式，过滤条件只针对row模式的数据有效(ps. mixed/statement因为不解析sql，所以无法准确提取tableName进行过滤)
|
||||
3. 检查下CanalConnector是否调用subscribe(filter)方法；有的话，filter需要和instance.properties的canal.instance.filter.regex一致，否则subscribe的filter会覆盖instance的配置，如果subscribe的filter是.\*\\..\*，那么相当于你消费了所有的更新数据 <span data-type="color" style="color:#F5222D">【特别注意】</span>
|
||||
4. canal 1.1.3+版本之后,会在日志里记录最后使用的filter条件，可以对比使用的filter看看是否和自己期望的是一致，如果不一致检查一下第3步
|
||||
```plain
|
||||
c.a.o.canal.parse.inbound.mysql.dbsync.LogEventConvert - --> init table filter : ^.*\..*$
|
||||
c.a.o.canal.parse.inbound.mysql.dbsync.LogEventConvert - --> init table black filter :
|
||||
```
|
||||
5. 检查一下[历史的issue列表](https://github.com/alibaba/canal/issues?utf8=%E2%9C%93&q=subscribe)，很有可能你的提问别人已经遇到过并解决了，比如表达式不对，特别是双斜杠的问题
|
||||
|
||||
---
|
||||
|
||||
问3：canal是否支持aliyun rds的binlog订阅？
|
||||
|
||||
答3： 参考issue：[https://github.com/alibaba/canal/issues/727](https://github.com/alibaba/canal/issues/727)
|
||||
|
||||
|
||||
---
|
||||
|
||||
问4：canal的整体性能怎么样？
|
||||
|
||||
答4：
|
||||
* 参考issue: [https://github.com/alibaba/canal/issues/726](https://github.com/alibaba/canal/issues/726)
|
||||
* 参考性能测试文档：[https://github.com/alibaba/canal/wiki/Performance](https://github.com/alibaba/canal/wiki/Performance)
|
||||
|
||||
|
||||
---
|
||||
|
||||
|
||||
问5： 出现batchId:73 is not the firstly:72 或 clientId:1001 batchId:50560 is not exist , please check
|
||||
|
||||
答5：
|
||||
1. 第一个原因是client在ack的问题，前两个异常是client没有按照顺序ack对应的batchId
|
||||
2. 最后一个是ack的batchId在服务端被清理了ps. 服务端发生清理只有两个原因：
|
||||
* client发起过一次rollback
|
||||
* server端发生过一次instance的重启，比如scan=true时发现文件变化自动restart了
|
||||
|
||||
|
||||
---
|
||||
|
||||
|
||||
问6：Received error packet: errno = 1236, sqlstate = HY000 errmsg = Could not find first log file name in binary log index file
|
||||
|
||||
答6: 出现dump异常时，建议把errno/errmsg到google上搜索一下，这个异常是mysql标准返回的，代表对应的binlog位点在mysql里不存在，应该是被mysql server清理掉了，解决办法：重置一下canal订阅位点
|
||||
Reference in New Issue
Block a user