chenjl/canal
1
0
Fork 0
mirror of https://github.com/alibaba/canal.git synced 2026-11-04 00:30:15 +08:00
Code Issues Packages Projects Releases Wiki Activity

Updated Canal prometheus (markdown)

lcybo
2018-08-18 11:50:32 +08:00
parent a39d5d32e6
commit 021a4474da
1 changed files with 40 additions and 35 deletions
Download Patch File Download Diff File Expand all files Collapse all files

75
Canal-prometheus.md

@@ -35,7 +35,7 @@ Canal server 性能指标监控基于prometheus的实现。
|Basic|Canal instance 基本信息。|是| |Basic|Canal instance 基本信息。|是|
|Network bandwith|网络带宽。包含inbound(binlog读取)和outbound(向client输出)|是| |Network bandwith|网络带宽。包含inbound(binlog读取)和outbound(向client输出)|是|
|Delay|Canal instance 相对于master的延时。|否| |Delay|Canal instance 相对于master的延时。|否|
|Idle|sink线程idle占比；dump线程idle占比(仅parallel mode)。|是| |[Idle](https://github.com/alibaba/canal/wiki/Canal-prometheus/_edit#idle)|sink线程idle占比；dump线程idle占比(仅parallel mode)。|是|
|TPS|Canal client instance消费binlog的TPS， 以MySQL binlog events为单位计算。|否| |TPS|Canal client instance消费binlog的TPS， 以MySQL binlog events为单位计算。|否|
|TPS(transaction)|Canal instance 处理binlog的TPS，以MySQL transaction为单位计算。|否| |TPS(transaction)|Canal instance 处理binlog的TPS，以MySQL transaction为单位计算。|否|
|TPS(rows VS rowData)|Canal instance处理binlog的TPS，table rows：以table变更行为单位。row data：rowData events为单位。|是| |TPS(rows VS rowData)|Canal instance处理binlog的TPS，table rows：以table变更行为单位。row data：rowData events为单位。|是|
@@ -67,7 +67,7 @@ clamp_max(rate(canal_instance_publish_blocking_time{destination="example"}[2m]),
**1. Sinking idle ratio也很高，则瓶颈是因为client的consume速度相对较慢。** **1. Sinking idle ratio也很高，则瓶颈是因为client的consume速度相对较慢。**
**2. Sinking idle ratio较低，那么server端parser是性能瓶颈，可参考[Performance](https://github.com/alibaba/canal/wiki/Performance)进行tuning.** **2. Sinking idle ratio较低，那么server端parser是性能瓶颈，可参考[Performance](https://github.com/alibaba/canal/wiki/Performance)进行tuning.**
***
##### Delay(seconds) ##### Delay(seconds)
![Image text](https://raw.githubusercontent.com/lcybo/canal/master/images/delay.PNG) ![Image text](https://raw.githubusercontent.com/lcybo/canal/master/images/delay.PNG)
@@ -79,7 +79,7 @@ canal_instance_traffic_delay / 1000
**1. 如果Canal已经消费至最新position，且binlog长时间未更新，delay的resolution受到master_heartbeat_period的影响，目前频率为15秒。** **1. 如果Canal已经消费至最新position，且binlog长时间未更新，delay的resolution受到master_heartbeat_period的影响，目前频率为15秒。**
**2. Limitations: 如果store满了(恰好store中的数据包含最新的position，且MySQL binlog停止更新)，且client暂停消费，那么delay会不断增长。当然，满足这些条件的概率极低。** **2. Limitations: 如果store满了(恰好store中的数据包含最新的position，且MySQL binlog停止更新)，且client暂停消费，那么delay会不断增长。当然，满足这些条件的概率极低。**
***
##### 网络带宽(KB/s) ##### 网络带宽(KB/s)
![Image text](https://github.com/lcybo/canal/blob/master/images/network.PNG?raw=true) ![Image text](https://github.com/lcybo/canal/blob/master/images/network.PNG?raw=true)
@@ -91,80 +91,85 @@ rate(canal_instance_received_binlog_bytes{destination="example"}[2m]) / 1024
rate(canal_instance_client_bytes{destination="example"}[2m]) / 1024 rate(canal_instance_client_bytes{destination="example"}[2m]) / 1024
``` ```
**Instance client订阅后消费所占用的带宽。** **Instance client订阅后消费所占用的带宽。**
***
##### TPS(Binlog events) ##### TPS(Binlog events)
![Image text](https://github.com/lcybo/canal/blob/master/images/TPS.PNG?raw=true)
``` ```
rate(canal_instance_store_consume_seq{destination="example"}[2m]) rate(canal_instance_store_consume_seq{destination="example"}[2m])
``` ```
**Canal instance client消费TPS(以binlog event为单位)，即canal instance最终的整体性能。** **Canal instance client消费TPS(以binlog event为单位)，即canal instance最终的整体性能。**
***
##### TPS(MySQL transaction)
* TPS(MySQL transaction) ![Image text](https://github.com/lcybo/canal/blob/master/images/transactions.PNG?raw=true)
``` ```
rate(canal_instance_transactions{destination="example"}[2m]) rate(canal_instance_transactions{destination="example"}[2m])
``` ```
**Canal instance处理transaction的TPS，以TRANSACTION_END事件为基准。** **Canal instance处理transaction的TPS，以TRANSACTION_END事件为基准。**
***
##### TPS(Table row)
* TPS(Table row) ![Image text](https://github.com/lcybo/canal/blob/master/images/rows.PNG?raw=true)
``` ```
rate(canal_instance_rows_counter{destination="example"}[2m]) rate(canal_instance_rows_counter{destination="example"}[2m])
``` ```
**Canal instance接受binlog中，数据库表变更行数的TPS.** **Canal instance接受binlog中，数据库表变更行数的TPS.**
* DML model indicator
``` ```
rate(canal_instance_rows_counter{destination="example"}[15s]) / rate(canal_instance_row_events{destination="example"}[15s]) rate(canal_instance_row_events{destination="example"}[2m])
``` ```
**RowData events包含行数。OLTP下，RowData大体上仅包含1行变更；批量DML下，该指标会上行。** **Canal instance接受binlog中，rowData类型events的TPS.**
**批量DML的场景下，rowData会包含多个table rows, tableRows指标会显著高于rowDatas.**
***
##### Client requests
* RPS(Client request) ![Image test](https://github.com/lcybo/canal/blob/master/images/reqs.PNG?raw=true)
``` ```
rate(canal_instance_client_packets{destination="example"}[2m]) canal_instance_client_packets{destination="example"}
``` ```
**Netty server处理client requests吞吐量。** **Netty server处理的client requests，以packetType为label分类统计。**
* QPS(空包) ##### Empty packets
![Image text](https://github.com/lcybo/canal/blob/master/images/empty.PNG?raw=true)
``` ```
rate(canal_instance_client_empty_batches{destination="example"}[2m]) rate(canal_instance_client_empty_batches{destination="example"}[2m])
``` ```
**client get返回每秒空包量。如果正常traffic下，该值很大，考虑使用connector的timeout机制，节省资源。** **client get返回每秒空包量。如果正常traffic下，该值很大，考虑使用connector的timeout机制，节省资源。**
* Client latency
_experimental_
``` ```
canal_instance_client_request_latency_bucket rate(canal_instance_client_packets{destination="example", packetType="GET"}[2m])
``` ```
**Histogram, client请求响应情况。关于[histogram](https://prometheus.io/docs/concepts/metric_types/#histogram).** **作为empty rate的参照。**
##### Client latency
![Image text](https://github.com/lcybo/canal/blob/master/images/latency.PNG?raw=true)
```
canal_instance_client_request_latency_bucket{destination="example"}
```
**Histogram, client请求响应情况统计。关于[histogram](https://prometheus.io/docs/concepts/metric_types/#histogram).**
* Event store占用 ##### Event store占用
![Image text](https://github.com/lcybo/canal/blob/master/images/remain_events.PNG?raw=true)
``` ```
canal_instance_store_produce_seq - canal_instance_store_consume_seq canal_instance_store_produce_seq{destination="example"} - canal_instance_store_consume_seq{destination="example"}
``` ```
**Event store内未ack的events数量，实时性受scrape_interval影响。** **Event store内未ack的events数量，实时性受scrape_interval影响。**
* Event store memory占用(KB, 仅memory mode) ##### Event store memory占用(KB, 仅memory mode)
![Image text](https://github.com/lcybo/canal/blob/master/images/remain_mem.PNG?raw=true)
``` ```
(canal_instance_store_produce_mem - canal_instance_store_consume_mem) / 1024 (canal_instance_store_produce_mem{destination="example"} - canal_instance_store_consume_mem{destination="example"}) / 1024
``` ```
**Event store内未ack的events所占用内存大小，实时性受scrape_interval影响。** **Event store内未ack的events所占用内存大小，实时性受scrape_interval影响。**
* 状态信息 ##### 状态信息
``` ```
canal_instance canal_instance{destination="example"}
canal_instance_parser_mode canal_instance_parser_mode{destination="example"}
canal_instance_store canal_instance_store{destination="example"}
``` ```
**通过labels展示状态信息。** **通过labels展示状态信息。**