From 021a4474da4c192c1dbb66796498e2bdb22b536d Mon Sep 17 00:00:00 2001 From: lcybo Date: Sat, 18 Aug 2018 11:50:32 +0800 Subject: [PATCH] Updated Canal prometheus (markdown) --- Canal-prometheus.md | 75 ++++++++++++++++++++++++--------------------- 1 file changed, 40 insertions(+), 35 deletions(-) diff --git a/Canal-prometheus.md b/Canal-prometheus.md index b8a9db6..be1a720 100644 --- a/Canal-prometheus.md +++ b/Canal-prometheus.md @@ -35,7 +35,7 @@ Canal server 性能指标监控基于prometheus的实现。 |Basic|Canal instance 基本信息。|是| |Network bandwith|网络带宽。包含inbound(binlog读取)和outbound(向client输出)|是| |Delay|Canal instance 相对于master的延时。|否| -|Idle|sink线程idle占比；dump线程idle占比(仅parallel mode)。|是| +|[Idle](https://github.com/alibaba/canal/wiki/Canal-prometheus/_edit#idle)|sink线程idle占比；dump线程idle占比(仅parallel mode)。|是| |TPS|Canal client instance消费binlog的TPS， 以MySQL binlog events为单位计算。|否| |TPS(transaction)|Canal instance 处理binlog的TPS，以MySQL transaction为单位计算。|否| |TPS(rows VS rowData)|Canal instance处理binlog的TPS，table rows：以table变更行为单位。row data：rowData events为单位。|是| @@ -67,7 +67,7 @@ clamp_max(rate(canal_instance_publish_blocking_time{destination="example"}[2m]), **1. Sinking idle ratio也很高，则瓶颈是因为client的consume速度相对较慢。** **2. Sinking idle ratio较低，那么server端parser是性能瓶颈，可参考[Performance](https://github.com/alibaba/canal/wiki/Performance)进行tuning.** - +*** ##### Delay(seconds) ![Image text](https://raw.githubusercontent.com/lcybo/canal/master/images/delay.PNG) @@ -79,7 +79,7 @@ canal_instance_traffic_delay / 1000 **1. 如果Canal已经消费至最新position，且binlog长时间未更新，delay的resolution受到master_heartbeat_period的影响，目前频率为15秒。** **2. Limitations: 如果store满了(恰好store中的数据包含最新的position，且MySQL binlog停止更新)，且client暂停消费，那么delay会不断增长。当然，满足这些条件的概率极低。** - +*** ##### 网络带宽(KB/s) ![Image text](https://github.com/lcybo/canal/blob/master/images/network.PNG?raw=true) @@ -91,80 +91,85 @@ rate(canal_instance_received_binlog_bytes{destination="example"}[2m]) / 1024 rate(canal_instance_client_bytes{destination="example"}[2m]) / 1024 ``` **Instance client订阅后消费所占用的带宽。** - - +*** ##### TPS(Binlog events) - +![Image text](https://github.com/lcybo/canal/blob/master/images/TPS.PNG?raw=true) ``` rate(canal_instance_store_consume_seq{destination="example"}[2m]) ``` **Canal instance client消费TPS(以binlog event为单位)，即canal instance最终的整体性能。** +*** - -* TPS(MySQL transaction) +##### TPS(MySQL transaction) +![Image text](https://github.com/lcybo/canal/blob/master/images/transactions.PNG?raw=true) ``` rate(canal_instance_transactions{destination="example"}[2m]) ``` **Canal instance处理transaction的TPS，以TRANSACTION_END事件为基准。** +*** - -* TPS(Table row) +##### TPS(Table row) +![Image text](https://github.com/lcybo/canal/blob/master/images/rows.PNG?raw=true) ``` rate(canal_instance_rows_counter{destination="example"}[2m]) ``` **Canal instance接受binlog中，数据库表变更行数的TPS.** - - -* DML model indicator ``` -rate(canal_instance_rows_counter{destination="example"}[15s]) / rate(canal_instance_row_events{destination="example"}[15s]) +rate(canal_instance_row_events{destination="example"}[2m]) ``` -**RowData events包含行数。OLTP下，RowData大体上仅包含1行变更；批量DML下，该指标会上行。** +**Canal instance接受binlog中，rowData类型events的TPS.** +**批量DML的场景下，rowData会包含多个table rows, tableRows指标会显著高于rowDatas.** +*** - -* RPS(Client request) +##### Client requests +![Image test](https://github.com/lcybo/canal/blob/master/images/reqs.PNG?raw=true) ``` -rate(canal_instance_client_packets{destination="example"}[2m]) +canal_instance_client_packets{destination="example"} ``` -**Netty server处理client requests吞吐量。** +**Netty server处理的client requests，以packetType为label分类统计。** -* QPS(空包) +##### Empty packets +![Image text](https://github.com/lcybo/canal/blob/master/images/empty.PNG?raw=true) ``` rate(canal_instance_client_empty_batches{destination="example"}[2m]) ``` **client get返回每秒空包量。如果正常traffic下，该值很大，考虑使用connector的timeout机制，节省资源。** - - -* Client latency - -_experimental_ ``` -canal_instance_client_request_latency_bucket +rate(canal_instance_client_packets{destination="example", packetType="GET"}[2m]) ``` -**Histogram, client请求响应情况。关于[histogram](https://prometheus.io/docs/concepts/metric_types/#histogram).** +**作为empty rate的参照。** + +##### Client latency +![Image text](https://github.com/lcybo/canal/blob/master/images/latency.PNG?raw=true) +``` +canal_instance_client_request_latency_bucket{destination="example"} +``` +**Histogram, client请求响应情况统计。关于[histogram](https://prometheus.io/docs/concepts/metric_types/#histogram).** -* Event store占用 +##### Event store占用 +![Image text](https://github.com/lcybo/canal/blob/master/images/remain_events.PNG?raw=true) ``` -canal_instance_store_produce_seq - canal_instance_store_consume_seq +canal_instance_store_produce_seq{destination="example"} - canal_instance_store_consume_seq{destination="example"} ``` **Event store内未ack的events数量，实时性受scrape_interval影响。** -* Event store memory占用(KB, 仅memory mode) +##### Event store memory占用(KB, 仅memory mode) +![Image text](https://github.com/lcybo/canal/blob/master/images/remain_mem.PNG?raw=true) ``` -(canal_instance_store_produce_mem - canal_instance_store_consume_mem) / 1024 +(canal_instance_store_produce_mem{destination="example"} - canal_instance_store_consume_mem{destination="example"}) / 1024 ``` **Event store内未ack的events所占用内存大小，实时性受scrape_interval影响。** -* 状态信息 +##### 状态信息 ``` -canal_instance -canal_instance_parser_mode -canal_instance_store +canal_instance{destination="example"} +canal_instance_parser_mode{destination="example"} +canal_instance_store{destination="example"} ``` **通过labels展示状态信息。**