我们线上的tendis集群运行一段时间就特别的慢,简单的get/set命令都要达到10几秒,有人能帮忙解决一下吗?

[zhaochunxue]

是最新版本的tendis2.3.1

[zhaochunxue]

这是我们的配置文件信息,磁盘是nvme的,目前磁盘读写压力并不大,error中没有异常日志,就是很慢很慢,慢到客户端一直超时

1. "aof-enabled"
2. "no"
3. "aof-psync-num"
4. "500"
5. "bind"
6. ""0.0.0.0""
7. "binlog-send-batch"
8. "256"
9. "binlog-send-bytes"
10. "16777216"
11. "binlog-using-defaultcf"
12. "no"
13. "binlogdelrange"
14. "1"
15. "binlogfilesecs"
16. "1200"
17. "binlogfilesizemb"
18. "64"
19. "binlogratelimitmb"
20. "64"
21. "checkkeytypeforsetcmd"
22. "no"
23. "chunksize"
24. "16384"
25. "cluster-enabled"
26. "yes"
27. "cluster-migration-barrier"
28. "1"
29. "cluster-migration-binlog-iters"
30. "10"
31. "cluster-migration-distance"
32. "10000"
33. "cluster-migration-rate-limit"
34. "32"
35. "cluster-migration-slots-num-per-task"
36. "10"
37. "cluster-node-timeout"
38. "30000"
39. "cluster-require-full-coverage"
40. "yes"
41. "cluster-single-node"
42. "no"
43. "cluster-slave-no-failover"
44. "no"
45. "cluster-slave-validity-factor"
46. "10"
47. "compactrange-after-deleterange"
48. "no"
49. "daemon"
50. "no"
51. "databases"
52. "16"
53. "delcntindexmgr"
54. "10000"
55. "deljobcntindexmgr"
56. "1"
57. "dir"
58. ""./store/db""
59. "domain-enabled"
60. "no"
61. "dumpdir"
62. ""./store/dump""
63. "executorthreadnum"
64. "16"
65. "executorworkpoolsize"
66. "8"
67. "fullpushthreadnum"
68. "4"
69. "fullreceivethreadnum"
70. "4"
71. "garbage-delete-size"
72. "30"
73. "garbagedeletethreadnum"
74. "1"
75. "generallog"
76. "no"
77. "incrpushthreadnum"
78. "4"
79. "jeprof-auto-dump"
80. "yes"
81. "keysdefaultlimit"
82. "100"
83. "kvstorecount"
84. "10"
85. "lockdbxwaittimeout"
86. "1"
87. "lockwaittimeout"
88. "3600"
89. "logdir"
90. ""./store/log""
91. "loglevel"
92. ""notice""
93. "logrecyclethreadnum"
94. "4"
95. "lua-time-limit"
96. "5000"
97. "luastatemaxidletime"
98. "3600000"
99. "masterauth"
100. ""YWExYTdmZmRlMGZk""
101. "maxbinlogkeepnum"
102. "1"
103. "maxclients"
104. "10000"
105. "migrate-gc-enabled"
106. "yes"
107. "migrate-snapshot-key-num"
108. "100000"
109. "migrate-snapshot-retry-num"
110. "1000"
111. "migratereceivethreadnum"
112. "4"
113. "migratesenderthreadnum"
114. "4"
115. "minbinlogkeepsec"
116. "3600"
117. "netbatchsize"
118. "1048576"
119. "netbatchtimeoutsec"
120. "10"
121. "netiothreadnum"
122. "8"
123. "noexpire"
124. "no"
125. "pausetimeindexmgr"
126. "10"
127. "pidfile"
128. ""./store/tendisplus.pid""
129. "port"
130. "6501"
131. "proto-max-bulk-len"
132. "536870912"
133. "requirepass"
134. ""YWExYTdmZmRlMGZk""
135. "rocks.blockcache_num_shard_bits"
136. "6"
137. "rocks.blockcache_strict_capacity_limit"
138. "no"
139. "rocks.blockcachemb"
140. "20971"
141. "rocks.compress_type"
142. ""snappy""
143. "rocks.disable_wal"
144. "no"
145. "rocks.flush_log_at_trx_commit"
146. "no"
147. "rocks.level0_compress_enabled"
148. "no"
149. "rocks.level1_compress_enabled"
150. "yes"
151. "rocks.wal_dir"
152. """"
153. "scancntindexmgr"
154. "1000"
155. "scandefaultlimit"
156. "10"
157. "scandefaultmaxiteratetimes"
158. "10000"
159. "scanjobcntindexmgr"
160. "1"
161. "slave-migrate-enabled"
162. "no"
163. "slavebinlogkeepnum"
164. "1"
165. "slowlog"
166. ""./store/log/slowlog""
167. "slowlog-file-enabled"
168. "yes"
169. "slowlog-flush-interval"
170. "1000"
171. "slowlog-log-slower-than"
172. "100000"
173. "slowlog-max-len"
174. "128"
175. "storage"
176. ""rocks""
177. "timeoutsecbinlogwaitrsp"
178. "30"
179. "truncatebinlogintervalms"
180. "1000"
181. "truncatebinlognum"
182. "50000"
183. "version-increase"
184. "yes"

[TendisDev]

你好，分析慢查询的方式

1. 判断当时系统负载
2. 判断慢查询的时间戳是否集中，慢查询的比例如何
3. 分析rocksdb日志，搜索compaction相关日志，看下在慢查询对应时间是否存在大量compaction
4. 通过monitor分析是否存在大key

另外，尝试修改这个参数 binlogdelrange 10000