apache
optimize: Optimize logging collection config
- [ ] I have registered the PR changes.
Ⅰ. Describe what this PR did
优化日志采集配置，将日志采集相关的配置抽取，交给Spring Environment管理
Ⅱ. Does this pull request fix one issue?
fixes #4648
Ⅲ. Why don't you add test cases (unit test/integration test)?
已经添加部分单元测试，但是Kafka测试依赖kafka环境，连接不到Kafka服务会导致单元测试失败，为了避免对整体测试结果影响，暂时先删除kafka相关测试
Ⅳ. Describe how to verify it
搭建logstash和kafka服务,修改seata-server项目application.yml
logging:
config: classpath:logback-spring.xml
file:
path: ${user.home}/logs/seata
extend:
logstash-appender:
destination: 127.0.0.1:4560 # 修改为logstash服务地址
kafka-appender:
bootstrap-servers: 127.0.0.1:9092 # 修改位kafka地址
topic: logback_to_logstash
# 修改kafka生产者参数
producer-configs:
acks: 0
linger:
ms: 1000
max:
block:
ms: 0
验证logstash-appender
使用如下配置启动logstash，观察控制台输出即可
input {
tcp {
port => 4560
codec => json_lines
}
}
output{
stdout {
codec => rubydebug
}
}
验证 kafka-appender
启动kafka，并且使用如下配置文件启动logstash,观察控制台输出即可
input {
kafka {
bootstrap_servers => "localhost:9092"
topics => ["logback_to_logstash"]
}
}
output{
stdout {
codec => rubydebug
}
}
Ⅴ. Special notes for reviews
logging.extend支持的全部配置示例如下
logging:
extend:
logstash-appender:
destination: 127.0.0.1:4560
pattern: '{"timestamp":"%date{yyyy-MM-dd HH:mm:ss.SSS}","level":"%p","app_name":"${spring.application.name:seata-server}","PORT":"${server.servicePort:0}","thread_name":"%t","logger_name":"%logger","X-TX-XID":"%X{X-TX-XID:-}","X-TX-BRANCH-ID":"%X{X-TX-BRANCH-ID:-}","message":"%m"}'
keep-alive-duration: 5 minutes
keep-alive-message: ping
keep-alive-charset: UTF-8
reconnection-delay: 30 seconds
connection-strategy: preferPrimary
connection-ttl: 10 seconds
write-buffer-size: 8192
write-timeout: 1 minute
connection-timeout: 5 seconds
ring-buffer-size: 8192
wait-strategy: blocking
kafka-appender:
bootstrap-servers: 127.0.0.1:9092
topic: logback_to_logstash
pattern: '{"@timestamp":"%d{yyyy-MM-dd HH:mm:ss.SSS}","level":"%p","app_name":"${spring.application.name:seata-server}","PORT":"${server.servicePort:0}","thread_name":"%t","logger_name":"%logger","X-TX-XID":"%X{X-TX-XID:-}","X-TX-BRANCH-ID":"%X{X-TX-BRANCH-ID:-}","message":"%m"}'
keying-strategy: noKey
producer-configs:
acks: 0
linger:
ms: 1000
max:
block:
ms: 0
logstash-appender 中配置项含义可参考链接https://gitee.com/mirrors/logstash-logback-encoder#tcp-appenders
logback 本身提供了 <springProperty/> 用来映射 logback 配置 与 springboot 配置，个人不是非常建议使用 Listener 这种方式去添加 appender，不是非常灵活。
appender 启停功能，可以考虑引入 logback 的 <if> 标签的扩展依赖：
<dependency>
<groupId>org.codehaus.janino</groupId>
<artifactId>janino</artifactId>
<version>${janino.version}</version>
</dependency>
将 enabled 配置映射到 springboot 中。
appender启停功能，可以考虑引入logback的<if>标签的扩展依赖：<dependency> <groupId>org.codehaus.janino</groupId> <artifactId>janino</artifactId> <version>${janino.version}</version> </dependency>将
enabled配置映射到springboot中。
appender启停功能，可以考虑引入logback的<if>标签的扩展依赖：<dependency> <groupId>org.codehaus.janino</groupId> <artifactId>janino</artifactId> <version>${janino.version}</version> </dependency>将
enabled配置映射到springboot中。
我觉得listener方式处理也不太灵活，已经按照<if>标签的配置方式进行修改，并且在原来的配置基础上添加了一些必要的配置项提供给用户进行动态修改。
我还是更倾向于直接使用listener方式来处理,因为这样用户需要关心的不再是yml+xml的方式,而是关注在yml中即可,而且对seata的维护者来说,使用listener方式对日志是拥有更高的掌控权,logback的配置掌控权在用户侧,我们只能通过文档等方式去帮助用户配置,如果使用listener方式的话,假设用户的日志输出是个"123",然后我们发送到kafka中的msg是{"msg":""123""} 这时就会出现json解析问题,如果我们用listener方式,我们可以不用用户去配置appender是哪个,我们可以直接集成kafkalogappender去对相关日志进行转义之类的
目前引入logback
<if>标签支持后，可以配合<springProperty/>标签动态的加载logback配置，seata维护者可以在xml中借助logback的<if then>和<if then else>根据用户在yml中指定的配置来动态加载标签。用户只需要按照文档指导修改相关的logging.extend配置来管理logback日志收集appender的配置。如果后续appender的提供者有新特性，使用xml+<if>标签支持也很方便，而listener方式的话需要直接操作API的方式来支持新特性，感觉不太灵活。
但是从对日志掌控权的方面考虑，listener的配置方式可能更好，虽然他不灵活，但是可以稳定的提供服务，appender的一些关键配置信息也会被更规范的管理，同时xml+
<if>配置方式使配置文件的修改成本增加，可能出现用户随意修改导致配置格式错误无法加载的问题。
xml+
<if>的方式可能配置上更加灵活，不需要通过java代码直接操作appender，但是无形增加了配置的复杂度，使logback配置文件看起来非常臃肿不好理解。 listener方式虽然配置不灵活，但是对日志掌控权高，可以规范日志采集的一些关键配置。 总的来说两种方式各有优劣，大家一起讨论一下，选择一个合适方式来管理日志采集功能
我最新的一次pr按照listener思路重新弄了一下，实现的思路和Springboot集成logback类似，如果用户xml添加了appender按照用户的xml配置为准，如果没有配置加载seata默认appender，并且透出自定义配置项供用户在yml中维护
我最新的一次pr按照listener思路重新弄了一下，实现的思路和Springboot集成logback类似，如果用户xml添加了appender按照用户的xml配置为准，如果没有配置加载seata默认appender，并且透出自定义配置项供用户在yml中维护
扫码加一下钉钉群，我们群内沟通下吧
appender这些可以先不用继承方式来做吧？
是的，现在还没有使用继承的方式来做，只是所有的appender都采用LoggingEventCompositeJsonEncoder来处理json转义，保证不会出现msg转义问题。 之前代码的appender类主要是指将appender添加到loggerContext，可能命名存在歧义，现在已经修改命名方式为AppenderProvider结尾，使用appendTo方法向loggerContext添加appender
我简单说一下当前的实现思路，当前实现的主要思路是当LoggingExtendApplicationListener监听到ApplicationEnvironmentPreparedEvent事件时，会实例化LogbackExtendConfigurator对象并且调用其中的doLoggingExtendConfiguration方法。
LogbackExtendConfigurator首先向loggerContext加载一些必要的配置信息，然后会通过seata SPI机制，获取所有实现LogbackLoggingExtendAppenderProvider的实现，调用其appendTo()方法，完成appender配置并添加到loggerContext中，。
其中涉及到的主要类及其关联关系如下：
This pull request introduces 1 alert when merging 2f81482f97d4908dcc8e2f775efe65dda9005a30 into 8d6c3ef51eb56abe46b9b8e1948109b4ac16eca9 - view on LGTM.com
new alerts:
- 1 for Dereferenced variable may be null
![lgtm-com[bot] avatar](https://avatars.githubusercontent.com/in/17324?v=4 "Published by a pub.dev verified publisher"){kind=small}
Codecov Report
Merging #4702 (c2ea9d6) into develop (b7dafbf) will increase coverage by
0.07%. The diff coverage is54.34%.
@@ Coverage Diff @@
## develop #4702 +/- ##
=============================================
+ Coverage 49.20% 49.28% +0.07%
- Complexity 4091 4148 +57
=============================================
Files 736 744 +8
Lines 25723 26091 +368
Branches 3177 3221 +44
=============================================
+ Hits 12658 12858 +200
- Misses 11716 11851 +135
- Partials 1349 1382 +33
| Impacted Files | Coverage Δ | |
|---|---|---|
| .../logging/extend/LoggingExtendAppenderProvider.java | 0.00% <0.00%> (ø) |
|
| ...g/listener/LoggingExtendLoggerContextListener.java | 0.00% <0.00%> (ø) |
|
| ...end/LogbackLoggingExtendKafkaAppenderProvider.java | 46.26% <46.26%> (ø) |
|
| .../AbstractLogbackLoggingExtendAppenderProvider.java | 48.48% <48.48%> (ø) |
|
| ...ver/logging/logback/LogbackExtendConfigurator.java | 54.66% <54.66%> (ø) |
|
| ...ing/listener/LoggingExtendApplicationListener.java | 57.14% <57.14%> (ø) |
|
| .../LogbackLoggingExtendLogstashAppenderProvider.java | 69.00% <69.00%> (ø) |
|
| .../logging/extend/LoggingExtendPropertyResolver.java | 76.00% <76.00%> (ø) |
|
| ...torage/file/store/FileTransactionStoreManager.java | 55.62% <0.00%> (-0.65%) |
:arrow_down: |
| ...erver/storage/file/session/FileSessionManager.java | 47.77% <0.00%> (-0.64%) |
:arrow_down: |
| ... and 1 more |
Thank you for your submission! We really appreciate it. Like many open source projects, we ask that you all sign our Contributor License Agreement before we can accept your contribution.
0 out of 2 committers have signed the CLA.
:x: wulingxiao
:x: lingxiao-wu
wulingxiao seems not to be a GitHub user. You need a GitHub account to be able to sign the CLA. If you have already a GitHub account, please add the email address used for this commit to your account.
You have signed the CLA already but the status is still pending? Let us recheck it.
Thank you for your submission! We really appreciate it. Like many open source projects, we ask that you all sign our Contributor License Agreement before we can accept your contribution.
0 out of 2 committers have signed the CLA.
:x: wulingxiao
:x: lingxiao-wu
wulingxiao seems not to be a GitHub user. You need a GitHub account to be able to sign the CLA. If you have already a GitHub account, please add the email address used for this commit to your account.
You have signed the CLA already but the status is still pending? Let us recheck it.
Due to code conflicts and age, if you are still interested in participating in the community, please resubmit this pr to the 2.x branch after resolving the conflict. 由于代码冲突以及年代久远，如果你还有兴趣参与到社区，请解决完冲突后重新请提交这份pr至2.x分支中