improve support for structured logging (Json and possibly other formats)

[justcoon]

in current implementation we have json logger however there is not way have to do case class logging to json in structured way

there is also limitation in relation to ZIO#log* interface which only supports strings, zio core log annotation also supporting just string values in relation to that currently it could be possible to do

object ConsoleJsonApp extends ZIOAppDefault {
  
  case class User(id: UUID, name: String) {
    def toJson = s"""{ "id": "$id", "name": "$name"}""".stripMargin
  }
  private val userLogAnnotation = LogAnnotation[User]("user", (_, i) => i, _.toJson)

  override val bootstrap: ZLayer[ZIOAppArgs, Any, Any] =
    Runtime.removeDefaultLoggers >>> consoleJson(
      LogFormat.default + LogFormat.annotation(LogAnnotation.TraceId) + LogFormat.annotation(
        userLogAnnotation
      )
    )

  private val users = List.fill(2)(UUID.randomUUID())

  override def run: ZIO[Scope, Any, ExitCode] =
    (for {
      traceId <- ZIO.succeed(UUID.randomUUID())
      _       <- ZIO.foreachPar(users) { uId =>
                   {
                     ZIO.logInfo("Starting operation") *>
                       ZIO.sleep(500.millis) *>
                       ZIO.logInfo("Stopping operation")
                   } @@ userLogAnnotation(User(uId, "aaa"))
                 } @@ LogAnnotation.TraceId(traceId)
      _       <- ZIO.logInfo("Done")
    } yield ExitCode.success)
}

however current json logger is not able recognise json string value

{"timestamp":"2023-01-26T23:39:19.142999+01:00","level":"INFO","thread":"zio-fiber-5","message":"Starting operation","trace_id":"01087a2a-a46c-4ff5-b4de-1c3c680b55a6","user":"{ \"id\": \"d894df3a-265f-4fc2-8abf-1808e52b5c4f\", \"name\": \"aaa\"}"}
{"timestamp":"2023-01-26T23:39:19.143007+01:00","level":"INFO","thread":"zio-fiber-6","message":"Starting operation","trace_id":"01087a2a-a46c-4ff5-b4de-1c3c680b55a6","user":"{ \"id\": \"3e9a023e-9f4c-4ebb-aa9d-e231c8e17400\", \"name\": \"aaa\"}"}

this could be fixed, but it is probably not very safe and future proof

also other thing which we could consider is support for zio-schema, which then could allow different structured output based on given schema

we need to analyze the possibilities how to add this functionality and make it more user friendly

[justcoon]

probably we should do it in phases:

1. modify current json logger to able recognise json string value and add it like json value
2. analyse the possibilities how do structured logging and add better support for structured logging, consider usage of zio-schema

[jdegoes]

To do (1), we should create a high-performance, zero-allocation JSON recognizer (String => Boolean), which looks through a string and returns true / false depending on whether it contains valid JSON.

If it contains valid JSON, then it can be inserted as-is, without the need to wrap it into a String and escape the contents.

If it does NOT contain valid JSON, then the current behavior should occur.

I think we can add logStructured("message", myValue) which uses ZIO Schema to log myValue in a structured way, by rendering to JSON and then passing to underlying.

[ipetkovic]

I could come up with something regarding (1), have something ready to show next week. Do we want strict json validation? Regarding (2), does it make sense to have support for structured logging of annotations as well? For example, if I write something like: logInfo("mesage") @@ myannotation(myValue) that myValue is logged in structured way.

[ipetkovic]

Could you have a look at the https://github.com/zio/zio-logging/pull/657 ? I also put some remarks/concerns in comment of the PR. I am happy to hear any feedback. Thanks!

[ipetkovic]

Once again, updated https://github.com/zio/zio-logging/pull/657 Can you have a look?