GKE Labels are not filled out correctly

[Algirdyz]

Environment details

• OS: GKE 1.20
• Node.js version: node:14-alpine
• npm version: 7.16.0
• @google-cloud/logging-winston version: 5.1.0

Steps to reproduce

I have written a simple application to test my logging issues throughout my nodejs based kubernetes apps.

const { LoggingWinston } = require("@google-cloud/logging-winston");
const winston = require("winston");

const logger = winston.createLogger({
    level: 'info',
    transports: [],
});

logger.add(new LoggingWinston({
    serviceContext: 'nodejs-test-app',
    redirectToStdout: true, // This is the variable I try to change to see different results
}));



setInterval(() => {
    logger.info('Hello, World!');
    logger.error('TestError!')
}, 2000);

If I set the redirectToStdout to true, I get the following log in logexplorer:

{
  "insertId": ".........YTaS7Z9xmO879ASpoCgZ3mx",
  "jsonPayload": {
    "message": {
      "message": "Hello, World!",
      "metadata": {}
    },
    "logName": "projects/{{projectId}}/logs/winston_log",
    "timestamp": "2022-06-29T12:57:46.648Z"
  },
  "resource": {
    "type": "k8s_container",
    "labels": {
      "container_name": "nodejs-test-app",
      "location": "europe-west4-c",
      "pod_name": "nodejs-test-app-b67dd5478-v6d6j",
      "project_id": "REDACTED",
      "cluster_name": "REDACTED",
      "namespace_name": "default"
    }
  },
  "timestamp": "2022-06-29T12:57:46.648969018Z",
  "severity": "INFO",
  "labels": {
    "k8s-pod/app": "nodejs-test-app",
    "compute.googleapis.com/resource_name": "REDACTED-node-pool-9d6cfdf5-q8ek",
    "k8s-pod/app_kubernetes_io/managed-by": "gcp-cloud-build-deploy",
    "k8s-pod/pod-template-hash": "b67dd5478"
  },
  "logName": "projects/REDACTED/logs/stdout",
  "receiveTimestamp": "2022-06-29T12:57:48.341112484Z"
}

This looks OK'ish, but as mentioned in another issue (https://github.com/googleapis/nodejs-logging-winston/issues/704) message has an inner field. Looking at this log inside GKE GUI shows an unexpandable entry like this:

In order to see the full message I have to go into the logs of the specific container and navigate away from GKE GUI in the console.

If, on the other hand I set redirectToStdout to false. Log entry in log explorer becomes like this:

{
  "insertId": "........0TyXbYD7xUlAOt3aVr1L.wAT",
  "jsonPayload": {
    "message": "Hello, World!",
    "metadata": {}
  },
  "resource": {
    "type": "k8s_container",
    "labels": {
      "location": "europe-west4-c",
      "container_name": "undefined",
      "namespace_name": "default",
      "cluster_name": "REDACTED",
      "pod_name": "nodejs-test-app-6989f5875d-bwmp9",
      "project_id": "REDACTED"
    }
  },
  "timestamp": "2022-06-29T12:53:31.634000062Z",
  "severity": "INFO",
  "logName": "projects/REDACTED/logs/winston_log",
  "receiveTimestamp": "2022-06-29T12:53:31.688181632Z"
}

As a result of missing labels, this log no longer shows up in GKE GUI at all, and even navigating to container logs from there does not yield any results. The only way to find it was to search by logname in log explorer logName="projects/REDACTED/logs/winston_log"

What is wrong with my setup? This is the simplest implementation I could do and replicated it, on several of my GKE clusters in every nodejs app. Python applications still log fine. It also used to work with nodejs but stopped at some point.

[mattnworb]

I had a similar problem, and noticed that the underlying @google-cloud/logging library has this note in the code that queries the metadata server to get labels:

https://github.com/googleapis/nodejs-logging/blob/eda96618d95938cf4c53682c78bb5ada86a54f87/src/utils/metadata.ts#L164-L166

[losalex]

@Algirdyz , thanks for opening the issue and really sorry for late response! As for you comment regarding extra message wrapper in payload - recently we added support for useMessageField config parameter which can be set to false to eliminate extra message wrapper. Of course it will work only when you set redirectToStdout to true. I will look for labels issue, but just want to confirm - you mentioned that labels worked correctly before, right?

[losalex]

@Algirdyz , I am going to close this issue - feel free to reopen if you have more questions/comments.