googleapis
cloud logging library consume disk space
description
recently I have "disk full" issues on my servers. the folder that takes the disk space is "tmp/systemd-private-29babb11a1c64801884d7fdf7ea53545-apache2.service-twEfbE/tmp/batch-daemon-fai lure" the content in this folder are the content from some of my logs. it took an 100 GB easily in 1 day.
https://github.com/googleapis/google-cloud-php/blob/b5109cfdc8e2b6032b96f1a99a50dd13dfcae356/Core/src/Batch/HandleFailureTrait.php#L77 (the code lines which write to local files)
Environment details
- OS: ubuntu debian 9.5
- PHP version: 7.3
- Package name and version: "google/cloud-logging": "^1.21",
Steps to reproduce
Code example
# Selects the log to write to
$GoogLogger = LoggingClient::psrBatchLogger('backend',['clientConfig'=> ['projectId'=> getenv('GCP_PROJECT_ID')]]);
//time that past since api start being process.
$processTimeSinceStart = (isset($_REQUEST['processStartTime'])) ? microtime(true) - $_REQUEST['processStartTime'] : 0;
# Creates the log entry
$logData = [
'traceId' => $_REQUEST['traceId'] ?? '',
'processName' => $_REQUEST['processName'] ?? '',
'processType' => $this->processType,
'processTimeSinceStart' => round($processTimeSinceStart, 3),
'WizecareAccountId' => $_REQUEST['accountId'] ?? '',
'longMessage' => substr($message, 0, 100000), //limit to 100K characters
'userAgent' => $_REQUEST['userAgent'] ?? '',
'logsCount' => $_REQUEST['logsCount'] ?? '',
'env' => getenv('APP_ENV'),
];
//if severity is warning or more, add error detection
if ($severityLevel>= LoggerInterface::WARNING_SEVERITY) {
$logData['@type'] = "type.googleapis.com/google.devtools.clouderrorreporting.v1beta1.ReportedErrorEvent";
}
$GoogLogger->log($severityLevel, substr($message, 0, 140), $logData);
example of content of 1 of the files
{"api":"channels/get-user-system-channel","accountId":1071})";i:1;i:101;}}s:14:"7: __construct";a:6:{s:4:"file";s:65:"/var/www/videotherapy.co/public_html/vt/api/DispatcherManager.php";s:4:"line";i:317;s:8:"function";s:11:"__construct";s:5:"c$ Accept: application/json, text/plain, / Accept-Encoding: gzip, deflate, br Accept-Language: en-US,en;q=0.9,he;q=0.8 Access-Control-Allow-Origin: * Connection: keep-alive Content-Length: 59 Content-Type: application/json;charset=UTF-8 Host: dev.wizecare.com Origin: http://localhost:8070 Referer: http://localhost:8070/ Sec-Ch-Ua: " Not A;Brand";v="99", "Chromium";v="96", "Google Chrome";v="96" Sec-Ch-Ua-Mobile: ?0 Sec-Ch-Ua-Platform: "macOS" Sec-Fetch-Dest: empty Sec-Fetch-Mode: cors Sec-Fetch-Site: cross-site User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/96.0.4664.110 Safari/537.36
{"api":"channels/get-user-system-channel","accountId":1071})";i:1;i:101;}}s:14:"6: __construct";a:6:{s:4:"file";s:65:"/var/www/videotherapy.co/public_html/vt/api/DispatcherManager.php";s:4:"line";i:317;s:8:"function";s:11:"__construct";s:5:"c$ Accept: application/json, text/plain, / Accept-Encoding: gzip, deflate, br Accept-Language: en-US,en;q=0.9,he;q=0.8 Access-Control-Allow-Origin: * Connection: keep-alive Content-Length: 59 Content-Type: application/json;charset=UTF-8 Host: dev.wizecare.com Origin: http://localhost:8070 Referer: http://localhost:8070/ Sec-Ch-Ua: " Not A;Brand";v="99", "Chromium";v="96", "Google Chrome";v="96" Sec-Ch-Ua-Mobile: ?0 Sec-Ch-Ua-Platform: "macOS" Sec-Fetch-Dest: empty Sec-Fetch-Mode: cors Sec-Fetch-Site: cross-site User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/96.0.4664.110 Safari/537.36
{"api":"channels/get-user-system-channel","accountId":1071})";i:1;i:101;}}s:14:"7: __construct";a:6:{s:4:"file";s:65:"/var/www/videotherapy.co/public_html/vt/api/DispatcherManager.php";s:4:"line";i:317;s:8:"function";s:11:"__construct";s:5:"c$ Accept: application/json, text/plain, / Accept-Encoding: gzip, deflate, br Accept-Language: en-US,en;q=0.9,he;q=0.8 Access-Control-Allow-Origin: * Connection: keep-alive Content-Length: 59 Content-Type: application/json;charset=UTF-8 Host: dev.wizecare.com Origin: http://localhost:8070 Referer: http://localhost:8070/ Sec-Ch-Ua: " Not A;Brand";v="99", "Chromium";v="96", "Google Chrome";v="96" Sec-Ch-Ua-Mobile: ?0 Sec-Ch-Ua-Platform: "macOS" Sec-Fetch-Dest: empty Sec-Fetch-Mode: cors Sec-Fetch-Site: cross-site User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/96.0.4664.110 Safari/537.36
@yeonatan we've recently introduced a feature to prevent logging to a temporary file on failure, if you'd like to use it while debugging your issue further:
export GOOGLE_CLOUD_BATCH_DAEMON_FAILURE_DIR="false"
As for why you are seeing such frequent failures while attempting to log, it isn't clear to me yet. Have you been able to add in some debugging statements to understand what could be causing the failures?
