MagicMirror icon indicating copy to clipboard operation
MagicMirror copied to clipboard

Weather module (using pirateweather provider) don't update/grow stale

Open JHenzi opened this issue 1 year ago • 28 comments

Platform: Raspberry Pi 4.

Node Version: 22.

MagicMirror² Version: latest

Description: Weather forecast (pirate weather) doesn't update without relaunching or reloading the page. Today, Tomorrow and the next day are all wrong after one day (today is yesterday, tomorrow is today). I don't think the values are updating either. Believe the current weather / temperature is wrong after it ages as well as it didn't update for a day.

Steps to Reproduce: Use Pirate Weather with this config:

	{
			module: "weather",
			position: "top_right",
			config: {
				weatherProvider: "pirateweather",
				type: "current",
				lat: 39,
				lon: -84,
				apiKey: "---",
			}
		},
		{
			module: "weather",
			position: "top_left",
			header: "Weather Forecast",
			config: {
				weatherProvider: "pirateweather",
				type: "forecast",
				lat: 39,
				lon: -84,
				apiKey: "---",
			}
		}

Expected Results: Live data.

Actual Results: Stale data

Configuration: See above

Additional Notes: Recently switched to Pirate Weather. Don't see any errors in the console.

JHenzi avatar Jan 11 '25 16:01 JHenzi

the documentation says there is a 10 minute cycle time, before next fetch of the data

weather is done all in browser, do if you waited more than 10 minutes, open the develoer window ctrl-shift-i select the console tab and put a unique part of the module name in the filter field (weath for this module)

see if there are any error messages (usually in red) and copy/paste them here

sdetweil avatar Jan 11 '25 17:01 sdetweil

No error messages in console. Happy to find any other logs I guess. Maybe better to explain the weather forecast will say:

  • today
  • tomorrow
  • mon

When today is Monday.

JHenzi avatar Jan 11 '25 22:01 JHenzi

today is saturday

sdetweil avatar Jan 11 '25 22:01 sdetweil

No error messages in console. Happy to find any other logs I guess. Maybe better to explain the weather forecast will say:

today tomorrow sun

When today is Sunday.

The module is broken.

JHenzi avatar Jan 12 '25 14:01 JHenzi

ok, I changed the title to include pirateweather, someone will have to look at it..

sdetweil avatar Jan 12 '25 14:01 sdetweil

I'm not seeing this still happening since I reported it. Users are migrating to pirate weather, maybe someone else will have this happen.

JHenzi avatar Jan 13 '25 15:01 JHenzi

maybe it was a Pirateweather problem

sdetweil avatar Jan 20 '25 22:01 sdetweil

so this was a temporary problem? If yes, can we close this issue? @JHenzi

khassel avatar Jan 31 '25 22:01 khassel

I've been experiencing this also; it's not a temporary problem but seemingly intermittent. It seems related to scheduling updates. When the mirror process starts, updates perform as expected, but over time, the process eventually stops updating without error.

Very painfully slowly working on identifying where or why this is coming up; likely something that Pirateweather is including in the response rarely.

Ryanclbryant avatar Feb 03 '25 15:02 Ryanclbryant

Also same problem. Def. not temporary… i am using yr as provider

merasil avatar Feb 06 '25 21:02 merasil

For information still have the same problem always show "Loading..."

dangherve avatar Feb 27 '25 15:02 dangherve

do not know if it will help but i have the same problem with weatherbit and envcanada

And on openmeteo I got "invalid date" NAN 0 or undefined

dangherve avatar Feb 27 '25 18:02 dangherve

For others experiencing this issue; I’m curious if you also use a third party module which consumes calendar and weather notifications?

This problem seemingly disappeared when I stopped broadcasting calendar events, and returned when I started them again. I suspect this is related to issue #3678

Ryanclbryant avatar Feb 27 '25 21:02 Ryanclbryant

#3678 is unsolved. for a few people broadcast/sendSocketNotification doesn't work.

no reliable debug found yet

sdetweil avatar Feb 27 '25 22:02 sdetweil

I finaly test several provider the only one woking is openweathermap but the hourly type is not working

dangherve avatar Feb 27 '25 23:02 dangherve

hourly is not implemented for all providers

Image

sdetweil avatar Feb 27 '25 23:02 sdetweil

not what the screenshot says

dangherve avatar Feb 27 '25 23:02 dangherve

For others experiencing this issue; I’m curious if you also use a third party module which consumes calendar and weather notifications?

This problem seemingly disappeared when I stopped broadcasting calendar events, and returned when I started them again. I suspect this is related to issue #3678

I'm not using anything like that.

The behavior described above makes sense - after "some time" Magic Mirror needs restarted to re-enable weather updates.

JHenzi avatar Mar 10 '25 15:03 JHenzi

I have this issue too. I'm using pirateweather, data updates when starting MM, and then seems to update for a couple of days.. then it's stuck with days and weather data.

papinist avatar May 02 '25 19:05 papinist

weather is all in browser

can you install MMM-LogExt or MMM-Logging to send the browser logs to npm start logs, so we have them in one place

sdetweil avatar May 02 '25 19:05 sdetweil

I switched to MMM-PirateSkyForecast that I just found. Will see if data in this will remain stuck too.

papinist avatar May 04 '25 21:05 papinist

I have this issue as well. Things are generally fine for a few days, and then it's usually a day or several before I notice the weather stopped updating.

My system is in that state right now. In the error log at the same time as the last logged weather update I have the following:

[2025-05-14 11:01:32.757] [ERROR] TypeError: invalid media type
    at Object.parse (/home/pi/MagicMirror/node_modules/content-type/index.js:126:11)
    at setCharset (/home/pi/MagicMirror/node_modules/express/lib/utils.js:252:28)
    at ServerResponse.send (/home/pi/MagicMirror/node_modules/express/lib/response.js:175:32)
    at cors (/home/pi/MagicMirror/js/server_functions.js:58:8)
    at process.processTicksAndRejections (node:internal/process/task_queues:95:5)
    at async /home/pi/MagicMirror/js/server.js:82:41
[2025-05-14 11:01:32.764] [ERROR] (node:1296) UnhandledPromiseRejectionWarning: TypeError: invalid media type
    at Object.parse (/home/pi/MagicMirror/node_modules/content-type/index.js:126:11)
    at setCharset (/home/pi/MagicMirror/node_modules/express/lib/utils.js:252:28)
    at ServerResponse.send (/home/pi/MagicMirror/node_modules/express/lib/response.js:175:32)
    at ServerResponse.json (/home/pi/MagicMirror/node_modules/express/lib/response.js:279:15)
    at ServerResponse.send (/home/pi/MagicMirror/node_modules/express/lib/response.js:163:21)
    at cors (/home/pi/MagicMirror/js/server_functions.js:62:7)
    at process.processTicksAndRejections (node:internal/process/task_queues:95:5)
    at async /home/pi/MagicMirror/js/server.js:82:41
(Use `electron --trace-warnings ...` to show where the warning was created)
[2025-05-14 11:01:32.765] [ERROR] (node:1296) UnhandledPromiseRejectionWarning: Unhandled promise rejection. This error originated either by throwing inside of an async function without a catch block, or by rejecting a promise which was not handled with .catch(). To terminate the node process on unhandled promise rejection, use the CLI flag `--unhandled-rejections=strict` (see https://nodejs.org/api/cli.html#cli_unhandled_rejections_mode). (rejection id: 1)
[2025-05-14 11:01:53.124] [ERROR] TypeError: invalid media type
    at Object.parse (/home/pi/MagicMirror/node_modules/content-type/index.js:126:11)
    at setCharset (/home/pi/MagicMirror/node_modules/express/lib/utils.js:252:28)
    at ServerResponse.send (/home/pi/MagicMirror/node_modules/express/lib/response.js:175:32)
    at cors (/home/pi/MagicMirror/js/server_functions.js:58:8)
    at process.processTicksAndRejections (node:internal/process/task_queues:95:5)
    at async /home/pi/MagicMirror/js/server.js:82:41
[2025-05-14 11:01:53.125] [ERROR] (node:1296) UnhandledPromiseRejectionWarning: TypeError: invalid media type
    at Object.parse (/home/pi/MagicMirror/node_modules/content-type/index.js:126:11)
    at setCharset (/home/pi/MagicMirror/node_modules/express/lib/utils.js:252:28)
    at ServerResponse.send (/home/pi/MagicMirror/node_modules/express/lib/response.js:175:32)
    at ServerResponse.json (/home/pi/MagicMirror/node_modules/express/lib/response.js:279:15)
    at ServerResponse.send (/home/pi/MagicMirror/node_modules/express/lib/response.js:163:21)
    at cors (/home/pi/MagicMirror/js/server_functions.js:62:7)
    at process.processTicksAndRejections (node:internal/process/task_queues:95:5)
    at async /home/pi/MagicMirror/js/server.js:82:41
[2025-05-14 11:01:53.125] [ERROR] (node:1296) UnhandledPromiseRejectionWarning: Unhandled promise rejection. This error originated either by throwing inside of an async function without a catch block, or by rejecting a promise which was not handled with .catch(). To terminate the node process on unhandled promise rejection, use the CLI flag `--unhandled-rejections=strict` (see https://nodejs.org/api/cli.html#cli_unhandled_rejections_mode). (rejection id: 2)

jbaboval avatar May 18 '25 23:05 jbaboval

generally this means that the data fetch failed and the contents were not as expected, and no effective error recovery was in place.

it appears the module tried again, and encountered the same error again, and again

sdetweil avatar May 19 '25 11:05 sdetweil

looking at the MM code for handling this request

[2025-05-14 11:01:32.757] [ERROR] TypeError: invalid media type
    at Object.parse (/home/pi/MagicMirror/node_modules/content-type/index.js:126:11)
    at setCharset (/home/pi/MagicMirror/node_modules/express/lib/utils.js:252:28)
    at ServerResponse.send (/home/pi/MagicMirror/node_modules/express/lib/response.js:175:32)
    at cors (/home/pi/MagicMirror/js/server_functions.js:58:8)
    at process.processTicksAndRejections (node:internal/process/task_queues:95:5)
    at async /home/pi/MagicMirror/js/server.js:82:41

from providers.js

	async fetchData (url, type = "json", requestHeaders = undefined, expectedResponseHeaders = undefined) {
		const mockData = this.config.mockData;
		if (mockData) {
			const data = mockData.substring(1, mockData.length - 1);
			return JSON.parse(data);
		}
		const useCorsProxy = typeof this.config.useCorsProxy !== "undefined" && this.config.useCorsProxy;
		return performWebRequest(url, type, useCorsProxy, requestHeaders, expectedResponseHeaders, config.basePath);
	}

in performWebRequest (modules/default/utils.js)

we don't check for an error from fetch , and ASSUME it is good

async function performWebRequest (url, type = "json", useCorsProxy = false, requestHeaders = undefined, expectedResponseHeaders = undefined, basePath = "/") {
	const request = {};
	let requestUrl;
	if (useCorsProxy) {
		requestUrl = getCorsUrl(url, requestHeaders, expectedResponseHeaders, basePath); <--- using cors proxy
	} else {
		requestUrl = url;
		request.headers = getHeadersToSend(requestHeaders);
	}
	const response = await fetch(requestUrl, request);  <----- after here (via proxy url)
	const data = await response.text();

	if (type === "xml") {
		return new DOMParser().parseFromString(data, "text/html");
	} else {
		if (!data || !data.length > 0) return undefined;  <----- would cause error on res.send in cors

		const dataResponse = JSON.parse(data);
		if (!dataResponse.headers) {
			dataResponse.headers = getHeadersFromResponse(expectedResponseHeaders, response);
		}
		return dataResponse;

cors also doesn't check js/server_functions/cors

			Log.log(`cors url: ${url}`);
			const response = await fetch(url, { headers: headersToSend }); <------

			for (const header of expectedReceivedHeaders) {
				const headerValue = response.headers.get(header);
				if (header) res.set(header, headerValue);
			}
			const data = await response.text();
			res.send(data);  <--- doesn't return fetch error 

sdetweil avatar May 19 '25 13:05 sdetweil

I think all these await have to be wrapped with try/catch and we have to to thow on error

sdetweil avatar May 19 '25 15:05 sdetweil

and we didn't check the response header for accept/json or xml matching the type

cors got invoked because it was in the url but it didnt throw, etc,,,

sdetweil avatar May 19 '25 17:05 sdetweil

I'm experiencing the same issue but with openweathermap provider. Worth mentioning, I don't know if it's related, but I'm having a similar issue with the newsfeed also. It fetches the data in the initial run, but does not fetch new data after that. I tried multiple configurations trying to discard problems with the default values, but couldn't get it to work. This is the last one I tried:

                        module: "weather",
                        position: "top_right",
                        config: {
                                weatherProvider: "openweathermap",
                                type: "current",
                                updateInterval: 1000 * 60 *10,
                                roundTemp: true,
                                lat: -25.28183,
                                lon: -57.61310,
                                lang: 'es',
                                weatherEndpoint: '/weather',
                                apiVersion: '2.5',
                                apiKey: '...'
                        }
                },
                {
                        module: "weather",
                        position: "top_right",
                        header: "Pronóstico del tiempo en",
                        config: {
                                weatherProvider: "openweathermap",
                                type: "forecast",
                                updateInterval: 1000 * 60 * 120,
                                fade: false,
                                roundTemp: true,
                                lat: -25.28183,
                                lon: -57.61310,
                                lang: 'es',
                                weatherEndpoint: '/forecast',
                                apiVersion: '2.5',
                                apiKey: ' ...'

                        }
                },
                {
                        module: "newsfeed",
                        position: "bottom_bar",
                        config: {
                                feeds: [
                                        {
                                                title: "ABC Color",
                                                url: "https://www.abc.com.py/arc/outboundfeeds/rss/nacionales/?outputType=xml"
                                        }
                                ],
                                ignoreOldItems: true,
                                ignoreOlderThan: 1000 * 60 * 60 * 24 * 3,
                                reloadInterval: 1000 * 60 * 60,
                                showSourceTitle: true,
                                showPublishDate: true,
                                broadcastNewsFeeds: true,
                                broadcastNewsUpdates: true
                        }
                },```

gabsruff avatar Jul 28 '25 17:07 gabsruff

@gabsruff

Try this unrelated test fix See https://forum.magicmirror.builders/post/127456

sdetweil avatar Jul 28 '25 19:07 sdetweil