icinga2 icon indicating copy to clipboard operation
icinga2 copied to clipboard
verified publisher icon Icinga

Recognize PENDING -> UNKNOWN as state change

Open Al2Klimov opened this issue 6 years ago • 24 comments

... to get notified about that state change.

fixes #7807 fixes #10217

Al2Klimov avatar Feb 05 '20 17:02 Al2Klimov

This changes the current behavior, and will possibly influence the way states are counted and notifications are sent. Imho this needs a deeper discussion with @lippserd

dnsmichi avatar Feb 05 '20 17:02 dnsmichi

Just for understanding: Why exactly is it not allowed?

Al2Klimov avatar Feb 11 '20 08:02 Al2Klimov

It is a change in the behavior of the state machine. Now, !cr (pending) -> first state is not treated as a state change. As such, people know exactly when event handlers and notifications are fired. If this is shifted by one, making this !cr (pending) -> first state a first state change, the counting logic changes and leads to unexpected behaviour with regard to event handlers and notifications.

If you really need to know about the change from pending (which isn't a state by definition in Icinga 2) to the first check result, our decision was to add a new signal where IcingaDB and other features may subscribe upon, leaving the current state machine intact.

dnsmichi avatar Feb 11 '20 09:02 dnsmichi

For me that notification behavior is absolutely expected, i.e. I if a service with one check attempt changes from a non-problem state (e.g. PENDING) to a problem state (e.g. UNKNOWN) I expect a notification.

By the way we've already changed that state machine indirectly. I opt for merging this at least after v2.12.

Al2Klimov avatar Feb 11 '20 14:02 Al2Klimov

I can only repeat myself, pending is not a state in Icinga 2. This is something which has been invented by the web views, and is now slowly moved into the core without re-evaluation. Imho this needs a proper thought and conceptual review on all the affected components.

Anyhow, pinging @Thomas-Gelf since this affects the whole architecture.

dnsmichi avatar Feb 11 '20 14:02 dnsmichi

Sigh. Okay, then it's not PENDING, but UNKNOWN w/o any check result so far which have been changed to not a problem by you and me.

As a user I'd like to get notifications on a change from UNKNOWN w/o any check result so far to UNKNOWN the same way as e.g. on a change from UNKNOWN w/o any check result so far to CRITICAL.

Al2Klimov avatar Feb 11 '20 14:02 Al2Klimov

Let's simplify this.

Please write a new proposal, similar to a draft concept with the label TBD, how the current state logic works, and how you'd like it being changed. Illustrate that in all its detail in a new issue, including the core and web components. 1000 words should be a good orientation for that. Then everyone can discuss based on your ideas.

That being said, I'll close this PR.

dnsmichi avatar Feb 11 '20 15:02 dnsmichi

IMHO the expected behaviour for related state transitions is as simple as follows:

From To Notify State Change
Pending Ok - x
Pending Up - x
Pending Warning x x
Pending Critical x x
Pending Unknown x x
Pending Down x x

State change should result in IDO/IcingaDB State & History updates, former (last) state fields should be NULL where a transition from Pending/NoState occurred. If you need more words please let me know.

Thomas-Gelf avatar Feb 11 '20 17:02 Thomas-Gelf

Thank you, @Thomas-Gelf, for your opinion. Yes, I need a bit more words, but I'll just add them by myself:

  1. Just to be clear, the thing you called Pending is actually UNKNOWN w/o any check result so far on our side
  2. You seem to expect an actual state change on transition from UNKNOWN w/o any check result so far to ANY w/ a check result
  3. This PR does no less and no more that that and should be re-opened for merging in the future

Al2Klimov avatar Feb 12 '20 09:02 Al2Klimov

Re-opening as there seem to be discussion potential. @dnsmichi and @N-o-X ("our decision") are against and @Thomas-Gelf and me are not (2 vs. 2).

Al2Klimov avatar Feb 12 '20 09:02 Al2Klimov

I'm not directly against it, I'm just not sure if this change would break stuff. I really think the behaviour @Thomas-Gelf proposed would be nice to have, but might break something during implementation. As @dnsmichi said, it would be nice to have a summary of what that change would touch (Core & Web) and if we're able to implement it without breaking those things.

N-o-X avatar Feb 12 '20 09:02 N-o-X

Even better. 1 vs. 2.5.

Al2Klimov avatar Feb 12 '20 09:02 Al2Klimov

A decision can only be made with a document which covers all the details. Therefore I'd suggest that @Al2Klimov creates a new issue and does exactly that.

dnsmichi avatar Feb 12 '20 09:02 dnsmichi

  1. We really don't need more than one issue for one PR.
  2. IMAO @Thomas-Gelf has already covered all details.

Al2Klimov avatar Feb 12 '20 09:02 Al2Klimov

I'm talking about a document, which includes the table, and describes the current situation, adds the changes and includes all dependencies. It should also discuss the possible impact on backends, features and the end-user. If you need an inspiration, look into #7814 how to write such a document/concept. Depending on the work load of yours, we should find a time slot in your schedule with @lippserd.

dnsmichi avatar Feb 12 '20 10:02 dnsmichi

To me this seems "slightly" exaggerated for a fix that adds a missing state change event. I do not know whether the pull request would lead to notifications being treated differently than state changes in the DB, so this is something @Al2Klimov could eventually verify and explain. It is important that we're not going to send notifications when transitioning from "pending" (no former result) to a non-problem state.

Web frontend: @dnsmichi, could you please elaborate why you're expecting this change to break anything there?

IDO: no new state type is being introduced. Fixing the NULL-issue I mentioned would be a change with potential impact. Skip that part if you feel unsure. SLA functions didn't trust this (last_state) column from the beginning, so who cares.

Notification commands: some might work with last_state. Could they fail when sending a notification for an penging-to-unknown transition? Only if they also failed for a pending-to-critical transition. Who is to blame? The script author. Is this worse than sending no notification at all? Definitively no.

Thomas-Gelf avatar Feb 12 '20 10:02 Thomas-Gelf

To me this seems "slightly" exaggerated for a fix that adds a missing state change event.

Full ack.

Al2Klimov avatar Feb 12 '20 10:02 Al2Klimov

I opt for changing this but we need a test protocol with all possible states, state changes and triggered notifications please.

lippserd avatar Jun 09 '20 08:06 lippserd

Test protocol

object Host NodeName {
	check_command = "passive"
	enable_active_checks = false
}

for (s in range(4)) {
	for (mca in range(1, 3)) {
		object Service mca + "x" + s use (s, mca) {
			host_name = NodeName
			check_command = "dummy"
			check_interval = 24h
			retry_interval = 24h
			max_check_attempts = mca
			vars.dummy_state = s
		}
	}
}

object NotificationCommand "noop" {
	command = [ "true" ]
}

object User "admin" {
}

apply Notification "noop" to Service {
	command = "noop"
	users = [ "admin" ]
	assign where host.name == NodeName
}

Before

➜  icinga2 git:(a96403331) prefix/sbin/icinga2 daemon -x notice
[2020-06-09 14:52:55 +0200] information/cli: Icinga application loader (version: v2.12.0-rc1-38-ga96403331; debug)
[2020-06-09 14:52:55 +0200] notice/cli: Spawning seemless worker process doing the actual work
[2020-06-09 14:52:55 +0200] notice/cli: Spawned worker process (PID 70392), waiting for it to load its config
[2020-06-09 14:52:55 +0200] information/cli: Loading configuration file(s).
[2020-06-09 14:52:55 +0200] notice/ConfigCompiler: Compiling config file: /Users/aklimov/NET/WS/icinga2/prefix/etc/icinga2/icinga2.conf
[2020-06-09 14:52:55 +0200] notice/ConfigCompiler: Compiling config file: /Users/aklimov/NET/WS/icinga2/prefix/etc/icinga2/constants.conf
[2020-06-09 14:52:55 +0200] notice/ConfigCompiler: Compiling config file: /Users/aklimov/NET/WS/icinga2/prefix/etc/icinga2/zones.conf
[2020-06-09 14:52:55 +0200] notice/ConfigCompiler: Compiling config file: /Users/aklimov/NET/WS/icinga2/prefix/share/icinga2/include/itl
[2020-06-09 14:52:55 +0200] notice/ConfigCompiler: Compiling config file: /Users/aklimov/NET/WS/icinga2/prefix/share/icinga2/include/command-icinga.conf
[2020-06-09 14:52:55 +0200] notice/ConfigCompiler: Compiling config file: /Users/aklimov/NET/WS/icinga2/prefix/share/icinga2/include/plugins
[2020-06-09 14:52:55 +0200] notice/ConfigCompiler: Compiling config file: /Users/aklimov/NET/WS/icinga2/prefix/share/icinga2/include/command-plugins.conf
[2020-06-09 14:52:55 +0200] notice/ConfigCompiler: Compiling config file: /Users/aklimov/NET/WS/icinga2/prefix/share/icinga2/include/plugins-contrib
[2020-06-09 14:52:55 +0200] notice/ConfigCompiler: Compiling config file: /Users/aklimov/NET/WS/icinga2/prefix/share/icinga2/include/plugins-contrib.d/big-data.conf
[2020-06-09 14:52:55 +0200] notice/ConfigCompiler: Compiling config file: /Users/aklimov/NET/WS/icinga2/prefix/share/icinga2/include/plugins-contrib.d/databases.conf
[2020-06-09 14:52:55 +0200] notice/ConfigCompiler: Compiling config file: /Users/aklimov/NET/WS/icinga2/prefix/share/icinga2/include/plugins-contrib.d/hardware.conf
[2020-06-09 14:52:55 +0200] notice/ConfigCompiler: Compiling config file: /Users/aklimov/NET/WS/icinga2/prefix/share/icinga2/include/plugins-contrib.d/icingacli.conf
[2020-06-09 14:52:55 +0200] notice/ConfigCompiler: Compiling config file: /Users/aklimov/NET/WS/icinga2/prefix/share/icinga2/include/plugins-contrib.d/ipmi.conf
[2020-06-09 14:52:55 +0200] notice/ConfigCompiler: Compiling config file: /Users/aklimov/NET/WS/icinga2/prefix/share/icinga2/include/plugins-contrib.d/logmanagement.conf
[2020-06-09 14:52:55 +0200] notice/ConfigCompiler: Compiling config file: /Users/aklimov/NET/WS/icinga2/prefix/share/icinga2/include/plugins-contrib.d/metrics.conf
[2020-06-09 14:52:55 +0200] notice/ConfigCompiler: Compiling config file: /Users/aklimov/NET/WS/icinga2/prefix/share/icinga2/include/plugins-contrib.d/network-components.conf
[2020-06-09 14:52:55 +0200] notice/ConfigCompiler: Compiling config file: /Users/aklimov/NET/WS/icinga2/prefix/share/icinga2/include/plugins-contrib.d/network-services.conf
[2020-06-09 14:52:55 +0200] notice/ConfigCompiler: Compiling config file: /Users/aklimov/NET/WS/icinga2/prefix/share/icinga2/include/plugins-contrib.d/operating-system.conf
[2020-06-09 14:52:55 +0200] notice/ConfigCompiler: Compiling config file: /Users/aklimov/NET/WS/icinga2/prefix/share/icinga2/include/plugins-contrib.d/raid-controller.conf
[2020-06-09 14:52:55 +0200] notice/ConfigCompiler: Compiling config file: /Users/aklimov/NET/WS/icinga2/prefix/share/icinga2/include/plugins-contrib.d/smart-attributes.conf
[2020-06-09 14:52:55 +0200] notice/ConfigCompiler: Compiling config file: /Users/aklimov/NET/WS/icinga2/prefix/share/icinga2/include/plugins-contrib.d/storage.conf
[2020-06-09 14:52:55 +0200] notice/ConfigCompiler: Compiling config file: /Users/aklimov/NET/WS/icinga2/prefix/share/icinga2/include/plugins-contrib.d/virtualization.conf
[2020-06-09 14:52:55 +0200] notice/ConfigCompiler: Compiling config file: /Users/aklimov/NET/WS/icinga2/prefix/share/icinga2/include/plugins-contrib.d/vmware.conf
[2020-06-09 14:52:55 +0200] notice/ConfigCompiler: Compiling config file: /Users/aklimov/NET/WS/icinga2/prefix/share/icinga2/include/plugins-contrib.d/web.conf
[2020-06-09 14:52:55 +0200] notice/ConfigCompiler: Compiling config file: /Users/aklimov/NET/WS/icinga2/prefix/share/icinga2/include/manubulon
[2020-06-09 14:52:55 +0200] notice/ConfigCompiler: Compiling config file: /Users/aklimov/NET/WS/icinga2/prefix/share/icinga2/include/command-plugins-manubulon.conf
[2020-06-09 14:52:55 +0200] notice/ConfigCompiler: Compiling config file: /Users/aklimov/NET/WS/icinga2/prefix/share/icinga2/include/windows-plugins
[2020-06-09 14:52:55 +0200] notice/ConfigCompiler: Compiling config file: /Users/aklimov/NET/WS/icinga2/prefix/share/icinga2/include/command-plugins-windows.conf
[2020-06-09 14:52:55 +0200] notice/ConfigCompiler: Compiling config file: /Users/aklimov/NET/WS/icinga2/prefix/share/icinga2/include/nscp
[2020-06-09 14:52:55 +0200] notice/ConfigCompiler: Compiling config file: /Users/aklimov/NET/WS/icinga2/prefix/share/icinga2/include/command-nscp-local.conf
[2020-06-09 14:52:55 +0200] notice/ConfigCompiler: Compiling config file: /Users/aklimov/NET/WS/icinga2/prefix/etc/icinga2/features-enabled/checker.conf
[2020-06-09 14:52:55 +0200] notice/ConfigCompiler: Compiling config file: /Users/aklimov/NET/WS/icinga2/prefix/etc/icinga2/features-enabled/mainlog.conf
[2020-06-09 14:52:55 +0200] notice/ConfigCompiler: Compiling config file: /Users/aklimov/NET/WS/icinga2/prefix/etc/icinga2/features-enabled/notification.conf
[2020-06-09 14:52:55 +0200] notice/ConfigCompiler: Compiling config file: /Users/aklimov/NET/WS/icinga2/prefix/etc/icinga2/conf.d/pusc.conf
[2020-06-09 14:52:55 +0200] notice/ConfigCompiler: Compiling config file: /Users/aklimov/NET/WS/icinga2/prefix/var/lib/icinga2/api/packages/_api/include.conf
[2020-06-09 14:52:55 +0200] notice/ConfigCompiler: Compiling config file: /Users/aklimov/NET/WS/icinga2/prefix/var/lib/icinga2/api/packages/_api/39c960e8-78e2-45dd-b2c0-a064dd88359a/include.conf
[2020-06-09 14:52:55 +0200] notice/ConfigCompiler: Compiling config file: /Users/aklimov/NET/WS/icinga2/prefix/var/lib/icinga2/api/packages/_api/39c960e8-78e2-45dd-b2c0-a064dd88359a/../active.conf
[2020-06-09 14:52:55 +0200] information/ConfigItem: Committing config item(s).
[2020-06-09 14:52:55 +0200] notice/WorkQueue: Spawning WorkQueue threads for 'DaemonUtility::LoadConfigFiles'
[2020-06-09 14:52:55 +0200] information/ConfigItem: Instantiated 235 CheckCommands.
[2020-06-09 14:52:55 +0200] information/ConfigItem: Instantiated 1 NotificationComponent.
[2020-06-09 14:52:55 +0200] information/ConfigItem: Instantiated 1 FileLogger.
[2020-06-09 14:52:55 +0200] information/ConfigItem: Instantiated 1 Host.
[2020-06-09 14:52:55 +0200] information/ConfigItem: Instantiated 1 Endpoint.
[2020-06-09 14:52:55 +0200] information/ConfigItem: Instantiated 1 IcingaApplication.
[2020-06-09 14:52:55 +0200] information/ConfigItem: Instantiated 8 Notifications.
[2020-06-09 14:52:55 +0200] information/ConfigItem: Instantiated 1 NotificationCommand.
[2020-06-09 14:52:55 +0200] information/ConfigItem: Instantiated 8 Services.
[2020-06-09 14:52:55 +0200] information/ConfigItem: Instantiated 3 Zones.
[2020-06-09 14:52:55 +0200] information/ConfigItem: Instantiated 1 User.
[2020-06-09 14:52:55 +0200] information/ConfigItem: Instantiated 1 CheckerComponent.
[2020-06-09 14:52:55 +0200] information/ScriptGlobal: Dumping variables to file '/Users/aklimov/NET/WS/icinga2/prefix/var/cache/icinga2/icinga2.vars'
[2020-06-09 14:52:55 +0200] notice/WorkQueue: Stopped WorkQueue threads for 'DaemonUtility::LoadConfigFiles'
[2020-06-09 14:52:55 +0200] notice/cli: Notifying umbrella process (PID 70378) about the config loading success
[2020-06-09 14:52:55 +0200] notice/cli: Worker process successfully loaded its config
[2020-06-09 14:52:55 +0200] notice/cli: Waiting for the umbrella process to let us doing the actual work
[2020-06-09 14:52:55 +0200] notice/cli: The umbrella process let us continuing
[2020-06-09 14:52:55 +0200] notice/ConfigCompiler: Compiling config file: /Users/aklimov/NET/WS/icinga2/prefix/var/lib/icinga2/modified-attributes.conf
[2020-06-09 14:52:55 +0200] information/ConfigItem: Triggering Start signal for config items
[2020-06-09 14:52:55 +0200] information/FileLogger: 'main-log' started.
[2020-06-09 14:52:55 +0200] information/NotificationComponent: 'notification' started.
[2020-06-09 14:52:55 +0200] information/CheckerComponent: 'checker' started.
[2020-06-09 14:52:55 +0200] information/ConfigItem: Activated all objects.
[2020-06-09 14:52:55 +0200] notice/WorkQueue: Stopped WorkQueue threads for 'DaemonCommand::Run'
[2020-06-09 14:52:55 +0200] notice/ApiListener: Updating object authority for local objects.
[2020-06-09 14:52:55 +0200] notice/CheckerComponent: Skipping check for host 'alexanders-mbp.int.netways.de': active host checks are disabled
[2020-06-09 14:52:58 +0200] notice/Checkable: State Change: Checkable 'alexanders-mbp.int.netways.de!2x3' hard state change from UNKNOWN to UNKNOWN detected.
[2020-06-09 14:52:58 +0200] information/Checkable: Checkable 'alexanders-mbp.int.netways.de!2x3' has 1 notification(s). Checking filters for type 'Problem', sends will be logged.
[2020-06-09 14:52:58 +0200] notice/Notification: Attempting to send notifications of type 'Problem' for notification object 'alexanders-mbp.int.netways.de!2x3!noop'.
[2020-06-09 14:52:58 +0200] information/Notification: Sending 'Problem' notification 'alexanders-mbp.int.netways.de!2x3!noop' for user 'admin'
[2020-06-09 14:52:58 +0200] notice/Process: Running command 'true': PID 70394
[2020-06-09 14:52:58 +0200] information/Notification: Completed sending 'Problem' notification 'alexanders-mbp.int.netways.de!2x3!noop' for checkable 'alexanders-mbp.int.netways.de!2x3' and user 'admin' using command 'noop'.
[2020-06-09 14:52:58 +0200] notice/Process: PID 70394 ('true') terminated with exit code 0
[2020-06-09 14:53:00 +0200] notice/CheckerComponent: Pending checkables: 0; Idle checkables: 9; Checks/s: 0.0166667
[2020-06-09 14:53:03 +0200] notice/Checkable: State Change: Checkable 'alexanders-mbp.int.netways.de!2x0' hard state change from UNKNOWN to OK detected.
[2020-06-09 14:53:05 +0200] notice/CheckerComponent: Pending checkables: 0; Idle checkables: 9; Checks/s: 0.0333333
[2020-06-09 14:53:08 +0200] notice/Checkable: State Change: Checkable 'alexanders-mbp.int.netways.de!1x2' hard state change from UNKNOWN to CRITICAL detected.
[2020-06-09 14:53:08 +0200] information/Checkable: Checkable 'alexanders-mbp.int.netways.de!1x2' has 1 notification(s). Checking filters for type 'Problem', sends will be logged.
[2020-06-09 14:53:08 +0200] notice/Notification: Attempting to send notifications of type 'Problem' for notification object 'alexanders-mbp.int.netways.de!1x2!noop'.
[2020-06-09 14:53:08 +0200] information/Notification: Sending 'Problem' notification 'alexanders-mbp.int.netways.de!1x2!noop' for user 'admin'
[2020-06-09 14:53:08 +0200] notice/Process: Running command 'true': PID 70395
[2020-06-09 14:53:08 +0200] information/Notification: Completed sending 'Problem' notification 'alexanders-mbp.int.netways.de!1x2!noop' for checkable 'alexanders-mbp.int.netways.de!1x2' and user 'admin' using command 'noop'.
[2020-06-09 14:53:08 +0200] notice/Process: PID 70395 ('true') terminated with exit code 0
[2020-06-09 14:53:10 +0200] notice/CheckerComponent: Pending checkables: 0; Idle checkables: 9; Checks/s: 0.05
[2020-06-09 14:53:15 +0200] notice/CheckerComponent: Pending checkables: 0; Idle checkables: 9; Checks/s: 0.05
[2020-06-09 14:53:20 +0200] notice/CheckerComponent: Pending checkables: 0; Idle checkables: 9; Checks/s: 0.05
[2020-06-09 14:53:23 +0200] notice/Checkable: State Change: Checkable 'alexanders-mbp.int.netways.de!2x2' hard state change from UNKNOWN to CRITICAL detected.
[2020-06-09 14:53:23 +0200] information/Checkable: Checkable 'alexanders-mbp.int.netways.de!2x2' has 1 notification(s). Checking filters for type 'Problem', sends will be logged.
[2020-06-09 14:53:23 +0200] notice/Notification: Attempting to send notifications of type 'Problem' for notification object 'alexanders-mbp.int.netways.de!2x2!noop'.
[2020-06-09 14:53:23 +0200] information/Notification: Sending 'Problem' notification 'alexanders-mbp.int.netways.de!2x2!noop' for user 'admin'
[2020-06-09 14:53:23 +0200] notice/Process: Running command 'true': PID 70396
[2020-06-09 14:53:23 +0200] information/Notification: Completed sending 'Problem' notification 'alexanders-mbp.int.netways.de!2x2!noop' for checkable 'alexanders-mbp.int.netways.de!2x2' and user 'admin' using command 'noop'.
[2020-06-09 14:53:23 +0200] notice/Process: PID 70396 ('true') terminated with exit code 0
[2020-06-09 14:53:25 +0200] notice/CheckerComponent: Pending checkables: 0; Idle checkables: 9; Checks/s: 0.0666667
[2020-06-09 14:53:27 +0200] notice/Checkable: State Change: Checkable 'alexanders-mbp.int.netways.de!1x1' hard state change from UNKNOWN to WARNING detected.
[2020-06-09 14:53:27 +0200] information/Checkable: Checkable 'alexanders-mbp.int.netways.de!1x1' has 1 notification(s). Checking filters for type 'Problem', sends will be logged.
[2020-06-09 14:53:27 +0200] notice/Notification: Attempting to send notifications of type 'Problem' for notification object 'alexanders-mbp.int.netways.de!1x1!noop'.
[2020-06-09 14:53:27 +0200] information/Notification: Sending 'Problem' notification 'alexanders-mbp.int.netways.de!1x1!noop' for user 'admin'
[2020-06-09 14:53:27 +0200] notice/Process: Running command 'true': PID 70397
[2020-06-09 14:53:27 +0200] information/Notification: Completed sending 'Problem' notification 'alexanders-mbp.int.netways.de!1x1!noop' for checkable 'alexanders-mbp.int.netways.de!1x1' and user 'admin' using command 'noop'.
[2020-06-09 14:53:27 +0200] notice/Process: PID 70397 ('true') terminated with exit code 0
[2020-06-09 14:53:30 +0200] notice/CheckerComponent: Pending checkables: 0; Idle checkables: 9; Checks/s: 0.0833333
[2020-06-09 14:53:35 +0200] notice/CheckerComponent: Pending checkables: 0; Idle checkables: 9; Checks/s: 0.0833333
[2020-06-09 14:53:36 +0200] notice/Checkable: State Change: Checkable 'alexanders-mbp.int.netways.de!1x3' hard state change from UNKNOWN to UNKNOWN detected.
[2020-06-09 14:53:36 +0200] information/Checkable: Checkable 'alexanders-mbp.int.netways.de!1x3' has 1 notification(s). Checking filters for type 'Problem', sends will be logged.
[2020-06-09 14:53:36 +0200] notice/Notification: Attempting to send notifications of type 'Problem' for notification object 'alexanders-mbp.int.netways.de!1x3!noop'.
[2020-06-09 14:53:36 +0200] information/Notification: Sending 'Problem' notification 'alexanders-mbp.int.netways.de!1x3!noop' for user 'admin'
[2020-06-09 14:53:36 +0200] notice/Process: Running command 'true': PID 70399
[2020-06-09 14:53:36 +0200] information/Notification: Completed sending 'Problem' notification 'alexanders-mbp.int.netways.de!1x3!noop' for checkable 'alexanders-mbp.int.netways.de!1x3' and user 'admin' using command 'noop'.
[2020-06-09 14:53:36 +0200] notice/Process: PID 70399 ('true') terminated with exit code 0
[2020-06-09 14:53:36 +0200] notice/Checkable: State Change: Checkable 'alexanders-mbp.int.netways.de!1x0' hard state change from UNKNOWN to OK detected.
[2020-06-09 14:53:40 +0200] notice/CheckerComponent: Pending checkables: 0; Idle checkables: 9; Checks/s: 0.116667
[2020-06-09 14:53:40 +0200] notice/Checkable: State Change: Checkable 'alexanders-mbp.int.netways.de!2x1' hard state change from UNKNOWN to WARNING detected.
[2020-06-09 14:53:40 +0200] information/Checkable: Checkable 'alexanders-mbp.int.netways.de!2x1' has 1 notification(s). Checking filters for type 'Problem', sends will be logged.
[2020-06-09 14:53:40 +0200] notice/Notification: Attempting to send notifications of type 'Problem' for notification object 'alexanders-mbp.int.netways.de!2x1!noop'.
[2020-06-09 14:53:40 +0200] information/Notification: Sending 'Problem' notification 'alexanders-mbp.int.netways.de!2x1!noop' for user 'admin'
[2020-06-09 14:53:40 +0200] notice/Process: Running command 'true': PID 70400
[2020-06-09 14:53:40 +0200] information/Notification: Completed sending 'Problem' notification 'alexanders-mbp.int.netways.de!2x1!noop' for checkable 'alexanders-mbp.int.netways.de!2x1' and user 'admin' using command 'noop'.
[2020-06-09 14:53:40 +0200] notice/Process: PID 70400 ('true') terminated with exit code 0
[2020-06-09 14:53:45 +0200] notice/CheckerComponent: Pending checkables: 0; Idle checkables: 9; Checks/s: 0.133333
[2020-06-09 14:53:50 +0200] notice/CheckerComponent: Pending checkables: 0; Idle checkables: 9; Checks/s: 0.133333
[2020-06-09 14:53:55 +0200] notice/CheckerComponent: Pending checkables: 0; Idle checkables: 9; Checks/s: 0.133333
[2020-06-09 14:54:00 +0200] notice/CheckerComponent: Pending checkables: 0; Idle checkables: 9; Checks/s: 0.116667
[2020-06-09 14:54:05 +0200] notice/CheckerComponent: Pending checkables: 0; Idle checkables: 9; Checks/s: 0.1
[2020-06-09 14:54:10 +0200] notice/CheckerComponent: Pending checkables: 0; Idle checkables: 9; Checks/s: 0.0833333
[2020-06-09 14:54:15 +0200] notice/CheckerComponent: Pending checkables: 0; Idle checkables: 9; Checks/s: 0.0833333
[2020-06-09 14:54:20 +0200] notice/CheckerComponent: Pending checkables: 0; Idle checkables: 9; Checks/s: 0.0833333
[2020-06-09 14:54:25 +0200] notice/CheckerComponent: Pending checkables: 0; Idle checkables: 9; Checks/s: 0.0666667
[2020-06-09 14:54:30 +0200] notice/CheckerComponent: Pending checkables: 0; Idle checkables: 9; Checks/s: 0.05
[2020-06-09 14:54:35 +0200] notice/CheckerComponent: Pending checkables: 0; Idle checkables: 9; Checks/s: 0.05
[2020-06-09 14:54:40 +0200] notice/CheckerComponent: Pending checkables: 0; Idle checkables: 9; Checks/s: 0
[2020-06-09 14:54:45 +0200] notice/CheckerComponent: Pending checkables: 0; Idle checkables: 9; Checks/s: 0
^C[2020-06-09 14:54:48 +0200] notice/cli: Got signal 2, forwarding to seemless worker (PID 70392)
[2020-06-09 14:54:48 +0200] information/Application: Received request to shut down.
[2020-06-09 14:54:48 +0200] information/Application: Shutting down...
[2020-06-09 14:54:48 +0200] information/CheckerComponent: 'checker' stopped.
[2020-06-09 14:54:48 +0200] information/NotificationComponent: 'notification' stopped.
[2020-06-09 14:54:48 +0200] information/ConfigObject: Dumping program state to file '/Users/aklimov/NET/WS/icinga2/prefix/var/lib/icinga2/icinga2.state'
[2020-06-09 14:54:48 +0200] information/IcingaApplication: Icinga has shut down.
[2020-06-09 14:54:48 +0200] notice/cli: Seemless worker (PID 70392) stopped, stopping as well
➜  icinga2 git:(a96403331)

After

➜  icinga2 git:(bugfix/pending-unknown-state-change-7807) prefix/sbin/icinga2 daemon -x notice
[2020-06-09 14:56:56 +0200] information/cli: Icinga application loader (version: v2.12.0-rc1-39-gfa758e245; debug)
[2020-06-09 14:56:56 +0200] notice/cli: Spawning seemless worker process doing the actual work
[2020-06-09 14:56:56 +0200] notice/cli: Spawned worker process (PID 70704), waiting for it to load its config
[2020-06-09 14:56:56 +0200] information/cli: Loading configuration file(s).
[2020-06-09 14:56:56 +0200] notice/ConfigCompiler: Compiling config file: /Users/aklimov/NET/WS/icinga2/prefix/etc/icinga2/icinga2.conf
[2020-06-09 14:56:56 +0200] notice/ConfigCompiler: Compiling config file: /Users/aklimov/NET/WS/icinga2/prefix/etc/icinga2/constants.conf
[2020-06-09 14:56:56 +0200] notice/ConfigCompiler: Compiling config file: /Users/aklimov/NET/WS/icinga2/prefix/etc/icinga2/zones.conf
[2020-06-09 14:56:56 +0200] notice/ConfigCompiler: Compiling config file: /Users/aklimov/NET/WS/icinga2/prefix/share/icinga2/include/itl
[2020-06-09 14:56:56 +0200] notice/ConfigCompiler: Compiling config file: /Users/aklimov/NET/WS/icinga2/prefix/share/icinga2/include/command-icinga.conf
[2020-06-09 14:56:56 +0200] notice/ConfigCompiler: Compiling config file: /Users/aklimov/NET/WS/icinga2/prefix/share/icinga2/include/plugins
[2020-06-09 14:56:56 +0200] notice/ConfigCompiler: Compiling config file: /Users/aklimov/NET/WS/icinga2/prefix/share/icinga2/include/command-plugins.conf
[2020-06-09 14:56:56 +0200] notice/ConfigCompiler: Compiling config file: /Users/aklimov/NET/WS/icinga2/prefix/share/icinga2/include/plugins-contrib
[2020-06-09 14:56:56 +0200] notice/ConfigCompiler: Compiling config file: /Users/aklimov/NET/WS/icinga2/prefix/share/icinga2/include/plugins-contrib.d/big-data.conf
[2020-06-09 14:56:56 +0200] notice/ConfigCompiler: Compiling config file: /Users/aklimov/NET/WS/icinga2/prefix/share/icinga2/include/plugins-contrib.d/databases.conf
[2020-06-09 14:56:56 +0200] notice/ConfigCompiler: Compiling config file: /Users/aklimov/NET/WS/icinga2/prefix/share/icinga2/include/plugins-contrib.d/hardware.conf
[2020-06-09 14:56:56 +0200] notice/ConfigCompiler: Compiling config file: /Users/aklimov/NET/WS/icinga2/prefix/share/icinga2/include/plugins-contrib.d/icingacli.conf
[2020-06-09 14:56:56 +0200] notice/ConfigCompiler: Compiling config file: /Users/aklimov/NET/WS/icinga2/prefix/share/icinga2/include/plugins-contrib.d/ipmi.conf
[2020-06-09 14:56:56 +0200] notice/ConfigCompiler: Compiling config file: /Users/aklimov/NET/WS/icinga2/prefix/share/icinga2/include/plugins-contrib.d/logmanagement.conf
[2020-06-09 14:56:56 +0200] notice/ConfigCompiler: Compiling config file: /Users/aklimov/NET/WS/icinga2/prefix/share/icinga2/include/plugins-contrib.d/metrics.conf
[2020-06-09 14:56:56 +0200] notice/ConfigCompiler: Compiling config file: /Users/aklimov/NET/WS/icinga2/prefix/share/icinga2/include/plugins-contrib.d/network-components.conf
[2020-06-09 14:56:56 +0200] notice/ConfigCompiler: Compiling config file: /Users/aklimov/NET/WS/icinga2/prefix/share/icinga2/include/plugins-contrib.d/network-services.conf
[2020-06-09 14:56:56 +0200] notice/ConfigCompiler: Compiling config file: /Users/aklimov/NET/WS/icinga2/prefix/share/icinga2/include/plugins-contrib.d/operating-system.conf
[2020-06-09 14:56:56 +0200] notice/ConfigCompiler: Compiling config file: /Users/aklimov/NET/WS/icinga2/prefix/share/icinga2/include/plugins-contrib.d/raid-controller.conf
[2020-06-09 14:56:56 +0200] notice/ConfigCompiler: Compiling config file: /Users/aklimov/NET/WS/icinga2/prefix/share/icinga2/include/plugins-contrib.d/smart-attributes.conf
[2020-06-09 14:56:56 +0200] notice/ConfigCompiler: Compiling config file: /Users/aklimov/NET/WS/icinga2/prefix/share/icinga2/include/plugins-contrib.d/storage.conf
[2020-06-09 14:56:56 +0200] notice/ConfigCompiler: Compiling config file: /Users/aklimov/NET/WS/icinga2/prefix/share/icinga2/include/plugins-contrib.d/virtualization.conf
[2020-06-09 14:56:56 +0200] notice/ConfigCompiler: Compiling config file: /Users/aklimov/NET/WS/icinga2/prefix/share/icinga2/include/plugins-contrib.d/vmware.conf
[2020-06-09 14:56:56 +0200] notice/ConfigCompiler: Compiling config file: /Users/aklimov/NET/WS/icinga2/prefix/share/icinga2/include/plugins-contrib.d/web.conf
[2020-06-09 14:56:56 +0200] notice/ConfigCompiler: Compiling config file: /Users/aklimov/NET/WS/icinga2/prefix/share/icinga2/include/manubulon
[2020-06-09 14:56:56 +0200] notice/ConfigCompiler: Compiling config file: /Users/aklimov/NET/WS/icinga2/prefix/share/icinga2/include/command-plugins-manubulon.conf
[2020-06-09 14:56:56 +0200] notice/ConfigCompiler: Compiling config file: /Users/aklimov/NET/WS/icinga2/prefix/share/icinga2/include/windows-plugins
[2020-06-09 14:56:56 +0200] notice/ConfigCompiler: Compiling config file: /Users/aklimov/NET/WS/icinga2/prefix/share/icinga2/include/command-plugins-windows.conf
[2020-06-09 14:56:56 +0200] notice/ConfigCompiler: Compiling config file: /Users/aklimov/NET/WS/icinga2/prefix/share/icinga2/include/nscp
[2020-06-09 14:56:56 +0200] notice/ConfigCompiler: Compiling config file: /Users/aklimov/NET/WS/icinga2/prefix/share/icinga2/include/command-nscp-local.conf
[2020-06-09 14:56:56 +0200] notice/ConfigCompiler: Compiling config file: /Users/aklimov/NET/WS/icinga2/prefix/etc/icinga2/features-enabled/checker.conf
[2020-06-09 14:56:56 +0200] notice/ConfigCompiler: Compiling config file: /Users/aklimov/NET/WS/icinga2/prefix/etc/icinga2/features-enabled/mainlog.conf
[2020-06-09 14:56:56 +0200] notice/ConfigCompiler: Compiling config file: /Users/aklimov/NET/WS/icinga2/prefix/etc/icinga2/features-enabled/notification.conf
[2020-06-09 14:56:56 +0200] notice/ConfigCompiler: Compiling config file: /Users/aklimov/NET/WS/icinga2/prefix/etc/icinga2/conf.d/pusc.conf
[2020-06-09 14:56:56 +0200] notice/ConfigCompiler: Compiling config file: /Users/aklimov/NET/WS/icinga2/prefix/var/lib/icinga2/api/packages/_api/include.conf
[2020-06-09 14:56:56 +0200] notice/ConfigCompiler: Compiling config file: /Users/aklimov/NET/WS/icinga2/prefix/var/lib/icinga2/api/packages/_api/39c960e8-78e2-45dd-b2c0-a064dd88359a/include.conf
[2020-06-09 14:56:56 +0200] notice/ConfigCompiler: Compiling config file: /Users/aklimov/NET/WS/icinga2/prefix/var/lib/icinga2/api/packages/_api/39c960e8-78e2-45dd-b2c0-a064dd88359a/../active.conf
[2020-06-09 14:56:56 +0200] information/ConfigItem: Committing config item(s).
[2020-06-09 14:56:56 +0200] notice/WorkQueue: Spawning WorkQueue threads for 'DaemonUtility::LoadConfigFiles'
[2020-06-09 14:56:56 +0200] information/ConfigItem: Instantiated 235 CheckCommands.
[2020-06-09 14:56:56 +0200] information/ConfigItem: Instantiated 1 NotificationComponent.
[2020-06-09 14:56:56 +0200] information/ConfigItem: Instantiated 1 FileLogger.
[2020-06-09 14:56:56 +0200] information/ConfigItem: Instantiated 1 Host.
[2020-06-09 14:56:56 +0200] information/ConfigItem: Instantiated 1 Endpoint.
[2020-06-09 14:56:56 +0200] information/ConfigItem: Instantiated 1 IcingaApplication.
[2020-06-09 14:56:56 +0200] information/ConfigItem: Instantiated 8 Notifications.
[2020-06-09 14:56:56 +0200] information/ConfigItem: Instantiated 1 NotificationCommand.
[2020-06-09 14:56:56 +0200] information/ConfigItem: Instantiated 8 Services.
[2020-06-09 14:56:56 +0200] information/ConfigItem: Instantiated 3 Zones.
[2020-06-09 14:56:56 +0200] information/ConfigItem: Instantiated 1 User.
[2020-06-09 14:56:56 +0200] information/ConfigItem: Instantiated 1 CheckerComponent.
[2020-06-09 14:56:56 +0200] information/ScriptGlobal: Dumping variables to file '/Users/aklimov/NET/WS/icinga2/prefix/var/cache/icinga2/icinga2.vars'
[2020-06-09 14:56:56 +0200] notice/WorkQueue: Stopped WorkQueue threads for 'DaemonUtility::LoadConfigFiles'
[2020-06-09 14:56:56 +0200] notice/cli: Notifying umbrella process (PID 70690) about the config loading success
[2020-06-09 14:56:56 +0200] notice/cli: Worker process successfully loaded its config
[2020-06-09 14:56:56 +0200] notice/cli: Waiting for the umbrella process to let us doing the actual work
[2020-06-09 14:56:56 +0200] notice/cli: The umbrella process let us continuing
[2020-06-09 14:56:56 +0200] notice/ConfigCompiler: Compiling config file: /Users/aklimov/NET/WS/icinga2/prefix/var/lib/icinga2/modified-attributes.conf
[2020-06-09 14:56:56 +0200] information/ConfigItem: Triggering Start signal for config items
[2020-06-09 14:56:56 +0200] information/FileLogger: 'main-log' started.
[2020-06-09 14:56:56 +0200] information/NotificationComponent: 'notification' started.
[2020-06-09 14:56:56 +0200] information/CheckerComponent: 'checker' started.
[2020-06-09 14:56:56 +0200] information/ConfigItem: Activated all objects.
[2020-06-09 14:56:56 +0200] notice/WorkQueue: Stopped WorkQueue threads for 'DaemonCommand::Run'
[2020-06-09 14:56:56 +0200] notice/ApiListener: Updating object authority for local objects.
[2020-06-09 14:56:56 +0200] notice/CheckerComponent: Skipping check for host 'alexanders-mbp.int.netways.de': active host checks are disabled
[2020-06-09 14:56:59 +0200] notice/Checkable: State Change: Checkable 'alexanders-mbp.int.netways.de!2x3' soft state change from UNKNOWN to UNKNOWN detected.
[2020-06-09 14:57:01 +0200] notice/CheckerComponent: Pending checkables: 0; Idle checkables: 9; Checks/s: 0.0166667
[2020-06-09 14:57:04 +0200] notice/Checkable: State Change: Checkable 'alexanders-mbp.int.netways.de!2x0' hard state change from UNKNOWN to OK detected.
[2020-06-09 14:57:06 +0200] notice/CheckerComponent: Pending checkables: 0; Idle checkables: 9; Checks/s: 0.0333333
[2020-06-09 14:57:09 +0200] notice/Checkable: State Change: Checkable 'alexanders-mbp.int.netways.de!1x2' hard state change from UNKNOWN to CRITICAL detected.
[2020-06-09 14:57:09 +0200] information/Checkable: Checkable 'alexanders-mbp.int.netways.de!1x2' has 1 notification(s). Checking filters for type 'Problem', sends will be logged.
[2020-06-09 14:57:09 +0200] notice/Notification: Attempting to send notifications of type 'Problem' for notification object 'alexanders-mbp.int.netways.de!1x2!noop'.
[2020-06-09 14:57:09 +0200] information/Notification: Sending 'Problem' notification 'alexanders-mbp.int.netways.de!1x2!noop' for user 'admin'
[2020-06-09 14:57:09 +0200] notice/Process: Running command 'true': PID 70706
[2020-06-09 14:57:09 +0200] information/Notification: Completed sending 'Problem' notification 'alexanders-mbp.int.netways.de!1x2!noop' for checkable 'alexanders-mbp.int.netways.de!1x2' and user 'admin' using command 'noop'.
[2020-06-09 14:57:10 +0200] notice/Process: PID 70706 ('true') terminated with exit code 0
[2020-06-09 14:57:11 +0200] notice/CheckerComponent: Pending checkables: 0; Idle checkables: 9; Checks/s: 0.05
[2020-06-09 14:57:16 +0200] notice/CheckerComponent: Pending checkables: 0; Idle checkables: 9; Checks/s: 0.05
[2020-06-09 14:57:21 +0200] notice/CheckerComponent: Pending checkables: 0; Idle checkables: 9; Checks/s: 0.05
[2020-06-09 14:57:24 +0200] notice/Checkable: State Change: Checkable 'alexanders-mbp.int.netways.de!2x2' soft state change from UNKNOWN to CRITICAL detected.
[2020-06-09 14:57:26 +0200] notice/CheckerComponent: Pending checkables: 0; Idle checkables: 9; Checks/s: 0.0666667
[2020-06-09 14:57:28 +0200] notice/Checkable: State Change: Checkable 'alexanders-mbp.int.netways.de!1x1' hard state change from UNKNOWN to WARNING detected.
[2020-06-09 14:57:28 +0200] information/Checkable: Checkable 'alexanders-mbp.int.netways.de!1x1' has 1 notification(s). Checking filters for type 'Problem', sends will be logged.
[2020-06-09 14:57:28 +0200] notice/Notification: Attempting to send notifications of type 'Problem' for notification object 'alexanders-mbp.int.netways.de!1x1!noop'.
[2020-06-09 14:57:28 +0200] information/Notification: Sending 'Problem' notification 'alexanders-mbp.int.netways.de!1x1!noop' for user 'admin'
[2020-06-09 14:57:28 +0200] notice/Process: Running command 'true': PID 70708
[2020-06-09 14:57:28 +0200] information/Notification: Completed sending 'Problem' notification 'alexanders-mbp.int.netways.de!1x1!noop' for checkable 'alexanders-mbp.int.netways.de!1x1' and user 'admin' using command 'noop'.
[2020-06-09 14:57:28 +0200] notice/Process: PID 70708 ('true') terminated with exit code 0
[2020-06-09 14:57:31 +0200] notice/CheckerComponent: Pending checkables: 0; Idle checkables: 9; Checks/s: 0.0833333
[2020-06-09 14:57:36 +0200] notice/CheckerComponent: Pending checkables: 0; Idle checkables: 9; Checks/s: 0.0833333
[2020-06-09 14:57:37 +0200] notice/Checkable: State Change: Checkable 'alexanders-mbp.int.netways.de!1x3' hard state change from UNKNOWN to UNKNOWN detected.
[2020-06-09 14:57:37 +0200] information/Checkable: Checkable 'alexanders-mbp.int.netways.de!1x3' has 1 notification(s). Checking filters for type 'Problem', sends will be logged.
[2020-06-09 14:57:37 +0200] notice/Notification: Attempting to send notifications of type 'Problem' for notification object 'alexanders-mbp.int.netways.de!1x3!noop'.
[2020-06-09 14:57:37 +0200] information/Notification: Sending 'Problem' notification 'alexanders-mbp.int.netways.de!1x3!noop' for user 'admin'
[2020-06-09 14:57:37 +0200] notice/Process: Running command 'true': PID 70714
[2020-06-09 14:57:37 +0200] information/Notification: Completed sending 'Problem' notification 'alexanders-mbp.int.netways.de!1x3!noop' for checkable 'alexanders-mbp.int.netways.de!1x3' and user 'admin' using command 'noop'.
[2020-06-09 14:57:37 +0200] notice/Process: PID 70714 ('true') terminated with exit code 0
[2020-06-09 14:57:37 +0200] notice/Checkable: State Change: Checkable 'alexanders-mbp.int.netways.de!1x0' hard state change from UNKNOWN to OK detected.
[2020-06-09 14:57:41 +0200] notice/CheckerComponent: Pending checkables: 0; Idle checkables: 9; Checks/s: 0.116667
[2020-06-09 14:57:42 +0200] notice/Checkable: State Change: Checkable 'alexanders-mbp.int.netways.de!2x1' soft state change from UNKNOWN to WARNING detected.
[2020-06-09 14:57:46 +0200] notice/CheckerComponent: Pending checkables: 0; Idle checkables: 9; Checks/s: 0.133333
[2020-06-09 14:57:51 +0200] notice/CheckerComponent: Pending checkables: 0; Idle checkables: 9; Checks/s: 0.133333
[2020-06-09 14:57:56 +0200] notice/CheckerComponent: Pending checkables: 0; Idle checkables: 9; Checks/s: 0.133333
[2020-06-09 14:58:01 +0200] notice/CheckerComponent: Pending checkables: 0; Idle checkables: 9; Checks/s: 0.116667
[2020-06-09 14:58:06 +0200] notice/CheckerComponent: Pending checkables: 0; Idle checkables: 9; Checks/s: 0.1
[2020-06-09 14:58:11 +0200] notice/CheckerComponent: Pending checkables: 0; Idle checkables: 9; Checks/s: 0.0833333
[2020-06-09 14:58:16 +0200] notice/CheckerComponent: Pending checkables: 0; Idle checkables: 9; Checks/s: 0.0833333
[2020-06-09 14:58:21 +0200] notice/CheckerComponent: Pending checkables: 0; Idle checkables: 9; Checks/s: 0.0833333
[2020-06-09 14:58:26 +0200] notice/CheckerComponent: Pending checkables: 0; Idle checkables: 9; Checks/s: 0.0666667
[2020-06-09 14:58:31 +0200] notice/CheckerComponent: Pending checkables: 0; Idle checkables: 9; Checks/s: 0.05
[2020-06-09 14:58:36 +0200] notice/CheckerComponent: Pending checkables: 0; Idle checkables: 9; Checks/s: 0.05
[2020-06-09 14:58:41 +0200] notice/CheckerComponent: Pending checkables: 0; Idle checkables: 9; Checks/s: 0.0166667
[2020-06-09 14:58:46 +0200] notice/CheckerComponent: Pending checkables: 0; Idle checkables: 9; Checks/s: 0
[2020-06-09 14:58:51 +0200] notice/CheckerComponent: Pending checkables: 0; Idle checkables: 9; Checks/s: 0
[2020-06-09 14:58:56 +0200] notice/CheckerComponent: Pending checkables: 0; Idle checkables: 9; Checks/s: 0
[2020-06-09 14:59:01 +0200] notice/CheckerComponent: Pending checkables: 0; Idle checkables: 9; Checks/s: 0
[2020-06-09 14:59:06 +0200] notice/CheckerComponent: Pending checkables: 0; Idle checkables: 9; Checks/s: 0
[2020-06-09 14:59:11 +0200] notice/CheckerComponent: Pending checkables: 0; Idle checkables: 9; Checks/s: 0
[2020-06-09 14:59:16 +0200] notice/CheckerComponent: Pending checkables: 0; Idle checkables: 9; Checks/s: 0
[2020-06-09 14:59:21 +0200] notice/CheckerComponent: Pending checkables: 0; Idle checkables: 9; Checks/s: 0
[2020-06-09 14:59:26 +0200] notice/CheckerComponent: Pending checkables: 0; Idle checkables: 9; Checks/s: 0
^C[2020-06-09 14:59:31 +0200] notice/cli: Got signal 2, forwarding to seemless worker (PID 70704)
[2020-06-09 14:59:31 +0200] information/Application: Received request to shut down.
[2020-06-09 14:59:31 +0200] notice/CheckerComponent: Pending checkables: 0; Idle checkables: 9; Checks/s: 0
[2020-06-09 14:59:32 +0200] information/Application: Shutting down...
[2020-06-09 14:59:32 +0200] information/CheckerComponent: 'checker' stopped.
[2020-06-09 14:59:32 +0200] information/NotificationComponent: 'notification' stopped.
[2020-06-09 14:59:32 +0200] information/ConfigObject: Dumping program state to file '/Users/aklimov/NET/WS/icinga2/prefix/var/lib/icinga2/icinga2.state'
[2020-06-09 14:59:32 +0200] information/IcingaApplication: Icinga has shut down.
[2020-06-09 14:59:32 +0200] notice/cli: Seemless worker (PID 70704) stopped, stopping as well
➜  icinga2 git:(bugfix/pending-unknown-state-change-7807)

Al2Klimov avatar Jun 09 '20 13:06 Al2Klimov

Let me summarize what I understand from the code change and please correct me if I am wrong:

  • This does not only apply to the state change PENDING [1] -> UNKNOWN, but to PENDING -> First State in general
  • This first state change is now always treated as a hard state change, which
  • Triggers our state change and notification handlers
  • PENDING is considered a non-problem state, so notifications are only triggered if the First State is a problem state
  • The first state (as state change) is now also reflected in our IDO history tables, event API and Redis streams

[1] I mean "PENDING" in the sense of "No check result yet".

lippserd avatar Nov 10 '20 10:11 lippserd

  1. Yes and no. Now it applies also to PENDING -> UNKNOWN, in addition to PENDING -> X.
  2. Hard state change? Why hard state change? (see below)
  3. -
  4. See #7685 and 2.
  5. Yes. (But see also 1.)
[2020-11-10 13:20:55 +0100] warning/PluginCheckTask: Check command for object 'Alexanders-MacBook-Pro.local!http' (PID: 88691, arguments: '/Users/aklimov/NET/WS/icinga2/prefix/usr/lib/nagios/plugins/check_http' '-I' '127.0.0.1' '-u' '/') terminated with exit code 128, output: execvpe(/Users/aklimov/NET/WS/icinga2/prefix/usr/lib/nagios/plugins/check_http) failed: No such file or directory

[2020-11-10 13:20:55 +0100] notice/Checkable: State Change: Checkable 'Alexanders-MacBook-Pro.local!http' soft state change from UNKNOWN to UNKNOWN detected.

Al2Klimov avatar Nov 10 '20 12:11 Al2Klimov

@Al2Klimov I find it a little confusing that the first state without cr is also referred to as UNKNOWN in the logs. Is there any chance that we log something like "first state change to UNKNOWN" instead of "UNKNOWN to UNKNOWN"?

lippserd avatar Mar 10 '21 12:03 lippserd

@cla-bot check

Al2Klimov avatar Aug 04 '21 12:08 Al2Klimov

ref/IP/44082 ref/NC/774135

Al2Klimov avatar Jan 25 '23 12:01 Al2Klimov