ci/lava: Ignore DUT feedback messages

In our process of monitoring LAVA logs, we typically skip numerous
messages to enhance log clarity. We already exclude `feedback` messages
from display. These messages were just used as a heartbeat signal,
indicating that if we are receiving them, the Device Under Test (DUT) is
active.

Practically, if we continuously receive feedback messages without any
other message level (either `debug` or `target`) for several minutes,
this could be a cause for concern, as it likely indicates the device is
in a kind of livelock state.

Therefore, it is more prudent to ignore feedback messages, as they tend
to occur frequently in unstable scenarios. However, it's important to
note that any other message level will still be considered as a
heartbeat signal.

Real case where several minutes of feedback messages indicate a hang:
https://gitlab.freedesktop.org/mesa/mesa/-/jobs/53546660

Signed-off-by: Guilherme Gallo <guilherme.gallo@collabora.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/26996>

1 files changed, 22 insertions, 0 deletions

diff --git a/.gitlab-ci/lava/utils/log_follower.py b/.gitlab-ci/lava/utils/log_follower.py
index 1fdf490bcb8..7d54f884465 100644
--- a/.gitlab-ci/lava/utils/log_follower.py
+++ b/.gitlab-ci/lava/utils/log_follower.py
@@ -187,6 +187,25 @@ class LogFollower:
 
         return False
 
+    def ignore_dut_feedback_lines(self, line: dict[str, str]) -> bool:
+        """
+        Ignores feedback lines from LAVA.
+        If we only receive this level of message for some time, it means that the job is
+        misbehaving. E.g Rebooting.
+
+        Args:
+            line: A dictionary representing a single log line.
+
+        Returns:
+            A boolean indicating whether the current line is a feedback line.
+        """
+        if line["lvl"] == "feedback" and line["ns"] == "dut":
+            return True
+        if line["lvl"] == "debug":
+            # This message happens after LAVA end receiving the feedback from the DUT
+            if line["msg"] == "Listened to connection for namespace 'dut' done":
+                return True
+        return False
 
     def feed(self, new_lines: list[dict[str, str]]) -> bool:
         """Input data to be processed by LogFollower instance
@@ -207,6 +226,9 @@ class LogFollower:
             if self.merge_carriage_return_lines(line):
                 continue
 
+            if self.ignore_dut_feedback_lines(line):
+                continue
+
             # At least we are fed with a non-kernel dump log, it seems that the
             # job is progressing
             is_job_healthy = True