UNOMI-944: Add IT developer tooling for run archival, cross-run comparison and live Karaf inspection#770
Merged
Merged
Conversation
…rison and live Karaf inspection
There was a problem hiding this comment.
Pull request overview
This PR adds opt-in integration-test (IT) developer tooling under itests/ to (1) archive artifacts from a completed IT run, (2) compare multiple archived runs to spot flaky vs systematic failures, and (3) inspect the live Pax Exam/Karaf instance during a run. It also updates the IT documentation and records a build trace for reproducibility.
Changes:
- Add
archive-it-run.sh+ shared libs to capture/triage Karaf logs, failsafe reports, engine logs, and run context intoitests/archives/. - Add
compare-it-runs.sh+ comparison engine to classify tests across multiple captures (systematic/flaky/regression/fixed). - Add
build.sh“IT run trace” output (itests/target/it-run-trace.properties) and expanditests/README.mddocumentation (plus new LLM analysis guide).
Reviewed changes
Copilot reviewed 12 out of 13 changed files in this pull request and generated 12 comments.
Show a summary per file
| File | Description |
|---|---|
itests/README.md |
Major rewrite/expansion of IT overview, best practices, debugging, and new tooling docs |
itests/llm-it-run-analysis-guide.md |
New guide explaining how to interpret archived artifacts/logs for analysis |
itests/kt.sh |
New helper to locate and inspect the latest Pax Exam/Karaf instance/logs during a run |
itests/it-run-ui.sh |
New UI/presentation helpers for archiver/comparator scripts |
itests/it-run-lib.sh |
New shared library for run discovery, parsing, indexing, and fingerprinting |
itests/it-run-karaf-lib.sh |
New Karaf log mining/triage implementation (rollover support, unexpected filtering, correlation) |
itests/it-run-context-lib.sh |
New run-context/manifest generation and environment snapshot helpers |
itests/it-run-compare-lib.sh |
New comparison/report engine for multi-run test outcome classification |
itests/it-run-bootstrap.sh |
New bootstrap module that sources shared tooling and initializes environment |
itests/compare-it-runs.sh |
New CLI entrypoint for comparing two+ archived runs |
itests/archive-it-run.sh |
New CLI entrypoint for capturing/archiving an IT run and optionally auto-comparing recent runs |
build.sh |
Adds IT run tracing file output for reproducibility (mvn options, heaps, engine, flags) |
.gitignore |
Ignores itests/archives/ to keep captured artifacts out of git |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
…move shared libs under itests/lib/
kt.sh was missing set -euo pipefail unlike the other entry-point scripts, and the start/stop/console/debug arms used `if [ $? -eq 0 ]` which always exits 0 even when find_karaf_exec fails. Replaced with `|| exit 1`. In it-run-karaf.sh, karaf_line_is_actionable_trigger was spawning three separate sed subshells for ANSI stripping — one per helper. Pre-strip once and pass the result to all three. Also switched the exception-index triage step to use run_karaf_triage_step consistently with the other two steps in process_karaf_log_dir.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
archive-it-run.shto capture IT run artifacts (Karaf logs, failsafe reports, engine config, build trace) before the next build wipesitests/target/compare-it-runs.shto diff two or more captured runs and classify each test as consistently failing, consistently passing, or flakykt.sh(backported from unomi-3-dev) for live Karaf inspection during a run: log, tail, grep, start, stop, debugbuild.shthat records the exact Maven options, search engine, heap sizes, and flags for every IT runit-run-lib.sh,it-run-karaf-lib.sh,it-run-context-lib.sh,it-run-compare-lib.sh,it-run-bootstrap.sh,it-run-ui.sh) used by the above scriptsitests/README.mdwith a full overview of how the IT suite works, how to write tests (BaseIT API, polling helpers, HTTP helpers, log checking), and documentation for all new toolingNotes
itests/archives/is gitignoredTest plan
./build.sh --integration-testsand verifyitests/target/it-run-trace.propertiesis written./itests/archive-it-run.shafter a completed IT run and verify the archive is created underitests/archives/./itests/compare-it-runs.sh --last 2with two captures and verify the comparison report./itests/kt.sh helpand verify it locates the Karaf directory