/usr/bin/jq: Argument list too long

[mahdirezaie336]

I ran the runSonarQube.sh file successfully. But at the end of the build process, it waits for a few minutes and then gives me this error:

...
[INFO] BUILD SUCCESS
[INFO] ------------------------------------------------------------------------
[INFO] Total time:  01:47 min
[INFO] Finished at: 2022-12-02T19:41:51+03:30
[INFO] ------------------------------------------------------------------------



./scripts/runSonarQube.sh: line 60: /usr/bin/jq: Argument list too long

My OS is ubuntu 2022 and I use SonarQube verison 9.7.1.

I guess it reaches the maximum size of ARG_MAX in linux which is related to size of args buffer. It is mentioned here and here.
I couldn't fix it by myself.

[darkspirit510]

Hi, thank you for your issue!

Are you talking about SonarQube CE or any paid version?

Sascha

[mahdirezaie336]

Thanks
I use SonarQube CE. Is there any differences?

[mahdirezaie336]

Any updates?

[darkspirit510]

I use SonarQube CE. Is there any differences?
Don't think so. I just wanted to be sure when trying to reproduce.

Any updates?
No, had a lot to do in the last weeks. Will hopefully check it today, but more likely after christmas.

[darkspirit510]

Could you checkout #189 and see, if it works? I reduced the "chunk" size. At least it works on a clean ubuntu installation. If it does not work, I have to rewrite the script completely.

[zoobinn]

@darkspirit510 tested with your changes. Less arguments too long but still the behavior is not stable. And also for Score calculation literally is not correct. Using SonarQube Enterprise 9.9 LTS, Can clearly see the run populates close to ~18K Vulnerability Issues and ~1200 Hotspot. However, the benchmark script is only processing ~200 Vulnerability. Guess the runSonarqube.sh scripts needs a fundamental re-write.

[darkspirit510]

@darkspirit510 did you enter enterprise credentials to the script? (the community edition reports way less results). i don't have access to an enterprise instance. But I guess, jq is not made for generating such a big json file (but I am not sure about this). Feel free to fix it 😉

[zoobinn]

yes, most probably JQ can not handle the size of SonarQube JSON, the more rules the bigger the size of JSON array . Please check this PR. To avoid JQ errors reduced page size to 20 while writing results incrementally in to 2 files.
https://github.com/OWASP-Benchmark/BenchmarkJava/pull/196/files

Tracking this with SonarSource as well (Re Broken Benchmark and PR): https://community.sonarsource.com/t/tech-story-takeaways-from-building-a-sast-product-and-why-owasp-benchmark-is-not-enough/15126/4

[darkspirit510]

Hi @mahdirezaie336 sorry for the late reply. Please retry if #286 solved this issue. I removed the custom jq part completely.

[TheAuditorTool]

Hi @mahdirezaie336 sorry for the late reply. Please retry if #286 solved this issue. I removed the custom jq part completely.

I have confirmed your fix and cleaned up the leftovers. There is still issues with 10k API truncation sizes but i deferred it.
Let me know if you want me to fix that too, would solve @zoobinn issue.

The jq ARG_MAX overflow was fixed in 32933c4 by replacing shell-based result aggregation with SonarReport.java. This commit removes leftover artifacts: the sonar-report tool check that blocks script execution despite no longer being used, and uncomments docker stop so the SonarQube container is properly cleaned up after a run.

[TheAuditorTool]

Both issues in this thread should be fixed. Please review PRs.

[davewichers]

@darkspirit510 - Another SonarQube related PR from @TheAuditorTool for you to review.