Can't replicate stats

[marklabz]

I spent an entire day working on trying to replicate the same result and it's impossible.

First, the code needed a lot of work to make it run from building the kg, triples,and graph. Mainly to use openrouter instead of ollama.

I created all the entities, triples, etc with gpt5-mini and I'm taking questions from each jsonl that it created and it can hardly answer any of them.

I'm not sure what I'm doing wrong, but I can't imagine the switch between ollama to openrouter would yield such a difference in performance.

[RaZzzyz]

The model used in our paper is deepseek-v3, and we employ the GraphExtraction knowledge graph construction method. (This is because CommonKG introduces Wikipedia entity filtering, which requires incorporating domain-specific entities for better performance.) Could you provide more detailed information, such as triple examples and dataset information, so we can continue our discussion?

[marklabz]

The model used in our paper is deepseek-v3, and we employ the GraphExtraction knowledge graph construction method. (This is because CommonKG introduces Wikipedia entity filtering, which requires incorporating domain-specific entities for better performance.) Could you provide more detailed information, such as triple examples and dataset information, so we can continue our discussion?

I was running this in code spaces and the data is lost now but I can recreate my steps again and share the data. Just to confirm, you're using Python 3.10? Because I had a hard time building the dependencies, due to version mismatches.

[RaZzzyz]

The Python version is 3.11

[marklabz]

@RaZzzyz I just created a pull request, but you can test it first in that branch. The data is stored in datasets/mix/mix_chunk

[RaZzzyz]

@marklabz Hello, I took a quick look at the entity list you generated. I suspect this issue is due to your use of CommonKG for graph generation. As I mentioned earlier, CommonKG requires a specialized entity list for optimal results. Our results in this article were also obtained using GraphExtraction. Additionally, your entity list seems too small. Is the NER capability of the model you are using weak? CommonKG typically extracts more entities during our evaluation.

[marklabz]

In the branch, it contains GraphExtraction data. I tried both approaches and shared the refactored code. Either way, my goal was to recreate your stats, and the current code structure wasn't working based on the instructions from the README. Thanks for looking into it, cheers.

[RaZzzyz]

@marklabz ，I see the questions. In my source code, I give a uncomplete file_chunk function to segement chunk. In your code, you segement the query as the context. In https://github.com/marklabz/LeanRAG/blob/feature-branch/file_chunk.py line 57,you should take the context value as text. Thanks to your contribution, I update my file_chunk.py in the latest version.