GhApi.get_repo_contents method

[KeremTurgutlu]

This PR adds couple of utility methods to get repo file contents, which can be useful for dynamically creating the LLM context. Tested these changes locally with my editable builld.

get_repo_files(self:GhApi, owner, repo, branch="main", inc=None, exc=None):

This method returns all the repo files recursively using git.get_tree as recommended in the official GH api docs and allows filtering by passing inc:included and exc:excluded Unix-style wildcards using fnmatch. If a file matches one of the patterns in both included and excluded lists, then * expression is prioritized, if both have * expression than the longer expression is prioritized. This is useful for some common filtering cases, for example "Exclude all .md files expect for README.md", or "Include all .py files except for tests".

get_file_content(self:GhApi, path, owner, repo, branch="main"):

Gets file content of given file and also decodes the file content.

get_repo_contents(self:GhApi, owner, repo, **kwargs):

This uses GhApi.get_repo_files + parallel GhApi.get_file_content to get all repo contents.

More examples and explanations can be found in the core nb.

[jph00]

@KeremTurgutlu sorry I forgot to look at this! It looks great :D My only request is that you cut back on the output so that the docs are more concise. We really like things to be concise! Having more stuff for the user to read is a burden that we should only require where the upside outweighs the downside. So have a look at each line of code and each line of output and think about whether reaches that bar. If it doesn't, cut it back until it does.

[jph00]

@KeremTurgutlu just bumping this so we don't lose it ^^^

[KeremTurgutlu]

@jph00 thanks for the reminder!

I wonder if this is still needed given that we already have repo2ctx and also the more powerful gh_query?

[jph00]

Repo2ctx requires a full clone, so your way (particularly the file listing) seems like a win to me! :)

[KeremTurgutlu]

@jph00 I've made the following changes to make the code simpler and cleaner. Not sure why all the cell ids changed after nbdev_export

Logic:

• Simplified _include to rsync/grep model (exclude always wins), which should help with LLM tool use
• Rewrote get_repo_files with L().filter() chaining

Tests:

• Updated test cases for "exclude wins" behavior

Documentation:

• Added note explaining rsync/grep model + LLM familiarity
• Condensed outputs (3-4 examples instead of 40+, sliced content to 500 chars / 2-3 files)
• Deleted "excluded files" list cell
• Removed build-up narrative (single clean definition now)
• Fixed misleading "exclude all md except README" notes

Minor Change

• Created a clone of ghapi-test repo in AAI org and using it now to avoid permission issues

[jph00]

Thanks @KeremTurgutlu ! :)

[jph00]

Oh figured out the ids issue - we had clean_ids=True. Fixed now :)