Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
54 changes: 35 additions & 19 deletions docs/internals/extensions/source_code_linker.md
Original file line number Diff line number Diff line change
Expand Up @@ -12,6 +12,7 @@ The feature is split into **two layers**:

1. **Bazel pre-processing (before Sphinx runs)**
Generates and aggregates JSON caches containing the raw `source_code_link` findings (and, if available, repository metadata like `repo_name/hash/url`).
Note that "hash" might also be a "version" tag.

2. **Sphinx extension (during the Sphinx build)**
Reads the aggregated JSON, validates and adapts it, and finally injects the links into Sphinx needs in the final layout (**RepoSourceLink**).
Expand All @@ -36,6 +37,7 @@ one JSON cache per repository.
It also adds metadata to each needlink that is needed in further steps.

Example of requirement tags:

```python
# Choose one or the other, both mentioned here to avoid detection
# req-Id/req-traceability: <NEED_ID>
Expand Down Expand Up @@ -83,20 +85,23 @@ This step also fills in url & hash if there is a known_good_json provided (e.g.

(repo-metadata-rules)=
#### Repo metadata rules

Here are some basic rules regarding the MetaData information

In a combo build, a known_good_json **must** be provided.
If known_good_json is found then the hash & url will be filled for each need by the script.

Combo build:
- `repo_name`: repository name (parsed from filepath in step 1)
- `hash`: revision/commit hash (as provided by the known_good_json)
- `url`: repository remote URL (as provided by the known_good_json)

- `repo_name`: repository name (parsed from filepath in step 1)
- `hash`: revision/commit hash (as provided by the known_good_json)
- `url`: repository remote URL (as provided by the known_good_json)

Local build:
- `repo_name`: 'local_repo'
- `hash`: will be empty at this point, and later filled via parsing the git commands
- `url`: will be empty at this point, and later filled via parsing the git commands

- `repo_name`: 'local_repo'
- `hash`: will be empty at this point, and later filled via parsing the git commands
- `url`: will be empty at this point, and later filled via parsing the git commands

---

Expand Down Expand Up @@ -140,6 +145,7 @@ def test_feature():
...

```

> Note: If you use the decorator, it will check that you have specified a docstring inside the function.

#### Data Flow
Expand All @@ -153,7 +159,7 @@ def test_feature():
- Result (e.g. passed, failed, skipped)
- Result text (if failed/skipped will check if message was attached to it)
- Verifications (`PartiallyVerifies`, `FullyVerifies`)
- Also parses metadata according to the [metadata rules](#repo-metadata-rules)
- Also parses metadata according to the [metadata rules](#repo-metadata-rules)

- Cases without metadata are logged out as info (not errors).
- Test cases with metadata are converted into:
Expand All @@ -168,7 +174,9 @@ def test_feature():
- Warns on missing need IDs.

#### Example JSON Cache (DataFromTestCase)

The DataFromTestCase depicts the information gathered about one testcase.

```json
[
{
Expand Down Expand Up @@ -246,6 +254,7 @@ Instead of repeating `repo_name/hash/url` for every single link entry, the final
- All links belonging to that repository are stored beneath it

This somewhat looks like this:

```{code-block} json
[
{
Expand Down Expand Up @@ -296,20 +305,22 @@ During the Sphinx build process, the extension applies the computed links to nee
## Known Limitations

### General
- In combo builds, known_good_json is required.
- inefficiencies in creating the links and saving the JSON caches
- Not compatible with **Esbonio/Live_preview**

- In combo builds, known_good_json is required.
- inefficiencies in creating the links and saving the JSON caches
- Not compatible with **Esbonio/Live_preview**

### Codelinks
- GitHub links may 404 if the commit isn’t pushed
- Tags must match exactly (e.g. #<!-- comment prevents parsing this occurance --> req-Id)
- `source_code_link` isn’t visible until the full Sphinx build is completed

- GitHub links may 404 if the commit isn’t pushed
- Tags must match exactly (e.g. #<!-- comment prevents parsing this occurance --> req-Id)
- `source_code_link` isn’t visible until the full Sphinx build is completed

### TestLink

- GitHub links may 404 if the commit isn’t pushed
- XML structure must be followed exactly (e.g. `properties & attributes`)
- Relies on test to be executed first
- GitHub links may 404 if the commit isn’t pushed
- XML structure must be followed exactly (e.g. `properties & attributes`)
- Relies on test to be executed first

---

Expand All @@ -332,16 +343,19 @@ rm -rf _build/
## Internal Overview

The bazel part:
```

```text
scripts_bazel/
├── BUILD # Declare libraries and filegroups needed for bazel
├── generate_sourcelinks_cli.py # Bazel step 1 => Parses sourcefiles for tags
├── merge_sourcelinks.py
└── tests
│ └── ...
```

The Sphinx extension
```

```text
score_source_code_linker/
├── __init__.py # Main Sphinx extension; combines CodeLinks + TestLinks
├── generate_source_code_links_json.py # Most functionality moved to 'scripts_bazel/generate_sourcelinks_cli'
Expand All @@ -356,6 +370,7 @@ score_source_code_linker/
```

---

## Clearing Cache Manually

To clear the build cache, run:
Expand All @@ -364,7 +379,8 @@ To clear the build cache, run:
rm -rf _build/
```

## Examples:
## Examples

To see working examples for CodeLinks & TestLinks, take a look at the Docs-As-Code documentation.

[Example CodeLink](https://eclipse-score.github.io/docs-as-code/main/requirements/requirements.html#tool_req__docs_common_attr_id_scheme)
Expand Down
7 changes: 6 additions & 1 deletion src/extensions/score_source_code_linker/helpers.py
Original file line number Diff line number Diff line change
Expand Up @@ -107,5 +107,10 @@ def parse_info_from_known_good(
for category in kg_json["modules"].values():
if repo_name in category:
m = category[repo_name]
return (m["hash"], m["repo"].removesuffix(".git"))
hash_or_version = m.get("hash") or m.get("version")
if hash_or_version is None:
raise KeyError(
f"Module {repo_name} has neither 'hash' nor 'version' key."
)
return (hash_or_version, m["repo"].removesuffix(".git"))
raise KeyError(f"Module {repo_name} not found in known_good_json.")
70 changes: 70 additions & 0 deletions src/extensions/score_source_code_linker/tests/test_helpers.py
Original file line number Diff line number Diff line change
Expand Up @@ -189,6 +189,76 @@ def test_parse_info_from_known_good_empty_repo_dict_in_json(tmp_path: Path):
parse_info_from_known_good(json_file, "any_repo")


VALID_KNOWN_GOOD_WITH_VERSION = {
"modules": {
"target_sw": {
"score_baselibs": {
"repo": "https://github.com/eclipse-score/baselibs.git",
"version": "v1.2.3",
},
},
"tooling": {
"score_docs_as_code": {
"repo": "https://github.com/eclipse-score/docs-as-code.git",
"version": "4.3-alpha",
}
},
}
}


@pytest.fixture
def known_good_json_with_version(tmp_path: Path):
"""Providing a known_good.json file that uses 'version' instead of 'hash'."""
json_file = tmp_path / "known_good_version.json"
_ = json_file.write_text(json.dumps(VALID_KNOWN_GOOD_WITH_VERSION))
return json_file


def test_parse_info_from_known_good_with_version(known_good_json_with_version: Path):
"""Test that 'version' is accepted as a fallback when 'hash' is absent."""
hash_result, repo_result = parse_info_from_known_good(
known_good_json_with_version, "score_baselibs"
)

assert hash_result == "v1.2.3"
assert repo_result == "https://github.com/eclipse-score/baselibs"


def test_parse_info_from_known_good_with_version_different_category(
known_good_json_with_version: Path,
):
"""Test that 'version' works for a module in a different category."""
hash_result, repo_result = parse_info_from_known_good(
known_good_json_with_version, "score_docs_as_code"
)

assert hash_result == "4.3-alpha"
assert repo_result == "https://github.com/eclipse-score/docs-as-code"


def test_parse_info_from_known_good_neither_hash_nor_version(tmp_path: Path):
"""Test that KeyError is raised when neither 'hash' nor 'version' is present."""
json_file = tmp_path / "broken.json"
_ = json_file.write_text(
json.dumps(
{
"modules": {
"target_sw": {
"score_baselibs": {
"repo": "https://github.com/eclipse-score/baselibs.git",
}
}
}
}
)
)

msg = "score_baselibs has neither 'hash' nor 'version'"
with pytest.raises(KeyError, match=msg):
parse_info_from_known_good(json_file, "score_baselibs")


# Tests for get_github_link_from_json
def test_get_github_link_from_json_happy_path():
"""
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -21,6 +21,8 @@
Once we enable those we will need to change the tests
"""

import json
import os
import xml.etree.ElementTree as ET
from collections.abc import Callable
from pathlib import Path
Expand Down Expand Up @@ -382,3 +384,85 @@ def test_clean_test_file_name_empty_path_raises_error():
ValueError, match="Filepath does not have 'bazel-testlogs' nor 'tests-report'"
):
xml_parser.clean_test_file_name(raw_path)


# ╭──────────────────────────────────────────────────────────╮
# │ Tests for get_metadata_from_test_path │
# ╰──────────────────────────────────────────────────────────╯

_KNOWN_GOOD_WITH_HASH = {
"modules": {
"tooling": {
"score_docs_as_code": {
"repo": "https://github.com/eclipse-score/docs-as-code.git",
"hash": "abc123hashvalue",
}
}
}
}

_KNOWN_GOOD_WITH_VERSION = {
"modules": {
"tooling": {
"score_docs_as_code": {
"repo": "https://github.com/eclipse-score/docs-as-code.git",
"version": "v2.1.0",
}
}
}
}

_COMBO_TEST_PATH = Path(
"/root/bazel-testlogs/external/score_docs_as_code+/src/ext/score_foo/test.xml"
)


def test_get_metadata_from_test_path_local():
"""Local builds produce empty hash/url without reading known_good.json."""
local_path = Path(
"/home/root/docs-as-code/bazel-testlogs/src/extensions/foo/test.xml"
)
md = xml_parser.get_metadata_from_test_path(local_path)
assert md["repo_name"] == "local_repo"
assert md["hash"] == ""
assert md["url"] == ""


def test_get_metadata_from_test_path_combo_with_hash(tmp_path: Path):
"""Combo builds with 'hash' in known_good.json populate metadata correctly."""
json_file = tmp_path / "known_good.json"
json_file.write_text(json.dumps(_KNOWN_GOOD_WITH_HASH))

old = os.environ.get("KNOWN_GOOD_JSON")
try:
os.environ["KNOWN_GOOD_JSON"] = str(json_file)
md = xml_parser.get_metadata_from_test_path(_COMBO_TEST_PATH)
finally:
if old is None:
os.environ.pop("KNOWN_GOOD_JSON", None)
else:
os.environ["KNOWN_GOOD_JSON"] = old

assert md["repo_name"] == "score_docs_as_code"
assert md["hash"] == "abc123hashvalue"
assert md["url"] == "https://github.com/eclipse-score/docs-as-code"


def test_get_metadata_from_test_path_combo_with_version(tmp_path: Path):
"""Combo builds with 'version' in known_good.json populate metadata correctly."""
json_file = tmp_path / "known_good.json"
json_file.write_text(json.dumps(_KNOWN_GOOD_WITH_VERSION))

old = os.environ.get("KNOWN_GOOD_JSON")
try:
os.environ["KNOWN_GOOD_JSON"] = str(json_file)
md = xml_parser.get_metadata_from_test_path(_COMBO_TEST_PATH)
finally:
if old is None:
os.environ.pop("KNOWN_GOOD_JSON", None)
else:
os.environ["KNOWN_GOOD_JSON"] = old

assert md["repo_name"] == "score_docs_as_code"
assert md["hash"] == "v2.1.0"
assert md["url"] == "https://github.com/eclipse-score/docs-as-code"
Loading