Vendor full tokenized WikiText-2 corpus into core benchmark fixtures

[Copilot]

• Inspect repository dataset handling, script conventions, and documentation touchpoints
• Run baseline solution build and tests before making changes
• Add a local script to download/process full TinyLlama and WikiText-2 data into repository paths
• Document how to run the script and what files it updates
• Validate the solution build/tests still pass after the change
• Run final code review and security validation

---

💡 You can make Copilot smarter by setting up custom instructions, customizing its development environment and configuring Model Context Protocol (MCP) servers. Learn more Copilot coding agent tips in the docs.

[chatgpt-codex-connector]

You have reached your Codex usage limits for code reviews. You can see your limits in the Codex usage dashboard.
To continue using code reviews, you can upgrade your account or add credits to your account and enable them for code reviews in your settings.

[Copilot]

Pull request overview

This PR vendors the full pre-tokenized WikiText-2 corpus into BitNetSharp.Core as embedded resources and updates benchmark fixtures/tests/docs so benchmark/audit paths use repository-local data without external downloads.

Changes:

• Embed wiki.train/valid/test.tokens into BitNetSharp.Core and load them via BitNetBenchmarkFixtures.
• Expand fixture surface with split-specific properties (train/valid/test) and keep perplexity wired to validation.
• Add tests to validate split loading/counts and update benchmarking docs to reflect the new local corpus source.

Reviewed changes

Copilot reviewed 4 out of 7 changed files in this pull request and generated 3 comments.

File	Description
tests/BitNetSharp.Tests/HostedAgentBenchmarksExecutionTests.cs	Uses a smaller validation subset for perplexity test runtime; adds split-loading assertions for the new fixtures.
src/BitNetSharp.Core/BitNetSharp.Core.csproj	Embeds the vendored WikiText-2 token files as resources.
src/BitNetSharp.Core/BitNetBenchmarkFixtures.cs	Replaces placeholder WikiText-2 data with lazy-loaded embedded-resource splits.
docs/benchmarking.md	Documents that benchmark perplexity now uses the vendored local WikiText-2 validation split.

[sharpninja]

@copilot apply changes based on the comments in this thread

[Copilot]

@copilot apply changes based on the comments in this thread

Addressed in 679d00d. The WikiText-2 loader now preserves blank separator rows line-for-line, the fixture tests use shorter stable assertions instead of full long literals, and the benchmarking docs were updated to match the actual in-memory behavior.

[sharpninja]

@copilot diagnose and fix test failures in build pipeline.

[Copilot]

@copilot diagnose and fix test failures in build pipeline.

Addressed in 527f9df. I traced the pipeline issue to the build.yml Test step hanging after the vendored WikiText-2 corpus made benchmark/audit perplexity tests process very large samples. The fix keeps full-corpus behavior in production, adds a compact repository-local WikiText-2 subset for the affected tests through a shared helper, and the targeted plus full solution test runs now complete locally.