diff --git a/contributor_docs/reference_generation_process.md b/contributor_docs/reference_generation_process.md new file mode 100644 index 0000000000..afd9fd66af --- /dev/null +++ b/contributor_docs/reference_generation_process.md @@ -0,0 +1,415 @@ + + +# Reference-generation process + +This document will describe the documentation-generation process which takes the p5.js v2.x source code and generates from it the pages of the [reference website](https://beta.p5js.org/reference/). + +In addition, it will also cover the generation of other artifacts from the same source: +* type declarations supporting autocompletion, intellisense, and type-checking +* parameter validation data for the Friendly Error System (FES) + +You don't need to know _any_ of this to successfully write and maintain the reference comments in the p5.js library! If you've questions about that, see [Contributing to the p5.js reference](./contributing_to_the_p5js_reference.md). + +Warning: This describes how things work in February 2026 and is quite likely out of date! + +## Flow diagram for reference generation + +```mermaid +--- +title: Flow diagram of reference-generation +--- +graph TD + npmRunDocs[[npm run
docs]] --> docBuild[[documentation
build
/src/**/*.js]] + docBuild --> /docs/data.json@{ shape: doc } + /docs/data.json --> convert[[convert.mjs]] + convert --> /docs/reference/data.json@{ shape: doc } + convert --> /docs/reference/data.min.json@{ shape: doc } + /docs/reference/data.json --> websiteNpmBuildReference[[website
npm run
build:reference]] + + contentReference["/src/content
/reference/**/*.mdx"]@{shape: docs} + + websiteNpmBuildReference --> convertDocsToMDX + convertDocsToMDX --> contentReference + +``` + +## Reference generation explained + + + +The process is started with the execution of the following command from the p5.js-website repo: + +```bash +npm build:reference +``` + +This clones the p5.js repo into a working directory on the website and there runs: + +```bash +npm run docs +``` + +### p5.js: npm run docs + +You can run it locally and observe the outputs. + +It is currently run by [a release process workflow](../.github/workflows/release-workflow-v2.yml) on Github Actions CI. + +In turn this command runs the following single-line command: + +```bash +"documentation build + ./src/**/*.js ./src/**/**/*.js + -o ./docs/data.json + && node ./utils/convert.mjs" +``` + +This is actually _two_ commands in sequence: First, `documentation build ...`, and then - if the first command was successful `node ./utils/convert.mjs`. We'll look at these in turn. + +#### Step 1: `documentation build...` command + +This command turns JSDoc comment blocks across all the javascript files into a single structured json file ready for further processing. + +In more detail: + +```bash +documentation build + ./src/**/*.js ./src/**/**/*.js + -o ./docs/data.json +``` + +`documentation build` is the [standard way](https://github.com/documentationjs/documentation?tab=readme-ov-file) to invoke the tool [documentation.js](https://documentation.js.org/) to build documentation. + +Next we tell it which files to work on. + +`./src/**/*.js ./src/**/**/*.js` is an expression which will expand to match all .js files under the src directory at depths of one and two subdirectories. That covers all the .js files that currently have documentation comment blocks in them. + +Examples of files which will match: + +* `src/image/pixels.js` +* `src/color/color_spaces/hsb.js` + +* The `-o` is used to specify the output file. +* `documentation.js`'s default output format is JSON, so JSON content is written into the output file. + +It's important to note this output does not generate HTML pages, even though that's a common use for tools like documentation.js. + +This initial JSON file will be converted further by the next process, to prepare the data for the website build process. + +It will _also_ be used by another build process to build the types - see [the type-generation process](#type-generation-process) + +You can run this command yourself locally and inspect the output file to see the gathered data. + +You can also run `npx documentation build --help` to read the manual page for documentation.js. + +#### Step 2: `node ./utils/convert.mjs` command + +The second command is +```bash +node ./utils/convert.mjs +``` + +This will _only_ run if the documentation build command runs successfully. + +The `./utils/convert.mjs` script reads the output file generated by the `documentation build` command (the JSON file `/docs/data.json`), and creates three new output files: +* `/docs/reference/data.json` - Used by the p5.js-website to generate the reference pages +* `/docs/reference/data.min.json` - A smaller version of the above +* `/docs/parameterData.json` - Used by the FES\* + +The file generated initially by `documentation build` is _very_ large with a lot of data our pages won't need. Further, pieces of information are also scattered over the file - we'd like to group them together into a structure that will make it easier to use in later webpage generation. +The convert script has taken only what's necessary and grouped it together into the new data file `/docs/reference/data.json` and its minified version. + +This file will be used by the p5.js-website to generate the final pages. + +\* The convert script has _also_ generated `/docs/parameterData.json` which is used by the friendly error system (FES) at runtime to validate function parameters. We won't discuss that further here. + +#### Excerpt of /docs/reference.data.json + +Here's an excerpt from `/docs/reference/data.json`, showing how our `sin` function documentation is now represented at the end of both steps taken by `npm run docs`: + +```json +{ + "name": "sin", + "file": "src/math/trigonometry.js", + "line": 526, + "itemtype": "method", + "description": "

Calculates the sine of an angle.

\n

sin() is useful for many geometric tasks in creative coding. The values\nreturned oscillate between -1 and 1 as the input angle increases. sin()\ncalculates the sine of an angle, using radians by default, or according to\nif angleMode() setting (RADIANS or DEGREES).

\n", + "example": [ + "
\n\nfunction setup() {\n createCanvas(100, 100);\n\n describe('A white ball on a string oscillates up and down.');\n}\n\nfunction draw() {\n background(200);\n\n // Calculate the coordinates.\n let x = 50;\n let y = 30 * sin(frameCount * 0.05) + 50;\n\n // Draw the oscillator.\n line(50, y, x, y);\n circle(x, y, 20);\n}\n\n
\n\n
\n\nfunction setup() {\n createCanvas(100, 100);\n\n background(200);\n\n describe('A series of black dots form a wave pattern.');\n}\n\nfunction draw() {\n // Calculate the coordinates.\n let x = frameCount;\n let y = 30 * sin(x * 0.1) + 50;\n\n // Draw the point.\n point(x, y);\n}\n\n
\n\n
\n\nfunction setup() {\n createCanvas(100, 100);\n\n background(200);\n\n describe('A series of black dots form an infinity symbol.');\n}\n\nfunction draw() {\n // Calculate the coordinates.\n let x = 30 * cos(frameCount * 0.1) + 50;\n let y = 10 * sin(frameCount * 0.2) + 50;\n\n // Draw the point.\n point(x, y);\n}\n\n
" + ], + "overloads": [ + { + "params": [ + { + "name": "angle", + "description": "the angle, in radians by default, or according to if angleMode() setting (RADIANS or DEGREES).", + "type": "Number" + } + ], + "return": { + "description": "sine of the angle.", + "type": "Number" + } + } + ], + "return": { + "description": "sine of the angle.", + "type": "Number" + }, + "class": "p5", + "static": false, + "module": "Math", + "submodule": "Trigonometry" +}, +``` + +### after p5.js npm run docs - p5.js-website continues + +Once the p5.js `npm run docs` script has finished, the p5.js-website's `npm run build:reference` command continues, as follows: + +#### Summary: +Loads the generated `./docs/reference/data.json` +Generates an MDX file for each entry in the data. + +That's it! (The Astro web app will read these MDX files at runtime.) + +#### More detail: + + +For each module: + * generates the file path to which generated pages will be stored +For each feature: + * generates the MDX content for the relevant page + + + +e.g. here's one item loaded from the reference file: +```json +{ + itemtype: "property", + name: "RGB", + file: "src/scripts/parsers/in/p5.js/src/color/creating_reading.js", + line: 16, + type: "RGB", + description: "", + module: "Color", + submodule: "Creating & Reading", + class: "p5", +} +``` +Here's the path decided for it: +`'src/content/reference/en/p5/constants'` + +(TODO: why?) +### convertDocsToMDX + +This builds an array of objects: +```js +{ mdx, savePath, name: doc.name } +``` +and fills in a modulePath tree as it goes, with discovered module, submodule hierarchy. + +```js +addDocToModulePathTree(): + itemPath = 'p5/constants/RGB' + modulePath = "Color" + subPath= "Creating & Reading", + merges the following into modulePathTree: { + Color: { + "Creating & Reading": { + RGB: "p5/constants/RGB", + }, + }, + } +//fix doc description +correctRelativeLinksInDescription(doc.description) +correctRelativeLinksToExampleAssets(doc.description) + +//fix doc.example (a string[]) +doc.example = correctRelativeLinksToExampleAssets(doc.example) + +//generate MDX from the doc object +mdx = convertToMDX(doc); +#### convertToMDX(doc) + let frontMatterArgs = { + title: sanitizeName(doc.name), + module: doc.module, + submodule: doc.submodule ?? "", + file: doc.file.replace(/.*p5\.js\/(.*)/, "$1"), + description: doc.description ?? "", + line: doc.line, + deprecated: doc.deprecated, + } + //generate front matter as a string + //e.g. "---title: RGBmodule: Constantssubmodule: Creating & Readingfile: src/color/creating_reading.jsdescription: ''line: 16isConstructor: falseitemtype: propertyclass: p5type: RGB---" + + // Convert the markdown content to MDX + const mdxContent = remark().use(remarkMDX).processSync(markdownContent); + //returns this text: + example="---title: RGBmodule: Constantssubmodule: Creating & Readingfile: src/color/creating_reading.jsdescription: ''line: 16isConstructor: falseitemtype: propertyclass: p5type: RGB---\n# RGB\n" + + //remove the old .mdx files from the repo + //save the new MDX content objects to their appropriate paths + + +## Bonus: Flow diagram for search index generation + + +On the p5js-website repo, we run `npm run build:search` + + +See [mermaid flowchart syntax reference](https://mermaid.ai/open-source/syntax/flowchart.html) + +```mermaid +graph TD + contentReference["/src/content/reference"]@{shape: docs} + contentExamples["/src/content/examples"]@{shape: docs} + + npmBuildSearch[[npm run
build:search]] + npmBuildSearch --> buildSearchIndices[[buildSearchIndices]] --> genSearchIndexForExamples[[generateSearchIndex
examples]] + buildSearchIndices --> genSearchIndexForReference[[generateSearchIndex
reference]] + contentReference --> genSearchIndexForReference + contentExamples --> genSearchIndexForExamples + buildSearchIndices --> saveSearchIndex[[saveSearchIndex]] --> outDir[[public/search-indices en.json, ja.json, ...]]@{shape: docs} +``` +## Bonus: The type-generation process + +The JSON file `/docs/data.json` generated by `documentation build` from the p5.js source code is also used to generate type declarations - .d.ts files - which enable: + +* auto-completion & intellisense + - inline documentation on hover + - display of available methods and properties on the current object + - display of possible constants or object keys to pass as the current function argument +* type-checking (of javascript or typescript code) + +It is more common for such type-declarations to be generated by the TypeScript compiler (tsc) or similar. + + +The process is started by running the following command on the p5js repo: + +`npm run generate-types` runs: +```bash +"npm run docs && node utils/typescript.mjs" +``` + +###