Skip to content

Add ShowSpecialCharacters and ShowStringCharacters options and round-trip FullForm output #1763

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
mmatera merged 12 commits into from
Mar 29, 2026
Merged

Add ShowSpecialCharacters and ShowStringCharacters options and round-trip FullForm output #1763

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
12 commits
Select commit Hold shift + click to select a range
40740d8
partial
mmatera Mar 29, 2026
5d7b713
adjust render OutputForm. adjust tests.
mmatera Mar 29, 2026
38e3a5d
Remove print statement from files.py
mmatera Mar 29, 2026
3c36644
Update layout.py
mmatera Mar 29, 2026
36e8908
Rename function to string_to_invertible_ascii
mmatera Mar 29, 2026
25504f5
Change import from string_to_invertible_ansi to ascii
mmatera Mar 29, 2026
848d87b
Update layout.py
mmatera Mar 29, 2026
bb738c6
remove not strictly needed changes
mmatera Mar 29, 2026
4339b0b
Merge remote-tracking branch 'origin/master' into full_form_encodes_i…
mmatera Mar 29, 2026
2dc1b94
adjust ShowStringCharacters in OutputForm
mmatera Mar 29, 2026
ade5665
revert changes in mathice.core.evaluation
mmatera Mar 29, 2026
27bbf63
update SYMBOLS_MANIFEST
mmatera Mar 29, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions SYMBOLS_MANIFEST.txt
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -1105,6 +1105,7 @@ System`ShortRightArrow
System`ShortUpArrow
System`Shortest
System`Show
System`ShowSpecialCharacters
System`ShowStringCharacters
System`Sign
System`Simplify
Expand Down
18 changes: 18 additions & 0 deletions mathics/builtin/box/layout.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -459,6 +459,24 @@ def is_multiline(self) -> bool:
return any(item.is_multiline for item in self.items)


class ShowSpecialCharacters(Builtin):
"""
<url>
:WMA link:
https://reference.wolfram.com/language/ref/ShowSpecialCharacters.html</url>
<dl>
<dt>'ShowSpecialCharacters'
<dd>is an option for 'Style' and 'Cell' that directs whether non-ASCII characters must be shown as special characters or by escaped sequences.
</dl>

<ul>
<li>With 'ShowSpecialCharacters' set to 'False', special characters are always displayed by name when possible.
</ul>
"""

summary_text = "cell and style option directing whether show special characters in a reversible ASCII format."


class ShowStringCharacters(Builtin):
"""
<url>
Expand Down
5 changes: 4 additions & 1 deletion mathics/builtin/layout.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -547,7 +547,10 @@ class Style(Builtin):
"""

summary_text = "wrapper for styles and style options to apply"
options = {"ImageSizeMultipliers": "Automatic"}
options = {
"ImageSizeMultipliers": "Automatic",
"$OptionSyntax": "Ignore",
}
rules = {
"MakeBoxes[Style[expr_, OptionsPattern[Style]], f_]": (
"StyleBox[MakeBoxes[expr, f], "
Expand Down
40 changes: 39 additions & 1 deletion mathics/core/convert/op.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -12,6 +12,21 @@
)

ascii_operator_to_symbol = NAMED_CHARACTERS_COLLECTION["ascii-operator-to-symbol"]
CHARACTER_TO_NAME = {
Copy link
Copy Markdown
Member

@rocky rocky Mar 29, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Please move this to Mathics3-scanner. Thanks.

Copy link
Copy Markdown
Contributor Author

@mmatera mmatera Mar 29, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I will, but first I wanted to be sure that this is the right approach. Thoughts?

Copy link
Copy Markdown
Member

@rocky rocky Mar 29, 2026 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I find this approach of putting in a PR, then discussing whatever it is that happens to be in it (you figure it out), really hard to follow.

Many people start with a problem and then go to a solution, instead of writing some code based on something that feels wrong (is this what is meant by "vibe" coding?) and then looking at what's been created and discussing that.

If that's the way you have to work, well. okay. But maybe after all the vibe coding, we can have a discussion (independent of the code) about what's wrong. Then discuss ways to address that.

I had thought we were going to start to do that for #1735, which I had imagined was taking that code and breaking it up into pieces. You know, like option information from a built-in (CharacterEncoding, of ToString ) is not filtering down to rendering routines. How do we do that? Do we add **kwargs parameters to the methods or split out the relevant ones (like encoding)?

I admit that there are bigger issues we want to solve, but I offer this as a specific example of something where we can break off a small, isolated problem (independent of the larger issue) and create a PR for that.

Or decide to hold off on that until the bigger picture is decided.

Instead, we are now on to a related topic with code that is outside of #1735.

So be it.

Okay. Now that you've come across this other thing and written some code so you might be able to understand something about it, can we just forget about the code (for now), and describe what the problem is in human language, and then what the approaches for handling this are?

Copy link
Copy Markdown
Member

@rocky rocky Mar 29, 2026 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Okay. Now that you've come across this other thing and written some code so you might be able to understand something about it, can we just forget about the code (for now), and describe what the problem is in human language, and then what the approaches for handling this are?

I get from the PR comment (probably written after the code) that we should add the option ShowSpecialCharacters, which is used in Style and StyleBox.

Instead of the code, though, describe in human language what the issues or approaches are and what implications those might have.

(I write "human language" because I understand English may be awkward for you (as it is for me)).

If you want to think and describe in Spanish, that's okay, I'll use Google Translate. The main thing is to express the idea independent of specific code.

Copy link
Copy Markdown
Contributor Author

@mmatera mmatera Mar 29, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I find this approach of putting in a PR, then discussing whatever it is that happens to be in it (you figure it out), really hard to follow.

OK: the wordy version of this PR would be: we need conversion tables for

  1. Take any str in the Mathics3 inner encoding, and convert it into an ASCII representation in an invertible way. This is required for FullForm.
  2. Take any str in the Mathics3 inner encoding, and convert it into an ASCII representation that be visually close to the character that the internal character represents.

I believe I mentioned this earlier; if not, my apologies.

Many people start with a problem and then go to a solution, instead of writing some code based on something that feels wrong (is this what is meant by "vibe" coding?) and then looking at what's been created and discussing that.

If that's the way you have to work, well. okay. But maybe after all the vibe coding, we can have a discussion (independent of the code) about what's wrong. Then discuss ways to address that.

Again, I though we have already that discussion. Now I am just proposing an implementation for it. And for it, I feel easier to show the code of the implementation instead of trying to figure out how to translate from Physicist-Spanish to Computer-Science English.

I had thought we were going to start to do that for #1735, which I had imagined was taking that code and breaking it up into pieces. You know, like option information from a built-in (CharacterEncoding, of ToString ) is not filtering down to rendering routines. How do we do that? Do we add **kwargs parameters to the methods or split out the relevant ones (like encoding)?

I am doing the work of spliting in pieces. Now I put another of these pieces, related to MathMLForm. There are coming more.

I admit that there are bigger issues we want to solve, but I offer this as a specific example of something where we can break off a small, isolated problem (independent of the larger issue) and create a PR for that.

What this PR tries to solve is to have an output from FullForm that can be copies from any front-end, copy to another front end, and produce exactly the same code. Then, we can compare in tests results and expected results disregarding of the encoding.

Or decide to hold off on that until the bigger picture is decided.

That is the bigger picture

Instead, we are now on to a related topic with code that is outside of #1735.

So be it.

Okay. Now that you've come across this other thing and written some code so you might be able to understand something about it, can we just forget about the code (for now), and describe what the problem is in human language, and then what the approaches for handling this are?

I am going to update the PR description to focus more on its central aspect.

char: rf"\[{name}]"
for name, char in NAMED_CHARACTERS_COLLECTION["named-characters"].items()
}


ESCAPE_CODE_BY_DIGITS = {
1: r"\.0",
2: r"\.",
3: r"\:0",
4: r"\:",
5: r"\|0",
6: r"\|",
}

builtin_constants = NAMED_CHARACTERS_COLLECTION["builtin-constants"]
operator_to_unicode = NAMED_CHARACTERS_COLLECTION["operator-to-unicode"]
operator_to_ascii = NAMED_CHARACTERS_COLLECTION["operator-to-ascii"]
Expand All @@ -22,7 +37,6 @@
UNICODE_TO_AMSLATEX = NAMED_CHARACTERS_COLLECTION.get("unicode-to-amslatex", {})
UNICODE_TO_LATEX = NAMED_CHARACTERS_COLLECTION.get("unicode-to-latex", {})


AMSTEX_OPERATORS = {
NAMED_CHARACTERS["Prime"]: "'",
NAMED_CHARACTERS["Prime"] * 2: "''",
Expand All @@ -48,6 +62,30 @@
}


def string_to_invertible_ascii(string: str):
"""
Replace non-ANSI characters with their names. If the character
does not have a name, use the WMA hex character code form.
Passing the string through `evaluation.parse` brings back
the original string.
This is used in particular for rendering `FullForm` expressions,
and when `Style` is called with both the options
`ShowStringCharacters->True` and `ShowSpecialCharacters->False`.
"""
result = ""
for c in string:
ord_c = ord(c)
if ord_c < 128:
result += c
else:
named = CHARACTER_TO_NAME.get(c, None)
if named is None:
named = hex(ord_c)[2:]
named = ESCAPE_CODE_BY_DIGITS[len(named)] + named
result += named
return result


def is_named_operator(str_op):
if len(str_op) < 3:
return False
Expand Down
55 changes: 45 additions & 10 deletions mathics/format/form/outputform.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -25,7 +25,14 @@
from mathics.core.expression import BoxError, Expression
from mathics.core.list import ListExpression
from mathics.core.number import dps
from mathics.core.symbols import Atom, Symbol, SymbolFullForm, SymbolList, SymbolTimes
from mathics.core.symbols import (
Atom,
Symbol,
SymbolFullForm,
SymbolList,
SymbolTimes,
SymbolTrue,
)
from mathics.core.systemsymbols import (
SymbolDerivative,
SymbolInfinity,
Expand Down Expand Up @@ -88,8 +95,8 @@ def _default_render_output_form(
if isinstance(expr, Atom):
result = expr.atom_to_boxes(SymbolOutputForm, evaluation)
if isinstance(result, String):
return result.value
return result.to_text()
return result.to_text(**kwargs)
return result.to_text(**kwargs)

expr_head = expr.head
head = render_output_form(expr_head, evaluation, **kwargs)
Expand Down Expand Up @@ -835,12 +842,24 @@ def _slotsequence_outputform_text(expr: Expression, evaluation: Evaluation, **kw

@register_outputform("System`String")
def string_render_output_form(expr: String, evaluation: Evaluation, **kwargs) -> str:
# lines = expr.value.split("\n")
# max_len = max([len(line) for line in lines])
# lines = [line + (max_len - len(line)) * " " for line in lines]
# return "\n".join(lines)
value = expr.value
return value
from mathics.format.render.text import string as render_string

# To render a string in OutputForm, we use the
# function that render strings from Boxed expressions.
# When a String object is converted into Boxes,
# MakeBoxes enclose the original string with quotes.
# Then, depending on the value of the option
# `System`ShowStringCharacters`, these quotes are render or not.
# If this option is set to `False`, the added quotes are removed.
# Here we are not going through that route: if
# `System`ShowStringCharacters` is set to True, add the quotes:
if kwargs.get("System`ShowStringCharacters", None) is SymbolTrue:
expr = String(f'"{expr.value}"')
else:
# if not, set "System`ShowStringCharacters" to True,
# to avoid remove quotes that was there before formatting:
kwargs["System`ShowStringCharacters"] = SymbolTrue
return render_string(expr, **kwargs)


@register_outputform("System`StringForm")
Expand Down Expand Up @@ -940,7 +959,23 @@ def style_to_outputform_text(expr: Expression, evaluation: Evaluation, **kwargs)
elements = expr.elements
if not elements:
raise _WrongFormattedExpression
return render_output_form(elements[0], evaluation, **kwargs)
elem, *style_and_options = elements
options = {}
if style_and_options:
style, *style_and_options = style_and_options
option = style.get_option_values(evaluation)
if option is not None:
options.update(option)

for opt_arg in style_and_options:
option = opt_arg.get_option_values(evaluation)
if option is None:
raise _WrongFormattedExpression
for opt, val in option.items():
options[opt] = val

kwargs.update(options)
return render_output_form(elem, evaluation, **kwargs)


@register_outputform("System`Symbol")
Expand Down
10 changes: 9 additions & 1 deletion mathics/format/render/text.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -20,13 +20,14 @@
TagBox,
)
from mathics.core.atoms import String
from mathics.core.convert.op import string_to_invertible_ascii
from mathics.core.exceptions import BoxConstructError
from mathics.core.formatter import (
add_render_function,
convert_box_to_format,
convert_inner_box_field,
)
from mathics.core.symbols import Atom, SymbolTrue
from mathics.core.symbols import Atom, SymbolFalse, SymbolTrue
from mathics.format.box.graphics import prepare_elements as prepare_elements2d
from mathics.format.box.graphics3d import prepare_elements as prepare_elements3d
from mathics.format.form.util import _WrongFormattedExpression, text_cells_to_grid
Expand Down Expand Up @@ -156,6 +157,13 @@ def string(s: String, **options) -> str:
show_string_characters = (
options.get("System`ShowStringCharacters", None) is SymbolTrue
)
show_special_characters = (not show_string_characters) or (
not (options.get("System`ShowSpecialCharacters", None) is SymbolFalse)
)

if not show_special_characters:
value = string_to_invertible_ascii(value)

if value.startswith('"') and value.endswith('"'): # nopep8
if not show_string_characters:
value = value[1:-1]
Expand Down
16 changes: 10 additions & 6 deletions test/format/format_tests.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -65,17 +65,19 @@
'"\[Pi] is a trascendental number"':
msg: A String
latex:
System`InputForm: "\\text{``$\\pi$ is a trascendental number''}"
# TODO: This should be fixed later
# System`InputForm: "\\text{``$\\pi$ is a trascendental number''}"
System`OutputForm: "\\text{$\\pi$ is a trascendental number}"
System`StandardForm: "\\text{$\\pi$ is a trascendental number}"
System`TraditionalForm: "\\text{$\\pi$ is a trascendental number}"
mathml:
System`InputForm: "<ms>\u03C0&nbsp;is&nbsp;a&nbsp;trascendental&nbsp;number</ms>"
# TODO: THis should be fixed later
# System`InputForm: "<ms>\u03C0&nbsp;is&nbsp;a&nbsp;trascendental&nbsp;number</ms>"
System`OutputForm: "<mtext>\u03C0&nbsp;is&nbsp;a&nbsp;trascendental&nbsp;number</mtext>"
System`StandardForm: "<mtext>\u03C0&nbsp;is&nbsp;a&nbsp;trascendental&nbsp;number</mtext>"
System`TraditionalForm: "<mtext>\u03C0&nbsp;is&nbsp;a&nbsp;trascendental&nbsp;number</mtext>"
text:
System`InputForm: "\"\u03C0 is a trascendental number\""
System`InputForm: '"\[Pi] is a trascendental number"'
System`OutputForm: "\u03C0 is a trascendental number"
System`StandardForm: "\u03C0 is a trascendental number"
System`TraditionalForm: "\u03C0 is a trascendental number"
Expand Down Expand Up @@ -440,7 +442,8 @@ Graphics[{}]:
"Grid[{{\"Spanish\", \"Hola!\"},{\"Portuguese\", \"Ol\xE0!\"},{\"English\", \"Hi!\"}}]":
msg: Strings in a GridBox
latex:
System`InputForm: \text{Grid[\{\{"Spanish", "Hola!"\}, \{"Portuguese", "Ol\`{a}!"\}, \{"English", "Hi!"\}\}]}
# TODO: This should be fixed later
# System`InputForm: \text{Grid[\{\{"Spanish", "Hola!"\}, \{"Portuguese", "Ol\`{a}!"\}, \{"English", "Hi!"\}\}]}
System`OutputForm: '\text{Spanish Hola!\newline

\newline
Expand All @@ -458,12 +461,13 @@ Graphics[{}]:
System`TraditionalForm: "\\begin{array}{cc} \\text{Spanish} & \\text{Hola!}\\\\\
\ \\text{Portuguese} & \\text{Ol\\`{a}!}\\\\ \\text{English} & \\text{Hi!}\\end{array}"
mathml:
System`InputForm: "<mtext>Grid[{{&quot;Spanish&quot;,&nbsp;&quot;Hola!&quot;},&nbsp;{&quot;Portuguese&quot;,&nbsp;&quot;Olà!&quot;},&nbsp;{&quot;English&quot;,&nbsp;&quot;Hi!&quot;}}]</mtext>"
# TODO: This should be adjusted later
# System`InputForm: "<mtext>Grid[{{&quot;Spanish&quot;,&nbsp;&quot;Hola!&quot;},&nbsp;{&quot;Portuguese&quot;,&nbsp;&quot;Olà!&quot;},&nbsp;{&quot;English&quot;,&nbsp;&quot;Hi!&quot;}}]</mtext>"
System`OutputForm: '<mtext>Spanish&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;Hola!<mspace linebreak="newline" /><mspace linebreak="newline" />Portuguese&nbsp;&nbsp;&nbsp;Olà!<mspace linebreak="newline" /><mspace linebreak="newline" />English&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;Hi!<mspace linebreak="newline" /></mtext>'
System`StandardForm: "<mtable columnalign=\"center\">\n <mtr>\n <mtd columnalign=\"center\">\n <mtext>Spanish</mtext>\n </mtd>\n <mtd columnalign=\"center\">\n <mtext>Hola!</mtext>\n </mtd>\n </mtr>\n <mtr>\n <mtd columnalign=\"center\">\n <mtext>Portuguese</mtext>\n </mtd>\n <mtd columnalign=\"center\">\n <mtext>Olà!</mtext>\n </mtd>\n </mtr>\n <mtr>\n <mtd columnalign=\"center\">\n <mtext>English</mtext>\n </mtd>\n <mtd columnalign=\"center\">\n <mtext>Hi!</mtext>\n </mtd>\n </mtr>\n</mtable>"
System`TraditionalForm: "<mtable columnalign=\"center\">\n <mtr>\n <mtd columnalign=\"center\">\n <mtext>Spanish</mtext>\n </mtd>\n <mtd columnalign=\"center\">\n <mtext>Hola!</mtext>\n </mtd>\n </mtr>\n <mtr>\n <mtd columnalign=\"center\">\n <mtext>Portuguese</mtext>\n </mtd>\n <mtd columnalign=\"center\">\n <mtext>Olà!</mtext>\n </mtd>\n </mtr>\n <mtr>\n <mtd columnalign=\"center\">\n <mtext>English</mtext>\n </mtd>\n <mtd columnalign=\"center\">\n <mtext>Hi!</mtext>\n </mtd>\n </mtr>\n</mtable>"
text:
System`InputForm: "Grid[{{\"Spanish\", \"Hola!\"}, {\"Portuguese\", \"Ol\xE0!\"\
System`InputForm: "Grid[{{\"Spanish\", \"Hola!\"}, {\"Portuguese\", \"Ol\\[AGrave]!\"\
}, {\"English\", \"Hi!\"}}]"
System`OutputForm: "Spanish Hola!\n\nPortuguese Ol\xE0!\n\nEnglish \
\ Hi!\n"
Expand Down
Loading