Sile needs a true verbatim environment

[pdenapo]

The verbatim package says that

"Despite the name, verbatim does not alter the way that SILE sees special characters. You still need to escape backslashes and braces: to produce a backslash, you need to write \."

However, this is not for some use cases. For instance, you would like to use SILE to create a book (or course materials) on a programming language. Then you would
like to include your code without any modifications. This is crucial so that you can
actually run the code and check that it still works!

So I would suggest to add something like

\verbatim[src= filename]

that inserts the file without any escaping.

Currently you can do something like,

\begin{verbatim}
\include[src=table.py]
\end{verbatim}

but this does not work if your program includes some \

Also if you do something like

\begin{verbatim}
\include[src=test.sil]
\end{verbatim}

and test is the very same file in which you are including your code, you get an infinite
loop!

Also something like

\begin{code}
\include[src=table.py]
\end{code}

is bad. Since the code environment breaks the indentation, which is meaningful in
many languages like python. (The verbatim environment example above is not also nice since it converts tabs characters to spaces)

(Also I saw that some code in the packages use the raw environment but it
seems to be documented nowhere in the book)

In the future, It would be nice to have an environment for syntax highlighting (something like the listings package in LaTeX). There is a library in rust that might help

https://github.com/trishume/syntect

and there are some minimal lua bindings at

https://github.com/hishamhm/lua-syntect

I've been playing with that with the idea of contributing a syntax highlighting
package to SILE, but I haven''t manage to make it work yet.

[Omikhleia]

First, this is very tied to the SIL languange, not SILE itself.

For SIL, the tools are somewhat here to do it, despite what the poorly named "verbatim" package says.

Have you considered using a raw handler? There are not fully documented (see #1854) but do work quite the expected way, with examples here and there.

To get a grasp, you might want to try....

\begin[type=text]{raw}
Some stuff
\end[type=text]{raw}

or type=autodoc:code in a raw environment too, to see what we do here.

or, finally, implement your own raw handler in a package.

When I say above that it is "tied to the SIL language", what I mean is that some other "inputter" (= support logic for an input syntax) might have its own provision for such things.

E.g. my markdown.sile package obviously supports Markdown or Djot code blocks (even with "naive" Lua code syntax highlighting, since this was easy to achieve with just what we have without additional library -- This "naive" approach to Lua code highlighting used in markdown.sile could be proposed here upstream, and I once fancied doing it so the manual could have nicer Lua code samples... but I never had the time to work on this).

There is one deeper thing though, whatever solution is used; The way SILE implements obeying to lines in such environments is not perfect (it doesn't honor blank lines, which are skipped). This would need to be addressed, possibly.

Then, as of "general" syntax highlighting, this is probably best to be handled in a 3rd-party package rather than in the core distribution. I once experimented using the (Lua-based) highlighters from Scintillua (formerly part of TextAdept, and see also some Lua module based on it), but never had the time to complete something clean either. Yet, this is typically a case where I wouldn't care the logic being in Lua or not -- and I'd certainly want to use a good syntax highlighting package if available, however it does the trick.

In other terms, your issue deals with 3 possibly separate topics:

1. How to process raw content in SIL, and there the raw environment might be of use (documentation issue, as of Explain raw handlers and reserved SIL keywords #1854)
2. How to honor lines properly in such environment(s), and there, something should be done in SILE (bug fix issue)
3. How to syntax highlight code, and there some good 3rd party package would be welcome, whether based on solutions in Lua, C or Rust. (enhancement, but probably not in the core distribution, so I am not adding the label here ;))

I'd be happy to see some progress on 2 or 3 ;)

BTW

For instance, you would like to use SILE to create a book (or course materials) on a programming language.

Personally, I wouldn't use the SIL language for such a book. Markdown or Djot are cool enough , and more ubiquitous. But heh, people use the languages of their choice ;)

[Omikhleia]

BTW, code syntax highlighting was also discussed here (Closed, but some ideas might still be worth)

[pdenapo]

Many thanks! Your markdown package indeed did a much better job for my use case!

[Omikhleia]

I tried to deal with (1) (= #1854) in PR #2031. As noted above (3) (= syntax highlighting) was the subject of prior discussions and would be best addressed in a 3rd-party package.

So we are only left with (2) in this issue, that is how to address the idiosyncrasies of the ill-named verbatim package.