Documentation rebuild #2837
Documentation rebuild #2837
Conversation
ethancrawford
force-pushed
the
documentation_rebuild
branch
from
6d4d48e
to
1260147
Compare
|
Hey @ethancrawford, Thank you for the great explanation and diagrams! |
|
Weird. I'll have to investigate that asap. I'll update in the next few hours... |
|
Ah. Probably a side effect of moving a bunch of those files into a sub-directory. Will fix it soon. |
|
@SunderB - out of curiosity, does |
|
|
|
Oh wait, it seems there were some left over untracked files in the wrong folder. (I first tried to rebase it on dev branch, but realised that had been done already) After stashing those, it works! |
|
I don't expect so, but how about |
|
I'm wondering where those files came from, since |
|
The website works, but there seems to be no css. (I think the css file was cleared in the git stash) I'll try resetting the repo to your branch again. |
|
Still no apparent CSS apart from a top loading bar, is that expected? Update: it seems like I'm missing Bootstrap |
|
hmm, no. I'll see if I can reproduce it maybe |
|
I downloaded Bootstrap 5.1.1 and placed the files in etc/docs/assets/static/(css/js), and it's working :) |
|
Ok 👍 |
|
Ah. It looks like the static folder is in the .gitignore 😂 😛 |
|
I've switched to using bootstrap from npm, so that it's easier to include it in the webpacker pipeline. The styling would likely be completely changed at some point if this was ever going to be used anyway. |
|
I'm just messing around with the phoenix HTML templates. Is there any way to set an attribute of an html element with a variables? (e.g. for links/anchors) |
|
My thoughts so far: General
Website
Qt Docs
|
|
Update: |
Yes, you can interpolate Elixir into the html attributes with (This is a feature of the latest version of embedded elixir, HEex - or 'Html aware Embedded Elixir). Replies to your next comments: General
Yes. My thinking was that if the docs metadata were to be copied into a new project such as an alternative GUI, or a plugin for a different text editor, then there would need to be some way to parse the metadata and create HTML in this external project, for it to be usable there. My thinking was that there would need to be some kind of indication in the metadata files about what each piece of data represented and/or how it was intended to be presented in the HTML. (Without this, it would be up to the contributor to attempt to compare the metadata with what they see in the Sonic Pi app itself to understand what it was and where it fit in in the UI). Maybe I am overthinking it, but those were my thoughts about that 🙂
Not especially, but I know that Sam is keen to minimise the number of new technologies introduced into the project where possible. Website
The test website looks good - Sounds fine to me 🙂
I hadn't decided what to do about the markdown docs yet. My preference is of course to have all of the docs in the same format, so that we don't have to have separate functions to handle parsing different formats. In theory I'd go with a once off manual effort to convert the markdown to TOML, (assuming we use that for the intermediate format - or script this conversion depending on how time consuming it is) and then make phoenix live view/live component templates for the tutorial & examples just like I have done for the others. Ie, yes - the tutorial would be included - it's just another part of the docs, and my instinct would be to replace the current tutorial HTML conversion process by including the tutorial in the same process I've used for the other docs sections here.
Although Phoenix is not built for this by default, there are ways people have adapted Phoenix to act as a static site generator - here's a few examples:
Qt Docs Maybe I'm misunderstanding your questions for this heading, but if you mean how is the documentation integrated into the GUI, I mention all that above in my introductory notes on the PR. Instead of embedding things like the contents list and links to the HTML files into the C++ code, we replace all that by making the documentation widget a Qt web view - meaning that we can render the docs website directly within the webview, navigation bar and all. The Qt app does not need to know anything about the docs website other than the URL to load into the web view. |
|
(Also, love what you've added to my experiment so far! - I was just under the impression that any work on our ideas was waiting until we decided how to move forward with either of them! 🤷♂️ 🙂 ) |
ethancrawford
force-pushed
the
documentation_rebuild
branch
from
9a22e06
to
8e7b9c2
Compare
|
I made a table to help start comparing different aspects of the implementations. Tell me if anything needs adding :)
|
|
Would having a Phoenix web server process in the background impact performance in any way? |
|
Table looks good! Just as long as we're aware that some of the choices I made like using a single TOML file per synth/function can be adjusted, using bootstrap was just a temporary hack for the POC, and Node.js is just a development dependency. (Re the markdown for the tutorials - as I mentioned above, if it was feasible, I'd probably convert them to TOML metadata just like the rest of the documentation, just so that we didn't have to have two separate methods for handling the conversion/parsing/displaying processes 🙂. That's just my view of it - and that's just assuming TOML/Phoenix is decided to be the way forward! 🤷♂️ ).
Unknown, so testing this will be useful. My initial instinct is that it's fine.
Yes, the daemon. Sam has said elsewhere that he would see any Elixir-driven serving of html, whether through Phoenix or otherwise, to be built as an OTP app loaded as part of Tau. (Perhaps this all uses an Umbrella app). That way the daemon could just load a single umbrella app which then starts the individual pieces (Tau and whatever serves the HTML). Looking forward to Sam joining the conversation too. He's flat out for the next few weeks with travelling and generating much needed income, so won't be available until after that. |
The markdown would still have to be rendered at some point though? Unless there's some other markup we could use? |
Technically, yes. It is more awkward to convert the markdown into structured data in a TOML file, unless maybe the 'structured data' is just a TOML multiline string that happens to contain the markdown verbatim. Which perhaps could be a little pointless. What I don't particularly much like about the tutorial being in markdown is that is forces the HTML into a certain structure, meaning that documentation designers who might want to adapt the tutorial content into their own project (a VS Code plugin for example) are forced to display the content exactly according to the HTML that the Markdown renders to - instead of having the freedom to structure the HTML how they like. The bonus of only storing the content/data of the tutorial in a portable form is that it separates it from the presentation format. The problem is that as I originally mentioned it's more involved to convert the markdown into a structured data format, since you have plain text with things like links interspersed through it. |
|
The more I think about it, the more I feel that having two distinct methods for encoding the documentation (markdown for rich text - Tutorial and Examples, TOML (or similar) for structured data - all other Help sections) makes sense. Attempting to encode rich text in TOML is probably more trouble than it is worth. Just a thought, but I wonder about the idea of using intermediate ERB templates like I do for the TOML files, but instead of the templates generating TOML files for the Tutorial and Examples, they generate the Markdown... |
|
One thing to consider is how it might be possible to use a new documentation system from a new language implementation. If, say, we were to re-implement things in Elixir, how might documentation be plugged in? This was the main motivation of suggesting something like toml to represent the structure and markdown as a way of representing the differences in text formatting. Personally I don't see any issue in throwing markdown strings into a toml document - although I'm totally happy to see other ideas explored. Using erb, or Ruby-specific things does throw some alarm bells up for me, but again, I'd be happy to see an overall plan of what you want to do so we can discuss it. I definitely think it makes sense to design the architecture, discuss and agree on it before any more coding is done just so we can all make sure we're happy with the direction :-) |
|
The only reason I've been working with the idea of ERB is that the Synth/FX/DSL metadata is currently in Ruby, and therefore it was trivial to access the metadata this way. The intention is not to leave the TOML generating code in Ruby indefinitely. Embedded templating scripts are nothing new; When we switch the DSL to something else (for which I understand Elixir is a good possibility) there are templating libraries available to use (Eex for example. I already show in the above code how to use it - the only difference if/when the entire DSL was switched over to Elixir would be that we would generate the TOML directly from Eex, as well as parsing this TOML with Eex to generate the HTML. Alternatively, even if the DSL ends up in Rust instead, there are templating script crates available there also, as I am confident there would be for other languages in turn). Markdown embedded in the TOML could work. It would be handy to do away with the custom markdown handling in i18n-tool.rb at least 🙂 I'd lean towards the idea I mention in the second paragraph of my previous reply, as using templates would allow us to include extraction of the translatable text in the process. |
|
(This is how the tutorial markdown embedded in the ERB TOML template might look): |
I get the approach, but a possible issue with this is that the order of parts of of sentence may differ between languages, so the positions of links relative to the other words may change. Also splitting parts of a sentence into different strings may make it harder to translate via weblate, as context may be lost? (I haven't used it before, so please correct me if I'm wrong on that) |
|
True - correct on both points. (Should have checked weblate first). |
|
Looking forward to continuing the discussion about this when you are available @samaaron - please let me know if there's extra information you might need as well 👍 |
|
butterfingers 😂 |
|
I noticed the comments about removal of Node/NPM as dependencies in the Phoenix 1.6 RC 🙂 |
ethancrawford
force-pushed
the
documentation_rebuild
branch
from
8e7b9c2
to
82f525d
Compare
|
@SunderB - are you up for a video chat about our PRs? I have sent a message on in_thread 🙂 |
ethancrawford
force-pushed
the
documentation_rebuild
branch
from
82f525d
to
a37c882
Compare
ethancrawford
force-pushed
the
documentation_rebuild
branch
from
a37c882
to
ba36b7c
Compare
The translation pot file needs to contain references to files from *all* of the different documentation sections, to reduce duplication. To achieve this, we now process all of the collections of documentation data (synths/fx/lang/etc) as a single group. Several minor updates to the base templates have also been made.
TODO: Add parsing code for TOML files, add live views etc
Both the generating of the documentation TOML files and the Phoenix app's build process are now hooked into the pre-build process. The TOML templates have also been updated.
This changes edits the styling of the view slightly with bootstrap and removes the default Phoenix styles. We also change the TOML documentation metadata structure slightly so that sliding information is included _within_ a synth/fx's data, not along-side it as an adjacent key/value.
The file organisation in the web app has been changed to group things togather a little more logically. Also, the app is now using the documentation metadata generated by the Ruby server.
The Pheonix web app has now had its Elixir and Node packages upgraded, to allow for easier development of some parts, such as HEex view templates.
The variable tracking the side panel navigation headings has been renamed, to more accurately reflect what it represents. We now show a basic title or heading on each page in each of the main categories. Lastly, the 'Tutorial' and 'Examples' categories have been disabled as adding pages to those categories requires more work, which will not be done unless future discussions confirm this Phoenix app as the right approach.
The utility functions were previously moved to a dedicated Paths module. This caused a conflict with the documentation rebuild code, but it has now been resolved.
ethancrawford
force-pushed
the
documentation_rebuild
branch
from
ba36b7c
to
a4edc16
Compare
Purpose of this PR
To generate discussion about the best way forward for rebuilding the documentation system. Since some parallel work was also carried out by SunderB, there will need to be some compromises somewhere about which parts of either implementation to carry forward.
There are currently several issues with the existing documentation in the help panel:
The work in this PR was originally started to address these issues: namely,
Implementation choices in this PR
Translation of the documentation
The proposed approach to making all of the documentation translatable is to stick with the existing translation framework, gettext. We would extract the text of the documentation metadata from the Ruby source code into a .pot file using Ruby's gettext gem.
(Since gettext will only work with string literals, it is necessary to first 'eval' the documentation metadata so that variable contents are output as raw strings. Once this is done, we can then extract the string literals into the .pot file).
This approach was deemed suitable since gettext is one of the most widely supported translation systems, and doing the translation from Ruby source code appeared reasonable since the prospect of later re-implementing the Sonic Pi DSL in a different language such as Elixir would most likely necessitate rewriting the documentation metadata from scratch anyway, so there would be more work down the line regardless of whether the metadata was extracted for translation from Ruby in the mean-time, or placed in some intermediate format first.
Portable documentation metadata
The proposed approach to making the documentation metadata portable/sharable between multiple 'host projects' is to extract the metadata into TOML files.
The reason for this particular data format is that it is already used in the Sonic Pi project for things such as configuration, and there is interest in using it further in future, such as for storage of custom synth metadata. Standardising on a single portable data format where possible seemed reasonable. TOML also appears to be fairly well supported in many programming languages.
Other data formats such as JSON and XML were considered, but TOML was ultimately chosen for the reason that it was already in use, and this would minimise the number of new technologies introduced into the project.
Parsing of documentation metadata from Sonic Pi DSL source code, HTML generation
The current approach of parsing metadata and generating html as interpolated strings directly from Ruby scripts mixes presentation code (html output) directly with server or app code. This is not very maintainable.
It is proposed to instead have dedicated templates that format and display the documentation metadata, separate from any Sonic Pi server code or build script.
The approach chosen in this PR was to use ERB to generate templates for the portable TOML files, instead of templates for HTML directly, to enable the portability of the documentation metadata as mentioned further above. Other templating engines were considered, such as Tilt and Slim, but ERB was chosen since it is already part of the Ruby standard library, and provides all of the functions we need. If the Ruby server code is later replaced with another language such as Elixir or Rust, there will be an equivalent library that allows code to be embedded into a text file in the same way, so the concept of using templates to produce intermediate files for the documentation is still suitable.
Storage, loading and displaying of documentation in the GUI
The documentation is currently partially embedded into the Qt GUI, so that it requires compiling the app to update it.
It is proposed to instead load the documentation into Sonic Pi via a web view/web browser embedded in the GUI, which avoids this problem.
Embedding a Qt web view into the GUI is straight forward, so the question then becomes how to create and package the HTML pages that are displayed through it.
Conversion of portable intermediate metadata files into HTML for the Sonic Pi GUI
The proposed approach was to use Elixir's Phoenix web framework to create a dynamic website, that kept the documentation metadata to be displayed in memory, rather than storing it in a database. A database was not as necessary in this case because there was not a large amount of information to store, and it was also mostly static data.
Phoenix was considered suitable for several reasons:
Using a custom-built set of scripts to template and serve the HTML pages without the use of an existing web framework is possible, but would require a fair amount of work to plumb in enough features to become suitably flexible and useful. (Also, other things like live views and live components are quite specialised and would be difficult to replicate).
Below are a few diagrams giving a rough overview of the components and processes that the proposed documentation rebuild uses.
I am interested in hearing about anyone's opinions, particularly @SunderB and @samaaron, regarding my proposed approaches and reasoning, and am happy to elaborate or answer any further questions.
The work done here is also just a proof of concept, with only the bare minimum of functionality to demonstrate the various technologies involved. (For example, starting the web server up was not yet integrated into the Sonic Pi boot process, and neither is the new documentation website yet integrated into the Sonic Pi GUI). I'm aware that various parts will potentially need to be adjusted/tidied up/completed properly depending on the outcome of any discussion about their suitability.
How to run a copy of this code demonstrating the proposed documentation rebuild
documentation_rebuildbranch ofethancrawford/sonic-picd etc/docsmix phx.serverlocalhost:4000in a browser to see a basic documentation website.Rough diagrams outlining the components and processes in the proposed documentation rebuild
Translation and conversion of Sonic Pi DSL metadata into a portable format
Basic overview of integrating a Phoenix web app and loading and displaying the documentation pages