RLiterate

RLiterate is a tool for authoring documents. You can think of it as a programmer's version of a word processor. This book describes RLiterate and also includes the complete implementation presented in small pieces.

Screenshot of RLiterate editing itself.

A tour of RLiterate

This chapter gives an overview of what types of document you can create with RLiterate and presents the main features of the program.

Main GUI

The main GUI consists of the table of contents and the workspace.

The main GUI window.

The table of contents shows the outline of the document, and from it, pages can be opened in the workspace. Each page is drawn with a border in the workspace.

Reading tool

RLiterate is not only a tool for authoring documents, but also a tool for reading documents. The following features support that.

Getting started

This chapter gives practical advice for using RLiterate on your computer.

Status

RLiterate is currently a prototype. Experimentation is encouraged, but it may have serious bugs.

Installing

The source code for RLiterate is in a Git repository hosted on Github:

https://github.com/rickardlindberg/rliterate

Assuming you have Git installed, you can clone the repository like this:

git clone https://github.com/rickardlindberg/rliterate.git

Before you can run RLiterate, you need Python with the following libraries installed:

Here is how they can be installed on a Fedora system:

dnf install python wxPython python2-pygments

Once they are installed, you can run RLiterate like this:

python rliterate.py rliterate.rliterate

This will open the RLiterate document. Enter a different filename as last parameter to create a new document.

Diffing *.rliterate files

When *.rliterate files are version controlled, the textual diff is hard to read. This problem can be sovled in Git by defining a textconv command that converts the *.rliterate file to text that is suitable for diffing. The --diff option to RLiterate outputs a file in a diff friendly format.

First associate *.rliterate files with rliterate using a Git attributes file. For example in $HOME/.config/git/attributes:

*.rliterate diff=rliterate

Then define the textconv command in your git config ($HOME/.gitconfig):

[diff "rliterate"]
      textconv=bash -c 'python $RLITERATE_ROOT/rliterate.py "$0" --diff'

Concepts

Document model

RLiterate documents have pages organized in a hierarchy. Pages have a title and paragraphs. Paragraphs can be of different types. RLiterate documents can also be exported to different formats for display in different mediums.

Paragraph types

The different paragraph types is what make RLiterate documents special. Code paragraphs for example enable literate programming by allowing chunks of code to be defined and then be automatically assembled into the final source file. Text paragraphs are used for writing prose.

Variables

Literate programming

TODO: Explain how code paragraphs enable literate programming.

Focus by hoisting

Hoisting a page in the table of contents allows you to focus on a subset of the document.

Page 1 hoisted.

Breath first reading

Openining a page and all immediate children allows you to read a subset of the document breath first. It's like reading only the first section of each chapter in an entire book.

Demo page with its children opened. Notice that Sub page is not an immediate child and therefore not shown.

History

This chapter explains how RLiterate came about.

The idea (winter 2017)

I started to think about what would become RLiterate when I read the paper Active Essays on the Web. In it they talk about embedding code in documents that the reader can interact with:

An Active Essay combines a written essay, program fragments, and the resulting live simulations into a single cohesive narrative.

They also mention literate programming as having a related goal of combining literature and programming.

At the time I was working on a program that I thought would be nice to write in this way. I wanted to write an article about the program and have the code for the program embedded in the article. I could have used a literate programming tool for this, but the interactive aspect of active essays made me think that a tool would be much more powerful if the document could be edited "live", similar to WYSIWYG editors. Literate programming tools I were aware of worked by editing plain text files with a special syntax for code and documentation blocks, thus lacking the interactive aspect.

To me, the most central idea in literate programming is that we should write programs for other humans. Only secondary for the machine. I find code easier to understand if I can understand it in isolated pieces. One example where it is difficult to isolate a piece without literate programming is test code. Usually the test code is located in one file, and the implementation in another. To fully understand the piece it is useful to both read the tests and the implementation. To do this we have to find this related information in two unrelated files. I wrote about this problem in 2013 in Related things are not kept together. Literate programming allows us to present the test and the implementation in the same place, yet have the different pieces written to different files. The compiler might require that they are in separate files, but with literate programming, we care first about the other human that will read our code, and only second about the compiler.

Another argument for literate programming is to express the "why". Why is this code here? Timothy Daly talks about it in his talk Literate Programming in the Large. He also argues that programmers must change the mindset from wring a program to writing a book. Some more can be read in Example of Literate Programming in HTML.

Some more resources about literate programming:

So the tool I envisoned was one that would allow literate programming and also allow more interaction with the content.

The prototype (winter 2018)

So I decided to build a prototype to learn what such a tool might be like.

First I came up with a document model where pages were organized in a hierarchy. Each page had paragraphs that could be of different types. This idea was stolen from Smallest Federated Wiki. The code paragraph would allow for literate programming, while the text paragraph would be for prose. I also envisioned other paragraph types that would allow for more interaction. Perhaps one paragraph type could be Graphviz code, and when edited, a generated graph would appear instead of the code.

After coming up with a document model, I implement a GUI that would allow editing such documents. This GUI had to be first class as it would be the primary way author (and read) documents.

The first version of the GUI was not first class though. I started to experiment with a workspace for showing pages.

First GUI prototype of workspace.

Then I contintued with a table of contents. At this point it was based on a tree control present in wxPython.

First GUI prototype of table of contents. Workspace pages also have titles.

All the data so far had been static and hard coded. I started to flesh out the document model and implement GUI controls for manipulating the document. Here is the add button on pages.

Add button that creates the factory paragraph.

Drag and drop was implemented fairly early. I wanted good visual feedback. This was part of the first class idea.

Drag and drop with red divider line indicating drop point.

Drag and drop with good visual feedback was hard to achieve with the tree control from wxWidgets, so at this point I rewrote the table of contents widget as a custom widget.

Custom table of contents that allows drag and drop and custom rendering.

I added more operations for adding and deleting pages.

Context menu that allows adding and deleting pages.

Then finally I added a second paragraph type: the code paragraph type that would enable literate programming.

Code paragraphs showing literate programming in action.

At this point I had all the functionality in place for writing documents and embedding code in them. I imported all the code into an RLiterate document (previously it was written as a single Python file) and started extracting pieces and adding prose to explain the program. This was a tremendously rewarding experience. RLiterate was now bootstrapped.

Improving the prototype (summer 2018)

Dogfeeding. I imporoved RLiterate inside RLiterate.

As I continued to improve the RLiterate document, I noticed features I was lacking. I added them to the program and iterated.

ProjecturED was an influence for introducing variables.

When researching ideas for RLiterate, I stumbled across ProjecturED. It is similar to RLiterate in the sense that it is an editor for more structured documents. Not just text. The most interesting aspect for me was that a variable name exists in one place, but can be rendered in multiple. So a rename is really simple. With RLiterate, you have to do a search and replace. But with ProjecturED you just change the name and it replicates everywhere. This is an attractive feature and is made possible by the different document model.

Writing "the program" (autumn 2018)

I started writing the article and noticed other features I was missing.

Direct manipulation (winter 2018-2019)

Started work on getting rid of modes.

Inspired by Alan Kay and Lary Tesler.

Successful experiment with no-modes editing of titles in dec 2018.

RLiterate works in modes: view and edit. This is in conflict with "no modes". ProjecturED is more direct. It can show multiple projections of the same data structure, but full direct editing is available in both projections. RLiterate can show two projections (view and edit) but only supports edits in one.

Support for more direct editing might be desierable. But I'm a Vim user, and I think modes are powerful. I need to investigate this area more. Can direct manipulation feel as "precise" as different modes? How can automatic save work if there is no explicit "edit ends"?

PDF by Lary Tesler about modes: A Personal History of Modeless Text Editing and Cut/Copy-Paste .

If modes must be used, it should be visually clearly indicated that you are in a mode.

On WYSIWYG by Alan Kay.

Implementation

This chapter gives a complete description of the implementation of RLiterate presented in small pieces.

Files

RLiterate is implemented in Python. The implementation consists of four files: one for a compiler of gui widgets, one for the program, one for the test code, and a Makefile for various operations.

  1. rlgui.py
# This file is extracted from rliterate.rliterate.
# DO NOT EDIT MANUALLY!
  1. rliterate.py
# This file is extracted from rliterate.rliterate.
# DO NOT EDIT MANUALLY!

<<imports>>
<<constants>>
<<decorators>>
<<base base base classes>>
<<base base classes>>
<<base classes>>
<<classes>>
<<functions>>

if __name__ == "__main__":
    <<entry point>>
  1. test_rliterate.py
# This file is extracted from rliterate.rliterate.
# DO NOT EDIT MANUALLY!

<<imports>>
<<fixtures>>
<<test cases>>
  1. Makefile
# This file is extracted from rliterate.rliterate.
# DO NOT EDIT MANUALLY!

<<rules>>

The remainder of this chapter will fill in the sections in those files to create the complete implementation.

Document model

An RLiterate document is represented as a Python dictionary with the following stucture:

{
  "root_page": {
    "id": "106af6f8665c45e8ab751993a6abc876",
    "title": "Root",
    "paragraphs": [
      {
        "id": "8f0a9f84821540e89d7c9ca93ed0fbe7",
        "type": "text",
        "fragments": [
          {"text": "This ", "type": "text" },
          {"text": "is",    "type": "strong" },
          {"text": "cool",  "type": "emphasis" }
        ]
      },
      ...
    ],
    "children": [ ... ]
  },
  "variables": { ... }
}

It has a root page and variables. The root page has a unique id, a title, paragraphs, and child pages. Paragraphs also have unique ids and different attributes depending on type. (The text paragraph has a list of text fragments with different styles.)

The Document class defined in this section provides a friendly interface for working with an RLiterate document.

Loading and saving

  1. rliterate.py
  2. classes
  3. Document 
@classmethod
def from_file(cls, path):
    if os.path.exists(path):
        return cls(path, load_json_from_file(path))
    else:
        return cls(path, new_document_dict())
  1. rliterate.py
  2. functions 
def new_document_dict():
    return {
        "root_page": new_page_dict(),
        "variables": {},
    }
  1. rliterate.py
  2. classes 
class Document(Observable):

    def __init__(self, path, document_dict):
        Observable.__init__(self)
        self.path = path
        self._load(document_dict)

    <<Document>>
  1. rliterate.py
  2. classes
  3. Document 
def _load(self, document_dict):
    self._history = History(
        self._convert_to_latest(document_dict),
        size=UNDO_BUFFER_SIZE
    )
  1. rliterate.py
  2. constants 
UNDO_BUFFER_SIZE = 10
  1. rliterate.py
  2. classes
  3. Document 
@property
def document_dict(self):
    return self._history.value
  1. rliterate.py
  2. classes
  3. Document 
def save(self):
    write_json_to_file(
        self.path,
        self.document_dict
    )

Modifying

There are only three ways in which a document should be modified:

  1. Via the `0397a216ff674867aa422a0778df2f44` method
  2. Via undo
  3. Via redo
  1. rliterate.py
  2. classes
  3. Document 
@contextlib.contextmanager
def transaction(self, name):
    with self.notify():
        with self._history.new_value(name, modify_fn=lambda x: x):
            yield

def modify(self, name, modify_fn):
    with self.notify():
        with self._history.new_value(name, modify_fn=modify_fn):
            pass

def get_undo_operation(self):
    def undo():
        with self.notify():
            self._history.back()
    if self._history.can_back():
        return (self._history.back_name(), undo)

def get_redo_operation(self):
    def redo():
        with self.notify():
            self._history.forward()
    if self._history.can_forward():
        return (self._history.forward_name(), redo)

Page

  1. rliterate.py
  2. functions 
def new_page_dict():
    return {
        "id": genid(),
        "title": "New page...",
        "children": [],
        "paragraphs": [],
    }
  1. rliterate.py
  2. classes
  3. Document 
def get_page(self, page_id):
    for page in self.iter_pages():
        if page.id == page_id:
            return page

def iter_pages(self):
    def iter_pages(page):
        yield page
        for child in page.children:
            for sub_page in iter_pages(child):
                yield sub_page
    return iter_pages(self.get_root_page())

def get_root_page(self):
    return Page(
        self,
        ["root_page"],
        self.document_dict["root_page"],
        None,
        None
    )
  1. rliterate.py
  2. classes 
class Page(DocumentFragment):

    def __init__(self, document, path, fragment, parent, index):
        DocumentFragment.__init__(self, document, path, fragment)
        self._parent = parent
        self._index = index

    <<Page>>
  1. rliterate.py
  2. classes
  3. Page 
@property
def parent(self):
    return self._parent

@property
def full_title(self):
    return " / ".join(page.title for page in self.chain)

@property
def chain(self):
    result = []
    page = self
    while page is not None:
        result.insert(0, page)
        page = page.parent
    return result

@property
def depth(self):
    return len(self.chain)

def iter_code_fragments(self):
    for paragraph in self.paragraphs:
        for fragment in paragraph.iter_code_fragments():
            yield fragment

def iter_text_fragments(self):
    for paragraph in self.paragraphs:
        for fragment in paragraph.iter_text_fragments():
            yield fragment

@property
def id(self):
    return self._fragment["id"]

@property
def title(self):
    return self._fragment["title"]

def set_title(self, title):
    self._document.modify("Change title", lambda document_dict:
        im_replace(document_dict, self._path+["title"], title)
    )
  1. rliterate.py
  2. classes
  3. Page 
def delete(self):
    if self.parent is not None:
        self.parent.replace_child_at_index(self._index, self._fragment["children"])
  1. rliterate.py
  2. classes
  3. Page 
def move(self, target_page, target_index):
    # Abort if invalid move
    page = target_page
    while page is not None:
        if page.id == self.id:
            return
        page = page.parent
    # Abort if no-op mode
    if target_page.id == self.parent.id and target_index in [self._index, self._index+1]:
        return
    # Do the move
    with self._document.transaction("Move page"):
        if target_page.id == self.parent.id:
            insert_first = target_index > self._index
        else:
            insert_first = target_page.depth > self.parent.depth
        if insert_first:
            target_page.insert_child_at_index(self._fragment, target_index)
            self.parent.replace_child_at_index(self._index, [])
        else:
            self.parent.replace_child_at_index(self._index, [])
            target_page.insert_child_at_index(self._fragment, target_index)
Paragraphs
  1. rliterate.py
  2. classes
  3. Page 
@property
def paragraphs(self):
    return [
        Paragraph.create(
            self._document,
            self._path+["paragraphs", index],
            paragraph_dict,
            index,
            self
        )
        for index, paragraph_dict
        in enumerate(self._fragment["paragraphs"])
    ]
  1. rliterate.py
  2. classes
  3. Page 
def get_paragraph(self, paragraph_id):
    for paragraph in self.paragraphs:
        if paragraph.id == paragraph_id:
            return paragraph
  1. rliterate.py
  2. classes
  3. Page 
def delete_paragraph_at_index(self, index):
    self._document.modify("Delete paragraph", lambda document_dict:
        im_modify(
            document_dict,
            self._path+["paragraphs"],
            lambda paragraphs: paragraphs[:index]+paragraphs[index+1:]
        )
    )
  1. rliterate.py
  2. classes
  3. Page 
def insert_paragraph_at_index(self, paragraph_dict, index):
    self._document.modify("Insert paragraph", lambda document_dict:
        im_modify(
            document_dict,
            self._path+["paragraphs"],
            lambda paragraphs: paragraphs[:index]+[paragraph_dict]+paragraphs[index:]
        )
    )
Children
  1. rliterate.py
  2. classes
  3. Page 
@property
def children(self):
    return [
        Page(
            self._document,
            self._path+["children", index],
            child_dict,
            self,
            index
        )
        for index, child_dict
        in enumerate(self._fragment["children"])
    ]
  1. rliterate.py
  2. classes
  3. Page 
def add_child(self):
    self.insert_child_at_index(new_page_dict(), len(self._fragment["children"]))
  1. rliterate.py
  2. classes
  3. Page 
def insert_child_at_index(self, page_dict, index):
    self._document.modify("Insert page", lambda document_dict:
        im_modify(
            document_dict,
            self._path+["children"],
            lambda children: children[:index]+[page_dict]+children[index:]
        )
    )
  1. rliterate.py
  2. classes
  3. Page 
def replace_child_at_index(self, index, page_dicts):
    self._document.modify("Replace page", lambda document_dict:
        im_modify(
            document_dict,
            self._path+["children"],
            lambda children: children[:index]+page_dicts+children[index+1:]
        )
    )

Paragraph

  1. rliterate.py
  2. classes
  3. Document 
def add_paragraph(self, page_id, target_index=None, paragraph_dict={"type": "factory"}):
    page = self.get_page(page_id)
    if target_index is None:
        target_index = len(page.paragraphs)
    page.insert_paragraph_at_index(
        dict(paragraph_dict, id=genid()),
        target_index
    )

def get_paragraph(self, page_id, paragraph_id):
    return self.get_page(page_id).get_paragraph(paragraph_id)
  1. rliterate.py
  2. classes 
class Paragraph(DocumentFragment):

    @staticmethod
    def create(document, path, paragraph_dict, index, page):
        return {
            "text": TextParagraph,
            "quote": QuoteParagraph,
            "list": ListParagraph,
            "code": CodeParagraph,
            "image": ImageParagraph,
            "expanded_code": ExpandedCodeParagraph,
        }.get(paragraph_dict["type"], Paragraph)(document, path, paragraph_dict, index, page)

    def __init__(self, document, path, paragraph_dict, index, page):
        DocumentFragment.__init__(self, document, path, paragraph_dict)
        self._index = index
        self._page = page

    @property
    def id(self):
        return self._fragment["id"]

    @property
    def type(self):
        return self._fragment["type"]

    @contextlib.contextmanager
    def multi_update(self):
        with self._document.transaction("Edit paragraph"):
            yield

    def update(self, data):
        self._document.modify("Edit paragraph", lambda document_dict:
            im_modify(
                document_dict,
                self._path,
                lambda paragraph: dict(paragraph, **data)
            )
        )

    def delete(self):
        self._page.delete_paragraph_at_index(self._index)

    def move(self, target_page, target_index):
        if target_page.id == self._page.id and target_index in [self._index, self._index+1]:
            return
        with self._document.transaction("Move paragraph"):
            if target_index > self._index:
                target_page.insert_paragraph_at_index(self._fragment, target_index)
                self._page.delete_paragraph_at_index(self._index)
            else:
                self._page.delete_paragraph_at_index(self._index)
                target_page.insert_paragraph_at_index(self._fragment, target_index)

    def duplicate(self):
        with self._document.transaction("Duplicate paragraph"):
            self._page.insert_paragraph_at_index(
                dict(copy.deepcopy(self._fragment), id=genid()),
                self._index+1
            )

    @property
    def filename(self):
        return "paragraph.txt"

    def iter_code_fragments(self):
        return iter([])

    def iter_text_fragments(self):
        return iter([])

    def insert_paragraph_before(self, **kwargs):
        self._document.add_paragraph(
            self._page.id,
            target_index=self._index,
            **kwargs
        )

    def insert_paragraph_after(self, **kwargs):
        self._document.add_paragraph(
            self._page.id,
            target_index=self._index+1,
            **kwargs
        )
Text
  1. rliterate.py
  2. classes 
class TextParagraph(Paragraph):

    @property
    def fragments(self):
        return TextFragment.create_list(
            self._document,
            self._path+["fragments"],
            self._fragment["fragments"]
        )

    @property
    def tokens(self):
        return [x.token for x in self.fragments]

    def get_text_index(self, index):
        return self._text_version.get_selection(index)[0]

    @property
    def text_version(self):
        return self._text_version.text

    @property
    def _text_version(self):
        text_version = TextVersion()
        for fragment in self.fragments:
            fragment.fill_text_version(text_version)
        return text_version

    @text_version.setter
    def text_version(self, value):
        self.update({"fragments": text_to_fragments(value)})

    def iter_text_fragments(self):
        return iter(self.fragments)
  1. rliterate.py
  2. functions 
def fragments_to_text(fragments):
    text_version = TextVersion()
    for fragment in fragments:
        fragment.fill_text_version(text_version)
    return text_version.text


def text_to_fragments(text):
    return TextParser().parse(text)
  1. rliterate.py
  2. classes 
class TextParser(object):

    SPACE_RE = re.compile(r"\s+")
    PATTERNS = [
        (
            re.compile(r"\*\*(.+?)\*\*", flags=re.DOTALL),
            lambda parser, match: {
                "type": "strong",
                "text": match.group(1),
            }
        ),
        (
            re.compile(r"\*(.+?)\*", flags=re.DOTALL),
            lambda parser, match: {
                "type": "emphasis",
                "text": match.group(1),
            }
        ),
        (
            re.compile(r"``(.+?)``", flags=re.DOTALL),
            lambda parser, match: {
                "type": "variable",
                "id": match.group(1),
            }
        ),
        (
            re.compile(r"`(.+?)`", flags=re.DOTALL),
            lambda parser, match: {
                "type": "code",
                "text": match.group(1),
            }
        ),
        (
            re.compile(r"\[\[(.+?)(:(.+?))?\]\]", flags=re.DOTALL),
            lambda parser, match: {
                "type": "reference",
                "text": match.group(3),
                "page_id": match.group(1),
            }
        ),
        (
            re.compile(r"\[(.*?)\]\((.+?)\)", flags=re.DOTALL),
            lambda parser, match: {
                "type": "link",
                "text": match.group(1),
                "url": match.group(2),
            }
        ),
    ]

    def parse(self, text):
        text = self._normalise_space(text)
        fragments = []
        partial = ""
        while text:
            result = self._get_special_fragment(text)
            if result is None:
                partial += text[0]
                text = text[1:]
            else:
                match, fragment = result
                if partial:
                    fragments.append({"type": "text", "text": partial})
                    partial = ""
                fragments.append(fragment)
                text = text[match.end(0):]
        if partial:
            fragments.append({"type": "text", "text": partial})
        return fragments

    def _normalise_space(self, text):
        return self.SPACE_RE.sub(" ", text).strip()

    def _get_special_fragment(self, text):
        for pattern, fn in self.PATTERNS:
            match = pattern.match(text)
            if match:
                return match, fn(self, match)
Quote
  1. rliterate.py
  2. classes 
class QuoteParagraph(TextParagraph):
    pass
List
  1. rliterate.py
  2. classes 
class ListParagraph(Paragraph):

    @property
    def child_type(self):
        return self._fragment["child_type"]

    @property
    def children(self):
        return [
            ListItem(self._document, self._path+["children", index], x)
            for index, x
            in enumerate(self._fragment["children"])
        ]

    def get_text_index(self, list_and_fragment_index):
        return self._text_version.get_selection(list_and_fragment_index)[0]

    @property
    def text_version(self):
        return self._text_version.text

    @property
    def _text_version(self):
        def list_item_to_text(text_version, child_type, item, indent=0, index=0):
            text_version.add("    "*indent)
            if child_type == "ordered":
                text_version.add("{}. ".format(index+1))
            else:
                text_version.add("* ")
            for fragment in item.fragments:
                fragment.fill_text_version(text_version)
            text_version.add("\n")
            for index, child in enumerate(item.children):
                with text_version.index(index):
                    list_item_to_text(text_version, item.child_type, child, index=index, indent=indent+1)
        text_version = TextVersion()
        for index, child in enumerate(self.children):
            with text_version.index(index):
                list_item_to_text(text_version, self.child_type, child, index=index)
        return text_version

    @text_version.setter
    def text_version(self, value):
        child_type, children = ListParser(value).parse_items()
        self.update({
            "child_type": child_type,
            "children": children
        })

    def iter_text_fragments(self):
        for item in self.children:
            for fragment in item.iter_text_fragments():
                yield fragment
  1. rliterate.py
  2. classes 
class ListItem(DocumentFragment):

    @property
    def fragments(self):
        return TextFragment.create_list(
            self._document,
            self._path+["fragments"],
            self._fragment["fragments"]
        )

    @property
    def child_type(self):
        return self._fragment["child_type"]

    @property
    def children(self):
        return [
            ListItem(self._document, self._path+["children", index], x)
            for index, x
            in enumerate(self._fragment["children"])
        ]

    @property
    def tokens(self):
        return [x.token for x in self.fragments]

    def iter_text_fragments(self):
        for fragment in self.fragments:
            yield fragment
        for child in self.children:
            for fragment in child.iter_text_fragments():
                yield fragment
  1. rliterate.py
  2. classes 
class ListParser(object):

    ITEM_START_RE = re.compile(r"( *)([*]|\d+[.]) (.*)")

    def __init__(self, text):
        self.lines = text.strip().split("\n")

    def parse_items(self, level=0):
        items = []
        list_type = None
        while True:
            type_and_item = self.parse_item(level)
            if type_and_item is None:
                return list_type, items
            else:
                item_type, item = type_and_item
                if list_type is None:
                    list_type = item_type
                items.append(item)

    def parse_item(self, level):
        parts = self.consume_bodies()
        next_level = level + 1
        item_type = None
        if self.lines:
            match = self.ITEM_START_RE.match(self.lines[0])
            if match:
                matched_level = len(match.group(1))
                if matched_level >= level:
                    parts.append(match.group(3))
                    self.lines.pop(0)
                    parts.extend(self.consume_bodies())
                    next_level = matched_level + 1
                    if "*" in match.group(2):
                        item_type = "unordered"
                    else:
                        item_type = "ordered"
        if parts:
            child_type, children = self.parse_items(next_level)
            return (item_type, {
                "fragments": TextParser().parse(" ".join(parts)),
                "children": children,
                "child_type": child_type,
            })

    def consume_bodies(self):
        bodies = []
        while self.lines:
            if self.ITEM_START_RE.match(self.lines[0]):
                break
            else:
                bodies.append(self.lines.pop(0))
        return bodies
Code
  1. rliterate.py
  2. classes 
class CodeParagraph(Paragraph):

    <<CodeParagraph>>
Path
  1. rliterate.py
  2. classes
  3. CodeParagraph 
@property
def path(self):
    return Path(
        [x for x in self._fragment["filepath"] if x],
        [x for x in self._fragment["chunkpath"] if x]
    )

@path.setter
def path(self, path):
    self.update({
        "filepath": copy.deepcopy(path.filepath),
        "chunkpath": copy.deepcopy(path.chunkpath),
    })

@property
def filepath(self):
    return self._fragment["filepath"]

@filepath.setter
def filepath(self, filepath):
    self.update({
        "filepath": filepath,
    })

@property
def chunkpath(self):
    return self._fragment["chunkpath"]

@chunkpath.setter
def chunkpath(self, chunkpath):
    self.update({
        "chunkpath": chunkpath,
    })
  1. rliterate.py
  2. classes
  3. CodeParagraph 
@property
def filename(self):
    return self.path.filename
  1. rliterate.py
  2. classes 
class Path(object):

    @classmethod
    def from_text_version(cls, text):
        try:
            filepath_text, chunkpath_text = text.split(" // ", 1)
        except:
            filepath_text = text
            chunkpath_text = ""
        return cls(
            filepath_text.split("/") if filepath_text else [],
            chunkpath_text.split("/") if chunkpath_text else [],
        )

    @property
    def text_version(self):
        if self.has_both():
            sep = " // "
        else:
            sep = ""
        return "{}{}{}".format(
            "/".join(self.filepath),
            sep,
            "/".join(self.chunkpath)
        )

    @property
    def text_start(self):
        return self.text_end - len(self.last)

    @property
    def text_end(self):
        return len(self.text_version)

    def extend_chunk(self, chunk):
        return Path(
            copy.deepcopy(self.filepath),
            copy.deepcopy(self.chunkpath)+copy.deepcopy(chunk)
        )

    @property
    def filename(self):
        return self.filepath[-1] if self.filepath else ""

    @property
    def last(self):
        if len(self.chunkpath) > 0:
            return self.chunkpath[-1]
        elif len(self.filepath) > 0:
            return self.filepath[-1]
        else:
            return ""

    @property
    def is_empty(self):
        return self.length == 0

    @property
    def length(self):
        return len(self.chunkpath) + len(self.filepath)

    def __init__(self, filepath, chunkpath):
        self.filepath = filepath
        self.chunkpath = chunkpath

    def __eq__(self, other):
        return (
            isinstance(other, Path) and
            self.filepath == other.filepath and
            self.chunkpath == other.chunkpath
        )

    def __ne__(self, other):
        return not (self == other)

    def is_prefix(self, other):
        if len(self.chunkpath) > 0:
            return self.filepath == other.filepath and self.chunkpath == other.chunkpath[:len(self.chunkpath)]
        else:
            return self.filepath == other.filepath[:len(self.filepath)]

    def has_both(self):
        return len(self.filepath) > 0 and len(self.chunkpath) > 0

    @property
    def filepaths(self):
        for index in range(len(self.filepath)):
            yield (
                self.filepath[index],
                Path(self.filepath[:index+1], [])
            )

    @property
    def chunkpaths(self):
        for index in range(len(self.chunkpath)):
            yield (
                self.chunkpath[index],
                Path(self.filepath[:], self.chunkpath[:index+1])
            )
  1. rliterate.py
  2. classes
  3. Document 
def rename_path(self, path, name):
    with self.transaction("Rename path"):
        for p in self._code_paragraph_iterator():
            filelen = len(p.path.filepath)
            chunklen = len(p.path.chunkpath)
            if path.is_prefix(p.path):
                if path.length > filelen:
                    self.modify("", lambda document_dict: im_replace(
                        document_dict,
                        p._path+["chunkpath", path.length-1-filelen],
                        name
                    ))
                else:
                    self.modify("", lambda document_dict: im_replace(
                        document_dict,
                        p._path+["filepath", path.length-1],
                        name
                    ))
            else:
                for f in p.fragments:
                    if f.type == "chunk":
                        if path.is_prefix(Path(p.path.filepath, p.path.chunkpath+f.path)):
                            self.modify("", lambda document_dict: im_replace(
                                document_dict,
                                f._path+["path", path.length-1-filelen-chunklen],
                                name
                            ))

def _code_paragraph_iterator(self):
    for page in self.iter_pages():
        for p in page.paragraphs:
            if p.type == "code":
                yield p
Fragments
  1. rliterate.py
  2. classes
  3. CodeParagraph 
@property
def fragments(self):
    return CodeFragment.create_list(
        self._document,
        self._path+["fragments"],
        self,
        self._fragment["fragments"]
    )

def iter_code_fragments(self):
    return iter(self.fragments)

@property
def body_data(self):
    data = {
        "fragments": copy.deepcopy(self._fragment["fragments"]),
        "ids": {},
    }
    for fragment in self.fragments:
        if fragment.type == "variable":
            data["ids"][fragment.id] = fragment.name
    return data
  1. rliterate.py
  2. classes 
class CodeFragment(DocumentFragment):

    @staticmethod
    def create_list(document, path, code_paragraph, code_fragment_dicts):
        return [
            CodeFragment.create(document, path+[index], code_paragraph, code_fragment_dict)
            for index, code_fragment_dict
            in enumerate(code_fragment_dicts)
        ]

    @staticmethod
    def create(document, path, code_paragraph, code_fragment_dict):
        return {
            "variable": VariableCodeFragment,
            "chunk": ChunkCodeFragment,
            "code": CodeCodeFragment,
            "tabstop": TabstopCodeFragment,
        }.get(code_fragment_dict["type"], CodeFragment)(document, path, code_paragraph, code_fragment_dict)

    def __init__(self, document, path, code_paragraph, code_fragment_dict):
        DocumentFragmen