Metadata-Version: 2.4
Name: djot-py
Version: 0.2.0
Summary: A pure Python implementation of the Djot markup language
Author: Victor Nee
Author-email: Victor Nee <neefrankie@gmail.com>
License-Expression: MIT
Classifier: Development Status :: 3 - Alpha
Classifier: Intended Audience :: Developers
Classifier: Topic :: Software Development :: Libraries :: Python Modules
Classifier: Topic :: Text Processing :: Markup
Classifier: Operating System :: OS Independent
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3 :: Only
Classifier: Programming Language :: Python :: 3.10
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Typing :: Typed
Requires-Python: >=3.12
Description-Content-Type: text/markdown

# djot-py

A Python library for parsing the light markup format [djot](https://github.com/jgm/djot).

This implementation is based on and modified from [djot.js](https://github.com/jgm/djot.js).

## Usage

```bash
pip install djot-py
```

## API

To parse text into an AST, call `parse`:

```python
from djot import parse

ast = parse("hi there\nfriend\n\nnew para")
```

To parse text into events:

```python
from djot import EventParser

text = "hello *world*\n\nfoo"

parser = EventParser(text)

# The parse method returns an iterator of events.
for event in parser.parse():
    print(event)
```

There are no rendering features available currently. Will be added in the future.

## Differences from Djot.js

- Support for comment is deliberately dropped.

### Comment

Comment should not be mixed with attributes.

I hope there's a specific comment syntax in the future. For example, there could be two types of comments:

- Line comment appeared at the end of each line, using symbols like `%%`
- Block comment wrapped in `%%%`, similar to code block.

### List

This is not hard to implement. However, too much choices might introduce future ambiguity.
I prefere simplicity rather than feature-rich. For example, Typst style simply uses `-` and `+`. Leave styling to a renderer. And block attributes could be used to convey some basic styling information.

## Proposal for new syntax

### Blockquote

Markdown style blockquote is unsual. It is prefix-based block element, different from other tokens.
How about a new syntax specific to blockquote:

```djot
>>> source="Source" author="Author"
This is a blockquote.
>>>
```

Will this simplify the parsing algorithm and make composing blockquote less tedious?

## Chunk-based Advancement

Djot.js adopt a strict line-based advancement. Traditional compiler adopts a character-based advancement.
Markup languages like djot/markdown fall into neither of them. Each parsing step might span a char, half a line,
or several lines. I call it chunk-based advancement. This idea might be already discussed in compiler theory.
I'm not sure since I haven't read enough theoretical books.

## The lifecycle of state-driven vs syntax-drive elements

Most block elements in djot (and also Markdown) are impplicitly closed elements.
They cannot be closed until a new element is visible.
That's the purpose of the `close` callback method,
passively closing previous elements.
There are a few exceptoions, however, like fenced div and code block, 
which are syntactically difference from other elements.

Containers likes table, list, paragraph do not have explicit closure symbol. 
You don't know they are terminated until you see a new element. 
That is, they don't know they are dead until new element is born.
That's why then are closed mostly in `on_close` method: you have
no way to figure out whether a container reached its end of life
until you are ready to push a new sibling container.
It's similar to how UI view works on iOS (or Android): a view does
not know it might be destroyed until a new view is pushed on stack,
hence `viewWillDisappear` triggered.
For such elmenents, `continue` method of the container 
could only tell whether it should yield to inner element or
stop immediately.
They could never tell whether they reached the end of life because
there are no such markup for such hints.

Those elements are expclitly close with closing markup.
Their presence interrupted the meaning of `open`, `continue` and `close` workflow.
Those elements deserve special treatment.

However, fenced div and code block are one of few exceptions with
a clear physical border declaring itself "I find myself reaching the end of life". `continue` could detect whether it should
close itself in addition to yielding to inner element or stopping immediately.

So a simple boolean value returned from `continue` 
could hardly expression those state. 
Those syntax-driven should at least return 3 states:

- `CONTINUE`: yield to inner element
- `STOP`: I cannot continue on the new element. But we should delay closing myself until we are sure there are no lazy content.
- `CLOSE`: Find closing markup so I could be closed immediately.

### Extra blankline in fenced div

In the test file block.spec.ts, there is an extra event `{ startpos: 24, endpos: 24, annot: "blankline" }` on line 246.
It is the position of newline char after ending colon:

```djot
::: warning

::: <- Here
```

In my view it should not be treated as a blankline.
This newline is undefined in Djot syntax.

Let's figure out why there is a trailing blankline.
When djot.js sees the ending `:::`, its continue method returns
`false`, and set cursor to `\r` if it exists, otherwise `\n`.
So the check continuations loops stops before this container.
When it comes to open stage, no rule works for current cursor
position which is the the return or newline char.
Since it is not lazy content, the container is closed.
Then it falls into `tip.content === ContentType.Block` and `isBlank`
branch, which creates a blankline event for the trailing return or newline char.

A improved approach should close the container at checking continuations step and jump to next line directly.
Fenced div is a explicitly closed element. There's no neeed to let it falltrough.
