---

## inclusion: always

# OCaml Org Mode Parser

## Project mission

Build a compact, high-quality OCaml library whose primary
responsibility is parsing Org Mode documents into a semantic, strongly
typed abstract syntax tree.

The library is intended to serve as infrastructure for downstream
consumers such as:

* HTML renderers;
* document-analysis tools;
* Patoline input support;
* formatters and printers maintained in separate libraries.

Keep the parser focused. Do not turn it into an Org execution
environment, editor engine, document renderer, or general
document-processing framework.

## Source of truth

The current official Org Mode syntax specification is the normative
reference for syntax and parsing behavior.

<https://orgmode.org/worg/org-syntax.html>

When the specification is ambiguous:

1. prefer the documented behavior of Org Mode;
2. prefer semantic usefulness over preservation of surface spelling;
3. prefer permissive recovery over rejecting the document;
4. document any deliberate simplification;
5. add a fixture that demonstrates the chosen behavior.

The target is practical coverage of approximately 80% of commonly used
Org syntax, not exact compatibility with every behavior of Emacs Org
Mode or `org-element`.

Do not claim exact Org compatibility unless it has been demonstrated
by tests.

## Core product principles

### One primary responsibility

The library parses Org syntax.

It must not:

* render HTML, LaTeX, Patoline, Markdown, or other output formats;
* evaluate Babel source blocks;
* execute inline source code;
* resolve or fetch external links;
* expand macros;
* evaluate table formulas;
* mutate or rewrite Org documents;
* provide editor-specific incremental parsing;
* depend on an Emacs process;
* reproduce the full Org runtime environment.

Downstream libraries may interpret or render the AST.

### Semantic AST

Prioritize semantic meaning over exact source representation.

The parser does not need to be lossless. It may normalize
insignificant syntax such as:

* redundant blank lines;
* line endings;
* ordinary physical newlines within paragraphs;
* equivalent keyword capitalization where Org treats it as
  insignificant.

Preserve text exactly where whitespace is semantically meaningful,
including:

* source block bodies;
* example block bodies;
* export block bodies;
* comment block bodies;
* code objects;
* verbatim objects;
* fixed-width content.

The AST is initially read-only. Do not design around arbitrary AST
editing or byte-for-byte serialization.

### Small public API

Keep the public API narrow and stable.

The initial public entry point should be conceptually equivalent to:

```ocaml
val parse : string -> Ast.document
```

A diagnostic API may also be provided:

```ocaml
val parse_with_diagnostics :
  string -> Ast.document * Diagnostic.t list
```

Do not expose lexer tokens, raw parser nodes, parser stacks, mutable
cursors, or normalization internals unless a demonstrated use case
requires it.

### Strongly typed representation

Use closed OCaml variants and records for known Org constructs.

Prefer:

```ocaml
type element =
  | Paragraph of inline list
  | Plain_list of plain_list
  | Table of table
  | Drawer of drawer
  | Block of block
  | Keyword of keyword
  | Comment of string
  | Fixed_width of string list
  | Horizontal_rule
```

Avoid generic stringly typed node maps such as:

```ocaml
type node = {
  kind : string;
  properties : (string * string) list;
  children : node list;
}
```

Use ordinary strings only for genuinely open namespaces, including:

* custom block names;
* unknown keywords;
* property names;
* language identifiers;
* link protocols;
* user-defined TODO keywords.

The type system should make important structural distinctions visible
without making extensions impossible.

## Compatibility model

### Supported configuration

External Emacs configuration is unavailable and out of scope.

Start with the defaults documented by the Org syntax
specification. Allow supported file-local declarations to override
those defaults.

Examples include file-local TODO declarations such as:

```org
#+TODO: TODO WAITING | DONE CANCELLED
```

Do not read:

* user Emacs configuration;
* directory-local Emacs variables;
* active minor modes;
* loaded Emacs packages;
* environment-specific Org variables.

### File-wide declarations

Syntax-affecting file declarations may occur anywhere in a document
and may affect constructs appearing before them.

Use a lightweight whole-file pre-scan to collect supported syntax
settings before the main parse.

This does not violate the single-grammar objective. The pre-scan only
discovers document configuration; it does not construct the document
AST.

Ordinary metadata keywords such as `#+TITLE` remain document elements
and must not disappear merely because the pre-scan inspected them.

### Tracking Org syntax

Target the latest published Org syntax rather than freezing
compatibility to an old Org release.

Changes to supported syntax must be:

* isolated in focused commits;
* covered by tests;
* recorded in release notes;
* assessed for AST compatibility.

Avoid speculative support for undocumented behavior.

## Parser architecture

### Menhir prototype

Use Menhir for the initial parser prototype.

Use its conventional deterministic LR parsing mode unless a concrete
grammar problem demonstrates that another mode is necessary.

Do not introduce GLR, Earley parsing, or generalized ambiguity
handling merely because Org has complex syntax. Org usually selects a
preferred interpretation through contextual and ordered recognition
rules rather than requiring multiple parse trees.

### OCaml-native dependencies

Dependencies are acceptable when they belong to the OCaml ecosystem
and do not rely on unstable external runtimes or system libraries.

Acceptable categories include:

* Menhir;
* pure-OCaml Unicode libraries;
* pure-OCaml test libraries;
* Dune tooling.

Avoid dependencies that require:

* C stubs, unless later explicitly approved;
* system-installed parsing libraries;
* an Emacs runtime;
* external language runtimes;
* network access during parsing;
* platform-specific services.

For the prototype, treat UTF-8 text as opaque OCaml
strings. Structural Org delimiters are ASCII. Do not add Unicode
classification until a supported production demonstrably requires it.

Never corrupt, reinterpret, or normalize non-ASCII byte sequences
unnecessarily.

### One grammar, multiple syntactic scales

Org contains two principal scales:

* structural elements such as headings, lists, blocks, drawers,
  tables, and paragraphs;
* inline objects such as emphasis, links, timestamps, entities, code,
  and verbatim text.

They may use separate Menhir start symbols or lexer modes while
remaining part of one parser implementation and one coherent grammar.

A suitable model is:

```menhir
%start <Raw_ast.document> document
%start <Ast.inline list> inline_document
```

Do not force block-level lines and inline punctuation through one
uniform token stream when that makes the grammar less understandable.

### Line-oriented structural lexer

Use a small line-oriented structural lexer.

Classify physical lines into high-level tokens such as:

```ocaml
type token =
  | Blank
  | Heading of int * string
  | Keyword of string * string
  | Begin_block of string * string option
  | End_block of string
  | Drawer_begin of string
  | Drawer_end
  | Property of string * string
  | List_item of raw_list_item
  | Table_row of string
  | Fixed_width of string
  | Comment_line of string
  | Horizontal_rule
  | Text_line of string
  | Eof
```

The exact constructors may evolve during the design phase.

Keep lexical recognition conservative. A malformed or uncertain
construct should normally become `Text_line`, allowing it to survive
as paragraph text.

Do not place complex document hierarchy or semantic interpretation in
the lexer.

### Raw and semantic representations

It is acceptable and encouraged to distinguish:

1. a private raw parse representation;
2. the public semantic AST.

Menhir should parse syntactic ordering and grouping. A narrow
normalization phase may then handle relationships that are unsuitable
for direct LR productions.

Examples include:

* converting flat headings into a hierarchy;
* interpreting heading title components;
* attaching planning data to headings;
* attaching property drawers to headings;
* nesting list items according to indentation;
* attaching affiliated keywords to eligible elements;
* joining paragraph lines;
* invoking inline parsing on bounded text fields.

The normalization phase must not become an undocumented second parser.

Every normalization rule must have:

* a clear invariant;
* focused unit tests;
* an explanation when the behavior is non-obvious.

### Whole-document parsing

Parse complete input strings.

Streaming and incremental reparsing are out of scope for the initial
library.

A channel-based convenience function may read the complete channel and
call the string parser, but do not expose a streaming guarantee.

## AST guidance

### Document structure

The semantic document should distinguish the content before the first
heading from the heading tree.

A representative shape is:

```ocaml
type document = {
  directives : directive list;
  preamble : element list;
  headings : heading list;
}
```

The final field names should be chosen during design, but the
distinction must remain representable.

### Headings

A heading should expose semantic components rather than only its raw
title:

```ocaml
type heading = {
  level : int;
  todo : string option;
  priority : char option;
  commented : bool;
  title : inline list;
  tags : string list;
  planning : planning option;
  properties : property list;
  contents : element list;
  children : heading list;
}
```

Accept skipped heading levels permissively.

For example, a level-three heading following a level-one heading may
become a child of the most recent level-one heading even when no
level-two heading intervenes.

Do not reject a whole document because its hierarchy is
unconventional.

### Blocks

Distinguish blocks with materially different semantics.

Opaque-body blocks should include at least:

* source;
* example;
* export;
* comment.

Recursively parsed blocks should include at least:

* quote;
* center;
* custom blocks where appropriate.

Unknown `#+begin_NAME` forms should become `Custom_block`, not parser
errors.

Preserve source bodies as opaque strings. Do not parse embedded
programming languages.

### Affiliated keywords

Represent affiliated keywords as metadata attached to the element they
modify, rather than as unrelated siblings.

Do not attach an affiliated keyword across an intervening blank line
or in a context where Org would not treat it as affiliated.

Unknown ordinary keywords should remain representable as generic
keyword nodes.

### Planning and properties

Planning information and heading property drawers should be
represented directly on headings where possible.

Avoid constructing semantically impossible states, such as arbitrary
planning lines freely appearing inside every element type.

An ordinary named drawer remains a regular element.

### Inline text

Inline parsing should eventually distinguish at least:

* plain text;
* bold;
* italic;
* underline;
* strike-through;
* code;
* verbatim;
* links;
* timestamps;
* explicit line breaks.

Malformed inline markup should degrade to plain text.

Do not lose surrounding text merely because one inline construct is
malformed.

### Whitespace normalization

Unless a construct preserves literal text:

* blank lines terminate paragraphs but need not become AST nodes;
* multiple blank lines may be treated as equivalent;
* physical lines in one paragraph may be joined with one space;
* explicit Org line breaks remain explicit inline nodes;
* indentation significant to lists must be interpreted before
  trimming.

Keep whitespace rules centralized rather than scattering `String.trim`
calls throughout the parser.

## Initial supported scope

The first useful release should aim to support the following.

### Structural syntax

* document preamble or zeroth section;
* heading hierarchy;
* file-local TODO declarations;
* heading TODO states;
* priorities;
* tags;
* planning information;
* property drawers;
* ordinary drawers;
* paragraphs;
* ordered lists;
* unordered lists;
* descriptive lists;
* source blocks;
* example blocks;
* export blocks;
* comment blocks;
* quote blocks;
* center blocks;
* verse blocks;
* custom blocks;
* keywords;
* affiliated keywords;
* Org tables;
* comments;
* fixed-width content;
* horizontal rules.

### Inline syntax

* plain text;
* bold;
* italic;
* underline;
* strike-through;
* code;
* verbatim;
* regular links;
* common plain links;
* angle links;
* active timestamps;
* inactive timestamps;
* timestamp ranges;
* explicit line breaks;
* basic entities or LaTeX fragments only after the core parser is
  stable.

### Deferred features

Unless a spec explicitly promotes them into scope, defer:

* citations;
* radio targets;
* radio links;
* macros;
* complex footnotes;
* inline Babel calls;
* inline source blocks;
* diary sexps;
* inlinetasks;
* table formula interpretation;
* link resolution;
* entity expansion;
* Babel execution;
* language-specific source parsing;
* exact compatibility with `org-element`;
* incremental editor parsing;
* lossless printing.

Do not silently expand a feature spec to include deferred behavior.

## Permissive parsing and diagnostics

The parser should produce a useful best-effort tree for malformed
input.

Use deterministic fallback rules.

Examples:

| Input condition | Preferred result |
|-----------------------------|-------------------------------------------------------|
| Unknown ordinary line | Paragraph text |
| Malformed inline markup | Plain text |
| Unknown keyword | Generic keyword |
| Unknown begin block | Custom block |
| Invalid timestamp-like text | Plain text |
| Invalid list marker | Paragraph text |
| Isolated end-block marker | Paragraph or keyword-like text |
| Broken table row | End the table and parse the line normally |
| Unterminated opaque block | Preserve body through the appropriate boundary or EOF |

Where recovery loses likely structure, emit a warning diagnostic.

Diagnostics should initially be lightweight. Line numbers are
acceptable even though full source spans are out of scope.

Do not throw exceptions for ordinary malformed Org input.

Exceptions may be used for:

* violated internal invariants;
* impossible generated-parser states;
* programming errors;
* resource failures.

Public parsing functions should convert recoverable syntax problems
into AST nodes and diagnostics.

## Quality requirements

### Tests are part of the feature

Every supported syntax feature requires tests.

Use several levels of testing:

1. lexer classification tests;
2. grammar production tests;
3. normalization tests;
4. end-to-end document tests;
5. malformed-input recovery tests;
6. regression tests for every fixed bug.

Prefer small readable Org fixtures with explicit expected ASTs.

Use examples from the official syntax specification where licensing
and attribution permit, supplemented by original fixtures.

Do not base correctness only on snapshot tests. Important semantic
fields should be asserted explicitly.

### No premature performance engineering

Favor clear grammar and predictable behavior.

Avoid:

* repeated concatenation that is obviously quadratic;
* unrestricted backtracking;
* repeated whole-document rescans beyond the deliberate configuration
  pre-scan;
* regex-heavy designs that obscure grammar rules.

Do not add complex caching, streaming, mutable arenas, or specialized
buffers without profiling.

Expected parsing complexity should normally be linear in input size,
apart from clearly bounded local analyses.

### Maintainability

Keep modules focused.

The module organization is:

```text
lib/
  ast.ml              (* Public semantic AST types *)
  ast.mli
  raw_ast.ml          (* Private raw parse tree from Menhir *)
  config.ml           (* Document config from prescan *)
  prescan.ml          (* Pre-scan for TODO keywords etc. *)
  lexer.ml            (* Handwritten structural line lexer *)
  parser.mly          (* Menhir structural grammar *)
  inline_lexer.ml     (* Handwritten inline lexer *)
  inline_parser.mly   (* Menhir inline grammar *)
  parse_bridge.ml     (* Bridges lexers to Menhir parsers *)
  normalize.ml        (* Raw_ast → Ast transformation *)
  diagnostic.ml       (* Diagnostic types *)
  org.ml              (* Public API entry points *)
test/
  test_main.ml        (* All tests: lexer, parser, inline, integration *)
  fixtures/           (* .org fixture files *)
```

This layout may evolve when the design discovers a clearer separation.

Avoid circular module dependencies.

Keep private types private through `.mli` files or Dune module
visibility.

Use descriptive names based on Org terminology.

### OCaml style

Use current stable OCaml and Dune conventions.

Prefer:

* algebraic data types;
* exhaustive pattern matching;
* immutable values;
* total helper functions where practical;
* explicit result types at module boundaries;
* small pure normalization functions;
* local mutation only when it clearly simplifies stack or cursor
  processing.

Avoid:

* objects without a demonstrated need;
* polymorphic variants for the central closed AST;
* global mutable parser state;
* exceptions as ordinary control flow;
* unsafe operations;
* PPX dependencies unless explicitly justified;
* clever type-level machinery that makes the library harder to
  consume.

Warnings should be treated seriously. New code should compile without
avoidable warnings.

### Documentation

Document:

* the public AST;
* parsing entry points;
* normalization rules visible to consumers;
* supported syntax;
* known deviations from Org;
* recovery behavior;
* version compatibility.

Do not document private implementation details as public guarantees.

Maintain a concise support matrix showing:

* supported;
* partially supported;
* deferred;
* intentionally unsupported.

## Kiro spec-driven workflow

### Use one coherent feature per spec

Each spec should address a bounded capability, such as:

* document and heading parsing;
* paragraph and inline text parsing;
* block parsing;
* drawer and property parsing;
* list parsing;
* table parsing;
* affiliated keywords;
* timestamps;
* diagnostics and recovery.

Do not create one enormous implementation spec for the entire Org
language.

Specs may depend on earlier foundations, but each should leave the
repository compiling and tested.

### Requirements phase

Write requirements in testable terms.

Each requirement should state:

* the syntax recognized;
* the resulting semantic representation;
* the relevant context restrictions;
* normalization behavior;
* malformed-input fallback;
* exclusions.

Use EARS-style acceptance criteria where useful.

Example:

> WHEN the parser encounters a heading whose first title token matches
> a TODO keyword declared anywhere in the same file, THE parser SHALL
> store that token as the heading TODO state.

Avoid vague requirements such as:

> The parser should handle headings correctly.

Explicitly identify deferred cases instead of leaving them ambiguous.

### Design phase

The design must explain:

* relevant AST additions;
* lexer tokens;
* Menhir productions;
* pre-scan behavior;
* normalization steps;
* recovery behavior;
* module boundaries;
* test strategy;
* compatibility implications.

Before introducing a new dependency, explain:

* why existing tools are insufficient;
* whether it is pure OCaml;
* whether it adds runtime or build-time coupling;
* whether it expands the public API.

Do not redesign unrelated modules during a feature unless the
requirement genuinely demands it.

### Task phase

Break implementation into small verifiable tasks.

A normal feature task sequence should resemble:

1. add or refine private raw types;
2. add lexer recognition;
3. add grammar productions;
4. add normalization;
5. add public AST exposure if needed;
6. add positive tests;
7. add malformed-input tests;
8. update support documentation.

Every implementation task should name:

* the expected files or modules;
* the behavior added;
* the tests that verify completion.

Do not mark a task complete while tests fail or the implementation
only contains placeholders.

### During implementation

Follow the accepted requirements and design.

When implementation reveals a design flaw:

1. stop expanding the workaround;
2. identify the inconsistency;
3. update the design or requirement;
4. then continue implementation.

Do not silently diverge from the spec.

Run focused tests after each task and the full suite before
considering the spec complete.

Keep commits and patches scoped to the active task.

### Definition of done

A feature is complete only when:

* requirements are implemented;
* the AST representation is documented;
* valid examples parse as specified;
* malformed examples recover as specified;
* tests pass;
* no deferred feature was accidentally introduced;
* no unnecessary dependency was added;
* public API changes are intentional;
* the syntax support matrix is updated.

## Decision-making rules

When several implementations are possible, prefer them in this order:

1. correct semantic behavior;
2. clear representation of Org concepts;
3. permissive and deterministic recovery;
4. small and stable public API;
5. maintainable grammar;
6. pure-OCaml portability;
7. acceptable performance;
8. minimal code size.

Do not optimize only for the smallest number of lines.

Do not sacrifice AST clarity merely to remove a private intermediate
type.

Do not generalize before at least two concrete features need the
abstraction.

## Explicit architectural decisions

Treat the following as established decisions unless the user
deliberately revises them:

* Menhir is the parser-generator prototype.
* UTF-8 is initially opaque string content.
* Parsing operates on complete documents.
* External Emacs configuration is ignored.
* supported file-local syntax configuration is honored;
* syntax-affecting file declarations may require a pre-scan;
* the parser is permissive;
* the AST is semantic rather than lossless;
* the public AST uses closed variants;
* source block bodies remain opaque;
* parsing is separate from rendering and evaluation;
* incremental reparsing is out of scope;
* exact `org-element` compatibility is not a goal;
* pure-OCaml ecosystem dependencies are acceptable;
* C and system-library dependencies require explicit approval.

When a requested change conflicts with one of these decisions, call
out the conflict rather than silently changing the architecture.

## Concrete implementation choices

The following choices were made during requirements elicitation and
are binding for the current implementation:

* **Package name**: `org`.
* **OCaml version**: 5.5.0 (local opam switch).
* **Test framework**: Alcotest.
* **Structural lexer**: handwritten OCaml (not ocamllex). The lexer
  classifies physical lines into structural tokens. It is
  context-aware (knows TODO keywords from the pre-scan).
* **Structural parser**: Menhir grammar consuming structural tokens,
  producing `Raw_ast.document`.
* **Inline parser**: Menhir grammar (separate `.mly` file with its own
  `%start` symbol) consuming inline tokens, producing `Ast.inline
  list`. Chosen for its good error messages and grammar reasoning.
* **Inline lexer**: handwritten OCaml tokenizer for bounded inline
  text.
* **Build system**: Dune 3.x with full opam package metadata.
* **First working increment**: parse a representative fixture document
  exercising the 80% most-used Org features, including inline objects.
* **Menhir mode**: deterministic LR (table mode with incremental API
  for error reporting).

## Implementation notes

These notes document key design decisions discovered during
implementation that go beyond the original architecture.

### Inline emphasis pre-scanning

The inline lexer pre-scans the entire input for matched emphasis
delimiter pairs *before* tokenizing. For each delimiter character
(`*`, `/`, `_`, `+`), it builds a hashtable mapping positions to
`Open` or `Close` roles. Only positions with confirmed matching
pairs emit `STAR_OPEN`/`STAR_CLOSE` etc. tokens; unmatched
delimiters become `TEXT`.

This eliminates grammar ambiguity entirely. The inline Menhir
grammar has zero conflicts because it never sees unmatched
delimiters. Fallback-to-plain-text is handled at the lexer level,
not in the grammar.

This was adopted after initial attempts with fallback grammar rules
caused reduce/reduce conflicts that Menhir resolved incorrectly.

### Record label prefixing

OCaml 5.x enforces warning 30 (duplicate-definitions) strictly in
mutually recursive type groups. The public AST uses prefixed record
labels to avoid collisions:

* `li_` for list items
* `src_` for source blocks
* `kw_` for keywords
* `dir_` for directives
* `prop_` for properties
* `aff_` for affiliated keywords

### Library wrapping

The library uses `org.ml` as a manual Dune wrapper module. All
public sub-modules are accessible under the `Org` namespace
(e.g. `Org.parse`, `Org.Ast`, `Org.Config`).

The public API entry point is `lib/org.ml` which serves both as
the wrapper and the top-level API, exposing `Org.parse` directly.

### Structural parser conflicts

The structural Menhir grammar has 5 shift/reduce conflicts from
`nonempty_list` / `list` macros where consecutive same-type elements
(e.g. multiple list items, text lines, table rows) can be extended
or ended. Menhir resolves these by preferring shift (extend the
current list), which is correct behavior.

### Default branch

The default branch is `master`.