SF.net SVN: docutils:[9733 ] trunk/docutils
milde--- via Docutils-checkins <[email protected]>
| Newsgroups | gmane.text.docutils.cvs |
|---|---|
| Message-ID | <[email protected]> |
Revision: 9733
http://sourceforge.net/p/docutils/code/9733
Author: milde
Date: 2024-06-06 14:01:22 +0000 (Thu, 06 Jun 2024)
Log Message:
-----------
Start development of a new "xml" parser for Docutils XML.
First draft implementation, documentation update, and tests.
Modified Paths:
--------------
trunk/docutils/HISTORY.txt
trunk/docutils/docs/ref/rst/directives.txt
trunk/docutils/docs/user/config.txt
trunk/docutils/docutils/parsers/__init__.py
trunk/docutils/test/test_parsers/test_get_parser_class.py
Added Paths:
-----------
trunk/docutils/docutils/parsers/docutils_xml.py
trunk/docutils/test/test_parsers/test_docutils_xml/
trunk/docutils/test/test_parsers/test_docutils_xml/__init__.py
trunk/docutils/test/test_parsers/test_docutils_xml/test_parse.py
Modified: trunk/docutils/HISTORY.txt
===================================================================
--- trunk/docutils/HISTORY.txt 2024-06-05 15:54:27 UTC (rev 9732)
+++ trunk/docutils/HISTORY.txt 2024-06-06 14:01:22 UTC (rev 9733)
@@ -37,6 +37,10 @@
normalize values,
raise ValueError for invalid attribute names or values.
+* docutils/parsers/docutils_xml.py
+
+ - New parser for Docutils XML sources. Provisional.
+
* docutils/parsers/recommonmark_wrapper.py
- New method `Parser.finish_parse()` to clean up (before validating).
@@ -1019,8 +1023,10 @@
* docutils/writers/pseudoxml.py
- - New option `detailled`_.
+ - New option `detailled`__.
+ __ detailed_
+
* test/DocutilsTestSupport.py
- Run python3 test like python2 against source not the build/-directory
Modified: trunk/docutils/docs/ref/rst/directives.txt
===================================================================
--- trunk/docutils/docs/ref/rst/directives.txt 2024-06-05 15:54:27 UTC (rev 9732)
+++ trunk/docutils/docs/ref/rst/directives.txt 2024-06-06 14:01:22 UTC (rev 9733)
@@ -1599,6 +1599,8 @@
``parser`` : text_ (parser name)
Parse the included content with the specified parser.
+ See the `"parser" configuration setting`_ for available parsers.
+
(New in Docutils 0.17)
``start-after`` : text_
@@ -2249,6 +2251,7 @@
.. _image_loading: ../../user/config.html#image-loading
.. _input_encoding: ../../user/config.html#input-encoding
.. _math_output: ../../user/config.html#math-output
+.. _"parser" configuration setting: ../../user/config.html#parser
.. _raw_enabled: ../../user/config.html#raw-enabled
.. _root_prefix: ../../user/config.html#root-prefix
.. _sectnum_xform: ../../user/config.html#sectnum-xform
Modified: trunk/docutils/docs/user/config.txt
===================================================================
--- trunk/docutils/docs/user/config.txt 2024-06-05 15:54:27 UTC (rev 9732)
+++ trunk/docutils/docs/user/config.txt 2024-06-06 14:01:22 UTC (rev 9733)
@@ -935,12 +935,11 @@
and it is left if the reference style is "brackets".
-.. _myst:
-
[myst parser]
-------------
-Provided by the 3rd party package `myst-docutils`_.
+Parser for Markdown (CommonMark_) with rST-compatibility extensions
+provided by the 3rd party package `myst-docutils`_.
See `MyST with Docutils`_ and MyST's `Sphinx configuration options`_
(some settings are not applicable with Docutils).
@@ -951,32 +950,49 @@
https://myst-parser.readthedocs.io/en/latest/sphinx/reference.html#sphinx-config-options
-.. _pycmark:
-
[pycmark parser]
----------------
-Provided by the 3rd party package `pycmark`__.
+Parser for Markdown (CommonMark_)
+provided by the 3rd party package `pycmark`_.
Currently no configuration settings.
-__ https://pypi.org/project/pycmark/
+.. _pycmark: https://pypi.org/project/pycmark/
-.. _recommonmark:
-
[recommonmark parser]
---------------------
+Parser for Markdown (CommonMark_)
+provided by the 3rd party package recommonmark_.
+
.. admonition:: Deprecated
- Depends on deprecated 3rd-party package recommonmark__.
+ Depends on deprecated 3rd-party package recommonmark_.
Support will be removed in Docutils 1.0.
Currently no configuration settings.
-__ https://pypi.org/project/recommonmark/
+.. _recommonmark: https://pypi.org/project/recommonmark/
+[xml parser]
+------------
+
+The `Docutils XML parser` processes an XML representation of a
+`Docutils Document Tree`_
+(e.g. the output of the `Docutils XML writer <[docutils_xml writer]_>`__).
+
+New in Docutils 0.22
+
+Parser Specific Defaults
+~~~~~~~~~~~~~~~~~~~~~~~~
+.. class:: run-in narrow
+
+:doctitle_xform_: False.
+:validate_: True.
+
+
[readers]
=========
@@ -2285,15 +2301,17 @@
parser
~~~~~~
Parser component name.
-Either "rst" (default) or the import name of a plug-in parser module.
+Either "`rst <[restructuredtext parser]_>`__" (default),
+"`xml <[xml parser]_>`__", or the import name of a plug-in parser module.
-Parsers for CommonMark_ known to work with Docutils include "pycmark_",
-"myst_", and "recommonmark_".
+Parsers for CommonMark_ known to work with Docutils include
+"`pycmark <[pycmark parser]_>`__", "`myst <[myst parser]_>`__",
+and "`recommonmark <[recommonmark parser]_>`__".
*Default*: "rst".
*Option*: ``--parser``
-.. _CommonMark: https://spec.commonmark.org/0.30/
+.. _CommonMark: https://spec.commonmark.org/current/
.. _writer [docutils application]:
@@ -2417,6 +2435,7 @@
.. References
+.. _Docutils Document Tree:
.. _Document Tree: ../ref/doctree.html
.. _Docutils Runtime Settings:
Modified: trunk/docutils/docutils/parsers/__init__.py
===================================================================
--- trunk/docutils/docutils/parsers/__init__.py 2024-06-05 15:54:27 UTC (rev 9732)
+++ trunk/docutils/docutils/parsers/__init__.py 2024-06-06 14:01:22 UTC (rev 9733)
@@ -82,6 +82,9 @@
'rest': 'docutils.parsers.rst',
'restx': 'docutils.parsers.rst',
'rtxt': 'docutils.parsers.rst',
+ # Docutils XML
+ 'docutils_xml': 'docutils.parsers.docutils_xml',
+ 'xml': 'docutils.parsers.docutils_xml',
# 3rd-party Markdown parsers
'recommonmark': 'docutils.parsers.recommonmark_wrapper',
'myst': 'myst_parser.docutils_',
Added: trunk/docutils/docutils/parsers/docutils_xml.py
===================================================================
--- trunk/docutils/docutils/parsers/docutils_xml.py (rev 0)
+++ trunk/docutils/docutils/parsers/docutils_xml.py 2024-06-06 14:01:22 UTC (rev 9733)
@@ -0,0 +1,104 @@
+#! /usr/bin/env python3
+# :Copyright: © 2024 Günter Milde.
+# :License: Released under the terms of the `2-Clause BSD license`_, in short:
+#
+# Copying and distribution of this file, with or without modification,
+# are permitted in any medium without royalty provided the copyright
+# notice and this notice are preserved.
+# This file is offered as-is, without any warranty.
+#
+# .. _2-Clause BSD license: https://opensource.org/licenses/BSD-2-Clause
+#
+# Revision: $Revision$
+# Date: $Date$
+
+"""A Docutils-XML parser.
+
+ Provisional: The API is not fixed yet.
+ Defined objects may be renamed or changed in any Docutils release
+ without prior notice.
+"""
+
+import xml.etree.ElementTree as ET
+
+from docutils import nodes, parsers
+
+
+class Parser(parsers.Parser):
+
+ """A Docutils-XML parser."""
+
+ supported = ('xml', 'docutils-xml')
+ """Aliases this parser supports."""
+
+ config_section = 'xml parser'
+ config_section_dependencies = ('parsers',)
+ settings_default_overrides = {'doctitle_xform': False,
+ 'validate': True,
+ }
+
+ def parse(self, inputstring, document):
+ """
+ Parse `inputstring` and populate `document`, a "document tree".
+
+ Provisional.
+ """
+ self.setup_parse(inputstring, document)
+
+ # get ElementTree
+ root = ET.fromstring(inputstring)
+ # convert ElementTree to Docutils Document Tree
+ if root.tag == 'document':
+ convert_attribs(document, root.attrib)
+ for element in root:
+ document.append(element2node(element))
+ else:
+ document.append(element2node(root))
+
+ self.finish_parse()
+
+
+def element2node(element):
+ """
+ Convert an `etree` element and its children to Docutils doctree nodes.
+
+ Return a `docutils.nodes.Element` instance.
+
+ Internal.
+ """
+ # Get the corresponding `nodes.Element` instance:
+ nodeclass = getattr(nodes, element.tag)
+ node = nodeclass()
+
+ # Attributes: convert and add to `node.attributes`.
+ convert_attribs(node, element.attrib)
+
+ # Append text (wrapped in a `nodes.Text` instance)
+ append_text(node, element.text)
+
+ # Append children and their tailing text
+ for child in element:
+ node.append(element2node(child))
+ # Text after a child node
+ append_text(node, child.tail)
+
+ return node
+
+
+def convert_attribs(node, a):
+ # Convert doctree element attribute values from string to their datatype,
+ for key, value in a.items():
+ if key.startswith('{'):
+ continue # skip duplicate attributes with namespace URL
+ node.attributes[key] = nodes.ATTRIBUTE_VALIDATORS[key](value)
+
+
+def append_text(node, text):
+ if not text:
+ return
+ if isinstance(node, nodes.TextElement):
+ node.append(nodes.Text(text))
+ elif text.strip():
+ # no TextElement: ignore formatting whitespace
+ # but append other text (node becomes invalid!)
+ node.append(nodes.Text(text.strip()))
Property changes on: trunk/docutils/docutils/parsers/docutils_xml.py
___________________________________________________________________
Added: svn:eol-style
## -0,0 +1 ##
+native
\ No newline at end of property
Added: svn:keywords
## -0,0 +1 ##
+Author Date Id Revision
\ No newline at end of property
Added: trunk/docutils/test/test_parsers/test_docutils_xml/__init__.py
===================================================================
--- trunk/docutils/test/test_parsers/test_docutils_xml/__init__.py (rev 0)
+++ trunk/docutils/test/test_parsers/test_docutils_xml/__init__.py 2024-06-06 14:01:22 UTC (rev 9733)
@@ -0,0 +1 @@
+"""Tests for `docutils.parsers.docutils_xml`."""
Property changes on: trunk/docutils/test/test_parsers/test_docutils_xml/__init__.py
___________________________________________________________________
Added: svn:eol-style
## -0,0 +1 ##
+native
\ No newline at end of property
Added: svn:keywords
## -0,0 +1 ##
+Author Date Id Revision
\ No newline at end of property
Added: trunk/docutils/test/test_parsers/test_docutils_xml/test_parse.py
===================================================================
--- trunk/docutils/test/test_parsers/test_docutils_xml/test_parse.py (rev 0)
+++ trunk/docutils/test/test_parsers/test_docutils_xml/test_parse.py 2024-06-06 14:01:22 UTC (rev 9733)
@@ -0,0 +1,136 @@
+#!/usr/bin/env python3
+# :Copyright: © 2024 Günter Milde.
+# :License: Released under the terms of the `2-Clause BSD license`_, in short:
+#
+# Copying and distribution of this file, with or without modification,
+# are permitted in any medium without royalty provided the copyright
+# notice and this notice are preserved.
+# This file is offered as-is, without any warranty.
+#
+# .. _2-Clause BSD license: https://opensource.org/licenses/BSD-2-Clause
+
+"""Tests for `docutils.parsers.docutils_xml.Parser.parse()`."""
+
+from pathlib import Path
+import sys
+import unittest
+
+if __name__ == '__main__':
+ # prepend the "docutils root" to the Python library path
+ # so we import the local `docutils` package.
+ sys.path.insert(0, str(Path(__file__).resolve().parents[3]))
+
+from docutils.frontend import get_default_settings
+from docutils.parsers import docutils_xml
+from docutils.utils import new_document
+
+parser = docutils_xml.Parser()
+
+
+class XmlParserTestCase(unittest.TestCase):
+ def test_parser(self):
+ settings = get_default_settings(docutils_xml.Parser)
+ # settings.warning_stream = ''
+ for name, cases in totest.items():
+ for casenum, (case_input, case_expected) in enumerate(cases):
+ with self.subTest(id=f'totest[{name!r}][{casenum}]'):
+ document = new_document('test data', settings.copy())
+ parser.parse(case_input, document)
+ output = document.pformat()
+ self.assertEqual(case_expected, output)
+
+
+totest = {}
+
+totest['simple'] = [
+["""\
+<document>
+ <paragraph>A paragraph.</paragraph>
+</document>
+""",
+"""\
+<document source="test data">
+ <paragraph>
+ A paragraph.
+"""],
+["""\
+<document source="test sample">
+ <paragraph>A paragraph
+ with two lines.</paragraph>
+</document>
+""",
+# TODO: unindent second line
+"""\
+<document source="test sample">
+ <paragraph>
+ A paragraph
+ with two lines.
+"""],
+["""\
+<document>
+ <paragraph>Paragraph 1.</paragraph>
+ <paragraph>Paragraph 2.</paragraph>
+</document>
+""",
+"""\
+<document source="test data">
+ <paragraph>
+ Paragraph 1.
+ <paragraph>
+ Paragraph 2.
+"""],
+["""\
+<paragraph>Sub-trees are parsed, too.
+Line breaks are preserved.</paragraph>
+""",
+"""\
+<document source="test data">
+ <paragraph>
+ Sub-trees are parsed, too.
+ Line breaks are preserved.
+"""],
+]
+
+totest['inline'] = [
+["""\
+<paragraph><emphasis>emphatically</emphasis></paragraph>
+""",
+"""\
+<document source="test data">
+ <paragraph>
+ <emphasis>
+ emphatically
+"""],
+["""\
+<paragraph><strong>strong</strong> week</paragraph>
+""",
+"""\
+<document source="test data">
+ <paragraph>
+ <strong>
+ strong
+ week
+"""],
+]
+
+totest['attributes'] = [
+["""\
+<paragraph classes="top secret">classified text</paragraph>
+""",
+"""\
+<document source="test data">
+ <paragraph classes="top secret">
+ classified text
+"""],
+[r"""
+<paragraph ids="ref-2 ref-1" names="ref\ 2 ref\ 1">target paragraph</paragraph>
+""",
+r"""<document source="test data">
+ <paragraph ids="ref-2 ref-1" names="ref\ 2 ref\ 1">
+ target paragraph
+"""],
+]
+
+
+if __name__ == '__main__':
+ unittest.main()
Property changes on: trunk/docutils/test/test_parsers/test_docutils_xml/test_parse.py
___________________________________________________________________
Added: svn:eol-style
## -0,0 +1 ##
+native
\ No newline at end of property
Added: svn:keywords
## -0,0 +1 ##
+Author Date Id Revision
\ No newline at end of property
Modified: trunk/docutils/test/test_parsers/test_get_parser_class.py
===================================================================
--- trunk/docutils/test/test_parsers/test_get_parser_class.py 2024-06-05 15:54:27 UTC (rev 9732)
+++ trunk/docutils/test/test_parsers/test_get_parser_class.py 2024-06-06 14:01:22 UTC (rev 9733)
@@ -29,10 +29,17 @@
class GetParserClassTestCase(unittest.TestCase):
- def test_registered_parser(self):
+ def test_registered_parsers(self):
+ get_parser_class('null')
get_parser_class('rst')
+ get_parser_class('docutils_xml')
# raises ImportError on failure
+ def test_registered_parsers_case_folding(self):
+ get_parser_class('reStructuredText')
+ get_parser_class('XML')
+ # raises ImportError on failure
+
def test_bogus_parser(self):
with self.assertRaises(ImportError):
get_parser_class('nope')
This was sent by the SourceForge.net collaborative development platform, the world's largest Open Source development site.
_______________________________________________
Docutils-checkins mailing list
[email protected]
https://lists.sourceforge.net/lists/listinfo/docutils-checkins