Externalize Visitor/Transformer from ast to visitor (#141), add docs

`fluent.syntax.visitor` is matching what `@fluent/syntax` does.
The documentation covers more than just Visitor/Transformer, but
now is as good a time as any to add that.

Author: Pike

fluent.syntax/CHANGELOG.md

@@ -4,6 +4,7 @@
 
   - Documentation is now on https://projectfluent.org/python-fluent/fluent.syntax/.
   - Removal of deprecated `BaseNode.traverse` method.
+  - Refactor `Visitor` and `Transformer` into `fluent.syntax.visitor` (from `.ast`)
 
 ## fluent.syntax 0.17 (September 10, 2019)
 

fluent.syntax/docs/ast.rst

@@ -1,5 +1,5 @@
-AST & Tooling
-=============
+AST
+===
 
 
 .. automodule:: fluent.syntax.ast

fluent.syntax/docs/index.rst

@@ -12,6 +12,7 @@ post-processing of Fluent source files.
    :maxdepth: 2
    :caption: Contents:
 
+   usage
    reference
 
 

fluent.syntax/docs/reference.rst

@@ -17,4 +17,5 @@ provide more fine-grained control and detail.
 
    parsing
    ast
+   visitor
    serializing

fluent.syntax/docs/usage.rst

@@ -0,0 +1,140 @@
+Using fluent.syntax
+===================
+
+The ``fluent.syntax`` package provides a parser, a serializer, and libraries
+for analysis and processing of Fluent files.
+
+Parsing
+-------
+
+To parse a full resource, you can use the :py:func:`fluent.syntax.parse`
+shorthand:
+
+.. code-block:: python
+
+   from fluent.syntax import parse
+   resource = parse("""
+   ### Fluent resource comment
+
+   first = creating a { $thing }
+   second = more content
+   """)
+
+To parse a single :py:class:`fluent.syntax.ast.Message` or :py:class:`fluent.syntax.ast.Term`, use
+:py:meth:`fluent.syntax.parser.FluentParser.parse_entry`:
+
+.. code-block:: python
+
+   from fluent.syntax.parser import FluentParser
+   parser = FluentParser()
+   key_message = parser.parse_entry("""
+   ### Fluent resource comment
+
+   key = value
+   """)
+
+Serialization
+-------------
+
+To create Fluent syntax from AST objects, use :py:func:`fluent.syntax.serialize` or
+:py:class:`fluent.syntax.serializer.FluentSerializer`.
+
+.. code-block:: python
+
+   from fluent.syntax import serialize
+   from fluent.syntax.serializer import FluentSerializer
+   serialize(resource)
+   serializer = FluentSerializer()
+   serializer.serialize(resource)
+   serializer.serialize_entry(key_message)
+
+Analysis (Visitor)
+------------------
+
+To analyze an AST tree in a read-only fashion, you can subclass
+:py:class:`fluent.syntax.visitor.Visitor`.
+
+You overload individual :py:func:`visit_NodeName` methods to
+handle nodes of that type, and then call into :py:func`self.generic_visit`
+to continue iteration.
+
+.. code-block:: python
+
+   from fluent.syntax import visitor
+   import re
+
+   class WordCounter(visitor.Visitor):
+       COUNTER = re.compile(r"[\w,.-]+")
+       @classmethod
+       def count(cls, node):
+           wordcounter = cls()
+           wordcounter.visit(node)
+           return wordcounter.word_count
+       def __init__(self):
+           super()
+           self.word_count = 0
+       def visit_TextElement(self, node):
+           self.word_count += len(self.COUNTER.findall(node.value))
+           self.generic_visit(node)
+
+   WordCounter.count(resource)
+   WordCounter.count(key_message)
+
+In-place Modification (Transformer)
+-----------------------------------
+
+Manipulation of an AST tree can be done in-place with a subclass of
+:py:class:`fluent.syntax.visitor.Transformer`. The coding pattern matches that
+of :py:class:`visitor.Visitor`, but you can modify the node in-place.
+You can even return different types, or remove nodes alltogether.
+
+.. code-block:: python
+
+   class Skeleton(visitor.Transformer):
+       def visit_SelectExpression(self, node):
+           # This should do more checks, good enough for docs
+           for variant in node.variants:
+               if variant.default:
+                   default_variant = variant
+                   break
+           template_variant = self.visit(default_variant)
+           template_variant.default = False
+           node.variants[:] = []
+           for key in ('one', 'few', 'many'):
+               variant = template_variant.clone()
+               variant.key.name = key
+               node.variants.append(variant)
+           node.variants[-1].default = True
+           return node
+       def visit_TextElement(self, node):
+         return None
+
+   skeleton = Skeleton()
+   skeleton.visit(key_message)
+   WordCounter.count(key_message)
+   # Returns 0.
+   new_plural = skeleton.visit(parser.parse_entry("""
+   with-plural = { $num ->
+     [one] Using { -one-term-reference } to hide
+    *[other] Using { $num } {-term-reference} as template
+   }
+   """))
+   print(serializer.serialize_entry(new_plural))
+
+This returns
+
+.. code-block:: fluent
+
+   with-plural =
+       { $num ->
+           [one] { $num }{ -term-reference }
+           [few] { $num }{ -term-reference }
+          *[many] { $num }{ -term-reference }
+       }
+
+
+.. warning::
+
+   Serializing an AST tree that was created like this might not produce
+   valid Fluent content.
+

fluent.syntax/docs/visitor.rst

@@ -0,0 +1,7 @@
+Visitor & Transformer
+=====================
+
+
+.. automodule:: fluent.syntax.visitor
+   :members:
+   :show-inheritance:

fluent.syntax/fluent/syntax/ast.py

@@ -6,67 +6,6 @@
 import six
 
 
-class Visitor(object):
-    '''Read-only visitor pattern.
-
-    Subclass this to gather information from an AST.
-    To generally define which nodes not to descend in to, overload
-    `generic_visit`.
-    To handle specific node types, add methods like `visit_Pattern`.
-    If you want to still descend into the children of the node, call
-    `generic_visit` of the superclass.
-    '''
-    def visit(self, node):
-        if isinstance(node, list):
-            for child in node:
-                self.visit(child)
-            return
-        if not isinstance(node, BaseNode):
-            return
-        nodename = type(node).__name__
-        visit = getattr(self, 'visit_{}'.format(nodename), self.generic_visit)
-        visit(node)
-
-    def generic_visit(self, node):
-        for propname, propvalue in vars(node).items():
-            self.visit(propvalue)
-
-
-class Transformer(Visitor):
-    '''In-place AST Transformer pattern.
-
-    Subclass this to create an in-place modified variant
-    of the given AST.
-    If you need to keep the original AST around, pass
-    a `node.clone()` to the transformer.
-    '''
-    def visit(self, node):
-        if not isinstance(node, BaseNode):
-            return node
-
-        nodename = type(node).__name__
-        visit = getattr(self, 'visit_{}'.format(nodename), self.generic_visit)
-        return visit(node)
-
-    def generic_visit(self, node):
-        for propname, propvalue in vars(node).items():
-            if isinstance(propvalue, list):
-                new_vals = []
-                for child in propvalue:
-                    new_val = self.visit(child)
-                    if new_val is not None:
-                        new_vals.append(new_val)
-                # in-place manipulation
-                propvalue[:] = new_vals
-            elif isinstance(propvalue, BaseNode):
-                new_val = self.visit(propvalue)
-                if new_val is None:
-                    delattr(node, propname)
-                else:
-                    setattr(node, propname, new_val)
-        return node
-
-
 def to_json(value, fn=None):
     if isinstance(value, BaseNode):
         return value.to_json(fn)

fluent.syntax/fluent/syntax/visitor.py

@@ -0,0 +1,65 @@
+# coding=utf-8
+from __future__ import unicode_literals, absolute_import
+
+from .ast import BaseNode
+
+
+class Visitor(object):
+    '''Read-only visitor pattern.
+
+    Subclass this to gather information from an AST.
+    To generally define which nodes not to descend in to, overload
+    `generic_visit`.
+    To handle specific node types, add methods like `visit_Pattern`.
+    If you want to still descend into the children of the node, call
+    `generic_visit` of the superclass.
+    '''
+    def visit(self, node):
+        if isinstance(node, list):
+            for child in node:
+                self.visit(child)
+            return
+        if not isinstance(node, BaseNode):
+            return
+        nodename = type(node).__name__
+        visit = getattr(self, 'visit_{}'.format(nodename), self.generic_visit)
+        visit(node)
+
+    def generic_visit(self, node):
+        for propname, propvalue in vars(node).items():
+            self.visit(propvalue)
+
+
+class Transformer(Visitor):
+    '''In-place AST Transformer pattern.
+
+    Subclass this to create an in-place modified variant
+    of the given AST.
+    If you need to keep the original AST around, pass
+    a `node.clone()` to the transformer.
+    '''
+    def visit(self, node):
+        if not isinstance(node, BaseNode):
+            return node
+
+        nodename = type(node).__name__
+        visit = getattr(self, 'visit_{}'.format(nodename), self.generic_visit)
+        return visit(node)
+
+    def generic_visit(self, node):
+        for propname, propvalue in vars(node).items():
+            if isinstance(propvalue, list):
+                new_vals = []
+                for child in propvalue:
+                    new_val = self.visit(child)
+                    if new_val is not None:
+                        new_vals.append(new_val)
+                # in-place manipulation
+                propvalue[:] = new_vals
+            elif isinstance(propvalue, BaseNode):
+                new_val = self.visit(propvalue)
+                if new_val is None:
+                    delattr(node, propname)
+                else:
+                    setattr(node, propname, new_val)
+        return node

fluent.syntax/tests/syntax/test_visitor.py

@@ -5,10 +5,11 @@
 
 from fluent.syntax.parser import FluentParser
 from fluent.syntax import ast
+from fluent.syntax import visitor
 from tests.syntax import dedent_ftl
 
 
-class MockVisitor(ast.Visitor):
+class MockVisitor(visitor.Visitor):
     def __init__(self):
         self.calls = defaultdict(int)
         self.pattern_calls = 0
@@ -80,7 +81,7 @@ def __call__(self, node):
         return node
 
 
-class VisitorCounter(ast.Visitor):
+class VisitorCounter(visitor.Visitor):
     def __init__(self):
         self.word_count = 0
 
@@ -104,7 +105,7 @@ def __call__(self, node):
         return node
 
 
-class ReplaceTransformer(ast.Transformer):
+class ReplaceTransformer(visitor.Transformer):
     def __init__(self, before, after):
         self.before = before
         self.after = after