"Invalid api version" when using a Panflute filter with Python version > 3.9

[rschram]

Hi everyone,

When rendering a Quarto project with a new version of Python3 I can no longer use a Panflute filter which had worked with Python 3.9. The issue seems to be with the "api version" used in Panflute. These are the error messages from the RStudio (2022.07.2) console when run with Python 3.10.12, Pandoc 2.17.1.1, Panflute 2.3.1, and Quarto 1.2.269:

Traceback (most recent call last):
File "/home/rschram/Current/Pandoc/filters/quotable", line 84, in
main()
File "/home/rschram/Current/Pandoc/filters/quotable", line 78, in main
doc = doc.walk(find_passage,doc=doc)
File "/home/rschram/.local/lib/python3.10/site-packages/panflute/base.py", line 274, in walk
child = child.walk(action, doc, stop_if)
File "/home/rschram/.local/lib/python3.10/site-packages/panflute/containers.py", line 85, in walk
ans = list(chain.from_iterable(ans))
File "/home/rschram/.local/lib/python3.10/site-packages/panflute/containers.py", line 83, in
ans = ((item,) if type(item) is not list else item for item in ans)
File "/home/rschram/.local/lib/python3.10/site-packages/panflute/containers.py", line 81, in
ans = (item.walk(action, doc, stop_if) for item in self)
File "/home/rschram/.local/lib/python3.10/site-packages/panflute/base.py", line 282, in walk
altered = action(self, doc)
File "/home/rschram/Current/Pandoc/filters/quotable", line 48, in find_passage
caption = pf.convert_text(elem.attributes['caption'])
File "/home/rschram/.local/lib/python3.10/site-packages/panflute/tools.py", line 489, in convert_text
out = json.loads(out, object_hook=from_json)
File "/usr/lib/python3.10/json/init.py", line 359, in loads
return cls(**kw).decode(s)
File "/usr/lib/python3.10/json/decoder.py", line 337, in decode
obj, end = self.raw_decode(s, idx=_w(s, 0).end())
File "/usr/lib/python3.10/json/decoder.py", line 353, in raw_decode
obj, end = self.scan_once(s, idx)
File "/home/rschram/.local/lib/python3.10/site-packages/panflute/elements.py", line 1362, in from_json
return Doc(*items, api_version=api, metadata=meta)
File "/home/rschram/.local/lib/python3.10/site-packages/panflute/elements.py", line 66, in init
raise TypeError("invalid api version", api_version)
TypeError: ('invalid api version', [1, 20])
Error running filter /home/rschram/Current/Pandoc/filters/quotable:
Filter returned error status 1

Apologies for not being able to produce a MWE or being able to provide more context. Any help would be appreciated because I would like to be reliably reproduce the Quarto project while keeping packages updated.

Best wishes,
Ryan

[machow]

I ran into an issue in https://github.com/machow/quartodoc, where I had to switch from panflute to a pure lua filter, due to a quarto release breaking compatibility (or something similar).

@cscheid (from the quarto team), my understanding is that quarto is unlikely to work with panflute in the future. If that's the case, it might be useful to put guidance for folks here?

edit: based on this quarto discussion, it seems like they should be able to work together.

[sergiocorreia]

Apologies for the delay replying; have been travelling and on holidays for the last weeks! I will be back in three days and have pinned down to look at the issue. I suspect it's just a new PA doc API version that needs to be validated, so should be straightforward.

[cscheid]

This error is coming not from Python's version (or Quarto), but from the fact that the version of Panflute running on OP's machine doesn't support Pandoc's JSON version:

File "/home/rschram/.local/lib/python3.10/site-packages/panflute/elements.py", line 66, in init
raise TypeError("invalid api version", api_version)
TypeError: ('invalid api version', [1, 20])

To avoid future confusion, I would suggest changing this error message to make it clear that what "API version" means here.

With that said:

Quarto will not, and generally cannot, prevent users from using Panflute or other Pandoc tooling that goes through JSON filters. But, from the perspective of Quarto, it's these filters' responsibility to preserve the structure of Quarto's custom AST nodes (such as Shortcode, Callout, FloatRefTarget and DecoratedCodeBlock).

Speaking as a Quarto developer, let me just say that I don't expect Panflute or other tooling to adapt to how Quarto operates, but Panflute users writing Pandoc JSON filters that operate on Div or Span nodes will need to be careful to preserve those structures when using Quarto.

For those interested users (and I'm really responding to @machow here), here's a pretty-printed version of the JSON output you're likely to encounter when using Quarto and running a JSON filter:

{
    "pandoc-api-version": [
        1,
        23,
        1
    ],
    "meta": {
        "biblio-config": {
            "t": "MetaBool",
            "c": true
        },
        "date-format": {
            "t": "MetaInlines",
            "c": [
                {
                    "t": "Str",
                    "c": "long"
                }
            ]
        },
        "document-css": {
            "t": "MetaBool",
            "c": false
        },
        "fig-responsive": {
            "t": "MetaBool",
            "c": true
        },
        "header-includes": {
            "t": "MetaList",
            "c": []
        },
        "include-after": {
            "t": "MetaList",
            "c": [
                {
                    "t": "MetaBlocks",
                    "c": [
                        {
                            "t": "RawBlock",
                            "c": [
                                "html",
                                "</main>\n<!-- /main column -->\n"
                            ]
                        }
                    ]
                },
                {
                    "t": "MetaBlocks",
                    "c": [
                        {
                            "t": "RawBlock",
                            "c": [
                                "html",
                                "<elided, unimportant>"
                            ]
                        }
                    ]
                },
                {
                    "t": "MetaBlocks",
                    "c": [
                        {
                            "t": "RawBlock",
                            "c": [
                                "html",
                                "</div> <!-- /content -->\n"
                            ]
                        }
                    ]
                }
            ]
        },
        "include-before": {
            "t": "MetaList",
            "c": [
                {
                    "t": "MetaBlocks",
                    "c": [
                        {
                            "t": "RawBlock",
                            "c": [
                                "html",
                                "<div id=\"quarto-content\" class=\"page-columns page-rows-contents page-layout-article\">\n<div id=\"quarto-margin-sidebar\" class=\"sidebar margin-sidebar\">\n</div>\n<main class=\"content\" id=\"quarto-document-content\">\n"
                            ]
                        }
                    ]
                }
            ]
        },
        "labels": {
            "t": "MetaMap",
            "c": {
                "abstract": {
                    "t": "MetaInlines",
                    "c": [
                        {
                            "t": "Str",
                            "c": "Abstract"
                        }
                    ]
                },
                "affiliations": {
                    "t": "MetaInlines",
                    "c": [
                        {
                            "t": "Str",
                            "c": "Affiliations"
                        }
                    ]
                },
                "authors": {
                    "t": "MetaInlines",
                    "c": [
                        {
                            "t": "Str",
                            "c": "Authors"
                        }
                    ]
                },
                "description": {
                    "t": "MetaInlines",
                    "c": [
                        {
                            "t": "Str",
                            "c": "Description"
                        }
                    ]
                },
                "doi": {
                    "t": "MetaInlines",
                    "c": [
                        {
                            "t": "Str",
                            "c": "Doi"
                        }
                    ]
                },
                "keywords": {
                    "t": "MetaInlines",
                    "c": [
                        {
                            "t": "Str",
                            "c": "Keywords"
                        }
                    ]
                },
                "modified": {
                    "t": "MetaInlines",
                    "c": [
                        {
                            "t": "Str",
                            "c": "Modified"
                        }
                    ]
                },
                "published": {
                    "t": "MetaInlines",
                    "c": [
                        {
                            "t": "Str",
                            "c": "Published"
                        }
                    ]
                },
                "related_formats": {
                    "t": "MetaInlines",
                    "c": [
                        {
                            "t": "Str",
                            "c": "Other Formats"
                        }
                    ]
                }
            }
        },
        "lang": {
            "t": "MetaInlines",
            "c": [
                {
                    "t": "Str",
                    "c": "en"
                }
            ]
        },
        "link-citations": {
            "t": "MetaBool",
            "c": true
        },
        "pagetitle": {
            "t": "MetaString",
            "c": "test-fig"
        },
        "quarto-template-params": {
            "t": "MetaMap",
            "c": {
                "title-block-categories": {
                    "t": "MetaInlines",
                    "c": [
                        {
                            "t": "Str",
                            "c": "true"
                        }
                    ]
                }
            }
        },
        "quarto-version": {
            "t": "MetaInlines",
            "c": [
                {
                    "t": "Str",
                    "c": "99.9.9"
                }
            ]
        },
        "toc-title": {
            "t": "MetaInlines",
            "c": [
                {
                    "t": "Str",
                    "c": "Table"
                },
                {
                    "t": "Space"
                },
                {
                    "t": "Str",
                    "c": "of"
                },
                {
                    "t": "Space"
                },
                {
                    "t": "Str",
                    "c": "contents"
                }
            ]
        }
    },
    "blocks": [
        {
            "t": "Div",
            "c": [
                [
                    "",
                    [],
                    [
                        [
                            "__quarto_custom",
                            "true"
                        ],
                        [
                            "__quarto_custom_type",
                            "FloatRefTarget"
                        ],
                        [
                            "__quarto_custom_context",
                            "Block"
                        ],
                        [
                            "__quarto_custom_id",
                            "1"
                        ]
                    ]
                ],
                [
                    {
                        "t": "Div",
                        "c": [
                            [
                                "",
                                [],
                                [
                                    [
                                        "__quarto_custom_scaffold",
                                        "true"
                                    ]
                                ]
                            ],
                            [
                                {
                                    "t": "Plain",
                                    "c": [
                                        {
                                            "t": "Image",
                                            "c": [
                                                [
                                                    "",
                                                    [],
                                                    []
                                                ],
                                                [],
                                                [
                                                    "https://example.com/image.png",
                                                    ""
                                                ]
                                            ]
                                        }
                                    ]
                                }
                            ]
                        ]
                    },
                    {
                        "t": "Div",
                        "c": [
                            [
                                "",
                                [],
                                [
                                    [
                                        "__quarto_custom_scaffold",
                                        "true"
                                    ]
                                ]
                            ],
                            [
                                {
                                    "t": "Plain",
                                    "c": [
                                        {
                                            "t": "Str",
                                            "c": "A"
                                        },
                                        {
                                            "t": "Space"
                                        },
                                        {
                                            "t": "Str",
                                            "c": "figure"
                                        },
                                        {
                                            "t": "Space"
                                        },
                                        {
                                            "t": "Str",
                                            "c": "with"
                                        },
                                        {
                                            "t": "Space"
                                        },
                                        {
                                            "t": "Str",
                                            "c": "a"
                                        },
                                        {
                                            "t": "Space"
                                        },
                                        {
                                            "t": "Str",
                                            "c": "caption."
                                        }
                                    ]
                                }
                            ]
                        ]
                    }
                ]
            ]
        },
        {
            "t": "Div",
            "c": [
                [
                    "3ade8a4a-fb1d-4a6c-8409-ac45482d5fc9",
                    [
                        "hidden"
                    ],
                    []
                ],
                []
            ]
        }
    ]
}

Let me draw attention to two notable features:

• a Div with __quarto_custom attribute set to true. This indicates a "Custom AST node" for Quarto, here a FloatRefTarget node.
• a Div with __quarto_custom_scaffold attribute set to true. This stores the "content" of custom AST nodes.

In most cases, I would recommend these nodes to not be changed in any way. (It's in principle possible for JSON filters to construct new Quarto Custom AST nodes. But I expect a very small set of users would be interested in eg. programmatically creating a Quarto callout node inside a JSON filter, so let's wait to cross that river until anyone's interested in doing so.)

[cscheid]

Finally, Quarto 1.2.269 is 18 months old and uses a correspondingly old version of Pandoc (2.19.2). I would fairly strongly recommend upgrading to the latest stable version.