xml.etree parser does not accept valid control characters

[romuald]

BPO	43703
Nosy	@scoder, @serhiy-storchaka, @romuald

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = None
closed_at = None
created_at = <Date 2021-04-02.11:03:35.614>
labels = ['expert-XML', '3.9']
title = 'xml.etree parser does not accept valid control characters'
updated_at = <Date 2021-04-02.14:39:51.136>
user = 'https://github.com/romuald'

bugs.python.org fields:

activity = <Date 2021-04-02.14:39:51.136>
actor = 'Romuald'
assignee = 'none'
closed = False
closed_date = None
closer = None
components = ['XML']
creation = <Date 2021-04-02.11:03:35.614>
creator = 'Romuald'
dependencies = []
files = []
hgrepos = []
issue_num = 43703
keywords = []
message_count = 3.0
messages = ['390050', '390065', '390066']
nosy_count = 4.0
nosy_names = ['scoder', 'eli.bendersky', 'serhiy.storchaka', 'Romuald']
pr_nums = []
priority = 'normal'
resolution = None
stage = None
status = 'open'
superseder = None
type = None
url = 'https://bugs.python.org/issue43703'
versions = ['Python 3.9']

[romuald]

Python XML parser (xml.etree) does not seems to allow control characters that are invalid in XML 1.0, but valid in XML 1.1 [1] [2]

Considering the following sample:

import xml.etree.ElementTree as ET

bad = '<?xml version="1.1"?><foo>bar � baz</foo>'
print(ET.fromstring(bad))

The parser raises the following error:
ParseError: reference to invalid character number: line 1, column 30

[1] https://www.w3.org/TR/xml11/Overview.html#charsets
[2] https://www.w3.org/TR/xml11/Overview.html#sec-xml11

[serhiy-storchaka]

It is a known issue, see bpo-11804 and bpo-39512.

In short, the underlying library for XML parsing (expat) does not support XML 1.1 and does not have plans to support it. And seems that XML 1.1 is a dead standard if it is not supported in popular parsing libraries.

From where you get your XML data? What programs generated them?

[romuald]

Thanks for the quick reply

We're getting data from about a hundred different providers around the world; some of them not really keen on standards, so we already have some hacks to fix invalid XML. We'll add one to the list

In that particular case, the XML was invalid anyway since it was an XML 1.0 document, and the character was sent as "binary" (\x19)