Regression in tokenizer handling of `\r`

[tusharsadhwani]

Bug report

Bug description:

Python 3.12 onwards we get a weird \r} token when trying to parse a file just containing '{\r}':

$ printf '{\r}' | python3.11 -m tokenize
1,0-1,1:            OP             '{'            
1,1-1,2:            ERRORTOKEN     '\r'           
1,2-1,3:            OP             '}'            
1,3-1,4:            NEWLINE        ''             
2,0-2,0:            ENDMARKER      ''             

$ printf '{\r}' | python3.12 -m tokenize
1,0-1,1:            OP             '{'            
1,1-1,3:            OP             '\r}'          
1,3-1,4:            NEWLINE        ''             
2,0-2,0:            ENDMARKER      ''   

Weirdly, AST generation passes just fine in both cases:

$ printf '{\r}' | python3.11 -m ast
Module(
   body=[
      Expr(
         value=Dict(keys=[], values=[]))],
   type_ignores=[])

$ printf '{\r}' | python3.12 -m ast
Module(
   body=[
      Expr(
         value=Dict(keys=[], values=[]))],
   type_ignores=[])

Expected behaviour

I'd expect the \r to yield a NL instead, and we get a } OP as expected.

CPython versions tested on:

3.11, 3.12, 3.13, 3.14

Operating systems tested on:

macOS

[tusharsadhwani]

There's one more interesting one, when the tokenizer seems to think that '\r ' is a non-whitespace token:

$ printf 'foo\n\r \nbar' | python3.11 -m tokenize
1,0-1,3:            NAME           'foo'          
1,3-1,4:            NEWLINE        '\n'           
2,0-2,3:            NL             '\r \n'        
3,0-3,3:            NAME           'bar'          
3,3-3,4:            NEWLINE        ''             
4,0-4,0:            ENDMARKER      ''             

$ printf 'foo\n\r \nbar' | python3.12 -m tokenize                  
1,0-1,3:            NAME           'foo'          
1,3-1,4:            NEWLINE        '\n'           
2,0-2,2:            OP             '\r '          
2,2-2,3:            NEWLINE        '\n'           
3,0-3,3:            NAME           'bar'          
3,3-3,4:            NEWLINE        ''             
4,0-4,0:            ENDMARKER      ''    

I would have expected the 2,2-2,3: NEWLINE '\n' case in Python 3.12 to be NL instead, as there is no semantic meaning to that newline. Python 3.11 categorizes that correctly as NL.

[serhiy-storchaka]

The behavior was changed because the tokenize module now uses the C tokenizer. And it turned out that a single CR is not always recognized as a newline separator. I tried to fix this today, but it is not so easy. Starting from the fact that the code is read by lines, but C's fgets() and Python's readline() read to the LF byte (this works for the CRLF newlines, but not for CR).

[tusharsadhwani]

Another weird case, where we also get a NL where we should be getting a NEWLINE:

$ python -m tokenize x.py           
0,0-0,0:            ENCODING       'utf-8'        
1,0-1,1:            NL             '\n'           
2,0-2,1:            COMMENT        '#'            
2,1-2,3:            NUMBER         '\r0'          
2,3-2,4:            NL             ''             
3,0-3,0:            ENDMARKER      ''

$ python3.11 -m tokenize x.py
0,0-0,0:            ENCODING       'utf-8'        
1,0-1,1:            NL             '\n'           
2,0-2,3:            COMMENT        '#\r0'         
2,3-2,3:            NL             ''             
3,0-3,0:            ENDMARKER      ''