diff --git a/developer-workflow/grammar.rst b/developer-workflow/grammar.rst index dfb6a2f9d..ee6bdbaa4 100644 --- a/developer-workflow/grammar.rst +++ b/developer-workflow/grammar.rst @@ -42,7 +42,7 @@ Note: sometimes things mysteriously don't work. Before giving up, try ``make cl Then run ``make regen-ast`` to regenerate :cpy-file:`Include/internal/pycore_ast.h` and :cpy-file:`Python/Python-ast.c`. -* :cpy-file:`Parser/tokenizer.c` contains the tokenization code. +* :cpy-file:`Parser/lexer/` contains the tokenization code. This is where you would add a new type of comment or string literal, for example. * :cpy-file:`Python/ast.c` will need changes to validate AST objects diff --git a/internals/compiler.rst b/internals/compiler.rst index 5dd949d9d..a4e8457c3 100644 --- a/internals/compiler.rst +++ b/internals/compiler.rst @@ -11,7 +11,7 @@ Abstract In CPython, the compilation from source code to bytecode involves several steps: -1. Tokenize the source code (:cpy-file:`Parser/tokenizer.c`). +1. Tokenize the source code (:cpy-file:`Parser/lexer/` and :cpy-file:`Parser/tokenizer/`). 2. Parse the stream of tokens into an Abstract Syntax Tree (:cpy-file:`Parser/parser.c`). 3. Transform AST into an instruction sequence (:cpy-file:`Python/compile.c`). diff --git a/internals/parser.rst b/internals/parser.rst index db9922a80..ac5f9ba49 100644 --- a/internals/parser.rst +++ b/internals/parser.rst @@ -599,8 +599,9 @@ If you are on Windows you can use the Visual Studio project files to regenerate ./PCbuild/build.bat --regen -How tokens are generated and the rules governing this is completely up to the tokenizer (:cpy-file:`Parser/tokenizer.c`) -and the parser just receives tokens from it. +How tokens are generated and the rules governing this are completely up to the tokenizer +(:cpy-file:`Parser/lexer/` and :cpy-file:`Parser/tokenizer/`); +the parser just receives tokens from it. Memoization -----------