From 401d3657da56f4709c3eae10bed73aff6c1466f1 Mon Sep 17 00:00:00 2001 From: Muhammad Mahad Date: Sun, 17 Dec 2023 16:23:28 +0000 Subject: [PATCH 1/3] Fixed token.c hyperlink in compiler design --- developer-workflow/grammar.rst | 2 +- internals/compiler.rst | 2 +- internals/parser.rst | 2 +- 3 files changed, 3 insertions(+), 3 deletions(-) diff --git a/developer-workflow/grammar.rst b/developer-workflow/grammar.rst index dfb6a2f9d..3aec24c1a 100644 --- a/developer-workflow/grammar.rst +++ b/developer-workflow/grammar.rst @@ -42,7 +42,7 @@ Note: sometimes things mysteriously don't work. Before giving up, try ``make cl Then run ``make regen-ast`` to regenerate :cpy-file:`Include/internal/pycore_ast.h` and :cpy-file:`Python/Python-ast.c`. -* :cpy-file:`Parser/tokenizer.c` contains the tokenization code. +* :cpy-file:`Parser/lexer/` & :cpy-file:`Parser/tokenizer/` contains the tokenization code. This is where you would add a new type of comment or string literal, for example. * :cpy-file:`Python/ast.c` will need changes to validate AST objects diff --git a/internals/compiler.rst b/internals/compiler.rst index 5dd949d9d..1c81d682b 100644 --- a/internals/compiler.rst +++ b/internals/compiler.rst @@ -11,7 +11,7 @@ Abstract In CPython, the compilation from source code to bytecode involves several steps: -1. Tokenize the source code (:cpy-file:`Parser/tokenizer.c`). +1. Tokenize the source code (:cpy-file:`Parser/lexer/` & :cpy-file:`Parser/tokenizer/`). 2. Parse the stream of tokens into an Abstract Syntax Tree (:cpy-file:`Parser/parser.c`). 3. Transform AST into an instruction sequence (:cpy-file:`Python/compile.c`). diff --git a/internals/parser.rst b/internals/parser.rst index db9922a80..59546e07b 100644 --- a/internals/parser.rst +++ b/internals/parser.rst @@ -599,7 +599,7 @@ If you are on Windows you can use the Visual Studio project files to regenerate ./PCbuild/build.bat --regen -How tokens are generated and the rules governing this is completely up to the tokenizer (:cpy-file:`Parser/tokenizer.c`) +How tokens are generated and the rules governing this is completely up to the tokenizer (:cpy-file:`Parser/lexer/` & :cpy-file:`Parser/tokenizer/`) and the parser just receives tokens from it. Memoization From ae8532b02bc943b36a2d5e4997063e8d7a9b4712 Mon Sep 17 00:00:00 2001 From: AlexWaygood Date: Sun, 17 Dec 2023 17:35:46 +0000 Subject: [PATCH 2/3] Wrap lines; small grammar fixes --- developer-workflow/grammar.rst | 3 ++- internals/compiler.rst | 2 +- internals/parser.rst | 5 +++-- 3 files changed, 6 insertions(+), 4 deletions(-) diff --git a/developer-workflow/grammar.rst b/developer-workflow/grammar.rst index 3aec24c1a..c8dcfb54a 100644 --- a/developer-workflow/grammar.rst +++ b/developer-workflow/grammar.rst @@ -42,7 +42,8 @@ Note: sometimes things mysteriously don't work. Before giving up, try ``make cl Then run ``make regen-ast`` to regenerate :cpy-file:`Include/internal/pycore_ast.h` and :cpy-file:`Python/Python-ast.c`. -* :cpy-file:`Parser/lexer/` & :cpy-file:`Parser/tokenizer/` contains the tokenization code. +* :cpy-file:`Parser/lexer/` and :cpy-file:`Parser/tokenizer/` contain the + tokenization code. This is where you would add a new type of comment or string literal, for example. * :cpy-file:`Python/ast.c` will need changes to validate AST objects diff --git a/internals/compiler.rst b/internals/compiler.rst index 1c81d682b..a4e8457c3 100644 --- a/internals/compiler.rst +++ b/internals/compiler.rst @@ -11,7 +11,7 @@ Abstract In CPython, the compilation from source code to bytecode involves several steps: -1. Tokenize the source code (:cpy-file:`Parser/lexer/` & :cpy-file:`Parser/tokenizer/`). +1. Tokenize the source code (:cpy-file:`Parser/lexer/` and :cpy-file:`Parser/tokenizer/`). 2. Parse the stream of tokens into an Abstract Syntax Tree (:cpy-file:`Parser/parser.c`). 3. Transform AST into an instruction sequence (:cpy-file:`Python/compile.c`). diff --git a/internals/parser.rst b/internals/parser.rst index 59546e07b..ac5f9ba49 100644 --- a/internals/parser.rst +++ b/internals/parser.rst @@ -599,8 +599,9 @@ If you are on Windows you can use the Visual Studio project files to regenerate ./PCbuild/build.bat --regen -How tokens are generated and the rules governing this is completely up to the tokenizer (:cpy-file:`Parser/lexer/` & :cpy-file:`Parser/tokenizer/`) -and the parser just receives tokens from it. +How tokens are generated and the rules governing this are completely up to the tokenizer +(:cpy-file:`Parser/lexer/` and :cpy-file:`Parser/tokenizer/`); +the parser just receives tokens from it. Memoization ----------- From 6eaed717b2a12068ee629d26f957ad88a1c548ad Mon Sep 17 00:00:00 2001 From: Muhammad Mahad Date: Tue, 19 Dec 2023 06:13:24 +0000 Subject: [PATCH 3/3] Point grammer.rst to pure lexer tokenization code --- developer-workflow/grammar.rst | 3 +-- 1 file changed, 1 insertion(+), 2 deletions(-) diff --git a/developer-workflow/grammar.rst b/developer-workflow/grammar.rst index c8dcfb54a..ee6bdbaa4 100644 --- a/developer-workflow/grammar.rst +++ b/developer-workflow/grammar.rst @@ -42,8 +42,7 @@ Note: sometimes things mysteriously don't work. Before giving up, try ``make cl Then run ``make regen-ast`` to regenerate :cpy-file:`Include/internal/pycore_ast.h` and :cpy-file:`Python/Python-ast.c`. -* :cpy-file:`Parser/lexer/` and :cpy-file:`Parser/tokenizer/` contain the - tokenization code. +* :cpy-file:`Parser/lexer/` contains the tokenization code. This is where you would add a new type of comment or string literal, for example. * :cpy-file:`Python/ast.c` will need changes to validate AST objects