Hacking CPython Grammar: Supporting Standalone Walrus Assignments

Python 3.8 introduced assignment expressions via the walrus operator (:=) under PEP 572. While standard assignment statements (such as a = 10) simply bind a value to a name, assignment expressions both assign the value and return it as an expression value, allowing inline assignment and conditional testing.

However, a curious syntax restriction exists in Python:

# This works:
a = 10

# This raises SyntaxError: invalid syntax:
a := 10

# Yet wrapping it in parentheses works completely fine:
(a := 10)

By exploring CPython’s grammar definition and AST compilation pipeline, we can diagnose why standalone walrus assignments fail and patch the grammar file to make a := 10 a valid first-class statement.

---

1. The Utility of Assignment Expressions

Before modifying compiler tables, consider why the walrus operator was introduced. A classic use case is avoiding duplicated code when consuming streams, prompts, or regex matches.

The Naive Loop Pattern

Consider a minimalist REPL shell implementation in Python:

import os

command = input("> ")
while command != "exit":
    os.system(command)
    command = input("> ")

In this pattern, command = input("> ") is duplicated once before the loop to prime the condition and once at the end of the loop body to fetch the next input. An alternative using while True: with an if command == "exit": break construct adds boilerplate and obfuscates the termination invariant.

The Walrus Operator Pattern

Using the assignment expression operator (:=), the value is bound to command and immediately evaluated against the exit condition:

import os

while (command := input("> ")) != "exit":
    os.system(command)

This makes the code concise and guarantees that command is updated and validated in a single atomic-looking step.

---

2. The Walrus Anomaly

Evaluating the walrus operator at the top level produces an unexpected inconsistency:

>>> a = 10       # Valid assignment statement
>>> (a := 10)    # Valid assignment expression: returns 10, sets a = 10
10
>>> a := 10
  File "<stdin>", line 1
    a := 10
       ^
SyntaxError: invalid syntax

Why does (a := 10) evaluate without issue, while a := 10 raises a SyntaxError?

To understand this, we must look into the CPython grammar. Note that Python 3.8 used the original pgen LL(1) parser generator (defined in Grammar/Grammar). Python 3.9+ transitioned to a PEG parser (defined in Grammar/python.gram), but the architectural flow from concrete syntax grammar to Abstract Syntax Tree (AST) remains conceptually identical.

---

3. Tracing the CPython 3.8 Grammar

In CPython 3.8, the language grammar is defined in Grammar/Grammar.

Where := Is Defined: Named Expressions

Searching for := in Grammar/Grammar reveals that the walrus operator is internally referred to as a named expression:

named_expr_test: test [':=' test]

Inspecting where named_expr_test is accepted reveals rules for conditional statements and list comprehensions:

if_stmt: 'if' named_expr_test ':' suite ('elif' named_expr_test ':' suite)* ['else' ':' suite]
while_stmt: 'while' named_expr_test ':' suite ['else' ':' suite]
testlist_comp: (named_expr_test|star_expr) ( comp_for | (',' (named_expr_test|star_expr))* [','] )

Why Parentheses Work: The atom Rule

Tracing testlist_comp leads directly to the atom production rule:

atom: ('(' [yield_expr|testlist_comp] ')' |
       '[' [testlist_comp] ']' |
       '{' [dictorsetmaker] '}' |
       NAME | NUMBER | STRING+ | '...' | 'None' | 'True' | 'False')

An atom represents the smallest syntactic unit in expressions. The grammar explicitly allows testlist_comp—which includes named_expr_test—when surrounded by parentheses ( ) or square brackets [ ].

Consequently, (a := 10) parses as a parenthesized atom wrapping a testlist_comp. Bare statements, however, do not match the atom rule.

---

4. The Anatomy of Assignment Statements

To see why a = 10 is allowed as a standalone statement, we trace how Python parses general statements:

graph TD
    Statement[stmt] --> SimpleStmt[simple_stmt]
    SimpleStmt --> SmallStmt[small_stmt]
    SmallStmt --> ExprStmt[expr_stmt]
    ExprStmt --> StandardAssign["testlist_star_expr ('=' ...)"]
    ExprStmt --> AugAssign["augassign (+=, -=, ...)"]
    ExprStmt --> AnnAssign["annassign (a: int = 1)"]

In Grammar/Grammar:

simple_stmt: small_stmt (';' small_stmt)* [';'] NEWLINE
small_stmt: (expr_stmt | del_stmt | pass_stmt | flow_stmt |
             import_stmt | global_stmt | nonlocal_stmt | assert_stmt)

Looking closer at expr_stmt:

expr_stmt: testlist_star_expr (annassign | augassign | ('=' (yield_expr|testlist_star_expr))*)

This single production handles:

1. Annotated assignments (annassign): e.g., a: int = 1
2. Augmented assignments (augassign): e.g., a += 10
3. Standard assignments: e.g., a = b = 10

Because ':=' was intentionally omitted from expr_stmt under PEP 572, the parser rejects a := 10 as an invalid statement.

---

5. Modifying the Grammar to Support Standalone :=

We can alter Grammar/Grammar so that expr_stmt accepts ':=' alongside standard assignment '='.

Step 1: Update expr_stmt

Modify Grammar/Grammar around line 50:

- expr_stmt: testlist_star_expr (annassign | augassign | ('=' (yield_expr|testlist_star_expr))*)
+ expr_stmt: testlist_star_expr (annassign | augassign | (('=' | ':=') (yield_expr|testlist_star_expr))*)

By adding ':=' as an alternative to '=', the parser recognizes bare walrus statements without requiring enclosing parentheses.

Step 2: Regenerate the Parser and Recompile

Because Grammar/Grammar is compiled into C header and parser tables, the auto-generated parser source files must be regenerated using pgen:

# Regenerate parser files from Grammar/Grammar
make regen-grammar

# Recompile the Python binary
make -j$(nproc)

Step 3: Verifying the Grammar Update

Launching the newly built ./python binary:

>>> a = 10
>>> a
10
>>> (a := 20)
20
>>> a := 30
>>> a
30

a := 30 now executes without raising a SyntaxError.

---

6. Why Did This Work Without Changing C Code?

Changing the grammar allowed the parser to accept the syntax, but what converted a := 30 into a functional variable assignment without adding C logic?

The answer lies in how CPython constructs its Abstract Syntax Tree in Python/ast.c.

AST Generation in ast_for_expr_stmt

When CPython parses an expression statement, it routes the parse tree node to ast_for_expr_stmt() in Python/ast.c:

static stmt_ty
ast_for_expr_stmt(struct compiling *c, const node *n)
{
    REQ(n, expr_stmt);
    int num = NCH(n); // Number of child nodes in the parse tree

    if (num == 1) {
        // Expression without assignment (e.g., bare variable or function call)
        return Expr(...);
    }
    else if (TYPE(CHILD(n, 1)) == augassign) {
        // Augmented assignment (e.g., a += 1)
        return AugAssign(...);
    }
    else if (TYPE(CHILD(n, 1)) == annassign) {
        // Annotated assignment (e.g., a: int = 1)
        return AnnAssign(...);
    }
    else {
        // Fallback branch: Handles standard assignments
        return Assign(...);
    }
}

Breakdown of Node Processing

For a := 30, the parse node expr_stmt has 3 children (NCH(n) == 3):

1. CHILD(n, 0): testlist_star_expr (a)
2. CHILD(n, 1): Operator token (':=')
3. CHILD(n, 2): Expression (30)

Let us trace the conditions:

1. num == 1: False (num is 3).
2. TYPE(CHILD(n, 1)) == augassign: False (:= is not an augmented assignment operator).
3. TYPE(CHILD(n, 1)) == annassign: False (:= is not a type annotation operator).
4. Fallback (else): CPython reaches the standard assignment handler and creates an Assign AST node.

Because the grammar modification routed a := 30 through the default Assign AST branch, CPython naturally emits an assignment bytecode sequence, binding the value 30 to variable a.

---

7. Rationale: Why Did Python Disallow Standalone :=?

The omission of bare a := 10 in PEP 572 was an intentional design decision by Guido van Rossum and the CPython core team:

Design Consideration	Rationale
Syntactic Ambiguity	In Python, = is an assignment statement, while == is an equality comparison. Allowing bare := would introduce a redundant, third assignment style at statement level, confusing developers coming from languages where := is standard (e.g., Go).
Visual Bug Prevention	Disallowing bare := prevents accidental typos when an author intends to write an equality check a == 10 or an assignment a = 10, but mistakenly types a := 10.
The Zen of Python	”There should be one— and preferably only one —obvious way to do it.” Standard assignment statements already serve this role via =.

---

Key Takeaways

• Assignment Expressions vs. Statements: Assignment statements (=) bind values to names within a scope, whereas assignment expressions (:=) simultaneously assign and return the value.
• The Grammar Rule Difference: (a := 1) works out-of-the-box because parentheses classify the construct as an atom containing a testlist_comp. Unparenthesized a := 1 falls under expr_stmt, which historically whitelisted only =, augassign, and annassign.
• Compilation Flow: Modifying Grammar/Grammar requires regenerating parser tables (make regen-grammar) so that the parser driver recognizes the new token sequences.
• AST Fallback: In CPython’s ast.c, the default non-augmented, non-annotated branch of ast_for_expr_stmt generates an Assign node, allowing bare walrus expressions to function automatically once permitted by the grammar.