Local names implementation

[andrewleech]

Summary

Started with #4 checked out. Then in a new Claude micropython session:

perform a comprehensive code review of the 6 latest commits, reviewing the changes in each one sequentially.. build up a detailed technical log in markdown of the sequence of changes

before these changes, local variable names are discarded and just the the values retained in an array which are then referred to by index. this process is handled in both the compiler and the vm

the initial commits focus on minimal change to expose the local variables by index only to python, via the profile framework. this allows them to be accessed in sys.settrace callbacks. consolidate just these changes into one minimal simplified commit, exclude the larger charge to retain variable names for now. ensure the author on this commits is retained as Jos as per . then think extra hard about how to add unit tests into the standard micropython test suite for this settrace local variable access (by index only)

micropython uses string interning, names of functions, classes, global attributes etc are stored as QSTR_* then referenced by the integer id of that QSTR. strings in C code are parsed by a custom pre-processor script and stored separately in the c binary. for the names of objects in python code the compiler must collect and retain them in the created bytecode. currently the names of local variables is not retained the way other things like functions are, this will need to change so they're retained the same way. the references to these names should be stored in a new list/array, then length of which should be calculated in an equivalent way to hiow the length of the existing local variables values array is calculated/allocated. ideally this new variable names lookup array is stored in such a way it can be accessed by the profile module, or as a new __ attribute on functions itself, or a new .co_ attribute on the code object; think deeply about options like this and find one that would not result in a change to the bytecode format. also prioritise options that minimise the size of code change. the end goal is the have a list of names that can be associated with the list of local values accessed in a systrace callback. think deeply about options, present them to me in a technical planning document.

Executive Summary

This document presents architectural options for preserving local variable names in MicroPython's compilation and runtime system, enabling proper name resolution in sys.settrace() callbacks without modifying the bytecode format.

Current Architecture Analysis

String Interning (QSTR) System

• MicroPython uses QSTR (interned strings) for all identifiers
• Function names, class names, and global attributes are already preserved as QSTRs
• Local variable names exist as QSTRs during compilation but are discarded after bytecode generation
• The QSTR system is highly optimized for memory efficiency

Compilation Flow

Source Code → Lexer (QSTRs created) → Parser → Scope Analysis → Bytecode Emission
                                                      ↓
                                              Local names exist here
                                              but are discarded

Current Data Structures

// During compilation (py/scope.h)
typedef struct _id_info_t {
    uint8_t kind;
    uint8_t flags;
    uint16_t local_num;
    qstr qst;  // Variable name as QSTR - currently discarded for locals
} id_info_t;

// Runtime structure (py/emitglue.h)
typedef struct _mp_raw_code_t {
    // ... existing fields ...
    mp_bytecode_prelude_t prelude;  // Contains n_state, n_pos_args, etc.
    // No local name information preserved
} mp_raw_code_t;

Design Constraints

1. No bytecode format changes - Existing .mpy files must remain compatible
2. Minimal code changes - Reduce implementation complexity
3. Memory efficiency - Only store when needed (conditional compilation)
4. QSTR integration - Leverage existing string interning system
5. Profile accessibility - Names must be accessible from sys.settrace() callbacks

Proposed Solutions

Option 1: Extend mp_raw_code_t (Recommended)

Implementation:

// py/emitglue.h
typedef struct _mp_raw_code_t {
    // ... existing fields ...
    #if MICROPY_PY_SYS_SETTRACE_LOCALNAMES
    const qstr *local_names;  // Array of QSTRs indexed by local_num
    #endif
} mp_raw_code_t;

Advantages:

• Natural location alongside other function metadata
• Already accessible from profile module via code_state->fun_bc->rc
• Follows existing pattern (line_info storage)
• No runtime overhead when disabled

Implementation Details:

1. Allocate during mp_emit_glue_new_raw_code()
2. Populate in scope_compute_things() after local_num assignment
3. Access in frame_f_locals() to map indices to names

Memory Cost:

• sizeof(qstr) * num_locals per function (typically 2-4 bytes per local)

Option 2: Function Object Attribute

Implementation:

// Add new attribute to function objects
// Accessed as: func.__localnames__ or func.co_varnames

Advantages:

• Python-accessible for introspection
• Compatible with CPython's co_varnames
• No change to raw_code structure

Disadvantages:

• Requires modifying function object creation
• Additional indirection at runtime
• More complex implementation

Option 3: Separate Global Mapping

Implementation:

// Global hash table: raw_code_ptr → local_names_array
static mp_map_t raw_code_to_locals_map;

Advantages:

• Completely decoupled from existing structures
• Can be added/removed without touching core structures

Disadvantages:

• Additional lookup overhead
• Memory overhead for hash table
• Cleanup complexity for garbage collection

Option 4: Encode in Bytecode Prelude

Implementation:

• Extend bytecode prelude with optional local names section
• Use a flag bit to indicate presence

Advantages:

• Data travels with bytecode
• Works with frozen bytecode

Disadvantages:

• Violates constraint: Changes bytecode format
• Breaks .mpy compatibility
• Increases bytecode size

Recommended Implementation Plan

Phase 1: Core Infrastructure (Option 1)

1. Add conditional field to mp_raw_code_t:

// py/emitglue.h
#if MICROPY_PY_SYS_SETTRACE_LOCALNAMES
const qstr *local_names;  // NULL if no locals or disabled
#endif

2. Allocate and populate during compilation:

// py/compile.c - in scope_compute_things()
#if MICROPY_PY_SYS_SETTRACE_LOCALNAMES
if (scope->num_locals > 0) {
    // Allocate array
    qstr *names = m_new0(qstr, scope->num_locals);
    
    // Populate from id_info
    for (int i = 0; i < scope->id_info_len; i++) {
        id_info_t *id = &scope->id_info[i];
        if (ID_IS_LOCAL(id->kind) && id->local_num < scope->num_locals) {
            names[id->local_num] = id->qst;
        }
    }
    
    scope->raw_code->local_names = names;
}
#endif

3. Use in profile module:

// py/profile.c - in frame_f_locals()
#if MICROPY_PY_SYS_SETTRACE_LOCALNAMES
const mp_raw_code_t *rc = code_state->fun_bc->rc;
if (rc->local_names != NULL) {
    qstr name = rc->local_names[i];
    if (name != MP_QSTR_NULL) {
        // Use actual name instead of local_XX
    }
}
#endif

Phase 2: Python Accessibility (Optional)

Add co_varnames attribute to code objects:

def func(a, b):
    x = 1
    y = 2

print(func.__code__.co_varnames)  # ('a', 'b', 'x', 'y')

Memory Optimization Strategies

1. Share common patterns:

   • Many functions have similar local patterns (i, j, x, y)
   • Could use a pool of common name arrays

2. Compress storage:

   • Store only non-parameter locals (parameters can be reconstructed)
   • Use bit flags for common names

3. Lazy allocation:

   • Only allocate when settrace is active
   • Use weak references for cleanup

Size Impact Analysis

Typical function with 4 locals:

• Storage: 4 * sizeof(qstr) = 8-16 bytes
• Overhead: ~0.5% of typical raw_code size

Mitigation:

• Only enabled with MICROPY_PY_SYS_SETTRACE_LOCALNAMES
• Zero cost when disabled

Testing Strategy

1. Correctness tests:

   • Verify name mapping matches source order
   • Handle edge cases (no locals, many locals)
   • Test with nested functions and closures

2. Memory tests:

   • Measure overhead with typical programs
   • Verify cleanup on function deallocation

3. Compatibility tests:

   • Ensure .mpy files work unchanged
   • Test frozen bytecode compatibility

Conclusion

Option 1 (Extend mp_raw_code_t) provides the best balance of:

• Minimal code changes
• Natural integration with existing architecture
• Zero overhead when disabled
• Direct accessibility from profile module

This approach preserves the bytecode format while enabling full local variable name resolution in debugging scenarios.

---

The implantation here started as suggested above, but then extended to also include the "bytecode modification" option as well behind second feature flag.

● Local Variable Names Schemes Summary ## Phase 1 (RAM Storage - MICROPY_PY_SYS_SETTRACE_LOCALNAMES) preserves local variable names in memory during source file compilation by extending the mp_raw_code_t structure with a local_names array. This approach provides excellent debugging experience for source-based development with minimal implementation complexity and zero bytecode format changes, ensuring complete backward compatibility. However, it has significant limitations: local variable names are only available when running from source files, .mpy files fall back to generic local_XX naming, and there's a runtime memory overhead of ~8 bytes plus 4-8 bytes per local variable per function. This makes it ideal for development and debugging workflows but less suitable for production deployments using pre-compiled .mpy files.

Phase 2 (Bytecode Persistence -

MICROPY_PY_SYS_SETTRACE_LOCALNAMES_PERSIST) extends the bytecode format to store local variable names
directly in the source info section of .mpy files, enabling debugging support for pre-compiled modules. The major advantages include complete debugging coverage for both source and .mpy files, persistent local names that survive compilation, graceful compatibility across MicroPython versions (files with local names work on older versions, just without the names), and relatively small file size
overhead (~1-5 bytes plus ~10 bytes per local
variable). The main drawbacks are increased .mpy
file sizes proportional to the number of local variables, slightly more complex implementation requiring bytecode format extensions, and additional
compilation time to encode the local names. This
scheme provides the most comprehensive solution for production debugging scenarios where .mpy files are the primary deployment format.

Testing

Trade-offs and Alternatives