How CPython Processes Python Source Code

Have you ever wondered how the language we know and love works under the hood? I know I have. I went digging under the hood to find out. I found the CPython Internals book very helpful, along with the official documentation and source code. I now understand why the CPython Internals book is highly recommended. Thanks, Anthony Shaw, for signing my book!

This article is a version of the notes I took while exploring how everything works. But before we conquer the world of CPython internals, we will start at the very beginning, processing source code from various inputs.

Buckle up, it's going to be chaotic!

With every journey, we need to know where we are heading. Our first destination is struct tok_state.

struct tok_state is the core abstraction layer CPython uses to read characters consistently, regardless of their source. Whether input originates from a file on disk, an in-memory string, or a Python readline() callable, struct tok_state feeds the tokenizer a uniform character stream.

The core fields inside Parser/lexer/state.h are:

struct tok_state {
    char *buf;          // input buffer
    char *cur;          // next character
    char *inp;          // end of available data
    const char *end;    // end of allocated buffer

    FILE *fp;           // file input, if any
    PyObject *readline; // readline() callable, if any

    char *encoding;
    enum decoding_state decoding_state;

    int (*underflow)(struct tok_state *);
    ...
};

Visually, this is what the process looks like:

                     source
                       |
        +--------------+--------------+
        |              |              |
        v              v              v
     FILE *        C string       readline()
        |              |              |
  _PyTokenizer_   _PyTokenizer_   _PyTokenizer_
   FromFile()      FromUTF8()     FromReadline()
                   FromString()
        |              |              |
        +--------------+--------------+
                       |
                       v
               struct tok_state
                       |
                       v
               _PyTokenizer_Get()
                       |
                    tokens
                       |
             +---------+---------+
             |                   |
             v                   v
        PEG parser        tokenize module
      (file, string)         (stream)
             |                   |
             v                   v
            AST          TokenInfo objects

File input

We will use the addition example shown in addition.py throughout the article.

# addition.py

x = 40
y = 2
print(x + y)

Run it with:

python addition.py

The interpreter reads and executes the script in addition.py.

Here is what happens in the background between hitting enter and the first token when processing the file input:

1. The C entry point: The OS (operating system) starts the process at main() in Programs/python.c.
2. Initialization in Modules/main.c: Py_BytesMain() packs argc/argv into a _PyArgv and calls pymain_main(), which runs pymain_init() and Py_RunMain().
   • pymain_init() builds the runtime by:
     • Pre-initialization: _Py_PreInitializeFromPyArgv picks the memory allocators and the UTF-8 mode before any Python object exists.
     • Configuration: PyConfig_SetBytesArgv() reads the command-line arguments and environment variables, and records the target path in run_filename (addition.py in our example). It doesn't open the file yet.
     • Initialization: Py_InitializeFromConfig() creates the main interpreter, thread state, garbage collector state, and import system. By default, the build always has the GIL (Global Interpreter Lock), but a free-threaded build runs without it.
   • Py_RunMain() runs and finalises the program.
3. Deciding what to run: Since we passed a path, CPython takes pymain_run_file(), which converts the path to a string and calls pymain_run_file_obj(). The function:
   • raises the cpython.run_file audit event,
   • opens the file in binary mode with Py_fopen(filename, "rb"),
   • skips the first line if you passed -x,
   • refuses directories,
   • calls _PyRun_AnyFileObject() with closeit = 1.
4. From opening the file to the parser: _PyRun_AnyFileObject() checks _Py_FdIsInteractive(). If it is not interactive, it goes to _PyRun_SimpleFileObject().
   • _PyRun_SimpleFileObject() sets __main__.__file__, then maybe_pyc_file() checks the extension and magic number.
   • A .pyc skips the parser entirely via run_pyc_file(). Since ours is source, it calls pyrun_file().
   • pyrun_file() calls _PyParser_ASTFromFile(), then compiles and evaluates the AST (Abstract Syntax Tree).
   • _PyParser_ASTFromFile() raises the compile audit event and calls _PyPegen_run_parser_from_file_pointer().
   • _PyPegen_run_parser_from_file_pointer() builds the tokenizer with _PyTokenizer_FromFile().

This completes the file-loading phase.

String input

If we pass the same addition of x and y with no file, like:

python -c "x = 40; y = 2; print(x + y)"

Here is what is happening:

1. The C entry point: similar to file input.
2. Initialization in Modules/main.c: similar to file input, with a distinction in what the config reader stores: -c sets run_command = "x = 40; y = 2; print(x + y)", while run_filename stays NULL.
3. Deciding what to run: Because we passed a command with -c, CPython branches to pymain_run_command(). It is the counterpart of pymain_run_file_obj(). The function:
   • raises the cpython.run_command audit event,
   • dedents the command,
   • encodes it to UTF-8 mode,
   • sets PyCF_IGNORE_COOKIE, because the text is already Unicode,
   • calls _PyRun_SimpleStringFlagsWithName() with the name "<string>".
4. From string to the parser: _PyRun_SimpleStringFlagsWithName() runs the code in __main__.__dict__ via _PyRun_StringFlagsWithName().
   • _PyRun_StringFlagsWithName() calls _PyParser_ASTFromString(), then compiles and evaluates the AST.
   • _PyParser_ASTFromString() raises the compile audit event and calls _PyPegen_run_parser_from_string().
   • _PyPegen_run_parser_from_string() builds the tokenizer. Because PyCF_IGNORE_COOKIE is set, it picks _PyTokenizer_FromUTF8().

This completes the string-loading phase.

Stream input

addition.py is read as a stream of lines by the tokenize module:

python -m tokenize addition.py

Background activities happening:

1. The C entry point: Identical to file input.
2. Initialization in Modules/main.c: Alike to file input, with a contradiction in what the config reader stores: -m sets run_module = "tokenize", and addition.py lands in sys.argv[1].
3. Deciding what to run: We passed a module with -m, so CPython switches to pymain_run_module(). The function:
   • raises the cpython.run_module audit event,
   • imports runpy._run_module_as_main,
   • calls it with "tokenize", which finds Lib/tokenize.py and executes it as __main__. Loading tokenize.py itself is a normal import, usually from its cached .pyc, so it doesn't go through the parser.
4. From readline() to the tokenizer: _main() opens addition.py in binary mode and calls tokenize(f.readline).
   • tokenize() detects the encoding,
   • then calls _generate_tokens_from_c_tokenizer(), which builds _tokenize.TokenizerIter(),
   • tokenizeriter_new_impl() in Python/Python-tokenize.c calls _PyTokenizer_FromReadline(readline, encoding, 1, 1),
   • _PyTokenizer_FromReadline() stores the callable on the tok_state and installs tok_underflow_readline as its refill function,
   • On each refill, tok_readline_string() calls readline() from C with PyObject_CallNoArgs(), StopIteration means end of input,
   • Every next() on the iterator calls _PyTokenizer_Get() and wraps the result as a TokenInfo, which _main() prints.

The stream path stops at tokens. There’s no public API that builds an AST from TokenInfo objects, so you have to untokenize them back into a string and send that through the string path.

Where the paths meet: tok_state

We are back to the tok_state. All three paths end the same way: a constructor returns the struct tok_state as illustrated previously. From this stage, the lexer does not know where the bytes came from.

Each constructor fills buf and installs its own underflow.

Buf is the temporary memory holding the batch of characters being tokenized and the underflow pointer holds a callback function that fetches the next chunk of data into buf

That function pointer is the only place the three inputs still differ:

• For a file, tok_underflow_file reads the next line from fp.
• At the REPL, tok_underflow_interactive prints the >>> prompt first, then reads the line.
• For a string, the whole source is already in memory, so tok_underflow_string moves to the next line.
• For a stream, _PyTokenizer_FromReadline() installs tok_underflow_readline, which calls the Python readline() callable for each new line.

The lexer reads one character at a time with tok_nextc(). When the buffer is empty, it calls underflow to get the next line.

Tokens come out of _PyTokenizer_Get():

int
_PyTokenizer_Get(struct tok_state *tok, struct token *token)
{
    int result = tok_get(tok, token);
    if (tok->decoding_erred) {
        result = ERRORTOKEN;
        tok->done = E_DECODE;
    }
    return result;
}

tok_get() picks a mode: - tok_get_normal_mode() for ordinary code, - tok_get_fstring_mode() inside f-strings and t-strings.

Each call returns one token type and fills the struct token with its position. Indentation becomes INDENT/DEDENT tokens here, per the lexical analysis rules.

For our file addition.py, python -m tokenize addition.py prints:

0,0-0,0:            ENCODING       'utf-8'
1,0-1,1:            NAME           'x'
1,2-1,3:            OP             '='
1,4-1,6:            NUMBER         '40'
1,6-1,7:            NEWLINE        '\n'
2,0-2,1:            NAME           'y'
2,2-2,3:            OP             '='
2,4-2,5:            NUMBER         '2'
2,5-2,6:            NEWLINE        '\n'
3,0-3,5:            NAME           'print'
3,5-3,6:            OP             '('
3,6-3,7:            NAME           'x'
3,8-3,9:            OP             '+'
3,10-3,11:          NAME           'y'
3,11-3,12:          OP             ')'
3,12-3,13:          NEWLINE        '\n'
4,0-4,0:            ENDMARKER      ''

The parser gets the same stream with two differences. ENCODING is added by the tokenize module only. And every OP reaches the parser as its exact type: EQUAL, LPAR, PLUS, RPAR.

Handing tokens to the parser

Here the file and string paths join. Both wrap the tok_state in a Parser, run it with _PyPegen_run_parser(), then free the tokenizer.

The parser asks for tokens one at a time. Each time a grammar rule needs the next one, _PyPegen_fill_token() calls _PyTokenizer_Get(). That call is where the lexer and the parser meet.

The stream path never gets here. Its tokens go back to Python as TokenInfo objects.

The parser returns an AST. run_mod() compiles it to bytecode and runs it, and 42 is printed as the sum of our program.

The next phase is going to dig into the steps from parsing to execution.

References:

• Python Initialization Configuration
• CPython's Internals
• Python Initialization Configuration
• Lexical analysis
• tokenize
• PEP 263
• PEP 617
• PEP 701
• Devguide: Guide to the parser