INNER CODE UNIT · Python

tokenize

kejilion/sh · site_access_nginx.py:42

def tokenize(text: str):
    """Yield decoded Nginx tokens together with their original source spans."""
    n, i = len(text), 0
    while i < n:
        if text[i].isspace():
            i += 1
            continue
        if text[i] == "#":
            while i < n and text[i] != "\n":
                i += 1
            continue
        if text[i] in "{};":
            yield Token(text[i], i, i + 1, True)
            i += 1
            continue
        start, quote, value = i, None, []
        while i < n:
            c = text[i]

View source record →