Allow for custom Token definiton entry

[markopaden]

Currently, this module only allows adding definitions by adding TokenDefn objects. TokenDefn constructor has this line: $this->regex = sprintf('%s^%s%s%s', $delimiter, $regex, $delimiter, $modifiers); Sometimes, you don't want to add ^ into your expression. There should be a way to insert regex token only by giving it regex(string) and token name.

[tmilos]

Any suggestion how to define such regex in array config like below?

$config = new LexerArrayConfig([
            '\\s' => '',
            '\\d+' => 'number',
            '\\+' => 'plus',
            '-' => 'minus',
            '\\*' => 'mul',
            '/' => 'div',
        ]);

[markopaden]

None really. I made CustomTokenDefn class that extends TokenDefn:

class CustomTokenDefn extends TokenDefn
{
    /**
     * CustomTokenDefn constructor.
     *
     * @param string $name
     * @param string $regex
     */
    public function __construct(string $name, string $regex)
    {
        parent::__construct($name, $regex);
        $this->name = $name;
        $this->regex = $regex;
    }
}

and I can add that to the config with addTokenDefinition() function. Still, the problem remains since order of tokens is important. Also, your php doc here:

/**
     * @param TokenDefn[] $tokenDefinitions
     */
    public function __construct(array $tokenDefinitions)
    {
        foreach ($tokenDefinitions as $k => $v) {
            if ($v instanceof TokenDefn) {
                $this->addTokenDefinition($v);
            } elseif (is_string($k) && is_string($v)) {
                $this->addTokenDefinition(new TokenDefn($v, $k));
            }
        }
    } 

says that the constructor accepts only an array of TokenDefn object, while it also accepts normal assoc array (even in your example). That makes PHPstan throw an error. Correct phpdoc would be:

/**
     * @param TokenDefn[]|array $tokenDefinitions
     */