ctpg icon indicating copy to clipboard operation
ctpg copied to clipboard

Is it valid to use custom_term alongside char_term, string_term, and regex_term within one terms() ?

Open 95833 opened this issue 3 months ago • 1 comments

It doesn't seem to be supported at the moment, but perhaps it could work if the code is modified from

if constexpr (generate_lexer)
{
    res = regex::dfa_match(lexer_sm, opts, ps.current_sp, ps.current_it, ps.buffer_end, ps.error_stream);
}
else
{
    lexer_type custom_lexer;
    res = custom_lexer.match(opts, ps.current_sp, ps.current_it, ps.buffer_end, ps.error_stream);
}

to

if constexpr (generate_lexer)
{
    res = regex::dfa_match(lexer_sm, opts, ps.current_sp, ps.current_it, ps.buffer_end, ps.error_stream);
}
else {
    lexer_type custom_lexer;
    res = custom_lexer.match(opts, ps.current_sp, ps.current_it, ps.buffer_end, ps.error_stream);

    if (res.term_idx == uninitialized16)
        res = regex::dfa_match(lexer_sm, opts, ps.current_sp, ps.current_it, ps.buffer_end, ps.error_stream);
}

? However, I'm not familiar with how terms() works internally, could you update the project to support this functionality?

95833 avatar Oct 27 '25 17:10 95833

With Copilot's help, I changed create_lexer to

template<size_t... I>
constexpr void create_lexer(std::index_sequence<I...>)
{
    regex::dfa_builder<lexer_dfa_size> b(lexer_sm);

    auto try_add_impl = [&b, this]<size_t Index>() constexpr {
        using Term = std::tuple_element_t<Index, decltype(term_tuple)>;
        if constexpr (has_get_data<Term>::value) {
            regex::add_term_data_to_dfa(std::get<Index>(term_tuple).get_data(), b, size16_t(Index));
        }
    };
    (try_add_impl.template operator()<I>(), ...);
}

and it appears to work successfully.

95833 avatar Oct 28 '25 08:10 95833