llama.cpp/grammars/japanese.gbnf

# A probably incorrect grammar for Japanese
root        ::= jp-char+ ([ \t\n] jp-char+)*
jp-char     ::= hiragana | katakana | punctuation | cjk
hiragana    ::= [ぁ-ゟ]
katakana    ::= [ァ-ヿ]
punctuation ::= [、-〾]
cjk         ::= [一-鿿]
llama : add grammar-based sampling (#1773) * llama, main : constrain sampling to grammar * allow loading grammar from file * fix whitespace errors * handle & print parser errors * add comments to grammar syntax and allow newlines where unambiguous * add missing include * support alternates in root rule * fix bugs with empty token and EOS * adjust JSON grammar * remove swp file * rewrite ternary expressions Co-authored-by: Henri Vasserman <henv@hot.ee> * use struct for grammar elements and add Unicode support * add unicode escapes * add inverse char ranges * only sample full tokens (no peeking or truncation) * llama : minor style changes blindly applied in online editor - hopefully I didn't break something * update help text * add warning message if EOS is disabled --------- Co-authored-by: Henri Vasserman <henv@hot.ee> Co-authored-by: Georgi Gerganov <ggerganov@gmail.com> 2023-07-24 03:58:10 +00:00			`# A probably incorrect grammar for Japanese`
			`root ::= jp-char+ ([ \t\n] jp-char+)*`
			`jp-char ::= hiragana \| katakana \| punctuation \| cjk`
			`hiragana ::= [ぁ-ゟ]`
			`katakana ::= [ァ-ヿ]`
			`punctuation ::= [、-〾]`
			`cjk ::= [一-鿿]`