Function count_tokens 

pub fn count_tokens(text: &str) -> u64

Tokens of text in TOKENIZER_ENCODING, special-token text counted as ordinary text.

A file that happens to contain <|endoftext|> is counted as the characters it holds, not as the single control token a model would see: what is counted is what a reader is handed, and a count must never depend on whether the text looks like a protocol marker. The count is of the text alone, with no framing a provider adds around a message.

use majordomus_cli::economics::context::count_tokens;
assert_eq!(count_tokens("hello world"), 2);
assert!(count_tokens("<|endoftext|>") > 1, "special-token text is ordinary text");
let prose = "Majordomus counts what it selects.";
assert!(count_tokens(prose) < prose.len() as u64, "a token spans several bytes");