pub const TOKENIZER_IMPLEMENTATION: &str = "tiktoken-rs 0.12.0";Expand description
The implementation and its version, as Cargo.lock resolved the pin in Cargo.toml:
the build script reads it from the lockfile, so the number is stated in one place and a
record can never name a version other than the one compiled in.
use majordomus_cli::economics::context::TOKENIZER_IMPLEMENTATION;
let (name, version) = TOKENIZER_IMPLEMENTATION.split_once(' ').unwrap();
assert_eq!(name, "tiktoken-rs");
assert!(version.split('.').count() == 3, "a resolved version, never `unknown`: {version}");