Skip to main content

TOKENIZER_IMPLEMENTATION

Constant TOKENIZER_IMPLEMENTATION 

Source
pub const TOKENIZER_IMPLEMENTATION: &str = "tiktoken-rs 0.12.0";
Expand description

The implementation and its version, as Cargo.lock resolved the pin in Cargo.toml: the build script reads it from the lockfile, so the number is stated in one place and a record can never name a version other than the one compiled in.

use majordomus_cli::economics::context::TOKENIZER_IMPLEMENTATION;
let (name, version) = TOKENIZER_IMPLEMENTATION.split_once(' ').unwrap();
assert_eq!(name, "tiktoken-rs");
assert!(version.split('.').count() == 3, "a resolved version, never `unknown`: {version}");