This C++ library is open source, part of the xbid.ai stack. I needed a low-overhead, fast Byte Pair Encoding (BPE) counter accurate enough for billing estimates and strategy comparisons. By skipping OpenAI template overhead we can trade exact parity for speed, with only ~1.5% deviation. The tool also provides support for Google’s sentencepiece binary models with a thin wrapper (100% parity).

• C++ BPE counter compatible with .tiktoken (OpenAI) encodings.
• Quasi-parity (no templates, <1.5% error)
• 60% faster than OpenAI’s official tiktoken (JS/WASM)
• No dependencies (standard C++20 toolchain)

Our initial code was using a naive byte-length heuristic, very fast but too inaccurate. For xbid-ai, I wanted something more reliable due to the nature of our prompts—trading signals are unbounded, and strategy outputs are compared for costs before routing across multi-LLM/model layer.