crates.io · crates:tokenizers
tokenizers active
Fastest implementation of the most used tokenizers in AI.
| Latest stable version | 0.23.2 |
|---|---|
| Published | 2026-09-03 08:49:28 UTC |
| Install | cargo add tokenizers |
| Minimum Rust version (rust-version) | none declared |
| License | Apache-2.0 |
| Advisories on latest version | none known |
| Last verified | 2026-10-09 15:20:24 UTC |
| Sources | api.osv.dev, crates.io |
Maintenance
| Activity | active |
|---|---|
| Last stable release | 2026-09-03 08:49:28 UTC · 36 days ago |
| Stable releases in the last 12 months | 3 |
| Published versions | 42 |
Heuristic based only on release dates; it does not measure issue or security response times.
Version history
| Version | Published |
|---|---|
0.23.2 | 2026-09-03 08:49:28 UTC |
0.23.1 | 2026-04-27 14:39:17 UTC |
0.22.2 | 2025-12-02 12:53:51 UTC |
0.22.1 | 2025-09-19 09:46:10 UTC |
0.22.0 | 2025-08-29 10:22:26 UTC |
0.21.4 | 2025-07-28 13:19:38 UTC |
0.21.2 | 2025-06-24 10:21:08 UTC |
0.21.1 | 2025-03-13 10:45:44 UTC |
0.21.0 | 2024-11-27 13:06:48 UTC |
0.20.4 | 2024-11-26 17:03:22 UTC |
0.20.3 | 2024-11-05 17:20:12 UTC |
0.20.2 | 2024-11-04 17:17:03 UTC |
0.20.1 | 2024-10-10 09:46:24 UTC |
0.20.0 | 2024-08-08 16:45:14 UTC |
0.19.1 | 2024-04-17 21:17:09 UTC |
0.19.0 | 2024-04-17 08:52:21 UTC |
0.15.2 | 2024-02-12 02:11:46 UTC |
0.15.1 | 2024-01-22 15:01:56 UTC |
0.15.0 | 2023-11-14 18:51:49 UTC |
0.14.1 | 2023-10-06 11:11:43 UTC |
Showing the 20 most recent of 38 versions. JSON
Give your AI agent these facts
Vigia is a free MCP server: your coding agent checks versions, breaking changes and vulnerabilities before it writes code. No signup, no API key.
claude mcp add --transport http vigia https://vigia.coredls.cloud/mcp
Add to CursorAdd to VS Code★ Star on GitHub
Any other MCP client
{ "mcpServers": { "vigia": { "url": "https://vigia.coredls.cloud/mcp" } } }Agent skill (SKILL.md): tells the agent when to use Vigia · GitHub Action for pull requests
Frequently asked questions
What is the latest version of tokenizers?
The latest stable version of tokenizers is 0.23.2, published on 2026-09-03. Verified by Vigia on 2026-10-09.
How do I install the latest version of tokenizers?
Run cargo add tokenizers. The latest stable version is 0.23.2 (verified 2026-10-09).
Is tokenizers deprecated?
No. tokenizers is not marked as deprecated in the registry.
Which Rust version does tokenizers require?
tokenizers 0.23.2 does not declare a Rust version requirement.
Does tokenizers 0.23.2 have known vulnerabilities?
No known advisories affect tokenizers 0.23.2 according to the data Vigia checks.
Is tokenizers still maintained?
tokenizers published 3 stable releases in the last 12 months; the most recent was 36 days ago. Activity: active.
Observed changes
No changes observed yet since Vigia started tracking this package.
Related packages
- monostate-impl
- inherent
- minijinja
- monostate
- crc16
- tabled_derive
- similar-asserts
- azure_core
- rustc-literal-escaper
- mac_address
- wry
- des
Badges for your README
Show the latest version, maintenance status and deprecation status of this package. Copy the Markdown:
[](https://vigia.coredls.cloud/crates/tokenizers) [](https://vigia.coredls.cloud/crates/tokenizers) [](https://vigia.coredls.cloud/crates/tokenizers)
For agents
GET https://vigia.coredls.cloud/v1/packages/crates/tokenizers
GET https://vigia.coredls.cloud/v1/packages/crates/tokenizers/versions/{version}