huggingface-tokenizers

Fast tokenizers optimized for research and production. Rust-based implementation tokenizes 1GB in <20 seconds. Supports BPE, WordPiece, and Unigram algorithms. Train custom vocabularies, track alignme

by ihatesea69· Repository·other

Run in AgentArea Browse All Skills

Also installable via skills CLI

npx skills add ihatesea69/HieuNghi-AI-Skills/airesearch_skills/02-tokenization/huggingface-tokenizers

Source

Repo:github.com/ihatesea69/HieuNghi-AI-Skills

Path:airesearch_skills/02-tokenization/huggingface-tokenizers/SKILL.md(main)

Related in other

agent-memory-yamadashy-repomix

Use this skill when the user asks to save, remember, recall, or organize memories. Triggers on: 'remember this', 'save t...

by yamadashy

21,427

task-execution-engine

CLI tool for configuring and monitoring Claude Code

by davila7

18,218

qiuzhi

指导Claude按照二哥的风格撰写求职类文章，包括公司薪资爆料、年终奖盘点、求职攻略、offer选择建议等内容。

by itwanger

16,619