mirror of
https://lezozoien.com/gitea/Le_Zozoien/LLM.git
synced 2026-11-16 00:08:51 +00:00
A custom tokenizer
- Python 100%
| data | ||
| output | ||
| tokenizer | ||
| utility | ||
| .gitattributes | ||
| .gitignore | ||
| main.py | ||
| README.md | ||
| requirements.txt | ||
To get all necessary files, run these from the root of the project:
pip install -r requirements.txtpython -m utility.parquetConverterpython tokenizer/tokenizer.pypython -m utility.fileSplitter