A custom tokenizer
Find a file
2026-11-15 23:51:02 +01:00
data Requirements, small changes 2026-11-06 15:34:48 +01:00
output Remove wiki.txt from output 2026-11-06 14:44:22 +01:00
tokenizer Major refactor, added saving and loading models 2026-11-15 22:51:09 +01:00
utility Requirements, small changes 2026-11-06 15:34:48 +01:00
.gitattributes Added lfs support for parquet files 2026-10-15 22:47:20 +02:00
.gitignore Progress on the model : Now able to train 2026-11-15 21:02:05 +01:00
main.py Began adding the generation pipeline ("je m'appelle 200") 2026-11-15 23:51:02 +01:00
README.md Set requirements in readme 2026-11-06 14:39:57 +01:00
requirements.txt Forgot the requirements 2026-11-09 17:32:42 +01:00

To get all necessary files, run these from the root of the project:

  1. pip install -r requirements.txt
  2. python -m utility.parquetConverter
  3. python tokenizer/tokenizer.py
  4. python -m utility.fileSplitter