Hacker Newsnew | past | comments | ask | show | jobs | submitlogin

> I don't really like how the author asked for exact character lengths, after clearly specifying that LLMs can't really count characters. It seems like an experiment set up to fail.

Some authors write a lot about GPT stuff but they don't have the slightest clue about how they work, that's why they have such expectations. I don't know about this author's credentials, but I know several people who are now the AI celebrities of our age simply because they a lot about other people's research findings.



Yes, I know how tokenizers work and have spent an embarrassing amount of time working with/training tokenizer models with Hugging Face tokenizers.


He knows what a tokenizer is.




Guidelines | FAQ | Lists | API | Security | Legal | Apply to YC | Contact

Search: