this post was submitted on 02 Aug 2026
43 points (97.8% liked)
Programming
28027 readers
496 users here now
Welcome to the main community in programming.dev! Feel free to post anything relating to programming here!
Cross posting is strongly encouraged in the instance. If you feel your post or another person's post makes sense in another community cross post into it.
Hope you enjoy the instance!
Rules
Rules
- Follow the programming.dev instance rules
- Keep content related to programming in some way
- If you're posting long videos try to add in some form of tldr for those who don't want to watch videos
Wormhole
Follow the wormhole through a path of communities !webdev@programming.dev
founded 3 years ago
MODERATORS
you are viewing a single comment's thread
view the rest of the comments
view the rest of the comments
So, it's basically saying that anything less than 1/256 (0.00390625f) is 0x0.
Anything equal to or over 0.99609375 (255/256) should be 0xff.
Divide by the count (256), not the max index (255). I'm not sure the 0.5 offset is really needed.
The offset is needed because this quantization dividing by 256 uses truncation whereas by 255 uses rounding.
So a value of 245 (for example) could be anything between 245.00000 and 245.99999. Therefore, when dequantizing, we set the value to 245.5, which is in the middle. Between 245.00000 and 245.99999.