[{"data":1,"prerenderedAt":59},["ShallowReactive",2],{"project-pytorch-bottleneck":3},{"id":4,"title":5,"body":6,"description":12,"extension":26,"heroImages":27,"highlights":31,"meta":41,"navigation":42,"overview":43,"path":44,"role":45,"seo":46,"stack":47,"stem":54,"tagline":55,"type":56,"year":57,"__hash__":58},"projects\u002Fprojects\u002Fpytorch-bottleneck.md","PyTorch Data Opti.",{"type":7,"value":8,"toc":22},"minimark",[9,13,16,19],[10,11,12],"p",{},"When I trained small CNNs like a ResNet18, I noticed the GPU was under 40% most of the time. The model was not the problem, the GPU was just waiting for data. The way I see it, the CPU is the kitchen assistant who prepares the ingredients, and the GPU is the expensive chef. If the assistant is too slow, the chef stands there doing nothing.",[10,14,15],{},"My guess was that the normal synchronous loading creates small stops, because of Python and the GIL. So I made a benchmark to test it, and I looked at the time of each batch, not only the average. The average hides these stops, but you see them in the jitter.",[10,17,18],{},"Then I tried the optimizations one by one, to know what each one really changes: async loading, prefetching and caching in memory. I think a stable speed matters as much as a high speed when you train for a long time.",[10,20,21],{},"At the end, the GPU went from 38% to more than 90% usage. For me this project was about understanding what is really happening between the framework and the hardware.",{"title":23,"searchDepth":24,"depth":24,"links":25},"",2,[],"md",[28,29,30],"\u002Fresources\u002Fimg\u002Fbottlenecks\u002Fthroughput-bench.png","\u002Fresources\u002Fimg\u002Fbottlenecks\u002Fgpu-utilization.png","\u002Fresources\u002Fimg\u002Fbottlenecks\u002Flatency-jitter.png",[32,35,38],{"title":33,"description":34},"GPU usage","From under 40% to around 94% by loading and prefetching the data in the background.",{"title":36,"description":37},"Speed","3 times more images per second, by finding what was slowing the CPU and the bus and going around it.",{"title":39,"description":40},"Jitter","I made a small tool to see how much the loading time changes from one batch to another, to keep the training steady.",{},true,"Why my GPU was waiting for data when I trained small CNNs, and how I went from 38% to over 90% GPU usage.","\u002Fprojects\u002Fpytorch-bottleneck","Developer",{"title":5,"description":12},[48,49,50,51,52,53],"Python","PyTorch","CUDA","Performance Profiling","Matplotlib","NumPy","projects\u002Fpytorch-bottleneck","Hardware Efficiency Research - Eliminating GPU starvation in Deep Learning pipelines.","Personal",2026,"rqEmcYcU8KAGz3AT1FfzQ2rGo-hTSJAsS_yrWwH-jnc",1791377584489]