light @reprompting
learning from first principles Joined May 2025-
Tweets183
-
Followers167
-
Following75
-
Likes119
my favorite thing about backpropagation is just how simple it is. it is literally just applying the chain rule for derivatives.
simple backpropagation by hand on 2 layers had fun revising it
Overall, the chapter shows that dynamic parallelism is mainly useful for workloads where the amount of work changes as the program runs. For more regular workloads, the normal host-launched execution model is usually the better option.
The chapter uses adaptive mesh refinement, Bezier curves and quadtrees as examples and then discusses things like synchronization between parent and child kernels, streams, memory visibility, launch overhead and when dynamic parallelism actually makes sense.
Summarizing chapter 21 of PMPP This chapter introduces CUDA Dynamic Parallelism where GPU kernels can launch other kernels directly instead of relying on the CPU to launc every piece of work.
I feel like the idea discussed here isnt entirely new, since overlapping data movement with computation is already commonly used in single-GPU training to keep the GPU busy.
The chapter then introduces the CUDA features that make this possible: 1. CUDA streams for concurrent execution 2. Asynchronous memory copies 3. Pinned memory to enable DMA transfers 4. CUDAaware MPI The common idea is to to reduce idle time by overlapping work wherever possible
Summarizing chapter 20 of PMPP This chapter introduces CUDA streams using a distributed 3D stencil computation as the running example. Unlike the previous chapters that focus on optimizing kernels on a single GPU, this one looks at scaling across multiple nodes using MPI
from choosing the right algorithm to decomposing the problem and optimizing the implementation. The chapter also revists many examples from earlier chapters to show that performance isnt just about writing faster kernels but also about making better algorithmic and design choices
Summarizing chapter 19 of PMPP. This chapter takes a step back from CUDA programming and focuses on computational thinking for parallel programming. Instead of introducing new optimization appraoches, it discusses how to approach parallel probelms -
From there, the implementation is gradually optimized using techniques like constant memory, memory coalescing, thread coarsening, and finally cutoff-based methods (binning) to reduce the amount of unnecessary computation.
The gather approach instead assigns each thread to a grid point and computes contributions from all charges. This increases the amount of work per thread, but avoids atomics entirely, which makes it more suitable for parallel execution.
Summarizing chapter 18 of PMPP. This chapter focuses on electrostatic potential map computation as an example of optimizing a real GPU application, and it brings together many of the optimization techniques introduced earlier.
CUDA SpMV using CSR format. Storing only non-zeros feels simple but irregular memory access is where things got messy.
I liked how the chapter showed optimization as an iterative process. They sped up the expensive MRI reconstruction kernel, only to find that a completely different part of the pipeline had become the bottleneck.
The chapter starts with a straightforward implementation and progressively improves it through better parallelization strategies, memory optimizations, and GPU-specific hardware features.
Summarizing chapter 17 of PMPP. This chapter is a case study on optimizing an iterative MRI reconstruction algorithm for GPUs. Unlike the previous chapters that introduced individual parallel patterns, this one brings many of those ideas together in a real application.
The chapter then ends by showing how convolutions can be reformulated as matrix multiplications (GEMM), allowing them to leverage highly optimized GPU libraries. This idea forms the foundation of libraries such as cuDNN.
The chapter expands on explaining why convolutions are such a good fit for GPUs (they expose massive amounts of parallelism across output pixels, feature maps, minibatches). It then shows how convolution kernels can be mapped onto CUDA and how performance can be improved.
Summarizing chapter 16 of PMPP. This chapter focuses on deep learning, specifically CNNs, as a case study for massively parallel computing. Since I already have a background in machine learning, the neural network concepts were mostly familiar.
Giason Pooni @giasonpooni
0 Followers 258 Following
Ravi Vijayakumar @thehiphopswami
4K Followers 4K Following Chip Implementation @ OpenAI. Prev: TPU Impl lead of Ironwood + TPU 8i @ Google. Pushing limits of physics gives me the thrills.
Magesh Devaprakash @magesh_d
51 Followers 2K Following
Kalyan @kalyann567
17 Followers 2K Following
Nikola @ThieuPhuc31706
6 Followers 108 Following
Anirudh Kamaraj @kakarot2805
8 Followers 214 Following Research Intern @ Together AI (Inference) MSCS (ML) @ Gatech | ML systems | Ex - SMTS@Oracle, Servicenow ML | NITW 22
Ankit pachouri @Ankit_pachouri
22 Followers 670 Following
Moshood Olawale @Quantymosh
169 Followers 2K Following
Sarthak Singh @Sarthak07616266
35 Followers 817 Following
Julie Clement @JulieClement
81 Followers 6K Following
Boosie B @b447017
1 Followers 18 Following
Noaman @Nomz223
13 Followers 1K Following
Devanshu Sinha @sinhadevanshu20
36 Followers 978 Following techie intern @ Pune | working with bits O_o | organized @hackhound2k23
Frank Chiarulli Jr. @_fcjr
125 Followers 974 Following Privacy Nerd building https://t.co/md5H492Htw & @moonfootlabs. Prev leadership @jpmorgan, @SvixHQ, & @Ghostery. Forever @recursecenter. You can just build things.
🌙 @pimavista
32 Followers 288 Following
fulanliu @liujiangtao8
35 Followers 3K Following
ママン @neko_jin7692
218 Followers 5K Following Hardware engineer, software developer, translator, imam, cook, driver, STEM advocate, bookworm, you name it 電気エンジニア、ソフトプログラマー、面白いな人、日本語おしゃべりなマレーシア人、いろいろ。彼女が欲しい
promptmaxing @promptmaxing
18 Followers 336 Following uncertainty is the first step toward self-determination, courage comes next. also, figuring out how this llm shit works.
Andrea Alberici @aalberici
4K Followers 7K Following A geek from the 80's, a consultant in the 90's. Consulting in Business Processes (CRM/ERP, BPO) and Data Science (Business Intelligence, Data Smithing)
mon dsfds @DsfdsMon18759
159 Followers 6K Following
S Sanjay @Sanjay_lm10
22 Followers 3K Following
advithkrishnan.eth @advith_krishnan
142 Followers 475 Following :D | Mashed a keyboard for @LinuxFoundation @OpenMFProject @chalmersuniv @eth @NASA | My views are personal | Building existential crisis | crisises? | crises?
TrackBTC @TrackBTCbot
0 Followers 66 Following
shahmeer @shahmeerwarma
1 Followers 63 Following u1 student @mcgillu math & cs major interested in world models & physical ai
Vikas Reddy @vikasreddyc
125 Followers 7K Following
Calvin | Impossible i... @calchulus
27K Followers 18K Following Year 6 @impossible_ Prev @binanceresearch '18-'21 finished all econ credits @uchicago in two years AMA
it’s real @Iwasthere01
2K Followers 7K Following ⛲ Maybe the true universe is the friends we made along the way..
Britannio Jarrett @Britannioj
764 Followers 4K Following Hacking on cool toys. Formerly LLM inference research @ Imperial. Software abundance is near.
Clément Lefebvre @ClementLef92400
7 Followers 116 Following
Senseii @Senseii8792101
83 Followers 3K Following
eristic @vol_ition
296 Followers 7K Following A pseudonym's pseudonym; 'you can be arrogant or ignorant, but never both'
Ben @Benmam97
36 Followers 2K Following
LostralianTrades @LostralianTrade
892 Followers 7K Following CitadelLounge | CEO @GanntecApp | Angel investor | 7Y futures trading
promptmaxing @promptmaxing
18 Followers 336 Following uncertainty is the first step toward self-determination, courage comes next. also, figuring out how this llm shit works.
Ravi Vijayakumar @thehiphopswami
4K Followers 4K Following Chip Implementation @ OpenAI. Prev: TPU Impl lead of Ironwood + TPU 8i @ Google. Pushing limits of physics gives me the thrills.
Pollux9437 @Orange41324306
1K Followers 2K Following Agent @yipitdata, RL Infra and pretrain research at night, sometimes photography and poetry. writer account @orange97648
Lelouch 👾 @lelouchdaily
12K Followers 485 Following Documenting my SWE & ML grind | https://t.co/NMr3mpuRDd | Posting progress reports & resharing quality memes.
Bryce, the CUDA Colon... @blelbach
19K Followers 3K Following Principal Engineer at @NVIDIA working on programming languages. @adspthepodcast co-host. C++ Library Evolution chair emeritus. Frequent flyer. Horology fan.
tsuki @tensorcore
4K Followers 464 Following Hiroyuki Ootomo. High-precision GEMM emulation on Tensor Cores. Work at #76B900. Cooking @cp_async. Hai-to-Yoka: https://t.co/jAdudlZfnb
Zhihu Frontier @ZhihuFrontier
10K Followers 165 Following 🚀Bringing China's AI & tech trends, voices and perspectives to the global stage. ⚡️Powered by 知乎/https://t.co/OkIemRZdcj, China's leading knowledge community.
Britannio Jarrett @Britannioj
764 Followers 4K Following Hacking on cool toys. Formerly LLM inference research @ Imperial. Software abundance is near.
gravito @Gravito841
5K Followers 2K Following In an abusive relationship with math. type 3 fun haver
Jeremy Dohmann @jecdohmann
268 Followers 848 Following Reasoning at @perceptroninc. Former @dbrxmosaicai, @realitylabs I write music https://t.co/QJl07cJ9zM
R @idksmthngbut
1 Followers 218 Following
Aayush @driftaware9
65 Followers 533 Following MLE @Expedia Ex - Razorpay, IIT Kharagpur '22 Interested in everything engineering
anaum @anaumghori
1K Followers 493 Following
Jared Roesch @roeschinc
2K Followers 825 Following Distinguished Engineer @nvidia; working on Tile IR Prev. Co-founder @octoml. PhD @uwcse. Attempting to write about AI @ https://t.co/toFSukgrzM
Ayush Goyal @goyalayus
199 Followers 432 Following @iitroorkee | 20 | reinforcement learning | ex - https://t.co/0QVBIHhQeD
Meronix @Meronix15
697 Followers 2K Following Retired CEO of my local crack house. Retarded and zero hoes
Ivan Yashchuk @IvanYashchuk
654 Followers 807 Following I work on @PyTorch at @NVIDIA Born in Ukraine | Raised in Siberia Yugra | Living in 🇫🇮 and 🇨🇭
𝑘𝑒𝑟𝑛𝑒�... @kernel_trick
146 Followers 303 Following still somewhat loves computers • some exp with llms • there are no claude-like models • tokens per petrodollar
𝓚𝓸𝓴𝓸 @DevKokooo
1K Followers 2K Following ✨ sustaining flow with gamified productivity @forgewaredev • cs, linguistics, learning math for ML • 🌾 interested in offworld industries, agtech, biotech
Ravi Chandra Veeramac... @ravichandra
244 Followers 2K Following Knowledge is cheap. Wisdom is expensive.
Miguel (in sf) @Kaweees1
1K Followers 2K Following Prev. Roboticist @dimensionalos, Humanoids & Chip design agents @nvidia GEAR.
Habibi Willi @WilliFetzner
388 Followers 1K Following professional player according to mifid Habibi | Doktor in Feminismus
Rob @Robcseals
85 Followers 2K Following Interested in several things but focusing on intelligent systems/machines | aspiring philomath | Prov 21:31, Phil 4:13 working on @acxlabs and other stuff
Rick Spanbauer @RickSpanbauer
55 Followers 567 Following
faraz @farazdotai
1K Followers 1K Following token min-maxer @nvidia, prev token juggler @cohere | studied token literacy @uwaterloo 🇨🇦
Federico Bruzzone @fedebruzzone7
135 Followers 164 Following CS PhD Candidate. Programming Languages and Compilers enthusiast. Diving into AI (ML/DL) compilers.
LaurieWired @lauriewired
156K Followers 293 Following researcher @google; serial complexity unpacker; https://t.co/Vl1seeNgYK ex @ msft & aerospace
MancerAI @MancerAI_
1K Followers 1K Following e/acc AI dev | Physics → ML | Apparently AI researcher nowadays. shitposts, cognition and consciousness | Building odd futures
s3nh @s3nhxx
466 Followers 858 Following In process of building something useful. Reading, coding. dm me
Ekaeo @Ekaeoq
11K Followers 421 Following Watchmaker with a strong CS background https://t.co/J5ylTJyA0O See my blog for projects: https://t.co/FtNPoC0WfA
Shengkun Tang @shengkun_t52337
188 Followers 85 Following Research Intern in @Alibaba_Qwen | PhD student in @mbzuai
levi @levidiamode
5K Followers 702 Following 365 days of GPU programming ▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓░░░░░░░░░░░░░░░░░░ 176/365
Charles 🎉 Frye @ A... @charles_irl
19K Followers 3K Following memer of technical staff at @modal. he/him. ex @full_stack_dl, @weights_biases (acq. @CoreWeave), phd Berkeley @Redwood_Neuro. try https://t.co/SYWVMCb7OB
Trends for United States
You might like
