buy me a coffee

I work on AI infra and inference, and love reading, writing, music and art.

Anshuman Agrawal

2026

merged into TheHPXProject/hpx

  1. #7607Get the Rostam Jenkins lanes passing again
  2. #7574Don't define HPX config macros while finding an installed HPX
  3. #7575Stop hiding errors in the PDF documentation build
the rest
  1. #7573Restore the default handler before aborting
  2. #7571Give each worker its own random generator for work stealing
  3. #7572Keep the real PAPI error and skip the test when the counter is missing
  4. #7590Stop cancelling running Jenkins builds
  5. #7567Keep LSU CI results when builds are interrupted
  6. #7527Keep the caching allocator's thread_local lookup out of line
  7. #7532Fix LSU GitHub status reporting
  8. #7533Stop failed LSU builds from publishing installs
  9. #7534Keep LSU matrix artifacts separate
  10. #7526Avoid communicator reuse between collectives test phases
  11. #7531Bound Slurm waits in LSU CI
  12. #7516Bound the Dijkstra termination probe retry when the token cannot be delivered
  13. #7521Whiten the Dijkstra locality before the token is transmitted
  14. #7506Fix the two shutdown regressions from #7471 that hang the distributed CIs
  15. #7507Make the nvcc CUDA configurations compile and pass again

Baton

Checks the handoffs inside inference systems (a request moved after a crash, KV cache from a shared pool, weights copied from an RL trainer) by redoing a sample the slow, trusted way.

on GitHub since 24 Sep

merged into TheHPXProject/hpx

  1. #7500Stop the colocated tests finalizing from inside the hpx_main loop
  2. #7495Restart the Dijkstra termination probe cursor for every probe
  3. #7491Make force_disconnect claim a locality before removing it
the rest
  1. #7492Check that parcels to a killed locality report an error
  2. #7493Set SLURM_MPI_TYPE=pmix for the rostam CIs
  3. #7481Cache channel communicators separately for each site
  4. #7449Fix pairwise all_to_all build with Clang 17 and OpenMP
  5. #7412Give the launched locality the endpoint the probe actually bound
  6. #7431Add size-based pairwise dispatch to all_to_all
  7. #7445Add collectives API documentation metadata
  8. #7426Fix the build with HPX_WITH_THREAD_QUEUE_WAITTIME=ON
  9. #7419Advertise the parcelport port that was actually bound

merged into Project-HAMi/HAMi-core

  1. #293Pass the fan index and timestamps to the v2 hooks
  2. #279Fix repeated RTLD_NEXT lookups returning null
  3. #278Call rate_limiter from cuLaunchCooperativeKernel
the rest
  1. #248Recover the post-init lock after process death

ceteris

Record benchmark conditions. Check what changed before comparing numbers. On PyPI.

first released on PyPI 29 Aug

three hundred processes waited for one pid

HAMi issue 1662 blamed a lock file for a minute of startup latency. By the time I got there the lock had been replaced, and the expensive part was never the lock: it was a CUDA context created so a process could learn which PID the host calls it. One process carried from Kubernetes allocation to cuInit, then 300 released at once.

the tensor was only the handle

A source-pinned reconstruction of PyTorch from one small training step: tensor identity, storage and views, ATen dispatch, modules, autograd, optimizers, modes, devices, transforms, compilation, export, data workers, distributed reducers, serialization, profiling, and custom operators.

One collective call used two generations

A two-locality demo hangs before the hierarchy is even visible. Fixing it leads through HPX communicator servers, uneven trees, all-to-all routing, shared generation clocks, scans, flattened payloads, convergent failures, MPI shutdown, and one still-open endpoint repair.

merged into TheHPXProject/hpx

  1. #7398Move the all_reduce reduction seed instead of copying it
  2. #7405Let the OS pick the parcelport port in the departed locality test
  3. #7404Make MPI parcelport teardown orderly: quiesce before MPI_Finalize
the rest
  1. #7401Cache the first collective finalizer failure and rethrow it for every site
  2. #7377Flatten hierarchical all-to-all exchange payloads
  3. #7378Move communicator basenames into owned storage
  4. #7375Flatten hierarchical gather and scatter payloads
  5. #7359Harden collectives validation paths
  6. #7369Reject explicit generations after auto generation on the same communicator
  7. #7364Avoid communicator reuse between scan test phases
  8. #7343Add hierarchical scan collectives

arrays become matrices

Six integers become a 2x3 matrix. Turning it sideways without moving them produces either the right answer or a very plausible wrong one, depending on two strides.

my canary got bigger

Running CommCanary end to end on a laptop: a compression tool that doubled the size of its input, an error budget the compiler refused, a search whose own receipts outweighed what it saved, and the one number in the whole pipeline that turned out to mean what it said.

one hundred events became one

A delta-debugging reducer deleted ninety nine of a trace's one hundred events and every configuration ranking survived. Why preserving the decision is the wrong gate on its own, and how CommCanary compresses a distributed LLM communication trace while proving the compression kept the answer.

merged into titoatwork/COLIDE

  1. #5Pin one GPU when nodes expose multiple devices
  2. #4Final polish 2
  3. #3Support per-profile SLURM partitions for multi-queue clusters
the rest
  1. #2Final polish
  2. #1Harden DICC multi-day campaign workflow

the address where the value lived

A pointer remembers the first element of a vector. One push_back later, the value survives and the pointer does not. Following the missing address through arrays, layout, lifetime, vector growth, arenas, and pools.

the head pointer came back

A delayed dequeue sees the same head address twice and succeeds with a stale compare-and-swap. Following that returned address through FIFO invariants, helping, linearizability, hazard pointers, reclamation, and the actual limits of lock-free progress.

the syscall the kernel refused

The same open call returns a file descriptor, and then returns Operation not permitted. Following the result through the arm64 trap, XNU's file authorization path, a process policy, Linux seccomp, and executable memory.

idle gaps are not compute

A collective benchmark put one NCCL configuration safely in the middle. The workload put it last. Following that disagreement through trace capture, replay scheduling, CUDA streams, ranking statistics, missing evidence, queueing, and one source line that changed the explanation.

Rings, Trees, and Butterflies: HPX Collectives

Eight ranks need the same sum. The HPX call leads into one communicator server, then into five executed all-reduce algorithms, a byte-and-message cost model, fresh MPI measurements, and a provenance-stamped cluster result that changes what the source alone seemed to imply.

the page that was not there

A mapping succeeds with no resident pages, then one ordinary load per page enters the kernel 256 times. Following those loads through virtual addresses, arm64 translation, XNU faults, zero fill, protection, copy-on-write, file sharing, wiring, and the measurements that cannot reveal a TLB.

how your computer fakes randomness

rand() returns the same numbers after every process start. The first value leads into state recovery, entropy, biased mappings, finite tails, forked streams, and benchmark results whose meaning changes when order and uncertainty are handled.

CommCanary

Distill a distributed-LLM communication trace into a small, model-free regression canary, and prove the distillation didn't change the answer.

on GitHub since 30 Jun

merged into TheHPXProject/hpx

  1. #7340Harden collectives validation paths
  2. #7326Hierarchical collectives: unify the per-call generation step for cross-collective sharing
  3. #7321Harden hierarchical collectives: flat fallback unification, input validation, tree test coverage
the rest
  1. #7307Add hierarchical all_to_all collective

one line to machine code

One line enters torch.compile. What comes back is a guarded region containing a library matrix multiply and a generated C++ loop, plus a crash that four threads can reproduce and one thread cannot.

teaching a model

A rank-8 patch fit the training target while the frozen model forgot unrelated text. The loss said it learned. The held-out prompts forced a more careful answer.

one gradient, backward

A thirty-line tape and PyTorch agree on one gradient. A finite difference does not. Following the discrepancy reaches the chain rule, reverse and forward accumulation, matrix backward, GPT-2, checkpointed activations, optimizer state, and the numbers fp16 cannot keep.

one forward pass

GPT-2 put Paris fifth. Following the exact rows behind that result reaches byte-pair tokens, matrix products, attention, residual state, a growing KV cache, batch reuse, and the difference between simulating 4-bit error and running a packed kernel.

how machines agree

One counter, two machines, one dropped packet, and the state that must survive retries, partitions, and a killed process before an acknowledged increment can be trusted.

where the time goes

One program that sums every byte of a 1 GiB file, and the same arithmetic made 4000x slower and then fast again, a walk through syscalls, the user/kernel boundary, 16 KiB pages, page faults, the page cache, mmap, P and E cores, sampling profilers, and using the roofline as a bottleneck detector.

a file is just bytes

A column read that looked thirty-eight times faster led into Parquet pages, byte ranges, encodings, pruning, Arrow buffers, and the commit race hidden behind a table.

the fast path out of python

Summing ten million numbers takes 290 ms in a Python loop and 3.4 ms in numpy. Following the gap reaches PyObject boxing, pairwise reduction, native buffers, ownership, the GIL, and the point where zero-copy stops being a sufficient claim.

merged into TheHPXProject/hpx

  1. #7193Add flat-collective fallback to hierarchical_communicator for small site counts
  2. #7198Add non-power-of-arity tests for hierarchical all_reduce and all_gather
  3. #7189Add unit tests for hierarchical all_reduce and all_gather
the rest
  1. #7160Add hierarchical all_reduce and all_gather via reduce+broadcast composition
  2. #7141Use is_default() consistently in hierarchical gather_here
  3. #7136Fix copy-paste errors in collectives Doxygen comments

two threads to a task runtime

Two threads each add one to a counter a million times, and it comes out to 1,012,274, a different wrong number every run. Following that lost million down through the load-add-store race, atomics, the weakly-ordered memory model, false sharing, mutexes and deadlock, and up to why real systems hand you tasks and futures instead of raw threads.

from a C loop to a GPU kernel

A walk from a slow 512x512 C matrix multiply through cache, SIMD, FMA, tiling, roofline thinking, Metal kernels, threadgroup memory, GEMM, reductions, softmax, attention, and fusion.

sixteen million and one

A full walk through computer numerics from integer representation and IEEE-754 float bits to rounding, subnormals, cancellation, compensated summation, SIMD/FMA/BLAS, mixed precision, and quantization.

from source to a running process

A C++ toolchain walk from preprocessing and templates to object files, ODR traps, static and dynamic linking, ABI symbols, relocations, GOT/stubs, lazy binding, dyld, and the first instruction of a process.

Python is a C program

Five lines of Python followed all the way down: a 52 KB Mach-O launcher, tokens, AST, bytecode, _PyEval_EvalFrameDefault, PyObject, refcounts and immortals, cyclic GC, the GIL, free-threading, specialization, the experimental JIT, and the C API.

std::move doesn't move anything

Value categories, RAII, move semantics, copy elision, smart pointers, templates, and concepts, traced from one line of libc++ source. With real benchmarks on an M4 and real HPX code throughout.

Triton 1.58

Post-training quantization of transformer weights to {-1, 0, +1}, with custom Triton kernels for the quantized matmul.

on GitHub since 31 Jan

2025

NeuroRank

A BERT reranker distilled into a smaller student, then ONNX and 8-bit quantization for low-latency search.

on GitHub since 1 Sep 2025

things I love

all of them, by kind

literary fiction
Norwegian Wood Haruki Murakami
The Kite Runner Khaled Hosseini
The Secret History Donna Tartt
The Namesake Jhumpa Lahiri
classics
Pride and Prejudice Jane Austen
White Nights Fyodor Dostoevsky
Notes from Underground Fyodor Dostoevsky
Crime and Punishment Fyodor Dostoevsky
The Stranger Albert Camus
computing
Programming Massively Parallel Processors Wen-mei W. Hwu, David B. Kirk, Izzat El Hajj
memoir
When Breath Becomes Air Paul Kalanithi
Surely You're Joking, Mr. Feynman! Richard Feynman
coming of age
The Perks of Being a Wallflower Stephen Chbosky
historical fiction
The Book Thief Markus Zusak
science fiction
Dark Matter Blake Crouch
Foundation Isaac Asimov
Dune Frank Herbert
The Hitchhiker's Guide to the Galaxy Douglas Adams
Project Hail Mary Andy Weir
The Martian Andy Weir
dystopian fiction
Brave New World Aldous Huxley

by kind, continued

essays
Meditations on Moloch Scott Alexander
The War of Art Steven Pressfield
philosophy
Gödel, Escher, Bach Douglas Hofstadter
Zen and the Art of Motorcycle Maintenance Robert M. Pirsig
science
The Pleasure of Finding Things Out Richard Feynman
Cosmos Carl Sagan
Algorithms to Live By Brian Christian, Tom Griffiths
poetry
The Pleasures of the Damned Charles Bukowski

songs

mostly folk, country, and ghazals.

  • Northern SkyNick DrakeEnglish
  • Chemtrails over the Country ClubLana Del ReyEnglish
  • Paris in the RainLauvEnglish
  • RaastonShaurHindi
  • Baawra MannShunyaHindi
  • Inner LightElderbrookEnglish
  • DreamersSavoir AdoreEnglish
  • Dil Mein Ek Lehar Si Uthi Hai AbhiGhulam AliUrdu
  • Ab Kya Ghazal SunaoonTalat AzizUrdu
  • Tere NaamKaavishUrdu
  • Wild Mountain ThymeThe CorriesEnglish
  • Made in JapanBuck OwensEnglish
  • Dasht-e-TanhaiIqbal BanoUrdu
  • Diamond DayVashti BunyanEnglish
  • I Lost Something in the HillsSibylle BaierEnglish
  • If I Needed YouTownes Van ZandtEnglish

writing

  1. 28three hundred processes waited for one pid31 Jul
  2. 27the tensor was only the handle29 Jul
  3. 26One collective call used two generations28 Jul
  4. 25arrays become matrices27 Jul
  5. 24my canary got bigger26 Jul
  6. 23one hundred events became one26 Jul
  7. 22the address where the value lived8 Jul
  8. 21the head pointer came back7 Jul
  9. 20the syscall the kernel refused7 Jul
  10. 19idle gaps are not compute6 Jul
  11. 18Rings, Trees, and Butterflies: HPX Collectives6 Jul
  12. 17the page that was not there6 Jul
  13. 16how your computer fakes randomness5 Jul
  14. 15one line to machine code23 Jun
  1. 14teaching a model15 Jun
  2. 13one gradient, backward3 Jun
  3. 12one forward pass27 May
  4. 11how machines agree22 May
  5. 10two machines talking to each other4 May
  6. 9where the time goes30 Apr
  7. 8a file is just bytes24 Apr
  8. 7the fast path out of python20 Apr
  9. 6two threads to a task runtime15 Apr
  10. 5from a C loop to a GPU kernel4 Apr
  11. 4sixteen million and one26 Mar
  12. 3from source to a running process21 Mar
  13. 2Python is a C program15 Mar
  14. 1std::move doesn't move anything11 Mar

work, infra + inference

I'm Anshuman Agrawal, a systems and AI infrastructure engineer. I spend my time deep in C++ parallel runtimes, CUDA, and the layers between user code and silicon.

HPX
a C++ standard library for parallelism and concurrency. I contribute to its collective communication primitives as part of GSoC.
CUDA / GPU programming
writing kernels, understanding memory hierarchies, fighting occupancy.
Distributed systems
how compute and data move across a cluster, and why that's hard.

I contribute to HPX's collective communication primitives as part of GSoC. 51 of my pull requests are merged there.

I also fix things in HAMi, which shares GPUs between pods on Kubernetes.

projects

  • CommCanary GitHub, since 30 Jun 2026

    Distill a distributed-LLM communication trace into a small, model-free regression canary, and prove the distillation didn't change the answer.

  • ceteris PyPI, first release 29 Aug 2026

    Record benchmark conditions. Check what changed before comparing numbers. On PyPI.

  • Baton GitHub, since 24 Sep 2026

    Checks the handoffs inside inference systems (a request moved after a crash, KV cache from a shared pool, weights copied from an RL trainer) by redoing a sample the slow, trusted way.

  • Triton 1.58 GitHub, since 31 Jan 2026

    Post-training quantization of transformer weights to {-1, 0, +1}, with custom Triton kernels for the quantized matmul.

  • NeuroRank GitHub, since 1 Sep 2025

    A BERT reranker distilled into a smaller student, then ONNX and 8-bit quantization for low-latency search.

  1. Dead Poets SocietyPeter Weir
  2. Good Will HuntingGus Van Sant
  3. The Secret Life of Walter MittyBen Stiller
  4. Eternal Sunshine of the Spotless MindMichel Gondry
  5. SoulPete Docter
  1. DarkBaran bo Odar, Jantje Friese
  2. Game of ThronesDavid Benioff, D. B. Weiss

internet home

AnshumanAgrawal

contents

  1. 4things I love
  2. 6songs
  3. 8books
  4. 10writing
  5. 12films and series
  6. 14work (infra + inference)
  7. 16find me

I work on AI infra and inference, and love reading, writing, music and art.