70th IT Press Tour: Clockwork pulled the power on a live AI training job, and it carried on …

Posted on Author AIJobNexus 0
This post was originally published on this site.

Live demos on real hardware are where confident startups go to be humbled. So when a Clockwork engineer in a South Bay data centre pulled on a pair of gloves, switched off a GPU server in the middle of a training run and yanked out its power cables, the room went very quiet.

The job stopped at step 5,040. About a minute later it resumed at step 5,040, on a different physical machine, with no checkpoint reload and no recomputed work.

That minute is the entire Clockwork pitch. GPU clusters fail constantly, and the AI jobs running on them should barely notice. Here is the video of the Clockwork presentation, upon which this article is based. Please watch, and read on!

Who was in the room

Clockwork took the second session on day two of the 70th IT Press Tour, on 6 October, one day after announcing a US$31 million funding round. The IT Press Tour Founder, Philippe Nicolas, introduced a face plenty of us already knew.

That was Clockwork CEO, Suresh Vasudevan, who ran product and R&D at NetApp, launched Nimble Storage at an IT Press Tour event years ago, took it through its IPO and sale to HPE, then led Sysdig.

With him were Clockwork Chief Business Officer, Dan Zheng, a founding team member and early Googler, Clockwork VP Products, Gavin Cohen, and Clockwork VP Marketing, Anita Pandey, who kept an eye on the clock (appropriately) through a session that ran close to two hours.

Australian readers may like to know Cohen is a UNSW graduate. He joined Clockwork from ScienceLogic, where he was VP of AI product management.

From stock exchange clocks to GPU clusters

Clockwork started in 2018 as TickTock Networks, built on Stanford research by Clockwork Co-founder, Balaji Prabhakar, a Stanford professor and co-inventor of the DCQCN congestion control used in RDMA networks, and his former student, Clockwork Co-founder and CTO, Yilong Geng. Clockwork Chief Scientist, Mendel Rosenblum, a VMware co-founder, was involved from the start.

The original problem was time. Their 2018 USENIX NSDI paper, Exploiting a Natural Network Effect for Scalable, Fine-grained Clock Synchronization, described synchronising clocks across huge fleets of machines to within tens of nanoseconds, using software alone.

“What we are able to do is use software, and scale to tens of thousands of nodes without requiring any proprietary hardware,” he said.

The first customers were in finance. Nasdaq needed precise timestamps to move its exchange to the cloud and prove which order arrived first. Bloomberg, Wells Fargo and Coinbase followed.

Then came the pivot. Once clocks agree to the nanosecond, you can measure exactly how long a packet takes to travel one way. The usual method halves the round-trip time and assumes both directions match, and they often don’t.

That precision matters enormously inside AI clusters, where GPU-to-GPU messages take single-digit microseconds. Clockwork turned to AI in 2023, and its FleetIQ platform became generally available in September 2025.

I couldn’t resist pointing out that Clockwork’s arrival was obviously very timely. Vasudevan took it with the patience of a man who’s heard every clock pun going.

Why one hot GPU can stall thousands

Training is a loop. Each forward and backward pass takes one to 10 seconds, repeated tens of thousands of times over weeks or months. Inference is a loop too, churning through tokens and reusing the KV cache each time.

At frontier scale, tens of thousands of GPUs act as one computer, and every GPU has to sync with its peers before the next step. One slow link or one overheated chip holds everyone up.

“The network is now part of the computing infrastructure,” Vasudevan said. “If the network has a tail latency, then the entire job is running slowly. Every GPU is now waiting.”

When something fails outright, it gets worse. NCCL, NVIDIA’s collective communications library, times out and the PyTorch job dies. Recovery means finding the fault, restarting thousands of GPUs (up to 30 minutes), reloading the last checkpoint and recomputing everything since.

The statistics on Clockwork’s slides were sobering. A Meta study puts mean time to failure at 7.9 hours for 1,024-GPU jobs. A ByteDance study found 42.5% of jobs ran at least 10% slower because of stragglers, a problem engineers call “fail slow”. Manual detection and repair can take one to three hours.

“We are used to cloud where we measure availability in four nines and five nines,” Vasudevan said. “In AI workloads you’re struggling to get to the low nineties.”

A champagne tower worth US$150 million

Cohen opened with a slide of 1,024 champagne glasses stacked in a pyramid (he warned us there’d be no champagne at lunch). Each glass was a GPU: 256 nodes of eight, about US$150 million of hardware.

“Touching any one of these GPUs or champagne glasses is essentially going to crash the entire workload running on it,” Cohen said.

Clockwork’s answer is three lines of defence.

LinkPass keeps traffic moving when a link flaps

The first is LinkPass, an NCCL plugin that needs zero model changes. When a network card or link dies, it catches the error at a low level and shifts the traffic, including in-flight packets and queue pairs, onto the other healthy NICs serving that GPU. When the link recovers, traffic moves back.

The whole failover takes one to two seconds. Vasudevan said NCCL’s transport timeout is about a minute and PyTorch’s default is 10 minutes.

“About 15 to 20% of failures stem from links breaking down,” Vasudevan said, adding that LinkPass is how Clockwork landed Oracle and LinkedIn. It’s been in production for more than a year, and Oracle Cloud Infrastructure now offers it to its customers.

Clockwork’s Three-Line AI Failover-1.png

TorchPass is vMotion for GPUs

The second line of defence is TorchPass, which live-migrates a GPU’s work to a spare, much as VMware’s vMotion moves a virtual machine.

It plugs into Kubernetes, Kubeflow, Slurm, Slinky and Soperator. When a GPU signals trouble (rising temperatures, ECC memory errors, or an NVIDIA XID error that says “drain this node”), TorchPass copies state from the still-living worker. When a GPU dies without warning, TorchPass rebuilds the lost state from a data-parallel replica elsewhere in the job. The hard bit, Cohen said, is “quiescing”, bringing hundreds or thousands of GPUs to exactly the same pause point when each runs on slightly different micro-timings.

Every conversation the failed GPU was part of has to carry on as if it never left. “Every single operation, we have to make sure that the replaced rank’s role is being substituted cleanly,” Vasudevan said.

I asked how many companies actually have a spare rack sitting around. “All of them,” Vasudevan said. Neoclouds increasingly ask tenants to hold 5% to 8% spare capacity, and those spares run lower-priority jobs that can be pre-empted when needed.

Snapshots, the newest layer

The third layer, announced on 5 October, is TorchPass platform snapshots. Clockwork calls them the industry’s first multi-node snapshots: a consistent capture of an entire running job, across every node, without touching the model code.

That covers GPU memory, CUDA state, CPU memory, open files and in-flight network state. Linux’s CRIU handles the CPU side, Vasudevan explained, and the trick is marrying that to the GPU state.

I asked how big they are. “They’re big,” Cohen said. Vasudevan put it at about 200GB per GPU.

Storage readers will like this bit. Snapshots can write straight to storage via GPUDirect Storage, or land in DRAM and drain to storage in the background.

According to the slides, a platform snapshot takes under 20 seconds, with about 1% overhead when taken every 25 minutes. Tenants who add Clockwork’s hooks (patches exist for TorchTitan, DeepSpeed and Megatron-LM) get model checkpoints in under five seconds, at about 1% overhead every five minutes.

Storage, Cohen added, “is usually the easiest, cheapest component in these environments”, a line I suspect a few storage vendors on this tour would dispute. Vasudevan admitted snapshot restore times are still being benchmarked.

Clockwork’s AI resilience stack-2.png

Pulling cables in the South Bay

Advertisement

Clockwork Tech Lead, Lin Shi, walked us through a two-node training job, each node with four GPUs and four RDMA NICs, running at about one step per second.

A colleague pulled a network cable. The step counter kept climbing while two surviving NICs picked up the slack, and traffic rebalanced when the cable went back in.

Next came the power cords. “These are the same cables that power my rice cooker,” Lin Shi said, as his gloved colleague pulled them out.

Training froze at step 5,040 for about a minute, then a hot spare took over at step 5,040 as a new Kubernetes pod on a different server. Cohen later told us Lin Shi hadn’t slept for a week, which I can believe, although presumably he was half-joking. 

Advertisement

FleetLens asks whether it’s the network

The other half of FleetIQ is observability. Cohen positioned FleetLens as a specialist for the fabric blind spots that Datadog, Splunk, New Relic and NVIDIA’s own DCGM and UFM tools share.

The scale explains why. Clockwork’s slide on an NVIDIA DGX GB300 SuperPOD counted 9,216 GPUs, 128 leaf switches, 72 spine switches, about 25,000 network links and about 50,000 transceivers.

Faults in there show up as grey failures, where every dashboard shows green and jobs still crawl. FleetLens fires lightweight probes between NIC pairs every 10 milliseconds, measuring true one-way delay thanks to the synced clocks. Switch counters, Zheng said, typically average over 30 to 60 seconds.

Each probe is mapped to the switches and ports it crossed, and triangulation finds the narrowest shared component. In a recorded demo, FleetLens pinpointed a lossy leaf-to-spine link down to the port and direction of loss.

“Because the network gets blamed for everything,” Cohen said. “So we want to definitively say: is it or isn’t the network?”

An AI SRE that explains root causes and fixes is next on the roadmap. I offered “elite lens” as a tagline, free of charge (it’s what journalists do).

FleetAudit runs the same technology as a pre-flight check before a cluster goes to a customer. WhiteFiber(NASDAQ: WYFI) uses it to catch ConnectX-7 and BlueField NIC issues, marginal transceivers, MPO cable faults and ECN storms.

WhiteFiber CTO, Tom Sanfilippo, said in the release that Clockwork’s fleet audit “localises faults in minutes, and lets us correct them before acceptance.”

Who’s buying, and how the numbers stack up

LinkedIn runs LinkPass across its AI fleet, including a 4,096-GPU cluster. Before that, one leaf port flap could knock eight to 16 GPUs offline and crash the job.

“At AI infrastructure scale, a single network issue should never sideline healthy GPUs or interrupt running workloads,” LinkedIn SVP, CTO Infrastructure, Raghu Hiremagalur, said in the release. Clockwork says LinkPass prevents tens of thousands of GPU-hours of downtime a month there.

Together AI is packaging LinkPass, TorchPass and, eventually, snapshots as a service. “Our customers grade us on goodput, the share of their GPU-hours that actually move the model forward,” Together AI Product Lead, Pavneet Ahluwalia, said in the release.

A global tech company running thousands of on-prem GPUs is a design partner for snapshots, since its platform team can’t force every research team to checkpoint. The release also lists Nebius, Nscale, Wells Fargo and DCAI as customers.

“Efficiency is a new capacity,” Zheng said. “You have to do with what you have and really squeeze more juice out of that lemon.”

SemiAnalysis now includes TorchPass in its ClusterMAX TCO calculator. On default values for a 4,096-GPU cluster, Clockwork’s slide shows goodput loss falling from 13.68% to 2.76% for a gold-tier provider, and from 28.51% to 10.31% for silver.

“In our ClusterMAX, TorchPass cuts training goodput loss from 14% to under 3% for a gold-rated neocloud,” SemiAnalysis Founder, CEO and Chief Analyst, Dylan Patel, said in the release.

Clockwork detailed 1.jpg

Competition, pricing and a guarantee

Vasudevan said pricing is typically per GPU per year, with capacity-based enterprise licence agreements for some large cloud deals. Neoclouds buy direct, and enterprises increasingly buy through partners such as Together AI.

The main rival, he said, is inertia. “It’s either DIY or sort of what is the urgency to act,” Vasudevan said, since the default is to live with checkpoint-restart and its lost hours.

There are alternatives. Meta’s open source TorchFT drops a failed replica group and keeps going without it. The New Stack reported that a SemiAnalysis test (a single configuration on 64 H200s) found TorchPass ahead of TorchFT on throughput with similar recovery times.

Futuriom points to Crusoe’s AutoClusters, which replaces failed GPUs in under five minutes, and CoreWeave’s stall-detection tools.

Clockwork’s differentiator is that it has put its claims in writing. Its YOCO (You Only Compute Once) guarantee, launched in July and effective from 3 August, promises that at least 90% of training failures on supported TorchPass workloads will be resolved through live migration, or customers get a 25% credit on renewal.

“There’s a big difference between a vendor making a slide that says their product works and them writing it into a contract,” SemiAnalysis Member of Technical Staff and ClusterMAX Lead Author, Jordan Nanos, said at the time.

AMD is “an equal citizen”, Vasudevan said, because AMD’s RCCL derives from NCCL. It also works on InfiniBand and on RoCE from NVIDIA or Arista. Google TPUs, AWS Trainium and scale-up fabrics such as NVLink and UALink aren’t supported yet.

The money

The US$31 million round was co-led by Premji Invest, Wing Venture Capital and Seligman Ventures, with NEA and e& Capital returning. Total funding is now US$73 million.

“The one thing that scales perfectly is unreliability: put enough GPUs in one machine and something is always failing,” NEA Venture Partner, Greg Papadopoulos, said in the release.

Why Australia should care

In September, NVIDIA and eight local partners (Firmus, Sharon AI, IREN, ResetData, Megaport, CDC, NEXTDC and AirTrunk) announced plans for up to 2GW of AI computing capacity by 2027.

Firmus is rated Silver in SemiAnalysis’ ClusterMAX 3.0, which is exactly the tier where Clockwork’s modelling shows the biggest goodput gains.

Australia’s real constraint is power, with grid connection queues in NSW and Victoria. When every new megawatt is hard won, squeezing more useful work out of GPUs already plugged in matters even more, and Zheng’s “efficiency is a new capacity” line starts to sound like local energy policy.

Clockwork detailed 2.jpg

My take

Clockwork has picked a genuinely hard problem, and it showed its work with real cables and real power cords. That’s rarer on these tours than it should be.

The open questions are snapshot restore times, how fast NVIDIA folds similar features into NCCL, and independent proof of that 90% figure beyond SemiAnalysis’ testing. I’d also like to see TPU and Trainium support before calling it universal.

For now, as Vasudevan put it in the release, “Failures are inevitable at AI scale. Losing hours of useful work to them should not be.” Watching step 5,040 survive a pulled power cord made that case better than any slide.

Here is the video again if you didn’t get a chance to watch it above, after which is the list of sources:

Sources

This article was written and researched with AI assistance and edited by Alex Zaharov-Reutt.

Alex Zaharov-Reutt travelled to the 70th Edition of the IT Press Tour as a guest of The IT Press Tour.

Leave a Reply

Your email address will not be published. Required fields are marked *