tl;dr: we built TorchPass because large distributed training jobs fail a lot, and checkpoint restart is expensive. TorchPass addresses this by migrating the state from failed resources to spares.
In large GPU clusters, even small failures (a GPU falling off the bus, a node crash, a network link flap) can bring down an entire distributed training job. And once you get into clusters with hundreds or thousands of GPUs, something is almost always failing. Research from Meta suggests mean time to failure drops to about 7.9 hours for a 1,024-GPU cluster. And when a single failure occurs, the entire distributed job crashes.
The usual recovery model is to take frequent checkpoints during training, and recover from the most recent checkpoint when a failure occurs. But:
all work since the last checkpoint is lost\ntime is wasted replacing nodes and checkpoint reloading\nmore time is lost restarting the entire distributed job
TorchPass uses a different approach: instead of restarting the job, it migrates the failed training rank to a spare GPU and resumes training at the same step.
TorchPass supports planned migration (triggered pre-emptively when an imminent failure is detected) or unplanned migration (triggered by a hard failure). Further details about how it works can be found here:https://clockwork.io/blog/torchpass-workload-fault-tolerance...
We ran a 3,000 step training benchmark using TorchTitan Llama-4 MoE Scout (109B) on 64 H200 GPUs with random failure injection to compare checkpoint restarts, TorchPass and TorchFT.
TorchPass completed in 405 min\nCheckpoint restart completed in 818 min. \nTorchFT in 930 min
Checkpoint restart was slower mainly because of the time taken to restore from checkpoint, restart the training and recompute the work since the last checkpoint.
TorchFT lost almost no time due to the failures, but was slower because it introduced a significant per-step overhead because it requires the using gloo (rather than NCCL) for cross replica all reduce operations.
Happy to answer questions about the implementation and benchmarks."},"story_title":{"matchLevel":"none","matchedWords":[],"value":"Fault Tolerance Benchmark: Clockwork TorchPass, TorchFT and Checkpoint Restart"},"story_url":{"fullyHighlighted":false,"matchLevel":"partial","matchedWords":["clockwork",".","io"],"value":"https://clockwork.io/blog/keeping-distributed-training-running-through-failures/"}},"_tags":["comment","author_danzheng","story_47338038"],"author":"danzheng","comment_text":"Hi all \u2014 I\u2019m Dan, founding team at clockwork.io. Today we launched TorchPass! We'd love to get your feedback.
tl;dr: we built TorchPass because large distributed training jobs fail a lot, and checkpoint restart is expensive. TorchPass addresses this by migrating the state from failed resources to spares.
In large GPU clusters, even small failures (a GPU falling off the bus, a node crash, a network link flap) can bring down an entire distributed training job. And once you get into clusters with hundreds or thousands of GPUs, something is almost always failing. Research from Meta suggests mean time to failure drops to about 7.9 hours for a 1,024-GPU cluster. And when a single failure occurs, the entire distributed job crashes.
The usual recovery model is to take frequent checkpoints during training, and recover from the most recent checkpoint when a failure occurs. But:
all work since the last checkpoint is lost\ntime is wasted replacing nodes and checkpoint reloading\nmore time is lost restarting the entire distributed job
TorchPass uses a different approach: instead of restarting the job, it migrates the failed training rank to a spare GPU and resumes training at the same step.
TorchPass supports planned migration (triggered pre-emptively when an imminent failure is detected) or unplanned migration (triggered by a hard failure). Further details about how it works can be found here:https://clockwork.io/blog/torchpass-workload-fault-tolerance...
We ran a 3,000 step training benchmark using TorchTitan Llama-4 MoE Scout (109B) on 64 H200 GPUs with random failure injection to compare checkpoint restarts, TorchPass and TorchFT.
TorchPass completed in 405 min\nCheckpoint restart completed in 818 min. \nTorchFT in 930 min
Checkpoint restart was slower mainly because of the time taken to restore from checkpoint, restart the training and recompute the work since the last checkpoint.
TorchFT lost almost no time due to the failures, but was slower because it introduced a significant per-step overhead because it requires the using gloo (rather than NCCL) for cross replica all reduce operations.
Happy to answer questions about the implementation and benchmarks.","created_at":"2026-03-11T16:56:00Z","created_at_i":1773248160,"objectID":"47338084","parent_id":47338038,"story_id":47338038,"story_title":"Fault Tolerance Benchmark: Clockwork TorchPass, TorchFT and Checkpoint Restart","story_url":"https://clockwork.io/blog/keeping-distributed-training-running-through-failures/","updated_at":"2026-03-11T22:31:03Z"},{"_highlightResult":{"author":{"matchLevel":"none","matchedWords":[],"value":"danzheng"},"comment_text":{"fullyHighlighted":false,"matchLevel":"full","matchedWords":["clockwork",".","io","gpu"],"value":"A very clever part of the HUYGENS algorithm is that it doesn\u2019t just sync clocks pair-wise, it leverages a natural network effect where a group of pair-wise synchronized clocks becomes transitively synchronized, helping reduce errors further without requiring specialized hardware. That\u2019s one of the key reasons it can achieve ~100 nanoseconds of software-based sync on commodity networks.
The authors\u2019 work forms the basis of what the team at Clockwork.io is building, enabling accurate one-way delay measurements (rather than just RTT/2) that improve latency visibility and telemetry across CPU and GPU infrastructure"},"story_title":{"matchLevel":"none","matchedWords":[],"value":"Clock synchronization is a nightmare"},"story_url":{"matchLevel":"none","matchedWords":[],"value":"https://arpitbhayani.me/blogs/clock-sync-nightmare/"}},"_tags":["comment","author_danzheng","story_46368177"],"author":"danzheng","comment_text":"A very clever part of the HUYGENS algorithm is that it doesn\u2019t just sync clocks pair-wise, it leverages a natural network effect where a group of pair-wise synchronized clocks becomes transitively synchronized, helping reduce errors further without requiring specialized hardware. That\u2019s one of the key reasons it can achieve ~100 nanoseconds of software-based sync on commodity networks.
The authors\u2019 work forms the basis of what the team at Clockwork.io is building, enabling accurate one-way delay measurements (rather than just RTT/2) that improve latency visibility and telemetry across CPU and GPU infrastructure","created_at":"2025-12-28T17:42:37Z","created_at_i":1766943757,"objectID":"46412826","parent_id":46404046,"story_id":46368177,"story_title":"Clock synchronization is a nightmare","story_url":"https://arpitbhayani.me/blogs/clock-sync-nightmare/","updated_at":"2026-03-05T23:14:40Z"}],"hitsPerPage":20,"nbHits":3,"nbPages":1,"page":0,"params":"query=clockwork.io+gpu&advancedSyntax=true&analyticsTags=backend","processingTimeMS":31,"processingTimingsMS":{"_request":{"queue":5,"roundTrip":17},"afterFetch":{"total":1},"fetch":{"query":11,"scanning":17,"total":29},"total":31},"query":"clockwork.io gpu","serverTimeMS":36}