Detecting neural replay non-invasively in humans is hard, and the methods used for it have lacked a validation against a known ground truth. We present FASTIMAGES, a benchmark dataset of seventy participants with parallel fMRI (n = 40) and MEG (n = 30) recordings containing known neural sequences evoked by fast visual stimulation. Using these data we evaluate two sequence-detection methods — Temporally Delayed Linear Modelling (TDLM) for MEG and Slope Order Dynamic Analysis (SODA) for fMRI. Both excel in their native modality with comparable effect sizes, whereas cross-modality transfer remains challenging. FASTIMAGES offers data with known, clearly expressed sequences as a shared reference point to validate future methods.
Jump to Results Summary ↓Both methods first use a brain decoder to estimate, at every time point, how strongly each stimulus is currently "reactivated", and then test whether those reactivations follow an expected sequence.
Seventy participants viewed grayscale images from five categories (house, cat, chair, face, shoe) in two task types:
The two task types, as seen by participants on screen:
Cross-validated decoders trained at each time point clearly surpassed chance (20%): peak accuracy reached ~55% in MEG (around 150 ms after onset, with a rapid rise and slow decay) and ~70% in fMRI (peaking around 4 s, reflecting the slower hemodynamic response). During the fast sequences, the presented order is visible by eye in the averaged decoded probabilities — even in the fastest MEG condition.
Applied to the MEG sequence trials, TDLM produced clear sequenceness peaks at the expected time lag in all four speed conditions, surpassing significance for a t-test, a sign-flip permutation test and a cluster-permutation test. Between-participant variance was high, however: only about half of participants reached individual significance, and sequenceness was strongly correlated with each participant's decoder quality (r = 0.70) but not with their behavioural performance. This underlines that TDLM should be interpreted at the group, not the individual, level.
Applied to the fMRI sequence trials, SODA showed the characteristic onset (negative) and offset (positive) slope dynamics. The offset period was significant in all speed conditions, while the onset period was harder to detect and only reached significance for the slower speeds. As in MEG, slope magnitude correlated with decoder quality but not behaviour. Notably, the theoretically derived onset/offset windows only partially overlapped the empirically significant clusters.
To build intuition, we ran both methods on contrived examples that partly violate their assumptions. TDLM produces a sharp peak only when the inter-item lag is constant, and smears or vanishes when lags vary, items are missing, or the order is wrong. SODA, because it is dominated by the first and last items of the sequence, can produce a convincing onset/offset signature even from partial sequences — making it robust to outliers but harder to interpret directionally.
Each method struggles in the other's modality. SODA on MEG finds onset peaks but no genuine offset phase, because MEG reactivations decay too fast to overlap — though this very property makes SODA a promising candidate for detecting fast "clustered" reactivation. TDLM on fMRI cannot recover any of the benchmark sequences, since the TR (1.25 s) is slower than the sequences; it only succeeds for a very slow 2148 ms condition, and only when trials are averaged.
Across speed conditions, TDLM showed stable medium effect sizes (mean Cohen's d = 0.77). SODA effect sizes were speed- and period-dependent (offset mean d = 0.9), and a combined onset+offset measure markedly increased both effect size and power. In this idealized, stimulus-driven benchmark — an absolute best-case scenario — a bootstrap analysis suggested that even roughly 20 participants and 20 trials are needed to reach 80% power, and only when the expected lag or period is known in advance. Real replay studies, with weaker, non-time-locked signals and unknown timing, should expect to need substantially more. When timing is unknown, the cluster-permutation test was consistently more powerful than a sign-flip permutation test — our recommended best practice.
FASTIMAGES validates that TDLM and SODA each perform well when the data meet their assumptions — temporally resolved peaks for TDLM (MEG) and overlapping responses for SODA (fMRI) — but that they do not transfer readily to the other modality. The two methods are therefore complementary rather than interchangeable. Across both, decoder quality emerged as the dominant limiting factor, suggesting clear room for improvement. Because the benchmark uses stimulus-driven, time-locked sequences decoded by classifiers trained on the same stimuli, it represents an upper bound on detection sensitivity.
We suggest using FASTIMAGES as:
The analysis code is available on GitHub, and the BIDS datasets are hosted on GIN: MEG and fMRI.