The difficulty of tracking multiple moving objects among identical distractors increases with the number of tracked targets. Previous research has shown that the number of targets tracked (i.e., load) modulates activity in brain areas related to visuospatial attention, giving rise to so-called attention response functions (ARFs). While the hemifield/hemispheric effects of spatial attention (e.g., hemispatial neglect, hemifield capacity limits) are well described, it had not previously been tested whether a hemispheric or hemifield imbalance exists among ARFs. By recording blood oxygenation level-dependent activity from human brains (n = 19, female and male) in a multiple-object tracking paradigm, we show that the number of tracked objects modulates activity in a large network of areas bilaterally. A significant effect of contralateral load was found in earlier areas throughout the dorsal and ventral visual streams, while the effects of ipsilateral load emerged in later areas. Both contra- and ipsilateral load significantly influenced activity in the parietal and frontal lobes, specifically the dorsal attention network. In addition, some brain regions in the occipital lobe were significantly more sensitive to contralateral than ipsilateral load. Our results are consistent with findings showing that a diverse set of brain areas contributes to tracking multiple targets. In particular, we extend the canonical view of load-based ARFs to include hemifield bias. Given the hemifield-specific nature of speed and capacity limits to multiple-object tracking, we conjecture that areas that show a strong hemifield preference may impose a bottleneck on processing that results in limits on the capacity and speed of tracking.