Extended reality (XR) technologies are rapidly advancing and becoming an integral part of our day-to-day life. XR technologies enable immersive multi-user applications such as virtual reality, collaborative augmented reality (AR), and shared virtual environments. Such multi-user XR (MuXR) applications place strict demands on local wireless networks, like dense Wi-Fi environments where latency, fairness, and stability directly impact user experience. In this paper, we propose a federated multi-link aggregation (FeMLA) framework for dense multi-link (ML-AP) Wi-Fi networks, designed to optimize quality of experience (QoE) for MuXR traffic under multi-link (multi-band) spectrum configurations. FeMLA exchanges lightweight local learning metrics among interfering neighbors and constructs a max-min global learning reward to explicitly optimize worst-user QoE. We evaluate our proposed framework with varying network densities (interference conditions) and XR traffic loads, using throughput, delay, and worst-user performance as key metrics. Simulation results show that in dense deployments with up to 16 ML-APs, FeMLA reduces mean XR delay to 28 ms, achieving an 86-93% latency reduction compared to fixed, random, and standalone reinforcement learning baselines, while improving mean QoE to 0.85. Moreover, FeMLA elevates the median worst-user QoE from near-zero or moderate levels to 0.81, demonstrating strong fairness and stability under severe interference. These results highlight federated, QoE-driven coordination as a scalable and effective approach for supporting next-generation multi-user Social XR over dense Wi-Fi networks.