Few-shot learning is one of the important research areas in machine learning. It aims to train models with extremely limited labeled samples and generalize to unseen categories or tasks. The key challenges are learning class-specific representations and modeling relationships between samples under limited supervision. In addition, data perturbations tend to have a more pronounced impact in few-shot settings. To address these challenges, we propose Cross-Memory Attention (CMA). CMA extracts memory features from both the support and query sets and integrates the support memories into the query representations. In this way, our method models long-range dependencies between the support and query sets while maintaining a lower parameter count than standard Transformer-based designs, thereby addressing relationship modeling between samples. Additionally, the Domain Adaptation Module mitigates the impact of data perturbations by training an additional branch with perturbed data. The module constructs a learnable classification space, whose introduction overcomes the limitations of fixed, non-learnable metric learning in the classification space. To enable the use of our Cross-Memory Attention in 5-shot scenarios, we propose a Multi-sample Adaptive Fusion Module that can be applied to any multi-sample learning framework. This module effectively extracts common features from multiple samples, making it versatile and adaptable. Finally, we conduct extensive experiments on four public datasets, validating the effectiveness of our model. The code is made publicly available at https://github.com/Mrliujunwen/cross-memory-fewshot.