Brain diseases significantly impact physical and mental health, making the development of models to identify biomarkers for early diagnosis essential. However, building high-quality models typically relies on large-scale datasets, while the privacy-sensitive nature of medical data often restricts its sharing and utilization. Multi-site studies provide a potential solution by integrating data from various sources, yet existing methods frequently neglect site-specific private features, such as demographic information. Therefore, in this paper, we propose a simple yet effective framework based on Tensor Decomposition and Personalized Federated Learning (TDPFL) for multi-site brain disease recognition, while protecting these private features. On the central server, we designed a dual feature aggregation module to facilitate efficient knowledge sharing among sites. On the client side, we introduced a personalized branch to safeguard private information (i.e., age, gender, and education) and developed a tensor decomposition module to extract features from subjects' brain scan data. Furthermore, we developed a dynamic prototype aggregation module to monitor evolving brain features over time. This mechanism enhances the model's capacity to capture these dynamics, thereby improving classification and prediction accuracy. Experiments on two publicly available rs-fMRI datasets across six sites showed that TDPFL outperformed baseline methods with a 4 % improvement in average classification accuracy. Additionally, we identified site-specific brain disease-related biomarkers, offering novel insights into early diagnosis. Code is available at https://github.com/ChaojunZ/TDPFL.git.