Purpose: Respiratory motion is one of the major challenges in radiotherapy. In this work, a comprehensive and clinically plausible set of 4D numerical phantoms, together with their corresponding “ground truths,” have been developed and validated for 4D radiotherapy applications. Methods: The phantoms are based on CTs providing density information and motion from multi-breathing-cycle 4D Magnetic Resonance imagings (MRIs). Deformable image registration (DIR) has been utilized to extract motion fields from 4DMRIs and to establish inter-subject correspondence by registering binary lung masks between Computer Tomography (CT) and MRI. The established correspondence is then used to warp the CT according to the 4DMRI motion. The resulting synthetic 4DCTs are called 4DCT(MRI)s. Validation of the 4DCT(MRI) workflow was conducted by directly comparing conventional 4DCTs to derived synthetic 4D images using the motion of the 4DCTs themselves (referred to as 4DCT(CT)s). Digitally reconstructed radiographs (DRRs) as well as 4D pencil beam scanned (PBS) proton dose calculations were used for validation. Results: Based on the CT image appearance of 13 lung cancer patients and deformable motion of five volunteer 4DMRIs, synthetic 4DCT(MRI)s with a total of 871 different breathing cycles have been generated. The 4DCT(MRI)s exhibit an average superior–inferior tumor motion amplitude of 7 ± 5 mm (min: 0.5 mm, max: 22.7 mm). The relative change of the DRR image intensities of the conventional 4DCTs and the corresponding synthetic 4DCT(CT)s inside the body is smaller than 5% for at least 81% of the pixels for all studied cases. Comparison of 4D dose distributions calculated on 4DCTs and the synthetic 4DCT(CT)s using the same motion achieved similar dose distributions with an average 2%/2 mm gamma pass rate of 90.8% (min: 77.8%, max: 97.2%). Conclusion: We developed a series of numerical 4D lung phantoms based on real imaging and motion data, which give realistic representations of both anatomy and motion scenarios and the accessible “ground truth” deformation vector fields of each 4DCT(MRI). The open-source code and motion data allow foreseen users to generate further 4D data by themselves. These numeric 4D phantoms can be used for the development of new 4D treatment strategies, 4D dose calculations, DIR algorithm validations, as well as simulations of motion mitigation and different online image guidance techniques for both proton and photon radiation therapy.