To develop and test the reliability of a new semiquantitative scoring system for the assessment of cartilage changes by ultrasound in a web-based exercise as well as a patient exercise of patients with RA. Methods: A taskforce of the Outcome Measures in Rheumatology Ultrasound Working Group performed a systematic literature review on the US assessment of cartilage in RA, followed by a Delphi survey on cartilage changes and a new semiquantitative US scoring system, and finally a web-based exercise as well as a patient exercise. For the web-based exercise, taskforce members scored a dataset of anonymized static images of MCP joints in RA patients and healthy controls, which also contained duplicate images. Subsequently, 12 taskforce members used the same US to score cartilage in MCP and proximal interphalangeal joints of six patients with RA in in a patient reliability exercise. Percentage agreement and prevalence of lesions were calculated, as intrareader reliability was assessed by weighted kappa and interreader reliability by Light's kappa. Results: The three-grade semiquantitative scoring system demonstrated excellent intrareader reliability (kappa: 0.87 and 0.83) in the web-based exercise and the patient exercise, respectively. Interreader reliability was good in the web-based exercise (kappa: 0.64) and moderate (kappa: 0.48) in the patient exercise. Conclusion: Our study demonstrates that ultrasound is a reliable tool for evaluating cartilage changes in the MCP joints of patients with RA and supports further development of a new reliable semiquantitative ultrasound scoring system for evaluating cartilage involvement in RA.