A valid method for trace silver (Ag) detection in geological samples was developed in this study using aerosol dilution inductively coupled plasma-mass spectrometry after extraction with inverse aqua regia. This was proposed primarily to reduce the interference from Nb and Zr during mass spectrometric measurements. Almost 93% of Nb and Zr was removed after the extraction. By mixing an appropriate amount of Ar with the sample aerosol using an aerosol dilution system prior to plasma, the residual Nb oxides and Zr oxides or hydroxides could be successfully removed. The relative yields of the interfering oxides and hydroxides were as low as 0.087% (NbO/Nb) and 0.013% (ZrOH/Zr), which were 3–5 times lower than those in the traditional mode without the addition of Ar. Moreover, the signal-to-noise ratio of Ag was five times higher than that in the traditional mode. The proposed method was applied to the determination of Ag in 68 standard reference materials (SRMs) of soil, sediment, and rock. The results for 47 of these geological SRMs were in good agreement with the reference values. The Ag levels in three SRMs (GSP-2 Granodiorite, STM-2, and SGR-1b) are being reported for the first time herein. For these SRMs, 10 separate aliquots of the sample were digested and analyzed over a period of three months, and analysis revealed that the determined values were reasonable. Thus, the proposed method shows significant potential for the accurate determination of trace Ag in various geological samples.