We explore the potential of Tb- and Yb-doped InVO4, InTaO4, and InNbO4 for applications as phosphors for light-emitting sources. Doping below 0.2% barely change the crystal structure and Raman spectrum but provide optical excitation and emission properties in the visible and near-infrared (NIR) spectral regions. From optical measurements, the energy of the first/second direct band gaps was determined to be 3.7/4.1 eV in InVO4, 4.7/5.3 in InNbO4, and 5.6/6.1 eV in InTaO4. In the last two cases, these band gaps are larger than the fundamental band gap (being indirect gap materials), while for InVO4, a direct band gap semiconductor, the fundamental band gap is at 3.7 eV. As a consequence, this material shows a strong self-activated photoluminescence centered at 2.2 eV. The other two materials have a weak self-activated signal at 2.2 and 2.9 eV. We provide an explanation for the origin of these signals taking into account the analysis of the polyhedral coordination around the pentavalent cations (V, Nb, and Ta). Finally, the characteristic green (5D4 → 7FJ) and NIR (2F5/2 → 2F7/2) emissions of Tb3+ and Yb3+ have been analyzed and explained.