Impact of Crime on Public Building Usage Act of 2023 This bill requires reports to Congress on the impact of crime and safety concerns on public building usage and associated costs. Specifically, the Government Accountability Office must conduct a review and report on, among other things the effects of increased crime rates and safety concerns, including the use of fentanyl and other drugs and homelessness, in urban areas on building usage for in-person work at federal buildings; the effects of low office utilization rates on safety around federal buildings; and any costs associated with safety issues impacting federal buildings. The Office of Inspector General of the General Services Administration must submit to Congress a report on the impacts on and costs associated with building operations related to crime and public safety in and around federal buildings.