Urban Mobility Models (UMMs) are fundamental tools for estimating the population in urban sites and their spatial movements over time. They have great value for such applications as managing the resources of cellular networks, predicting traffic congestion, and city planning. Most existing UMMs were developed primarily in 2D. However, we argue that people's movements and living patterns involve 3D space, i.e., buildings, which can heavily affect the accuracy of UMMs. In this paper, we for the first time conduct a comprehensive study on the impacts of buildings on human movements, and the effect on UMMs. In particular,we start from an extensive trace analysis of two different real-world datasets. Our key observation is that human patterns of movement among urban sites are affected by buildings, with buildings being able to "temporarily hold" human mobility. We innovatively capture this property by extending Markov processes, which have been widely used in developing UMMs, with semi-absorbing states. We then develop a Semi-absorbing Urban Mobility model (SUM) and theoretically prove its properties to capture the intrinsic impacts of buildings with an analysis of SUM on its difference from that of previous UMMs. Our evaluation also demonstrates that, as a basis for supporting mobile applications in an intracity and hourly scale, the SUM is far superior to previous UMMs. Our real-world case study on cellular network resource allocations further reveals the effectiveness of our SUM model. We show that the performance of the resource allocation scheme in a cellular network substantially improves by using SUM, with a reduction in the packet loss probability of 3.19 times.