In this paper, an extremely efficient pipeline architecture is proposed for the realization of the prime factor algorithm (PFA). By using the extended diagonal feature of the Chinese Remainder Theorem (CRT) mapping, we show that the data transactions during the computation can be efficiently carried out with the simplest control strategy. Due to this reason, the present approach has the least hardware requirement as compared to the previous propositions. Furthermore, in the case of multi-dimensional PFA computation, it does not require the computation to be split up into a number of two-dimensional ones. Consequently, the overhead which is required for data loading and data retrieval in each two-dimensional stage can be saved. In fact, all these savings are achieved by only using one more connection link which connects all the memory buffers via the extended diagonal of a multi-dimensional array.