Performance metrics evaluation of multi-objective optimization for transportation

Various multi-objective optimization methods are widely applied in the transportation field, enabling decision-makers to find solutions that balance trade-off objectives. Since comparing the performance of multi-objective optimization methods is generally difficult, many performance metrics are introduced to quantitatively evaluate the performance of multi-objective optimization methods. However, the effectiveness of the performance metrics needs to be further investigated. Thus, we first critically analysed a series of performance metrics, including number of solutions obtained (NOSO), overall nondominated solutions number (ONSN), normalized maximum spread (NMS), error ratio (ER), nearest ideal distance (NID), mean ideal distance (MID), spacing (SP), inverted generational distance (IGD), and hypervolume-based ratio (HR), which were extensively adopted to assess the performance of multi-objective optimization methods. We found that these performance metrics cannot always accurately reflect the quality of solutions obtained and may be misleading. Thereafter, two axioms were proposed to define the criteria for reliable performance metrics. Additionally, whether these performance metrics satisfied the two axioms was rigorously proved. The performance metrics that satisfied both axioms, i.e., NOSO, ONSN, NMS, ER, and HR, were considered reliable. Furthermore, a real-world cargo transportation case was investigated, indicating the unreliability of metrics MID and SP.