In the context of multimedia retrieval, the goal of accuracy is to a certain extent contradictory with that of efficiency. The former relies on exploiting sophisticated features, whereas the latter favors using simple features with reduced dimensionality. As an endeavor to strike the balance between these two goals, this paper presents a self-adaptive semantic schema mechanism (SSM) for multimedia databases. The SSM is implemented based on an object-oriented data model, with classes being organized into a semantic hierarchy. As its most distinguishable feature, when the conditions of certain ECA-rules are satisfied, SSM supports adaptive evolution of a schema in the form of expansion with new classes and/or compaction by removing inefficient ones. This self-adaptive evolution strategy allows a schema to optimize for the requirements of each specific application, thereby achieving a dynamic, application-specific balance between accuracy and efficiency. A prototype system for multimedia retrieval, 2M2Net, has been built based on this mechanism and validated for its feasibility.