In this paper, a novel optical logic operator based on the multifunctional metasurface driven by all‐optical diffractive neural network is reported, which can perform four principal quantum logic operations (Pauli‐X, Pauli‐Y, Pauli‐Z, and Hadamard gates). The two ground states and are characterized by two orthogonal linear polarization states. The proposed spatial‐ and polarization‐multiplexed all‐optical diffractive neural network only contains a hidden layer physically mapped as a metasurface with simple and compact unit cells, which dramatically reduces the volume and computing resources required for the system. The designed optical quantum operator is proven to achieve high fidelities for all four quantum logical gates, up to 99.96% numerically and 99.88% experimentally. The solution will facilitate the construction of large‐scale optical quantum computing systems and scalable optical quantum devices.