Competition between spin liquids and valence-bond order in the frustrated spin-1/2 Heisenberg model on the honeycomb lattice

Using variational wave functions and Monte Carlo techniques, we study the antiferromagnetic Heisenberg model with first-neighbor J1 and second-neighbor J2 antiferromagnetic couplings on the honeycomb lattice. We perform a systematic comparison of magnetically ordered and nonmagnetic states (spin liquids and valence-bond solids) to obtain the ground-state phase diagram. Néel order is stabilized for small values of the frustrating second-neighbor coupling. Increasing the ratio J2/J1, we find strong evidence for a continuous transition to a nonmagnetic phase at J2/J1≈0.23. Close to the transition point, the Gutzwiller-projected uniform resonating valence-bond state gives an excellent approximation to the exact ground-state energy. For 0.23 J2/J1 0.36, a gapless Z2 spin liquid with Dirac nodes competes with a plaquette valence-bond solid. In contrast, the gapped spin liquid considered in previous works has significantly higher variational energy. Although the plaquette valence-bond order is expected to be present as soon as the Néel order melts, this ordered state becomes clearly favored only for J2/J1 0.3. Finally, for 0.36 J2/J1≤0.5, a valence-bond solid with columnar order takes over as the ground state, being also lower in energy than the magnetic state with collinear order. We perform a detailed finite-size scaling and standard data collapse analysis, and we discuss the possibility of a deconfined quantum critical point separating the Néel antiferromagnet from the plaquette valence-bond solid.