Introduction

Recently, FeSe has attracted considerable interest because of its atypical magnetism1,2,3,4,8, 9), or in the mono layer limit (Tc∼65-109 K, ref. 10). The unique superconducting properties of FeSe are presumably related to its magnetism, which has also been shown to be uncommon. FeSe displays nematic, but not stripe magnetic order1,11 that is unexpected because nematic order has been argued to be the consequence of stripe magnetic order, and both break the C4 lattice symmetry11. In iron pnictides, the stripe magnetic order invariably occurs at or immediately below the nematic-(tetragonal-to-orthorhombic) ordering temperature11. Although previous works have shown that the nematic order could be driven by spin fluctuations without the requirement of magnetic order2,3,4,12, the microscopic origin of the absence of the long-range stripe magnetic order in FeSe remains elusive.

Theoretical studies have suggested that the stripe magnetic order in FeSe is absent due to the development of other competing instabilities2,3,4,27. To further elucidate the role of the Néel spin fluctuations in iron-based superconductivity, a detailed study of the pressure-/electron-do** dependence of the spin correlations in FeSe would be desirable.

To conclude, we have reported the observation of the Néel spin fluctuations over a wide energy range in an iron-based superconductor. We show that the absence of the long-range magnetic order in FeSe is due to the competition between the Néel and stripe magnetic instabilities. This differs from the parent compounds of the cuprate and iron pnictide high-temperature superconductors, where only one type of magnetic order is observed. Our findings agree with a theoretical prediction that FeSe is a novel S=1 nematic quantum-disordered paramagnet interpolating between the Néel and stripe magnetic instabilities2, which indicates a connection between the magnetism of the cuprate- and iron-based superconductors. The experimental determination of the nematic magnetic ground state of FeSe will be extremely valuable in identifying the microscopic mechanism of superconductivity in FeSe-based materials8,9,10,24,25,26.

Methods

Sample growth and characterizations

Our FeSe single crystals were grown under a permanent gradient of temperature (∼400–330 °C) in the KCl–AlCl3 flux28. The single-crystal X-ray and neutron diffraction refinements on our samples indicated a stoichiometric chemical composition to within the error bars, and no interstitial atoms or impurity phases were observed (Supplementary Note 1; Supplementary Figs 1–3; Supplementary Table 1). The specific heat, magnetic susceptibility and resistivity measurements performed on randomly selected FeSe single crystals further demonstrate that our sample is a bulk superconductor without detectable impurities. (Supplementary Note 1; Supplementary Figs 4 and 5).

Neutron-scattering experiments

Our inelastic neutron-scattering measurements were carried out on the ARCS time-of-flight chopper spectrometer at the Spallation Neutron Source of Oak Ridge National Laboratory, USA, 4SEASONS chopper spectrometer at the Japan Proton Accelerator Research Complex and MERLIN chopper spectrometer at the Rutherford Appleton Laboratory, Didcot, UK. The large detector arrays on these instruments allowed us to measure spin excitations over a wide range of energy and momentum. The |Q|-dependent background is subtracted for the data below the aluminium phonon cutoff energy of ∼40 meV (Supplementary Note 2; Supplementary Fig. 7). To facilitate comparison with theory and previous measurements, our data were normalized into absolute units by using the elastic incoherent scattering of a standard vanadium sample. The absolute intensity of the resonance mode is consistent with previous low-energy data12 normalized with acoustic phonons (Supplementary Fig. 6). The incident neutron beam was aligned parallel to the c axis. The wavevector Q at (qx, qy, qz) is defined as (H, K, L)=(qxa/2π, qyb/2π, qzc/2π) in the reciprocal lattice units in the orthorhombic unit cell. In this unit cell, the magnetic wavevectors associated with the stripe and Néel magnetic order are Q=(1, 0) and Q=(1, 1), which correspond to the ordering wavevectors of the parent compounds of the iron pnictides and the cuprates, respectively.

Data availability

All relevant data that support the findings of this study are available from the corresponding author on request.

Additional information

How to cite this article: Wang, Q. et al. Magnetic ground state of FeSe. Nat. Commun. 7:12182 doi: 10.1038/ncomms12182 (2016).