Introduction

Flat-band systems have been proposed as interesting theoretical models to prove the existence of ferromagnetic ordering with itinerant electrons1,2,3,4. Theoretical developments in such flat-band systems have been made almost in parallel with those in the widely discussed topological insulators (TIs)5,6,7,8. The nontrivial topology of electronic bands in a kagome lattice, one of those flat-band systems, has been extensively studied9,10,11,12,13,14,15,16,17,18.

The experimental quests for topological materials with kagome lattice have also been carried out. Many of such experimental efforts were stimulated by the prediction of Weyl semimetals19,20, including intermetallic compounds involving Co21,22,23,24,25, Fe26,27,28,29, Mn30,31,32, and van-der-Waals compounds33, as well as optical lattices34,35. More recently, the coexistence of superconductivity and nontrivial band topology was reported in a kagome compound36,37,38,39.

When a flat band is partially occupied by electrons, the Coulomb repulsive interactions could become dominant over the electronic kinetic energy. This situation is already realized in two-dimensional electron gases under applied magnetic fields, where flat bands correspond to Landau levels. Fractional quantum Hall (FQH) effects were thus discovered40,41. An exact numerical analysis made an important contribution by demonstrating that quantum fluctuations are essential to stabilize FQH states over charge density wave states42. Once the charge excitation gap is induced at a fractional filling, the property of FQH states is elegantly explained using effective theory43.

Recently, further intriguing proposals were put forward by considering flat bands with nontrivial topology and repulsive interactions, whereby FQH states could be generated without having Landau levels, called fractional Chern insulators (FCIs). These proposals considered single-band models on a kagome lattice44,45, checkerboard lattices46,47,48,49,50, a Haldane model on a honeycomb lattice, and a ruby lattice45, as well as multi-band models on a buckled honeycomb lattice51, a triangular lattice52, and a square lattice for the mercury-telluride TI45. It was later revealed that quantum Hall states realized in flat band systems and those realized under an applied magnetic field are adiabatically connected53. When realized in real materials, FCI states in flat band systems could become a vital element of topological quantum computing54,55. Based on numerical results44,45,46,47,48,49,50,51,52, the possibility of FCI states was suggested in some flat band systems12,15,17. However, the material realization of such FCI states has yet to be demonstrated as theoretical proposals often focus on simple one-band models and other proposed systems have small band gaps.

Motivated by the recent experimental realization of kagome materials, where multiple transition-metal d orbitals are active near the Fermi level, we consider in this work a multiorbital itinerant-electron model on a two-dimensional kagome lattice. With the atomic spin–orbit coupling (SOC), this model shows multiple topological phases, including spin Hall insulators when spin splitting is absent and Chern insulators when spin splitting is induced. Furthermore, this model exhibits flat bands having nonzero Chern number as in a single-band kagome system10. We found that non-local Coulomb interactions induce FCI states when such a flat band is fractionally occupied by electrons. Note that our approach employs the original on-site local source of the SOC, while most of simplified models widely employed in other efforts assume the form of SOC simply based on symmetry considerations. Thus, our work relies on more fundamental foundations. Our model calculation is particularly relevant to CoSn-type intermetallic compounds when a single kagome layer becomes available.

Results

Theoretical model

To begin with, we set up a multi-orbital tight-binding model on a kagome lattice

$${H}_{{{{{{{{\rm{t}}}}}}}}}=-\mathop{\sum}\limits_{\left\langle {{{{{{{\bf{r}}}}}}}}\,{{{{{{{{\bf{r}}}}}}}}}^{\prime}\right\rangle }\mathop{\sum}\limits_{\alpha \beta \sigma }\left({t}_{{{{{{{{\bf{r}}}}}}}}\,{{{{{{{{\bf{r}}}}}}}}}^{\prime}}^{\alpha \beta }{c}_{{{{{{{{\bf{r}}}}}}}}\alpha \sigma }^{{{{\dagger}}} }{c}_{{{{{{{{{\bf{r}}}}}}}}}^{\prime}\beta \sigma }+{{{{{{{\rm{H.c.}}}}}}}}\right),$$
(1)

as schematically shown in Fig. 1. Here, \({c}_{{{{{{{{\bf{r}}}}}}}}\alpha \sigma }^{({{{\dagger}}} )}\) is the annihilation (creation) operator of an electron at site r, orbital α, and with spin σ = ↑ or ↓. As discussed by Meier et al.25, CoSn-type kagome systems have several flat bands with {yz, xz}, {xy, x2 − y2}, or 3z3 − r2 character. We focus on a {yz, xz} subset for simplicity and use α = a for the yz orbital and b for the xz orbital. With this basis, nearest-neighbor hop** intensities \({t}_{{{{{{{{\bf{r}}}}}}}}\,{{{{{{{\bf{r}}}}}}}}^{\prime} }^{\alpha \beta }\) can be parameterized using Slater integrals56. Between site 1 and site 2, \({\hat{t}}_{{{{{{{{\bf{1}}}}}}}}{{{{{{{\bf{2}}}}}}}}}\) is diagonal in orbital indices as \({t}_{{{{{{{{\bf{1}}}}}}}}\,{{{{{{{\bf{2}}}}}}}}}^{aa}={t}_{\delta }\) and \({t}_{{{{{{{{\bf{1}}}}}}}}\,{{{{{{{\bf{2}}}}}}}}}^{bb}={t}_{\pi }\), corresponding to (ddδ) and (ddπ), respectively, by Slater and Koster56. Other components are obtained by rotating the basis a and b as shown in the Methods section. From now on, tπ is used as the unit of energy.

Fig. 1: Schematics of our theoretical model.
figure 1

a Kagome lattice with three sublattices, labeled 1, 2, and 3. The two arrows are lattice translation vectors a1,2. b Local orbitals a = yz and b = xz. Colored ellipsoids indicate regions of electron wave functions, where the sign is positive. c Nearest-neighbor hop** integrals. yz(xz) orbitals between site 1 and site 2 are hybridized via diagonal hop** tδ(π), i.e., δ(π) bonding. Other hop** integrals between site 2 and site 3 and between site 1 and site 3 are obtained via the Slater rule56 as shown in the Methods section.

Full size image

Because yz and xz are written using the eigenfunctions of angular momentum lz = ±1 for l = 2 as \(\left|yz\right\rangle =\frac{{{{{{{{\rm{i}}}}}}}}}{\sqrt{2}}(\left|1\right\rangle +\left|-1\right\rangle )\) and \(\left|xz\right\rangle =-\frac{1}{\sqrt{2}}(\left|1\right\rangle -\left|-1\right\rangle )\), respectively, the SOC \(\lambda \overrightarrow{l}\cdot \overrightarrow{s}\) in the {yz, xz} subset is written as

$${H}_{{{{{{{{\rm{soc}}}}}}}}}=\frac{\lambda }{2}\mathop{\sum}\limits_{{{{{{{{\bf{r}}}}}}}}\,\sigma }\left({{{{{{{\rm{i}}}}}}}}{\sigma }_{\sigma \sigma }^{z}{c}_{{{{{{{{\bf{r}}}}}}}}a\sigma }^{{{{\dagger}}} }{c}_{{{{{{{{\bf{r}}}}}}}}b\sigma }+{{{{{{{\rm{H.c.}}}}}}}}\right),$$
(2)

where \({\hat{\sigma }}^{z}\) is the z component of the Pauli matrices.

As shown in Supplementary Note 1, an effective model for the {xy, x2 − y2} doublet has the same form as the above Ht + Hsoc. By symmetry, there is no hop** matrix between the {yz, xz} doublet and the other orbitals xy, x2 − y2, and 3z2 − r2, but the {xy, x2 − y2} doublet and the 3z2 − r2 singlet could be hybridized. As discussed briefly later, the degeneracy in the {yz, xz} doublet and in the {xy, x2 − y2} doublet could be lifted by a crystal field. Such band splitting is also induced by the difference between tδ and tπ. Furthermore, all d orbitals could in principle be mixed by the SOC. Including these complexities is possible but depends on the material and they usually induce smaller perturbations, therefore, here they are left for future analyses.

Non-interacting band topology

By diagonalizing the single-particle Hamiltonian Ht + Hsoc, one obtains dispersion relations as shown in Fig. 2. In the simplest case, where the hop** matrix \({t}_{{{{{{{{\bf{r}}}}}}}}\,{{{{{{{\bf{r}}}}}}}}^{\prime} }^{\alpha \beta }\) does not distinguish tδ and tπ and the SOC is absent, the dispersion relation is identical to the one for the single-band tight-binding model, consisting of flat bands and graphene-like bands as shown by gray lines in Fig. 2a. Note that each band is fourfold degenerate because of two orbitals and two spins per site. Including SOC does not change the dispersion curve but simply shifts \(\overrightarrow{l}\cdot \overrightarrow{s}=\pm 1/2\) bands (see Supplementary Note 1).

Fig. 2: Dispersion relations of the non-interacting model.
figure 2

Bulk dispersion relations without the spin-orbit coupling (SOC) (a) and with the SOC λ = 0.2 (b). In both cases, energy E is scaled by the π-bond hop** integral tπ. Gray lines in a indicate the dispersion with the δ-bond hop** integral tδ = 1, which realizes the ideal dispersion in a kagome lattice. Blue lines in (a) and red lines in (b) are dispersions with tδ = 0.5. The inset shows the first Brillouin zone with high-symmetry lines used in (a), (b). The Chern number \({{{{{{{{\mathcal{C}}}}}}}}}_{n}\) for the spin up component of each band is also shown in (b). c Dispersion relations with tδ = 0.5 and λ = 0.2 in the ribbon geometry, which is periodic along the a1 direction and contains 20 unit cells along the perpendicular direction. Gapless edge modes are indicated by red and blue lines.

Full size image

Including orbital dependence as tδ ≠ tπ without SOC instead splits the fourfold degeneracy except for two points at the Γ point and two points at the K point. Quite intriguingly, Dirac dispersions emerge from the topmost flat bands as shown as blue lines in Fig. 2a (see Supplementary Note 1 for more discussion). Turning on the SOC further splits such fourfold degeneracy, leading to nontrivial band topology. In this particular example, the spin component along the z axis is conserved giving unique characteristics to this case. As shown in Fig. 2b, the spin up component of each band is characterized by a nonzero Chern number \({{{{{{{{\mathcal{C}}}}}}}}}_{n}\). Because of the time-reversal symmetry, spin down bands have opposite Chern numbers. The topological property is also confirmed by gapless modes in the dispersion relation with the ribbon geometry, as shown in Fig. 2c. Here, there appear one (two) pair of gapless modes between the highest and the second highest (between the second lowest and the third lowest) bands, shown as red (blue) curves, corresponding to the sum of Chern numbers below the gap, −1(−2). The other edge states are invisible because of the overlap with the bulk continuum.

A multi-orbital kagome model thus naturally shows quasi flat bands with nontrivial topology. However, close inspection revealed that, with tδ = 0.5 and λ = 0.2, the minimum of the highest band at the K point is slightly lower than the maximum of the second highest band at the Γ point. Thus, instead of a TI, a topological semimetal is realized when the Fermi level is located between the highest band and the second highest band. In fact, there are ways to make the gap positive. Here, we consider second-neighbor hop** matrices \({\hat{t}}_{{{{{{{{\bf{r}}}}}}}}\,{{{{{{{\bf{r}}}}}}}}^{\prime} }^{(2)}\). As explained in Supplementary Note 1, these are also parametrized by π-bonding (ddπ) and δ-bonding (ddδ), \({t}_{\pi }^{(2)}\) and \({t}_{\delta }^{(2)}\), respectively. For simplicity, we fix the ratio between tπ and \({t}_{\pi }^{(2)}\) and between tδ and \({t}_{\delta }^{(2)}\) as \({t}_{\pi }^{(2)}/{t}_{\pi }={t}_{\delta }^{(2)}/{t}_{\delta }={r}_{2}\), and analyze the sign and magnitude of the band gap Δgap between the highest band and the second highest band, as well as the flatness of the highest band defined by \({{\Delta }}\varepsilon \equiv {\varepsilon }_{1,\max }-{\varepsilon }_{1,\min }\).

Figure 3a plots Δε as a function of tδ and r2 with λ = 0.2. As mentioned previously, the perfectly flat band with Δε = 0 is realized at tδ = 1 and r2 = 0, but band gap Δgap is zero. The flatness is immediately modified by reducing tδ from 1. As indicated by an open square in the plot, tδ = 0.5 and r2 = 0 gives Δε ~ 0.88 and negative band gap Δgap ~ −0.027. Nonzero r2 controls the relative energy between the zone center and the zone boundary. In particular, negative r2 pushes up the energy at the K point, hereby the flatness is recovered. Naturally, the flatness and the positive gap are correlated as indicated by red loops in the second and forth quadrants because the separation between the highest band and the second highest band is fixed by the SOC strength. As indicated by a filled circle, tδ = 0.5 and r2 = −0.2 gives Δε ~ 0.22 and positive band gap Δgap ~ 0.17. Corresponding dispersion relation is shown in Fig. 3b. The Chern numbers remain unchanged by this r2.

Fig. 3: Control of the band flatness.
figure 3

a Color map of the flatness Δε of the highest band as a function of tδ and the ratio between the nearest-neighbor and the second-neighbor hop** r2 with λ = 0.2. Open square (filled circle) locates tδ = 0.5 with r2 = 0 (−0.2). Red closed loops show the areas where the band gap is positive Δgap > 0. b Bulk band structure with tδ = 0.5, r2 = −0.2, and λ = 0.2. Band-dependent Chern number is also shown.

Full size image

Many-body effects

Having established the topological properties at the single-particle level, we turn our attention to many-body effects focusing on the highest-energy flat band. A unique property of the current model is that the topmost quasi flat band has Chern number \(| {{{{{{{\mathcal{C}}}}}}}}| =1\). Thus, a large spin polarization can be induced by many-body interactions57 or by a small magnetic field. Further intriguing possibilities are FCI states when a topological flat band has a fractional filling and the insulating gap is induced by correlation effects44,46,47,48,49,50,21,22,23,24,26,27,28,30,31,32,36,37,38,39, reducing the thickness of such materials down to a few unit cells, or growing thin films of such materials and tuning the Fermi level to a topological flat band by chemical substitution or gating, might be a promising route to observe the phenomena predicted here. The sign and the magnitude of the parameter r2 could depend on details of the material, such as the species of ligand ions, and might be further controlled by compressive or tensile strain. First principles calculations would help to construct realistic material-dependent models12,15,17. It is anticipated that the separation between {yz, xz}, {xy, x2 − y2}, or 3z3 − r2 subsets will be enhanced by reducing the film thickness compared with that in the bulk so that one can focus on one of the subsets only. In addition to a kagome lattice, topological flat bands appear in dice and Lieb lattices59,60,61. Study of FCI states in such lattice geometries and material search is another important direction.

To summarize, we have demonstrated the close interplay between the spatial frustration and the orbital degree of freedom in a kagome lattice. With the relativistic SOC, such an interplay not only affects the band dispersion, but also induces nontrivial topology. Specifically, we showed that the original flat bands in a kagome lattice become dispersive and topologically nontrivial. When such topological bands are fractionally occupied by electrons, many-body interactions drive further intriguing phenomena, i.e., fractional Chern insulating states. Our work may bridge the gap between idealized theoretical studies and real materials.

Methods

Non-interacting {y z, x z} model

Here we deduce the hop** matrices of the {yz, xz} model in the Slater–Koster approximation.

For nearest-neighbor bonds, in addition to the diagonal matrix \({\hat{t}}_{{{{{{{{\bf{1}}}}}}}}{{{{{{{\bf{2}}}}}}}}}\) presented in the main text, we have

$${\hat{t}}_{{{{{{{{\bf{13}}}}}}}}}=\, \frac{1}{4}\left[\begin{array}{cc}3{t}_{\pi }+{t}_{\delta }&\sqrt{3}({t}_{\pi }-{t}_{\delta })\\ \sqrt{3}({t}_{\pi }-{t}_{\delta })&{t}_{\pi }+3{t}_{\delta }\end{array}\right],\\ {\hat{t}}_{{{{{{{{\bf{23}}}}}}}}}=\, \frac{1}{4}\left[\begin{array}{cc}3{t}_{\pi }+{t}_{\delta }&-\sqrt{3}({t}_{\pi }-{t}_{\delta })\\ -\sqrt{3}({t}_{\pi }-{t}_{\delta })&{t}_{\pi }+3{t}_{\delta }\end{array}\right].$$
(3)

Similarly, second neighbor hop** matrices can be written as

$${\hat{t}}_{{{{{{{{\bf{1}}}}}}}}\,{{{{{{{\bf{2}}}}}}}}}^{(2)}=\, \left[\begin{array}{cc}{t}_{\pi }^{(2)}&0\\ 0&{t}_{\delta }^{(2)}\end{array}\right],\\ {\hat{t}}_{{{{{{{{\bf{1\,3}}}}}}}}}^{(2)}=\, \frac{1}{4}\left[\begin{array}{cc}{t}_{\pi }^{(2)}+3{t}_{\delta }^{(2)}&-\sqrt{3}({t}_{\pi }^{(2)}-{t}_{\delta }^{(2)})\\ -\sqrt{3}({t}_{\pi }^{(2)}-{t}_{\delta }^{(2)})&3{t}_{\pi }^{(2)}+{t}_{\delta }^{(2)}\end{array}\right],\\ {\hat{t}}_{{{{{{{{\bf{2\,3}}}}}}}}}^{(2)}=\, \frac{1}{4}\left[\begin{array}{cc}{t}_{\pi }^{(2)}+3{t}_{\delta }^{(2)}&\sqrt{3}({t}_{\pi }^{(2)}-{t}_{\delta }^{(2)})\\ \sqrt{3}({t}_{\pi }^{(2)}-{t}_{\delta }^{(2)})&3{t}_{\pi }^{(2)}+{t}_{\delta }^{(2)}\end{array}\right],$$
(4)

where subscript (2) is introduced to highlight the difference from the nearest-neighbor bonds. These are schematically shown in Fig. 6. \({t}_{\pi }^{(2)}\) and \({t}_{\delta }^{(2)}\) correspond to (ddπ) and (ddδ), respectively, by Slater and Koster56.

Fig. 6: Second neighbor hop** matrices.
figure 6

yz(xz) orbitals between site 1 and site 2 are hybridized via diagonal hop** \({t}_{\pi (\delta )}^{(2)}\), i.e., π(δ) bonding. Other hop** integrals between site 1 and site 3 and between site 2 and site 3 are given by \({\hat{t}}_{{{{{{{{\bf{1\,3}}}}}}}}}^{(2)}\) and \({\hat{t}}_{{{{{{{{\bf{2\,3}}}}}}}}}^{(2)}\), respectively, obtained via the Slater rule56.

Full size image

Non-interacting Berry curvature

The band-dependent Berry curvature of non-interacting electrons is given as a function of momentum k as

$${{{\Omega }}}_{n{{{{{{{\bf{k}}}}}}}}}={{{{{{{\rm{i}}}}}}}}\mathop{\sum}\limits_{m(\ne n)}\frac{\left\langle n\right|{\hat{v}}_{x{{{{{{{\bf{k}}}}}}}}}\left|m\right\rangle \left\langle m\right|{\hat{v}}_{y{{{{{{{\bf{k}}}}}}}}}\left|n\right\rangle -({\hat{v}}_{x{{{{{{{\bf{k}}}}}}}}}\leftrightarrow {\hat{v}}_{y{{{{{{{\bf{k}}}}}}}}})}{{({\varepsilon }_{m{{{{{{{\bf{k}}}}}}}}}-{\varepsilon }_{n{{{{{{{\bf{k}}}}}}}}})}^{2}},$$
(5)

where, using the Hamiltonian matrix in momentum space \({\hat{H}}_{{{{{{{{\bf{k}}}}}}}}}\), \({\hat{v}}_{\eta {{{{{{{\bf{k}}}}}}}}}\) is given by \({\hat{v}}_{\eta {{{{{{{\bf{k}}}}}}}}}=\partial {\hat{H}}_{{{{{{{{\bf{k}}}}}}}}}/\partial {k}_{\eta }\). With this Berry curvature, the band dependent Chern number \({{{{{{{{\mathcal{C}}}}}}}}}_{n}\) is given by

$${{{{{{{{\mathcal{C}}}}}}}}}_{n}=\frac{1}{2\pi }{\int}_{{{{{{{{\rm{BZ}}}}}}}}}{{{{{\rm{d}}}}}}^{2}k\,{{{\Omega }}}_{n{{{{{{{\bf{k}}}}}}}}},$$
(6)

where the momentum integral is taken in the first Brillouin zone.

Many-body Chern number

The many-body Chern number is computed by introducing a twist boundary condition to a single-particle wave function as \(\psi ({{{{{{{\bf{r}}}}}}}}+{N}_{j}{{{{{{{{\bf{a}}}}}}}}}_{j})={{{{{\rm{e}}}}}}^{{{{{{{{\rm{i}}}}}}}}{\theta }_{j}}\psi ({{{{{{{\bf{r}}}}}}}})\), where Nj=1,2 are the numbers of unit cells along lattice translation vectors aj=1,2, with phase factors θj=1,2. This corresponds to inserting magnetic fluxes. When one flux quantum is inserted, θj changes from 0 to 2π and discretized momentum k moves from its original position to its neighbor along the bj direction with the momentum shift given by Δk = bj/Nj.

Many-body Chern number of the ground state (k1, k2) is computed via \({{{{{{{{\mathcal{C}}}}}}}}}_{({k}_{1},{k}_{2})}=\frac{1}{2\pi }\int\nolimits_{0}^{2\pi }{{{{{\rm{d}}}}}}{\theta }_{1}\int\nolimits_{0}^{2\pi }{{{{{\rm{d}}}}}}{\theta }_{2}{F}_{({k}_{1},{k}_{2})}({\theta }_{1},{\theta }_{2})\)58 where F(θ1, θ2) is the Berry curvature given by

$${F}_{({k}_{1},{k}_{2})}({\theta }_{1},{\theta }_{2})={{{{{{{\rm{Im}}}}}}}}\left\{\left\langle \frac{\partial {{{\Phi }}}_{({k}_{1},{k}_{2})}}{\partial {\theta }_{2}}\left|\frac{\partial {{{\Phi }}}_{({k}_{1},{k}_{2})}}{\partial {\theta }_{1}}\right.\right\rangle -({\theta }_{1}\leftrightarrow {\theta }_{2})\right\}.$$
(7)

Here, \(|{{{\Phi }}}_{({k}_{1},{k}_{2})}\rangle\) is the many-body wave function constructed using single-particle wave functions with a twist boundary condition ψ(r) after the Fourier transformation to momentum space. The momentum index (k1, k2) will be omitted in the following discussion for simplicity.

Partial derivative of a wave function with respect to θj is approximated by a finite difference as \(\left|\partial {{\Phi }}/\partial \theta \right\rangle \approx \frac{1}{| {{\Delta }}{{{{{{{\boldsymbol{\theta }}}}}}}}| }[\left|{{\Phi }}({{{{{{{\boldsymbol{\theta }}}}}}}}+{{\Delta }}{{{{{{{\boldsymbol{\theta }}}}}}}})\right\rangle -\left|{{\Phi }}({{{{{{{\boldsymbol{\theta }}}}}}}})\right\rangle ]\). Here, the vector notation is used for θ = (θ1, θ2), and Δθ = (Δθ1, 0) or (0, Δθ2). Then, it is required to compute a product of two wave functions as \(\left\langle {{\Phi }}({{{{{{{\boldsymbol{\theta }}}}}}}})| {{\Phi }}({{{{{{{\boldsymbol{\theta }}}}}}}}^{\prime} )\right\rangle\) with \({{{{{{{\boldsymbol{\theta }}}}}}}}\,\ne\, {{{{{{{\boldsymbol{\theta }}}}}}}}^{\prime}\). Because we are using a multiorbital model projected onto the flat band, special care is needed, as detailed in Supplementary Note 2.