Outcomes of the EMDataResource cryo-EM Ligand Modeling Challenge

Abstract

The EMDataResource Ligand Model Challenge aimed to assess the reliability and reproducibility of modeling ligands bound to protein and protein–nucleic acid complexes in cryogenic electron microscopy (cryo-EM) maps determined at near-atomic (1.9–2.5 Å) resolution. Three published maps were selected as targets: Escherichia coli beta-galactosidase with inhibitor, SARS-CoV-2 virus RNA-dependent RNA polymerase with covalently bound nucleotide analog and SARS-CoV-2 virus ion channel ORF3a with bound lipid. Sixty-one models were submitted from 17 independent research groups, each with supporting workflow details. The quality of submitted ligand models and surrounding atoms were analyzed by visual inspection and quantification of local map quality, model-to-map fit, geometry, energetics and contact scores. A composite rather than a single score was needed to assess macromolecule+ligand model quality. These observations lead us to recommend best practices for assessing cryo-EM structures of liganded macromolecules reported at near-atomic resolution.

Extended data

Extended Data Fig. 1 Growth of cryo-EM structures and novel ligands derived from them.

(a) Cryo-EM maps released into the EM Data Bank (EMDB) archive by year and resolution range (source: www.emdataresource.org) up to the end of 2023. (b) Novel non-polymer ligands included in cryo-EM structures by year of release into the Protein Data Bank (PDB) through 2023. Inset: major categories of novel ligands found in cryo-EM-derived models (through 2021). See Online Methods for details.

Extended Data Fig. 2 Selected submitted ligand models for each of the Challenge targets.

Panels are labeled by team ID and model # (see Table 1), in order of decreasing ligand Q-scores (see Fig. 3, row 1) from top to bottom. The portion of the map corresponding to the ligand is shown as a semi-transparent surface, along with the model of the ligand. Ligand Q-score is the average Q-score of all non-H atoms in the ligand. For each atom, the Q-score is measured by correlation of map density to the expected gaussian function, at points within 2 Å of the atom and closer to the atom than any other non-H atom in the the model¹⁰. Higher-scoring ligand models fit better in the cryo-EM density than lower-scoring models.

Extended Data Fig. 3 Evaluation of ions in submitted models (stereo images).

(a) Target 1 6cvm reference model Mg A2002 (gray sphere) with water ligands (orange spheres), located near the PETG ligand, with density for classic octahedral coordination. Only six of 23 submitted Target 1 models included the Mg²⁺ and all three coordinating waters. Others had either only Mg²⁺, Mg²⁺ plus one or two waters, Mg²⁺ plus waters with zero occupancy, no atoms modeled, or atoms significantly displaced. (b) Some groups placed metal ions with weak justification, as exemplified by the Na⁺ (grey sphere) shown here in model EM005_1 for Target 3.

Extended Data Fig. 4 Q-score rankings for ligand + extended vicinity and for full models.

(a-c) LIVQ10 (Ligand + extended vicinity ≤10 Å) Q-scores (black bars) and full model Q-scores (gray bars) are plotted for each submitted model and each reference model, with order according to ligand + extended vicinity rank.

Extended Data Fig. 5 Alternative Group Ranking by sum of Ligand, Ligand+Environment, Full Model Coordinates-only, Full Model Fit-to-Map composite scores.

(a) Group ranking (left-to-right) according to the sum of four composite z-scores, as described below. Only groups that submitted models for all 3 targets and have rank similar to or better than PDB reference models are shown. (b) Correlation table (n = 61) of scores used to create z-scores and rankings in panel (A) and/or Fig. 4. Group composite scores were calculated per team as follows. For each submitted model, and for each score type, a composite z-score was calculated. For each target (T1, T2, T3), the model submitted by that group with maximum composite z-score was selected for inclusion in the final average score over all targets. Ligand: z = (0.33*z.MogulComposite + 0.33*z.StrainEnergyMM + 0.33*z.Q-ligand). Ligand+environment: z = (0.33*z.Pharmacore + 0.33*z.Probescore + 0.33*z.LIVQ5). Full model coordinates-only: z = (0.25*z.Clash + 0.5*z.CablamConf + 0.25*z.CablamCa). Full model fit-to-map: z = (0.25*z.EMRinger + 0.25*z.Q-Protein + 0.25*z.TEMPySMOC + 0.25*z.PhenixFCS05).

Extended Data Fig. 6 Ligand/Ligand Environment Probescores.

(a) Molprobity Probescore³² distributions for ligands in Targets 1–3 (reference models: red triangles; submitted model scores are plotted as gray circles with following exceptions: Target 1, yellow boxes if PTQ sugar ring position was flipped relative to reference; Target 2, asterisk if F86 was set to half-occupancy; Target 3, blue diamonds if PEE was modeled as head-group+tails). Scores are plotted in horizontal axis lanes with small random vertical shifts to visually separate clustered points. Notably, score distributions have wide spreads independent of noted model features: PTQ sugar orientation, F86 occupancy, or PEE inclusion of tails–although for PEE the score distribution is noticeably broader when the larger and more variable tails are included. (b) T2 density map with reference model in the region of the F86 ligand, showing half-strength density for the remdesivir ligand, implying that only half the molecules have covalently bound inhibitor. Image is reproduced from Figure 6 of reference ³⁸ (open access CC-BY license, no permission required for reuse). (c-e) T2 F86 + pyrophosphate ligand environments for the reference model (PDBid 7BV2), model EM004_2, and model EM008_1, respectively. All-atom contact dots are from Probescore, with all-atom clashes in hot pink and favorable H-bonds and vdW contacts in green and blue. Molecular graphics are shown in KiNG⁸⁸.

Extended Data Table 1 Ligand and Ligand+environment Q-scores for submitted models with highest ligand Q-scores