What did we find?

In a previous post I talked about our plan to perform MD simulations of Mac1 crystals under different conditions. We had just started an MD simulation of Mac1 in complex with ADP-ribose (ADPr), and were waiting for it to finish. We weren’t sure how different it would be from the initial simulations of Mac1 without ADPr, in which a CHES buffer occupies the ADPr binding site.

Now we know more. Visual comparisons (above images) show that the rich anisotropic diffuse diffuse features in the maps with CHES buffer (left) vs. ADPr (center) in the binding pocket have similar patterns of peaks and troughs that differ in detail. Subtracting them reveals the differences more clearly (right). Quantitative analysis shows that the variations in the difference map are comparable in strength to the intensities in either individual map. This result is encouraging as it indicates that such differences might be observed in experiments.

It’s a trap!

Along the way we encountered a common pitfall in making these kinds of comparisons: due to an indexing ambiguity in the P43 space group, the structures of Mac1 used for simulations with and without ADPr were solved using different definitions of the lattice vectors, with the h and k axis swapped, and the l axis reversed (compare PDB IDs 7TX0 and 7TX3). The diffUSE modeling team worked out how to make the simulated diffuse maps consistent at our recent all hands meeting, enabling us to perform a controlled comparison of the simulations.

What next?

The next step is to compare both of these simulations with data recently collected at CHESS (see logbook), in one of a series of diffUSE beam times that are expected to yield a large number of datasets. These runs already have revealed that diffuse data are reproducible between CHESS and ALS beamlines. Data from Mac1 +/- ADPr are now in the processing pipeline; we’re eager to see how Mac1 diffuse scattering changes upon ligand binding, and whether MD simulations can help explain what we see.

Why This Retreat Mattered

In late January, the DiffUSE Project team gathered in person for our first progress meeting at Astera’s headquarters in Emeryville, California. Since our October online meeting, every team has made substantial progress.

The retreat brought together team members working on data collection, data processing, molecular dynamics simulations, machine learning modeling, infrastructure, and open science to assess progress against our six-month goals and chart the path forward.

Perhaps the most exciting development is a deceptively simple one: diffuse scattering data collected at CHESS (Cornell) and ALS (Berkeley) are reproducible. This cross-country validation marks a critical step toward making diffuse scattering a routine tool for structural biology.

What Have We Accomplished Since October?

Data Collection

Kara Zielinski (Fraser Lab, UCSF) reported on an intensive fall data collection campaign:

• 9 beamtimes since project inception across two synchrotrons (CHESS and ALS)
• 18 participants contributed to data collection
• 7 protein systems: Mac1, NrdE, Lysozyme, DNA fibers, ATCase, Insulin, and Huwe1
• 129 “good” datasets collected (no data collection errors)

The team systematically explored experimental perturbations:

• Temperature: Data collected at 100K (cryo), 220-275K (intermediate), and 310-315K (elevated), though sample handling for intermediate temperatures requires further optimization
• Ligands: Mac1 + ADPr (11 datasets from CHESS and ALS combined) and Mac1 + small molecule “opener” (6 datasets from ALS)
• Radiation damage mitigation: Vector scans implemented at CHESS to spread dose across radiation-sensitive samples like NrdE and ATCase

Beamline-specific improvements included:

• ALS: Explored dose dependence, wavelength effects, and exposure time optimization; addressed collimator ring scatter issues at 14 keV
• CHESS: Continued X-ray aperture optimization for background reduction

Slide shared at the DiffUSE project’s retreat showcasing crystals of initially tested protein systems.

Data Processing

Steve Meisburger (Cornell/CHESS) presented major advances in data processing tools and a landmark reproducibility result.

mdx2 is an open-source software package for processing and analyzing diffuse X-ray scattering data. Development has accelerated with a new team (Steve Meisburger, Justin Biel, Joseph Lee) and modern development practices, including version control, issue tracking, and code review. Version 10.3 was released in December 2025 with:

• Containerized deployment via conda install -c conda-forge mdx2
• Jupyter Lab environment integration
• Live processing capability on Voltage Park during beam times

Reference datasets from CHESS now span multiple systems (Mac1, NrdE, DNA, ATCase, Insulin) with systematic tracking through integration, merging, and fine map generation stages.

The headline result: Diffuse scattering is reproducible between CHESS and ALS. Side-by-side comparisons of Mac1 diffuse maps from both beamlines show consistent features, validating that the signal is robust across different detector systems, beam profiles, and facilities. This East-meets-West reproducibility is foundational for any future multi-site data collection campaigns.

Slides shared at the DiffUSE project’s retreat showcasing reproducibility across beamlines.

Additional findings from DNA crystal analysis revealed that correlated disorder differs between room temperature and 100K conditions, even when the static structures appear similar—and that diffuse signal extends beyond the Bragg resolution limit, suggesting untapped information content.

A Galaxy platform prototype was demonstrated, pointing toward a vision of “Cryosparc for diffuse,” making diffuse data processing accessible through a GUI with integrated workflows and interactive visualizations.

Molecular Dynamics Simulations

Mike Wall presented substantial progress on crystallographic MD simulations.

Apo Mac1 baseline results show exceptional agreement between simulation and experiment:

• Total correlation coefficient: CC = 0.96
• Anisotropic correlation coefficient: CC = 0.56
• Simulation: 2×2×2 supercell with OPC3 waters (279,004 atoms), neutron crystal structure 7TX3, 1100 ns unrestrained trajectory

MD optimization methods are advancing on two fronts:

• Enrichment: Selectively removing MD frames to increase diffuse correlation
• Reweighting: Using JAX to optimize frame weights via differentiable Pearson CC maximization. Initial test on experimental diffuse data achieved CC = 0.97 with 47,150 reflections to 3.5 Å resolution (work by Karson Chrispens, documented in a DiffUSE blog post)

Ligand perturbations are now being simulated: Mac1 + ADPr shows distinct diffuse patterns compared to baseline Mac1, with protonation state variations (ASP157 → ASH157) under investigation.

Second system: Dihydrofolate reductase (DHFR, PDB: 7FPV) is being developed as a generalization target, expanding beyond the Mac1 test case.

Simulated diffraction capabilities using nanoBragg (James Holton) enable validation of data processing pipelines—simulated diffuse intensity can be extracted using mdx2, closing the loop between simulation and experiment.

Slides shared at the DiffUSE project’s retreat showcasing simulation results of a new protein system, Dihydrofolate reductase (DHFR, PDB: 7FPV).

Machine Learning Modeling

Marcus Collins presented the ML modeling roadmap focused on using experimental data to reveal hidden protein conformations.

Key insight: Current ML structure predictors (AlphaFold3-like models, including Boltz-2, Protenix, RF3) do not reliably predict alternate conformations (altlocs) even with multiple random seeds, indicating they have not learned about underlying ensembles. This gap motivates developing density-guided ensemble generation (Sampleworks).

Density guidance approach: The team is implementing training-free guidance from experimental density maps (2Fo-Fc), using the difference between experimental and calculated maps to steer diffusion model sampling toward conformations consistent with crystallographic data. Early results are promising but mixed: Boltz-2 with density guidance can capture both altlocs in some test cases like PTP1B (6B8X), though performance varies across systems.

Sampleworks pipeline is being built as a plug-and-play guidance framework to use different structure prediction models, experimental data, and guidance strategies.

• Model wrappers implemented for RF3, Protenix, Boltz-1, and Boltz-2 (MD and X-ray modes)
• Initial test set of ~50 structures from PDB prepared with altlocs; electron density maps being generated
• Evaluation metrics: RSCC, LDDT, clash scores, backbone and sidechain geometry

Water modeling emerges as a critical challenge for advancing to reciprocal space. Our first attempt is to improve the modeling of explicit solvent. Current models achieve ~0.3 precision/recall at 0.5 Å—insufficient for improving Rwork/Rfree. The team is exploring flow-matching approaches and evaluating whether a single unified model or separate protein/water models will be more effective. Ordered waters coupled to protein altlocs are particularly important targets.

Infrastructure and Publishing

Justin Biel presented the computational infrastructure supporting DiffUSE, built around a three-pillar model: Data, Compute, and Publishing.

Compute Infrastructure uses Voltage Park as the backbone:

• H100 SXM5 GPUs available via bare metal (8× GPU configurations)
• Two usage patterns supported:
  • Workspaces: Interactive environments for experimental work, debugging, and visualization
  • Workflows: Hardened, scalable pipelines for production analysis
• The DiffUSE web app now provides resource checkout, visibility into running resources, and SSH/Jupyter access
• Custom container management enables workspace pausing and environment customization
• Workflow orchestration via Prefect and Docker

Data Infrastructure centers on the DiffUSE web app:

• Storage: Core Backblaze storage (S3-compatible) with OSN bucket integration for beamline data
• Access: Automatic mounting to Voltage Park resources, plus web app download, CLI, Python SDK, and API
• Metadata: Experiments have artifacts, optional markdown content (like logbook entries), relationships to other experiments, and tags
• Automation: Beam-trip data automatically triggers experiment registration; dataset files populate metadata fields
• Governance: Standards compliance checking, staging-to-public workflows, DOI attachment decisions

Publishing workflow discussions focused on:

• When to stage data privately vs. make everything open immediately
• When to attach DOIs (content should be largely immutable)
• External database destinations: SBGrid Databank, PDB, Zenodo

Open Science

Prachee Avasthi (Head of Open Science, Astera) led a discussion on publishing expectations and open science practices. Discussion explored barriers to sharing, evidence of downstream reuse, orphan artifacts without ideal homes, and prioritization of unaddressed data sharing issues.

Reflections on Our Distributed Model

This retreat underscored how the diffUSE’s distributed structure works. By embedding team members across institutions (Cornell, UCSF, Berkeley Lab, and beyond) we maintain direct access to beamlines, computational expertise, and scientific communities that would be impossible to replicate in a single location. The “Diffuse East ≈ Diffuse West” result is itself a product of this model: data collected by different teams at facilities 2,500 miles apart, processed with shared tools, yielding consistent results. Our infrastructure investments (the DiffUSE web app, Voltage Park compute, standardized containerized environments) bridge the geographic gaps, allowing a scientist at Cornell to spin up the same analysis environment as a colleague in California.

The in-person retreat revealed how much asynchronous collaboration had already accomplished, sessions focused on integration and next steps rather than catching people up. Open science practices (shared logbooks, blog posts, open repositories) keep everyone aligned between meetings. The challenge ahead is scaling this approach: as we add systems, datasets, and collaborators, maintaining the coherence that makes distributed work effective will require continued investment in documentation, automation, and the human connections that make a dispersed team feel like one group working toward a shared goal.

The science described here represents the output of a significant and coordinated resource investment. Since DiffUSE’s start in July, Astera has committed $3.2M to stand up the project: $2.63M in research grants distributed directly to our partner labs at Fraser Lab/LBL, Ando Lab/CHESS, and Wankowicz Lab, $567K in Astera personnel and contractor support, and $30K in computational infrastructure. On top of this, CHESS contributed an estimated $700K in beamtime, bringing the total resource investment to roughly $3.9M. Looking ahead, an additional $2.4M is projected for 2026 as the project scales toward its core scientific goals.

A screenshot from our data management infrastructure, demonstrated at the retreat. This is in active development with Prophet Town and Voltage Park.

Mike Wall presents progress on MD optimization of diffuse scattering to a full house at the Astera Institute.

What’s Next?

Data Collection (3-month goals)

• Collect data on additional systems; collect lysozyme at ALS
• Optimize sample handling for intermediate temperatures (oil-based approaches)
• Explore serial crystallography approaches (chip types, small wedges, crystal size variation)
• Continue investigating cryo options (traditional, NANUQ, high-pressure cryocooling)

Data Processing (2026 goals)

• Improve mdx2 performance (~2× speedup)
• Implement GOODVIBES and DISCOBALL in Python (JAX)
• Fully explore serial crystallography processing
• Deploy on Ando lab Galaxy server; add mdx2 tools
• Develop “Cryosparc for diffuse” project roadmap

MD Simulations

• Continue model/data comparisons and refine MD models (protonation states, parameterization)
• Expand to new systems and additional ligand/mutation perturbations
• Explore how MD optimizations can support other DiffUSE activities (ML modeling, diffraction image simulation, data processing validation)

ML Modeling

• Scale up Sampleworks evaluation across initial test set
• Improve water prediction models (retrain SuperWater with better data, explore flow matching vs. diffusion)
• Quantify water model precision requirements by systematically perturbing well-supported waters
• Progress toward reciprocal space/Bragg peak guidance, ultimately targeting diffuse data guidance

Infrastructure

• Finalize containerized workspace management with pause/resume capability
• Expand workflow orchestration options
• Refine data governance workflows for staging → public → external database publication

Open Science

• Address identified barriers to sharing
• Establish timelines for DOI attachment and external database deposition
• Continue documentation through blog posts and logbooks

Special thanks to Astera for hosting the retreat in Emeryville.

Glossary

Diffuse Scattering in the Cloud

While out on a walk, as I looked up at the sky, a certain cloud formation (above left) reminded me of the l = 0 slice through the MD simulation of Mac1 diffuse scattering (above right). That got me thinking about the next steps for the diffUSE MD simulations (which are, of course, being performed using cloud computing resources).

As described in the Quarterly All Hands Meeting post, we recently shared our short-term plans for the various components of the diffUSE project. We’ve already performed baseline comparisons of crystalline MD simulations of Nsp3 macrodomain (Mac1) to diffuse scattering data (see the 3-2-1 Contact post). Now we want to improve the models and see what happens in the simulations when we make changes. With this in mind, we’re planning to: (1) improve the current MD model of Mac1; (2) simulate Mac1 crystals under different conditions; and (3) develop a model of a new system.

Thinking about (2), I contacted James Fraser to chat about what to do for the next MD simulations of Mac1. We decided to look at Mac1 in complex with ADP-ribose. This choice is timely, as Kara Zielinski just collected diffUSE diffraction data from crystals of this complex at the Cornell High-Energy Synchrotron Source (CHESS), during a recent trip from UCSF to Nozomi Ando’s lab at the Cornell.

What will the MD simulation of diffuse scattering from crystals of Mac1 in complex with ADPr look like? Probably a lot like the ones we’ve done already, with some small changes. We’re planning to analyze the differences and find out what happens to the dynamics when different ligands bind. But we don’t really know yet what we’ll see. These moments of suspense are very common in science, but they’re absent from the stories we usually tell in the literature. The open science model we’re using on the diffUSE project enables us to document these periods of uncertainty as a part of the public narrative of the project. It feels kind of liberating.

We haven’t gathered all together since our June kick-off meeting, so in Mid October, we met (online) with all members of the diffUSE project to discuss our overall goals for each project team, progress made over the first few months, and goals for the next three months. We emphasized how the different pieces of the project integrate to build methods, data, models, and encodings so the community can routinely use diffuse scattering in basic biology and drug discovery.

What have we completed in the first few months?

Data collection:

• We have collected ambient-temperature datasets from CHESS for lysozyme, macrodomain, NrdE, and DNA fibers.
• We have collected data at ALS on Mac1 and Huwe1 using humidity boxes and watershed sleeves with controlled transmission and beam size.
• We have played around with temperature modulation used to probe dose dependence and mosaicity effects with samples shipped] from UCSF to Cornell
• We implemented standardized background frames, uniform sleeve lengths, and precise humidity control to enhance map quality and cross-beamline comparability with an eye toward future multi-site data collection campaigns.
• We have documented all collection procedures on the diffUSE website logbooks.

Data processing:

• Developing xia2.multiplex for automated data merging, mdx2 for data extraction, and comprehensive data quality control (QC) workflows.

• Building graphical user interfaces (GUIs) for mdx2 to improve usability and accessibility.

• Identifying and resolving bugs that arise when multiple users concurrently process the same datasets.

• Implementing differentiable refinement by treating molecular dynamics (MD) frame weights as trainable parameters in a Pearson correlation–based objective function.

Machine Learning Modeling:

• Developing pipeline scaffolds to integrate experimental structural data directly into generative model training and evaluation.
• Creating quantitative metrics for assessing and benchmarking ensemble data.
• Building a generative water model that learns to predict water molecule positions from protein structure, designed for future integration into broader generative modeling frameworks.

Simulations:

• We simulated a Mac1 2×2×2 supercell with OPC3 waters and 279,004 atoms reaches 150 ns per day on Voltage Park. With refined masking and resampling, total CC is 0.96 and anisotropic CC is 0.56 on the H8 dataset, which sets a clear target for larger supercells and ligand or mutant comparisons.
• Taylor completed his rotation developing a simulator.

Encoding:

• We continue to advocate for conformational and compositional heterogeneity-encoding strategies.
• We have developed a script to translate between encodings for multiconformer and ensemble representations.
• We are working on developing a standalone script and a COOT integration script for conformational heterogeneity.

Infrastructure and Open Science:

• We have access to Voltage Park compute and S3 storage via the command line, which will make sharing maps and models easier.
• 16 blog posts and 6 beamtime logbooks to date!

What is up for the next 3 months?

Data Collection: Catalog all existing data, fill gaps, complete background series at ALS, finalize hardened collection procedures that travel across LBL and CHESS, and post collection reports on the site. Developing shared checklists to coordinate and standardize future data-collection cycles.

Data Processing: Converge on a single, documented workflow, generate preliminary maps for all ALS and CHESS datasets, produce fine maps for GOODVIBES and DISCOBALL, stand up a CHESS 2026-1 pipeline, and publish processing reports on the site.

Modeling: Ship an initial pipeline that accepts maps for guided sampling.

Encoding: Land final working group approval, publish the schema and examples, and connect the web app to our catalog so processed maps, models, and metadata are searchable and shareable.

Infrastructure and sharing science: More blog posts!

We will meet in the Bay Area in January and report back more after that.

Overview

This post describes the first systematic comparisons between molecular dynamics (MD)–derived diffuse scattering and experimental measurements on the diffUSE project. These early analyses establish a baseline for evaluating how well MD simulations reproduce the observed isotropic and anisotropic features of the data. They also suggest possible improvements in the way we compare MD simulations to diffuse data.

Initial Comparisons

Data from the first diffUSE experiments were used for comparisons (see Getting our feet wet post). The H8 dataset was used, which was obtained using a medium radiation dose (see preliminary analysis of the diffuse scattering).

The H8 dataset was originally sampled on a grid using 2×2×4 points per integer hkl value. To align with the 2x2x2 supercell MD simulation output, it was resampled onto a 2×2×2 grid for direct comparison.

This initial processing yielded a correlation with the MD simulation of 0.88 for the total diffuse intensity, and a correlation of 0.32 for just the anisotropic component.

Refinement and Improvements

The MD simulations were performed using 2x2x2 supercell, which is too small to include contributions from long-range correlations in the lattice. We therefore wondered whether the agreement of the MD with the diffuse data might be diminished in the immediate neighborhood of the Bragg peak, which is associated with lattice vibrations (GOODVIBES is specifically designed to accurately model this part of the signal). We also noticed certain outlier intensity values in the data (specifically, negative values), and wished to avoid using those in comparisons.

Another round of data processing was performed considering these ideas. First, Steve re-processed the diffraction images, sampling more finely onto a 4x4x4 grid. The intensities at integer hkl, correspinding to the immediate neighborhood of the Bragg peaks, were then masked out prior to downsampling to 2x2x2. Negative intensities were additionally masked out.

The revised processing improved the agreement with the MD substantially:

• Total correlation: 0.96
• Anisotropic correlation: 0.56

Outlook

We now have our first assessment of the agreement of MD simulations with diffuse data collected on the diffUSE project. This is our baseline for improving the MD models. We also identified specific regions of the diffuse map – close to the Bragg peaks – where the MD might be currently lacking. These regions of the model might be improved by using a larger supercell for the MD.

This post was initially drafted in ChatGPT based on a Slack exchange between Steve Meisburger and Michael Wall, and was rewritten and posted by Michael Wall on October 20, 2025.

Our structural biology techniques capture an enormous amount of conformational heterogeneity that is often lost in the transition from experimental data to deposited models. Part of this loss stems from a lack of sufficiently sophisticated algorithmic methods, which is an active area of development in this project and elsewhere. Still, an equally important factor is how we choose to encode structural heterogeneity in the models themselves.

In the majority of structures deposited in the Protein Data Bank (PDB), conformational heterogeneity is represented only in a harmonic sense, through atomic displacement parameters (B-factors) or translation–libration–screw (TLS) parameters. These parameters can be incorporated into a single structural model, describing the amplitude and anisotropy of atomic fluctuations around a mean position. However, they do not encode anharmonic or discrete conformational variability. To capture non-harmonic conformational heterogeneity, models have emerged that explicitly include multiple atomic coordinate sets. Two dominant strategies have emerged from X-ray crystallography and cryo-EM: multiconformer models and multi-model (ensemble) models[1].

A multiconformer model represents conformational diversity locally, without duplicating the entire macromolecule. When a region of the electron density is well described by a single conformation, only one set of coordinates with appropriate B-factors is modeled. When the density indicates discrete alternative conformations, such as side chain rotamers or backbone flips, the relevant atoms are copied and assigned alternate location (ALTLOC) identifiers in the PDB file. We have previously demonstrated that this modeling approach can yield substantial improvements in fitting to experimental data and also reduce geometric distortions and eliminate many rotamer outliers[2].

Multi-model approaches model heterogeneity by encoding multiple complete copies of the system, which can sometimes more effectively capture structural motions like backbone shifts. However, ensembles containing tens to hundreds of models can lead to a high parameter-to-data ratio. The most common ensemble models used in Bragg peak analysis today are time-averaged ensembles, generated by molecular dynamics simulations. These ensembles are restrained by time-averaged X-ray structure factors to produce a large number of models, often hundreds, each representing a snapshot from a single trajectory [3]. Further, crystalline MD simulations are currently the best model to describe the diffuse data[4].

Converting between the two representations is a non-trivial task, as the generation and data contained in the two model types are unique. However, there are many times when transferring between the two representations may be needed. For example, the primary methods we use to model and represent diffuse data are molecular dynamics, which generate ensemble models. However, as described by members of this project and others, this MD data has a poor correspondence with the Bragg peak data. However, currently, there are no approaches to further refine this MD model against Bragg peak data. One way we can represent and refine this against Bragg peaks is with multiconformer models, which are compatible with traditional refinement software and allow for manual manipulation.

Converting multiconformer to ensemble models

In general, there is an exponential number of ways to combine single-residue conformations. For this reason, enumerating all combinations becomes infeasible as soon as structures contain a higher number of residues with AltLoc conformations. Previously, Gutermuth et al proposed an algorithm to convert multiconformer models into ensemble models (AltLocEnumerator)[5]. This fast branch-and-bound algorithm to generate valid alternative protein structure conformations is described through AltLoc annotations. The algorithm searches for compatible residue conformations, maximizing the probabilities of conformational states by scoring the AltLoc occupancy values. We aimed to convert a qFit multiconformer PDB into an ensemble structure (PDB: 5iu1). While we attempted multiple options, including enumerating all, optimizing the occupancy score, and changing the number of models, all resulted in multiple models that were almost always overlapping (distribution of all heavy atom RMSD shown below).

We then decided to make five different models (as qFit models make up to 5 alt locs per residue) using the option ‘atllocid’, which provided us with models that were separated with more realistic RMSD (distribution of all heavy atom RMSD shown below).

AltLocEnumerator –file 5i1u_final_qFit.pdb –altlocid A

This command needs to be repeated for each altloc ID, and then the models should be concatendated with MODEL/ENDMODEL lines. Of note, this algorithm is not open source but available to academics for free.

Converting ensemble models to multiconformer

There was no existing tool to convert an ensemble model to a multiconformer model, prompting us to design one. We did this using qFit. Our approach systematically collapses an ensemble by iterating over each residue across all models and clustering equivalent residues. Taking the first model as the reference, we assign each subsequent residue to an existing cluster if its RMSD to the cluster centroid is within 1 Å (default parameter). If the RMSD exceeds this threshold, a new conformation is created. We then used the relabel function in qFit, which uses simulated annealing (SA) optimization of a Lennard-Jones potential to reassign altloc labels, ensuring that conformers of different residues/ligands have consistent altloc labels. Note that while we collapsed many conformers, many residues still have 26+ conformers, meaning these cannot be represented with the historic PDB format.

While we can create a multiconformer model, a few issues remain. 1) We do not have a correct occupancy for any residue (all residues are currently assigned occupancy of 0.50), 2) There may be issues in the geometry of backbone atoms due to removing conformations on a residue level. The other thing to note is that this is currently incredibly slow (~45 minutes for 400 residues with 70 models). While imperfect, the multiconformer models enable us to feed this into other algorithms, such as refinement or qFit.

This tool is available in the qFit repository, calling multimodel_2_multiconformer.py, only requiring an input PDB.

References

1. Woldeyes RA, Sivak DA, Fraser JS. E pluribus unum, no more: from one crystal, many conformations. Curr Opin Struct Biol. 2014;28: 56–62.
2. Wankowicz SA, Ravikumar A, Sharma S, Riley BT, Raju A, Flowers J, et al. Uncovering Protein Ensembles: Automated Multiconformer Model Building for X-ray Crystallography and Cryo-EM. bioRxiv. 2024. doi:10.1101/2023.06.28.546963
3. Burnley BT, Afonine PV, Adams PD, Gros P. Modelling dynamics in protein crystal structures by ensemble refinement. Elife. 2012;1: e00311.
4. Wall ME. Internal protein motions in molecular-dynamics simulations of Bragg and diffuse X-ray scattering. IUCrJ. 2018;5: 172–181.
5. Gutermuth T, Sieg J, Stohn T, Rarey M. Modeling with Alternate Locations in X-ray Protein Structures. J Chem Inf Model. 2023;63: 2573–2585.

The Scientific Setup

For each 3D pixel in reciprocal space (indexed by h), we have:

• Observed data, $y(h)$ from experiment
• Predicted data, $x(h)$ computed from molecular dynamics (MD) trajectories

We evaluate agreement using the Pearson correlation coefficient:

Each prediction $x(h)$ is derived from structure factors $F(h, t)$ across time points in the MD simulation:

The goal is to assign weights $w(t)$ to each time point to maximize $\mathrm{CC}$:

If we can find optimal weights, we can identify which regions of the trajectory best match experimental reality — potentially distinguishing “good” frames from those that detract from agreement.

Community Brainstorming

Steve suggested asking whether $\mathrm{CC}$ is the right target — perhaps a likelihood might better capture the physics.

Karson Chrispens proposed leveraging machine learning frameworks like JAX or PyTorch to treat the weights as trainable parameters.
By backpropagating through the Pearson correlation, an optimizer like Adam could efficiently learn the optimal weights.

James Holton suspected this approach could outperform traditional non-linear least-squares optimization and shared example MTZ datasets for testing.

Steve also mentioned using a genetic algorithm if the weights were binary ($0$ or $1$), though he acknowledged the continuous formulation might not have a unique minimum.

Prototyping the Optimizer

Karson quickly implemented a JAX-based prototype using reciprocalspaceship for MTZ I/O and optax for optimization.
The loss function was simply $-\mathrm{CC}$, and weights were constrained to $(0, 1)$ via a sigmoid transform.

When tested on toy datasets and real MTZ files, the optimizer:

• Successfully recovered 50:50 weights for mixtures of two “ground-truth” structures
• Produced sensible intermediate values when one or both inputs were “wrong”
• Converged robustly from different initializations

Example output for a ground-truth mixture:

Final weights: [0.46, 0.54]
Final CC: 1.0000

And for mismatched data:

Final weights: [0.76, 0.24]
Final CC: 0.78

Discussion

Marcus Collins noted that this approach resembles computing Boltzmann-like factors for each configuration and suggested PyTorch could be an equally good (and more common) platform.
He also cautioned that Pearson $\mathrm{CC}$ may not be the optimal objective function.

Karson confirmed that JAX runs efficiently on GPUs and planned to scale the approach to larger datasets by stacking multiple MTZ files.

Where This Might Go Next

This prototype demonstrates that gradient-based optimization can efficiently identify the contribution of different MD frames to observed diffuse scattering patterns.
Future directions include:

• Expanding to full MD trajectories with thousands of frames
• Experimenting with alternate objectives (e.g., likelihood, cross-entropy)
• Incorporating crystal symmetry and resolution weighting
• Exploring physical interpretations of the resulting weights

Code and Data

Karson’s implementation, pearson_target.py, is available here, and the test MTZ data can be downloaded from
here.

TL;DR:
By treating MD frame weights as trainable parameters in a differentiable Pearson correlation objective, we can use ML optimizers like Adam to rapidly identify which parts of a trajectory best explain experimental diffuse scattering — turning a brute-force search into a smooth, data-driven optimization problem.

I walked into Jaime Fraser’s office on a Friday in June, shortly before my early summer rotation at UCSF was about to start. As a physicist by training, I found his work on conformational entropy in proteins fascinating. But he quickly turned our conversation towards a topic I had never heard of before: diffuse X-Ray crystallography and somehow using it to solve protein ensembles and dynamics. “We’re kicking off the initiative at a conference on Monday, you should come!”

Come I did. While I was very lost on the specifics of most of the science during those two days, I could sense the excitement. Whole labs had traveled from across the country to attend as well as esteemed scientists from more than one national laboratory. They were proposing a new way to do science, more open, more flexible, and faster than we have come to expect from the traditional academy. So I did what any 1st-year PhD student would given the opportunity to join such a project, I took it and ran with it.

There was one small problem with joining a structural biology lab on the pretense of innovating in the theory of X-Ray crystallography during a summer rotation: I hadn’t touched diffraction since my bachelor’s degree. I too was coming off the heels of a master’s degree at Utrecht University and a follow-up internship at the University of Vienna both in theoretical physics. So I started at the beginning.

I was already reading the basics of Thomson scattering of a free electron before the end of day 2 at the diffUSE kick-off conference. “OH, it’s just the oscillation of the electron under electromagnetic radiation that emits scattered rays of the same frequency”. I’m not sure why but I always expected there to be a lot more quantum mechanics involved in scattering. As is often the case, the classical understanding of even very small systems is very much good enough.

The beginning of my rotation was nose to the grind stone in the form of review papers and textbooks, including some from the middle of the last century. I needed to understand classical structural X-Ray crystallography before I even knew how to ask Jaime what diffuse X-Ray crystallography is. Finally, after a few weeks, I knew what a Bragg peak and a structure factor were, and I had actually worked through the derivation of what I consider the central result of crystallographic theory: the electron density of the unit cell (and therefore the molecular structure of the contents of the unit cell) is the Fourier Transform of the scattered structure factors in diffraction space (and vice versa!) So what about diffuse?

Example scattering image (right) from a perfect crystal (left) with 6-atom hexagonal molecules at each lattice site. Structural information is contained in the sharp reflections (referred to as Bragg Peaks) of the scattered X-Rays. The scattering imafe is devoid of diffuse scattering. Fig. 5.5, Blundell and Johnson (1976).

diffUSE X-Ray Scattering

Now this central tenant is only 100% correct if each unit cell of a crystal is identical, but as every structural biologist knows, it is nigh impossible for a molecule as complicated as a protein to crystallize into the exact same form in each and every unit cell throughout a crystal. Besides the possibility of ligand-bound complexes, the protein itself can be in subtlely different folds between each unit cell. These are known as the protein’s conformers, and they result from small adjustments to a protein’s structure like bond rotations. All of the conformers available to a protein are known as its ensemble.

So do Bragg peaks still give you the protein structure when all different kinds of conformers might be present in a protein crystal? Yes and no - the structure you get from analyzing the scattering pattern is the averaged structure due to the averaged structure factors between all the unit cells. Yet the signal is not actually the same as if every unit cell were replaced with this averaged structure; the fact that the crystal is composed of different conformers is encoded in the scattering image as an unphased diffuse signal between the Bragg peaks.

X-Ray scattering image of insulin. The signal between the sharp Bragg peaks is the diffuse scattering. The isotropic ring is due to water’s presence in the crystal as a solvent, but non-isotropic diffuse signal is due in part to insulin’s conformational ensemble. Meisberger and Ando (2023).

The dominant term contributing to this diffuse signal is known as Guinier’s Equation:

$I$ is the intensity reading on the detector and $F$ is the complex structure factor. These values are averaged (a phased average in the case of the structure factor) over each and every unit cell at each spot on the X-Ray detector. If each unit cell is truly identical, then the average intensity is the same as the square of the average structure factor and the diffuse signal is exactly zero. Now if you have ever learned that the intensity is the complex square of the structure factor, then you can see Guinier’s Equation in a new light (transitioning to the angle brackets notation for ensemble averages):

The diffuse signal is really the statistical variance in the structure factor over every unit cell. James Holton, an expert beamline scientist at the Lawrence Berkeley National Laboratory, has a great interpretation of this variance up on his blog: the diffuse signal allows you to differentiate between correlated motions of the crystal’s unit cells. Now if we determine the correlated motions between the various conformers of a protein, we are really solving for the available dynamics of that protein! This is what makes diffuse protein X-Ray crystallography a big deal, and what the diffUSE initiative aims to accomplish. Much of protein behavior requires dynamics: catalysis, bond formation, active transport - solving the diffuse problem will allow for a whole new insight into the correlated dynamics that allow proteins to perform their functions.

Two conformational states of the F1-ATPase complex. This energy-providing molecular machine found in the mitochondria is a famous example of motion-induced function in molecular biology. Sobti, M. et al.(2024).

So what did I actually do?

After learning the essential background theory, I still had to actually do something to show for the rotation! Innovating on a theory or model of physical phenomena is hardwork, but we are lucky as 21st century scientists. We can use our ample computational resources to simulate the results of an experiment that are predicted from a proposed model. We then compare these simulations with real experimental observations, if the the simulations and the observations look the same, your model likely has explanatory power. If the comparison is not very similar, then the model needs some work.

So this was the guiding question of my rotation: Can we simulate diffuse scattering from an ensemble of protein conformers and process it as if it were real data? If so, the simulated data can be compared to real data to see how well our current models of diffuse scattering fit the real thing (recall that I said Guinier’s Equation is the dominant term in the diffuse signal merely due to the presence of distinct conformers, but other terms become not insignificant when there are correlation lengths of and between certain conformers).

James Holton was one of my main scientific mentors for this project, and he had already spent years developing extremely accurate, realistic simulations of diffraction datasets. So my initial task was to understand his simulation scripts and help modify them to account for diffuse scattering due to multiconformer crystals. In order to do so, I needed to familiarize myself with a whole suite of computational crystallographic tools that had been communally developed for decades (CCP4 and Phenix chief among them). I also needed to learn how to read (and write a few lines of!) tcsh command line scripts, James H.’s preferred scripting language. A totally new language to me, but the good news about command line scripting is that there are only so many layers of complexity that can be associated with each line of code. James helped me get set up to work on the LBL Beamline cluster, and after some googling (as well as LLM queries), I was able to understand what I needed to.

James had already scripted the computation of the diffuse scattering from the presence of two conformers, but we wanted to see what the diffuse signal looks like for whole ensembles of proteins, so we modified the script to account for more than just two. That meant parallelization to keep the computation time reasonable. After some great mentorship from James and the reverse-engineering of some of his other parallelized programs, we came up with a working script.¹

Next step was to process these simulated datasets and see if we could actually extract the diffuse signal from them. In order to do so, I needed to utilize another specially designed software tool, mdx2 - a diffuse scattering signal extractor and processor developed by Steve Meisberger and Nozomi Ando at Cornell University. I recall asking Steve if there was some tutorial that I could access, and he pointed me to an awesome paper written by him and Nozomi that not only linked to a github of step-by-step jupyter tutorials, but also a detailed guide on how to install the necessary Python packages to run mdx2 right in the paper itself! “Whoa!”, I thought. “So that’s what open science is really about.” I was able to get it working with our datasets, and managed to extract the Davinci Dude pattern from one of our simulated diffraction sets as a proof of concept.

Fully simulated diffraction image (left) of the two conformers of 1aho which generate the “Davinci Dude” diffuse scattering featured in James H.’s post. A slice of the extracted diffuse signal in reciprocal space (right) was generated by processing a dataset of these diffraction images using mdx2.

We moved on to proper ensemble modelling with the SARS-CoV-2 NSP3 macrodomain in its 7kqo P43 crystal form. Using the ensemble_refinement function from Phenix, we created a 200-conformer pseudoensemble (James H. ran the refinement for 3 days to get it), generated an associated diffraction dataset, and processed it with mdx2 to extract the diffuse signal.²

Three representative conformers of the 200-conformer pseudoensemble of 7kqo we used to generate a diffraction image dataset. One is colored green, red, and tan respectively.

Simulated diffuse scattering (left). Full diffraction image (center). Slice of the extracted diffuse signal in reciprocal space (right). All from the 200-conformer pseudoensemble of 7kqo.

In essense, the result of my summer project is a functional pipeline for simulating the diffuse X-Ray scattering of a multiconformer protein ensemble and subsequent processing of it through mdx2. These simulated signals are now ready to be compared to real signals, and the model of diffuse scattering incremented on. Sadly, my involvement in the story ends here as I need to move on to my next rotation. I’m very grateful for getting to do such awesome science and learning so much under the guidance of James Holton and Jaime Fraser! I would also like to thank the IMSD program and the diffUSE initiative for making this work possible and for helping me get off to a great start at UCSF!

Shipping Method

Two types of shipments were prepared. The first was a dry ice box containing four samples stored frozen in solution. For one of these samples, we also included a frozen seed stock essential for crystallization. The second shipment utilized a Crystal Positioning Systems thermal shipper, which maintains a consistent temperature at 20 degrees Celsius. In this box, we provided stocks of protein buffers and precipitant components to maximize reproducibility. Additionally, we included a 96-well crystallization plate containing pre-grown Huwe1 crystals of the fifth sample. We utilized foam inserts to hold the plate for safe transport.

By shipping both frozen proteins and a crystallization plate, we aim to determine which shipping method is most effective for sample preparation and crystal quality.

Looking Ahead

The next steps involve the Ando Lab assessing the quality of the shipped crystals to determine whether crystallization plates are a feasible option for sample transport. The Ando Lab will also use the shipped protein to grow new crystals. While we anticipate crystallization to be largely reproducible, some optimization may be necessary to produce crystals of sufficient size for diffuse scattering experiments, especially if plate formats and drop sizes differ from our established methods.

The Fraser Lab remains available to provide troubleshooting support and to ship additional protein as needed. Once we establish confidence in our shipping methods and crystal growth reproducibility, we will also prepare shipments of ligands to conduct soaking experiments.

Protein crystals are wet and tiny. Because of this, they dry out in air very fast, and this drying out crushes the lattice and ruins the diffraction. The first person to realize this was Dorothy Crowfoot (who later won the Nobel Prize under her married name: Dorothy Hodgkin). Pictured above is the sample holder she used. Glass flake is an excellent moisture barrier, but it is hard to work with and also glass has relatively high scattering and absorption of X-rays. This scattering makes X-ray background on the detector that can easily bury the faint diffuse-scatter signal from the protein crystal that we are trying to measure. Many different ways of mounting protein crystals have been developed over the years, and each has its compromises. For diffUSE, we will need to really push this technology to make high quality diffuse scatter data a routine undertaking.

The central challenge of window materials for protein crystals is not just the X-ray background, but the permeability. In general, thinner stuff gives you less background, but also dries out the crystals faster. Permeability of a thin film is hard to measure, but the potato chip industry has already done a lot of work on this (Rob Thorne says to google: “barrier materials”). Nearly all polymer-plastic based stuff has the same compromise because plastic is held together by van der Waals forces, and these easily open up to let gas molecules through via thermal motions. A “pile of worms” on the molecular level. Covalent solids, like soda lime glass, are ~1e6x less permeable, but with ~10x more scattering/absorption per unit thickness. I have looked for large sheets of 1-micron thick glass, but nobody seems to make them. You can get broken glass of this thickness and it is very cheap: “glass flake” is a paint additive. I have some, but it is hard to work with.

Recent results in the serial crystallography world have had good results with “SOS” chips, which use 2.5 µm thick Etnom foil (Chemplex). This will have very low background, but variability between crystals has been observed and is still being worked out.

You can also get very low permeability with metal foils. Metallized mylar is a popular barrier material, and you only need nanometers of metal to form a good barrier. Downside is you can’t see through it with optical microscopes, but if you are doing blind, grid-scanning serial data collection then it doesn’t matter anyway. I, however, would like to do the data collection a bit smarter and try to center the crystals in the beam. But that is my bias.

From the X-ray point of view, the background level is very predictable. This is because the cross sections of scattering and absorption depend only on the element, so all you need to do is calculate how many atoms are in the beam and you got your answer. Materials that have been rolled, stretched or pulled will have some orientation bias and therefore fiber diffraction. But, all that does is move the scattered photons around in reciprocal space, the number of scattered photons depends on the elemental composition and little else.

Experience at Cornell has settled on “free mounting” as a solution, but I have found this is not broadly general. About half of all crystals in my experience do not like being exposed to surface tension. The “HARE” chip solution from T-REXX at Hamburg is attractive because the silicon substrate makes a good scatterguard, but you need to surround it with a “rain forest” humidity container. That said, the T-REXX folks have been optimizing it for years. I need to follow up with David von Stetten to talk about the details.

Something else I learned at the most recent Serial Workshop was that people are having good luck with agarose as a carrier for fragile crystals (rubisco). That made me realize: we want whatever is touching the crystals to be softer than the crystals themselves. And I mean “softer” in the Mohs hardness scale kind of way. We don’t want the mount to “scratch” the protein. Agarose also has the advantage that it is easily cast, and cast materials tend to have no fiber diffraction. The trick will be getting it thin and manipulable in a reproducible way.

I want to stress: hardness of the window material is going to be really important. Consider that you not only want the window to be thin, but close to all the crystals. This is because any gap between the windows and the crystal will no doubt be filled with liquid, and that will contribute to the background. Crystals never all grow in exactly the same size and shape, so there is always going to be a compromise between the size of the crystal-window gap for the smaller crystals, and how many of the big crystals are going to be in contact with the windows. If the windows are harder than the crystal, then the crystal “loses”. It gets smooshed. And those were all your biggest crystals.

If we have a layer of soft, amorphous material coating hard windows, then it will move out of the way as the crystal is pressed into it, filling the gaps between them. In this situation the overall thickness of the material in the beam will be fixed from shot to shot, with the only variability being the fraction of the window-window gap that is filled with crystal vs filled with gel. We could then calculate exactly how much gel is in the beam and quantify its contribution to background. I predict this will be a very valuable feature.

Single-crystal windows I haven’t looked at in a while, but I think the SAXS people are the best ones to turn to for such things. We learned early on that diamond windows suck. Too much diffuse scatter from defects in the lattice. They can make “single” crystal diamond, but the mosaicity is pretty poor and the grain boundaries make DS. Beryllium is better. In fact, you want to use etched beryllium. This is because when they roll the Be foil you get tiny rocks and bits of the rollers stuck in the surface. These pits and defects make diffuse scatter. If you dissolve the surface away in acid it makes for a lower background window. These days, of course, they coat all the Be foils with plastic. The plastic makes it a crappier window, but it is more acceptable to our friends in charge of chemical safety.

That said, mica has met with a lot of success as a SAXS window. Ilme Schlichting described to me how to thin it down by pulling off layers with Scotch tape. You look for a color change that indicates it is only a few hundred nm thick. Then it is a good window. Just fragile.

I also have to admit it has been a while since we looked into these things. It may well be that single-crystal diamond has improved over the years. The only way to find out, however, is to buy some and try it. The companies have no idea, but last I checked were interested in the results.

Also, the ultimate thin covalent solid window is graphene. Jeney Wierman tells me she is still traumatized from working with it in grad school, but something she said that stuck with me is that a 1 micron backing material is fine. That also means, I think, that you don’t have to have just one layer of graphene. Ten would probably be ok. Martin Caffrey experimented years ago with graphene-impregnated plastics, but I think he went back to glass. Still. A graphene filled plastic that you spin cast and then UV-set should have a uniform background, and could be very very thin and water tight. You can get “bulk” graphene pretty cheap these days. Something I’d like to test.

@Steve found a paper comparing 10 µm quartz to various polymers and glasses: https://pmc.ncbi.nlm.nih.gov/articles/PMC6057835/. “Looks promising, provided you avoid hitting a Bragg condition of the quartz.

This reminds me of a big caveat that I’m not sure this work addressed: self-absorption. If all you do is look at the amount of background on the detector, then depleted uranium will look like a good window material: no photons on the detector at all.

Another important principle to remembner is that the scattering cross section of an atom is a fixed quantity. It is independent of the structure of the material it is in: crystal, amorphous, gas, or otherwise. A given number of oxygen atoms in the beam scatters a knowable number of photons. The only thing the structure of the material does is push those photons around on the detector. So, at the end of the day, the only way a window can be “X-ray transparent” is to be thin. And light. And it would be great if it is also cheap and easy to work with.

Looking forward to everyone else’s thoughts and to learning how to do this right!