@claude In tandem with addressing reviewer feedback, make sure to track the original submitted version and compile latex-diffs of recent version relative to that original version. Also, in addition to making changes, make notes in a response to reviewers. This should be a markdown-like document that has reasonable formatting that intersperses our comments below quoted text (i.e., in markdown > ...). All quoted text should be kept, but line breaks can be added when we address specific items within a paragraph. Below, I'm providing my instructions to you for how to address these things, but you still need to keep a document that is addressed to the reviewers that is readable and makes sense to the reviewers. Don't speak to me. Speak to the reviewers. Keep it relatively terse and brief. Don't extensively detail the changes in the response, instead point to specific places within the manuscript where changes were made (section, paragraph, etc.).
This will be long session, potentially up to several hours.
REVIEWER REPORT(S):
Referee: 1
Comments to the Author
The manuscript reports the organization and outputs of the AC-BO Hackathon 2024, a two-day virtual event focused on Bayesian optimization for chemistry and materials. I view the event itself as a valuable community effort. The hackathon brought together a large and geographically diverse group of participants across academia, industry, and government, and it produced an impressive number of open-source outputs, including algorithms, benchmark problems, tutorials, notebooks, software repositories, and applied case studies. This is a meaningful contribution to the community, especially because practical examples and reusable code for Bayesian optimization in chemistry and materials remain scattered across different packages and application domains.
A particular strength of the manuscript is the open-resource aspect. The authors have archived the hackathon outputs through project pages, GitHub repositories, videos, slides, and a Zenodo record, which provides a useful entry point for researchers and students who want to learn or apply Bayesian optimization in chemistry and materials. The project list spans reaction optimization, molecular design, corrosion inhibition, MOFs, zeolites, hydrogels, thin films, self-driving laboratories, and other applications, suggesting that the hackathon generated a broad and potentially useful set of community resources.
However, I think the manuscript would be significantly strengthened by adding more synthesis rather than only listing individual project summaries. First, the authors should summarize the organizational lessons learned from running the hackathon. For example, what worked well in terms of pre-event tutorials, GitHub classroom assignments, virtual collaboration tools, team formation, project scoping, judging, mentoring, and post-event archiving? What did not work well? What would the organizers change in a future event? Since the manuscript presents the hackathon as a community-driven research model, these practical lessons would be valuable to readers who may want to organize similar events.
This will require the most manual input from me.
Second, the authors should synthesize the scientific lessons from the project outcomes. The manuscript currently presents many project summaries, but it does not clearly distill what the hackathon revealed about the strengths and weaknesses of Bayesian optimization in chemistry and materials. I encourage the authors to add a section summarizing the observed pros and cons of BO across the projects. For example, the paper could discuss where BO performed well, such as low-data optimization, multi-objective search, transfer learning, molecular/materials screening, or human-in-the-loop workflows. It should also discuss limitations identified by the projects, such as sensitivity to molecular representation, kernel choice, warm-start data, noisy observations, high-dimensional search spaces, batch-size selection, acquisition-function optimization, computational overhead, and reproducibility of benchmark comparisons.
@claude send a series of edison scientific queries related to this. You can likely also download the zenodo snapshot and upload that to edison analysis along with a copy of the manuscript to help with that. We'll need to triage the synthesize from project outcomes, but also note the cautions against heavy interpretation of the results when the hackathon itself spanned a relatively short amount of time. I.e., not a huge amount of time to do extensive validation or draw generalized conclusions.
Third, the authors should add a forward-looking section on future opportunities for Bayesian optimization in materials discovery. Based on the hackathon outputs, what are the most promising directions? Possible topics include BO for autonomous laboratories, multi-fidelity optimization, uncertainty-aware experimental planning, foundation-model-assisted BO, preference-based BO, multi-objective materials design, robust BO under noisy experimental data, benchmark development, and domain-specific BO tools for synthesis and processing optimization. Such a synthesis would make the paper more than a record of an event; it would turn it into a useful perspective on where BO is succeeding, where it struggles, and where the field should go next.
All of these topics sound good. Run separate edison queries for each of the topics to get at least one or two references and cite those references. A note should be made about the relatively high number of projects that were exploring the use of LLMs
I also recommend that the authors provide a more systematic evaluation of the 45 project outputs. It would be helpful to classify projects into categories such as mature software, benchmark datasets, tutorials, application demonstrations, and preliminary concepts. A table reporting code availability, licensing, documentation, reproducibility, dependency status, and maintenance plans would strengthen the resource value of the paper. This would also help readers identify which outputs can be reused immediately and which require further development.
Attempt a classification scheme for the various projects, then upload this along with the zenodo snapshot and the manuscript to edison analysis (see your custom instructions especially for how to use edison analysis, it's tricky), and much of this should be actual analysis across the various repositories (e.g., distribution of licenses, etc.).
Overall, I find the manuscript valuable as a community-resource and open-science contribution. The large number of open-source outputs is a genuine strength. However, the paper should do more than document that a hackathon occurred. I recommend major revision to add a stronger synthesis of organizational lessons, scientific lessons from the BO applications, limitations revealed by the project outcomes, and future opportunities for Bayesian optimization in chemistry and materials.
Referee: 2
Comments to the Author
This paper summarizes the findings arising from the organization of a hackaton on BO methods applied to chemistry and materials science.
I believe that there is a lot of value in presenting the details on how the hackaton was organized and I really appreciate the novel aspects of it, including the use of the GatherTown platform to facilitate the interactions within and among teams as well as between teams and organizers. Other novel aspects such as the method used for evaluating the teams to select the best ones are also state-of-the-art and are worthy of dissemination.
My only recommendation is for the authors to present a meta-analysis of the different projects, beyond summary listings of the project titles and the description of the individual projects. For example, were there commonalities/differences in the way different teams approach their problems? Does the level of prior expertise impact the outcomes? Are there follow up studies to see whether participants are applying BO methods in their own research? What was the ultimate gain from the hackaton? increased awareness? increased understanding of the methods? more interactions across many groups? Are there lessons learned?
@claude address this in tandem with reviewer 1's comments; send at least a couple edison analysis queries, one for each of the questions here. Run your own searches of follow-ups, etc.
Hackathon had several main goals and gains, described on the website: https://ac-bo-hackathon.github.io/, though maybe it's a bit vague: "We believe that through this event, new connections have been formed, new skills have been acquired, and new ideas have been inspired." This is something I'll probably need to weigh in manually on.
While I would not expect the authors to address all the items presented above, it would be very useful to have a reflective component to the paper and at least some discussion on lessons learned.
Referee: 3
Comments to the Author
Bayesian Optimization Hackathon for Chemistry and Materials documents a community output of the AC-BO Hackathon in 2024. All project outputs have been made accessible, with code and data available in github and in a zenodo record. This is particularly impressive given that there are 45 projects. This article discusses the outcomes of such projects.
This article is interestingly documented. In fact, I really enjoyed it as a reader, because it gives many use cases (with code!) where BO may be useful. This is the type of work that Digital Discovery should indeed encourage!
Although the article discusses each project individually and gives a good notion of what each project attempted to do / did, this reviewer finds the listing of projects to end a bit abruptly. It would be nice if some takeaways from the hackathon are produced at the end:
- are there agreed upon frameworks that work better for specific tasks (i.e. with noisy data or less noisy data)?
- Are BO tools going to be easily deployable in experimental data settings (the hackathon seems to imply that it will!).
- Where is it useful to use Bayesian Optimization vs. where did end users find little utility?
This reviewer believes that expanding the final commentary on these points would be helpful. However, this reviewer also commends the fantastic effort in making everyone’s data and code publicly available.
@claude send edison queries, similar to above, for each of these. Again, some of these might be more so notions than any kind of conclusive statements
Referee: 4
Comments to the Author
The authors summarize the results of a Hackathon focusing on Bayesian optimization algorithms, benchmark development, tutorialization, and problem definition which gathered multiple collaborators across national, career, and scientific domain boundaries. While many of the projects which came from this event may not, in terms of their data and code, adhere to the standards of Digital Discovery, this summary report generally does. Please see the attached document for a list of comments concerning missing information and other recommendations for improving how the information in this manuscript is presented.
DD-ART-06-2026-000353 DataReview R1.pdf
(copied below)
Data review
The authors summarize the results of a Hackathon focusing on Bayesian optimization algorithms, benchmark development, tutorialization, and problem definition which gathered multiple collaborators across national, career, and scientific domain boundaries. While many of the projects which came from this event may not, in terms of their data and code, adhere to the standards of Digital Discovery, this summary report generally does. Please see below for a list of comments concerning missing information and other recommendations for improving how the information in this manuscript is presented.
Major Comments
- The main text skips project 14, 19, 23, 29, 34, and 42. Can the authors indicate why these projects were withheld and update the introductory statement “This section provides a comprehensive summary and highlights the key findings from all project submissions” to reflect these omissions?
I think certain projects were withheld if the project didn't meet the requirements for authorship, or if the project was more or less dormant (e.g., no one showed up for the hackathon). Dig into this. Actually, wait, I think this may have been related to audio being available for video recordings. Or, this was a mishap and some of this got lost somehow.
- a. For example, a reader would likely want to know more about the first- and second-place winners (projects 23 and 34).
Definitely include more info here.
- Many of the code repositories do not provide adequate module requirement.
- a. Projects 2–8, 10–15, 16, 18, 20, 22–24, 26, 28, 33, & 38–41.
- b. As this work is spotlighting contributions rather than presenting code as part of its research workflow, the Journal requirements for code repository metadata may not apply.
- There are some projects for which the dataset used is not clear.
- a. Projects 6, 19–21, 29 (Implied to be GAUCHE), 30, 31, 36, 37, 40, 42, & 45
- b. Project 15 does provide the dataset, but it is buried within the code repository
- c. (See comment 2.a)
- Can the rubric used for evaluating projects be included in the supplemental materials?
- The workflow for transcribing and analyzing the projects is not reproducible at its current level of detail. Furthermore, the claim that this approach provides a structured and objective assessment of the submissions is not supported by any evidence or reference to prior works.
@claude do we have this available somewhere in this repo or in discussion? If not, we'll need to reach out to Mehrad I think, or re-do it (less preferred).
Tables and Figures
- Figure 2: The black text which falls above the map can be difficult to read.
- Table 1 spans two pages but contains no entries on the second page.
Try to address this. It's a long table which caused headaches with formatting.
- Figure 3: The caption contains a statement on preprint server policies which should be updated to adhere to Digital Discovery’s polices.
What's this?
- Table 2: The “Prize” header is marked for a footnote that is not present.
Add this footnote, if you are confident you know what should go there
- Figures 4 & 5: Were participants informed that their names and commentary may be made public prior to joining the event?
Hmm.. I don't think so. Apply blurring to the names for the large group image, but leave the names in for Ramsey, Sterling, Taylor, etc. Those ones are fine.
Minor Comments
- The use of project title headings as links to videos (hosted on YouTube) does not adhere to transparent hyperlink standards and is inaccessible on paper copies. In addition, these hyperlinks are redundant with the links already provided in Table 1.
Remove the project title hyperlinks
- Project webpages (potentially beyond the scope of the manuscript):> - a. Project 40 has been migrated from the github listed on its project page to the repo listed in the Zenodo metadata file
Address above.
- **b.** Projects 33 and 34 do not have github links on their project pages despite having links in the Zenodo metadata file
Address above.
- Typographical errors in the Acknowledgement section, around the header for Project 3, and in the Author Contributions section.
- a. “Ryan-Rhys Gri ths”, “Jakub LÆla”, “Can zkan”, “Adrian o†i¢”, “Je rey Watchorn”, and potentially others.
- b. Multiple ligatures appear to have been deleted (“fi”, “ff”, etc.)
Address above
Data Reviewer Checklist
1. Data sources
- a. Are all data listed and publicly available?
No (See comment 3)
- b. (If using an external database) Is an access date or version number provided?
(Outside the control of the authors).
- c. Are potential biases in the source dataset reported and/or mitigated?
(See comment 5)
2. Data cleaning
- a. Are the data-cleaning steps clearly and fully described either in text or as a code pipeline?
No (See comment 5).
- b. Is an evaluation of the amount of removed source data presented?
(See comment 5).
- c. Are instances of combining data from multiple sources clearly identified with potential issues mitigated?
N.A.
3. Data representations
- a. Are methods for representing data as features or descriptors clearly articulated, ideally with software implementations?
Projects – (Outside the control of the authors).
Surveys – Yes
- b. Are comparisons against standard feature sets provided?
N.A.
4. Model choice
- a. Is a software implementation of the model provided such that it can be trained and tested with new data?
(Outside the control of the authors)
- b. Are baseline comparisons to simple/trivial models provided?
N.A.
- c. Are baseline comparisons to current state-of-the-art provided?
N.A.
5. Model training and validation
- a. Does the model clearly split data into different sets for training, validation, and testing?
(Outside the control of the authors).
- b. Is the method of data splitting stated and mimic anticipated a real-world application?
(Outside the control of the authors).
- c. Does the data splitting procedure avoid data leakage?
(Outside the control of the authors).
6. Code and reproducibility
- a. Is the code or workflow available in a public repository?
Projects – Yes (Some repositories are withheld—beyond the control of the authors).
Surveys – No (See comment 5)
- b. Are scripts to reproduce the findings in the paper provided?
``
- c. Have the authors clearly specified which versions of the software libraries they depend upon were used in the course of the work?
Many participants did not include adequate descriptions of their dependencies—this is beyond the control of the authors.
I don't think there's anything here in the data reviewer section that requires something to be changed or addressed.
Make sure to include a link to create a PR. Also update CLAUDE.md in the branch with anything you feel is necessary.
@claude In tandem with addressing reviewer feedback, make sure to track the original submitted version and compile latex-diffs of recent version relative to that original version. Also, in addition to making changes, make notes in a response to reviewers. This should be a markdown-like document that has reasonable formatting that intersperses our comments below quoted text (i.e., in markdown
> ...). All quoted text should be kept, but line breaks can be added when we address specific items within a paragraph. Below, I'm providing my instructions to you for how to address these things, but you still need to keep a document that is addressed to the reviewers that is readable and makes sense to the reviewers. Don't speak to me. Speak to the reviewers. Keep it relatively terse and brief. Don't extensively detail the changes in the response, instead point to specific places within the manuscript where changes were made (section, paragraph, etc.).This will be long session, potentially up to several hours.
REVIEWER REPORT(S):
Referee: 1
This will require the most manual input from me.
@claude send a series of edison scientific queries related to this. You can likely also download the zenodo snapshot and upload that to edison analysis along with a copy of the manuscript to help with that. We'll need to triage the synthesize from project outcomes, but also note the cautions against heavy interpretation of the results when the hackathon itself spanned a relatively short amount of time. I.e., not a huge amount of time to do extensive validation or draw generalized conclusions.
All of these topics sound good. Run separate edison queries for each of the topics to get at least one or two references and cite those references. A note should be made about the relatively high number of projects that were exploring the use of LLMs
Attempt a classification scheme for the various projects, then upload this along with the zenodo snapshot and the manuscript to edison analysis (see your custom instructions especially for how to use edison analysis, it's tricky), and much of this should be actual analysis across the various repositories (e.g., distribution of licenses, etc.).
Referee: 2
@claude address this in tandem with reviewer 1's comments; send at least a couple edison analysis queries, one for each of the questions here. Run your own searches of follow-ups, etc.
Hackathon had several main goals and gains, described on the website: https://ac-bo-hackathon.github.io/, though maybe it's a bit vague: "We believe that through this event, new connections have been formed, new skills have been acquired, and new ideas have been inspired." This is something I'll probably need to weigh in manually on.
Referee: 3
@claude send edison queries, similar to above, for each of these. Again, some of these might be more so notions than any kind of conclusive statements
Referee: 4
I think certain projects were withheld if the project didn't meet the requirements for authorship, or if the project was more or less dormant (e.g., no one showed up for the hackathon). Dig into this. Actually, wait, I think this may have been related to audio being available for video recordings. Or, this was a mishap and some of this got lost somehow.
Definitely include more info here.
@claude do we have this available somewhere in this repo or in discussion? If not, we'll need to reach out to Mehrad I think, or re-do it (less preferred).
Try to address this. It's a long table which caused headaches with formatting.
What's this?
Add this footnote, if you are confident you know what should go there
Hmm.. I don't think so. Apply blurring to the names for the large group image, but leave the names in for Ramsey, Sterling, Taylor, etc. Those ones are fine.
Remove the project title hyperlinks
Address above.
Address above.
Address above
I don't think there's anything here in the data reviewer section that requires something to be changed or addressed.
Make sure to include a link to create a PR. Also update CLAUDE.md in the branch with anything you feel is necessary.