Building Human-Agent Systems for Science

Imagine we had perfect agents for doing science. What then would good science process look like? In this class, we try to answer this question. We look at the interfaces where people will work with agents and develop new kinds of infrastructure to help them work together better. We start by building a simple scientific discovery agent and exploring mechanisms for human steering and review of agentic work. Then we expand focus. We review the broad array of processes where agents may disrupt scientific workflows, like collaborative sensemaking over data, sharing results, and building repositories of knowledge for the lab and community. In a series of assignments, we build prototype infrastructure to support envisioned scientific workflows and use them on example research problems. In a final project, students build pieces of agentic scientific infrastructure inspired by their own research or interests.

This course has a strong human-computer interaction flavor. We will of course read papers and hear from speakers in AI and build many prototypes that use agen, but be ready to dive deep into concepts from human-computer interaction that help point the way to new solutions.

Audience: Ph.D. students, Master’s, Bachelor’s students, and others engaged in research from any field. This course is for researchers who are interested in trying out new processes for themselves and their group. Those not actively engaged in research might still be considered but will be given lower priority.

Public option: I’m planning to allow a select set of non-Penn students with strong interest to call into lectures and guest talks in real-time. If you want to participate in this capacity, fill out this form.

Prerequisites: We welcome students from the gamut of fields! But — you must know how to write code. For instance,you should be able to write Python yourself, pick and call programming libraries, and inspect what your code is doing. You won’t necessarily be writing much code yourself. But you will be on the responsible for making sure your tools aren’t accidentally abusing your research data or running up a huge bill.

Format: Every week, there will be 1 instructor-led lecture on Monday, and 1 discussion section where we discuss papers on the week’s lecture topic on Wednesday.

Semester: Spring 2027

When: Mondays and Wednesdays, 1:45-3:15pm

Location: TBD

Instructor: Andrew Head

Schedule

This is a rough schedule, and it will be continually updated for the next few weeks (this message written on Oct. 2, 2026).

Assignments

This class will have 7 assignments (also liable to be updated). Assignment #1-4 focus on augmenting individual scientific activity, and #5-6 on collaborative and community-level activity. Almost all assignments will provide opportunities for students to use the systems they are building in their own research.

• Assignment 1. Build your scientific “digital twin.” This is a guided activity in building a scientific discovery agent. It will take in a dataset as input, come up with research questions, explore them, and report out results. Tune it until it resembles your scientific “digital twin,” producing results and outputs you fully stand behind without intervention.

• Assignment 2. Ideate with a custom LLM wiki. Agents are supposed to help us keep track of more information in knowledge-intensive tasks. Build up a local knowledge base that actually does this for you. Four UI requirements: it has to be dead easy to add data to it; it has to produce an compact and accurate wiki that you actually want to use use; it has to ask for clarification when it doesn’t know what’s in your data; and you have to be able to query against it. Test it out by using it as a partner in ideating research project ideas.

• Assignment 3. Make agentic output make sense. You will be given an output trace from a scientific discovery agent from a domain you have never worked in before. Put it in an interface that makes it make sense. A non-expert should be able to quickly assess: Does it come up with a valid result? Use appropriate methods? Does its result really matter? But they don’t have the expertise! Figure out what explanations and context you have to provide in the interface so that it doesn’t matter.

• Assignment 4. A norm-enforcing analysis environment. Analyses have norms. One such analysis is qualitative analysis, where it is expected that a human reviews data, comes to an independent judgment of the how to describe it, and bases that judgment on a complete reading of the data. Is it possible to build AI tools that provide AI support while ensuring the norms are upheld? Build a qualitative analysis environment that allows the use of AI but structures the way that data is shown and processed to guarantee the norms are upheld.

• Assignment 5. A proactive knowledge base with many humans and agents. Turn your wiki from Assignment 2 into a substrate that can (and will!) support collaborative science among you and your final project teammates. It must have features for—descriptions of claims you are working on, evidence you have collected, ability to collaboratively edit the wiki without conflict, agents can contribute to wiki, and participants are notified when new evidence is brought to bear on key claims. Use this wiki to support your group’s creation of a final project proposal.

• Assignment 6. Bifurcating the research report. Create a set of publications that represent one possibility of the future of publication. Given a trace of scientific activity, generate (1) a document that represents the sophistication of human decisions of the scientist, which could be used in training and promotion settings (2) a document representing the actual results in a more compact but expandable form. Could this be the future of scientific publishing?

• Assignment 7. Final project. To be worked on in the last 2 months of the class. Develop a piece of scientific infrastructure designed to fit a human-agent scientific future. Work on something inspired by your own research or interests—you are strongly encouraged to make something that you yourself or your lab would use.