Skip to content

Commit 5aa79ad

Browse files
patrickpcleepatrickpclee
authored
ATC '26 AE (#184)
1 parent e7444d9 commit 5aa79ad

5 files changed

Lines changed: 158 additions & 1 deletion

File tree

‎_conferences/atc.md‎

Lines changed: 3 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -2,4 +2,6 @@
22
title: ATC
33
---
44

5-
The [USENIX Annual Technical Conference](https://www.usenix.org/conferences/byname/131) introduced artifact evaluation in 2022.
5+
The [ACM SIGOPS Annual Technical Conference (ATC)](https://sigops.org/s/conferences/atc/2026/) (formerly the USENIX Annual Technical Conference) is organized by [ACM SIGOPS](https://sigops.org/). Research artifact evaluation was first introduced in 2022.
6+
7+

‎_conferences/atc2026/aec-call.md‎

Lines changed: 25 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,25 @@
1+
---
2+
title: Call for Evaluators
3+
order: 20
4+
---
5+
6+
Join the Artifact Evaluation committee, learn about the latest research before everyone else, and help make science more reusable and reproducible!
7+
8+
**Apply now**: [fill this self-nomination form](https://forms.gle/63xtgr1dcu7uejLo6) (deadline: **August 15, 2026 (AoE)**)
9+
10+
You can be located anywhere in the world as all committee discussions will happen online. AEC membership is especially suitable if you are early in your career, such as the first or second year of your PhD. You are welcome to join the AEC if you are working in any practical aspects related to computer systems, such as operating systems; runtime systems; parallel and distributed systems; storage; networking; ML for system and systems for ML; security and privacy; virtualization; software-hardware interactions; performance evaluation and workload characterization; reliability, availability, and scalability; energy and power management; and bug-finding, tracing, analyzing, and troubleshooting.
11+
12+
As an artifact evaluator, you will:
13+
- Read the papers whose artifacts you are assigned to review
14+
- Install, audit, and run artifacts
15+
- Discuss with other evaluators to award [badges](./badges)
16+
17+
Check out the [evaluator guide](/evaluator-guide) for an overview of what artifact evaluation involves. You must carry out evaluations independently, without sharing data nor information with people outside the committee.
18+
19+
Please ensure you have available time during the evaluation period stated in the [home page](./index)'s important dates section. We hope to assign up to 2 artifacts per evaluator, each taking around **10-20 hours** to review.
20+
21+
This work can usually be done on your own computer, any moderately recent desktop or laptop will do. In specific cases and to the extent possible, authors will arrange their artifacts so as to run in community research testbeds or will provide remote access to their systems. If you have access to special hardware, that might come in handy, but it is by no means required.
22+
23+
**Peer review policy:** Please refer to [ACM Peer Review Policy](https://www.acm.org/publications/policies/peer-review) regarding the use of generative AI. In particular, please refrain from uploading submissions / artifacts to any generative AI tools for reviews. Note that these submissions / artifacts may not be the finalized versions, and they should not be disclosed outside of the review process.
24+
25+
Contact the [chair](./committee) if you have any questions.

‎_conferences/atc2026/badges.md‎

Lines changed: 76 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,76 @@
1+
---
2+
title: Badges
3+
order: 10
4+
---
5+
6+
<style>
7+
img { width: 4em; }
8+
</style>
9+
10+
ATC is an ACM conference and thus uses [ACM's badges](https://www.acm.org/publications/policies/artifact-review-and-badging-current).
11+
12+
Authors can apply for, and be awarded, one of three combinations for their artifacts and associated papers:
13+
14+
| ![Available](/images/acm_available_1.1.png) ![Functional](/images/acm_functional_1.1.png) ![Reproduced](/images/acm_reproduced_1.1.png)<br>**Available, Functional, and Reproduced** | For the vast majority of software artifacts,<br>and for hardware artifacts whenever possible. |
15+
| ![Available](/images/acm_available_1.1.png) ![Functional](/images/acm_functional_1.1.png)<br>**Available and Functional** | For data sets,<br>as well as artifacts that require custom environments authors can't give access to. |
16+
| ![Functional](/images/acm_functional_1.1.png) ![Reproduced](/images/acm_reproduced_1.1.png)<br>**Functional and Reproduced** | For software and hardware artifacts that the authors cannot make public. |
17+
18+
Artifacts submitted for the first target may be awarded one of the other two if availability or reproducibility evaluations fail, respectively.
19+
Authors cannot apply for other combinations of badges as these make little sense, such as "an artifact that is not public, does not appear functional, but outputs the right numbers".
20+
21+
22+
## Checklists
23+
24+
To provide fair evaluation across artifacts, to help authors prepare, and to help evaluators work efficiently, each badge has an associated checklist.
25+
26+
### "Available" checklist
27+
28+
- The artifact is available on a **public archive with irrevocable versioning and long-term storage**, such as Zenodo but not GitHub
29+
- The artifact has a **license that allows comparison and extension**, such as the [CC-BY](https://creativecommons.org/licenses/by/4.0/) or [MIT](https://opensource.org/license/mit/) licenses
30+
- The artifact has a **"read me" file referencing the paper**
31+
32+
These criteria must be met *at the time artifact evaluation finishes*.
33+
Authors only need to put the data in long-term storage once evaluators are otherwise satisfied. Development may take place on a platform like GitHub.
34+
Promises of future availability are *not* acceptable, such as uploading the artifact to a private repository with the goal of "eventually" making it public.
35+
36+
### "Functional" checklist
37+
38+
- The artifact has a **"read me" file** with:
39+
- A description of each artifact component and how it relates to the paper
40+
- A description of the exact environment the authors used, such as OS version and hardware
41+
- If the artifact includes code that deliberately performs malicious or destructive operations, appropriate warnings and context
42+
- The artifact includes **all code and data relevant to the paper**, and only those
43+
- The artifact must not include obsolete or unrelated code nor data
44+
- If existing code or data has been modified, the artifact should clearly separate the modifications from the original
45+
- If the paper makes soundness claims, such as proofs, there should be simple scripts to verify these, such as listing proof assumptions
46+
- If the paper makes quantifiable claims, such as code size per module, there should be simple scripts to output these
47+
- For data, **modifications made to the raw data are documented**
48+
- For instance, whether parts of the raw data were anonymized or discarded
49+
- For executable artifacts, the "read me" file also contains **documentation** to:
50+
- Run and extend a "minimal working example"
51+
- Compile and execute the artifact, including pre-installation steps
52+
- Configure the artifact, such as selecting IP addresses or disks
53+
- Know the expected resource use per kind of experiment, such as "5 minutes, 10 GB of disk space"
54+
- Know what unusual behavior to expect, such as warning messages emitted by another system used as baseline for experiments
55+
- For executable artifacts, the artifact includes a **precise list of dependencies**:
56+
- Whenever possible, it should be usable by a package manager
57+
- Exotic dependencies must have associated automation to download and build them
58+
- OS-level dependencies must involve a VM/container, accompanied by a script to generate the VM/container
59+
- Proprietary dependencies must have associated instructions to obtain them along with "mock" versions to demonstrate their use
60+
- The artifact includes an **example input and configuration for each kind of experiment** in the paper
61+
- Authors are encouraged, but not required, to provide inputs, configurations, and outputs for all experiments described in the paper
62+
63+
Artifacts must be usable on other environments than the authors', though software may require specific hardware such as one model of network card.
64+
Manual work such as writing configuration files must be minimized. There must be no redundant manual steps such as writing the same configuration values in multiple places, as this inevitably leads to human error.
65+
66+
### "Reproduced" checklist
67+
68+
- The artifact includes a **single script to run each experiment** and output results, given the necessary input and configuration
69+
- The scripts must be documented, allowing researchers to ensure they correspond to the claims, merely producing the right output is not enough
70+
- The scripts must handle common edge cases in a reasonable fashion, such as forgetting arguments or running the same script twice
71+
- The artifact includes a **script to convert each experiment's results into human-readable ones** as close to the paper presentation as possible
72+
- For simple results presentation such as tables, this and the previous script can be merged into one
73+
- The artifact may contain separate installation steps for the dependencies of plotting scripts, subject to the same criteria
74+
75+
The expected workflow for an evaluator or a researcher looking to reuse the artifact is to install the artifact using a handful of commands, run experiments with one command each, and plot data as necessary.
76+
In the absence of problems requiring debugging, active time must not exceed a few minutes.

‎_conferences/atc2026/committee.md‎

Lines changed: 12 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,12 @@
1+
---
2+
title: Committee
3+
order: 30
4+
---
5+
6+
**Chair**:
7+
- Patrick P. C. Lee, The Chinese University of Hong Kong
8+
9+
_Please do not contact the chairs to self-nominate for the committee; use the form instead._
10+
11+
**Members**:
12+
- TBA

‎_conferences/atc2026/index.md‎

Lines changed: 42 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,42 @@
1+
---
2+
title: Artifact Evaluation
3+
order: 0
4+
---
5+
6+
A scientific paper consists of a constellation of artifacts that extend beyond the document itself: software, hardware, evaluation data and documentation, raw survey results, mechanized proofs, models, test suites, benchmarks, and so on. In some cases, the quality of these artifacts is as important as that of the document itself. Last year, [46% of accepted USENIX ATC '25 papers](https://sysartifacts.github.io/atc2025/results) participated in the artifact evaluation process. Based on the success of artifact evaluation for past editions, ACM ATC '26 will have an artifact evaluation process.
7+
8+
The artifact evaluation process considers the reusability and reproducibility of artifacts associated with papers. Artifact evaluation is _single blind_: authors do not know evaluators' identity, but evaluators know who the authors are.
9+
10+
**Call for evaluators**: Apply [here](./aec-call) to join the artifact evaluation committee!
11+
12+
**Artifact registration and submission**: TBA
13+
14+
## Process
15+
16+
Artifact evaluation is a cooperative process, unlike paper reviewing, meaning that authors have the opportunity to fix problems found during evaluation if they can do so in reasonable time.
17+
18+
1. Authors decide which [badges and associated checklists](./badges) they target.
19+
2. Authors _register_ their artifact indicating which badges they target, along with the paper PDF and some metadata.
20+
3. Authors then _submit_ their artifact, packaged according to [the artifact packaging guide](/packaging-guide) using a service such as GitHub.
21+
4. Evaluators begin with a short "kick the tires" phase to double-check that the artifacts are complete and appear usable and contact authors if that is not the case.
22+
5. Evaluators proceed with the main evaluation period, operating mostly independently but occasionally contacting authors if an issue arises.
23+
6. Authors upload the final version of their artifact to permanent storage such as Zenodo and submit their artifact DOI.
24+
7. Authors are awarded badges which they add to their camera-ready paper submission.
25+
26+
## Important Dates
27+
28+
- Paper acceptance notification: **September 18, 2026**
29+
- Artifact submission deadline: **September 22, 2026 (AoE)**
30+
- Artifact "kick the tires" evaluation period ends: **September 29, 2026**
31+
- Artifact decisions: **October 14, 2026**
32+
- Paper camera-ready deadline: **October 16, 2026**
33+
34+
At least one author for each artifact submission must be reachable via email and respond to questions in a timely manner during the kick-the-tires period. Ideally, authors should also be available to respond to reasonable requests during the main evaluation period.
35+
36+
## Committee
37+
38+
Chair:
39+
40+
- Patrick P. C. Lee, The Chinese University of Hong Kong
41+
42+
Full membership list [here](./committee).

0 commit comments

Comments
 (0)