Public CGO-PPoPP'18 artifact evaluation discussion session on the 26th of February

We have successfully completed PPoPP'18 artifact evaluation (AE). Just like at CGO'18, we received a record number of artifact submissions: 15. Results are now available at http://ctuning.org/ae/artifacts.html !


For the first time, we used the new ACM Artifact Review and Badging policy which we co-authored last year. Note that it is now possible to search for papers with specific badges in the ACM Digital Library: go to https://dl.acm.org/advsearch.cfm?coll=DL&dl=ACM and select "Artifact Badge" for field and then select badges to search! Since we see AE as a cooperative process to improve reproducibility of experiments, authors, reviewers and chairs worked closely together to improve artifacts and pass evaluation. We would like to thank them all for their hard work: http://cTuning.org/ae/committee.html !


Though there were no major problems, we noticed that "reusability/customization" criteria in the new guidelines are quite vague and caused ambiguity in evaluation of several complex artifacts.

Another problem is that all artifacts have their own ad-hoc formats and scripts, while we now have to automate this process as much as possible to make AE sustainable. ACM is now evaluating several technologies to pack, share and evaluate artifacts automatically: https://www.acm.org/publications/artifacts

We plan to evaluate those techniques further during the 1st open tournament on reproducible and Pareto-efficient co-design of the whole software/hardware/model stack for deep learning and other emerging workloads: http://cKnowledge.org/request

We would like to discuss all these issues with the community to improve next AE during an open CGO-PPoPP AE discussion session on the 26th of February 2018 (17:15). Please join us and feel free to provide your feedback!

• CGO'18 website: http://cgo.org/cgo2018
• PPoPP'18 website: https://ppopp18.sigplan.org
• Artifact evaluation for systems conferences: http://cTuning.org/ae

My CNRS webcast "Enabling open and reproducible research at computer systems conferences: good, bad and ugly"

This spring I was kindly invited by Dr. Arnaud Legrand (CNRS research scientist promoting reproducible research in France) to present our practical experience while enabling open and reproducible research at computer systems conferences (good, bad and ugly).

This CNRS webinar took place in Grenoble on March 14, 2017 with a very lively audience.

You can find the following online resources related to this talk:

• Info about this presentation at GitHub
• Slides
• Notes during the talk
• Video

Microsoft sponsors non-profit cTuning foundation

We would like to thank to Microsoft for providing an Azure sponsorship for our non-profit cTuning foundation to host our public repository of cross-linked artifacts and optimization results in a unified and reusable Collective Knowledge format.

Many thanks to Dr. Aaron Smith for his assistance!

Successful PhD defense at the University of Paris-Saclay (advised by cTuning foundation members)

We would like to congratulate Abdul Memon (PhD student advised by Dr. Grigori Fursin from the cTuning foundation) for successfully defending his thesis "Crowdtuning: Towards Practical and Reproducible Auto-tuning via Crowdsourcing and Predictive Analytics" in the University of Paris-Saclay.

Most of the software, data sets and experiments are shared in a unified, reproducible and reusable way using Collective Mind framework and later converted to the new Collective Knowledge framework.

We helped prepare ACM policy on Result and Artifact Review and Badging

After arranging many Artifact Evaluations to reproduce and validate experimental results from published papers at various ACM and IEEE computer systems conferences (CGO, PPoPP, PACT, SC), we saw the need for a common reviewing and badging methodology.

In 2016, the cTuning foundation joined ACM internal workgroup on reproducibility and provided feedback and suggestions to develop a common methodology for artifact evaluation. The outcome of this collaborative effort is the common ACM policy on Result and Artifact Review and Badging published here:

• http://www.acm.org/publications/policies/artifact-review-badging

We started aligned artifact submission and reviewing procedures for computer systems conferences:

• http://cTuning.org/ae/submission.html
• http://cTuning.org/ae/reviewing.html

We expect that this document will gradually evolve based on our AE experience - please, stay tuned for more news!

Collective Knowledge summer digest is now available online

Join our experiment on public discussion of ADAPT'16 paper submissions!

Dear colleagues,

As a part of our ongoing initiative towards open, collaborative and reproducible computer systems' research, we cordially invite you to participate in the public pre-reviewing of ADAPT'16 paper submissions co-located with HiPEAC'16(6th international workshop on adaptive, self-tuning

computing systems).

Each submission is now available at ArXiv and has a separate Reddit discussion thread here:

• https://www.reddit.com/r/adaptworkshop
• https://www.reddit.com/r/reviewpapers/comments/3sogux/join_our_exciting_experiment_on_reddit

Note that several papers have shared artifacts (benchmarks, data sets, models) to help you validate presented techniques and even build upon them!

Please, feel free to comment on these papers, exchange ideas, reproduce results, suggest extensions, note missing references and related work, etc. We hope such public pre-reviewing will speed up dissemination of novel ideas while also letting authors actively engage in discussions and eventually improve their open articles before the final reviewing by the ADAPT Program Committee!

You can find more details about this publication model at http://adapt-workshop.org/motivation2016.html

Looking forward to your participation!