It had been quite a while since I last had the pleasure of returning to Dagstuhl Castle. This week is also a return to an old topic: competitions and empirical evaluations in automated reasoning.
During my PhD in the 2000s, I became interested in this issue that is anything but anecdotal: a good experimental results table can completely change whether a scientific paper gets accepted.
Back then, I realized that some papers focused on subsets of problems or used different metrics, which made it difficult to assess the methods objectively. I therefore created SATEX, a site that gathered benchmarks and solvers and presented the results objectively—a kind of SAT-flavored Kaggle (albeit very rudimentary).
From 2002 onward, I got swept up in the whirlwind of organizing the first international competitions for SAT solvers. Those were remarkable scientific and human adventures.
Today a few key principles we established with Daniel Le Berre (and others) are still in place for competitions, yet the landscape has changed dramatically. The core questions remain: how do we define which problems are worth solving or including in result tables? How do we showcase the diversity of approaches? How do we cope with the growing complexity of algorithms that makes their behavior harder to analyze? SAT solvers have become so sophisticated that their inner workings are now very difficult to grasp.
This workshop week highlighted how important competitions are within our scientific communities. We probably need more federation efforts across competitions. Bridges between problems (allowing for multiple formalisms or solvers) must be built across events.
And of course, all of this happens in the amazing Dagstuhl setting, with the joy of seeing the movers and shakers of this silent AI revolution again.
If you have never been to Dagstuhl, know that it is a unique place for research and the cross-pollination of ideas. Incidentally, INRIA is one of the main supporters of the Dagstuhl center. Keeping such places alive is crucial!

As I do in every Dagstuhl post, play “Where’s Charlie?” with photos from my previous stays
2022

2019

2017

2015

2014

2009
