Task Force 6 - Subgroup 7: Benchmarking
The aim of this Task Force Subgroup is to provide a scalable and FAIR benchmarking platform for data-driven solutions with a focus on AI (especially ML) solutions. This group will also provide corresponding benchmarks, key performance indicators, benchmarking tools and services for the independent, repeatable and scalable benchmarking of data-driven (especially AI) technologies.
Moreover, it facilitates the systematic selection, improvement and objective comparison of benchmarks and generalization of knowledge from open-source benchmarking technologies.
Finally, it aims at detecting potential use cases and categories of users as well as potential synergies with existing benchmarking organizations (e.g., MLPerf) and other AI and Big Data benchmarking activities.
Benchmarks and key performance indicators are central for the use of AI solutions in industry and academia as they allow the data-driven comparison of the results of corresponding algorithms. Addressing these challenges at a European scale is central for the development of competitive benchmarking solutions that can be used to advance AI in Europe and the world.
