New CK released with 500+ reusable, portable and customizable AI/ML components (libraries, models, data sets)

We have the pleasure to announce a new release of our open-source community-driven Collective Knowledge framework (CK) with a completely redesigned website: http://cKnowledge.org.

With CK, you can convert your code and data into unified CK components with a common Python API and JSON meta description, and share them with others via private or public repositories (e.g. GitHub). The community can then reuse your components and help adapt them to new research scenarios by extending APIs, meta descriptions and functionality. The open and decentralized nature of CK liberates the community from being locked into any proprietary tools, formats and services.

For example, you can now take advantage of over 540 CK packages shared by our community to automatically install various AI/ML frameworks and libraries (TensorFlow, TFLite, MXNet, NNVM, TVM, VTA, Caffe, Caffe2, CNTK, cuDNN, ArmCL, PyTorch), models and data sets on Linux, MacOS, Windows and Android.

You can also quickly reuse over 340 customizable CK programs from traditional systems benchmarks to emerging AI applications. This includes all workflows from the 1st ACM ReQuEST tournament to collaboratively benchmark and co-design the efficient SW/HW stack for deep learning inference from the cloud to the edge!

All CK programs automatically manage dependencies using CK packages, unified compilation and customized execution across diverse platforms, frameworks, libraries, models and data sets. Adding new CK program has also become easier: just invoke “ck add program:my-new-program” and select one of the multiple shared templates! This approach simplifies developing customizable, portable and extensible benchmarks, and can assist new benchmarking initiatives such as MLPerf.

We also continue improving our universal and ML/AI-based CK autotuner/crowd-tuner with new practical use-cases to perform multi-objective autotuning/co-design of MobileNets across the full software/hardware stack, to crowdsource benchmarking of different AI frameworks and libraries (TFlite, TensorFlow, Caffe, ArmCL) across Android devices provided by volunteers, and to automatically generate adaptive libraries!

Based on user feedback, we have introduced a virtual CK environment with over 200 CK plugins to automatically detect software and data dependencies required by CK programs and experimental workflows. We have also shared over 150 CK modules and over 50 CK productivity functions with a common API which can help you automate and unify various AI/ML/systems research tasks.

We have updated CK documentation including first steps, portable package manager and how to add your own workflows and components . We also plan to redesign our public repository with crowdsourced experiments to make more dynamic and user-friendly: http://cKnowledge.org/repo

Please join us to discuss CK and related technology at ResCuE-HPC at Supercomputing’18, the 1st workshop on reproducible, customizable and portable workflows which we co-organize with Todd Gamblin (Lawrence Livermore National Laboratory, USA), Michela Taufer (University of Delaware, USA) and Milos Puzovic (The Hartree Centre, UK).

We are now preparing many exciting CK-based projects with our academic and industrial partners around automating artifact evaluation across different AI/ML/systems conferences (SysML, CGO, PPoPP, PACT, SC), collaboratively co-designing efficient SW/HW stack for emerging AI/ML and quantum workloads, starting new ReQuEST tournaments, and much more! Please get in touch if you are interested to know more or participate!

Enjoy,

Your Collective Knowledge team.