Please always reference the paper titled "Building a Corpus of Task-based Grading and Feedback Systems for Learning and Teaching Programming" (DOI 10.3991/ijep.v12i5.31283, published in the iJEP journal) and link to https://systemscorpus.strickroth.net/. Other URLs may change, this should be the static entry point.
| System | General information | Websites | |||||||||||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| Full system name | OC | Lang | FUY | Target Group | Self-Characterization | Infowebsite | Demosystem | Source Code Repository | |||||||||||||
| C | H | I | M | S | U | X | Focus | General Characterization | Purpose Characterization | Adjectives | Tasks and Activities | Keywords | Prog.Langs | ||||||||
| 2TSW [0] [1] [2] | ITA | en | 2018 | ● | system | assessment;correction | automated;gamified | assignments | automated assessment;computer programming learning;gamification;testing | C | |||||||||||
| Algo+ [0] [1] [2] | ALG | en | 2011 | ● | ● | Assessment tool | tool | assessment | assessment;automated grading;computer education;cs1;dynamic analysis;moocs;programming assignments;static analysis | C++ | |||||||||||
| AlgoWeb [0] | BRA | en | 2010 | ● | environment | algorithms;introductory programming;learning tools;web-based learning | Structured Portuguese | ||||||||||||||
| Android Appium Grader [0] | FIN | en | 2020 | ● | Assessment System/grader | system | assessment | automated | exercises | android;automated grading;computer science education;full-stack;mobile app development;online learning;software containers;ui testing | Java | https://cloudscape.aalto.fi/ | https://github.com/aolecloudscape/android-grader | ||||||||
| APOGEE [0] [1] [2] | Automated PrOject Grading and instant fEEdback system for web computing | USA | en | 2008 | ● | Grading tool,feedback system | system,tool | Feedback;Grading | automated | projects | active learning;automated grading;automated testing;trial-and-failure | any weblanguage (J2EE and ASP.Net) | https://sites.hofstra.edu/xfu/projects/apogee/ | ||||||||
| (Arifi2015) [0] | MAR | en | 2015 | ● | Assessment tool | tool | automated program evaluation;dynamic analysis;static analysis | C | |||||||||||||
| ArTEMiS [0] [1] | AuTomated assEssmentManagement System | DEU | en | 2018 | ● | E-Assessment System | system | assessment | automated | automated feedback;learning programming;programming exercises | any | https://github.com/ls1intum/ArTEMiS | |||||||||
| ASB [0] [1] [2] [3] | Automatische Software-Bewertung | DEU | en,de | 2005 | ● | Submission system | anwendung/application | Bewertung;einreichen | automatisch | automatische bewertung;checkstyle;javafx;junit;monocle;testfx;web-anwendung;websocket | Java, JavaFX, Python, C++, Android | ||||||||||
| Ask-Elle [0] [1] [2] | NLD | en | 2010 | ● | intelligent tutor | system | assessment | automatic;intelligent;interactive | automatic assessment;functional programming;haskell;strategies | Haskell | http://ideas.cs.uu.nl/ProgTutor/ | http://ideas.cs.uu.nl/AskElle/ | |||||||||
| ASURA [0] | PRT | en | 2018 | ● | Assessment environment | environment | assessment | game-based;turnament | games;graphical feedback;learning;programming;tournament | ||||||||||||
| Asys (JavAssess) [0] [1] | ESP | en | 2015 | ● | Java | http://personales.upv.es/josilga/ASys/ | |||||||||||||||
| Athene [0] [1] | USA | en | 2007 | ● | Assessment tool | system,tool | assessment;checker;submit | automated | homework | automated assessment tools;automated feedback;computer aided instruction;computer science education | C/C++ | ||||||||||
| AuDoscore [0] | DEU | en | 2017 | ● | extension,application | grading | exercises | grading;java;scala;student code submission | Java, Scala | https://github.com/FAU-Inf2/AuDoscore | |||||||||||
| AutoGrader-1 [0] | USA | en | 2007 | ● | Grading Framework | framework | grading | automatic | assignments | automated grading;interfaces;java | Java | https://github.com/mikehelmick/AutoGrader | |||||||||
| AutoGrader-2 [0] | USA | en | 2007 | ● | Grader | autograder;validate | autograder | Java | http://cs.sou.edu/~nordquip/ag/ag.html | ||||||||||||
| AutoLEP [0] [1] [2] | CHN | en | 2004 | ● | learning and examination environment | environment,system | examination | automated | excercise | intelligent tutoring systems;interactive learning environments;programming and programming languages | C | ||||||||||
| Automata [0] [1] [2] | IND | en | 2013 | ● | evaluation platform | system,framework,platform | evaluation;grading;testing | automatically;machine learning | automatic grading;feature engineering;mooc;one-class learning;question independent learning;recruitment;supervised learning | C | https://www.aspiringminds.com | ||||||||||
| Automatic Marker [0] | ZAF | en | 2008 | ● | automation system/marker | system | assessing;marker;marking | automation | assignments | assessment;automation;interoperability;sakai | Java | ||||||||||
| AutoStyle [0] [1] | USA | en | 2015 | ● | ● | Feedback | autograding;coding style;moocs | Ruby, Python | |||||||||||||
| Autotool [0] [1] | DEU | de | 2002 | ● | system | assessment;grades | automatically | exercise,solutions | Haskell | https://gitlab.imn.htwk-leipzig.de/autotool/all0 | |||||||||||
| AWAT [0] [1] | Automated Web application Testing | USA | en | 2008 | ● | tutoring system/testing system | system | testing | automated;smart | automated testing;test cases;web applications | any weblanguage | ||||||||||
| (Ball2017) [0] | USA | en | 2017 | ● | Grader | system | autograder | autograder | Snap! | https://github.com/snap-cloud/autograder | |||||||||||
| (BenacEarle2016) [0] | ESP | en | 2016 | ● | Grading Framework | framework | grading;testing | automatic | exercises | automated assessment;java;testing | Java | ||||||||||
| bitfip [0] | CAN | en | 2016 | ● | practice tool | tool | ungraded | practice | cs1;programming practice tool;student experience | Java | https://github.com/ModSquad-AVA/BitFit | ||||||||||
| Bottlenose [0] | USA | en | 2013 | ● | Assessment tool | framework,tool | assessment;feedback;submissions | automatic | C | ||||||||||||
| (Burke2018) [0] | DEU | en | 2018 | ● | tutoring system | system,platform | intelligent | algorithm learning;language-independent programming;tutoring system | |||||||||||||
| canary-framework [0] | USA | en | 2013 | ● | Grading Framework | framework,collection | grading | automatic | automated grading;continuous integration;software engineering best practice;version control | Java, C | https://sos-cer.github.io/projects/auto-grading | ||||||||||
| checkpoint [0] [1] [2] | GBR | en | 2005 | ● | Assessment System/Framework | system,framework | assessment | automated | automated assessment;feedback;multiple attempts;plagiarism;self-paced learning;survey analysis;‘little and often’ assessment | C#, Java, Ada, C, Assembly, ASP, Unix shell scripting | |||||||||||
| cloudcoder [0] | USA | en | 2013 | ● | exercise system | platform,system | excercises | outcomes;programming exercises;student work patterns | C/C++, Java, Python, Ruby | https://github.com/cloudcoderdotorg/CloudCoder | |||||||||||
| COALA [0] | Computer Assisted Environment for Learning Algorithms | ESP | en | 2009 | ● | Tutoring environment | environment,system | adaptive environments;instructional planning;intelligent tutoring;problem-based learning | Java | ||||||||||||
| code [0] [1] | MKD | en | 2012 | ● | Assessment System | system | assessment;grading | autograding | problems | automatic assessment system;evaluation;plagiarism | C, C++, Java | ||||||||||
| code hunt [0] [1] [2] | USA | en | 2014 | ● | ● | educational game / serious game | game | educational platforms;gaming for learning;symbolic execution | C#, Java | https://www.microsoft.com/en-us/research/project/code-hunt/ | |||||||||||
| code.org [0] | PRT | en | 2020 | coding platform | platform | code.org;computational thinking;early-age;technology-enhanced learning | Something block-based | https://code.org/ | |||||||||||||
| codeinsghts [0] | PRT | en | 2018 | ● | tool | autonomously | PHP, java, Python | ||||||||||||||
| CodeLab [0] | USA | en | 2016 | ● | ● | exercise system | system,tool | interactive | excercises | Python, Java, C++, C, C#, JavaScript, VB, SQL | https://www.turingscraft.com/ | https://www.turingscraft.com/ | |||||||||
| codeOcean [0] [1] | DEU | en,de | 2016 | ● | excercise platform | platform,tool | excercises | autograder;code repository;ooc;programming;scalability;sharing | Java, Python, Ruby, JavaScript; possible diverse as runs in docker | https://github.com/openHPI/codeocean | |||||||||||
| CodeQ [0] | SVN | en | 2018 | ● | programming system | system | exercises | automatic debugging;computer-assisted instruction;intelligent tutoring systems;program synthesis | Python, Prolog | https://codeq.si/ | https://ailab.si/code/ | ||||||||||
| Coderunner [0] [1] | NZL | en | 2016 | ● | Plugin | tool,plugin | assessing | automatic | assignments | automarking;automatic assessment;learning;opengl;teaching | C, Java, Python, PHP, JavaScript and Octave (Matlab), Clojure, Websites | https://coderunner.org.nz/ | https://www.youtube.com/watch?v=I6AO5CobNyo | https://github.com/trampgeek/CodeRunner | |||||||
| codetraining [0] | MEX | en | 2017 | ● | Learning Environment | environment | authoring tool;gamification;intelligent learning environment | Java | |||||||||||||
| CodeWorkout [0] | USA | en | 2015 | ● | drill and practice system | system | drill-and-practice,exercises | codeworkout;coding;homework;practice.exam;skill development | Java, Ruby, Python | http://codeworkout.org | https://github.com/web-cat/code-workout | ||||||||||
| codewrite [0] | NZL | en | 2011 | ● | drill and practice tool | tool | testing | drill-and-practice | assessment;codewrite;drill and practice;java;syntax errors | Java | https://codewrite.cs.auckland.ac.nz/ | ||||||||||
| Codex [0] | PRT | en | 2020 | ● | system | testing | exercises | c language;haskell language;property-based testing;teaching programming | C | https://github.com/pbv/codex | |||||||||||
| (Coffman2010) [0] | USA | en | 2010 | ● | Lab | lab | virtual | lab | electronic commerce;evaluation;guided instruction;hands-on programming;software laboratory;virtual labs | any (PHP) | |||||||||||
| CourseMarker [0] [1] [2] | GBR | en | 1998 | ● | Assessment System | system | assessment | architectures for educational technology;authoring tools and methods;evaluation of cal systems;interactive learning environments;system | Java, C++ | ||||||||||||
| cpp-tutor [0] [1] | PSE | en | 2008 | ● | tutoring system | system | intelligent | c++;experiment;intelligent tutoring system;learning;programming;tutor | C++ | ||||||||||||
| CrowdSorcerer [0] | FIN | en | 2017 | ● | system | evaluating | automatically | assignments | assignment creation;automated assessment;crowdsourcing;peer review;programming | Java | https://tmc.mooc.fi/user/new | https://github.com/rage/crowdsorcerer | |||||||||
| CS50 [0] | USA | en | 2012 | ● | tool,framework | assessing;correctness | assessment;autograding;feedback | Python, C (principally "any") | https://cs50.readthedocs.io/projects/check50/ | https://github.com/cs50/check50 | |||||||||||
| CSTutor [0] [1] | AUS | en | 2013 | ● | tutoring system | system | intelligent | c#;intelligent tutoring system;natural learning process | C# | ||||||||||||
| DGRADER [0] | CHN | en | 2020 | ● | grading system/grader | system,platform | grading;judgement | automatically | C, C++, Java | ||||||||||||
| Dodona [0] [1] | BEL | en | 2016 | ● | learning environment/tutoring system | system,environment,platform | assessment;feedback | intelligent | challenge,assignments,exercise | active learning;automated assessment;classroom management;computer programming;computer-assisted learning;computer-assisted teaching;decentralized authentication;education;educational data mining;feedback;intelligent tutoring system;learning analytics;plagiarism | Bash, C, C++, C#, Haskell, Java, JavaScript, Prolog, Python, Scheme, R, SQL, HTML, … | https://dodona.be/ | https://dodona.ugent.be/ | https://github.com/dodona-edu/dodona/ | |||||||
| DOMjudge [0] | COL | en | 2018 | judge | tool | grading | automate | contest | automatic assessment;computer programming;computer science education;continuous assessment | https://www.domjudge.org/about | https://www.domjudge.org/demo | ||||||||||
| DSLab [0] | Distributed Systems Lab | ESP | en | 2012 | ● | Assessment tool/lab | tool,lab | assessment | automated | lab | automatic assessment;distributed programming assignment;online learning;students' perceptions | http://dpcs.uoc.edu/projects/dslab/ | http://dpcs.uoc.edu/projects/dslab/files/download.html | ||||||||
| DUESIE [0] | Das UEbungsSystem der Informatik Einführung | DEU | de | 2007 | ● | exercise system | system | übungs | Java, SML | ||||||||||||
| eAIXESSOR [0] | DEU | en | 2008 | ● | Assessment framework | framework | assessment | automatic | caa system;eassessment concept;elearning;feedback;software and hardware systems forweb-based education | Java, C# | |||||||||||
| EASy [0] [1] | E-Assessment System | DEU | en,de | 2013 | ● | Assessment System | platform,system | assessment | exercises | Java | |||||||||||
| easyaccept [0] [1] | BRA | en | 2004 | ● | testing tool | tool | testing | acceptance testing;software development teaching;test-driven development | Java | http://easyaccept.sourceforge.net/ | |||||||||||
| eduComponents [0] [1] [2] | DEU | de,en | 2003 | ● | Python, Haskell, Scheme, CommonLisp and Prolog, Java, Erlang, XML | https://github.com/collective/ECSpooler | |||||||||||||||
| EduJudge [0] | ESP | en | 2008 | ● | ● | system | distributed learning environments;interactive learning environments;programming and programming languages;teaching/learning strategies | Pascal, Java, C and Cpp | http://eduvalab.uva.es/en/projects/edujudge-project | ||||||||||||
| (Eggert2013) [0] | DEU | de | 2013 | ● | system | bewertung | automatisch | Java, Python | |||||||||||||
| egrader [0] | ARE | en | 2011 | ● | Grading system | system | grading | computer aided assessment;computer aided education;control dependence graphs;java;programming | Java | ||||||||||||
| Eiffel 4 MOOC [0] | CHE | en | 2013 | ● | infrastructure | infrastructure | cs1;mooc;pedagogy;spoc | Eiffel | http://se.inf.ethz.ch/data/spoc/ | ||||||||||||
| ELM-ART [0] [1] | ELM Adaptive Remote Tutor | DEU | en | 1997 | ● | Tutoring system | system | adaptive; interactive;intelligent | adaptive hypermedia;adaptive learning environment;intelligent tutoring system;learning programming | LISP | http://art.ph-freiburg.de/Lisp-Course | ||||||||||
| Evaluate [0] | PRT | en | 2010 | ● | evaluation service | service,framework | evaluation;mark and grade | contest,exercise | e-learning;interoperability;soa | ||||||||||||
| (Fangohr2015) [0] | USA | en | 2009 | ● | Feedback provisioning system/assessment system | system | Feedback;assessment;submission | automatic | automatic assessment tools;automatic feedback provision;programming education;python;self-assessment technology | Python | |||||||||||
| fit-java-tutor [0] | DEU | en | 2013 | ● | learning environment/tutor | system,middleware,environment | adaptive;intelligent | tutor | intelligent tutoring system architecture | Java | https://javafit.de/ | ||||||||||
| fitchfork [0] | ZAF | en | 2006 | ● | Assessment software | software | assessment | assessment software;automatic assessment;mooc | C++ | ||||||||||||
| flop [0] [1] | Free Laboratory Of Programming | ESP | en | 2012 | ● | Assessment system/lab | system,lab | assessment | automated | lab | automated assessment systems;flop;learning of programming;user-centered design | C,C++,Pascal,Haskell,Python, Java (GCJ) | |||||||||
| FW4Ex [0] | framework for exercises | FRA | en | 2008 | ● | Grading infrastructure/exercise framework | infrastructure,framework | grading | mechanized | grading experimentation;mechanised grading | Java, PHP, C | https://www.codegradx.org/ | |||||||||
| (Galan2019) [0] | ESP | en | 2019 | ● | Assessment System | system | assessment | automatic | automated assessment;computer programming training;distance education;online education | C++ | |||||||||||
| GAME [0] [1] [2] | Generic Automated Marking Enviroment | AUS | en | 2004 | ● | Marking Environemt | environment | assessment;marking | automatic | architectures for educational technology system;post-secondary education;programming and programming languages;teaching/learning strategies | Java, C, C++ | ||||||||||
| GAME-2 [0] | Generic Automated Marking Enviroment | AUS | en | 2007 | ● | Marking Environemt | environment | marking | automatic | area measurement;fuzzy logic;human factors;programming languages | Java, C, C++ | ||||||||||
| GAME-2+ [0] | Generic Automated Marking Enviroment | AUS | en | 2009 | ● | Marking Environemt | environment | marking | automatic | automated marking;e-learning;fuzzy logic;programming assignment | C, C++, JAVA, Delphi, PHP | ||||||||||
| GATE [0] [1] [2] [3] [4] | Generic Assessment and Testing Environment | DEU | de,en | 2009 | ● | E-Assessment System/submission system/testing environment | system,environment | assessment;feedback;testing | intelligent | exercises | code review;feedback;mass education;peer code review;peer review;peer teaching;programming education | Java,Haskell | https://cses.informatik.hu-berlin.de/de/software/details/gate/ | https://github.com/csware/si/ | |||||||
| GitGrade [0] | USA | en | 2020 | ● | Grader | platform,system | accepting;grading | automatically | assignments | computer science education;grading;technology and teaching | JavaScript (+ others?) | https://gitgrade.cs.washington.edu/ | |||||||||
| Graja [0] [1] [2] | Grading Java programs | DEU | de | 2013 | ● | Grader | grading | autograder | autobewerter;bewertungsaspekt;bewertungsmaßstab;e-assessment;grader;graja;java;proforma-aufgabenformat;programmieraufgabe | Java | http://graja.hs-hannover.de | ||||||||||
| HackerRank [0] | JAM | en | 2018 | ● | programming platform | platform | auto-grading | auto-grading | compitetive,assignments | competitive programming;managing enrolment growth | over 30 in all | https://www.hackerrank.com/ | |||||||||
| (Harley2012) [0] | CAN | en | 2012 | ● | wizard | wizard | computer programming;computer-aided assessment;e-learning;encryption;wizard | Java, C, Perl, Ruby, Python | |||||||||||||
| Hop3X [0] [1] | FRA | en | 2008 | ● | learning system | system | adaptation;domain specific modeling;learning context;learning design;model driven engineering;open learning scenario | Java | http://eiah.univ-lemans.fr/HOP3X/HOP3X.xml | ||||||||||||
| hsExprTest [0] | CZE | en | 2014 | ● | test generation | test | automatic | assignments | automatic testing;haskell;programming education;quickcheck | Haskell | https://github.com/vlstill/hsExprTest | ||||||||||
| (Huang2015) [0] | USA | en | 2015 | ● | compiling system | system | compiling;feedback;grading | automatic | java;linux;mysql;php;programming assignments;web server | Java | http://www.austinhuang.net/research | ||||||||||
| ICE [0] | Interactive C Environment | AUS | en | 2017 | ● | tool,environment,system | automated;interactive | automated assessment;c-programming;programming based learning | C | ||||||||||||
| ILE Affective Java [0] | MEX | en | 2015 | ● | Learning Environment | environment | intelligent | affective computing;affective detection;eeg recognition;face recognition;intelligent learning environment | Java | ||||||||||||
| iList [0] [1] | USA | en | 2008 | ● | tutoring system | system | intelligent | computer science education;computer-assisted instruction;constraint-based modeling;education;evaluation/methodology;intelligent tutoring systems | C++, Java subset | ||||||||||||
| incom [0] [1] | DEU | en | 2009 | ● | tutoring system | system | homework,assignments,tutoring | constraint-based approach;evaluation;intelligent tutoring systems | Prolog | https://nats-www.informatik.uni-hamburg.de/INCOM/Dokumentation | |||||||||||
| Infandango [0] | GBR | en | 2011 | ● | grading system | system | grading | automated | automatic grading;django;java;restructured-text;sphinx | Java | https://infandango.readthedocs.io | https://bitbucket.org/ewan/infandango | |||||||||
| INGInious [0] | BEL | en | 2014 | ● | ● | Grader | grader | automatic | exercises | any; Docker images for C++, Java 7/8, Scala, Python 2/3, OZ | https://inginious.org/ | https://inginious.org/ | https://github.com/UCL-INGI/INGInious | ||||||||
| IPTutor [0] | NLD | en | 2014 | ● | tutor | feedback;hints | feedback generation;programming strategies;programming tutor | Java, PHP | http://ideas.cs.uu.nl/ | ||||||||||||
| it4all [0] | DEU | de | 2018 | ● | Assessment framework | framework | assessment;korrektur/correction | e-assessment;e-learning | Python, HTML, JavaScript | http://www.is.informatik.uni-wuerzburg.de/en/research/tool-it4all/ | https://gitlab2.informatik.uni-wuerzburg.de/bje40dc/it4all | ||||||||||
| ITAP [0] | Intelligent Teaching Assistant for Programming | USA | en | 2017 | ● | Tutoring system/teaching assistant | system | intelligent | automatic hint generation;data-driven tutoring;programming tutors;solution space | Python | |||||||||||
| J-Latte [0] [1] | Java Language Acquisition Tile Tutoring Environment | NZL | en | 2007 | ● | tutoring system/environment | system,environment | intelligent | Java | ||||||||||||
| JACK [0] [1] [2] [3] [4] [5] [6] | DEU | en,de | 2008 | ● | E-Assessment System | system | assessment;marking | automated | exercise | c-test;codeverständnis;kompetenzmessung;lückentext;programmierausbildung | Java, C++, Python | http://www.s3.uni-duisburg-essen.de/jack/ | https://jack-demo.s3.uni-due.de/ | ||||||||
| Java PBL [0] | Java Problem Based Learning | SRB | en | 2010 | ● | Learning Environment | environment | intelligent tutoring systems;learning management systems;problem-based learning;web applications | Java | ||||||||||||
| javamarker [0] [1] | IRN | en | 2010 | ● | marking system/plugin | plugin,system | mark | excercises | automated assessment;computer based assessment;gui grading;gui marking;java | Java | |||||||||||
| (Jin2014) [0] | USA | en | 2014 | ● | Tutor framework | framework | automatic hint generation;intelligent tutoring systems;programming tutors | Java | |||||||||||||
| JITS [0] [1] | Java Intelligent Tutoring System | CAN | en | 2003 | ● | ● | Tutoring system | system | intelligent | assessment of programming tutors;designing programming tutors;intelligent tutoring systems;programming tutors | Java | ||||||||||
| JPL [0] | Java Programming Laboratory | AUS | en | 2013 | ● | Learning Environment | environment | lab | java programming;problem-based learning;real-time progress tracking;scaffolded development | Java | |||||||||||
| jsassess [0] | JavaScript Assessment | FIN | en | 2010 | tool | assessment | automatically | exercise | automatic assessment;javascript;programming exercises | JS | https://github.com/vkaravir/jsassess-demo | ||||||||||
| JUG [0] | JUnit Generation | USA | en | 2012 | ● | Generation system | system | feedback | automated testing;evaluation;generative programming | Java | http://www.csl.mtu.edu/~cdbrown/JUG/ | ||||||||||
| jutge.org [0] [1] | ESP | en | 2003 | ● | ● | learning environment/judge | environment, judge | judge | virtual | problem | formal verification;logic circuit design;online judge;verilog | 22 programming languages with 31 different compilers | https://jutge.org/ | ||||||||
| Jype [0] [1] | FIN | en | 2009 | ● | Exercise tool/vizualization tool | tool | excercise | automatic assessment;computer science education;jype;visual debugging | Python | ||||||||||||
| Kattis [0] | SWE | en | 2005 | ● | Assessment System | system | assessment | automated | automated assessment;competition;data structures and algorithms;feedback | C/C++, Java | https://open.kattis.com/ | ||||||||||
| (Kitaya2016) [0] | JPN | en | 2016 | ● | scoring system | system | scoring | automated | automated scoring;java programming assignments;learning systems;web systems | Java | |||||||||||
| klee [0] | USA | en | 2015 | ● | Feedback framework | framework | feedback | automated | auto grader;computer science education;concolic testing | C++ | |||||||||||
| Kodr [0] | EGY | en | 2016 | ● | learning platform | platform | evaluate | gamified | coding challenges | computer education;debugging;gamification;offline-ready;python;web-based | JavaScript, Java, Python | ||||||||||
| Labtool [0] | USA | en | 2022 | ● | Assessment tool/lab assistant | assistant | assessment;feedback;semi-automated grading | automated | automated assistant;immediate feedback;semi-automated grading | C | https://gitlab.bucknell.edu/amm042/labtool | ||||||||||
| Learn OCaml [0] [1] [2] | FRA | en | 2016 | ● | ● | learning platform | platform,application,center | automated grading;browser based education platform;functional programming;mooc;programming education;safe program introspection | OCaml | https://github.com/ocaml-sf/learn-ocaml | |||||||||||
| Learn2mine [0] | USA | en | 2013 | ● | learning system | system,environment | scaffold | gamified | assignments | data mining;data science;gamification;programming;python;r | Python, R | https://github.com/Anderson-Lab/Learn2Mine-Main | |||||||||
| (Lee2021) [0] | USA | en | 2021 | ● | tutoring system | system | evaluate;feedback;grading;hints | automated | automated grading;personalized learning;real-time feedback | Java | |||||||||||
| Llschecker [0] | DEU | de | 2005 | ● | Assessment System | system | assessment | assisted | Haskell, Scheme, CommonLisp | ||||||||||||
| LUD [0] | Let's Use Data | ESP | en | 2022 | ● | scoring system,feedback system | system | assessment;feedback;scoring | automated | assignments | assessments;automatic feedback;automatic scoring;programming | Python | |||||||||
| M-Plat [0] [1] | Multi-Programming Language Adaptive Tutor | ESP | en | 2008 | ● | tutoring system | system | adaptive;intelligent | Java | ||||||||||||
| Mailing it in [0] | CAN | en | 2005 | ● | assessment | automated | automated assessment;client-based;cs1/2;email;experience report;instructional technologies;tools | ||||||||||||||
| MATLAB Critiquer [0] | USA | en | 2019 | ● | system | critique;examine;feedback;submissions | autograder;code critiquer;engineering fundamentals;matlab | MATLAB | |||||||||||||
| MATLAB Grader [0] | NOR | en | 2020 | ● | grader | tool,environment | MATLAB | https://www.mathworks.com/products/matlab-grader.html | |||||||||||||
| (Miller2014) [0] [1] | CHE | en | 2012 | ● | ● | Grading System | system | grading | automated grading;mooc;programming education;software engineering education | Java, Scala, C++ | |||||||||||
| Moe [0] | CZE | en | 2009 | Grading System | system | grading | automatic grading;mo-eval;moe | Pascal, C/C++, C#, Haskell | http://www.ucw.cz/moe/ | http://www.ucw.cz/gitweb/?p=moe.git;a=summary | |||||||||||
| Mooshak [0] [1] [2] [3] [4] | PRT | en | 2001 | ● | contest management system | system | contest | animation;automatic evaluation;feedback;learning;programming;students | C/C++, Java | https://mooshak2.dcc.fc.up.pt/ | https://mooshak2.dcc.fc.up.pt/wp-content/uploads/test-drive.html | https://svn.dcc.fc.up.pt/projects/Mooshak/Mooshak/ | |||||||||
| Mumuki [0] | ARG | en | 2016 | ● | ● | coding tool | tool | assessment;feedback | automatic assessment;coding tools;functional programming;haskell;replication | 17 programming languages, including Haskell, Prolog, Python, JavaScript, C and Ruby | https://mumuki.io/home/ | https://github.com/mumuki | |||||||||
| (Munoz2012) [0] | ESP | en | 2008 | ● | education tool | tool | evaluation | automating | exercises | automatic evaluation;e-learning;ict on education;lcms | MathLab, C/C++ | ||||||||||
| nbgrader [0] | USA | en | 2014 | ● | tool | grading | assignments | https://github.com/jupyter/nbgrader | |||||||||||||
| NoobLab [0] [1] [2] [3] | GBR | en | 2010 | ● | Learning Environment/development environment | environment,platform | intelligent | automated assessment;gamification;learning;learning analytics;plagiarism detection;programming;teaching;virtual environment | Python, Java, JavaScript, PHP | https://www.nooblab.com/nooblab/ | |||||||||||
| OASIS [0] [1] | Online Automated in-class Instant Assessment System | USA | en | 2010 | ● | Assessment System | system | assessment | automated | cs1;inquiry-based learning;java;online assessment | Java | ||||||||||
| OK [0] | USA | en | 2016 | ● | hint generation system | system | hint | automated tutor;computer-aided education;program analysis;program synthesis | Python Scheme | https://okpy.org | https://github.com/okpy/ok | ||||||||||
| (Oliveira2018) [0] | BRA | en | 2018 | ● | Teaching tool | tool,environment | automated;interactive | e-learning;interactive didactic material;teaching of programming | Potigol | ||||||||||||
| OnExSy [0] | online exercise system | DEU | en | 2009 | ● | exercise system | system | correction;submit | automatic | exercises,solutions | automatic correction;cal;cat;computer aided learning;computer aided teaching;online exercise system (onexsy);submission;web-based exercise tools | Assembler | |||||||||
| OOPS [0] | Object Oriented Programming System | ESP | en | 2009 | ● | Problem-solving environment/programming system | environment,tool,system | problem-solving,exercises | assessment for learning;blended learning;constraint-based modeling;learning reinforcement;self-assessment in complex domains | Pseudocode | |||||||||||
| Oto [0] [1] | CAN | en | 2008 | ● | marking tool | tool | marking;submission | assignment | automated marking;domain-specific language;educational software;scripting | Java, C, MPD, Assembler | http://oto.uqam.ca/ | ||||||||||
| PABS [0] [1] | ProgrammierAufgaben BewertungsSystem | DEU | en,de | 2010 | ● | feedback system, grading system | system | feedback;submit | automatically | assignment,solutions | automatic assessment;e-learning;feedback | Java, Scala, Groovy | |||||||||
| PCRS [0] [1] | Python Classroom Response System | CAN | en | 2013 | ● | classroom response system | system,tool | classroom setting | active learning;cs1;pair programming;peer investigation | Python, C, Java | https://mcs.utm.utoronto.ca/~pcrs/pcrs/ | ||||||||||
| Pedal [0] | USA | en | 2020 | ● | feedback system | infrastructure,library | feedback | automated | cs education;feedback architecture;immediate feedback;student model | Python | https://github.com/pedal-edu/pedal/ | ||||||||||
| Petcha [0] | Programming Exercises TeaCHing Assistant | PRT | en | 2012 | ● | Teaching Assistant | automated | automatic evaluation;interoperability;learning objects;programming exercises;teaching assistant | C# | https://sites.google.com/site/elspectoo/home/tools/petcha | |||||||||||
| pex4fun [0] [1] | USA | en | 2010 | ● | ● | educational gaming environment | game, environment | C#, VisualBasic, F# | https://www.microsoft.com/en-us/research/project/pex4fun/ | ||||||||||||
| PHP ITS [0] | AUS | en | 2012 | ● | tutoring system | system | intelligent | intelligent tutoring system;knowledge base;php;program analysis | PHP | ||||||||||||
| PLAY & CODE [0] | DEU | de | 2019 | learning platform | platform | serious gaming | grundlagen der programmierung;javascript;lernplattform;serious games | JavaScript | |||||||||||||
| PLWeb [0] | TWN | en | 2013 | ● | Exercise Management System | system | exercise,practice | architectures for educational technology system;computer science education;exercises design;plagiarism detection;programming;programming learning tools | Java | ||||||||||||
| Praktomat [0] [1] [2] [3] [4] | DEU | de,en | 1999 | ● | system | grader;submission;test | automated | programmierausbildung;software-review;software-testen | Java, C/C++, Fortran, Haskell, Python, R, Isabelle | https://pp.ipd.kit.edu/projects/praktomat/praktomat.php | https://github.com/KITPraktomatTeam/Praktomat/ | ||||||||||
| ProgEdu [0] | TWN | en | 2017 | ● | Assessment platform | platform | assessment | automated | automatic assessment;e-learning;git;software quality;version control system | ||||||||||||
| ProgTest [0] [1] | BRA | en | 2011 | ● | tool | evaluation;submission;testing | automatic | practical, assignments | automatic assessment tools;novice programmers;programming assignments;software testing | Java, C | |||||||||||
| Prutor [0] | PRogramming tUTOR | IND | en | 2016 | ● | Tutoring system | system,platform | automated tutors;education;intelligent tutoring systems;programming;programming languages | C | https://www.cse.iitk.ac.in/users/karkare/prutor/ | http://prutor.com | ||||||||||
| PyTA [0] | CAN | en | 2018 | ● | wrapper, analysis tool | wrapper,tool | checks;analysis | Python | http://www.cs.toronto.edu/~david/pyta/quick_start.html | https://github.com/pyta-uoft/pyta | |||||||||||
| pythia [0] [1] | BEL | en | 2012 | ● | learning platform/grader | platform,framework | grader;testing | intelligent | code grader;education;unit testing | Python | http://www.pythia-project.org/ | https://github.com/pythia-project/pythia | |||||||||
| Pythy [0] | USA | en | 2013 | ● | programming environment | environment | browser;interactive;media computation;online;python;skulpt;web-based | Python | https://github.com/web-cat/pythy | ||||||||||||
| qtype_junittest [0] | DEU | de | 2019 | ● | plugin | webservice,plugin | assessing;submitted | solutions | automated code evaluation;feedback generation;grading;java;java class isolation;junit;learning management system;moodle;signature validation;web service | Java | |||||||||||
| Quimera [0] [1] | PRT | en | 2012 | ● | contest management system/judge | system,judge | assessing;judge | automatic | contest,exercises | domain specific languages;dsl;dynamic analysis;grading systems | C | ||||||||||
| Quizly [0] | ITA | en | 2015 | Assessment platform | platform | assessment;grading | automatically | exercises | assessment;assessment tools and platforms;computer science curricula;visual languages | Blockly / App Inventor | https://github.com/ram8647/quizly | ||||||||||
| (Rahman2007) [0] [1] | MYS | en | 2007 | ● | Assessment application | application | assessment | automated | automated programming assessment;non-structural similarity;pseudo-code comparison;pseudo-code generator;static analysis | C | |||||||||||
| reveal [0] | USA | en | 2018 | ● | exam environment | environment | exam | assessment;autograding;exam;programming | Ruby | https://github.com/ace-lab/reveal | |||||||||||
| RPT [0] | Refactoring Programming Tutor | NLD | en | 2021 | ● | system | feedback;hints | practice,exercises | code quality;learning programming;refactoring;tutoring systems | Java,Python,Kotlin,Javascript | https://ideas.science.uu.nl/rpt/ | ||||||||||
| (Ru2015) [0] | FRA | en | 2014 | ● | Assessment System | system | assessment | automatic | assignments | automatic assessment;programming;tests;web-based system | C, Java | ||||||||||
| SAC [0] | System for Automated Assistance in Correction of Programming Exercises | ESP | en | 2008 | ● | Framework | platform,system | correction/validation | automated | computer aided assessment;computer science;education;source code evaluation | Java | ||||||||||
| SAM [0] | Seeded Auto Marker | AUS | en | 2014 | ● | Assessment System | system | assessment;marker;submissions | automated | automated assessment;technology in education;unit testing | Java | https://bitbucket.org/samuelbr/automarker-inb370 | |||||||||
| SAUCE [0] [1] | System for AUtomated Code Evaluation | DEU | en | 2015 | ● | Assessment tool/teaching and learning tool | tool,application,system | assessment;evaluation | automated;interactive | black box testing;parallel programming;teaching and learning;web application | Python, C++, CUDA | http://sauce.informatik.uni-mainz.de/ | http://sauce.informatik.uni-mainz.de/events/demo | https://github.com/moschlar/SAUCE | |||||||
| (Seemann2014) [0] | DEU | en | 2014 | ● | system | validate | automatically | exercises | assessment;computer programming;grading;online courses;teaching;tutoring;unit testing | Java | |||||||||||
| Semi-Marker [0] | CHN | en | 2008 | ● | Assessment System | tool,system | assessment;marker | automatic | assembly language;assess program;assessing strategy;assessment tool | Assembly | |||||||||||
| Senecode [0] | COL | en | 2019 | ● | feedback platform | platform | feedback-oriented;graded | automatically | assignments | automated grading;deliberate practice;feedback;programming;python | Python | ||||||||||
| Siette [0] [1] | ESP | en | 1998 | ● | Assessment environment | environment,system | assessment;evaluation | automatic | assignments | automatic assessment;computerized adaptive testing;item response theory | C, Java, Haskell | https://wiki.siette.org/doku.php?id=:en:start | |||||||||
| Sifu [0] [1] | PRT | en | 2020 | ● | awareness platform,serious game | platform,tool | assessment | automated;intelligent;serious game | challenge | awareness;capture-the-flag;cybersecurity;education;industry;intelligent coach;programming;secure coding;teaching;training | C, Java | ||||||||||
| (Singh2013) [0] | USA | en | 2013 | ● | feedback | automatically | problems | automated grading;computer-aided education;program synthesis | Python, C# | ||||||||||||
| SpecCheck [0] | USA | en | 2012 | ● | unit test generator | automatically | homework | automated grading;interface conformance;large classes | Java | ||||||||||||
| (Su2016) [0] | CHN | en | 2016 | ● | teaching system,judging system | platform,system | grade;judging | automatically;interactive | online judge system;software fault localization;test cases;visualization | C | |||||||||||
| Subato [0] | DEU | de | 2018 | ● | programming system | application,system | e-learning;programmierausbildung | C, Java, Python, Octave, Haskell | https://procomp.cs.hs-rm.de/subato/static/help_de.html | http://subato.org | https://gitlab.com/Sveneric/subato/tree/master | ||||||||||
| Submitty [0] [1] [2] [3] [4] | USA | en | 2014 | ● | management platform | platform | grading;submission | automated | assignments | autograding;autograding platforms;information technology | Python, C/C++, OpenGL in C++, Java, Scheme, Prolog, SQL, SPIM, and anything available on GNU / Linux | https://submitty.org/ | https://github.com/Submitty/ | ||||||||
| Svetovid [0] [1] [2] | SRB | en | 2008 | ● | development system/submission system | system | submission | interactive | development | computer assisted testing;computer supported learning;programming and problem solving | Java, MODULA-2, Scheme, evtl. auch weitere | ||||||||||
| t-flip [0] | NOR | en | 2016 | ● | system | assessment | automatic assessment;education;programming;test-driven development | Java | http://tflip.uia.no/ | ||||||||||||
| testmycode [0] [1] | FIN | en | 2013 | ● | Assessment System | system,service | assessment;feedback | automated | exercises | automatic assessment;extreme apprenticeship;programming;situated learning;testing;verification | Java | http://github.com/testmycode | |||||||||
| TEx-Sys [0] [1] [2] | Tutor–Expert System | HRV | en | 1992 | ● | Tutoring system | system | intelligent | architectures for educational technology system;authoring tools and methods;distributed learning environments;evaluation methodologies;intelligent tutoring systems | QBASIC, C, Pascal | |||||||||||
| URI Online Judge [0] [1] | BRA | en | 2013 | ● | judge | environment | exercises | academic tool;algorithms;online judge;programming practice | Java, C++ | ||||||||||||
| UVA OnlineJudge [0] | ESP | en | 2008 | contest system/judging system | system | judging | automatic | contest | competitive learning;informatics;online judge;programming contests | ANSI C, Java, C++ C++11, PASCAL, PYTH3 | https://uva.onlinejudge.org/index.php | https://uva.onlinejudge.org/index.php | |||||||||
| VILLE [0] [1] [2] [3] [4] | FIN | en | 2007 | ● | learning environment | tool,environment,framework | visualization | exercise-based | automatic assessment;novice programming;program visualization;teaching programming | Java, C++, JavaScript, Python, PHP | http://ville.cs.utu.fi | https://github.com/villeteam/vexer | |||||||||
| Viope [0] [1] | FIN | en | 2002 | ● | learning platform | environment,tool | course redesign;cs1;hygiene factors;intrinsic and extrinsic motivators;systems approach;theory of constraints;two-factor theory | Java, C | https://www.viope.com/ | ||||||||||||
| ViPLab [0] [1] [2] [3] | Virtual Programming Laboratory | DEU | en,de | 2010 | ● | Lab/plugin | plugin | virtual | lab | eexam;elearning management system;virtual laboratory | C/C++, Dumux, MATLAB and Octave | https://www.tik.uni-stuttgart.de/forschung/projekte/vip/ | https://www.tik.uni-stuttgart.de/forschung/projekte/vip/beispiel/ | ||||||||
| ViPS [0] [1] [2] | virtuelles Prüfungssystem | DEU | de | 2004 | ● | examination system | system | exam | virtual | Prolog, Lisp, Haskell, Octave | https://develop.studip.de/studip/plugins.php/pluginmarket/presenting/details/81097da5ef66a002998b75d5eeece1f0 | https://mvc.ikw.uni-osnabrueck.de/vips/vips.php?adm=1&mgc=!ViPS!&srv=prolog | |||||||||
| Virtual-C IDE [0] [1] | DEU | en | 2015 | ● | testing framework | framework | assessment;testing | exercises | automated assessment system;computer science education;computer-supported collaborative learning;programming environments;secure programming;teaching programming;tool based learning | C | https://sites.google.com/site/virtualcide/ | ||||||||||
| VPL [0] [1] [2] | Virtual Programming Lab | ESP | en | 2011 | ● | Lab | laboratory | virtual | lab | automatic assessment;plagiarism;programming learning;virtual lab | Ada, Assembly, C, C++, C#, FORTRAN, Haskell, java, Octave, Pascal, Perl, PHP, Prolog, Python, Ruby, Scheme, SQL and VHDL | http://vpl.dis.ulpgc.es/ | https://github.com/jcrodriguez-dis/moodle-mod_vpl | ||||||||
| VPL-Plugin [0] | EST | en | 2017 | ● | plugin | plugin | automatic assessment;automatic grading;computer graphics;image recognition;mooc;programming | Python | |||||||||||||
| Web-CAT [0] [1] [2] [3] [4] [5] [6] | Web-based Center for Automated Testing | USA | en | 2003 | ● | tool,center | grades;testing | automated | adherence;automated testing;instructional technology;test-driven development (tdd);test-first;unit testing | Java, Scheme, Python, Prolog, Pascal, Haskell, C++ | http://web-cat.org/ | https://github.com/web-cat | |||||||||
| WebIDE [0] [1] [2] | USA | en | 2010 | ● | development environment/tutoring system | environment,system,framework | feedback | intelligent;interactive | development | computer science education;cs0;cs1;intelligent tutor | C, Java, Python, Ruby | http://www.web-ide.org/ | http://www.web-ide.org/java | ||||||||
| WebLab [0] | NLD | en | 2010 | ● | Learning Management System | system | concepts of programming languages;definitional interpreters;scala;teaching;testing | Scala, Java, Python, JavaScript, C | https://weblab.tudelft.nl/wl101/ | ||||||||||||
| WebTA [0] [1] | USA | en | 2013 | ● | tool | critique | automatic;interactive | autograder;critiquing systems;cs1;design patterns | Java | ||||||||||||
| WebTasks [0] | DEU | en | 2006 | ● | platform | submitting;testing | discussing,solutions | java;online submission;programming tasks;testing | Java | ||||||||||||
| WeBWorK-JAG [0] [1] [2] | Java Auto Grader | USA | en | 2001 | ● | Assessment System/programming platform/grader | system,environment | assessment;grader | automated grading;eclipse;java;junit;open source;peer-review;testing | Java, Python, SML | http://www.csis.pace.edu/~scharff/webwork/ | ||||||||||
| (Wilcox2015) [0] | USA | en | 2013 | ● | automated assessment;automated grading;peer instruction | Java | |||||||||||||||
| xLx [0] [1] [2] | eXtreme e-Learning eXperience | DEU | en,de | 2002 | ● | learning platform | platform | exercises | automatic pre-correction;blended learning;e-learning;programming exercises | Java | |||||||||||
| YOJ [0] | CHN | en | 2012 | ● | Judge | system,judge | judge | practice,homework | computer programming;online judge;pedagogy | C | |||||||||||
The corpus is based on a systematic review of major sources of publications. The coverage will increase in further versions. Click here for an overview table on the current source coverage.