Intelligent Grading Platforms

At a glance

Intelligent grading platforms such as Gradescope, Crowdmark, and Akindi provide computer-aided assistance to reduce the time and challenges of the grading process and enable instructors in focusing on providing detailed, richer feedback.

Gradescope

gradescope

Crowdmark

crowdmark

Akindi

Akindi

Who to contact

Learning Innovation and Faculty Engagement: Gemma Henderson and Amanda Valdespino

Usage scenarios

The following scenario is generated from University of Miami.

Dr. Arun Malhotra, Associate Professor of Biochemistry and Molecular Biology

During the Spring 2019 semester, Dr. Arun Malhotra implemented a free version of the intelligent grading platform, Gradescope within the undergraduate level course BMB 507. Protein Structure, Function, and Biology, with 16 students enrolled. Within this course, students are assessed through homework assignments and three paper-based exams where students respond to short-answer questions, often requiring at least a paragraph per question.

After learning about Gradescope, Dr. Malhotra went through the set-up process to create a course, assignments, and add a student roster to the grading platform.  Once the paper-based exam was delivered in class, Dr. Malhotra scanned the papers and uploaded them to Gradescope for review. Aside from exams, students also submitted two problem sets on Gradescope, which required students to answer each question on a separate blank sheet, and upload answers as images (one per question), or as a single .PDF file.

A question-by-question grading approach allowed Dr. Malhotra to assign a rubric criteria to each question and use the criteria to grade each question, finding the transition easier to review and grade, than by per student. Due to transparency of the criteria, students could review both the comments about their exams, but also establish where points were provided/deducted.

Overall, Dr. Malhotra confirmed that while less time was taken to grade, yet initial set-up of criteria and scanning did take some time. Due to the success of using Gradescope, Dr. Malhotra is interested in using this again next semester and expanding its use across multiple Biochemistry and Molecular Biology courses.

References

  1. Blumenstyk, G. (2018, April 8). Can Artificial Intelligence Make Teaching More Personal? The Chronicle of Higher Education. Retrieved from https://www.chronicle.com/article/Can-Artificial-Intelligence/243023
  2. Bode, A. M. (2018). Integrating Collaborative Online Grading Platforms into the Coordination of Calculus: A Case Study. PRIMUS, 28(6), 550–561. https://doi.org/10.1080/10511970.2017.1388317
  3. Chen, Y.-Y., Liu, C.-L., Chang, T.-H., & Lee, C.-H. (2010). An Unsupervised Automated Essay Scoring System. IEEE Intelligent Systems. https://doi.org/10.1109/MIS.2010.3
  4. Goc, R. (2018). Improve Feedback to Students and Decrease Time Spent Grading: Weinberg College - Northwestern University. Retrieved June 28, 2019, from https://www.weinberg.northwestern.edu/weinbergit/about/news/fall-2018/crowdmark-newsletter.html
  5. Khare, K., Stewart, B., & Khare, A. (2018). Artificial Intelligence and the Student Experience: An Institutional Perspective. IAFOR Journal of Education, 6(3), 63–78. https://doi.org/10.22492/ije.6.3.04
  6. Ling, L., & Tan, C. W. (2018). Human-Assisted Computation for Auto-Grading. 2018 IEEE International Conference on Data Mining Workshops (ICDMW), 360–364. https://doi.org/10.1109/ICDMW.2018.00059
  7. Page, E. B. (1994). Computer Grading of Student Prose, Using Modern Concepts and Software. The Journal of Experimental Education, 62(2), 127–142. https://doi.org/10.1080/00220973.1994.9943835
  8. Pauna, M. (2017). Calculus Courses’ Assessment Data. Journal of Learning Analytics, 4(2), 12–21. https://doi.org/10.18608/jla.2017.42.3
  9. Shehab, A., Faroun, M., & Rashad, M. (2018). An Automatic Arabic Essay Grading System based on Text Similarity Algorithms. International Journal of Advanced Computer Science and Applications, 9(3). https://doi.org/10.14569/IJACSA.2018.090337
  10. Vahid, F., & Lysecky, R. (2019). Auto-Graded Programming Labs: Dos and Don’ts for Less-Stressed Higher-Performing Students, Reduced Grading Time, and Happier Teachers,. Proceedings of the 50th ACM Technical Symposium on Computer Science Education  - SIGCSE ’19, 1250–1250. https://doi.org/10.1145/3287324.3293727
  11. Zhang, Y., Shah, R., & Chi, M. (2016). Deep Learning + Student Modeling + Clustering: A Recipe for Effective Automatic Short Answer Grading. Retrieved from https://eric.ed.gov/?id=ED592661

Research Team

Gemma Henderson: Senior Instructional Designer, Academic Technologies

Amanda Valdespino: Instructional Designer, Academic Technologies

What is it?

Designed to accommodate a common paper-grading workflow, Gradescope, Crowdmark, and Akindi are intelligent grading platforms that digitize and process paper-based assessment online to allow for a faster grading process. Assessments are scanned and uploaded, responses are categorized, and a rubric is created by the assigned grader. Intelligent grading can reveal valuable statistics about students’ work and provide a path for students to request re-grades or further assessment. These platforms potentially provide faster student feedback and standardize the grading process.

How does it work?

Gradescope and Crowdmark share similar features and workflows, while Akindi is positioned as a Scantron alternative structured by its bubble sheet templates and auto-grading platform. Each tool has its own unique set up procedures and recommendations, but common workflows are detailed below. 

  1. Create a course and populate the student roster: Without integration with the Learning Management System (e.g. Blackboard Learn), first, create a course and add identifiable information. The student roster is then generally populated through uploading a course roster manually through a .csv file. Alternatively, Gradescope allows instructors to generate an ‘enrollment code’ for students to sign up for the course.
  2. Create an assessment: There are two types of assessment workflows, ‘administered assessment’ managed by instructors who upload the scans of student work, or ‘assigned assessment’ where students submit their own scan or image of completed work. Exams are usually ‘administered’ while homework assignments are ‘assigned.’ Before selecting the type of assessment, instructors are recommended to follow the formatting guidance or templates to optimize optical character recognition (OCR) and AI-assisted grading. Assessments usually require uploading a blank copy of question, adding fields to indicate student name, location of questions and their point values.
  3. Managing submissions: For administered assessments, instructors upload all scans of submissions through a mobile device, or .PDF file, and the platform will split and process them according to the structure of the template PDF, and assign them to a student automatically. For assigned assessments, submissions are processed as each student uploads their images or .PDF files through the mobile or web. For bubble-sheet assessments, submissions are automatically processed and graded by the platform using an answer key, with discrepancies flagged for review by an instructor. Unassigned students will require instructor review. 
  4. Grading submissions: Once assessments are processed, grading occurs on a question-by-question basis for an instructor to review. Instructors can review each question and add descriptive rubrics or comments during the grading process, which can be re-applied throughout the selected assessment and imported into other assessments at a later stage. AI-assisted grading in Gradescope occurs by grouping or ‘clustering’ similar student answers together, for review by the instructor to associate point values and feedback. Annotation tools provide instructors to add additional feedback directly on the document (text, drawing, highlighting). Once grading is complete and reviewed, grades are published with students notified via email to review. With paper-based assessments digitized, performance data is also available on a per question (or rubric basis).

Who’s doing it?

The use of intelligent grading platforms have been growing among universities and are being implemented in classrooms both with small and large student enrollments, although their popularity is still primarily in STEM related courses. In an effort to improve retention and success rates of their Calculus I courses, The University of Illinois at Chicago used Crowdmark and Gradescope to increase their grading and feedback (Bode, 2018). The use of these intelligent grading platforms allowed faculty to grade at a faster rate, and also better monitor their grading in terms of consistency and providing feedback. Therefore, the results were not only higher success rates for students in Calculus I, but also an improvement in the quality of grading from faculty.

Most responses from instructors that utilize intelligent grading tools are positive. Faculty members at the Weinberg College of Northwestern University have found Crowdmark a beneficial platform that grants the grading process to be more structured as answers are automatically grouped together, which results in less time trying to organize assessments (Goc, 2018). Faculty are also able to provide detailed feedback that can be reused through a common  “comment bank” that other assigned graders can view. This cuts time in writing the same comment repeatedly and it also helps assigned graders better identify common errors in student’s work.

Due to increased grading productivity, faculty are able to schedule more assignments/assessments than they typically would if grading the traditional way. Alegra Eroy-Reveles, an assistant professor of chemistry and biochemistry at San Francisco State University, found grading much easier with Gradescope and changed her course from including three major exams to eight smaller exams (Blumenstyk, 2018). With its increased usage, university  IT departments have created tutorials and webpages that center around recently adopted intelligent grading platforms. The University of Tennessee has included an entire section on Akindi complete with a list of how it is used and training videos from Akindi.

Why is it significant?

Digital grading or computer-assisted grading is not a relatively new concept. Educators and researchers have been engaged in finding ways for computers to automatically score assessment, since the early 1960s (Page, 1994; Khare, Stewart, & Khare, 2018). For instance, Project Essay Grade (PEG) was initially formed to evaluate and provide feedback on prose submissions of high school students (Page, 1994). There are also multiple explorations by educators employing algorithmic grading to reduce the feedback-grading loop (Chen, Liu, Chang, & Lee, 2010; Ling & Tan, 2018; Pauna, 2017; Shehab, Faroun, & Rashad, 2018; Vahid & Lysecky, 2019; Zhang, Shah, & Chi, 2016). Similarly, Gradescope and Crowdmark were co-founded by educators and teaching assistants within higher education institutions, to respond larger classes and ensure collaborative grading could occur. For decades, learning management systems (LMS) such as Blackboard Learn, Canvas, Moodle, offer tools to create and manage digital assessments, that include multiple question types, accept multiple file formats and include mobile-friendly grading views. These systems also have similar grading workflows to intelligent grading platforms where you can grade on a student-to-student basis or a question-to-question basis. However, this approach often requires manually converting paper based exams into digitised versions, and implementing digital proctoring procedures for exams, often requiring software to prevent access to other applications on the computer. The evolution and introduction of quick-to-start and user-friendly tools like Gradescope, Crowdmark and Akindi allow educators to digitize existing paper-based exams and assign scoring and criteria for each assessment question.

What are the downsides?

  • Initial set-up: Similar to any learning platform that creates student assignments, manages grades and offers feedback mechanisms, intelligent grading platforms require some initial set-up and configuration. In consequence,  these platforms may not necessarily save time grading, particularly for smaller classes with less paper-based assessments. For instance, paper assignments will often need formatting to the platforms’ standard, before printing and digitizing. 
  • STEM-focused: Intelligent grading platforms are also currently aimed towards STEM disciplines and their associated assessment types. For instance, AI-assisted grouping of answers in Gradescope is also limited to multiple choice and fill in the blank (text or math-based) question types while programming assignments 
  • Limited assessment types: There are limited avenues for peer-feedback or digital assessments within the platforms that are not paper-based. 
  • Ad-hoc rubrics: The process also focuses on providing rubric criteria on a question by question basis, which is not always available to students beforehand in the system, and still requires an instructor to guide and provide this to students. 
  • External grades: If an intelligent grading platform is used for only some assignments and exams, students will often rely on another source to review other and final grades. As learning management systems already offer a central gradebook connected to the student roster, using an additional grading platform requires integration with the LMS or the exporting and importing of grades to allow students to review final grades. In particular, if students are paying for an additional grading platform, these graded assignments are potentially highlighted as more important than formative exercises that may be ungraded.
  • Limited educational research: While there are a number of institutions who have adopted intelligent grading tools, there are still limited peer-reviewed studies that compare intelligent grading tools to existing practices, particularly outside disciplines of computer science and mathematics.
  • Data Management: Instructors should also consider the implications of uploading student work and grades. Students should be aware that their student data (grades, email) is held by a third-party tool, generally governed by the platform according to the Family Educational Rights and Privacy Act (FERPA), although some institutions initiate additional agreements.

Where is it going?

Gradescope, Crowdmark and Akindi are proactive and transparent when engaging with their users' feedback (instructors, TAs and administrators) to shape the development of their platforms. Gradescope's roadmap invites users to engage with suggestions and developments of the platform via a Trello board. In consideration that Gradescope is part of the plagiarism detection powerhouse Turnitin, it’s not surprising that features such as ‘Code Similarity’ have emerged, where instructors can compare programming assignments to other students enrolled within the course. Alternatively, Crowdmark is focusing on the development of administrative dashboards, group assignments and remote exam proctoring. Akindi is catching up with Gradescope and Crowdmark, offering a collaboration feature allowing multiple TAs and other instructors to grade in courses or sections. As intelligent grading platforms are web-based applications that require internet connection, providing mobile-based companion apps for offline grading is certainly on the roadmap for most applications.

What are the implications for teaching and learning?

Richer feedback: Intelligent grading platforms are promoted to alleviate the administrative burden of the grading process, for instructors to focus on and share richer feedback. Due to existing administrative and time constraints with traditional grading, students may not receive timely or detailed feedback, disconnected from the grading criteria. With intelligent grading platforms, instructors can use existing rubrics and comments developed over time to respond and aid students in improving their performance in class. Assigned graders can also provide detailed comments that can include photos, links and further breakdowns of responses like complex formulas, something that might not be possible when trying to grade and provide feedback within the margins of paper-based assessments. Assessments can then be sent directly back to students digitally to have a better reflection where they need to improve. 

Statistical reporting: Similar to other learning platforms (e.g Blackboard Learn’s Item Analysis) intelligent grading platforms provide in-depth statistical reporting and visualizations about grades, question scores, and grading time, that are easy to access. Faculty can detect which topics might need further instruction, as well as monitor the progress of assigned graders to see if they are matching the intended timeline.  

Organized grading workflow: Assessments are auto-assigned to students, and organized on a question-by-question basis, no matter if it is a bubble sheet, fill in the blank, or short response. Regrade requests are also possible within some of the platforms, allowing students to specify particular assessments and assigned graders can review where a human or computer error has occurred.

Collaborative grading: Faculty or students can upload assessments, and multiple assigned graders are able to collaborate at any distance or time while still all adhering to the same existing rubric and feedback comments. If one assigned grader decides to make a change to the rubric, it applies to all student assessments across the board, and all assigned graders are able to view the change, creating a grading environment that is fair to students.