A research seminar: Predicting success, excellence, and retention from students' early course performance: progress results from a machine learning-based decision support system in a first year tertiary education programme

A research seminar: Predicting success, excellence, and retention from students’ early course performance: progress results from a machine learning-based decision support system in a first year tertiary education programme

Wednesday 27 April, 2011, 12 noon
Location: Room 172-4024, Building 172, Unitec Institute of Technology, Carrington Road, Mt Albert

Peter J Mellalieu

Department of Management & Marketing
Faculty of Creative Industries and Business
Unitec Institute of Technology, Auckland, New Zealand

Higher educational institutions are focussing increased attention on identifying which students are likely to succeed - or fail - in their tertiary studies. Historically, academics have been keen to identify the ‘bright young things’ they view as prospects for recruiting to postgraduate courses. Less attention has been paid to those students who fall by the wayside: there have been plenty of ambitious and talented students to take their place. More recently, institutions have been obliged to pay attention to identifying those students ‘at risk’ of failure or marginal grades and conducting re-mediative activity. For instance, Culver (2010, 2011) reports on the services provided by the Noel-Levitz consultancy for improving institutional retention in North America. Culver argues that the business case for active management of student retention is compelling.

New Zealand higher educational institutions are beginning to recognise the need to follow the trend to manage student retention and success using approaches similar to those adopted by North American institutions. The driver for the New Zealand initiative is that government funding for higher education is increasingly being redirected towards a focus on outputs rather than inputs. Specifically, this entails a refocus of government funding towards funding student completions of academic programmes rather than simply student enrollments. It is noteworthy that in New Zealand, around one-half of higher education is funded by government funds.

Given this context, I constructed a prototype Decision Support System (DSS) that provides a student the means to predict their personal academic success and final grade as they progress through their course. The DSS is implemented as an interlocked series of spreadsheets, known as ReXS - for Retention, Excellence, and Success (Mellalieu, 2011a,b).

Inputs to ReXS include the student’s grade achieved on assignment components throughout the period of the course, demographic data, the student’s five talents as identified by the Gallup StrengthsFinder 2.0 instrument, and other psychometric data. The prediction system underpinning the DSS is based on several rules and regression equations derived from a test data set of student results from a previous delivery of the course in 2010. A schematic overview of the inter-relationship between several pertinent factors is presented in Mellalieu (2010 f). Figure 1 presents a snapshot of the dashboard user interface for the ReXS DSS.

A machine learning/data mining investigation using the WEKA Explorer workbench (Witten & Frank 2005; Hall, Frank et al, 2009) identified the feasibility of predicting accurately students’ final course performance from formative and summative assessments conducted within the first three weeks of a 12-week semester. The prediction system underpinning ReXS was subsequently derived from a set of rules and regression equations determined using WEKA Explorer (Mellalieu 2010 a,b,c,d,e,f,g).

Figure 1: Snapshot of the dashboard for ReXS

The course is a first year tertiary education course in innovation and entrepreneurship that is compulsory for all students in the management and marketing majors of a Bachelor of Business programme at Unitec Institute of Technology. The DSS outputs a student’s estimated grade and class percentile ranking at the start of the course (Week 1). These statistics are updated following assessments submitted by the student in Week 3, Week 6 (mid-way), and Week 12. A ‘last chance’ final assessment occurs during the examination period, Week 14, the result for which can be predicted from all the student’s previous assessment results… and other demographic data. The DSS output also includes a measure of the precision of the estimate, such as the statistical standard error of the grade estimate. As the student progresses through the course, the precision improves, since there are an increased number of data items upon which to make the prediction.

A most potent indicator of overall course performance revealed by the data mining exercises identified the crucial importance of a student’s ability to write formal academic English in response to a written case study assignment. This finding suggests that if a student undertakes personal coaching in the antecedents required to rite gude inglish [sic] then their chances of achieving an overall higher grade will increase. I suspect these antecedent competencies include: reading for comprehension, paraphrasing information, writing persuasive and logical arguments, selecting appropriate words to express thoughts, writing Global English sentence structures, paragraphing using topic sentences, and presenting/organising arguments and evidence in an appropriate genre such as a formal business report (Mellalieu 2007a,b,c, 2008, 2010 g,h). The foregoing are a selection of core, generic academic competencies that might well underpin success in many other business and tertiary education courses. Consequently, a DSS that provides students with ‘early warning’ of the likelihood of their academic success or failure based on an early assessment of these competencies may encourage students, their instructors, and advisors to take early, proactive action to remedy deficiencies both within and outside the classroom.

The machine learning analysis also revealed unexpected insights into the level of academic performance of a student team assignment. These results will be presented at the seminar.

Results from the ReXS DSS based on 2010 data have now been made available to students enrolled in the 2011 course. Students have completed their first three weeks of assessment. Accordingly each student has been provided with feedback on the prognosis for their subsequent course grades. Students have also been provided with advice on actions they can undertake to improve their grades.

The seminar will discuss

illustrations of the predictions made by the DSS
how the principles underlying the DSS can be extended to other courses
opportunities for improving the utility of the DSS to students and academic staff.

About the speaker
Peter Mellalieu teaches innovation, strategy, and entrepreneurship at Unitec Institute of Technology, Auckland, New Zealand. His first Decision Support System was used for strategic planning studies including factory location, and company merger-takeover in the New Zealand dairy products manufacturing industry (Mellalieu, 1982; Mellalieu & Hall, 1983). At Unitec, he taught strategic thinking for several years using Thompson et al’s DSS-based Business Strategy Game. Peter also has a current focus on developing trans-disciplinary academic literacies through eco-innovation, eco-strategy, and eco-entrepreneurship. His recent interest in machine learning/data mining grew from his postgraduate studies in operations research and systems modeling conducted at New Zealand’s national physics and engineering laboratory and his undergraduate studies in industrial engineering and information technology.

References
Bouckaert, R. R., Frank, E., Hall, M., Kirkby, R., Reutemann, P., Seewald, A., & Scuse, D. (2010). WEKA Manual for Version 3-6-3. Hamilton, NZ: The University of Waikato.

Culver, T. (2010, January 21). Setting student retention goals and developing student retention strategy. Noel-Levitz. Retrieved January 23, 2011, from http://blog.noellevitz.com/2011/01/21/6-keys-setting-realistic-student-retention-goals-developing-successful-student-retention-strategy/

Culver, T. (2011). Mid-Year Retention Indicators Report for Two-Year and Four-Year, Public and Private Institutions: Benchmark Research Study Conducted Fall 2010. Higher Ed Benchmarks. Noel-Levitz. Retrieved from https://www.noellevitz.com/NR/rdonlyres/C72E0608-2024-4175-9A94-9669C394253E/0/2011MIDYEARINDICATORSREPORT.pdf

Decision support system - Wikipedia, the free encyclopedia. (n.d.). . Retrieved February 3, 2010, from http://en.wikipedia.org/wiki/Decision_support_system

Guillaume, D., & Khachikian, C. (2011). The effect of time-on-task on student grades and grade expectations. Assessment & Evaluation in Higher Education, 36(3), 251-261. Retrieved from http://dx.doi.org/10.1080/02602930903311708

Hall, M., Frank, E., Holmes, G., Pfahringer, B., Reutemann, P., & Witten, I. H. (2009). The WEKA Data Mining Software: An Update. SIGKDD Explorations, 11(1).

Mellalieu, P. J. (1982). A Decision Support System for Corporate Planning in the New Zealand Dairy Industry, Doctor of Philosophy in mathematics, statistics and operations research,. Victoria University of Wellington, Wellington, New Zealand. Retrieved from http://nzresearch.org.nz/index.php/record/viewSchema/21040/3

Mellalieu, P. J. (1999). Creating the A+ assignment: A project management approach (Working Paper). Massey University, College of Business. Retrieved from http://web.mac.com/petermellalieu/Teacher/Blog/Entries/2007/10/21_Creating_the_A%2B_assignment%3A_A_project_management_approach.html

Mellalieu, P. J. (2007a, June 3). Let’s all learn and teach Global English in our business schools! Peter Mellalieu - Teacher. Retrieved April 21, 2010, from http://web.mac.com/petermellalieu/Teacher/Blog/Entries/2007/7/3_Let%E2%80%99s_all_Learn_and_Teach_Global_English_in_our_Business_School!.html

Mellalieu, P. J. (2007b, July 6). The Massey Writing Across the Curriculum Model: A manifesto for the renaissance of an international business school? Retrieved October 11, 2009, from http://web.mac.com/petermellalieu/Teacher/Blog/Entries/2007/7/6_The_Massey_Writing_Across_the_Curriculum_Model%3A_A_Manifesto_for_the_Renaissance_of_an_International_Business_School.html

Mellalieu, P. J. (2007c, October 18). Model answer: A “Five Paragraph” essay in management. Retrieved July 27, 2009, from http://web.mac.com/petermellalieu/Teacher/Blog/Entries/2007/10/18_Model_answer%3A_A_%E2%80%9CFive_Paragraph%E2%80%9D_essay_in_management.html

Mellalieu, P. J. (2008). Writing to learn argument and persuasion: A “Trojan Horse” for promoting the adoption of “Writing Across the Curriculum” (WAC) principles (Working paper). Auckland, NZ: Unitec New Zealand Centre for Innovation & Entrepreneurship. Retrieved from http://web.mac.com/petermellalieu/Teacher/Blog/Entries/2008/7/31_Writing_to_learn_argument_and_persuasion%3A_A_Trojan_Horse_for_promoting_the_adoption_of_Writing_Across_the_Curriculum_(WAC)_principles.html

Mellalieu, P. J., & Hall, K. R. (1983). An Interactive Planning Model for the New Zealand Dairy Industry. Journal of the Operational Research Society, 34, 521-532. doi:10.1057/jors.1983.119

Mount, J. (2009, August 19). A Demonstration of Data Mining. Win-Vector Blog (The Applied Theorist’s Point of View). Retrieved December 4, 2010, from http://www.win-vector.com/blog/2009/08/a-demonstration-of-data-mining/

Paper for PedR meeting: The effect of time-on-task on student grades and grade expectations Â« DrBadgr. (n.d.). . Retrieved April 12, 2011, from http://drbadgr.wordpress.com/2011/04/11/paper-for-pedr-meeting-the-effect-of-time-on-task-on-student-grades-and-grade-expectations/

Springer, S. P., Franck, M. R., & Reider, J. (2009). What Do Selective Colleges Look for in an Applicant? The Academic Record. Education.com. Retrieved April 12, 2011, from http://www.education.com/reference/article/selective-colleges-applicant-academic/

Springer, S. P., Reider, J., & Franck, M. R. (2009). Admission Matters: What Students and Parents Need to Know About Getting into College (2nd ed.). Jossey-Bass.

Thompson, A. A., Stappenbeck, G. J., Reidenbach, M. A., Thrasher, I. F., & Harms, C. C. (n.d.). Business Strategy Game Simulation. Retrieved July 7, 2009, from http://www.bsg-online.com/

Weka 3 - Data Mining with Open Source Machine Learning Software in Java. (n.d.). . Retrieved November 21, 2010, from http://www.cs.waikato.ac.nz/ml/weka/

Witten, I. H., & Frank, E. (2005). Data mining: practical machine learning tools and techniques (2nd ed.). Morgan Kaufmann.

Author’s progress results
Click on the ReXS tag for related postings on my tumblr blog.