Posts tagged Machine learning

Predicting success, excellence, and retention from students’ early course performance (I’m off to the USA!)

A composed satellite photograph of North Ameri...
Image via Wikipedia

I’m off to San Francisco, Boston and Miami: 21 July - 4 August….. I’m presenting two papers at:

XXIX International Conference of the International Council for Higher Education. Presented at Innovation and Development in Higher Education, Miami/Ft Lauderdale.

Here is the abstract for one paper.

Higher educational institutions are focussing increased attention on identifying which students are likely to succeed - or fail - in their tertiary studies. Culver (2010, 2011), for instance, reports on the business case for, and services provided by the Noel-Levitz consultancy for improving institutional retention in North America. In New Zealand, government funding for higher education is increasingly being redirected towards a focus on outputs (such as course completions) rather than inputs (student enrollments) (Ministry of Education, 2010).

Anticipating this context, I constructed a prototype Decision Support System (ReXS) to provide my students the means to predict their personal academic success and final grade as they progressed through a first-year (freshman) course ‘Innovation and Entrepreneurship’. Data mining of previous semesters’ course results identified the crucial importance of a student’s ability to write formal academic English as demonstrated in a written case study assignment.

Several students’ immediate reaction to a presentation introducing them to ReXS was unexpectedly enthusiastic and they became ‘early adopter’ users. ‘Late adopter’ users of ReXS also gained confidence in identifying the degree of effort they needed to apply to complete the course succesfully through their Final Test.

Whilst ReXS is a bespoke solution tailored to the particular assessment regime of a particular course, I believe the principles of its design and construction can be applied to any assessed course in higher education. Certainly, my students indicated they would welcome widespread adoption of the approach in other courses in their study program. The presentation provides an opportunity to discuss: Reactions from the student users of ReXS; Illustrations of the predictions made by the ReXS; How the principles underlying the Decision Support System can be extended to other courses; Opportunities for improving the utility of ReXS for students, academic, and administrative staff.

Citation and full paper

Mellalieu, P. J. (2011). Predicting success, excellence, and retention from students’ early course performance: progress results from a data-mining-based decision support system in a first year tertiary education programme. XXIX International Conference of the International Council for Higher Education. Presented at Innovation and Development in Higher Education, Miami/Ft Lauderdale: International Council for Higher Education. Retrieved from http://tinyurl.com/4935qol

Enhanced by Zemanta

Using the ReXS Decision Support System for predicting success and excellence in student performance (video)

Note: If you find the video too blury, you can view a High Definition version on the Vimeo site. Click on HD.

A machine learning/data mining exercise using the WEKA Explorer workbench identified the feasibility of predicting accurately students’ final course performance from formative and summative assessments conducted within the first three weeks of a 12-week semester (Mellalieu 2010, 2011; Witten & Frank 2005; Hall, Frank et al, 2009). This finding led to the decision of the course instructor to initiate construction of a prototype Decision Support System (DSS) to provide a student and their academic advisors the means to predict the student’s personal academic success and final grade as they progressed through the course (Mellalieu, 1982).

The DSS is implemented as an .xls spreadsheet, known as ReXS (for Retention, eXcellence, and Success). Inputs include the student’s grade achieved on assignment components throughout the period of the course, demographic data, and other psychometric data. The prediction system underpinning the DSS is based on several rules and regression equations derived from a test data set of student results from a previous delivery of the course in 2010.


The course is a first year tertiary education course in innovation and entrepreneurship that is compulsory for all students in the management and marketing majors of a Bachelor of Business programme. The DSS outputs a student’s estimated grade and class percentile ranking at the start of the course (Week 1), and updated following assessments submitted by the student in Week 3, Week 6 (mid-way), and Week 12. A ‘last chance’ final assessment occurs during the examination period, Week 14, the result for which can be predicted from all the student’s previous assessment results. As the student progresses through the course, we expect the precision of the grade estimate improves, since their are increased number of data items upon which to make the prediction. Furthermore, the student will know their actual ‘earned grade’ contribution to the final course grade.

Using the ReXS .xls spreadsheet
The video explains use of the ReXS dashboard that a student uses to ‘drive’ the DSS. The cells in green show where the user enters data, including:

  • The grade they achieved for a specific assignment
  • The actual class mean and  standard deviation for a specific assignment. If the data is not available, the statistics from a prediction dataset are used (for example, projected from previous semester’s results).


The DSS displays several outputs for the user:

  • Under MY GRADES (in orange): the per cent mark,  letter grade, and weighted contribution to the final course grade - nothing special there!
  • Under CLASS PERFORMANCE (in blue): an estimate of the student’s relative placing in the class, measured by the per cent of students below them in the class. This is derived simply from their mark and the class statistics (assuming a normal distribution of grades)
  • Under  PREDICTED GRADES (Yellow): an estimate for the student’s grades for all outstanding assignments in the course. This also includes a prediction of the student’s overall relative placing in the class. This is the ‘magic’ part of the DSS.


Furthermore, the student can enter their actual grades and conduct ‘what if’ analyses to explore what grade for assignment components they need to achieve to gain their target grade for the course as a whole.

Tabs at the bottom of the DSS dashboard permit the user to explore the inner workings of the DSS. These inner workings include the prediction equations resulting from the machine learning data analysis: Predictors, Class stats, etc. The Babbage tab is a giant spreadsheet that shows all the calculations in full.

ReXS spreadsheet
Available for download here:


Mellalieu, P. J. (2011, April 26). ReXS: Decision Support System for Retention, eXcellence, and Success (.xls spreadsheet). Peter Mellalieu - Teacher. Retrieved April 26, 2011, from http://preview.tinyurl.com/rexsdss1-2

References
Hall, M., Frank, E., Holmes, G., Pfahringer, B., Reutemann, P., & Witten, I. H. (2009). The WEKA Data Mining Software: An Update. SIGKDD Explorations, 11(1).

Mellalieu, P. J. (2011, April 27). Predicting success, excellence, and retention from students’ early course performance (video). Presented at the research seminar, Department of Management & Marketing, Unitec Institute of Technology, Auckland. Retrieved from http://vimeo.com/22877834

Mellalieu, P. J. (2011b, April 12). A Decision Support System for predicting Retention, eXcellence, & Success: The Dashboard for ReXS. Innovation & chaos … in search of optimality. Retrieved April 12, 2011, from http://pogus.tumblr.com/post/4548163448/image-via-wikipedia-a-decision-support-system-for


Mellalieu, P. J. (2011, April 18). Predicting success, excellence, and retention from students’ early course performance: progress results from a machine learning-based decision support system in a first year tertiary education programme (Research seminar announcement). Innovation & chaos … in search of optimality. Retrieved April 17, 2011, from http://pogus.tumblr.com/post/4702098630/a-research-seminar-predicting-success-excellence-and

Witten, I. H., & Frank, E. (2005). Data mining: practical machine learning tools and techniques (2nd ed.). Morgan Kaufmann.

Enhanced by Zemanta

Mellalieu, P. J. (2011, April 26). Predicting success, excellence, and retention from students’ early course performance: progress results from a machine learning-based decision support system in a first year tertiary education programme (video)

If you get bored waiting for the video to commence then please go directly to the Vimeo site. You will be able to view immediately in stream mode: http://vimeo.com/22877834


Runtime: 55 minutes


Related

Mellalieu, P. J. (2011, April 18). Predicting success, excellence, and retention from students’ early course performance: progress results from a machine learning-based decision support system in a first year tertiary education programme (Research seminar announcement). Innovation & chaos … in search of optimality. Retrieved April 17, 2011, from http://pogus.tumblr.com/post/4702098630/a-research-seminar-predicting-success-excellence-and Enhanced by Zemanta

Predicting success, excellence, and retention from students’ early course performance: progress results from a machine learning-based decision support system in a first year tertiary education programme (slideshow)

Predicting success, excellence, and retention from students’ early course performance: progress results fro…

Mellalieu, P. J. (2011, April 18). Predicting success, excellence, and retention from students’ early course performance: progress results from a machine learning-based decision support system in a first year tertiary education programme (Research seminar announcement). Innovation & chaos … in search of optimality. Retrieved April 17, 2011, from http://pogus.tumblr.com/post/4702098630/a-research-seminar-predicting-success-excellence-and
Mellalieu, P. J. (2011, April 26). ReXS: Decision Support System for Retention, eXcellence, and Success (.xls spreadsheet). Peter Mellalieu - Teacher. Retrieved April 26, 2011, from http://preview.tinyurl.com/rexsdss1-2
Enhanced by Zemanta

A research seminar: Predicting success, excellence, and retention from students’ early course performance: progress results from a machine learning-based decision support system in a first year tertiary education programme

Wednesday 27 April, 2011, 12 noon
Location: Room 172-4024, Building 172, Unitec Institute of Technology, Carrington Road, Mt Albert

Peter J Mellalieu

Department of Management & Marketing
Faculty of Creative Industries and Business
Unitec Institute of Technology, Auckland, New Zealand

Higher educational institutions are focussing increased attention on identifying which students are likely to succeed - or fail - in their tertiary studies. Historically, academics have been keen to identify the ‘bright young things’ they view as prospects for recruiting to postgraduate courses. Less attention has been paid to those students who fall by the wayside: there have been plenty of ambitious and talented students to take their place. More recently, institutions have been obliged to pay attention to identifying those students ‘at risk’ of failure or marginal grades and conducting re-mediative activity. For instance, Culver (2010, 2011) reports on the services provided by the Noel-Levitz consultancy for improving institutional retention in North America. Culver argues that the business case for active management of student retention is compelling.


New Zealand higher educational institutions are beginning to recognise the need to follow the trend to manage student retention and success using approaches similar to those adopted by North American institutions. The driver for the New Zealand initiative is that government funding for higher education is increasingly being redirected towards a focus on outputs rather than inputs. Specifically, this entails a refocus of government funding towards funding student completions of academic programmes rather than simply student enrollments. It is noteworthy that in New Zealand, around one-half of higher education is funded by government funds.


Given this context, I constructed a prototype Decision Support System (DSS) that provides a student the means to predict their personal academic success and final grade as they progress through their course. The DSS is implemented as an interlocked series of spreadsheets, known as ReXS - for Retention, Excellence, and Success (Mellalieu, 2011a,b).

Inputs to ReXS include the student’s grade achieved on assignment components throughout the period of the course, demographic data, the student’s five talents as identified by the Gallup StrengthsFinder 2.0 instrument, and other psychometric data. The prediction system underpinning the DSS is based on several rules and regression equations derived from a test data set of student results from a previous delivery of the course in 2010. A schematic overview of the inter-relationship between several pertinent factors is presented in Mellalieu (2010 f). Figure 1 presents a snapshot of the dashboard user interface for the ReXS DSS.

A machine learning/data mining investigation using the WEKA Explorer workbench (Witten & Frank 2005; Hall, Frank et al, 2009) identified the feasibility of predicting accurately students’ final course performance from formative and summative assessments conducted within the first three weeks of a 12-week semester. The prediction system underpinning ReXS was subsequently derived from a set of rules and regression equations determined using WEKA Explorer (Mellalieu 2010 a,b,c,d,e,f,g).


Figure 1: Snapshot of the dashboard for ReXS

image

The course is a first year tertiary education course in innovation and entrepreneurship that is compulsory for all students in the management and marketing majors of a Bachelor of Business programme at Unitec Institute of Technology. The DSS outputs a student’s estimated grade and class percentile ranking at the start of the course (Week 1). These statistics are updated following assessments submitted by the student in Week 3, Week 6 (mid-way), and Week 12. A ‘last chance’ final assessment occurs during the examination period, Week 14, the result for which can be predicted from all the student’s previous assessment results… and other demographic data. The DSS output also includes a measure of the precision of the estimate, such as the statistical standard error of the grade estimate. As the student progresses through the course, the precision improves, since there are an increased number of data items upon which to make the prediction.

A most potent indicator of overall course performance revealed by the data mining exercises identified the crucial importance of a student’s ability to write formal academic English in response to a written case study assignment. This finding suggests that if a student undertakes personal coaching in the antecedents required to rite gude inglish [sic] then their chances of achieving an overall higher grade will increase. I suspect these antecedent competencies include: reading for comprehension, paraphrasing information, writing persuasive and logical arguments, selecting appropriate words to express thoughts, writing Global English sentence structures, paragraphing using topic sentences, and presenting/organising arguments and evidence in an appropriate genre such as a formal business report (Mellalieu 2007a,b,c, 2008, 2010 g,h). The foregoing are a selection of core, generic academic competencies that might well underpin success in many other business and tertiary education courses. Consequently, a DSS that provides students with ‘early warning’ of the likelihood of their academic success or failure based on an early assessment of these competencies may encourage students, their instructors, and advisors to take early, proactive action to remedy deficiencies both within and outside the classroom.


The machine learning analysis also revealed unexpected insights into the level of academic performance of a student team assignment. These results will be presented at the seminar.

Results from the ReXS DSS based on 2010 data have now been made available to students enrolled in the 2011 course. Students have completed their first three weeks of assessment. Accordingly each student has been provided with feedback on the prognosis for their subsequent course grades. Students have also been provided with advice on actions they can undertake to improve their grades.

The seminar will discuss

  • illustrations of the predictions made by the DSS
  • how the principles underlying the DSS can be extended to other courses
  • opportunities for improving the utility of the DSS to students and academic staff.

About the speaker
Peter Mellalieu teaches innovation, strategy, and entrepreneurship at Unitec Institute of Technology, Auckland, New Zealand. His first Decision Support System was used for strategic planning studies including factory location, and company merger-takeover in the New Zealand dairy products manufacturing industry (Mellalieu, 1982; Mellalieu & Hall, 1983). At Unitec, he taught strategic thinking for several years using Thompson et al’s DSS-based Business Strategy Game. Peter also has a current focus on developing trans-disciplinary academic literacies through eco-innovation, eco-strategy, and eco-entrepreneurship. His recent interest in machine learning/data mining grew from his postgraduate studies in operations research and systems modeling conducted at New Zealand’s national physics and engineering laboratory and his undergraduate studies in industrial engineering and information technology.

References
Bouckaert, R. R., Frank, E., Hall, M., Kirkby, R., Reutemann, P., Seewald, A., & Scuse, D. (2010). WEKA Manual for Version 3-6-3. Hamilton, NZ: The University of Waikato.

Culver, T. (2010, January 21). Setting student retention goals and developing student retention strategy. Noel-Levitz. Retrieved January 23, 2011, from http://blog.noellevitz.com/2011/01/21/6-keys-setting-realistic-student-retention-goals-developing-successful-student-retention-strategy/

Culver, T. (2011). Mid-Year Retention Indicators Report for Two-Year and Four-Year, Public and Private Institutions: Benchmark Research Study Conducted Fall 2010. Higher Ed Benchmarks. Noel-Levitz. Retrieved from https://www.noellevitz.com/NR/rdonlyres/C72E0608-2024-4175-9A94-9669C394253E/0/2011MIDYEARINDICATORSREPORT.pdf

Decision support system - Wikipedia, the free encyclopedia. (n.d.). . Retrieved February 3, 2010, from http://en.wikipedia.org/wiki/Decision_support_system

Guillaume, D., & Khachikian, C. (2011). The effect of time-on-task on student grades and grade expectations. Assessment & Evaluation in Higher Education, 36(3), 251-261. Retrieved from http://dx.doi.org/10.1080/02602930903311708

Hall, M., Frank, E., Holmes, G., Pfahringer, B., Reutemann, P., & Witten, I. H. (2009). The WEKA Data Mining Software: An Update. SIGKDD Explorations, 11(1).

Mellalieu, P. J. (1982). A Decision Support System for Corporate Planning in the New Zealand Dairy Industry, Doctor of Philosophy in mathematics, statistics and operations research,. Victoria University of Wellington, Wellington, New Zealand. Retrieved from http://nzresearch.org.nz/index.php/record/viewSchema/21040/3

Mellalieu, P. (2013, March 15). Creating The A Plus Assignment: A Project Management Approach (Audio). Innovation & chaos … in search of optimality. Retrieved from http://pogus.tumblr.com/post/45403052813/this-audio-tutorial-helps-you-plan-out-the-time

Mellalieu, P. J. (2007, June 3). Let’s all learn and teach Global English in our business schools! Peter Mellalieu - Teacher. Retrieved April 21, 2010, from http://unitec.academia.edu/PeterMellalieu/Papers/1571526/Lets_all_learn_and_teach_Global_English_in_our_business_schools_

Mellalieu, P. J. (2007, July 6). The Massey Writing Across the Curriculum Model: A manifesto for the renaissance of an international business school? Peter Mellalieu - Teacher. Retrieved October 11, 2009, from http://www.academia.edu/1513127

Mellalieu, P. J. (2007, October 18). Model answer: A “Five Paragraph” essay in management. Retrieved July 27, 2009, from http://www.academia.edu/1465813

Mellalieu, P. J. (2008). Writing to learn argument and persuasion: A “Trojan Horse” for promoting the adoption of “Writing Across the Curriculum” (WAC) principles (Working paper). Auckland, NZ: Unitec New Zealand Centre for Innovation & Entrepreneurship. Retrieved from http://unitec.academia.edu/PeterMellalieu/Papers/1518834/Writing_to_learn_argument_and_persuasion_ATrojan_Horsefor_promoting_the_adoption_ofWriting_Across_the_Curriculum_WAC_principles_in_an_international_

Mellalieu, P. J., & Hall, K. R. (1983). An Interactive Planning Model for the New Zealand Dairy Industry. Journal of the Operational Research Society, 34, 521-532. doi:10.1057/jors.1983.119

Mount, J. (2009, August 19). A Demonstration of Data Mining. Win-Vector Blog (The Applied Theorist’s Point of View). Retrieved December 4, 2010, from http://www.win-vector.com/blog/2009/08/a-demonstration-of-data-mining/

Paper for PedR meeting: The effect of time-on-task on student grades and grade expectations « DrBadgr. (n.d.). . Retrieved April 12, 2011, from http://drbadgr.wordpress.com/2011/04/11/paper-for-pedr-meeting-the-effect-of-time-on-task-on-student-grades-and-grade-expectations/

Springer, S. P., Franck, M. R., & Reider, J. (2009). What Do Selective Colleges Look for in an Applicant? The Academic Record. Education.com. Retrieved April 12, 2011, from http://www.education.com/reference/article/selective-colleges-applicant-academic/

Springer, S. P., Reider, J., & Franck, M. R. (2009). Admission Matters: What Students and Parents Need to Know About Getting into College (2nd ed.). Jossey-Bass.

Thompson, A. A., Stappenbeck, G. J., Reidenbach, M. A., Thrasher, I. F., & Harms, C. C. (n.d.). Business Strategy Game Simulation. Retrieved July 7, 2009, from http://www.bsg-online.com/

Weka 3 - Data Mining with Open Source Machine Learning Software in Java. (n.d.). . Retrieved November 21, 2010, from http://www.cs.waikato.ac.nz/ml/weka/

Witten, I. H., & Frank, E. (2005). Data mining: practical machine learning tools and techniques (2nd ed.). Morgan Kaufmann.

Author’s progress results
Click on the ReXS tag for related postings on my tumblr blog.

Enhanced by Zemanta

A Decision Support System for predicting Retention, eXcellence, & Success: Feedback to my students using ReXS

A mathematics lecture, apparently about linear...
Image via Wikipedia

This is a memo I recently submitted to my students based on my analysis of their performance in the first three weeks of their course with me….

Their first assignment in BSNS 5391 Innovation & Entrepreneurship introduces the students to the traditional business case analysis. I achieve this introduction through a series of comprehension-like questions that lead the students to write a Formal Business Report.

Students submit their answers to 1/4 of the questions for the assignment in Week 3 of the course (Assignment 1a). I provide formative feedback based SOLELY on their writing quality assessed against six criteria adapted from a rubric from Maryvale School. My Machine Learning analysis identified that students’ performance on this draft assignment had extraordinary predictive power. Not only for the completed Assignment 1, but also for such tasks as an open book multi-choice test!

The memo to my students

Based on the mark for your Assignment 1a, I am able to predict what your grade will be for future assignments in this class. Table 3.3 shows the predictions.

If you do not like the predictions then take heart:

  • If you work smarter and harder than the ‘average’ student you may be able to gain a better mark than the prediction given your mark thus far.
  • The forecast error is (plus or minus) +/- 5 marks. For example, if you gained 15 marks for assignment 1a, then I predict you will gain 62 +/- 5 (57 to 67) marks for Assignment 1, and 73 +/- 4 (68 to 78) marks for the Final Mark for the course overall.
  • If you take advantage of the many resources within and beyond Unitec to improve your formal reading, comprehension, and writing skills, the benefits will extend through and beyond this course (Mellalieu, 2008).

These forecasts are NOT guarantees. The forecasts are based on statistical analysis of previous students grades for this class. This class appears to be quite different from the class from which I derived my forecast equations. So the error (plus or minus) may be higher. However, by the time I grade your Assignment 1bc and 3b, I will be able to present predictions to +/- 3 marks.

Table 3.3: Final Mark predicted from Formal Writing Quality (Assignment 1a) – (2011-1 Actual)

The following Table 3.4 shows Table 3.3 converted to Grades according to Unitec’s grade equivalence system.

Is it possible to get an A+ as a Final Mark in this course? YES. You must achieve an A or A+ for your Assignment 2. The grade shown in Table 3.3 for Assignment 2 is the class average I anticipate (81, A-). Since Ass 2 is a group assignment, the grade for Assignment 2 is not directly dependant on the grade you get for Assignment 1. But see my comments below.

Table 3.4 Final Grade predicted from Formal Writing Quality (Assignment 1a) – (2011-1 Actual)



For those with the #StrengthsQuest talents of Competition and Achiever, you may like to know your performance compared to others in the class. Table 3.5 shows how many students are expected to grade below you given your mark for Assignment 1a. I calculate this data based on the mean and standard deviation of the class’s results combined with my guestimates of future class performance.

Example: Given a student achieves a writing quality of 20 (Assignment 1a) then I predict that the student will achieve a grade better than 66.4 % of all other students in the class for Assignment 1. The table also suggests that only 23.7 per cent of students will achieve the required writing level of 24/30 for Assignment 1. (24 = 100 - 76.3).


Table 3.5: Student’s relative performance predicted from Formal Writing Quality (Assignment 1a) – (2011-1 Actual)

In a later posting, I’ll show you how the following two factors predict the likely grade you will gain for Assignment 2:

  • Average writing quality of team members (as measured by Assignment 1a)
  • Average team level of entrepreneurship (as measured by the talents that Bolton & Thompson argue are required for a serial entrepreneur, namely those identified in their FACETS model)

The BSNS 5391 boffins welcome with eagerness your questions and comments about the data presented here!

Further information from our laboratories

Mellalieu, P. J. (2011, April 12). A Decision Support System for predicting Retention, eXcellence, & Success: The Dashboard for ReXS. Innovation & chaos … in search of optimality. Retrieved April 12, 2011, from http://pogus.tumblr.com/post/4548163448/image-via-wikipedia-a-decision-support-system-for

Mellalieu, P. J. (2011). Unitec BSNS 5391 Innovation and Entrepreneurship. Course Handbook and Syllabus. Auckland, NZ: Unitec New Zealand. Retrieved from http://www.scribd.com/doc/36191676/Course-Handbook-and-Syllabus-Unitec-BSNS-5391-Innovation-and-Entrepreneurship

Mellalieu, P. J. (2010a, November 29). Predicting success, excellence, and retention from early course performance: a comparison of statistical and machine learning methods in a tertiary education programme: Part 1: Statistical analysis. Innovation & chaos … in search of optimality. Retrieved November 29, 2010, from http://pogus.tumblr.com/post/1724117822/predicting-success-excellence-and-retention-from

Mellalieu, P. J. (2010b, November 29). Predicting success, excellence, and retention from early course performance: a comparison of statistical and machine learning methods in a tertiary education: Part 1: Statistical analysis - Figures. Innovation & chaos … in search of optimality. Retrieved November 29, 2010, from http://pogus.tumblr.com/post/1723717009/predicting-success-excellence-and-retention-from

Mellalieu, P. J. (2010c, November 30). Predicting success, excellence, and retention from early course performance: a comparison of statistical and machine learning methods in a tertiary education programme: Part 2a: Preparing for data mining. Innovation & chaos … in search of optimality. Retrieved December 3, 2010, from http://pogus.tumblr.com/post/1983876991/predicting-success-excellence-and-retention-from

Mellalieu, P. J. (2010d, November 30). Predicting success, excellence, and retention from early course performance: a comparison of statistical and machine learning methods in a tertiary education programme: Part 2a: Preparing for data mining - Figures. Innovation & chaos … in search of optimality. Retrieved November 30, 2010, from http://pogus.tumblr.com/post/1983667421/predicting-success-excellence-and-retention-from

Mellalieu, P. J. (2010f, December 3). Predicting success, excellence, and retention from early course performance: a comparison of statistical and machine learning methods in a tertiary education programme: Part 2b: Data mining with WEKA Explorer. Innovation & chaos … in search of optimality. Retrieved December 3, 2010, from http://pogus.tumblr.com/post/2079452955/predicting-success-excellence-and-retention-from

Mellalieu, P. J. (2010g, December 5). Predicting success, excellence, and retention from early course performance: a comparison of statistical and machine learning methods in a tertiary education programme: Part 3a: A comprehensive time-staged model - Figures. Innovation & chaos … in search of optimality. Retrieved December 5, 2010, from http://pogus.tumblr.com/post/2100181658/predicting-success-excellence-and-retention-from

Mellalieu, P. J. (2010h, December 6). A Decision Support System for predicting success, excellence, and retention from students’ early course performance: a machine learning approach in a tertiary education programme in innovation and entrepreneurship: Part 1: Project summary. Innovation & chaos … in search of optimality. Retrieved January 10, 2011, from http://pogus.tumblr.com/post/2110849868/a-decision-support-system-for-predicting-success

Mellalieu, P. J. (2008). Writing to learn argument and persuasion: A “Trojan Horse” for promoting the adoption of “Writing Across the Curriculum” (WAC) principles (Working paper). Auckland, NZ: Unitec New Zealand Centre for Innovation & Entrepreneurship. Retrieved from http://web.mac.com/petermellalieu/Teacher/Blog/Entries/2008/7/31_Writing_to_learn_argument_and_persuasion%3A_A_Trojan_Horse_for_promoting_the_adoption_of_Writing_Across_the_Curriculum_(WAC)_principles.html

Enhanced by Zemanta

A Decision Support System for predicting success, excellence, and retention from students’ early course performance: a machine learning approach in a tertiary education programme in innovation and entrepreneurship: Part 1: Project summary

imageA machine learning/data mining exercise using the WEKA Explorer workbench identified the feasibility of predicting accurately students’ final course performance from formative and summative assessments conducted within the first three weeks of a 12-week semester (Mellalieu 2010 a,b,c,d,e; Witten & Frank 2005; Hall, Frank et al, 2009). This finding led to the decision of the course instructor to initiate construction of a prototype Decision Support System (DSS) to provide a student and their academic advisors the means to predict the student’s personal academic success and final grade as they progressed through the course (Mellalieu, 1982).

The DSS is to be implemented as a spreadsheet. Inputs include the student’s grade achieved on assignment components throughout the period of the course, demographic data, the student’s five talents as identified by the Gallup StrengthsFinder 2.0 instrument, and other psychometric data. The prediction system underpinning the DSS is based on several rules and regression equations derived from a test data set of student results from a previous delivery of the course in 2010. A schematic overview of the inter-relationship between several pertinent factors is presented in Mellalieu (2010 f).

The course is a first year tertiary education course in innovation and entrepreneurship that is compulsory for all students in the management and marketing majors of a Bachelor of Business programme. The DSS will output a student’s estimated grade and class percentile ranking at the start of the course (Week 1), and updated following assessments submitted by the student in Week 3, Week 6 (mid-way), and Week 12. A ‘last chance’ final assessment occurs during the examination period, Week 14, the result for which can be predicted from all the student’s previous assessment results… and possibly other demographic data. The DSS output will also include an measure of the precision of the estimate, such as the statistical standard error of the grade estimate. As the student progresses through the course, we expect the precision to improve, since their will be an increased number of data items upon which to make the prediction. Furthermore, the student will know their actual ‘earned grade’ contribution to the final course grade.

An enhanced version of the DSS could be developed to provide feedback about what steps the student can undertake to ‘beat the system’ in order to obtain higher grades than their projected grade. For example, a most potent indicator of overall course performance revealed by the data mining exercises identified the crucial importance of a student’s ability to write formal academic English in response to a written case student assignment. This finding suggests that if a student undertakes personal coaching in the antecedents required to rite gude inglish [sic] then their chances of achieving an overall higher grade will increase. I suspect these antecedent competencies include: reading for comprehension, paraphrasing information, writing persuasive and/or logical arguments, selecting appropriate words to express thoughts, writing Global English sentence structures, paragraphing using topic sentences, and presenting/organising arguments and evidence in an appropriate genre such as a formal business report, short essay, or memo (Mellalieu 2007, 2008, 2010 g,h). The foregoing are a selection of core, generic academic competencies that might well underpin success in many other business and tertiary education courses. Consequently, a DSS that provides students with ‘early warning’ of the likelihood of their academic success or failure based on an early assessment of these competencies may encourage students, their instructors, and advisors to take early, proactive action to remedy deficiencies both within and outside the classroom.

References
Hall, M., Frank, E., Holmes, G., Pfahringer, B., Reutemann, P., & Witten, I. H. (2009). The WEKA Data Mining Software: An Update. SIGKDD Explorations, 11(1).  


Witten, I. H., & Frank, E. (2005). Data mining: practical machine learning tools and techniques (2nd ed.). Morgan Kaufmann.
 
Author’s supporting studies

Mellalieu, P. J. (2011a, April 12). A Decision Support System for predicting Retention, eXcellence, & Success: Feedback to my students using ReXS. Innovation & chaos … in search of optimality. Retrieved April 12, 2011, from http://pogus.tumblr.com/post/4550061605/a-decision-support-system-for-predicting-retention

Mellalieu, P. J. (2011b, April 12). A Decision Support System for predicting Retention, eXcellence, & Success: The Dashboard for ReXS. Innovation & chaos … in search of optimality. Retrieved April 12, 2011, from http://pogus.tumblr.com/post/4548163448/image-via-wikipedia-a-decision-support-system-for


Mellalieu, P. J. (1982). A Decision Support System for Corporate Planning in the New Zealand Dairy Industry (Doctor of Philosophy in mathematics, statistics and operations research). Victoria University of Wellington. Retrieved from http://hdl.handle.net/10063/568

Mellalieu, P. J. (2007, June 3). Let’s all learn and teach Global English in our business schools! Peter Mellalieu - Teacher. Retrieved April 21, 2010, from http://unitec.academia.edu/PeterMellalieu/Papers/1571526/Lets_all_learn_and_teach_Global_English_in_our_business_schools_

Mellalieu, P. J. (2008). Writing to learn argument and persuasion: A “Trojan Horse” for promoting the adoption of “Writing Across the Curriculum” (WAC) principles (Working paper). Auckland, NZ: Unitec New Zealand Centre for Innovation & Entrepreneurship. Retrieved from http://unitec.academia.edu/PeterMellalieu/Papers/1518834/Writing_to_learn_argument_and_persuasion_ATrojan_Horsefor_promoting_the_adoption_ofWriting_Across_the_Curriculum_WAC_principles_in_an_international_


Mellalieu, P. J. (2010a, November 29). Predicting success, excellence, and retention from early course performance: a comparison of statistical and machine learning methods in a tertiary education programme: Part 1: Statistical analysis. Innovation & chaos … in search of optimality. Retrieved November 29, 2010, from http://pogus.tumblr.com/post/1724117822/predicting-success-excellence-and-retention-from

Mellalieu, P. J. (2010b, November 29). Predicting success, excellence, and retention from early course performance: a comparison of statistical and machine learning methods in a tertiary education: Part 1: Statistical analysis - Figures. Innovation & chaos … in search of optimality. Retrieved November 29, 2010, from http://pogus.tumblr.com/post/1723717009/predicting-success-excellence-and-retention-from

Mellalieu, P. J. (2010c, November 30). Predicting success, excellence, and retention from early course performance: a comparison of statistical and machine learning methods in a tertiary education programme: Part 2a: Preparing for data mining. Innovation & chaos … in search of optimality. Retrieved December 3, 2010, from http://pogus.tumblr.com/post/1983876991/predicting-success-excellence-and-retention-from


Mellalieu, P. J. (2010d, November 30). Predicting success, excellence, and retention from early course performance: a comparison of statistical and machine learning methods in a tertiary education programme: Part 2a: Preparing for data mining - Figures. Innovation & chaos … in search of optimality. Retrieved November 30, 2010, from http://pogus.tumblr.com/post/1983667421/predicting-success-excellence-and-retention-from

Mellalieu, P. J. (2010e, December 3). Predicting success, excellence, and retention from early course performance: a comparison of statistical and machine learning methods in a tertiary education programme: Part 2b: Data mining with WEKA Explorer. Innovation & chaos … in search of optimality. Retrieved December 3, 2010, from http://pogus.tumblr.com/post/2079452955/predicting-success-excellence-and-retention-from

Mellalieu, P. J. (2010f, December 5). Predicting success, excellence, and retention from early course performance: a comparison of statistical and machine learning methods in a tertiary education programme: Part 3a: A comprehensive time-staged model - Figures. Innovation & chaos … in search of optimality. Retrieved December 5, 2010, from http://pogus.tumblr.com/post/2100181658/predicting-success-excellence-and-retention-from

Mellalieu, P. J. (2010g, November 8). Preparing an essay for a written test or examination (Part A). Innovation & chaos … in search of optimality. Retrieved November 8, 2010, from http://pogus.tumblr.com/post/1512039456/short-essay-advice

Mellalieu, P. J. (2010h, November 8). Preparing an essay for a written test or examination (Part B): The Pogorific A+ exam-sitting method. Innovation & chaos … in search of optimality. Retrieved November 8, 2010, from http://pogus.tumblr.com/post/1512453524/scheduling-time-in-a-test


Enhanced by Zemanta