This annotated bibliography describes research that explains and defines the field of data analytics in education. This annotated bibliography also includes applications of methods to analyze data for educational use. The bibliography begins with publications that provide an overview of the field, followed by applications of data analytics in education, organized by purpose or method of analysis.
Merceron and Yacef argue for the use of cosine and added value (similar to lift) as measures of interestingness, which are separate from two standard measures of association rules - confidence and support. They substantiate their argument by working through a case study using LMS data. Cosine and added value were used to measure the usefulness of LMS-based instruction aimed to augment the face-to-face instruction in the classroom.
This research reports on the connection between student achievement and LMS usage and student characteristics. This research made more detailed categorization of LMS interaction (e.g., discussion group participation) to study the the relationship between LMS use and achievement. One finding is the concern over the accuracy of time measures in the LMS, which was found to be skewed to 0. Another finding was that lower-income students spend more time in the LMS than their counterpart. And that lower-income students experience a lower “efficiency” in terms of time on task.
Overview of the Field
Baker, R. S. J. D., & Yacef, K. (2009). The state of educational data mining in 2009 : A review and future visions. Journal of Educational Data Mining, 1(1), 3–16.
Provides an overview of the field of educational data mining. Defines educational data mining, identifies common methodologies, and examines current applications. Reviews the top cited research in the field between 1995 and 2005. Identifies past trends in research (applications and methodologies). Highlights potential future trends and research needs.
Baker, R. S. J. d., & Siemens, G. (2014). Educational Data Mining and Learning Analytics. Manuscript to be published in Cambridge Handbook of the Learning Sciences.
An updated overview of the field of educational data mining in addition to an overview of the field of learning analytics. Published for the Learning Sciences audience. Gives definitions of educational data mining and learning analytics. Describes methods of analysis, tools for analysis, applications, and future trends for educational data mining and learning analytics in the Learning Sciences. One noteworthy application discussed was methods used to detect disengagement in students (using prediction methods and knowledge engineering).
Bienkowski, M., Feng, M., & Means, B. (2012). Enhancing teaching and learning through educational data mining and learning analytics: An issue brief. Washington, DC: SRI International. Retrieved from http://www.ed.gov/edblogs/technology/files/2012/03/edm-la-brief.pdf
In this report, researchers review the growing work done on educational data mining and learning analytics. Researchers discuss applications to education, review possibilities and current examples, examine challenges to the field, and identifies future areas of research. This report provides a list of websites using these techniques that one can explore.
Campbell, B. J. P., Deblois, P. B., & Oblinger, D. G. (2007). Academic analytics: A new tool for a new era. Educause Review, (August 2007), 41–57.
Introduces the idea of academic analytics ("actionable intelligence to improve teaching, learning, and student success"), and reviews examples of research and technology done that matches ideals of the initiative. Also identifies needs and challenges facing this area of research and development, such as data ownership and privacy, sharing information, and profiling.
Ferguson, R. (2012). Learning analytics: Drivers, developments and challenges. International Journal of Technology Enhanced Learning, 4(5/6), 304–317. doi:10.1504/IJTEL.2012.051816
This article gives an introduction to the field of learning analytics. It discusses definitions, societal drivers that led to the development of learning analytics, and the history of the emergence of data analytics in education. The author also distinguishes learning analytics from academic analytics and educational data mining. Learning analytics is more about the learner's perspective and needs, about social perspectives on learning, and has close ties to the learning sciences. Future challenges to learning analytics are described.
Long, P., & Siemens, G. (2011). Penetrating the fog: Analytics in learning and education. Educause Review, 46(5), 31–40.
Describes definitions and distinctions in the field of learning analytics, potential uses, and challenges. Distinguishes between learning analytics and academic analytics. Presents an argument on the importance of learning analytics. Technology is allowing us greater capacity to capture data about student learning behavior. However there is a need to create new approaches for making sense of the data, and for looking beyond behavioral data.
Mayer-Schönberger, V., Cukier, K. (2013). Big Data: A Revolution That Will Transform How We Live, Work, and Think. New York, NY: Houghton Mifflin Harcourt. Kindle Edition.
This book outlines the fundamental philosophy behind big data. The amount of data, the importance of correlations, digitizing data and creating more data about things are all important to a new paradigm about how to look at the world and what we can learn from it. The book includes many examples to illustrate ideas presented.
Provost, F. & Fawcett, T. (2013). Data Science for Business: What you need to know about data mining and data-analytic thinking. Sebastopol, CA: O'Reilly Media. Kindle Edition.
This book describes the fundamental data science methods for analyzing data, including classification, regression, clustering, Bayesian, and association methods and statistics. The book also outlines a general process for solving data problems in business. The book includes some of the mathematical formulas for completing these tasks, but tries to take the least-technical description of methods as possible. Many examples are given to illustrate how methods are used.
Mayer-Schönberger, V., Cukier, K. (2013). Big Data: A Revolution That Will Transform How We Live, Work, and Think. New York, NY: Houghton Mifflin Harcourt. Kindle Edition.
This book outlines the fundamental philosophy behind big data. The amount of data, the importance of correlations, digitizing data and creating more data about things are all important to a new paradigm about how to look at the world and what we can learn from it. The book includes many examples to illustrate ideas presented.
Provost, F. & Fawcett, T. (2013). Data Science for Business: What you need to know about data mining and data-analytic thinking. Sebastopol, CA: O'Reilly Media. Kindle Edition.
This book describes the fundamental data science methods for analyzing data, including classification, regression, clustering, Bayesian, and association methods and statistics. The book also outlines a general process for solving data problems in business. The book includes some of the mathematical formulas for completing these tasks, but tries to take the least-technical description of methods as possible. Many examples are given to illustrate how methods are used.
Romero, C., & Ventura, S. (2010). Educational data mining: A review of the state of the art. IEEE Transactions on Systems, Man, and Cybernetics-Part C: Applications and Reviews, 40(6), 601–618.
This provides a very useful and detailed library of sources to turn to on what has been done to improve learning through educational data mining. Also reviews definitions, trends in publications and conferences, applications, and future directions. Main applications of educational data mining include analysis and visualization of data, providing feedback for supporting instructors, recommendations for students, predicting student performance, student modeling, detecting undesirable student behaviors, grouping students, Social Network Analysis, developing concept maps, constructing courseware, and planning and scheduling.
This important article reviews the current status of the development of the Learning Analytics field and includes reviews of such topics as the need to increase data scope, privacy and ownership of data, and the immaturity of the legal infrastructure in place in terms of privacy and ethics. The article argues that Learning Analytics has become and deserves to be viewed as, “an emerging research field.”
Siemens, G. (2013). Learning analytics: The emergence of a discipline. American Behavioral Scientist, 57(10), 1380–1400. doi:10.1177/0002764213498851
Applications
1. Prediction
Arnold, K. E., & Pistilli, M. D. (2012). Course Signals at Purdue: Using learning analytics to increase student success. In Proceedings of the 2nd International Conference on Learning Analytics and Knowledge (pp. 267–270). New York. doi:10.1145/2330601.2330666
Reports on a system that faculty can use to determine which students in their class are likely to succeed and which may be at risk for failing or dropping out. The system uses an algorithm that analyzes data from multiple databases, including user-behavior data on the LMS, academic history (GPA, college-entrance test scores) as provided by administration databases. Researchers report a decline in dropout rates among students who used the system. However, current research calls those findings into question: ttp://www.insidehighered.com/
Baker, R. S. J., Gowda, S. M., Wixon, M., Kalka, J., Wagner, A. Z., Aleven, V., … Rossi, L. (2012). Towards sensor-free affect detection in cognitive tutor algebra. In K. Yacef, O. Zaïane, H. Hershkovitz, M. Yudelson, & J. Stamper (Eds.), Proceedings of the 5th International Conference on Educational Data Mining (pp. 126–133). Chania, Greece.
This conference presentation explains the development of a model to detect user affect from log data obtained from an intelligent tutoring system. Two observers coded student behavior with the system as exhibiting a particular emotion (confusion, engaged concentration, frustration, or boredom). Log data corresponding to the times students were being observed was analyzed for patterns that correlated with the affective state coded by the observers. J48 decision trees, step regression, JRip, Naïve Bayes, and REP-Trees were used to develop the detector models. The resulting models had between 70%-99% accuracy of identifying student affect, and the models had an average Kappa of 30%.
Researchers present a probabilistic model of a learner’s emotional experience to an educational game that could be used by an intelligent tutoring system to appropriately respond to learner affect during gameplay. The model takes into account a learner’s primary goal for playing the game, compares it to log data of user interaction with the system, and identifies the probable emotional reaction to gameplay. The goal of the model is to help students be emotionally engaged in positive ways while playing the game, specifically to have a positive experience with gameplay and the interactive computer agent.
Conati, C., & Maclaren, H. (2009). Empirically building and evaluating a probabilistic model of user affect. User Modeling and User-Adapted Interaction, 19(3), 267–303. doi:10.1007/s11257-009-9062-8
To validate the model, student emotional responses were recorded from a pop-up survey that would be presented to students while playing the game. Other studies had been mentioned that videotaped or had researchers observe participants during game play, and coded emotional reactions given in facial expressions (Kapoor & Picard, 2005; D’Mello et al., 2008). Conati and Maclaren (2009) first used this method, but found it difficult for coders to come to sufficient agreement on emotional responses with minimal facial expression. Student self-report was found as a useful validator, though the researchers planned to use physiological sensors in future studies so as not to disrupt students during gameplay. While others would argue against the use of self-report for emotional response because of its interference with engagement and the possibility that people do not fully understand their emotional responses (Woolf et al., 2009; Picard et al. (2004), Conati and Maclaren (2009) found that quick, pop-up self-report requests could be useful to a degree in identifying student emotional response in a way that was minimally interfering and annoying to students.
Cocea, M., & Weibelzahl, S. (2011). Disengagement detection in online learning: Validation studies and perspectives. IEEE Transactions on Learning Technologies, 4(2), 114–124. doi:10.1109/TLT.2010.14
Researchers turned to log data to find indicators of engagement of students using intelligent tutoring systems. By finding these indicators, the researchers hoped to develop a method of detecting student disengagement that could alert the system to intervene and help the student. In a previous study, the researchers built a model that would classify student behavior as recorded in log data as engaged, neutral, or disengaged. The researchers used the same methods to determine whether the indicators identified in the previous study would hold true in the 2011 study.
Human coders reviewed log files from 32 students in a web development course. Coders labeled behavior as engaged, neutral, or disengaged based on how long users were on a page, and whether it appeared that users were on task. Data analysis methods were used to discover meaningful relationships in behavior as recorded in log data, including Bayesian Nets, decision trees, and regression. Analysis determined the most important indicators of engagement or disengagement: average time spent on pages, number of pages viewed, number of tests taken, average time spent on tests, number of correctly answered tests, and the number of incorrectly answered tests. The use of the model on new data proved to be accurate at least 85% of the time. Indicators identified in this study were similar to indicators identified in the previous one, showing that the researchers methods may be independent of learning systems.
This study is important to researchers and developers hoping to identify indicators of engagement or disengagement that can be used in any learning context. One weakness of this study, however, is a lack of detail about the relationship between human coding and statistical analysis and how the two were used to develop and validate the model. The article explains that both types of analysis were done, but does not explain whether human coding was compared to statistical analysis in order to validate statistical models, or whether both methods were used to create the model. This may present a challenge to researchers hoping to replicate their methods.
Human coders reviewed log files from 32 students in a web development course. Coders labeled behavior as engaged, neutral, or disengaged based on how long users were on a page, and whether it appeared that users were on task. Data analysis methods were used to discover meaningful relationships in behavior as recorded in log data, including Bayesian Nets, decision trees, and regression. Analysis determined the most important indicators of engagement or disengagement: average time spent on pages, number of pages viewed, number of tests taken, average time spent on tests, number of correctly answered tests, and the number of incorrectly answered tests. The use of the model on new data proved to be accurate at least 85% of the time. Indicators identified in this study were similar to indicators identified in the previous one, showing that the researchers methods may be independent of learning systems.
This study is important to researchers and developers hoping to identify indicators of engagement or disengagement that can be used in any learning context. One weakness of this study, however, is a lack of detail about the relationship between human coding and statistical analysis and how the two were used to develop and validate the model. The article explains that both types of analysis were done, but does not explain whether human coding was compared to statistical analysis in order to validate statistical models, or whether both methods were used to create the model. This may present a challenge to researchers hoping to replicate their methods.
D’mello, S., & Graesser, A. (2012). AutoTutor and affective autotutor. ACM Transactions on Interactive Intelligent Systems, 2(4), 1–39. doi:10.1145/2395123.2395128
This article reviews work to develop systems that enable an avatar tutor on an intelligent tutoring system to respond to student's cognitive and affective states. The tutor can assess how much students know in relation to the content being taught by asking questions and analyzing answers. The tutor can also sense when a student is frustrated or bored by analyzing dialogue between the tutor and the student, tracking facial features and body language on web cams and a seat pressure detection system. Based on this data, the tutor can personalize a response and instruction for student users.
In developing the cognitive-aware AutoTutor, latent semantic analysis, regular expressions, content word overlap metrics, and logical entailment were used to analyze student input, and dialogue between tutor and student. Latent semantic analysis determines how alike conceptually two ideas are. This analysis helps the system determine if a student response is close to a correct answer or contains a misconception.
Expert physicists rated student responses as being correct or incorrect and compared it to AutoTutor's assessment. The correlation between the comparisons was between .35-.50.
AutoTutor was found to improve learning by 0-2.1 sigma (a mean of 0.8 when compared to students reading instruction of similar content from a textbook).
For the Affective AutoTutor, classifiers were built for target emotions by using human observers and correlating human assessments with the values recorded by the sensors. Latent semantic analysis, Naive Bayes logistic regression, support vector machines, and other standard classifiers were also used to categorize types of student responses for certain affective states. These classifiers had an accuracy of 42%-78% (depending on the affective state being categorized) when compared to assessments made by human judges. The seat pressure detection system algorithms for classifying affect had accuracies of 70%-83%. The facial recognition system had mixed success, with accuracies of 17% to 72% depending on the emotion being categorized. Affective AutoTutor had reported learning gains of .51 sigma when compared to students just using different versions of AutoTutor.
The abstract for this article is a very good summary:
“Csikszentmihalyi’s Flow theory states that a balance between challenge and skill leads to high engagement, overwhelming challenge leads to anxiety or frustration, and insufficient challenge leads to boredom. In this paper, we test this theory within the context of student interaction with an intelligent tutoring system. Automated detectors of student affect and knowledge were developed, validated, and applied to a large data set. The results did not match Flow theory: boredom was more common for poorly-known material, and frustration was common both for very difficult material and very easy material. These results suggest that design for optimal engagement within online learning may require further study of the factors leading students to become bored on difficult material, and frustrated on very well-known material.”
The authors used a Naive Bayesian model to predict student dropout risk. The model was created using LMS participation data. Visual and correlational analysis lead to the development of different types of intervention based on the risk level for a particular student. Intervention included “direct and informal contact via telephone.”
One key take away from this article is that, “Without the feedback from face-to-face interactions, lecturers using virtual learning environments may find it difficult to identify and focus on students who are struggling in class.” This is logical because diagnosing struggling students happens many times from gestural, or non-verbal communication cues. Another key take away is the emergence of predictive data to not only identify struggling students but to be able to improve the virtual learning experience. The context of this research is the UK’s Open University.
Woolf, B., Burleson, W., Arroyo, I., Dragon, T., Cooper, D., & Picard, R. (2009). Affect-aware tutors: Recognising and responding to student affect. International Journal of Learning Technology, 4(3/4), 129–164.
This article is a review of several studies done by the authors in detecting the emotion of students using e-learning systems, and the results of efforts made by the system to intervene based on certain detected affective states. Of particular interest are methods to recognize or measure a student’s emotional response to learning. The authors conducted a number of studies of student emotion using human observation, sensors, or machine learning.
Human observers tracked facial, physical, and verbal expressions as well as on-task and off-task behavior while students used a tutoring software in a computer lab. Expressions were labeled using four categories: positive or negative valence (nature of emotion) and high or low arousal (amount of physical activity in relation to the emotion). Expressions were also labeled as desirable or undesirable based on on-task/off-task behavior. The combination of labels were used to define different emotions. For example, positive valence and high arousal was defined as being joyful or excited, while negative valence and low arousal was defined as tired or bored. The researchers found statistically significant correlations with certain coded emotions and pre-test scores and learning orientation surveys (N = 34, p < .01). On-task behavior correlated with post-test math scores: (N=34, R=.640, p<.018).
In other studies, the researchers used sensors to detect student emotion. Sensors included facial expression recognition using video analysis, posture analysis using seat pressure systems, hand pressure on mouse, and skin conductance systems. The researchers found a mix of significant results from using the sensors. Using the measurements from all the sensors was best in detecting frustration, which combination had an accuracy of predicting frustration 89% of the time; however, looking at only measurements from the mouse pressure system was best for detecting interest, which had a detection accuracy of 72.67%.
Finally, the researchers used machine learning to detect student emotion. Learner behavior as recorded in log data (i.e. number of pages visited, time spent on a page) and results of a motivation survey were analyzed using Bayesian networks to discover meaningful relationships. The resulting model was accurate 80-90% of the time in identifying a student’s emotional response as revealed in the motivation survey based on values of recorded log data.
In developing the cognitive-aware AutoTutor, latent semantic analysis, regular expressions, content word overlap metrics, and logical entailment were used to analyze student input, and dialogue between tutor and student. Latent semantic analysis determines how alike conceptually two ideas are. This analysis helps the system determine if a student response is close to a correct answer or contains a misconception.
Expert physicists rated student responses as being correct or incorrect and compared it to AutoTutor's assessment. The correlation between the comparisons was between .35-.50.
AutoTutor was found to improve learning by 0-2.1 sigma (a mean of 0.8 when compared to students reading instruction of similar content from a textbook).
For the Affective AutoTutor, classifiers were built for target emotions by using human observers and correlating human assessments with the values recorded by the sensors. Latent semantic analysis, Naive Bayes logistic regression, support vector machines, and other standard classifiers were also used to categorize types of student responses for certain affective states. These classifiers had an accuracy of 42%-78% (depending on the affective state being categorized) when compared to assessments made by human judges. The seat pressure detection system algorithms for classifying affect had accuracies of 70%-83%. The facial recognition system had mixed success, with accuracies of 17% to 72% depending on the emotion being categorized. Affective AutoTutor had reported learning gains of .51 sigma when compared to students just using different versions of AutoTutor.
Frankfort, J., Salim, K., Carmean, C., & Haynie, T. (2012, July). Analytics, nudges, and learner persistence. Educause Review.
A report on a system that uses self-report from students (how are you feeling today?) by using a mobile app and text messages, along with information about the course, student activity, and performance to provide personalized "nudges" to encourage students and help students have better study skills (e.g. deciding when to study for the next test). Initial studies found that students who used this system performed better than students who did not. There also seemed to be positive feedback from students on the system, and good indications that the system is actually used. How the system works is not explained in detail (algorithms, data analysis, specific types of data collected), but this does show how different types of data can work together to help learners.
Lynch-Holmes, K., & Mason, T. (2012, August). Building a purpose network to increase student engagement and retention. Educause Review.
This Educause Review article reports on a system at Fort Hays State University, partnered with ConnectEDU, that helps alert teachers, parents, resident assistants or student leaders when a student is at risk for dropping out of college. The data that is used to identify an at risk student is not mentioned, but it seems to come from many sources within the university.
Macfadyen, L. P., & Dawson, S. (2010). Mining LMS data to develop an “early warning system” for educators: A proof of concept. Computers & Education, 54(2), 588–599. doi:10.1016/j.compedu.2009.09.008
In this study, Macfadyen & Dawson (2010) investigate log data obtained from 5 sections of an online biology course offered on the BlackBoard Vista learning management system to determine which engagement factors best predict academic success. Thirteen variables were found to have a statistically significant correlation with students’ final grades (p < .01), including total number of discussion messages posted, total time online, and the number of web links viewed. A linear multiple regression analysis was used to develop a predictive model of behaviors that best predicted student final grades. Analysis found three highly correlated variables of significance (p=.00) to make that model: Total number of discussion messages posted, total number of assessments finished, and number of mail messages sent. A binary logistic regression was then used to categorize students at risk for failure using the validated model. The model was found to categorize accurately 73.3% of the time. The results of this study do not seem to say anything surprising about learning: those who do the work tend to succeed in class. The developed model also relies on data obtained over the course of an entire semester, which may not be useful for identifying at-risk students early on in a course. The study does show, however, that log data can be used to productively describe online learner behavior.
Frequency and duration of participation measures were tracked for a completely asynchronous online course to try to better understand student engagement. Differences in online participation were categorized in multiple ways--withdrawers vs completers, and successful completers and non-successful completers. The study explained 31% of the variance in achievement based on the participation measures.
Macfadyen, L. P., & Dawson, S. (2010). Mining LMS data to develop an “early warning system” for educators: A proof of concept. Computers & Education, 54(2), 588–599. doi:10.1016/j.compedu.2009.09.008
Morris, L. V., Finnegan, C., & Wu, S.-S. (2005). Tracking student behavior, persistence, and achievement in online courses. The Internet and Higher Education, 8(3), 221–231. doi:10.1016/j.iheduc.2005.06.009
San Pedro, M. O. Z., Baker, R. S. J., Gowda, S. M., & Heffernan, N. T. (2013). Towards an understanding of affect and knowledge from student interaction with an intelligent tutoring system. In Proceedings of the 16th International Conference on Artificial Intelligence and Education.
“Csikszentmihalyi’s Flow theory states that a balance between challenge and skill leads to high engagement, overwhelming challenge leads to anxiety or frustration, and insufficient challenge leads to boredom. In this paper, we test this theory within the context of student interaction with an intelligent tutoring system. Automated detectors of student affect and knowledge were developed, validated, and applied to a large data set. The results did not match Flow theory: boredom was more common for poorly-known material, and frustration was common both for very difficult material and very easy material. These results suggest that design for optimal engagement within online learning may require further study of the factors leading students to become bored on difficult material, and frustrated on very well-known material.”
Smith, V. C., Ph, D., & Lange, A. (2012). Predictive modeling to forecase student outcomes and drive effective interventions in online community college courses. Journal of Asynchronous Learning Networks, 16(3), 51–61.
Wolff, A., & Zdrahal, Z. (2012, July). Improving retention by identifying and supporting “at-risk” students. Educause Review.
Woolf, B., Burleson, W., Arroyo, I., Dragon, T., Cooper, D., & Picard, R. (2009). Affect-aware tutors: Recognising and responding to student affect. International Journal of Learning Technology, 4(3/4), 129–164.
This article is a review of several studies done by the authors in detecting the emotion of students using e-learning systems, and the results of efforts made by the system to intervene based on certain detected affective states. Of particular interest are methods to recognize or measure a student’s emotional response to learning. The authors conducted a number of studies of student emotion using human observation, sensors, or machine learning.
Human observers tracked facial, physical, and verbal expressions as well as on-task and off-task behavior while students used a tutoring software in a computer lab. Expressions were labeled using four categories: positive or negative valence (nature of emotion) and high or low arousal (amount of physical activity in relation to the emotion). Expressions were also labeled as desirable or undesirable based on on-task/off-task behavior. The combination of labels were used to define different emotions. For example, positive valence and high arousal was defined as being joyful or excited, while negative valence and low arousal was defined as tired or bored. The researchers found statistically significant correlations with certain coded emotions and pre-test scores and learning orientation surveys (N = 34, p < .01). On-task behavior correlated with post-test math scores: (N=34, R=.640, p<.018).
In other studies, the researchers used sensors to detect student emotion. Sensors included facial expression recognition using video analysis, posture analysis using seat pressure systems, hand pressure on mouse, and skin conductance systems. The researchers found a mix of significant results from using the sensors. Using the measurements from all the sensors was best in detecting frustration, which combination had an accuracy of predicting frustration 89% of the time; however, looking at only measurements from the mouse pressure system was best for detecting interest, which had a detection accuracy of 72.67%.
Finally, the researchers used machine learning to detect student emotion. Learner behavior as recorded in log data (i.e. number of pages visited, time spent on a page) and results of a motivation survey were analyzed using Bayesian networks to discover meaningful relationships. The resulting model was accurate 80-90% of the time in identifying a student’s emotional response as revealed in the motivation survey based on values of recorded log data.
2. Visualization
Duval, E. (2011). Attention Please ! Learning Analytics for Visualization and Recommendation. In Proceedings of LAK11: 1st International Conference on Learning Analytics and Knowledge (pp. 9–17). Banff, Canada.
Gives examples of visualization applications in industry and then in education. Outlines steps to help make learning analytics useful to education, including ontologies that track, label, and group types of learning activities, how to present information obtained through data mining (visualization through dashboards), to recommend learning choices based on user rating feedback data, to make data shareable, and to identify which data (and patterns of data) is important.
3. Relationship Mining
Arroyo, I., & Woolf, B. P. (2005). Inferring learning and attitudes from a Bayesian Network of log file data. In C. K. Looie, G. McCalla, B. Bredeweg, & J. Breuker (Eds.), Proceedings of the 12th International Conference on Artificial Intelligence in Education (pp. 33–40). Amsterdam: IOS Press.
The researchers' goal was to discover relationships between "observable variables" (student actions on the system recorded as log data) and "hidden variables" (student self-report on attitudes and perceived learning). Correlations were mined among variables gathered. Researchers used Bayesian statistics to try to predict what the value of a "hidden variable" would be from correlating "observable variables." The accuracy of this model was then tested (average accuracy of 80%).
In this conference presentation, Beer, Clark, and Jones (2009) review the results of their analysis of student behavior on BlackBoard and Moodle learning management systems. The purpose of this review was to identify whether variables from the log data files could be profitably used to indicate learner engagement. Data was obtained from over 90,000 undergraduate students in 2,714 courses for a period of five years. A general correlation was found between the number of recorded clicks from a student using the learning management system and final grades. The average number of clicks from students who failed was 219.47, 555.22 for those with average grades, and 730.06 for those with the highest grades. While making more clicks is assumed to not make someone more intelligent, measuring clicks may be a useful indicator of how well a student is engaging in online learning material. Though this conference presentation did not provide a thorough review of statistical analyses, it is useful in identifying the value of studying log data to better understand learning in online contexts.
Beer, C., Clark, K., & Jones, D. (2010). Indicators of engagement. In Proceedings of ASCILITE 2010 (pp. 75–86). Sydney, Australia.
Dawson, S., & McWilliam, E. (2008). Investigating the application of IT generated data as an indicator of learning and teaching performance.
This report for the Australian Learning and Teaching Council describes efforts to use user activity data on an LMS extracted from log files to predict student learning and evaluate instruction. Data extracted include read mail messages, sent mail messages, read discussion messages, posted discussion messages, viewed calendar entries, assessments started, assessments finished, total time spend on assessments, assignments read, assignments submitted, total time spent on assignment. Correlational, Mann-Whitney, Kruskal Wallis, and social network analysis were used to analyze the data. Those who participated more (as measured by number of content views, discussion postings, time spend on the LMS, tended to have higher grades.
Merceron, A., & Yacef, K. (2008). Interestingness measures for association rules in educational data. In The 1st International Conference on Educational Data Mining (pp. 57–66). Montreal, Canada.
Whitmer, J., Fernandes, K., & Allen, W. R. (2012, August). Analytics in progress: Technology use, student characteristics, and student achievement. Educause Review. Retrieved from http://www.educause.edu/ero/article/analytics-progress-technology-use-student-characteristics-and-student-achievement
4. Structure Discovery
Kinnebrew, J. S., Biswas, G., Sulcer, B., & Sta, B. (2008). Modeling and measuring self-regulated learning in teachable agent environments. Journal of E-Learning and Knowledge Society, 7(2), 19–35.
The researchers report on an analysis of student behaviors, performance, and learning with an intelligent tutoring system under different instructional approaches. The instructional approaches included an intelligent coaching system, and two learning by teaching approaches, one of which used self-regulated learning suggestions. Types of behaviors with the system were identified (e.g. applied reading, uniformed editing, checking, probing). A hidden Markov model was used to look for patterns in the sequence of behaviors to see if certain behaviors were more prevalent in one instructional condition than in others. Hidden Markov models are used to develop pattern recognition. It was found that students learning actions were more effective and led to greater learning results that were statistically significant than the actions of students in the other interventions (p<.1, effect size d=.72).
In this study, the authors try to identify variables from log files that best fit in a framework of motivation. The motivation framework was made by doing a literature review. Variables from log data were then identified. They did hierarchical clustering using Pearson Correlation Distance as measure and Between-groups Linkage as the clustering method to identify which variables grouped together. Then, they inferred which category in the theoretical framework of motivation that the grouped variables belonged.
The outcome of this article is to help teams improve their effectiveness by applying a methodology called mirror information. The context of this research is a software development team. Because the group work can be tracked through online tools, data can be collected and analyzed to identify “better” groups from “weaker” groups. Best practices can be identified. New groups can be given this information and evaluated based on these findings.
This research reports on the effect error-rate, hint-use, and time-spent have on learning in the context of multiple representations. The context is elementary school math. Students were instructed and outcomes measured that found that error-rate, hint-use, and time-spent were the best variables to use to regress outcome in the presence of multiple representations. The study used path analysis and sequential equation modeling methods to test their hypothesis.
Thompson, K., Kennedy-Clark, S., Markauskaite, L., & Southavilay, V. (2014). Discovering processes and patterns of learning in collaborative learning environments using multi-modal discourse analysis. Research and Practice in Technology Enhanced Learning, 9(2), 215–240.
This study used first order Markov models and heuristic mining algorithm to discover patterns of behavior in a computer-supported collaborative learning activity. Student discourse was recorded and coded using the Decision-Function Coding Scheme and matched with activity with the system recorded in the log data. The process mining algorithms were then used to see the relationships between certain decision-making and group work actions between structured and unstructured groups and groups who successfully completed the assignment and those that did not. The researchers found that group orientation, agreement, and actions taken to implement a decision can be important indicators of successful group work.
Hershkovitz, A., & Nachmias, R. (2009). Developing a log-based motivation measuring tool. In Proceedings of 1st International Conference on Educational Data Mining (pp. 99–106). Montreal, Canada.
Perera, D., Kay, J., Koprinska, I., Yacef, K., & Zaiane, O. R. (2009). Clustering and sequential pattern mining of online collaborative learning data. IEEE Transactions on Knowledge and Data Engineering, 21(6), 759–772. doi:10.1109/TKDE.2008.138
Rau, M. A., & Scheines, R. (2012). Searching for variables and models to investigate mediators of learning from multiple representations. In Proceedings of the 5th International Conference on Educational Data Mining (pp. 110–117).
Thompson, K., Kennedy-Clark, S., Markauskaite, L., & Southavilay, V. (2014). Discovering processes and patterns of learning in collaborative learning environments using multi-modal discourse analysis. Research and Practice in Technology Enhanced Learning, 9(2), 215–240.
This study used first order Markov models and heuristic mining algorithm to discover patterns of behavior in a computer-supported collaborative learning activity. Student discourse was recorded and coded using the Decision-Function Coding Scheme and matched with activity with the system recorded in the log data. The process mining algorithms were then used to see the relationships between certain decision-making and group work actions between structured and unstructured groups and groups who successfully completed the assignment and those that did not. The researchers found that group orientation, agreement, and actions taken to implement a decision can be important indicators of successful group work.
5. Discovery with Models
Aleven, V., Mclaren, B., Roll, I., & Koedinger, K. (2006). Toward meta-cognitive tutoring : A model of help seeking with a cognitive tutor. International Journal of Artificial Intelligence in Education, 16(2), 101–128.
Researchers developed a model that a system could use to identify when students should seek help while learning on an intelligent tutoring system. Their model was based on the researchers' intuition and experience developing and using a cognitive tutor. The model was then applied to preexisting student data from using the system. Changes were made to the model after analysis and then used on real students. The model was shown to be useful in helping students learn when to seek help.
Hershkovitz, a., de Baker, R. S. J., Gobert, J., Wixon, M., & Pedro, M. S. (2013). Discovery with models: A case study on carelessness in computer-based science inquiry. American Behavioral Scientist, 57(10), 1480–1499. doi:10.1177/0002764213479365
This article discusses a new method in educational data mining research called discovery with models. The article outlines this method and presents a case study where the method is used. Discovery with models involves building a model either through machine learning or knowledge engineering techniques, ensuring that the model is valid and generalizable, and using that model to discover patterns in new types of data. In the case study, the authors built a model of student carelessness in completing learning activities (a virtual lab) using Bayesian statistics and decision trees. They then applied the model to student responses on surveys of motivation and goal-orientation using correlational and cluster analysis. They found that students who exhibited careless behaviors tended to feel confident about their learning abilities.
6. Other
Liu, H., Member, S., Yu, L., & Member, S. (2005). Toward integrating feature selection algorithms for classification and clustering. IEEE Transactions on Knowledge and Data Engineering, 17(4), 491–502.
This article is a review of feature selection approaches--the process of deciding which variables actually matter for what one is trying to do with the data. Feature selection is useful/necessary when one is dealing with hundreds and thousands of different variables in a data set. The process of feature selection is described. A framework is presented to categorize different algorithms for feature selection. The authors discuss ways to best decide which algorithms are best for which purposes. The authors conclude with real-world applications where feature selection is used, and challenges to address to improve feature selection methods.
Mauger, A. J., Schwartz, C. M., & Grieco, S. J. (2012, July). Efficiencies, learning outcomes bolstered by analytics, data-informed decision making. Educause Review.
Alphabetized Version
Researchers developed a model that a system could use to identify when students should seek help while learning on an intelligent tutoring system. Their model was based on the researchers' intuition and experience developing and using a cognitive tutor. The model was then applied to preexisting student data from using the system. Changes were made to the model after analysis and then used on real students. The model was shown to be useful in helping students learn when to seek help.
Arnold, K. E., &
Pistilli, M. D. (2012). Course Signals at Purdue: Using learning analytics to
increase student success. In Proceedings
of the 2nd International Conference on Learning Analytics and Knowledge
(pp. 267–270). New York. doi:10.1145/2330601.2330666
Reports on a system that faculty can use to determine which students in their class are likely to succeed and which may be at risk for failing or dropping out. The system uses an algorithm that analyzes data from multiple databases, including user-behavior data on the LMS, academic history (GPA, college-entrance test scores) as provided by administration databases. Researchers report a decline in dropout rates among students who used the system. However, current research calls those findings into question: ttp://www.insidehighered.com/
Arroyo, I., &
Woolf, B. P. (2005). Inferring learning and attitudes from a Bayesian Network
of log file data. In C. K. Looie, G. McCalla, B. Bredeweg, & J. Breuker
(Eds.), Proceedings of the 12th
International Conference on Artificial Intelligence in Education (pp.
33–40). Amsterdam: IOS Press.
The researchers' goal was to discover relationships between "observable variables" (student actions on the system recorded as log data) and "hidden variables" (student self-report on attitudes and perceived learning). Correlations were mined among variables gathered. Researchers used Bayesian statistics to try to predict what the value of a "hidden variable" would be from correlating "observable variables." The accuracy of this model was then tested (average accuracy of 80%).
Baker, R. S. J.,
Gowda, S. M., Wixon, M., Kalka, J., Wagner, A. Z., Aleven, V., … Rossi, L.
(2012). Towards sensor-free affect detection in cognitive tutor algebra. In K.
Yacef, O. Zaïane, H. Hershkovitz, M. Yudelson, & J. Stamper (Eds.), Proceedings of the 5th International
Conference on Educational Data Mining (pp. 126–133). Chania, Greece.
This conference presentation explains the development of a model to detect user affect from log data obtained from an intelligent tutoring system. Two observers coded student behavior with the system as exhibiting a particular emotion (confusion, engaged concentration, frustration, or boredom). Log data corresponding to the times students were being observed was analyzed for patterns that correlated with the affective state coded by the observers. J48 decision trees, step regression, JRip, Naïve Bayes, and REP-Trees were used to develop the detector models. The resulting models had between 70%-99% accuracy of identifying student affect, and the models had an average Kappa of 30%.
Baker, R. S. J. D.,
& Yacef, K. (2009). The state of educational data mining in 2009 : A review and future visions. Journal of Educational Data Mining, 1(1), 3–16.
Provides an overview of the field of educational data
mining. Defines educational data mining, identifies common methodologies, and
examines current applications. Reviews the top cited research in the field
between 1995 and 2005. Identifies past trends in research (applications and
methodologies). Highlights potential future trends and research needs.
Baker, R. S. J. d.,
& Siemens, G. (2014). Educational Data Mining and Learning Analytics.
Manuscript to be published in Cambridge
Handbook of the Learning Sciences.
An updated overview of the field of educational data mining
in addition to an overview of the field of learning analytics. Published for
the Learning Sciences audience. Gives definitions of educational data mining
and learning analytics. Describes methods of analysis, tools for analysis,
applications, and future trends for educational data mining and learning
analytics in the Learning Sciences. One noteworthy application discussed was
methods used to detect disengagement in students (using prediction methods and
knowledge engineering).
Beer, C., Clark, K., & Jones, D. (2010). Indicators of engagement. In Proceedings of ASCILITE 2010 (pp. 75–86). Sydney, Australia.
Bienkowski, M., Feng,
M., & Means, B. (2012). Enhancing
teaching and learning through educational data mining and learning analytics:
An issue brief. Washington, DC: SRI International. Retrieved from http://www.ed.gov/edblogs/technology/files/2012/03/edm-la-brief.pdf
In this report, researchers review the growing work done on educational data mining and learning analytics. Researchers discuss applications to education, review possibilities and current examples, examine challenges to the field, and identifies future areas of research. This report provides a list of websites using these techniques that one can explore.
Campbell, B. J. P.,
Deblois, P. B., & Oblinger, D. G. (2007). Academic analytics: A new tool
for a new era. Educause Review,
(August 2007), 41–57.
Introduces the idea of academic analytics ("actionable
intelligence to improve teaching, learning, and student success"), and
reviews examples of research and technology done that matches ideals of the
initiative. Also identifies needs and challenges facing this area of research
and development, such as data ownership and privacy, sharing information, and
profiling.
Conati, C., & Maclaren, H. (2009). Empirically building and evaluating a probabilistic model of user affect. User Modeling and User-Adapted Interaction, 19(3), 267–303. doi:10.1007/s11257-009-9062-8
To validate the model, student emotional responses were recorded from a pop-up survey that would be presented to students while playing the game. Other studies had been mentioned that videotaped or had researchers observe participants during game play, and coded emotional reactions given in facial expressions (Kapoor & Picard, 2005; D’Mello et al., 2008). Conati and Maclaren (2009) first used this method, but found it difficult for coders to come to sufficient agreement on emotional responses with minimal facial expression. Student self-report was found as a useful validator, though the researchers planned to use physiological sensors in future studies so as not to disrupt students during gameplay. While others would argue against the use of self-report for emotional response because of its interference with engagement and the possibility that people do not fully understand their emotional responses (Woolf et al., 2009; Picard et al. (2004), Conati and Maclaren (2009) found that quick, pop-up self-report requests could be useful to a degree in identifying student emotional response in a way that was minimally interfering and annoying to students.
In this study, the authors try to identify variables from log files that best fit in a framework of motivation. The motivation framework was made by doing a literature review. Variables from log data were then identified. They did hierarchical clustering using Pearson Correlation Distance as measure and Between-groups LInkage as the clustering method to identify which variables grouped together. Then, they inferred which category in the theoretical framework of motivation that the grouped variables belonged.
Liu, H., Member, S., Yu, L., & Member, S. (2005). Toward integrating feature selection algorithms for classification and clustering. IEEE Transactions on Knowledge and Data Engineering, 17(4), 491–502.
This paper describes how one institution began creating a data-informed culture to improve course offerings and allocation of facilities on campus. Examples include improving the use of classrooms, optimizing the offerings in music, dance, and theatre courses to better match students’ interests, and launching a career coach website to educate students and job seekers about the job market. Data used in this example include enrollment data, course and student outcome data, classroom scheduling data, and tech support trouble ticket data.
Merceron and Yacef argue for the use of cosine and added value (similar to lift) as measures of interestingness, which are separate from two standard measures of association rules - confidence and support. They substantiate their argument by working through a case study using LMS data. Cosine and added value were used to measure the usefulness of LMS-based instruction aimed to augment the face-to-face instruction in the classroom.
Frequency and duration of participation measures were tracked for a completely asynchronous online course to try to better understand student engagement. Differences in online participation were categorized in multiple ways--withdrawers vs completers, and successful completers and non-successful completers. The study explained 31% of the variance in achievement based on the participation measures.
The outcome of this article is to help teams improve their effectiveness by applying a methodology called mirror information. The context of this research is a software development team. Because the group work can be tracked through online tools, data can be collected and analyzed to identify “better” groups from “weaker” groups. Best practices can be identified. New groups can be given this information and evaluated based on these findings.
This research reports on the effect error-rate, hint-use, and time-spent have on learning in the context of multiple representations. The context is elementary school math. Students were instructed and outcomes measured that found that error-rate, hint-use, and time-spent were the best variables to use to regress outcome in the presence of multiple representations. The study used path analysis and sequential equation modeling methods to test their hypothesis.
Romero, C., Ventura, S., & Garcia, E. Data mining in course management systems: Moodle case study and tutorial. Computers and Education, 51(1), 368-384.
This article reviews the purpose and process of doing data analytics in education (collecting, preprocessing, mining, interpreting, and applying). It discusses specific applications in doing data mining with Moodle, providing examples of data mining decisions and tools.
The abstract for this article is a very good summary:
“Csikszentmihalyi’s Flow theory states that a balance between challenge and skill leads to high engagement, overwhelming challenge leads to anxiety or frustration, and insufficient challenge leads to boredom. In this paper, we test this theory within the context of student interaction with an intelligent tutoring system. Automated detectors of student affect and knowledge were developed, validated, and applied to a large data set. The results did not match Flow theory: boredom was more common for poorly-known material, and frustration was common both for very difficult material and very easy material. These results suggest that design for optimal engagement within online learning may require further study of the factors leading students to become bored on difficult material, and frustrated on very well-known material.”
This important article reviews the current status of the development of the Learning Analytics field and includes reviews of such topics as the need to increase data scope, privacy and ownership of data, and the immaturity of the legal infrastructure in place in terms of privacy and ethics. The article argues that Learning Analytics has become and deserves to be viewed as, “an emerging research field.”
The authors used a Naive Bayesian model to predict student dropout risk. The model was created using LMS participation data. Visual and correlational analysis lead to the development of different types of intervention based on the risk level for a particular student. Intervention included “direct and informal contact via telephone.”
This research reports on the connection between student achievement and LMS usage and student characteristics. This research made more detailed categorization of LMS interaction (e.g., discussion group participation) to study the the relationship between LMS use and achievement. One finding is the concern over the accuracy of time measures in the LMS, which was found to be skewed to 0. Another finding was that lower-income students spend more time in the LMS than their counterpart. And that lower-income students experience a lower “efficiency” in terms of time on task.
One key take away from this article is that, “Without the feedback from face-to-face interactions, lecturers using virtual learning environments may find it difficult to identify and focus on students who are struggling in class.” This is logical because diagnosing struggling students happens many times from gestural, or non-verbal communication cues. Another key take away is the emergence of predictive data to not only identify struggling students but to be able to improve the virtual learning experience. The context of this research is the UK’s Open University.
Woolf, B., Burleson, W., Arroyo, I., Dragon, T., Cooper, D., & Picard, R. (2009). Affect-aware tutors: Recognising and responding to student affect. International Journal of Learning Technology, 4(3/4), 129–164.
This article is a review of several studies done by the authors in detecting the emotion of students using e-learning systems, and the results of efforts made by the system to intervene based on certain detected affective states. Of particular interest are methods to recognize or measure a student’s emotional response to learning. The authors conducted a number of studies of student emotion using human observation, sensors, or machine learning.
Human observers tracked facial, physical, and verbal expressions as well as on-task and off-task behavior while students used a tutoring software in a computer lab. Expressions were labeled using four categories: positive or negative valence (nature of emotion) and high or low arousal (amount of physical activity in relation to the emotion). Expressions were also labeled as desirable or undesirable based on on-task/off-task behavior. The combination of labels were used to define different emotions. For example, positive valence and high arousal was defined as being joyful or excited, while negative valence and low arousal was defined as tired or bored. The researchers found statistically significant correlations with certain coded emotions and pre-test scores and learning orientation surveys (N = 34, p < .01). On-task behavior correlated with post-test math scores: (N=34, R=.640, p<.018).
In other studies, the researchers used sensors to detect student emotion. Sensors included facial expression recognition using video analysis, posture analysis using seat pressure systems, hand pressure on mouse, and skin conductance systems. The researchers found a mix of significant results from using the sensors. Using the measurements from all the sensors was best in detecting frustration, which combination had an accuracy of predicting frustration 89% of the time; however, looking at only measurements from the mouse pressure system was best for detecting interest, which had a detection accuracy of 72.67%.
Finally, the researchers used machine learning to detect student emotion. Learner behavior as recorded in log data (i.e. number of pages visited, time spent on a page) and results of a motivation survey were analyzed using Bayesian networks to discover meaningful relationships. The resulting model was accurate 80-90% of the time in identifying a student’s emotional response as revealed in the motivation survey based on values of recorded log data.
Cocea, M., & Weibelzahl, S. (2011). Disengagement detection in online learning: Validation studies and perspectives. IEEE Transactions on Learning Technologies, 4(2), 114–124. doi:10.1109/TLT.2010.14
Researchers turned to log data to find indicators of engagement of students using intelligent tutoring systems. By finding these indicators, the researchers hoped to develop a method of detecting student disengagement that could alert the system to intervene and help the student. In a previous study, the researchers built a model that would classify student behavior as recorded in log data as engaged, neutral, or disengaged. The researchers used the same methods to determine whether the indicators identified in the previous study would hold true in the 2011 study.
Human coders reviewed log files from 32 students in a web development course. Coders labeled behavior as engaged, neutral, or disengaged based on how long users were on a page, and whether it appeared that users were on task. Data analysis methods were used to discover meaningful relationships in behavior as recorded in log data, including Bayesian Nets, decision trees, and regression. Analysis determined the most important indicators of engagement or disengagement: average time spent on pages, number of pages viewed, number of tests taken, average time spent on tests, number of correctly answered tests, and the number of incorrectly answered tests. The use of the model on new data proved to be accurate at least 85% of the time. Indicators identified in this study were similar to indicators identified in the previous one, showing that the researchers methods may be independent of learning systems.
This study is important to researchers and developers hoping to identify indicators of engagement or disengagement that can be used in any learning context. One weakness of this study, however, is a lack of detail about the relationship between human coding and statistical analysis and how the two were used to develop and validate the model. The article explains that both types of analysis were done, but does not explain whether human coding was compared to statistical analysis in order to validate statistical models, or whether both methods were used to create the model. This may present a challenge to researchers hoping to replicate their methods.
D’mello, S., & Graesser, A. (2012). AutoTutor and affective autotutor. ACM Transactions on Interactive Intelligent Systems, 2(4), 1–39. doi:10.1145/2395123.2395128
Human coders reviewed log files from 32 students in a web development course. Coders labeled behavior as engaged, neutral, or disengaged based on how long users were on a page, and whether it appeared that users were on task. Data analysis methods were used to discover meaningful relationships in behavior as recorded in log data, including Bayesian Nets, decision trees, and regression. Analysis determined the most important indicators of engagement or disengagement: average time spent on pages, number of pages viewed, number of tests taken, average time spent on tests, number of correctly answered tests, and the number of incorrectly answered tests. The use of the model on new data proved to be accurate at least 85% of the time. Indicators identified in this study were similar to indicators identified in the previous one, showing that the researchers methods may be independent of learning systems.
This study is important to researchers and developers hoping to identify indicators of engagement or disengagement that can be used in any learning context. One weakness of this study, however, is a lack of detail about the relationship between human coding and statistical analysis and how the two were used to develop and validate the model. The article explains that both types of analysis were done, but does not explain whether human coding was compared to statistical analysis in order to validate statistical models, or whether both methods were used to create the model. This may present a challenge to researchers hoping to replicate their methods.
D’mello, S., & Graesser, A. (2012). AutoTutor and affective autotutor. ACM Transactions on Interactive Intelligent Systems, 2(4), 1–39. doi:10.1145/2395123.2395128
This article reviews work to develop systems that enable an avatar tutor on an intelligent tutoring system to respond to student's cognitive and affective states. The tutor can assess how much students know in relation to the content being taught by asking questions and analyzing answers. The tutor can also sense when a student is frustrated or bored by analyzing dialogue between the tutor and the student, tracking facial features and body language on web cams and a seat pressure detection system. Based on this data, the tutor can personalize a response and instruction for student users.
In developing the cognitive-aware AutoTutor, latent semantic analysis, regular expressions, content word overlap metrics, and logical entailment were used to analyze student input, and dialogue between tutor and student. Latent semantic analysis determines how alike conceptually two ideas are. This analysis helps the system determine if a student response is close to a correct answer or contains a misconception.
Expert physicists rated student responses as being correct or incorrect and compared it to AutoTutor's assessment. The correlation between the comparisons was between .35-.50.
AutoTutor was found to improve learning by 0-2.1 sigma (a mean of 0.8 when compared to students reading instruction of similar content from a textbook).
For the Affective AutoTutor, classifiers were built for target emotions by using human observers and correlating human assessments with the values recorded by the sensors. Latent semantic analysis, Naive Bayes logistic regression, support vector machines, and other standard classifiers were also used to categorize types of student responses for certain affective states. These classifiers had an accuracy of 42%-78% (depending on the affective state being categorized) when compared to assessments made by human judges. The seat pressure detection system algorithms for classifying affect had accuracies of 70%-83%. The facial recognition system had mixed success, with accuracies of 17% to 72% depending on the emotion being categorized. Affective AutoTutor had reported learning gains of .51 sigma when compared to students just using different versions of AutoTutor.
In developing the cognitive-aware AutoTutor, latent semantic analysis, regular expressions, content word overlap metrics, and logical entailment were used to analyze student input, and dialogue between tutor and student. Latent semantic analysis determines how alike conceptually two ideas are. This analysis helps the system determine if a student response is close to a correct answer or contains a misconception.
Expert physicists rated student responses as being correct or incorrect and compared it to AutoTutor's assessment. The correlation between the comparisons was between .35-.50.
AutoTutor was found to improve learning by 0-2.1 sigma (a mean of 0.8 when compared to students reading instruction of similar content from a textbook).
For the Affective AutoTutor, classifiers were built for target emotions by using human observers and correlating human assessments with the values recorded by the sensors. Latent semantic analysis, Naive Bayes logistic regression, support vector machines, and other standard classifiers were also used to categorize types of student responses for certain affective states. These classifiers had an accuracy of 42%-78% (depending on the affective state being categorized) when compared to assessments made by human judges. The seat pressure detection system algorithms for classifying affect had accuracies of 70%-83%. The facial recognition system had mixed success, with accuracies of 17% to 72% depending on the emotion being categorized. Affective AutoTutor had reported learning gains of .51 sigma when compared to students just using different versions of AutoTutor.
Dawson, S., &
McWilliam, E. (2008). Investigating the
application of IT generated data as an indicator of learning and teaching
performance.
This report for the Australian Learning and Teaching Council describes efforts to use user activity data on an LMS extracted from log files to predict student learning and evaluate instruction. Data extracted include read mail messages, sent mail messages, read discussion messages, posted discussion messages, viewed calendar entries, assessments started, assessments finished, total time spend on assessments, assignments read, assignments submitted, total time spent on assignment. Correlational, Mann-Whitney, Kruskal Wallis, and social network analysis were used to analyze the data. Those who participated more (as measured by number of content views, discussion postings, time spend on the LMS, tended to have higher grades.
Duval, E. (2011).
Attention Please !
Learning Analytics for Visualization and Recommendation. In Proceedings of LAK11: 1st International
Conference on Learning Analytics and Knowledge (pp. 9–17). Banff, Canada.
Gives examples of visualization applications in industry and
then in education. Outlines steps to help make learning analytics useful to
education, including ontologies that track, label, and group types of learning
activities, how to present information obtained through data mining
(visualization through dashboards), to recommend learning choices based on user
rating feedback data, to make data shareable, and to identify which data (and
patterns of data) is important.
Ferguson, R. (2012).
Learning analytics: Drivers, developments and challenges. International
Journal of Technology Enhanced Learning, 4(5/6), 304–317.
doi:10.1504/IJTEL.2012.051816
This article gives an introduction to the field of learning analytics. It discusses definitions, societal drivers that led to the development of learning analytics, and the history of the emergence of data analytics in education. The author also distinguishes learning analytics from academic analytics and educational data mining. Learning analytics is more about the learner's perspective and needs, about social perspectives on learning, and has close ties to the learning sciences. Future challenges to learning analytics are described.
Frankfort, J., Salim,
K., Carmean, C., & Haynie, T. (2012, July). Analytics, nudges, and learner
persistence. Educause Review.
A report on a system that uses self-report from students (how are you feeling today?) by using a mobile app and text messages, along with information about the course, student activity, and performance to provide personalized "nudges" to encourage students and help students have better study skills (e.g. deciding when to study for the next test). Initial studies found that students who used this system performed better than students who did not. There also seemed to be positive feedback from students on the system, and good indications that the system is actually used. How the system works is not explained in detail (algorithms, data analysis, specific types of data collected), but this does show how different types of data can work together to help learners.
Hershkovitz, a., de
Baker, R. S. J., Gobert, J., Wixon, M., & Pedro, M. S. (2013). Discovery
with models: A case study on carelessness in computer-based science inquiry. American
Behavioral Scientist, 57(10), 1480–1499.
doi:10.1177/0002764213479365
This article discusses a new method in educational data mining research called discovery with models. The article outlines this method and presents a case study where the method is used. Discovery with models involves building a model either through machine learning or knowledge engineering techniques, ensuring that the model is valid and generalizable, and using that model to discover patterns in new types of data. In the case study, the authors built a model of student carelessness in completing learning activities (a virtual lab) using Bayesian statistics and decision trees. They then applied the model to student responses on surveys of motivation and goal-orientation using correlational and cluster analysis. They found that students who exhibited careless behaviors tended to feel confident about their learning abilities.
Hershkovitz, A.,
& Nachmias, R. (2009). Developing a log-based motivation measuring tool. In
Proceedings of 1st International Conference on Educational Data Mining
(pp. 99–106). Montreal, Canada.
Kinnebrew, J. S.,
Biswas, G., Sulcer, B., & Sta, B. (2008). Modeling and measuring
self-regulated learning in teachable agent environments. Journal of
E-Learning and Knowledge Society, 7(2), 19–35.
The researchers report on an analysis of student behaviors, performance, and learning with an intelligent tutoring system under different instructional approaches. The instructional approaches included an intelligent coaching system, and two learning by teaching approaches, one of which used self-regulated learning suggestions. Types of behaviors with the system were identified (e.g. applied reading, uniformed editing, checking, probing). A hidden Markov model was used to look for patterns in the sequence of behaviors to see if certain behaviors were more prevalent in one instructional condition than in others. Hidden Markov models are used to develop pattern recognition. It was found that students learning actions were more effective and led to greater learning results that were statistically significant than the actions of students in the other interventions (p<.1, effect size d=.72).
This article is a review of feature selection approaches--the process of deciding which variables actually matter for what one is trying to do with the data. Feature selection is useful/necessary when one is dealing with hundreds and thousands of different variables in a data set. The process of feature selection is described. A framework is presented to categorize different algorithms for feature selection. The authors discuss ways to best decide which algorithms are best for which purposes. The authors conclude with real-world applications where feature selection is used, and challenges to address to improve feature selection methods.
Long, P., & Siemens,
G. (2011). Penetrating the fog: Analytics in learning and education. Educause Review, 46(5), 31–40.
Describes definitions and distinctions in the field of
learning analytics, potential uses, and challenges. Distinguishes between learning
analytics and academic analytics. Presents an argument on the importance of
learning analytics. Technology is allowing us greater capacity to capture data
about student learning behavior. However there is a need to create new
approaches for making sense of the data, and for looking beyond behavioral
data.
Lynch-Holmes, K.,
& Mason, T. (2012, August). Building a purpose network to increase student
engagement and retention. Educause Review.
This Educause Review article reports on a system at Fort Hays State University, partnered with ConnectEDU, that helps alert teachers, parents, resident assistants or student leaders when a student is at risk for dropping out of college. The data that is used to identify an at risk student is not mentioned, but it seems to come from many sources within the university.
Macfadyen, L. P.,
& Dawson, S. (2010). Mining LMS data to develop an “early warning system”
for educators: A proof of concept. Computers & Education, 54(2),
588–599. doi:10.1016/j.compedu.2009.09.008
In this study, Macfadyen & Dawson (2010) investigate log data obtained from 5 sections of an online biology course offered on the BlackBoard Vista learning management system to determine which engagement factors best predict academic success. Thirteen variables were found to have a statistically significant correlation with students’ final grades (p < .01), including total number of discussion messages posted, total time online, and the number of web links viewed. A linear multiple regression analysis was used to develop a predictive model of behaviors that best predicted student final grades. Analysis found three highly correlated variables of significance (p=.00) to make that model: Total number of discussion messages posted, total number of assessments finished, and number of mail messages sent. A binary logistic regression was then used to categorize students at risk for failure using the validated model. The model was found to categorize accurately 73.3% of the time. The results of this study do not seem to say anything surprising about learning: those who do the work tend to succeed in class. The developed model also relies on data obtained over the course of an entire semester, which may not be useful for identifying at-risk students early on in a course. The study does show, however, that log data can be used to productively describe online learner behavior.
Mayer-Schönberger, V., Cukier, K. (2013). Big Data: A Revolution That Will Transform How We Live, Work, and Think. New York, NY: Houghton Mifflin Harcourt. Kindle Edition.
This book outlines the fundamental philosophy behind big data. The amount of data, the importance of correlations, digitizing data and creating more data about things are all important to a new paradigm about how to look at the world and what we can learn from it. The book includes many examples to illustrate ideas presented.
Mauger, A. J., Schwartz, C. M., & Grieco, S. J. (2012, July). Efficiencies, learning outcomes bolstered by analytics, data-informed decision making. Educause Review.
This book outlines the fundamental philosophy behind big data. The amount of data, the importance of correlations, digitizing data and creating more data about things are all important to a new paradigm about how to look at the world and what we can learn from it. The book includes many examples to illustrate ideas presented.
Mauger, A. J., Schwartz, C. M., & Grieco, S. J. (2012, July). Efficiencies, learning outcomes bolstered by analytics, data-informed decision making. Educause Review.
Merceron, A., &
Yacef, K. (2008). Interestingness measures for association rules in educational
data. In The 1st International Conference on Educational Data Mining
(pp. 57–66). Montreal, Canada.
Morris, L. V.,
Finnegan, C., & Wu, S.-S. (2005). Tracking student behavior, persistence,
and achievement in online courses. The Internet and Higher Education, 8(3),
221–231. doi:10.1016/j.iheduc.2005.06.009
Perera, D., Kay, J.,
Koprinska, I., Yacef, K., & Zaiane, O. R. (2009). Clustering and sequential
pattern mining of online collaborative learning data. IEEE Transactions on
Knowledge and Data Engineering, 21(6), 759–772.
doi:10.1109/TKDE.2008.138
Provost, F. & Fawcett, T. (2013). Data Science for Business: What you need to know about data mining and data-analytic thinking. Sebastopol, CA: O'Reilly Media. Kindle Edition.
This book describes the fundamental data science methods for analyzing data, including classification, regression, clustering, Bayesian, and association methods and statistics. The book also outlines a general process for solving data problems in business. The book includes some of the mathematical formulas for completing these tasks, but tries to take the least-technical description of methods as possible. Many examples are given to illustrate how methods are used.
Rau, M. A., & Scheines, R. (2012). Searching for variables and models to investigate mediators of learning from multiple representations. In Proceedings of the 5th International Conference on Educational Data Mining (pp. 110–117).
This book describes the fundamental data science methods for analyzing data, including classification, regression, clustering, Bayesian, and association methods and statistics. The book also outlines a general process for solving data problems in business. The book includes some of the mathematical formulas for completing these tasks, but tries to take the least-technical description of methods as possible. Many examples are given to illustrate how methods are used.
Rau, M. A., & Scheines, R. (2012). Searching for variables and models to investigate mediators of learning from multiple representations. In Proceedings of the 5th International Conference on Educational Data Mining (pp. 110–117).
Romero, C., &
Ventura, S. (2010). Educational data mining: A review of the state of the art. IEEE Transactions on Systems, Man, and
Cybernetics-Part C: Applications and Reviews, 40(6), 601–618.
This provides a very useful and detailed library of sources
to turn to on what has been done to improve learning through educational data
mining. Also reviews definitions, trends in publications and conferences,
applications, and future directions. Main applications of educational data
mining include analysis and visualization of data, providing feedback for
supporting instructors, recommendations for students, predicting student
performance, student modeling, detecting undesirable student behaviors,
grouping students, Social Network Analysis, developing concept maps,
constructing courseware, and planning and scheduling.
Romero, C., Ventura, S., & Garcia, E. Data mining in course management systems: Moodle case study and tutorial. Computers and Education, 51(1), 368-384.
This article reviews the purpose and process of doing data analytics in education (collecting, preprocessing, mining, interpreting, and applying). It discusses specific applications in doing data mining with Moodle, providing examples of data mining decisions and tools.
San Pedro, M. O. Z.,
Baker, R. S. J., Gowda, S. M., & Heffernan, N. T. (2013). Towards an
understanding of affect and knowledge from student interaction with an
intelligent tutoring system. In Proceedings of the 16th International
Conference on Artificial Intelligence and Education.
“Csikszentmihalyi’s Flow theory states that a balance between challenge and skill leads to high engagement, overwhelming challenge leads to anxiety or frustration, and insufficient challenge leads to boredom. In this paper, we test this theory within the context of student interaction with an intelligent tutoring system. Automated detectors of student affect and knowledge were developed, validated, and applied to a large data set. The results did not match Flow theory: boredom was more common for poorly-known material, and frustration was common both for very difficult material and very easy material. These results suggest that design for optimal engagement within online learning may require further study of the factors leading students to become bored on difficult material, and frustrated on very well-known material.”
Siemens, G. (2013).
Learning analytics: The emergence of a discipline. American Behavioral
Scientist, 57(10), 1380–1400. doi:10.1177/0002764213498851
Smith, V. C., Ph, D.,
& Lange, A. (2012). Predictive modeling to forecase student outcomes and
drive effective interventions in online community college courses. Journal
of Asynchronous Learning Networks, 16(3), 51–61.
Thompson, K., Kennedy-Clark, S., Markauskaite, L., & Southavilay, V. (2014). Discovering processes and patterns of learning in collaborative learning environments using multi-modal discourse analysis. Research and Practice in Technology Enhanced Learning, 9(2), 215–240.
This study used first order Markov models and heuristic mining algorithm to discover patterns of behavior in a computer-supported collaborative learning activity. Student discourse was recorded and coded using the Decision-Function Coding Scheme and matched with activity with the system recorded in the log data. The process mining algorithms were then used to see the relationships between certain decision-making and group work actions between structured and unstructured groups and groups who successfully completed the assignment and those that did not. The researchers found that group orientation, agreement, and actions taken to implement a decision can be important indicators of successful group work.
Whitmer, J., Fernandes, K., & Allen, W. R. (2012, August). Analytics in progress: Technology use, student characteristics, and student achievement. Educause Review. Retrieved from http://www.educause.edu/ero/article/analytics-progress-technology-use-student-characteristics-and-student-achievement
This study used first order Markov models and heuristic mining algorithm to discover patterns of behavior in a computer-supported collaborative learning activity. Student discourse was recorded and coded using the Decision-Function Coding Scheme and matched with activity with the system recorded in the log data. The process mining algorithms were then used to see the relationships between certain decision-making and group work actions between structured and unstructured groups and groups who successfully completed the assignment and those that did not. The researchers found that group orientation, agreement, and actions taken to implement a decision can be important indicators of successful group work.
Whitmer, J., Fernandes, K., & Allen, W. R. (2012, August). Analytics in progress: Technology use, student characteristics, and student achievement. Educause Review. Retrieved from http://www.educause.edu/ero/article/analytics-progress-technology-use-student-characteristics-and-student-achievement
Wolff, A., &
Zdrahal, Z. (2012, July). Improving retention by identifying and supporting
“at-risk” students. Educause Review.
This article is a review of several studies done by the authors in detecting the emotion of students using e-learning systems, and the results of efforts made by the system to intervene based on certain detected affective states. Of particular interest are methods to recognize or measure a student’s emotional response to learning. The authors conducted a number of studies of student emotion using human observation, sensors, or machine learning.
Human observers tracked facial, physical, and verbal expressions as well as on-task and off-task behavior while students used a tutoring software in a computer lab. Expressions were labeled using four categories: positive or negative valence (nature of emotion) and high or low arousal (amount of physical activity in relation to the emotion). Expressions were also labeled as desirable or undesirable based on on-task/off-task behavior. The combination of labels were used to define different emotions. For example, positive valence and high arousal was defined as being joyful or excited, while negative valence and low arousal was defined as tired or bored. The researchers found statistically significant correlations with certain coded emotions and pre-test scores and learning orientation surveys (N = 34, p < .01). On-task behavior correlated with post-test math scores: (N=34, R=.640, p<.018).
In other studies, the researchers used sensors to detect student emotion. Sensors included facial expression recognition using video analysis, posture analysis using seat pressure systems, hand pressure on mouse, and skin conductance systems. The researchers found a mix of significant results from using the sensors. Using the measurements from all the sensors was best in detecting frustration, which combination had an accuracy of predicting frustration 89% of the time; however, looking at only measurements from the mouse pressure system was best for detecting interest, which had a detection accuracy of 72.67%.
Finally, the researchers used machine learning to detect student emotion. Learner behavior as recorded in log data (i.e. number of pages visited, time spent on a page) and results of a motivation survey were analyzed using Bayesian networks to discover meaningful relationships. The resulting model was accurate 80-90% of the time in identifying a student’s emotional response as revealed in the motivation survey based on values of recorded log data.
No comments:
Post a Comment