Knowledge Inference or Latent Knowledge Estimation

Ryan Baker, in week four of the Big Data in Education Course, provides various examples of methodologies all focused on "measuring what a student knows at a specific time," or knowledge inference.  This is "often operationalized as measuring what relevant knowledge components a student knows at a specific time."  He then describes what a knowledge component is.  It is, "anything a student can know, that's meaningful to the current learning situation, which might include skills, knowledge of facts, knowledge of concepts, knowledge of principles, knowledge of schemas."  In short, he says, "anything a student can know, can be a knowledge component."

Baker suggests at least three reasons why it is useful to measure what a student knows:

• Because education's primary goal is enhancing student knowledge, by measuring it, "you know whether you're making it better."
• In addition, if you measure what a student knows, you can inform instructors and other stakeholders about it
• Finally, "if you can measure it, you can make automated pedagogical decisions."

He adds the caution measuring  what a student knows is different than measuring performance.  Specifically he notes, "inferring if a student's performance right now is associated with successfully demonstrating a skill, is not the same as knowing whether the student has this latent skill."  A student could guess and perform well.  Or a student could slip up when they actual know the skill.  Baker suggests one difference is not just looking at performance "at one specific moment," but looking at performance over time to observe patterns in performance.

Observing patterns in performance is at the center of the research I am conducting in the context of online spreadsheet learning.  An individual's performance over time does show signs of misconceptions of knowledge.  By identifying the underlying misconceptions, in terms of knowledge components, intervention efforts can be focused to specific knowledge components instead of the traditional response - please try again.

One methodology that seems especially relevant to this research is Learning Factors Analysis.  I will write a followup post about this particular methodology.

Datafication and Digitization

I found another important thought from Big Data: A Revolution - the concept of datafication and digitization. Datafication is making information about the world into a reusable, reviewable form. In old days, it was book keeping, logs, and journals. Nowadays, it includes logs of user behavior with systems stored in electronic databases. Digitization is taking hard copy forms of information and making them electronic. This would include Google's efforts to scan the world's books and store them in electronic format. This effort becomes datafication when Google OCRs the books so that people can search for words and patterns between words.

We are digitizing learning. We put the syllabus and needed files online so we no longer have to worry about printing costs. We ask learners to turn things in online, to watch videos made available online, or participate in an online discussion. We datafy this when we have information about these objects and interactions--where things are stored, how it was used, how many times it was used, who used it.

The power of Big Data is when we have lots of data to analyze. Being able to do Big Data in education, however, is limited. We have digitized a lot of learning, but have barely scratched the surface of what data is currently available and what data could yet be made. It is common to be able to access page views, and loggin counts, and number of messages sent from the LMS, but more could be done. Length of time learning, mouse tracking, and capturing information about on-task and off-task behavior could be harnessed. Woolf et al. (2009) describe efforts to capture learner's emotions using mouse pressure, seat pressure, facial recognition, and other biosensor systems. The challenge of our day is to continue to find ways to datafy the learning experience.

Learning Management Systems are trying to datafy more and more of the interactions of users and make that data available to others; however, not all of learning is taking place in an LMS. School administrative systems contain useful learner data, such as GPA, course completion statuses, exam scores, and final grades for courses. Tin Can is a service that tries to collect data from LMSs, website visits, and other online interactions and makes all that data available in useful ways. Still, a challenge is that much of learning is not taking place online. We can not get the true value of Big Data analysis on learning until we have lots of data about the full range of the learning experience. Finding ways to capture data about face-to-face interactions is key.

A final challenge, of course, is student privacy rights. Who should have access to student data? How should it be ethically used? How should it be stored. What can students do about access to their data? As more and more data becomes available, and using that data becomes more widespread, these privacy questions will demand more attention.

  

Woolf, B., Burleson, W., Arroyo, I., Dragon, T., Cooper, D., & Picard, R. (2009). Affect-aware tutors: Recognising and responding to student affect. International Journal of Learning Technology, 4(3/4), 129–164.

Case Study Reviews (2 of 2)

This is the second in a two-part series reviewing case studies of learning analytics.  The examples in this and the previous post come directly from the LAK 2013 online course by George Seimens.  The purpose of these posts is to look more closely at the kind of data being used and the ways these data are being used.

Building a Purpose Network to Increase Student Engagement and Retention
Purpose: Improve Retention
How: Increasing student engagement, parent engagement, tracking success measures
Data: (from picture) late to class, missed assignments, absence, illness/fatigue, substance abuse, other

This system provides dashboards and comparisons grouped by course, instructor, year in school. Also, this system tracks action plans and recurrence of issues.  I can't help but think that most intervention-focused systems, such as this one, implement intervention outside of the instructor.  This may not be a bad thing, but it limits the effectiveness of the intervention to generalizable academic skills or responsibility or social scaffolds, which are different from the academic experience the student has with the learning content. 



Efficiencies, Learning Outcomes Bolstered by Analytics, Data-Informed Decision Making
Purpose: Improve scheduling (room resources), success rate for Math 010, course offerings,
Data: Student enrollment (by course), success rate for Math 010, career interest data, transferability (to other schools)

This article is a list of 3 or 4 efforts of applying data to making school-level decisions--how to allocate campus resources more effectively, how to improve the success rate of Math 010, how to align course offerings to meet students' transfer objectives.


Improving Retention by Identifying and Supporting "At-Risk" Students
Purpose: Identify, support "at-risk" students
Data: Virtual Learning Environment(VLE) for online course provides interesting data from students

This kind of data is relatively new compared to a traditional classroom.  This data affords new measures of engagement for the student.  Interestingly, the recommendation and observation is that comparing students to a single engagement profile is not as effective at identifying risk because one student may have a high engagement profile in the VLE, but may not be learning as well as other students who measure low on VLE activity, but score high in the course.  Thus, student VLE activity should be compared to previous VLE activity for that student.  But the article also notes that different modules in the course will also affect the student's VLE activity.  Overall, the application introduces new data that is harder to obtain in traditional face-to-face classes, and provides initial recommendations for how to use that data to identify at-risk students.

Dethroning Causation

I'm reading a book called Big Data: A Revolution That Will Transform How We Live, Work, and Think, by Mayer-Shonberger and Cukier. I just got to the chapter, "Correlation," which discusses how causation is being dethroned by correlation. I remember my introduction to research and statistics 1 classes, where causation was trumpeted as the gold-standard of research. The aim of traditional research has been to take data from a small group and be able to make claims about a larger group.

Big data shakes things up because we can now get lots of data from the larger group. The book reviews an example about Google, who was able to predict the spread of the flu based on the search terms people were putting into Google. The search terms didn't cause the spread, but the correlation was extremely valuable to health organizations and governments wanting to target interventions and to track or stop the spread.

Big data in education means we can look for meaningful relationships among all sorts of data being produced in electronic learning contexts. Much of the data we can get from the electronic world would be extremely unreasonable to track by people in a traditional face-to-face classroom. Machines can capture, store, process, and analyze information much more efficiently than people can. We can look at big groups, like all student data across the university to a single user's data from on learning activity. And generalizing doesn't seem so important any more. The machine can learn and revise predicting and categorizing models as it gathers more data.

I have thought more and more about what value theory or traditional empirical research has any more. If you just need to let the data speak to find meaningful relationships and patterns, what does the "why" matter as long as the "what" helps us get things done? From what I have learned from Dr. Gibbons Explore, Explain, Design framework (Gibbons & Bunderson 2005), design work, which is focused more on achieving outcomes rather than explaining why outcomes were achieved, is becoming more valuable in the world of big data than traditional explain work done by science. This very notion was argued by Chris Anderson in Wired: http://www.wired.com/science/discoveries/magazine/16-07/pb_theory

So, why does theory and explain research matter? As Mayer-Shonberger and Cukier argue, we use theories and models to build the algorithms that analyze data, and we use theories and models to make sense of the relationships discovered. "After all," they said, "Google used search terms as a proxy for the flu, not the length of people's hair." Theory and models increasingly matter to enable us to make sense of all the data the digital world makes available to us.

Case Study Reviews (1 of 2)

Recently, I read some examples of cases where Learning Analytics was being applied.  Below is a short description of the kinds of data in each of the cases and what was trying to be explained.  It is interesting to me that these four do not center on data that a teacher would use to evaluate instruction or learning.  Most of this data is centered on institutional statistics rather than more traditional teacher-based data (except for grades), which include evaluating a learner on a particular skill or knowledge component.  Another observation is that there is relatively little NEW data being captured.  In most cases, there are new connections between existing data that are examined, evaluated, and regressed.  But essentially no new data that is being captured here.  I suppose one reason for this is it is easier to use available data than to have to create a new data collection process and infrastructure at the institution level.  Maybe instructors have the advantage of measuring or observing new data points that could improve the diagnostic/intervening capacity of an instructor or instructional technology.

Cases and Examples of learning Analytics:	Notes
SNAPP Overview	A pre, 2002-built system using social network analysis to identify, measure, track, and explain interactionas (IMHO, a proxy for learning)
Austin Peay State University: Degree Compass	Recommends courses a student should take based on the predicted grade they will receive in the course using course and course grade data from past students
Two Case Studies of Learner Analytics in the University System of Maryland	UMED - Using pre-enrollment (housing paid, tuition paid, enrolled/no data, admissions) data to address retention and graduation rates for at-risk student populations. BSU - fascinating collection of data, but implementation too early to tell how data is connected to intervention yet (lots of ideas, but little tried yet), technology integration is impressive (student services, financial aid, recruitment/admissions, registration/advising, academic services(testing services, learneng communities, tutorng), Curriculum & instruction( course roster, class sched, attendance, assignment scores, official grades)
Analytics in Progress: Technology Use, Student Characteristics, and Student Achievement	Using LMS data (use) and student characteristics to correlate with student achievement, student characteristics (gender, race/ethnicity, family income, previous academic achievement)

Classification

Much of my focus this week has been on methods of classification. The methods I have reviewed so far are based in regression, including step regression, logistic regression, and decision trees.

First, a disclaimer: I am a stats beginner! A lot of this stuff is going over my head. I'm trying to get the gist of these different statistical analyses by turning to other resources in addition to the EDM MOOC I am following. I feel like I am getting a decent understanding of things, but I still have questions about particulars, and I am sure I've missed something here and there. Take my explanations of this week's analyses below with a grain of salt!

Classification is good when you want to sort your data into groups that share patterns. For example, as you study the data, you notice that those who succeed in class have shared data patterns (similar number of discussion posts, similar amount of time spent on assignments, etc.), while those who tend to drop out have shared data patterns. You could set up a classification algorithm made from a previous data set to sort data in a new data set into the two groups (those who succeed and those who tend to drop out). Those with X number of page views would go into this group while those with less that X would go in the other. You could then compare your algorithm's accuracy rate to see if it is good enough to use on new data sets, or if it needs to be adjusted (like changing the number of page views required to be classified as a successful student).

Remember kindergarten, where you are given a shape and you have to find all the other shapes that match it? When building an algorithm, it's like you are taking a model shape that data for a specific category or group should look like. Maybe your successful group's data should look like a star, while those likely to dropout looks like a moon. When you use statistical analysis, it's like you are taking the data shape a student makes and comparing it to your two group's models, asking "Is this student's data shape close to a star or closer to a moon?"

Step regression and logistic regression are two types of analysis that determine whether one student's data matches better to one group's predetermined value or another group's. Both are binary types of analysis (you are dealing with two groups). With step regression (not step-wise regression!), you set a cut-off value that determines whether a data point will be classified as a 0 or 1 (0 means it belongs in one group and 1 means it belongs in the other). The data point is made up of all the variables you are analyzing. You assign weights to each variable group. Maybe you want to give more precedence to time on task for a week rather than how many emails the student sent in a week. You give a higher weight value to time on task, and a lesser value to emails sent. In the end, you would build an algorithm that looked something like this: Y=0.2a + 0.5b + 0.3c - 0.1d, where Y is the combined value of all the variables in a data point, a b c d are values of your different variables, and 0.2, etc. are your assigned weight values. After applying the algorithm to the data point, you would take the score of Y and compare it to the cut-off value you set that determines whether a data point will be a 0 or 1. If it is greater than the set value, it would go in one group. If it was less, it would go in the other.

With logistic regression, you determine the frequency or odds of a specific value of a dependent variable. Rather than a simple multiplication and addition problem like step regression, logic regression does something more complex: p(m)= 1/(1+e^-m). I don't even know what the function means, but there it is. By applying the function to a dependent variable value, you'd be able to compute whether the variable belongs to one group or another (even a trip to other resources, like Wikipedia, couldn't explain this better for me).

One problem with either of these approaches is that they do not take into account interaction effects. Baker gives this example to explain the problem: You have an algorithm made up of a value of the effectiveness of an educational software and a value of how on task a student is. Bad educational software is bad and being off task isn't any good, but maybe being off-task while you are supposed to be using bad educational software is a good thing if you are doing something more productive. The combination of certain values of your variables could lead to a need for more than two groups in a case like Baker describes. This is where decision trees can be handy.

Decision trees is another classification algorithm that allows you to apply different functions to your data set based on the value of your variables, allowing you to obtain a variety of outcomes. For instance, (another Baker example) you are trying to predict if a student will get quiz answers right or wrong. If a student taking the quiz doesn't have a lot of knowledge on a topic, but they spend little time answering the question, they tend to get the answer right. This is one group or outcome. If they don't have a lot of knowledge on the quiz topic and spend a lot of time trying to answer, they tend to get the answer wrong. On the flip side, if the student has a lot of knowledge on the quiz topic, maybe time spend isn't an appropriate variable to look at, but rather how many actions is taken to complete an answer. Perhaps a certain range of number of actions tends to lead to correct answers while a different range of task actions tends to lead to wrong answers. Using decision trees, you can have different paths that lead to your categories based on the values of specific variables.

In regards to the project I am working on dealing with data analysis, I can see methods for classification as being very important to what I am doing. I am gathering a number of different types of data that can be used to build algorithms that determine whether a student is engaged or disengaged. I don't really see interaction effects having much bearing with the data I am collecting, but it is hard to say until I see the data. I would probably first start with step regression or logic regression (if I figure out how to use it!). If the different combinations of variables tends to signify greater complexity, I might try decision trees.

How are Big Data & Analytics Being Used in Education?

One of the questions I've had regarding Learning Analytics and Educational Data Mining is, how are they being used in education?  Or to put it another way, what is the "job" that is trying to get done by appling LA & EDM?  This question seeks to understand the value that these methodologies, processes, and networks add to education.  This isn't answering the how question, but instead tries to understand what problems, or opportunities, or applications form the context of LA & EDM.
 Siemens and Long (2011) provide a wide breadth of value-adding activity.  I have transformed the narrative description into a numerical list simply to discuss further detail about select activities. 

1. Improve administrative decision-making and resource allocation
2. Flag at-risk students and provide intervention
3. Create shared understanding of the institution's successes and challenges
4. Transform college/university system
5. Transform academic models and pedagogical approaches
6. Make sense of complex topics
7. Conduct what-if scenarios and experiments to explore elements in a complex discipline (i.e., retaining students, reducing costs)
8. Provide up-to-date information to handle immediate challenges
9. Determine value generated by faculty including hard (research,patents) and soft (reputation, profile, teaching quality)
10. Inform learners about their own learning habits, give recommendations for improvement

Let me regroup this list into 3 buckets:

A) University Administrators (#1, #3, #4, #7, #9)
B) Faculty (#2, #5, #9)
C) Learners (#2, #6, #10)

 The A bucket seems to line up well with what is called Academic Analytics.  This is discussed more in Curtis' post here.

Bucket B deals with #2 (flagging at-risk students), #5 (transforming academic models and pedagogical approaches), and #9 (determining value generated by faculty).  The example used in the article regarding flagging at-risk students uses data such as hours viewing content and the number of pages viewed as the kind of data used. 

Bucket C identifies opportunities for LA & EDM that benefit the learner including #2 (identifying at-risk students and providing intervention), #6 (making sense of complex topics), and #10 (informing learners about their learning habits). 

Thus, LA & EDM seek to support the full range of participating constituents in higher education. 

Methods of Statistical Analysis

This week, I turned to the MOOC on Educational Data Mining put on by Baker at Coursera. Most of that course (like 98% of it) is dedicated to reviewing methods of statistical analysis that have been used in Educational Data Mining research. John and I started a page dedicated to reviewing these methods. The page is called Methodology. You can access it in the navigation box to the right. It will be a page in progress as I synthesize there what I am learning from the Coursera MOOC.

In the Methodology page we built, you will see major types of methods, broken down by sub-methods. Our purpose isn't to describe how to use the method, but rather to help someone get an idea of which type of method would be most useful for the research questions and data they are trying to tackle. We will describe what the method is, research questions the method could be used to answer, and how the results could be applied to improving learning. Anyways, keep tabs on its progress!

Similarities & Differences of LAK, EDM using Ontology Learning Tools

I would like to summarize a recent analysis by Zouaq, Joksimović, & Gasević (2013) who used ontology learning tools to identify the similarities and differences between Learning Analytics (LAK) and Educational Data Mining (EDM).  I highly recommend the article as it describes, in detail, the conceptual analysis methodology.  Here is a summary of the data used and the results found.

The data included the following (as described by Taibi & Dietze, 2013):

Results

Similarities:
"... both the LAK and EDM conferences have students, data and models as shared concepts."


Differences:
"LAK papers also focus on teachers/instructors, informal learning, and social, networked, and group learning."
"EDM papers focus on (data mining) methods and approaches, intelligent tutoring systems, features (extraction), and various types of parameters."

"LAK papers are more focused on teachers in order to empower them with learning analytics and to help them guide students. Moreover, there is an emphasis on (promoting) reflection of both students and instructors. Various aspects of social learning such as role playing and impact of communities appear to be highly popular topics in the LAK papers."

"EDM papers are much more focused on intelligent tutoring systems, accuracy of different types of (predictive) models, and revealing unexpected patterns."


Similarities:
"focus on data is shared by both the LAK and EDM communities"

Differences:
"LAK also seems to be focused on data collected by and for instructors, not only for students. This probably indicates a trend that the LAK community has so far acknowledged the role of instructors in the learning process and aimed at supporting them as much as learners."

"The EDM community has however focused more on measuring and predicting specific types of skills.  This is consistent with their focus on intelligent tutoring systems in which automated assessment of learners’ skills is of paramount importance." 

Finally:
"[L]earning analytics is an integral part of teaching profession, is an important step for teachers of tomorrow and learners, and offers a new approach. This figure reveals also the nature of learning analytics to promote qualitative understanding of context of information. Learning analytics is also (strongly) related to discourse analytics, which seems to be consistent with the strong emphasis of learning analytics on social learning and which is further confirmed by extracted relationships of discourse learning analytics with sense-making argumentation and social, all of which are types of skills recognized as important for the modern society."

What I found uniquely interesting in this summary is the distinction between LAK's focus on a teacher perspective both in terms of data and in terms of purpose while the EDM focus is primarily on learner skill.  I think both of these perspectives are useful in the classroom and in pushing the boundaries of our understanding in teaching and and learning.  I'm not ready yet to make the decision that one is more important that the other or one is more useful than the other.  It may be that each is appropriate for a different audience or a different set of stakeholders or scale.  LAK may be a better fit for instructors or others who interact frequently with learners.  EDM's analysis, conclusions and recommendations may find a better fit with the needs and interests of instructional designers, institutional entities, and others with a role outside of direct instructional delivery. 


References

 

Taibi, D., & Dietze, S. (2013). Fostering analytics on learning analytics research: the LAK dataset. Retrieved from http://ceur-ws.org/Vol-974/lakdatachallenge2013_preface.pdf

Zouaq, A., Joksimović, S., & Gasević, D. (2013). Ontology Learning to Analyze Research Trends in Learning Analytics Publications. Retrieved from http://ceur-ws.org/Vol-974/lakdatachallenge2013_08.pdf

 

Educational Data Mining, Learning Analytics... and Academic Analytics!

This week starts off with a discussion about general similarities and differences in the research areas interested in large data sets, computational mathematics, and education.

For a while, I thought there were two research areas: Educational Data Mining and Learning Analytics. But, as I read Long and Siemens (2011), I saw there was another research area called Academic Analytics. In the Learning Analytics (LA) MOOC put out by the Society for Learning Analytics Research (SOLAR), Siemens explains that Academic Analytics is not as well known as the other two, but still has its niche.

Anyways, here's a breakdown of some of the comparisons and contrasts in these different, but related, research areas:

• Similarities
• Has a focus on improving education
• Makes interpretations of large data sets to enable educational planning, strategy, and decision making
• Has interest in developing methods of data analysis to enable these interpretations
• Differences (and perhaps more similarities)
• Learning Analytics
• Focus "is exclusively on the learning process" (Long & Siemens, 2011, p. 34).
• Discourse analysis, social network analysis, content analysis, personalization and adaptation, prediction and intervention
• Educational Data Mining
• Especially interested in how to collect and analyze data
• Data visualization, recommendations, predictions, student modeling, grouping, social network analysis, concept mapping, planning and scheduling (Romero & Ventura, 2010).
• Academic Analytics
• "The application of business intelligence in education" (Long & Siemens, 2011, p. 34).
• Has a wider focus, especially in institutional level data analysis (enrollment, grades, GPA, college entrance exam scores, etc).
• Recommendations, predictions, scheduling, planning (Campbell, Deblois, & Oblinger, 2007).
• Doesn't seem to have its own journal or conference as LA and EDM do.

Reading introductory articles to these areas of research really makes me think that they could all be lumped together. I think Learning Analytics tries to separate itself by focusing solely on learning, whereas educational data mining and learning analytics include work on things like predicting which prospective students will most likely be admitted, or how many textbooks to order. As of yet, I still need to better understand the differences between educational data mining and academic analytics, as these two seem to be the most similar.

Which area does my research fit? Personally, I could see it belonging to any of them. I've seen examples of research similar to my project being presented in conferences centered on Learning Analytics and Educational Data Mining. I believe the focus of my research is on learning, and personalizing and adapting learning pathways based on user behavior and learner characteristics, thus pushing me more towards the Learning Analytics area. I don't think I could go wrong, however, in publishing and presenting in any of these areas.

References

Campbell, B. J. P., Deblois, P. B., & Oblinger, D. G. (2007). Academic analytics: A new tool for a new era. Educause Review, (August 2007), 41–57.

Long, P., & Siemens, G. (2011). Penetrating the Fog: Analytics in Learning and Education. Educause Review, 46(5), 31–40.

Romero, C., & Ventura, S. (2010). Educational data mining: A review of the state of the art. IEEE Transactions on Systems, Man, and Cybernetics-Part C: Applications and Reviews, 40(6), 601–618.