Classification

Much of my focus this week has been on methods of classification. The methods I have reviewed so far are based in regression, including step regression, logistic regression, and decision trees.

First, a disclaimer: I am a stats beginner! A lot of this stuff is going over my head. I'm trying to get the gist of these different statistical analyses by turning to other resources in addition to the EDM MOOC I am following. I feel like I am getting a decent understanding of things, but I still have questions about particulars, and I am sure I've missed something here and there. Take my explanations of this week's analyses below with a grain of salt!

Classification is good when you want to sort your data into groups that share patterns. For example, as you study the data, you notice that those who succeed in class have shared data patterns (similar number of discussion posts, similar amount of time spent on assignments, etc.), while those who tend to drop out have shared data patterns. You could set up a classification algorithm made from a previous data set to sort data in a new data set into the two groups (those who succeed and those who tend to drop out). Those with X number of page views would go into this group while those with less that X would go in the other. You could then compare your algorithm's accuracy rate to see if it is good enough to use on new data sets, or if it needs to be adjusted (like changing the number of page views required to be classified as a successful student).

Remember kindergarten, where you are given a shape and you have to find all the other shapes that match it? When building an algorithm, it's like you are taking a model shape that data for a specific category or group should look like. Maybe your successful group's data should look like a star, while those likely to dropout looks like a moon. When you use statistical analysis, it's like you are taking the data shape a student makes and comparing it to your two group's models, asking "Is this student's data shape close to a star or closer to a moon?"

Step regression and logistic regression are two types of analysis that determine whether one student's data matches better to one group's predetermined value or another group's. Both are binary types of analysis (you are dealing with two groups). With step regression (not step-wise regression!), you set a cut-off value that determines whether a data point will be classified as a 0 or 1 (0 means it belongs in one group and 1 means it belongs in the other). The data point is made up of all the variables you are analyzing. You assign weights to each variable group. Maybe you want to give more precedence to time on task for a week rather than how many emails the student sent in a week. You give a higher weight value to time on task, and a lesser value to emails sent. In the end, you would build an algorithm that looked something like this: Y=0.2a + 0.5b + 0.3c - 0.1d, where Y is the combined value of all the variables in a data point, a b c d are values of your different variables, and 0.2, etc. are your assigned weight values. After applying the algorithm to the data point, you would take the score of Y and compare it to the cut-off value you set that determines whether a data point will be a 0 or 1. If it is greater than the set value, it would go in one group. If it was less, it would go in the other.

With logistic regression, you determine the frequency or odds of a specific value of a dependent variable. Rather than a simple multiplication and addition problem like step regression, logic regression does something more complex: p(m)= 1/(1+e^-m). I don't even know what the function means, but there it is. By applying the function to a dependent variable value, you'd be able to compute whether the variable belongs to one group or another (even a trip to other resources, like Wikipedia, couldn't explain this better for me).

One problem with either of these approaches is that they do not take into account interaction effects. Baker gives this example to explain the problem: You have an algorithm made up of a value of the effectiveness of an educational software and a value of how on task a student is. Bad educational software is bad and being off task isn't any good, but maybe being off-task while you are supposed to be using bad educational software is a good thing if you are doing something more productive. The combination of certain values of your variables could lead to a need for more than two groups in a case like Baker describes. This is where decision trees can be handy.

Decision trees is another classification algorithm that allows you to apply different functions to your data set based on the value of your variables, allowing you to obtain a variety of outcomes. For instance, (another Baker example) you are trying to predict if a student will get quiz answers right or wrong. If a student taking the quiz doesn't have a lot of knowledge on a topic, but they spend little time answering the question, they tend to get the answer right. This is one group or outcome. If they don't have a lot of knowledge on the quiz topic and spend a lot of time trying to answer, they tend to get the answer wrong. On the flip side, if the student has a lot of knowledge on the quiz topic, maybe time spend isn't an appropriate variable to look at, but rather how many actions is taken to complete an answer. Perhaps a certain range of number of actions tends to lead to correct answers while a different range of task actions tends to lead to wrong answers. Using decision trees, you can have different paths that lead to your categories based on the values of specific variables.

In regards to the project I am working on dealing with data analysis, I can see methods for classification as being very important to what I am doing. I am gathering a number of different types of data that can be used to build algorithms that determine whether a student is engaged or disengaged. I don't really see interaction effects having much bearing with the data I am collecting, but it is hard to say until I see the data. I would probably first start with step regression or logic regression (if I figure out how to use it!). If the different combinations of variables tends to signify greater complexity, I might try decision trees.

How are Big Data & Analytics Being Used in Education?

One of the questions I've had regarding Learning Analytics and Educational Data Mining is, how are they being used in education?  Or to put it another way, what is the "job" that is trying to get done by appling LA & EDM?  This question seeks to understand the value that these methodologies, processes, and networks add to education.  This isn't answering the how question, but instead tries to understand what problems, or opportunities, or applications form the context of LA & EDM.
 Siemens and Long (2011) provide a wide breadth of value-adding activity.  I have transformed the narrative description into a numerical list simply to discuss further detail about select activities. 

1. Improve administrative decision-making and resource allocation
2. Flag at-risk students and provide intervention
3. Create shared understanding of the institution's successes and challenges
4. Transform college/university system
5. Transform academic models and pedagogical approaches
6. Make sense of complex topics
7. Conduct what-if scenarios and experiments to explore elements in a complex discipline (i.e., retaining students, reducing costs)
8. Provide up-to-date information to handle immediate challenges
9. Determine value generated by faculty including hard (research,patents) and soft (reputation, profile, teaching quality)
10. Inform learners about their own learning habits, give recommendations for improvement

Let me regroup this list into 3 buckets:

A) University Administrators (#1, #3, #4, #7, #9)
B) Faculty (#2, #5, #9)
C) Learners (#2, #6, #10)

 The A bucket seems to line up well with what is called Academic Analytics.  This is discussed more in Curtis' post here.

Bucket B deals with #2 (flagging at-risk students), #5 (transforming academic models and pedagogical approaches), and #9 (determining value generated by faculty).  The example used in the article regarding flagging at-risk students uses data such as hours viewing content and the number of pages viewed as the kind of data used. 

Bucket C identifies opportunities for LA & EDM that benefit the learner including #2 (identifying at-risk students and providing intervention), #6 (making sense of complex topics), and #10 (informing learners about their learning habits). 

Thus, LA & EDM seek to support the full range of participating constituents in higher education. 

Methods of Statistical Analysis

This week, I turned to the MOOC on Educational Data Mining put on by Baker at Coursera. Most of that course (like 98% of it) is dedicated to reviewing methods of statistical analysis that have been used in Educational Data Mining research. John and I started a page dedicated to reviewing these methods. The page is called Methodology. You can access it in the navigation box to the right. It will be a page in progress as I synthesize there what I am learning from the Coursera MOOC.

In the Methodology page we built, you will see major types of methods, broken down by sub-methods. Our purpose isn't to describe how to use the method, but rather to help someone get an idea of which type of method would be most useful for the research questions and data they are trying to tackle. We will describe what the method is, research questions the method could be used to answer, and how the results could be applied to improving learning. Anyways, keep tabs on its progress!

Similarities & Differences of LAK, EDM using Ontology Learning Tools

I would like to summarize a recent analysis by Zouaq, Joksimović, & Gasević (2013) who used ontology learning tools to identify the similarities and differences between Learning Analytics (LAK) and Educational Data Mining (EDM).  I highly recommend the article as it describes, in detail, the conceptual analysis methodology.  Here is a summary of the data used and the results found.

The data included the following (as described by Taibi & Dietze, 2013):

Results

Similarities:
"... both the LAK and EDM conferences have students, data and models as shared concepts."


Differences:
"LAK papers also focus on teachers/instructors, informal learning, and social, networked, and group learning."
"EDM papers focus on (data mining) methods and approaches, intelligent tutoring systems, features (extraction), and various types of parameters."

"LAK papers are more focused on teachers in order to empower them with learning analytics and to help them guide students. Moreover, there is an emphasis on (promoting) reflection of both students and instructors. Various aspects of social learning such as role playing and impact of communities appear to be highly popular topics in the LAK papers."

"EDM papers are much more focused on intelligent tutoring systems, accuracy of different types of (predictive) models, and revealing unexpected patterns."


Similarities:
"focus on data is shared by both the LAK and EDM communities"

Differences:
"LAK also seems to be focused on data collected by and for instructors, not only for students. This probably indicates a trend that the LAK community has so far acknowledged the role of instructors in the learning process and aimed at supporting them as much as learners."

"The EDM community has however focused more on measuring and predicting specific types of skills.  This is consistent with their focus on intelligent tutoring systems in which automated assessment of learners’ skills is of paramount importance." 

Finally:
"[L]earning analytics is an integral part of teaching profession, is an important step for teachers of tomorrow and learners, and offers a new approach. This figure reveals also the nature of learning analytics to promote qualitative understanding of context of information. Learning analytics is also (strongly) related to discourse analytics, which seems to be consistent with the strong emphasis of learning analytics on social learning and which is further confirmed by extracted relationships of discourse learning analytics with sense-making argumentation and social, all of which are types of skills recognized as important for the modern society."

What I found uniquely interesting in this summary is the distinction between LAK's focus on a teacher perspective both in terms of data and in terms of purpose while the EDM focus is primarily on learner skill.  I think both of these perspectives are useful in the classroom and in pushing the boundaries of our understanding in teaching and and learning.  I'm not ready yet to make the decision that one is more important that the other or one is more useful than the other.  It may be that each is appropriate for a different audience or a different set of stakeholders or scale.  LAK may be a better fit for instructors or others who interact frequently with learners.  EDM's analysis, conclusions and recommendations may find a better fit with the needs and interests of instructional designers, institutional entities, and others with a role outside of direct instructional delivery. 


References

 

Taibi, D., & Dietze, S. (2013). Fostering analytics on learning analytics research: the LAK dataset. Retrieved from http://ceur-ws.org/Vol-974/lakdatachallenge2013_preface.pdf

Zouaq, A., Joksimović, S., & Gasević, D. (2013). Ontology Learning to Analyze Research Trends in Learning Analytics Publications. Retrieved from http://ceur-ws.org/Vol-974/lakdatachallenge2013_08.pdf

 

Educational Data Mining, Learning Analytics... and Academic Analytics!

This week starts off with a discussion about general similarities and differences in the research areas interested in large data sets, computational mathematics, and education.

For a while, I thought there were two research areas: Educational Data Mining and Learning Analytics. But, as I read Long and Siemens (2011), I saw there was another research area called Academic Analytics. In the Learning Analytics (LA) MOOC put out by the Society for Learning Analytics Research (SOLAR), Siemens explains that Academic Analytics is not as well known as the other two, but still has its niche.

Anyways, here's a breakdown of some of the comparisons and contrasts in these different, but related, research areas:

• Similarities
• Has a focus on improving education
• Makes interpretations of large data sets to enable educational planning, strategy, and decision making
• Has interest in developing methods of data analysis to enable these interpretations
• Differences (and perhaps more similarities)
• Learning Analytics
• Focus "is exclusively on the learning process" (Long & Siemens, 2011, p. 34).
• Discourse analysis, social network analysis, content analysis, personalization and adaptation, prediction and intervention
• Educational Data Mining
• Especially interested in how to collect and analyze data
• Data visualization, recommendations, predictions, student modeling, grouping, social network analysis, concept mapping, planning and scheduling (Romero & Ventura, 2010).
• Academic Analytics
• "The application of business intelligence in education" (Long & Siemens, 2011, p. 34).
• Has a wider focus, especially in institutional level data analysis (enrollment, grades, GPA, college entrance exam scores, etc).
• Recommendations, predictions, scheduling, planning (Campbell, Deblois, & Oblinger, 2007).
• Doesn't seem to have its own journal or conference as LA and EDM do.

Reading introductory articles to these areas of research really makes me think that they could all be lumped together. I think Learning Analytics tries to separate itself by focusing solely on learning, whereas educational data mining and learning analytics include work on things like predicting which prospective students will most likely be admitted, or how many textbooks to order. As of yet, I still need to better understand the differences between educational data mining and academic analytics, as these two seem to be the most similar.

Which area does my research fit? Personally, I could see it belonging to any of them. I've seen examples of research similar to my project being presented in conferences centered on Learning Analytics and Educational Data Mining. I believe the focus of my research is on learning, and personalizing and adapting learning pathways based on user behavior and learner characteristics, thus pushing me more towards the Learning Analytics area. I don't think I could go wrong, however, in publishing and presenting in any of these areas.

References

Campbell, B. J. P., Deblois, P. B., & Oblinger, D. G. (2007). Academic analytics: A new tool for a new era. Educause Review, (August 2007), 41–57.

Long, P., & Siemens, G. (2011). Penetrating the Fog: Analytics in Learning and Education. Educause Review, 46(5), 31–40.

Romero, C., & Ventura, S. (2010). Educational data mining: A review of the state of the art. IEEE Transactions on Systems, Man, and Cybernetics-Part C: Applications and Reviews, 40(6), 601–618.