Lasang Tamang - Academia.edu (original) (raw)

Papers by Lasang Tamang

Zenodo (CERN European Organization for Nuclear Research), Jul 18, 2022

Domain modeling is a central component in education technologies as it represents the target doma... more Domain modeling is a central component in education technologies as it represents the target domain students are supposed to train on and eventually master. Automatically generating domain models can lead to substantial cost and scalability benefits. Automatically extracting key concepts or knowledge components from, for instance, textbooks can enable the development of automatic or semi-automatic processes for creating domain models. We explore in this work the use of transformer based pre-trained models for the task of keyphrase extraction. Specifically, we investigate and evaluate four different variants of BERT, a pre-trained transformer based architecture, that vary in terms of training data, training objective, or training strategy to extract knowledge components from textbooks for the domain of intro-toprogramming. We report results obtained using the following BERT-based models: BERT, CodeBERT, SciBERT and RoBERTa.

Lecture Notes in Computer Science, 2022

Intelligent Tutoring Systems

Lecture Notes in Computer Science, 2020

Reported here are the findings of a comparative study on the effects of using a Socratic Intellig... more Reported here are the findings of a comparative study on the effects of using a Socratic Intelligent Tutoring System for source code comprehension and learning computer programming. The result shows there are significant differences between the two groups where students who used Socratic Tutor ITS improved their knowledge by 45% in term of learning gain, developed a better understanding of concepts such as nested if-else and for loop, and improved their confidence level by 13%. Furthermore, the result of the Pearson product-moment correlation coefficient shows a positive correlation (r = 0.68) between feedback from the ITS and learning gain.

Proceedings of the ... International Florida Artificial Intelligence Research Society Conference, May 4, 2022

This paper presents a novel method to automatically assess self-explanations generated by student... more This paper presents a novel method to automatically assess self-explanations generated by students during code comprehension activities. The self-explanations are produced in the context of an online learning environment that asks students to freely explain Java code examples line-by-line. We explored a number of models consisting of textual features in conjunction with machine learning algorithms such as Support Vector Regression (SVR), Decision Trees (DT), and Random Forests (RF). Support Vector Regression (SVR) performed best having a correlation score with human judgments of 0.7088. The best model used a combination of features such as semantic measures obtained using a Sentence BERT pre-trained model and from previously developed semantic algorithms used in a stateof-the-art intelligent tutoring system.

We present in this paper a summary analysis of log files collected during an experiment designed ... more We present in this paper a summary analysis of log files collected during an experiment designed to test the hypothesis that prompting for free self-explanations leads to better comprehension of computer code examples. Indeed, the results indicate that students who were prompted to selfexplain while trying to understand code examples performed significantly better at predicting the correct output of the examples than students who were just prompted to read the code examples and predict their output.

This paper reports the findings of an empirical study on the effects and nature of self explanati... more This paper reports the findings of an empirical study on the effects and nature of self explanation during source code comprehension learning activities in the context of learning computer programming language Java. Our study shows that self explanation helps learning and there is a strong positive correlation between the volume of self-explanation students produce and how much they learn. Furthermore, selfexplanations as an instructional strategy has no discrepancy based on student's prior knowledge. We found that participants explain target code examples using a combination of language, code references, and mathematical expressions. This is not surprising given the nature of the target item, a computer program, but this indicates that automatically evaluating such self-explanations may require novel techniques compared to self-explanations of narrative or scientific texts.

Intelligent Tutoring Systems have been proven to generate excellent learning outcomes in many dom... more Intelligent Tutoring Systems have been proven to generate excellent learning outcomes in many domains such as physics, mathematics and computer programming. However, they have seen relatively little use in training and school classrooms due to the time and cost of designing and authoring. We developed an authoring tool for dialogue-based intelligent tutoring system for programming called Auto-author to reduce the time and cost. The tool allows teachers to create fully functional Socratic tutoring dialogue for learning programming from Java code. First, we conducted a controlled experiment on 45 introductory to programming students to assess auto-authored tutoring dialogues’ learning outcomes. The result shows that the auto-authored dialogues improved students’ programming knowledge by 43% in terms of learning gain. Secondly, we conducted a survey of auto-authored tutoring dialogues by introductory to programming course instructors to evaluate the dialogues’ quality. The result shows...

Computer Science (CS) education is critical in today’s world, and introductory programming course... more Computer Science (CS) education is critical in today’s world, and introductory programming courses are considered extremely difficult and frustrating, often considered a major stumbling block for students willing to pursue computer programming related careers. In this paper, we describe the design of Socratic Tutor, an Intelligent Tutoring System that can help novice programmers to better understand programming concepts. The system was inspired by the Socratic method of teaching in which the main goal is to ask a set of guiding questions about key concepts and major steps or segments of complete code examples. To evaluate the Socratic Tutor, we conducted a pilot study with 34 computer science students and the results are promising in terms of learning gains.

Negation plays a significant role in spoken and written natural languages. Negation is used in la... more Negation plays a significant role in spoken and written natural languages. Negation is used in language to deny something or to reverse the polarity or the sense of a statement. This paper presents a novel approach to automatically handling negation in tutorial dialogues using deep learning methods. In particular, we explored various Long Short Term Memory (LSTM) models to automatically detect negation focus, scope and cue in tutorial dialogues collected from experiments with actual students interacting with the state-of-the-art intelligent tutoring system, DeepTutor. The results obtained are promising.

We present a novel approach to intro-to-programming domain model discovery from textbooks using a... more We present a novel approach to intro-to-programming domain model discovery from textbooks using an over-generation and ranking strategy. We first extract candidate key phrases from each chapter in a Computer Science textbook focusing on intro-to-programming and then rank those concepts according to a number of metrics such as the standard tf-idf weight used in information retrieval and metrics produced by other text ranking algorithms. Specifically, we conduct our work in the context of developing an intelligent tutoring system for source code comprehension for which a specification of the key programming concepts is needed - the system monitors students' performance on those concepts and scaffolds their learning process until they show mastery of the concepts. Our experiments with programming concept instruction from Java textbooks indicate that the statistical methods such as KP Miner method are quite competitive compared to other more sophisticated methods. Automated discover...

The International FLAIRS Conference Proceedings

We present in this paper an automated method to assess the quality of Jupyter notebooks. The qual... more We present in this paper an automated method to assess the quality of Jupyter notebooks. The quality of notebooks is assessed in terms of reproducibility and executability. Specifically, we automatically extract a number of expert-defined features for each notebook, perform a feature selection step, and then trained supervised binary classifiers to predict whether a notebook is reproducible and executable, respectively. We also experimented with semantic code embeddings to capture the notebooks' semantics. We have evaluated these methods on a dataset of 306,539 notebooks and achieved an F1 score of 0.87 for reproducibility and 0.96 for executability (using expert-defined features) and an F1 score of 0.81 for reproducibility and 0.78 for executability (using code embeddings). Our results suggest that semantic code embeddings can be used to determine with good performance the reproducibility and executability of Jupyter notebooks, and since they can be automatically derived, they ...

The International FLAIRS Conference Proceedings

The transfer learning pretraining-finetuning paradigm has revolutionized the natural language pr... more The transfer learning pretraining-finetuning paradigm has revolutionized the natural language processing field yielding state-of the art results in several subfields such as text classification and question answering. However, little work has been done investigating pretrained language models for the open student answer assessment task. In this paper, we fine tune pretrained T5, BERT, RoBERTa, DistilBERT, ALBERT and XLNet models on the DT-Grade dataset which contains freely generated (or open) student answers together with judgment of their correctness. The experimental results demonstrated the effectiveness of these models based on the transfer learning pretraining-finetuning paradigm for open student answer assessment. An improvement of 8%-15% in accuracy was obtained over previous methods. Particularly, a T5 based method led to state-of-the-art results with an accuracy and F1 score of 0.88.

Understanding effective human tutors’ strategies is one approach to discovering effective tutoria... more Understanding effective human tutors’ strategies is one approach to discovering effective tutorial strategies. These strategies are described in terms of actions that tutors take while interacting with learners. To this end, we analyze in this paper dialoguebased interactions between professional tutors and tutees. There are two challenges when exploring patterns in such dialoguebased tutorial interactions. First, we need to map utterances, by the tutor and by the tutee, into actions. To address this challenge, we rely on the language-as-action theory according to which when we say something we do something. A second challenge is detecting effective tutorial sessions using objective measurements of learning. To tackle this challenge we align tutorial conversations with preand postmeasures of student mastery obtained from an intelligent tutoring system with which the students interacted before and after interacting with the human tutor. We present performance results of the automated...

Proceedings of the 52nd ACM Technical Symposium on Computer Science Education

We describe our system (DT Team) submitted at SemEval-2017 Task 1, Semantic Textual Similarity (S... more We describe our system (DT Team) submitted at SemEval-2017 Task 1, Semantic Textual Similarity (STS) challenge for English (Track 5). We developed three different models with various features including similarity scores calculated using word and chunk alignments, word/sentence embeddings, and Gaussian Mixture Model (GMM). The correlation between our system's output and the human judgments were up to 0.8536, which is more than 10% above baseline, and almost as good as the best performing system which was at 0.8547 correlation (the difference is just about 0.1%). Also, our system produced leading results when evaluated with a separate STS benchmark dataset. The word alignment and sentence embeddings based features were found to be very effective.

Zenodo (CERN European Organization for Nuclear Research), Jul 18, 2022

Domain modeling is a central component in education technologies as it represents the target doma... more Domain modeling is a central component in education technologies as it represents the target domain students are supposed to train on and eventually master. Automatically generating domain models can lead to substantial cost and scalability benefits. Automatically extracting key concepts or knowledge components from, for instance, textbooks can enable the development of automatic or semi-automatic processes for creating domain models. We explore in this work the use of transformer based pre-trained models for the task of keyphrase extraction. Specifically, we investigate and evaluate four different variants of BERT, a pre-trained transformer based architecture, that vary in terms of training data, training objective, or training strategy to extract knowledge components from textbooks for the domain of intro-toprogramming. We report results obtained using the following BERT-based models: BERT, CodeBERT, SciBERT and RoBERTa.

Lecture Notes in Computer Science, 2022

Intelligent Tutoring Systems

Lecture Notes in Computer Science, 2020

Reported here are the findings of a comparative study on the effects of using a Socratic Intellig... more Reported here are the findings of a comparative study on the effects of using a Socratic Intelligent Tutoring System for source code comprehension and learning computer programming. The result shows there are significant differences between the two groups where students who used Socratic Tutor ITS improved their knowledge by 45% in term of learning gain, developed a better understanding of concepts such as nested if-else and for loop, and improved their confidence level by 13%. Furthermore, the result of the Pearson product-moment correlation coefficient shows a positive correlation (r = 0.68) between feedback from the ITS and learning gain.

Proceedings of the ... International Florida Artificial Intelligence Research Society Conference, May 4, 2022

This paper presents a novel method to automatically assess self-explanations generated by student... more This paper presents a novel method to automatically assess self-explanations generated by students during code comprehension activities. The self-explanations are produced in the context of an online learning environment that asks students to freely explain Java code examples line-by-line. We explored a number of models consisting of textual features in conjunction with machine learning algorithms such as Support Vector Regression (SVR), Decision Trees (DT), and Random Forests (RF). Support Vector Regression (SVR) performed best having a correlation score with human judgments of 0.7088. The best model used a combination of features such as semantic measures obtained using a Sentence BERT pre-trained model and from previously developed semantic algorithms used in a stateof-the-art intelligent tutoring system.

We present in this paper a summary analysis of log files collected during an experiment designed ... more We present in this paper a summary analysis of log files collected during an experiment designed to test the hypothesis that prompting for free self-explanations leads to better comprehension of computer code examples. Indeed, the results indicate that students who were prompted to selfexplain while trying to understand code examples performed significantly better at predicting the correct output of the examples than students who were just prompted to read the code examples and predict their output.

This paper reports the findings of an empirical study on the effects and nature of self explanati... more This paper reports the findings of an empirical study on the effects and nature of self explanation during source code comprehension learning activities in the context of learning computer programming language Java. Our study shows that self explanation helps learning and there is a strong positive correlation between the volume of self-explanation students produce and how much they learn. Furthermore, selfexplanations as an instructional strategy has no discrepancy based on student's prior knowledge. We found that participants explain target code examples using a combination of language, code references, and mathematical expressions. This is not surprising given the nature of the target item, a computer program, but this indicates that automatically evaluating such self-explanations may require novel techniques compared to self-explanations of narrative or scientific texts.

Intelligent Tutoring Systems have been proven to generate excellent learning outcomes in many dom... more Intelligent Tutoring Systems have been proven to generate excellent learning outcomes in many domains such as physics, mathematics and computer programming. However, they have seen relatively little use in training and school classrooms due to the time and cost of designing and authoring. We developed an authoring tool for dialogue-based intelligent tutoring system for programming called Auto-author to reduce the time and cost. The tool allows teachers to create fully functional Socratic tutoring dialogue for learning programming from Java code. First, we conducted a controlled experiment on 45 introductory to programming students to assess auto-authored tutoring dialogues’ learning outcomes. The result shows that the auto-authored dialogues improved students’ programming knowledge by 43% in terms of learning gain. Secondly, we conducted a survey of auto-authored tutoring dialogues by introductory to programming course instructors to evaluate the dialogues’ quality. The result shows...

Computer Science (CS) education is critical in today’s world, and introductory programming course... more Computer Science (CS) education is critical in today’s world, and introductory programming courses are considered extremely difficult and frustrating, often considered a major stumbling block for students willing to pursue computer programming related careers. In this paper, we describe the design of Socratic Tutor, an Intelligent Tutoring System that can help novice programmers to better understand programming concepts. The system was inspired by the Socratic method of teaching in which the main goal is to ask a set of guiding questions about key concepts and major steps or segments of complete code examples. To evaluate the Socratic Tutor, we conducted a pilot study with 34 computer science students and the results are promising in terms of learning gains.

Negation plays a significant role in spoken and written natural languages. Negation is used in la... more Negation plays a significant role in spoken and written natural languages. Negation is used in language to deny something or to reverse the polarity or the sense of a statement. This paper presents a novel approach to automatically handling negation in tutorial dialogues using deep learning methods. In particular, we explored various Long Short Term Memory (LSTM) models to automatically detect negation focus, scope and cue in tutorial dialogues collected from experiments with actual students interacting with the state-of-the-art intelligent tutoring system, DeepTutor. The results obtained are promising.

We present a novel approach to intro-to-programming domain model discovery from textbooks using a... more We present a novel approach to intro-to-programming domain model discovery from textbooks using an over-generation and ranking strategy. We first extract candidate key phrases from each chapter in a Computer Science textbook focusing on intro-to-programming and then rank those concepts according to a number of metrics such as the standard tf-idf weight used in information retrieval and metrics produced by other text ranking algorithms. Specifically, we conduct our work in the context of developing an intelligent tutoring system for source code comprehension for which a specification of the key programming concepts is needed - the system monitors students' performance on those concepts and scaffolds their learning process until they show mastery of the concepts. Our experiments with programming concept instruction from Java textbooks indicate that the statistical methods such as KP Miner method are quite competitive compared to other more sophisticated methods. Automated discover...

The International FLAIRS Conference Proceedings

We present in this paper an automated method to assess the quality of Jupyter notebooks. The qual... more We present in this paper an automated method to assess the quality of Jupyter notebooks. The quality of notebooks is assessed in terms of reproducibility and executability. Specifically, we automatically extract a number of expert-defined features for each notebook, perform a feature selection step, and then trained supervised binary classifiers to predict whether a notebook is reproducible and executable, respectively. We also experimented with semantic code embeddings to capture the notebooks' semantics. We have evaluated these methods on a dataset of 306,539 notebooks and achieved an F1 score of 0.87 for reproducibility and 0.96 for executability (using expert-defined features) and an F1 score of 0.81 for reproducibility and 0.78 for executability (using code embeddings). Our results suggest that semantic code embeddings can be used to determine with good performance the reproducibility and executability of Jupyter notebooks, and since they can be automatically derived, they ...

The International FLAIRS Conference Proceedings

The transfer learning pretraining-finetuning paradigm has revolutionized the natural language pr... more The transfer learning pretraining-finetuning paradigm has revolutionized the natural language processing field yielding state-of the art results in several subfields such as text classification and question answering. However, little work has been done investigating pretrained language models for the open student answer assessment task. In this paper, we fine tune pretrained T5, BERT, RoBERTa, DistilBERT, ALBERT and XLNet models on the DT-Grade dataset which contains freely generated (or open) student answers together with judgment of their correctness. The experimental results demonstrated the effectiveness of these models based on the transfer learning pretraining-finetuning paradigm for open student answer assessment. An improvement of 8%-15% in accuracy was obtained over previous methods. Particularly, a T5 based method led to state-of-the-art results with an accuracy and F1 score of 0.88.

Understanding effective human tutors’ strategies is one approach to discovering effective tutoria... more Understanding effective human tutors’ strategies is one approach to discovering effective tutorial strategies. These strategies are described in terms of actions that tutors take while interacting with learners. To this end, we analyze in this paper dialoguebased interactions between professional tutors and tutees. There are two challenges when exploring patterns in such dialoguebased tutorial interactions. First, we need to map utterances, by the tutor and by the tutee, into actions. To address this challenge, we rely on the language-as-action theory according to which when we say something we do something. A second challenge is detecting effective tutorial sessions using objective measurements of learning. To tackle this challenge we align tutorial conversations with preand postmeasures of student mastery obtained from an intelligent tutoring system with which the students interacted before and after interacting with the human tutor. We present performance results of the automated...

Proceedings of the 52nd ACM Technical Symposium on Computer Science Education

We describe our system (DT Team) submitted at SemEval-2017 Task 1, Semantic Textual Similarity (S... more We describe our system (DT Team) submitted at SemEval-2017 Task 1, Semantic Textual Similarity (STS) challenge for English (Track 5). We developed three different models with various features including similarity scores calculated using word and chunk alignments, word/sentence embeddings, and Gaussian Mixture Model (GMM). The correlation between our system's output and the human judgments were up to 0.8536, which is more than 10% above baseline, and almost as good as the best performing system which was at 0.8547 correlation (the difference is just about 0.1%). Also, our system produced leading results when evaluated with a separate STS benchmark dataset. The word alignment and sentence embeddings based features were found to be very effective.