Roman Sitelew - Academia.edu (original) (raw)
Uploads
Book Reviews by Roman Sitelew
![Research paper thumbnail of [in Russian] A review of the monograph: Wessen K.P. Simulating Human Origins and Evolution, Cambridge University Press, Cambridge, 2005, 241pp.](https://attachments.academia-assets.com/58197840/thumbnails/1.jpg)
](Компьютерное моделирование эволюции человека. Рецензия на монографию: Wessen K.P. Simulating Huma... more Компьютерное моделирование эволюции человека. Рецензия на монографию: Wessen K.P. Simulating Human Origins and Evolution, Cambridge University Press, Cambridge, 2005, 241pp.
Papers by Roman Sitelew
Cornell University - arXiv, Jun 9, 2022
Language models demonstrate both quantitative improvement and new qualitative capabilities with i... more Language models demonstrate both quantitative improvement and new qualitative capabilities with increasing scale. Despite their potentially transformative impact, these new capabilities are as yet poorly characterized. In order to inform future research, prepare for disruptive new model capabilities, and ameliorate socially harmful effects, it is vital that we understand the present and near-future capabilities and limitations of language models. To address this challenge, we introduce the Beyond the Imitation Game benchmark (BIG-bench). BIG-bench currently consists of 204 tasks, contributed by 444 authors across 132 institutions. Task topics are diverse, drawing problems from linguistics, childhood development, math, common-sense reasoning, biology, physics, social bias, software development, and beyond. BIG-bench focuses on tasks that are believed to be beyond the capabilities of current language models. We evaluate the behavior of OpenAI's GPT models, Google-internal dense transformer architectures, and Switch-style sparse transformers on BIG-bench, across model sizes spanning millions to hundreds of billions of parameters. In addition, a team of human expert raters performed all tasks in order to provide a strong baseline. Findings include: model performance and calibration both improve with scale, but are poor in absolute terms (and when compared with rater performance); performance is remarkably similar across model classes, though with benefits from sparsity; tasks that improve gradually and predictably commonly involve a large knowledge or memorization component, whereas tasks that exhibit "breakthrough" behavior at a critical scale often involve multiple steps or components, or brittle metrics; social bias typically increases with scale in settings with ambiguous context, but this can be improved with prompting. The BIG-bench github code infrastructure and documentation was developed by Guy Gur-Ari,
Cornell University - arXiv, Dec 5, 2021
Data augmentation is an important component in the robustness evaluation of models in natural lan... more Data augmentation is an important component in the robustness evaluation of models in natural language processing (NLP) and in enhancing the diversity of the data they are trained on. In this paper, we present NL-Augmenter, a new participatory Pythonbased natural language augmentation framework which supports the creation of both transformations (modifications to the data) and filters (data splits according to specific features). We describe the framework and an initial set of 117 transformations and 23 filters for a variety of natural language tasks. We demonstrate the efficacy of NL-Augmenter by using several of its tranformations to analyze the robustness of popular natural language models. The infrastructure, datacards and robutstness analysis results are available publicly on the NL-Augmenter repository (https://github. com/GEM-benchmark/NL-Augmenter).
![Research paper thumbnail of [in Russian] Simple computer modelling of biological processes on the example of "Petri" app](https://attachments.academia-assets.com/58197895/thumbnails/1.jpg)
](Омская биологическая школа. Ежегодник. Вып.5: Межвуз. сб. науч. тр. / Под ред.Б.Ю. Kaccaлa. – Омск: Изд-во ОмГПУ, 2008. – 164 с. – с.112-118, 2008
Simple models of "organism" and "environment" used in " Petri" app are described, the preliminary... more Simple models of "organism" and "environment" used in " Petri" app are described, the preliminary results of simulations are demonstrated
![Research paper thumbnail of [in Russian] Computer modeling in paleontology: a brief review](https://attachments.academia-assets.com/58197964/thumbnails/1.jpg)
](Омская биологическая школа. Ежегодник. Вып.5: Межвуз. сб. науч. тр. / Под ред.Б.Ю. Kaccaлa. – Омск: Изд-во ОмГПУ, 2008. – 164 с. – с.118-126, 2008
The main directions of using computer modeling in paleontology are considered, and the analysis o... more The main directions of using computer modeling in paleontology are considered, and the analysis of the use of computer technology in paleontological research in the time perspective is carried out
Drafts by Roman Sitelew
Компьютерное моделирование эволюции человека. Рецензия на монографию: Wessen K.P. Simulating Huma... more Компьютерное моделирование эволюции человека. Рецензия на монографию: Wessen K.P. Simulating Human Origins and Evolution, Cambridge University Press, Cambridge, 2005, 241pp.
Cornell University - arXiv, Jun 9, 2022
Language models demonstrate both quantitative improvement and new qualitative capabilities with i... more Language models demonstrate both quantitative improvement and new qualitative capabilities with increasing scale. Despite their potentially transformative impact, these new capabilities are as yet poorly characterized. In order to inform future research, prepare for disruptive new model capabilities, and ameliorate socially harmful effects, it is vital that we understand the present and near-future capabilities and limitations of language models. To address this challenge, we introduce the Beyond the Imitation Game benchmark (BIG-bench). BIG-bench currently consists of 204 tasks, contributed by 444 authors across 132 institutions. Task topics are diverse, drawing problems from linguistics, childhood development, math, common-sense reasoning, biology, physics, social bias, software development, and beyond. BIG-bench focuses on tasks that are believed to be beyond the capabilities of current language models. We evaluate the behavior of OpenAI's GPT models, Google-internal dense transformer architectures, and Switch-style sparse transformers on BIG-bench, across model sizes spanning millions to hundreds of billions of parameters. In addition, a team of human expert raters performed all tasks in order to provide a strong baseline. Findings include: model performance and calibration both improve with scale, but are poor in absolute terms (and when compared with rater performance); performance is remarkably similar across model classes, though with benefits from sparsity; tasks that improve gradually and predictably commonly involve a large knowledge or memorization component, whereas tasks that exhibit "breakthrough" behavior at a critical scale often involve multiple steps or components, or brittle metrics; social bias typically increases with scale in settings with ambiguous context, but this can be improved with prompting. The BIG-bench github code infrastructure and documentation was developed by Guy Gur-Ari,
Cornell University - arXiv, Dec 5, 2021
Data augmentation is an important component in the robustness evaluation of models in natural lan... more Data augmentation is an important component in the robustness evaluation of models in natural language processing (NLP) and in enhancing the diversity of the data they are trained on. In this paper, we present NL-Augmenter, a new participatory Pythonbased natural language augmentation framework which supports the creation of both transformations (modifications to the data) and filters (data splits according to specific features). We describe the framework and an initial set of 117 transformations and 23 filters for a variety of natural language tasks. We demonstrate the efficacy of NL-Augmenter by using several of its tranformations to analyze the robustness of popular natural language models. The infrastructure, datacards and robutstness analysis results are available publicly on the NL-Augmenter repository (https://github. com/GEM-benchmark/NL-Augmenter).
Омская биологическая школа. Ежегодник. Вып.5: Межвуз. сб. науч. тр. / Под ред.Б.Ю. Kaccaлa. – Омск: Изд-во ОмГПУ, 2008. – 164 с. – с.112-118, 2008
Simple models of "organism" and "environment" used in " Petri" app are described, the preliminary... more Simple models of "organism" and "environment" used in " Petri" app are described, the preliminary results of simulations are demonstrated
Омская биологическая школа. Ежегодник. Вып.5: Межвуз. сб. науч. тр. / Под ред.Б.Ю. Kaccaлa. – Омск: Изд-во ОмГПУ, 2008. – 164 с. – с.118-126, 2008
The main directions of using computer modeling in paleontology are considered, and the analysis o... more The main directions of using computer modeling in paleontology are considered, and the analysis of the use of computer technology in paleontological research in the time perspective is carried out