Keeping it under lock and keywords: exploring new ways to open up the web archives with notebooks (original) (raw)
The UK Government Web Archive (UKGWA) has been archiving government websites since 1996 and now holds regular snapshots of over 5000 sites. Currently, this material can be accessed through browsing or a simple keyword search interface on their website and has also been catalogued in The National Archives’ online catalogue, Discovery. However, the scale of the UKGWA exposes the limits of the current search interface, and there is no facility to understand the archive in aggregate. This article seeks to go beyond the simple keyword search by exploring the data sources available, from APIs to web crawling, for computational analysis of the UKGWA. The article is accompanied by two Python Notebooks which present examples of analysis using each data source. Notebooks lower the technical barriers for the reader to explore and interpret the UKGWA as data, while surfacing the challenges around making web material computationally accessible.
Loading Preview
Sorry, preview is currently unavailable. You can download the paper by clicking the button above.
Related papers
In Proceedings of the 37th Annual International ACM SIGIR Conference on Research and Development in Information Retrieval. ACM Press, New York NY, 2014.