Common Crawl News Dataset (original) (raw)

The news dataset includes articles from news sites all over the world. WARC files are released on a daily basis. The news crawl was started in 2016, please see the news dataset announcement for further information.

The source code of the news crawler is available on our GitHub account. Please, report issues there and share your suggestions for improvements with us.

News Dataset WARC File Location

The WARC file names of the news data set follow the pattern:

crawl-data/CC-NEWS/yyyy/mm/CC-NEWS-yyyymmddHHMMSS-nnnnn.warc.gz

with

yyyy

year

mm

month (01..12)

dd

day of month (01, etc.)

HH

hour (00..23)

MM

minute (00..59)

SS

second (00..59)

nnnnn

serial WARC file number. The serial number is reset when the crawl process is resumed.

The timestamp (yyyymmddHHMMSS) indicates the time the first record in the WARC file was created.

We provide WARC file listings by month. The path listings are found at

s3://commoncrawl/crawl-data/CC-NEWS/yyyy/mm/warc.paths.gz

resp.

https://data.commoncrawl.org/crawl-data/CC-NEWS/yyyy/mm/warc.paths.gz

For accessing the data please see our Get Started page.

News Dataset Size By Year

For every year (linked) we provide an overview by month including links to the WARC file listings.