GitHub - wabarc/wayback: An archiving tool with an IM-style interface that prioritizes privacy and accessibility, integrated with various archival services including Internet Archive, archive.today, Ghostarchive, IPFS, Telegraph, and file systems. (original) (raw)

Wayback

Wayback is a web archiving and playback tool that allows users to capture and preserve web content. It provides an IM-style interface for receiving and presenting archived web content, and a search and playback service for retrieving previously archived pages. Wayback is designed to be used by web archivists, researchers, and anyone who wants to preserve web content and access it in the future.

Features

Free and open-source
Expose prometheus metrics
Cross-platform compatibility
Batch wayback URLs for faster archiving
Built-in CLI (wayback) for convenient use
Serve as a Tor Hidden Service or local web entry for added privacy and accessibility
Easier wayback to Internet Archive, archive.today, IPFS and Telegraph integration
Interactive with IRC, Matrix, Telegram bot, Discord bot, Mastodon, Twitter, and XMPP as a daemon service for convenient use
Supports publishing wayback results to Telegram channel, Mastodon, and GitHub Issues for sharing
Supports storing archived files to disk for offline use
Download streaming media (requires FFmpeg) for convenient media archiving.

Getting Started

For a comprehensive guide, please refer to the complete documentation.

Installation

The simplest, cross-platform way is to download from GitHub Releases and place the executable file in your PATH.

From source:

go install github.com/wabarc/wayback/cmd/wayback@latest

From GitHub Releases:

curl -fsSL https://get.wabarc.eu.org | sh

or via Bina:

curl -fsSL https://bina.egoist.dev/wabarc/wayback | sh

Using Snapcraft (on GNU/Linux)

sudo snap install wayback

Via APT:

curl -fsSL https://repo.wabarc.eu.org/apt/gpg.key | sudo gpg --dearmor -o /usr/share/keyrings/packages.wabarc.gpg echo "deb [arch=amd64,arm64,armhf signed-by=/usr/share/keyrings/packages.wabarc.gpg] https://repo.wabarc.eu.org/apt/ /" | sudo tee /etc/apt/sources.list.d/wayback.list sudo apt update sudo apt install wayback

Via RPM:

sudo rpm --import https://repo.wabarc.eu.org/yum/gpg.key sudo tee /etc/yum.repos.d/wayback.repo > /dev/null <<EOT [wayback] name=Wayback Archiver baseurl=https://repo.wabarc.eu.org/yum/ enabled=1 gpgcheck=1 gpgkey=https://repo.wabarc.eu.org/yum/gpg.key EOT

sudo dnf install -y wayback

Via Homebrew:

brew tap wabarc/wayback brew install wayback

Usage

Command line

$ wayback -h

A command-line tool and daemon service for archiving webpages.

Usage: wayback [flags]

Examples: wayback https://www.wikipedia.org wayback https://www.fsf.org https://www.eff.org wayback --ia https://www.fsf.org wayback --ia --is -d telegram -t your-telegram-bot-token WAYBACK_SLOT=pinata WAYBACK_APIKEY=YOUR-PINATA-APIKEY
WAYBACK_SECRET=YOUR-PINATA-SECRET wayback --ip https://www.fsf.org

Flags: --chatid string Telegram channel id -c, --config string Configuration file path, defaults: ./wayback.conf, ~/wayback.conf, /etc/wayback.conf -d, --daemon strings Run as daemon service, supported services are telegram, web, mastodon, twitter, discord, slack, irc, xmpp --debug Enable debug mode (default mode is false) --ga Wayback webpages to Ghostarchive (default true) -h, --help help for wayback --ia Wayback webpages to Internet Archive (default true) --info Show application information --ip Wayback webpages to IPFS (default true) --ipfs-host string IPFS daemon host, do not require, unless enable ipfs (default "127.0.0.1") -m, --ipfs-mode string IPFS mode (default "pinner") -p, --ipfs-port uint IPFS daemon port (default 5001) --is Wayback webpages to Archive Today (default true) --migrate Run SQL migrations --ph Wayback webpages to Telegraph (default true) --print Show application configurations -t, --token string Telegram Bot API Token --tor Snapshot webpage via Tor anonymity network --tor-key string The private key for Tor Hidden Service -v, --version version for wayback

Examples

Wayback one or more url to Internet Archive and archive.today:

wayback https://www.wikipedia.org

wayback https://www.fsf.org https://www.eff.org

Wayback url to Internet Archive or archive.today or IPFS:

// Internet Archive $ wayback --ia https://www.fsf.org

// archive.today $ wayback --is https://www.fsf.org

// IPFS $ wayback --ip https://www.fsf.org

For using IPFS, also can specify a pinning service:

$ export WAYBACK_SLOT=pinata $ export WAYBACK_APIKEY=YOUR-PINATA-APIKEY $ export WAYBACK_SECRET=YOUR-PINATA-SECRET $ wayback --ip https://www.fsf.org

// or

$ WAYBACK_SLOT=pinata WAYBACK_APIKEY=YOUR-PINATA-APIKEY
$ WAYBACK_SECRET=YOUR-PINATA-SECRET wayback --ip https://www.fsf.org

More details about pinning service.

With telegram bot:

wayback --ia --is --ip -d telegram -t your-telegram-bot-token

Publish message to your Telegram channel at the same time:

wayback --ia --is --ip -d telegram -t your-telegram-bot-token --chatid your-telegram-channel-name

Also can run with debug mode:

wayback -d telegram -t YOUR-BOT-TOKEN --debug

Both serve on Telegram and Tor hidden service:

wayback -d telegram -t YOUT-BOT-TOKEN -d web

URLs from file:

Configuration Parameters

Look at the full list of configuration options.

Deployment

Docker/Podman

docker pull wabarc/wayback docker run -d wabarc/wayback wayback -d telegram -t YOUR-BOT-TOKEN # without telegram channel docker run -d wabarc/wayback wayback -d telegram -t YOUR-BOT-TOKEN -c YOUR-CHANNEL-USERNAME # with telegram channel