Build a Site Connectivity Checker in Python (original) (raw)

Watch Now This tutorial has a related video course created by the Real Python team. Watch it together with the written tutorial to deepen your understanding: Building a Site Connectivity Checker

Building a site connectivity checker in Python is an interesting project to level up your skills. With this project, you’ll integrate knowledge related to handling HTTP requests, creating command-line interfaces (CLI), and organizing your application’s code using common Python project layout practices.

By building this project, you’ll learn how Python’s asynchronous features can help you deal with multiple HTTP requests efficiently.

In this tutorial, you’ll learn how to:

To get the most out of this project, you need to know the basics of handling HTTP requests and using argparse to create CLIs. You should also be familiar with the asyncio module and the async and await keywords.

But don’t worry! The topics throughout the tutorial will be introduced in a step-by-step fashion so that you can grasp them as you go. Additionally, you can download the complete source code and other resources for this project by clicking the link below:

Demo: A Site Connectivity Checker

In this step-by-step project, you’ll build an application that checks if one or more websites are online at a given moment. The app will take a list of target URLs at the command line and check them for connectivity either synchronously or asynchronously. The following video shows how the app works:

Your site connectivity checker can take one or more URLs at the command line. Then it creates an internal list of target URLs and checks them for connectivity by issuing HTTP requests and processing the corresponding responses.

Using the -a or --asynchronous option makes the application perform the connectivity checks asynchronously, potentially resulting in lower execution times, especially when you’re processing a long list of websites.

Project Overview

Your website connectivity checker app will provide a few options through a minimal command-line interface (CLI). Here’s a summary of these options:

By default, your application will run the connectivity checks synchronously. In other words, the app will perform the checks one after another.

With the -a or --asynchronous option, you can modify this behavior and make the app run the connectivity checks concurrently. To do this, you’ll take advantage of Python’s asynchronous features and the aiohttp third-party library.

Running asynchronous checks can make your website connectivity checker faster and more efficient, especially when you have a long list of URLs to check.

Internally, your application will use the standard-library http.client module to create a connection to the target website. Once you have a connection, then you can make an HTTP request to the website, which will hopefully react with an appropriate response. If the request is successful, then you’ll know that the site is online. Otherwise, you’ll know that the site is offline.

To display the result of every connectivity check on your screen, you’ll provide your app with a nicely formatted output that will make the app appealing to your users.

Prerequisites

The project that you’ll build in this tutorial will require familiarity with general Python programming. Additionally, it’ll require basic knowledge of the following topics:

Knowing the basics of the aiohttp third-party library would also be a plus but not a requirement. However, if you don’t have all this knowledge yet, then that’s okay! You might learn more by going ahead and giving the project a shot. You can always stop and review the resources linked here if you get stuck.

With this short overview of your website connectivity checker project and the prerequisites, you’re almost ready to start Pythoning and having fun while coding. But first, you need to create a suitable working environment and set up your project’s layout.

Step 1: Set Up Your Site Connectivity Checker Project in Python

In this section, you’ll get ready to start coding your site connectivity checker app. You’ll start by creating a Python virtual environment for the project. This environment will allow you to isolate the project and its dependencies from other projects and your system Python installation.

The next step is to set up the project’s layout by creating all the required files and the directory structure.

To download the code for this first step, click the following link and navigate to the source_code_step_1/ folder:

Set Up the Development Environment

Before you start coding a new project, you should do some preparation. In Python, you typically start by creating a virtual environment for the project. A virtual environment provides an isolated Python interpreter and a space to install your project’s dependencies.

To kick things off, go ahead and create the project’s root directory, called rpchecker_project/. Then move to this directory and run the following commands on your system’s command line or terminal:

The first command creates a fully functional Python virtual environment called venv inside the project’s root directory, while the second command activates the environment. Now run the following command to install the project’s dependencies with pip, the standard Python package manager:

With this command, you install aiohttp into your virtual environment. You’ll use this third-party library along with Python’s async features to handle asynchronous HTTP requests in your site connectivity checker app.

Cool! You have a working Python virtual environment with all the dependencies that you’ll need to start building your project. Now you can create the project’s layout to organize your code following Python best practices.

Organize Your Site Connectivity Checker Project

Python is surprisingly flexible when it comes to structuring applications, so you may find pretty different structures from project to project. However, small installable Python projects typically have a single package, which is often named after the project itself.

Following this practice, you can organize your site connectivity checker app using the following directory structure:

rpchecker_project/ │ ├── rpchecker/ │ ├── __init__.py │ ├── __main__.py │ ├── checker.py │ └── cli.py │ ├── README.md └── requirements.txt

You can use any name for this project and its main package. In this tutorial, the project will be named rpchecker as a combination of Real Python (rp) and checker, which points out the app’s main functionality.

The README.md file will contain the project’s description and instructions for installing and running the application. Adding a README.md file to your projects is a best practice in programming, especially if you’re planning to release the project as an open source solution. To learn more about writing good README.md files, check out How to write a great README for your GitHub project.

The requirements.txt file will hold the list of your project’s external dependencies. In this case, you only need the aiohttp library, because the rest of the tools and modules that you’ll use are available out of the box in the Python standard library. You can use this file to automatically reproduce the appropriate Python virtual environment for your app using pip, the standard package manager.

Inside the rpchecker/ directory, you’ll have the following files:

Now go ahead and create all these files as empty files. You can do this by using your favorite code editor or IDE. Once you finish creating the project’s layout, then you can start coding the app’s main functionality: checking if a website is online or not.

Step 2: Check Websites’ Connectivity in Python

At this point, you should have a suitable Python virtual environment with your project’s dependencies installed in it. You should also have a project directory containing all the files that you’ll use throughout this tutorial. It’s time to start coding!

Before jumping into the really fun stuff, go ahead and add the application’s version number to the __init__.py module in your rpchecker package:

The __version__ module-level constant holds your project’s current version number. Because you’re creating a brand-new app, the initial version is set to 0.1.0. With this minimal setup, you can start implementing the connectivity-checking functionality.

To download the code for this step, click the following link and look inside the source_code_step_2/ folder:

Implement a Connectivity Checker Function

There are several Python tools and libraries that you can use to check if a website is online at a given time. For example, a popular option is the requests third-party library, which allows you to perform HTTP requests using a human-readable API.

However, using requests has the drawback of installing an external library just to use a minimal part of its functionality. It’d be more efficient to find an appropriate tool in the Python standard library.

With a quick look at the standard library, you’ll find the urllib package, which provides several modules for handling HTTP requests. For example, to check if a website is online, you can use the urlopen() function from the urllib.request module:

The urlopen() function takes a URL and opens it, returning its content as a string or a Request object. But you just need to check if the website is online, so downloading the entire page would be wasteful. You need something more efficient.

What about a tool that gives you lower-level control over your HTTP request? That’s where the http.client module comes in. This module provides the HTTPConnection class, representing a connection to a given HTTP server.

HTTPConnection has a .request() method that allows you to perform HTTP requests using the different HTTP methods. For this project, you can use the HEAD HTTP method to ask for a response containing only the headers of the target website. This option will reduce the amount of data to download, making your connectivity checker app more efficient.

At this point, you have a clear idea of the tool to use. Now you can do some quick tests. Go ahead and run the following code in a Python interactive session:

In this example, you first create an HTTPConnection instance targeting the pypi.org website. The connection uses port 80, which is the default HTTP port. Finally, the timeout argument provides the number of seconds to wait before timing out connection attempts.

Then you perform a HEAD request on the site’s root path, "/", using .request(). To get the actual response from the server, you call .getresponse() on the connection object. Finally, you inspect the response’s headers by calling .getheaders().

Your website connectivity checker just needs to create a connection and make a HEAD request. If the request is successful, then the target website is online. Otherwise, the site is offline. In the latter case, it’d be appropriate to display an error message to the user.

Now go ahead and open the checker.py file in your code editor. Then add the following code to it:

Here’s a breakdown of what this code does line by line:

Your site_is_online() function returns True if the target website is available online. Otherwise, it raises an exception pointing out the problem it encountered. This latter behavior is convenient because you need to show an informative error message when the site isn’t online. Now it’s time to try out your new function.

Run Your First Connectivity Checks

To try out your site_is_online() function, go ahead and get back to your interactive session. Then run the following code:

In this code snippet, you first import site_is_online() from the checker module. Then you call the function with "python.org" as an argument. Because the function returns True, you know that the target site is online.

In the final example, you call site_is_online() with a non-existing website as a target URL. In this case, the function raises an exception that you can later catch and process to display an error message to the user.

Great! You’ve implemented the application’s main functionality of checking a website’s connectivity. Now you can continue with your project by setting up its CLI.

Step 3: Create Your Website Connectivity Checker’s CLI

So far, you have a working function that allows you to check if a given website is online by performing an HTTP request using the http.client module from the standard library. At the end of this step, you’ll have a minimal CLI that will allow you to run your website connectivity checker app from the command line.

The CLI will include options for taking a list of URLs at the command line and loading a list of URLs from a text file. The application will also display the connectivity check results with a user-friendly message.

To create the application’s CLI, you’ll use argparse from the Python standard library. This module allows you to build user-friendly CLIs without installing any external dependencies, such as Click or Typer.

To get started, you’ll write the required boilerplate code for working with argparse. You’ll also code the option to read URLs from the command line.

Click the link below to download the code for this step so that you can follow along with the project. You’ll find what you need in the source_code_step_3/ folder:

Parse Website URLs at the Command Line

To build the application’s CLI with argparse, you need to create an ArgumentParser instance so that you can parse arguments provided at the command line. Once you have an argument parser, then you can start adding arguments and options to your app’s CLI.

Now go ahead and open the cli.py file in your code editor. Then add the following code:

In this code snippet, you create read_user_cli_args() to keep the functionality related to the argument parser in a single place. To build the parser object, you use two arguments:

After creating the argument parser, you add a first command-line argument using .add_argument(). This argument will allow the user to enter one or more URLs at the command line. It’ll use the -u and --urls switches.

The rest of the arguments to .add_argument() work as follows:

Finally, your function returns the result of calling .parse_args() on the parser object. This method returns a Namespace object containing the parsed arguments.

Load Website URLs From a File

Another valuable option to implement in your site connectivity checker is the ability to load a list of URLs from a text file on your local machine. To do this, you can add a second command-line argument with the -f and --input-file flags.

Go ahead and update read_user_cli_args() with the following code:

To create this new command-line argument, you use .add_argument() with almost the same arguments as in the section above. In this case, you aren’t using the nargs argument, because you want the application to accept only one input file at the command line.

Display the Check Results

An essential component of every application that interacts with the user through the command line is the application’s output. Your application needs to show the result of its operations to the user. This feature is vital for ensuring a pleasant user experience.

Your site connectivity checker doesn’t need a very complex output. It just needs to inform the user about the current status of the checked websites. To implement this functionality, you’ll code a function called display_check_result().

Now get back to the cli.py file and add the function at the end:

This function takes the connectivity check result, the checked URL, and an optional error message. The conditional statement tests to see if result is true, in which case an "Online!" message is printed to the screen. If result is false, then the else clause prints "Offline?" along with an error report about the actual problem that has just occurred.

That’s it! Your website connectivity checker has a command-line interface to allow the user to interact with the application. Now it’s time to put everything together in the application’s entry-point script.

Step 4: Put Everything Together in the App’s Main Script

So far, your site connectivity checker project has a function that checks if a given website is online. It also has a CLI that you quickly built using the argparse module from the Python standard library. In this step, you’ll write the glue code—the code that will bring all these components together and make your application work as a full-fledged command-line app.

To kick things off, you’ll start by setting up the application’s main script or entry-point script. This script will contain the main() function and some high-level code that will help you connect the CLI in the front-end with the connectivity-checking functionality in the back-end.

To download the code for this step, click the link below, then check out the source_code_step_4/ folder:

Create the Application’s Entry-Point Script

The next step in building your website connectivity checker app is to define the entry-point script with a suitable main() function. To do this, you’ll use the __main__.py file that lives in the rpchecker package. Including a __main__.py file in a Python package enables you to run the package as an executable program using the command python -m <package_name>.

To start populating __main__.py with code, go ahead and open the file in your code editor. Then add the following:

After importing read_user_cli_args() from the cli module, you define the app’s main() function. Inside main(), you’ll find a few lines of code that don’t work yet. Here’s what this code should do after you provide the missing functionality:

With main() in place, you can start coding the missing pieces to make it work correctly. In the following sections, you implement _get_websites_urls() and _synchronous_check(). Once they’re ready to go, you’ll be able to run your website connectivity checker app for the first time.

Build the List of Target Website URLs

Your site connectivity checker app will be able to check multiple URLs in every execution. Users will feed URLs into the app by listing them at the command line, providing them in a text file, or both. To create the internal list of target URLs, the app will first process URLs provided at the command line. Then it’ll add additional URLs from a file, if any.

Here’s the code that accomplishes these tasks and returns a list of target URLs that combines both sources, the command line and an optional text file:

The first update in this code snippet is to import pathlib to manage the path to the optional URLs file. The second update is to add the _get_websites_urls() helper function, which does the following:

At the same time, _read_urls_from_file() runs the following actions:

The else clause on lines 25 to 26 prints an error message to point out that the input file doesn’t exit. If the function runs without returning a valid list of URLs, it returns an empty list.

Wow! That was a lot, but you made it to the end! Now you can continue with the final part of __main__.py. In other words, you can implement the _synchronous_check() function so that the app can perform connectivity checks on multiple websites.

Check the Connectivity of Multiple Websites

To run connectivity checks over multiple websites, you need to iterate through the list of target URLs, do the checks, and display the corresponding results. That’s what the _synchronous_check() function below does:

In this piece of code, you first update your imports by adding site_is_online() and display_check_result(). Then you define _synchronous_check(), which takes a list of URLs as arguments. The function’s body works like this:

To wrap up the __main__.py file, you add the typical Python if __name__ == "__main__": boilerplate code. This snippet calls main() when the module is run as a script or executable program. With these updates, your application is now ready for a test flight!

Run Connectivity Checks From the Command Line

You’ve written a ton of code without having the chance to see it in action. You’ve coded the site connectivity checker’s CLI and its entry-point script. Now it’s time to give your application a try. Before doing that, make sure you’ve downloaded the bonus material mentioned at the beginning of this step, especially the sample-urls.txt file.

Now get back to your command line and execute the following commands:

Your website connectivity checker works great! When you run rpchecker with the -h or --help option, you get a usage message that explains how to use the app.

The application can take several URLs at the command line or from a text file and check them for connectivity. If an error occurs during the check, then you get a message on the screen with information about what’s causing the error.

Go ahead a try some other URLs and features. For example, try to combine URLs at the command line with URLs from a file using the -u and -f switches. Additionally, check what happens when you provide a URLs file that’s empty or nonexistent.

Cool! Your site connectivity checker app works nicely and smoothly, doesn’t it? However, it has a hidden issue. The execution time can be overwhelming when you run the application with a long list of target URLs, because all the connectivity checks run synchronously.

To work around this issue and improve the application’s performance, you can implement asynchronous connectivity checks. That’s what you’ll do in the following section.

Step 5: Check Websites’ Connectivity Asynchronously

By performing the connectivity checks on multiple websites concurrently through asynchronous programming, you can improve the overall performance of your application. To do this, you can take advantage of Python’s asynchronous features and the aiohttp third-party library, which you already have installed in your project’s virtual environment.

Python supports asynchronous programming with the asyncio module and the async and await keywords. In the following sections, you’ll write the required code to make your app run the connectivity checks asynchronously using these tools.

To download the code for this final step, click the following link and look into the source_code_step_5/ folder:

Implement an Asynchronous Connectivity Checker Function

The first step in the process of making your website connectivity checker work concurrently is to write an async function that allows you to perform a single connectivity check on a given website. This will be the asynchronous equivalent to your site_is_online() function.

Get back to the checker.py file and add the following code:

In this update, you first add the required imports, asyncio and aiohttp. Then you define site_is_online_async() on line 10. It’s an async function that takes two arguments: the URL to check and the number of seconds before the requests time out. The function’s body does the following:

This implementation of site_is_online_async() is similar to the implementation of site_is_online(). It returns True if the target website is online. Otherwise, it raises an exception pointing out the encountered problem.

The main difference between these functions is that site_is_online_async() performs the HTTP requests asynchronously using the aiohttp third-party library. This distiction can help you optimize your app’s performance when you have a long list of websites to check.

With this function in place, you can proceed to update your application’s CLI with a new option that allows you to run the connectivity checks asynchronously.

Add an Asynchronous Option to the Application’s CLI

Now you need to add an option to your site connectivity checker app’s CLI. This new option will tell the app to run the checks asynchronously. The option can just be a Boolean flag. To implement this type of option, you can use the action argument of .add_argument().

Now go ahead and update read_user_cli_args() on the cli.py file with the following code:

This call to .add_argument() on the parser object adds a new -a or --asynchronous option to the application’s CLI. The action argument is set to "store_true", which tells argparse that -a and --asynchronous are Boolean flags that will store True when provided at the command line.

With this new option in place, it’s time to write the logic for checking the connectivity of multiple websites asynchronously.

Check the Connectivity of Multiple Websites Asynchronously

To check the connectivity of multiple websites asynchronously, you’ll write an async function that calls and awaits site_is_online_async() from the checker module. Get back to the __main__.py file and add the following code to it:

In this piece of code, you first update your imports to access site_is_online_async(). Then you define _asynchronous_check() on line 10 as an asynchronous function using the async keyword. This function takes a list of URLs and checks their connectivity asynchronously. Here’s how it does that:

Okay! You’re almost ready to try out the asynchronous capabilities of your site connectivity checker app. Before doing that, you need to take care of a final detail: updating the main() function to integrate this new feature.

Add Asynchronous Checks to the App’s Main Code

To add the asynchronous functionality to your application’s main() function, you’ll use a conditional statement to check if the user provided the -a or --asynchronous flag at the command line. This conditional will allow you to run the connectivity checks with the right tool according to the user’s input.

Go ahead and open the __main__.py file again. Then update main() like in the following code snippet:

The conditional statement on lines 12 to 15 checks if the user has provided the -a or --asynchronous flag at the command line. If that’s the case, main() runs the connectivity checks asynchronously using asyncio.run(). Otherwise, it runs the checks synchronously using _synchronous_check().

That’s it! You can now test this new feature of your website connectivity checker in practice. Get back to your command line and run the following:

The first command shows that your application now has a new -a or --asynchronous option that will run the connectivity checks asynchronously.

The second command makes rpchecker run the connectivity checks synchronously, just like you did in the previous section. This is because you don’t provide the -a or --asynchronous flags. Note that the URLs are checked in the same order that they’re entered at the command line.

Finally, in the third command, you use the -a flag at the end of the line. This flag makes rpchecker run the connectivity checks concurrently. Now the check results aren’t displayed in the same order as the URLs are entered but in the order in which the responses come from the target websites.

As an exercise, you can try to run your site connectivity checker application with a long list of target URLs and compare the execution time when the app runs the checks synchronously and asynchronously.

Conclusion

You’ve built a functional site connectivity checker application in Python. Now you know the basics of handling HTTP requests to a given website. You also learned how to create a minimal yet functional command-line interface (CLI) for your application and how to organize a real-world Python project. Additionally, you’ve tried out Python’s asynchronous features.

In this tutorial, you learned how to:

With this foundation, you’re ready to continue leveling up your skills by creating more complex command-line applications. You’re also better prepared to continue learning about HTTP requests with Python.

To review what you’ve done to build your app, you can download the complete source code below:

Next Steps

Now that you’ve finished building your site connectivity checker application, you can go a step further by implementing a few additional features. Adding new features by yourself will push you to learn about new and exciting coding concepts and topics.

Here are some ideas for new features:

To implement these features, you can take advantage of Python’s time module, which will allow you to measure the execution time of your code.

Once you implement these new features, then you can change gears and jump into other cool and more complex projects. Here are some great next steps for you to continue learning Python and building projects:

Watch Now This tutorial has a related video course created by the Real Python team. Watch it together with the written tutorial to deepen your understanding: Building a Site Connectivity Checker