Python UrlLib Introduction

Python UrlLib Introduction

Introduction to Url-Lib Module

Pre-requisite for Urllib:

1. Before delving with the various options in url-lib, it is expected that the readers have basic knowledge about python. urllib in Python3 is slightly different from urllib2 in Python2, so any knowledge in that module would be helpful for understanding the article even more.


1. Urllib module is the URL handling module provide by python.

2. This module provides a high-level interface for retrieving data across the World Wide Web(www). 

3. It is used to fetch URLs (Uniform Resource Locators). It uses the urlopen function and is able to fetch URLs using a variety of different protocols.

4. The urlopen() function is similar to the built-in function open(), but accepts Universal Resource Locators (URLs) instead of filenames.


Through urllib you can do following things-

    1. Can access websites.
    2. Read data from websites.
    3. Parse data.
    4. Modify headers.
    5. do any GET and POST requests.
As urllib is a standard library it is already present in the environment, you just need to import the package.

>>>import urllib

If it is not present in the environment then use the following code

>>>pip install urllib

Urllib vs Urllib2 vs Request:

urllib and urllib2 are both Python modules that provide URL request functionalities but offer different functionalities.

1. urllib2 accepts a Request object to set the headers for a URL request,but urllib does not have any option to accept Request it accepts only a URL.

2. urllib provides the urlencode method which is used for the generation of GET query strings, urllib2 doesn't possess any such function. 

3. It is one of the reasons why urllib is often used along with urllib2.

3. Python Requests has the ability to encode the parameters automatically so one can just pass them as simple arguments, unlike in the case of urllib, where you need to call the method urllib.encode() for the parameters to get encoded.

Urllib sub-modules includes:

urllib is a package that combines several modules for working with URLs-

1. urllib.requests - It is used for opening and reading the URLs

2. urllib.error - It contains the exceptions raised by urllib.requests.

3. urllib.parse - Used for parsing URLs.

4. urllib.robotparser - It is used for parsing robots.txt files.

Sample Program:

Let's look at a sample program.


import urllib.request

request_url = urllib.request.urlopen('') 



The piece of code outputs the source code of the URL i.e.

More Articles of Siddhesh Borkar:

Name Views Likes
Python MySQLdb - Creating Triggers using Python 3654 2
Python MySQLdb - Writing Dynamic SQL Query 2159 2
Python MySQLdb Introduction 392 2
Python UrlLib - How to access the Web using proxies in Python 1393 2
Python How to Set and Get Environment Variables 568 2
Python UrlLib - How to Log-in to a website using urllib 1843 2
Python MySQLdb - Retrieving images from a MySQL table. 751 2
Python UrlLib :- Parse a website using re and urllib 439 2
Python How to Upgrade PIP 291 2
Python Set Environment Variables 393 2
Python MySQLdb - Creating a database table using MySQLdb. 422 2
Python UrlLib:- urllib.parse Module 637 2
Python UrlLib - How to scrape images from a Website 997 2
Python UrlLib Introduction 371 2
Python MySQLdb - Deleting data using MySQLdb 356 2
Python MySQLdb - Updating data into a database. 402 2
Python UrlLib - How to deal with 403 Forbidden Error 3121 2
Python - PIP Remove Package 498 2
Student Management System with database connectivity using tkinter 7078 3
Python UrlLib - urllib.robotparser Module 479 2
Python MySQLdb - Dropping table using MySQLdb 364 2
Python UrlLib :- urlencode Method 429 2
Python UrlLib Module- How to scrap links from a Webpage 501 2
Python MySQLdb - Storing Images in MySQL as a Blob 2195 2
Python UrlLib :- How to download files from the Internet 960 2
Python How to Upgrade PIP3 1664 2
Python UrlLib-urllib.error Module 412 2
Python MySQLdb - Orderby and Limit Clause 444 2
Python Get Environment Variable 442 2
Python UrlLib- Downloading Images as JPG from URL. 871 2
Python MySQLdb - Read Operation using MySQLdb 362 2
Python MySQLdb - Inserting data into a database table using MySQLdb. 386 2
Python Reading Environment Variables 575 2
Python MySQLdb - SQL Joins 554 2
Python How and When to use __str__ 510 2
Python MySQLdb - Performing Transactions & Handling Errors. 510 3
Python UrlLib- urllib.requests Module 498 2