Hướng dẫn web crawler python geeksforgeeks

pip install requests

https://www.geeksforgeeks.org/python-programming-language/
200

pip install beautifulsoup4

title
html

tag with id as main. Now lets’s get the HTML content under this tag. Now let’s inspect more of the page get the content of the leftbar. 

We can see that the list in the leftbar is under the

tag with the class as leftBarList and our task is to find all the li under this ul.

Python3

import requests

from bs4 import BeautifulSoup

soup = BeautifulSoup(r.content, 'html.parser')

s = soup.find('div', id= 'main')

leftbar = s.find('ul', class_='leftBarList')

content = leftbar.find_all('li')

print(content)

Output:

Extracting Text from the tags

In the above examples, you must have seen that while scraping the data the tags also get scraped but what if we want only the text without any tags. Don’t worry we will discuss the same in this section. We will be using the text property. It only prints the text from the tag. We will be using the above example and will remove all the tags from them.

Example 1: Removing the tags from the content of the page 

Python3

Output:

Example 2: Removing the tags from the content of the leftbar

Python3

import requests

from bs4 import BeautifulSoup

soup = BeautifulSoup(r.content, 'html.parser')

s = soup.find('div', id= 'main')

leftbar = s.find('ul', class_='leftBarList')

lines = leftbar.find_all('li')

for line in lines:

    print(line.text)

Output:

Extracting Links

Till now we have seen how to extract text, let’s now see how to extract the links from the page.

Example: Python BeautifulSoup Extracting Links

Python3

Output:

Extracting Image Information

On again inspecting the page, we can see that images lie inside the img tag and the link of that image is inside the src attribute. See  the below image – 

Example: Python BeautifulSoup Extract Image 

Python3

import requests

from bs4 import BeautifulSoup

soup = BeautifulSoup(r.content, 'html.parser')

images_list = []

images = soup.select('img')

for image in images:

    src = image.get('src')

    alt = image.get('alt')

    images_list.append({"src": src, "alt": alt})

for image in images_list:

    print(image)

Output:

Scraping multiple Pages

Now, there may arise various instances where you may want to get data from multiple pages from the same website or multiple different URLs as well, and manually writing code for each webpage is a time-consuming and tedious task. Plus, it defines all basic principles of automation. Duh!  

To solve this exact problem, we will see two main techniques that will help us extract data from multiple webpages:

• The same website
• Different website URLs

Example 1: Looping through the page numbers 

page numbers at the bottom of the GeeksforGeeks website

Most websites have pages labeled from 1 to N. This makes it really simple for us to loop through these pages and extract data from them as these pages have similar structures. For example:

page numbers at the bottom of the GeeksforGeeks website

Here, we can see the page details at the end of the URL. Using this information we can easily create a for loop iterating over as many pages as we want (by putting page/(i)/ in the URL string and iterating “i” till N) and scrape all the useful data from them. The following code will give you more clarity over how to scrape data by using a For Loop in Python.

Python3

import requests

from bs4 import BeautifulSoup as bs

req = requests.get(URL)

soup = bs(req.text, 'html.parser')

titles = soup.find_all('div',attrs = {'class','head'})

print(titles[4].text)

Output:

7 Most Common Time Wastes During Software Development

Now, using the above code, we can get the titles of all the articles by just sandwiching those lines with a loop.

Python3

import requests

from bs4 import BeautifulSoup as bs

for page in range(1, 10):

    req = requests.get(URL + str(page) + '/')

    soup = bs(req.text, 'html.parser')

    titles = soup.find_all('div', attrs={'class', 'head'})

    for i in range(4, 19):

        if page > 1:

            print(f"{(i-3)+page*15}" + titles[i].text)

        else:

            print(f"{i-3}" + titles[i].text)

Output:

Example 2: Looping through a list of different URLs

The above technique is absolutely wonderful, but what if you need to scrape different pages, and you don’t know their page numbers? You’ll need to scrape those different URLs one by one and manually code a script for every such webpage.

Instead, you could just make a list of these URLs and loop through them. By simply iterating the items in the list i.e. the URLs, we will be able to extract the titles of those pages without having to write code for each page. Here’s an example code of how you can do it.

Python3

import requests

from bs4 import BeautifulSoup as bs

for url in range(0,2):

    req = requests.get(URL[url])

    soup = bs(req.text, 'html.parser')

    titles = soup.find_all('div',attrs={'class','head'})

    for i in range(4, 19):

        if url+1 > 1:

            print(f"{(i - 3) + url * 15}" + titles[i].text)

        else:

            print(f"{i - 3}" + titles[i].text)

Output:

For more information, refer to our Python BeautifulSoup Tutorial.

Saving Data to CSV

First we will create a list of dictionaries with the key value pairs that we want to add in the CSV file. Then we will use the csv module to write the output in the CSV file. See the below example for better understanding.

Example: Python BeautifulSoup saving to CSV

Python3

import requests

from bs4 import BeautifulSoup as bs

import csv

soup = bs(req.text, 'html.parser')

titles = soup.find_all('div', attrs={'class', 'head'})

titles_list = []

count = 1

for title in titles:

    d = {}

    d['Title Number'] = f'Title {count}'

    d['Title Name'] = title.text

    count += 1

    titles_list.append(d)

filename = 'titles.csv'

with open(filename, 'w', newline='') as f:

    w = csv.DictWriter(f,['Title Number','Title Name'])

    w.writeheader()

    w.writerows(titles_list)

Output: