Hướng dẫn web crawler using python

<div class="container">

  <h3 id="top-site">Danhsách website</h3>

  <div class="item col-xs-6">

    <p class="title">luyencode.net</p>

  </div>

  <div class="item col-xs-6">

    <p class="title">nguyenvanhieu.vn</p>

  </div>

</div>

0

1

2

3

4

5

6

7

8

9

10

11

12

13

14

15

16

17

18

19

20

21

22

23

24

25

26

27

28

29

30

31

32

33

34

35

36

37

38

39

40

41

42

43

44

45

importrequests

frombs4 importBeautifulSoup

url="https://edition.cnn.com/2021/10/03/uk/everard-uk-police-gbr-cmd-intl/index.html"

# Gửi 1 request đến url phía trên và nhận lại source page của nó

r= requests.get(url)

print(r.status_code)# 200 là thành công

# In thử page source ra coi sao, mình in thử 1000 ký tự đầu tiên thôi

# Bạn có thể thử so sánh với source mà bạn thấy trên trình duyệt

print(r.content[:1000])

# Làm đẹp source

# Mình dùng lxml parser cho nhanh,

# Ngoài ra có html.parser, lxml-xml, html5lib (https://www.crummy.com/software/BeautifulSoup/bs4/doc/)

soup=BeautifulSoup(r.content,'lxml')

print("============TITLE=============")

# Lấy thẻ tiêu đề bài báo

print(soup.title)

# Lấy nội dung tiêu đề

print(soup.title.text)

# Lấy ảnh đại diện của bài báo

# Ta tìm selector của nó theo hướng dẫn ở trên rồi dùng hàm select_one để tìm 1 phần tử đầu tiên

print("============FEATURE IMAGE=============")

feature_image=soup.select_one('#large-media > div > img')

print(feature_image.get('src-large'))

# Lấy nội dung bài

print("============CONTENT=============")

# Mình kiểm tra phần tử thì thấy nội dung bài nằm trong section id="body-text"

content_ele=soup.select_one("#body-text")

print(content_ele.text)

# Lấy tác giả bài viết, ngay phía dưới tiêu đề và phía trên ảnh đại diện

# By Kara Fox, CNN

print("============AUTHOR=============")

author_ele=soup.select_one('body > div.pg-right-rail-tall.pg-wrapper > article > div.l-container > div.metadata > div > p.metadata__byline > span > a')

print(author_ele.text)

200

b'

============TITLE=============

Sarah Everard was murdered by a UK police officer. Without urgent changes to policing, campaigners fear she won't be the last - CNN

============FEATURE IMAGE=============

//cdn.cnn.com/cnnnext/dam/assets/210314050931-04-vigil-sarah-everard-london-0313-super-169.jpg

============CONTENT=============

(CNN)When thousands gathered to mourn the death of Sarah Everard, a 33-year-old woman who was kidnapped, raped and murdered in March, London police responded with force. Scenes of grieving women being pinned to the ground and brutalized by Metropolitan Police officers shocked people across the United Kingdom. Everard was just walking home when she was snatched from the street -- to the mourners it felt like it could have happened to them. They had gathered to "Reclaim These Streets," as the group that organized the vigil is called, and to honor a lost sister. But London's police called the event an illegal gathering and handcuffed some participants, citing Covid regulations. Prosecutors would later say Covid-19 rules were used by one of their own officers to detain and then kidnap Everard. On the night of March 3, Wayne Couzens, a serving Met police officer, spent the entire evening "hunting a lone female to kidnap and rape," according to the judge that sentenced him on Thursday. Couzens stopped Everard on the street by identifying himself as police, "arresting" her under the pretense of breaking Covid rules. He raped her later that evening and strangled her with his police belt. A week later, her remains were found in a woodland in Ashford, Kent -- more than 50 miles from where she was last seen. Everard is not the first woman to be killed by a British policeman. And campaigners fear she won't be the last. Read MoreSarah Everard's murderer kidnapped her using police ID and handcuffsAt least 16 women have been killed by serving or retired police officers over the last 13 years in the UK, according to the Femicide Census, a group that collects data on women killed by men, and campaigners feel that tackling gender-based violence is not a police priority. There are hundreds of allegations of gender-based violence by police officers every year. Nearly 700 domestic abuse allegations were launched against police officers and staff from April 2015-2017, according to a 2019 Bureau of Investigative Journalism investigation. It also found that domestic abuse at the hands of the police are treated differently in court, with just 3.9% of domestic abuse allegations among police in England and Wales ending in a conviction, compared with 6.2% among the general population.  As fatal intimate partner violence is typically the culmination of years of abuse and coercive control, there is an urgent need to reform the criminal justice system, activists say. And that needs to start from the ground up. A woman attending the Sarah Everard vigil in south London on March 13, 2021 is arrested by police.  Harriet Wistrich, solicitor and director of the Centre for Women's Justice (CWJ), told CNN that there is a "kind of boys' locker room type culture within policing, which means that often, officers are loyal to their fellow colleagues over and above undertaking proper investigations -- and that women are fearful of reporting to the police. And if they do, sometimes they're the ones that are victimized."The CWJ filed a "super complaint" to the UK police watchdog in 2019, highlighting the difficulties that around 150 domestic abuse survivors faced in trying to navigate a path to justice. This August, the Independent Office for Police Conduct published its findings, saying that the criminal justice system is failing to function effectively in responding to domestic abuse -- despite the government's claim that it's a priority. (The Domestic Violence Act, which was passed this spring, creates a statutory definition of domestic abuse and established the role of a domestic abuse commissioner.) Since the complaint was filed, more women have come forward with allegations about domestic violence by police, Wistrich said, allegations that fuel deeper concerns about gender-based violence in the force. Speaking outside the courthouse where Couzens was handed a rare whole life sentence on Thursday, Met Police Commissioner Cressida Dick -- who in June dismissed Couzens as an isolated "bad 'un," -- said that she was horrified that he had used a "position of trust to deceive and coerce Sarah." Dick apologized on behalf of the Met and acknowledged that trust in the police had been "shaken." She said as commissioner that she would "do everything in my power to ensure we learn any lessons." On the frontline with the British feminists trying to close the gap between rights and realitySix years ago, as an officer with the Kent Police force, Couzens was accused of indecent exposure. Three days before Everard's murder, he was accused of exposing himself at a fast food restaurant in south London. An investigation has since been opened into the alleged failures by the Met to investigate two allegations against Couzens of indecent exposure in February. Another investigation is looking into allegations that five serving officers and one former officer shared grossly offensive material with Couzens on a WhatsApp group in 2019. And a separate investigation was also opened into allegations that an officer taking part in the search for Everard shared an inappropriate graphic depicting violence against women in a WhatsApp group with colleagues at the time of her disappearance. Inappropriate behavior is not new in the force. In June 2020, after sisters Nicole Smallman and Bibaa Henry were brutally murdered in London, two Met police officers took selfies next to their bodies and shared them on WhatsApp. Six other officers failed to report it.  There is no room for police officers with "question marks" over their previous conduct in the force, Wistrich said.  People protest violence against women and a bill that would give police greater powers in Cardiff, Wales, on March 17. The Met said in a statement that Couzens had been vetted when he joined and had "no criminal convictions or cautions," but that: "Vetting is a snapshot in time and unfortunately, can never 100% guarantee an individual's integrity."  But it also has a history of abetting sexual misconduct. As Couzens was sentenced on Thursday, a landmark judgment from the investigatory powers tribunal (IPT) ruled that the Met had violated an environmental activist's human rights after she had been deceived into a long-term sexual relationship by an undercover officer. Senior officers probably had a "a lack of interest in protecting women" from breaching their human rights and privacy, it said.  The IPT ruling thrusts the spotlight onto other documented abuses of power in the force. From March 2017-2019, 415 referrals were made for officers that had abused their position to sexually assault someone, with domestic and sexual violence victims, sex workers and drug users most at risk of being abused by an on-duty police officer, according to a 2018 Her Majesty's Inspectorate of Constabulary and Fire & Rescue Services report.  Nearly 500 Met police officers were accused of sexual misconduct, including assault and harassment from April 1, 2014-March 31, 2020, according to data released by a Freedom of Information request. Of the 493 complaints, 148 resulted in an investigation. Violence against women in the force is "not an isolated incident," says Wistrich. "It's something, much more kind of rotten within the system that that has to be rooted out."On Saturday, UK Prime Minister Boris Johnson described the police's failure to take violence against women and girls sufficiently seriously as "infuriating," in an interview with The Times. "Are the police taking this issue seriously enough? It's infuriating. I think the public feel that they aren't and they're not wrong," Johnson said."There is an issue about how we handle sexual violence, domestic violence, the sensitivity, the diligence, the time, the delay ... that's the thing we need to fix," he said," adding that the government is trying to "compress" the time between complaints being filed by women and at the point at which action is taken. Rape allegations continue to rise, but cases charged by the Crown Prosecution Service have dropped markedly, according to an October 2020 Victims Commissioner report, which said that in 2019-20 there were 55,000 reports of rape to the police, but only 1,867 cases charged. Plus, the proportion of survivors who chose to withdraw their case is on the rise (from 25% in 2015-16 to 41% in 2019-20). And the government's current strategy doesn't fill many women's rights activists with hope. The government has separated domestic abuse from their new Violence Against Women and Girls strategy, a move that many say misses the crucial point that women's experiences of violence and abuse are interconnected.In response to criticisms of the Met made in this article, a spokesperson told CNN in a statement: "There are numerous wide-ranging points here -- mostly the views of critics and commentators, and we're not setting out to knock back any criticisms they may wish to make."Disconnect and consentSince Couzens' sentencing, high-ranking police have been on a media blitz to signal they understand trust has been broken with the public, saying they will take steps to regain it. They've also outlined measures they believe women should take to feel safer. Women approached by lone police officers have a right to seek verification that the officer detaining them is legitimate, the Met said in a Friday statement.But that wouldn't have stopped Couzens from "arresting" Everard. He wasn't disguised as a police officer, he was a Met police officer -- with an officially issued warrant card to hand.The statement also said that if a woman has doubts about the validity of the arrest, they'd advise "running into a house, knocking on a door, waving a bus down" or calling police. Tributes to Sarah Everard are seen at Clapham Common on March 15 in London.Activists argue the police guidance is tone deaf, places the onus on women to take responsibility for avoiding crimes against them and shifts the focus away from the real problem: How to best identify and stamp out predators in their ranks. A member of Sisters Uncut, a direct-action feminist group who led the Everard vigil in March called the advice "ridiculous."Mirry, who asked CNN to be identified by her first name for safety, said that the Met's "advice shows the disconnect between the management of the police and the day-to-day experiences of the people who assumedly they serve."She added: "I think the focus of this advice is to make it seem as if Couzens was a cop gone rogue, when in fact we know that he was very much of part of a product of the institution."Activists also argue that crucially, such advice fails to take into consideration the concerns of people of color and minority backgrounds, who are disproportionately stopped by authorities in comparison to their White counterparts. They also say it's ableist, with police having not considered the rights of disabled people.London police warn women to be wary of lone officers after Sarah Everard murderThe Met's new guidance comes as the UK Home Secretary calls for an overhaul of policing. On Thursday, Priti Patel said that "all of us want to feel safe and be safe."Patel has also been backing a bill that will give more powers to the police, including tougher stop and search powers. For months, thousands have demonstrated against the Police, Crime, Sentencing and Courts Bill -- which is currently making its way through Parliament. The details of Everard's murder could add to that resistance.  "There's a kind of myth in this country of policing by consent, that the police are able to exercise force over us because they have the confidence of the public because we consent to it," Mirry said, adding: "In the case of Sarah Reed (a victim of police brutality who died in prison) it's used to beat us up; in the case of Sarah Everard, it's used to kill us."We do not consent to have force used against us in the name of our own protection," she said.

============AUTHOR=============

Kara Fox

0

1

2

3

4

5

6

7

8

9

10

11

12

13

14

15

16

17

18

19

20

21

22

23

24

25

26

27

28

29

30

31

32

33

34

35

36

37

38

39

40

41

42

43

44

45

46

47

48

49

50

51

52

53

importrequests

frombs4 importBeautifulSoup

url="https://stackoverflow.com/tags?page={}&tab=popular"

page=1

# Ý tưởng:

# 1.Mình sẽ kéo page source của url về

# 2. Làm đẹp source với bs4

# 3. Tìm tìm block chứa tất cả các block tag của page hiện tại

# 4. Tìm tất cả các block tag, mỗi block là 1 tag

# 5. Với mỗi block, ta viết 1 hàm để đọc các thông tin của nó

# 1. Gửi request tới url với page được truyền vào

r= requests.get(url.format(page))

print('status_code',r.status_code)

# 2. Làm đẹp source

soup= BeautifulSoup(r.content,'lxml')

# 3. Lấy block cha chứa tất cả các tags

block_tag=soup.select_one('div#tags-browser')

# 4. Các thẻ tag là con của block_tag, là các thẻ div có các class dưới đây

all_tags =block_tag.select('div.s-card.js-tag-cell.d-flex.fd-column')

print('Total tags count:',len(all_tags))

# 5. Viết hàm đọc các thông tin từ thẻ tag đó.

# Nếu option xử lý được bật, mình sẽ bỏ qua các khoảng trắng thừa, chỉ lấy giá trị số lượng

defget_tag_info(tag,preprocess=False):

    tag_name=tag.select_one('div.d-flex.jc-space-between.ai-center.mb12 > div > a').text

    tag_description =tag.select_one('div.flex--item.fc-medium.mb12.v-truncate4').text

    question_total=tag.select_one('div.mt-auto.d-flex.jc-space-between.fs-caption.fc-black-400 > div:nth-child(1)').text

    question_today=tag.select_one('div.mt-auto.d-flex.jc-space-between.fs-caption.fc-black-400 > div.flex--item.s-anchors.s-anchors__inherit > a:nth-child(1)').text

    question_this_week= tag.select_one('div.mt-auto.d-flex.jc-space-between.fs-caption.fc-black-400 > div.flex--item.s-anchors.s-anchors__inherit > a:nth-child(2)').text

    ifpreprocess:

        tag_description= tag_description.strip()

        question_total=question_total.split()[0]

        question_today= question_today.split()[0]

        question_this_week=question_this_week.split()[0]

    return tag_name,tag_description,question_total,question_today,question_this_week

# Thử in ra mà ko xử lý

print(get_tag_info(all_tags[0]))

print()

# Thử in ra và qua xử lý

print(get_tag_info(all_tags[0],True))

# Ta có thể lấy toàn bộ bằng cách duyệt lần lượt

# for tag in all_tags:

#     print(get_tag_info(tag, True))

status_code 200

Total tags count: 36

('javascript', '\r\n                        For questions regarding programming in ECMAScript (JavaScript/JS) and its various dialects/implementations (excluding ActionScript). Please include all relevant tags on your question; e.g., [node.js],…\r\n                    ', '2277839 questions', '471 asked today', '4511 this week')

('javascript', 'For questions regarding programming in ECMAScript (JavaScript/JS) and its various dialects/implementations (excluding ActionScript). Please include all relevant tags on your question; e.g., [node.js],…', '2277839', '471', '4511')

0

1

2

3

4

5

6

7

8

9

10

11

12

13

14

15

16

17

18

19

20

21

22

23

24

25

importrequests

# Thay user, pass bằng username và pesonal access token của bạn

r=requests.get('https://api.github.com/user',auth=('user','pass'))

# Kiểm tra status_code

print(r.status_code)

print()

# Lấy content-type mã request trả về từ header

print(r.headers['content-type'])

print()

# Lấy encoding của response được trả về

print(r.encoding)

print()

# Lấy kết quả trả về dạng text

print(r.text)

print()

# Phía trên ta thấy content type là application/json.

# Ta có thấy lấy về dạng dict như sau

print(r.json())

200

application/json; charset=utf8

utf-8

{"type":"User"... # đoạn này là kiểu str

{'private_gists': 219, 'total_private_repos': 7, ...} # đoạn này là kiểu dict