Flexible Question-Answer Generation: HTML and RSS Parsing

In the previous episode, we implemented custom Question-Answer. But it was too rigid.

So, besides arithmetic problems, what other ways can we generate flexible Question-Answer pairs?

For example, what is the title of the latest post on my blog?

HTML Parsing

If you're a human, how would you find the latest post on my blog? 

Open https://zelikk.blogspot.com/

Find this location, the title is the answer.

So, how does a computer program get this?

We press F12 to open the browser developer tools. Use the selection tool to select the post title we need. 

You can see that in the HTML, the post title corresponds to the element with class="post-title entry-title"

So, we use GPT to program.

Implement the following functionality in Python,

Visit https://zelikk.blogspot.com/

Find the 1st element with class="post-title entry-title" in the HTML, 

Output the text content of this element

Combine the result of GPT programming with the 4 lines of code in the original youtube.py, and you get this blog.py 

Place it in the pset directory and it will take effect. 

import requests

from bs4 import BeautifulSoup

def buildQA():

  question = '我的博客的最新一期博文标题是什么?'

  correct_answer = ''

  url = "https://zelikk.blogspot.com/"

  # 请求网页

  response = requests.get(url)

  # 解析 HTML

  soup = BeautifulSoup(response.text, "html.parser")

  # 找到第一个 class="post-title entry-title" 的元素

  element = soup.find(class_="post-title entry-title")

  if element:

    correct_answer = element.get_text(strip=True)

  else:

    print("没有找到目标元素")

  return question, correct_answer

Note that in order for the above code to run properly, you need to install the corresponding Python libraries.

pip3 install requests BeautifulSoup4 --break-system-packages

RSS XML Parsing

Some people's blogs have dynamically generated pages. If you request the page, you'll get an HTML containing JS, which needs JS to run (usually requesting data from the backend) before you know what posts are there.

So what to do? If the blog supports RSS subscriptions, we can request the RSS feed file for analysis.

Taking my blog as an example, open https://zelikk.blogspot.com/rss.xml

You can see it lists the recent posts

Same as before, use GPT to program

Implement the following functionality in Python

Open https://zelikk.blogspot.com/rss.xml

Get the title of the 1st post

You get rss.py, place it in the pset directory the same way.

import requests

import xml.etree.ElementTree as ET

def buildQA():

  question = '我的博客的最新一期博文标题是什么?'

  correct_answer = ''

  url = "https://zelikk.blogspot.com/rss.xml"

  # 请求 RSS 数据

  response = requests.get(url)

  # 解析 XML

  root = ET.fromstring(response.content)

  # RSS 结构一般是