Flexible Question-Answer Generation: HTML and RSS Parsing

查看相关主题:
In the previous episode, we implemented custom Question-Answer. But it was too rigid.
So, besides arithmetic problems, what other ways can we generate flexible Question-Answer pairs?
For example, what is the title of the latest blog post on my blog?

html parsing

If you were a human, how would you find the latest blog post on my blog? 
Find this spot, and the title is the answer.

So, how does a computer program fetch this entry?
Press F12 to open the browser's developer tools. Use the element selector to pick the blog post title we need. 

As you can see in the HTML, the blog post title corresponds to the element with class="post-title entry-title"

So, let's use GPT to code it.
Implement the following function in python,
Visit https://zelikk.blogspot.com/
In the HTML, find the 1st element with class="post-title entry-title", 
Output the text content of that element
Merge the result from GPT with the original 4 lines of code in youtube.py, and you get this blog.py 
Drop it in the pset directory and it'll take effect. 
import requests
from bs4 import BeautifulSoup


def buildQA():
  question = '我的博客的最新一期博文标题是什么?'
  correct_answer = ''


  url = "https://zelikk.blogspot.com/"


  # Request the webpage
  response = requests.get(url)


  # Parse HTML
  soup = BeautifulSoup(response.text, "html.parser")


  # Find the first element with class="post-title entry-title"
  element = soup.find(class_="post-title entry-title")


  if element:
    correct_answer = element.get_text(strip=True)
  else:
    print("Target element not found")


  return question, correct_answer
Note: in order for the code above to run properly, you need to install the corresponding python libraries.
pip3 install requests BeautifulSoup4 --break-system-packages


rss xml parsing

Some blogs render their pages dynamically. If you request the page directly, you'll get an HTML full of JavaScript, and you won't know what the posts are until the JS runs (usually fetching data from a backend).
So what do we do? If the blog supports RSS subscriptions, we can request the RSS feed file and analyze it.
Taking my blog as an example, open https://zelikk.blogspot.com/rss.xml
You can see that it lists the most recent few blog posts
Same as before, use GPT to code it
Implement the following function in python
Open https://zelikk.blogspot.com/rss.xml
Get the title of the 1st blog post
Get rss.py. Same as before, just drop it in the pset directory.
import requests
import xml.etree.ElementTree as ET


def buildQA():
  question = '我的博客的最新一期博文标题是什么?'
  correct_answer = ''


  url = "https://zelikk.blogspot.com/rss.xml"


  # Request the RSS data
  response = requests.get(url)


  # Parse XML
  root = ET.fromstring(response.content)


  # RSS structure is generally <rss><channel><item><title>
  first_item_title = root.find(".//channel/item/title")
  
  if first_item_title:
    correct_answer = first_item_title.text.strip()
  else:
    print("Blog post title not found")


  return question, correct_answer
Note: in order for the code above to run properly, you need to install the corresponding python libraries.
pip3 install requests --break-system-packages

Project Updated

Github


Comments
Post a Comment