爬虫 | 使用html.xpath、driver.xpath两种方式实现查找相似节点的元素

Selenium Webdriver 使用 xpath 查找所有 class 节点具有相似名称的元素

Posted by Haauleon on November 1, 2023

html.xpath

  使用这种 xpath 查询方式的前提是要先通过 webdriver 拿到页面的字符串类型的 html 源代码,然后使用 lxml.tree() 方法将源代码转成可通过 DOM 树的形式进行层级遍历。代码如下:

1
2
3
4
5
6
7
8
9
10
11
12
from lxml import etree


text = driver.page_source
html = etree.HTML(text)

review_ids = html.xpath('//*[@class="a-section review aok-relative"]/@id')
print(review_ids)
if review_ids:
    for review_id in review_ids:
        print(review_id)



driver.xpath

1
2
3
4
5
6
7
8
9
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
from selenium.webdriver.common.by import By


elements = WebDriverWait(driver, 30).until(EC.presence_of_all_elements_located((By.XPATH, '//*[@class="a-section review aok-relative"]')))
for element in elements:
    print(element.get_attribute('id'))




相关链接:
Python Selenium Webdriver查找所有div节点具有相似名称的元素