python模拟网页操作

Pre-requisites: Python installed and Selenium installed as package along with the web driver (.exe file)

Browser Automation Using Selenium

举例

from selenium import webdriver
import time
import re
opt = webdriver.ChromeOptions()                 #创建浏览器
driver = webdriver.Chrome(options=opt)          #创建浏览器对象
driver.get('https://www.baidu.com/')            #打开网页
# driver.maximize_window()                      #最大化窗口
time.sleep(2)                                   #加载等待
driver.find_element_by_id('kw').send_keys('天气')
driver.find_element_by_xpath("//span[@class='bg s_btn_wr']/input").click()

更新：

You need to import Selenium WebDriver using this code:

from selenium.webdriver.common.by import By

Next use this API:

and so on.

Source: Fixing Selenium AttributeError: ‘WebDriver’ object has no attribute ‘find_element_by_xpath’

non blocking wait （加载出某些元素之后就进行操作，而不是等到整个网页加载完）visibility_of_element_located

https://www.geeksforgeeks.org/non-blocking-wait-in-selenium-using-python/

When we want to do web automation, we require to wait for some javascript elements to load before we perform some action.

The possible solution to this is to wait until a element appears and not wait for more than that.

Let’s say you want to login on GeeksForGeeks through web automation and fill the login credentials as soon as username and password elements are visible on the web page and not wait until the whole page is loaded.

Step1:
You configure the webdriver as follows:

from selenium import webdriver

options = webdriver.ChromeOptions()
options.add_argument("--start-maximized")
options.add_argument("disable-infobars")

chrome = webdriver.Chrome(the_path_of_webdriver_which_is_an_exe,
                        chrome_options = options, service_args =['--ignore-ssl-errors = true'])

login_uri = 'https://auth.geeksforgeeks.org/'
username = 'something'
password = 'anything'
username_xpath = '//*[@id ="luser"]'
password_xpath = '//*[@id ="password"]'
sign_in_xpath = '//*[@id ="Login"]/button'

chrome.get(login_uri)

Here I’ve used chrome web driver which would start maximized (full window) with no infobars i.e it won’t say that chrome is being controlled by automation code and load the sign page of GFG without any hassle.

Do Note that in order to find the xpath of these elements you need to get into the developer mode and inspect these elements.

Step 2:

from selenium.webdriver.support import expected_conditions as EC
from selenium.webdriver.support.wait import WebDriverWait
# return True if element is visible within 30 seconds, otherwise False
def is_visible(locator, timeout = 30):
    try:
        WebDriverWait(chrome, timeout).until(EC.visibility_of_element_located((By.XPATH, locator)))
        return True
    except TimeoutException:
        return False

The above function is_visible is facilitator of the non blocking call we intend to discuss here.
Explanation:
\1) locator – the xpath of the element
\2) timeout – until when to wait for the element to appear (because we don’t want to wait forever)
\3) chrome – the webdriver object we initialized earlier
\4) It utilizes the inbuild utility of ui to make the web driver wait until the element is visible (identified by xpath)
\5) if it does appear within the timeout it returns True else False

Step 3:
This is how we utilize the function:

if not is_visible(username_xpath): raise RuntimeError("Something went wrong with the username field :(")
username_field = chrome.find_element_by_xpath(username_xpath)
username_field.send_keys(username)

if not is_visible(password_xpath): raise RuntimeError("Something went wrong with the password field :(")
password_field = chrome.find_element_by_xpath(password_xpath)
password_field.send_keys(password)

if not is_visible(sign_in_xpath): raise RuntimeError("Something went wrong with the sign in field :(")
sign_in_btn = chrome.find_element_by_xpath(sign_in_xpath)
sign_in_btn.click()

Here we call the is_visible function and pass the xpath of username, password and sign_in button respectively and wait for the element to appear within timeout (here 30s). If not visible then we raise an RuntimeError with appropriate message.
If it appears anytime earlier than 30s it proceeds and find the element by xpath (as now it is visible on the webpage so this call wouldn’t throw exception error.

We then send the data and click on sign in and you can enjoy learning on GFG without any blocking call