[Python] SSL Error & Max retries exceeded with url - Request 모듈 과 셀레니움 동시 사용으로 인한 트래픽 중복
qpldocs.dla.mil/search/parts.aspx?qpl=1780 상기의 url에서 아래 이미지에 나오는 리스트들을 텍스트로 추출하고자 했다. 아래는 코드 import requests from bs4 import BeautifulSoup import os from selenium import webdriver from selenium.webdriver.common.by import By from selenium.webdriver.common.keys import Keys from selenium.webdriver.chrome.service import Service from webdriver_manager.chrome import ChromeDriverManager import time fro..
[Eng/Kor, Python for crawling] The way to put 0(zero) in front of 1 digit such as 01, 001 with using format() in for loop // for 문 사용 시 01, 001 등 일의자리 앞에 0을 넣는 방법
In summary, when you using format() in for loop, if you want to put 0(zero) in front of the number of 1 digits, USE {:02d} format When I analyze HTML which I want to crawling web, List's id class pattern start from 03 to 27. //*[@id='Lu_gov_DG_ctl03_btnGovPartNo'] ~ //*[@id='Lu_gov_DG_ctl27_btnGovPartNo'] However, if I use format() in for loop, there is no way to put 0 in front of 1 digit. for i..
[Eng/Kor, HTML, Python] Discrepancy between id, class, name elements in HTML / id, class, name 역할의 차이
Id element -. Use for unique identification purpose when build HTML in order to categorize or name well. FRONT-END-DEVELOPEMENT NEWS ... FRONT-END DEVELOPEMENT HEROS ... -. Use for connection between label and input control. 이메일 Class attribute -. Use for re-use purpose for many times. -. Use for a consistent design or format needed to apply for other several element, like button. -. Application..
for loop in python is a bit difference than for loop in C. In C, #include int main() { for (int i = 0; i >> test_list = ['one', 'two', 'three'] >>> for i in test_list: ... print(i) Above for loop wi..
Alt vs Title element Alt element in HTML, the statement in Alt element appear if image not loaded. Title element in HTML, the statement in Title element appear when your mouse is located on image. DeprecationWarning You can see below warning mention even though your code is running well. DeprecationWarning: find_elements_by_* commands are deprecated. Please use find_elements() instead The reason..
Web Crawling project - QPD site HTML code analysis(1) 품목 검색 시 URL 규칙 찾기 및 세부 리스트 table의 규칙 찾기
QPD 사이트에서 노가다로 획득해야 하는 Mil-spec 정보를 웹 크롤링을 통해 엑셀 데이터로 끌어와서 업무 효율성을 높이려는 프로젝트를 진행 중이다. 그 프로젝트 과정에 있어 일부 느낀 점, 혹은 중요 아이디어 등을 기록으로 남긴다. 1. 품목 검색 시 URL 주소 변화를 관찰했다. M83536을 검색해서 세부 항목으로 타고 들어가자 아래와 같이 URL이 나왔다. https://qpldocs.dla.mil/search/parts.aspx?qpl=1780¶m=M83536/06-017L&type=8 그 결과 qpl=1780 이 M83536 QPL 번호라고 추측 가능했다. 실제로 https://qpldocs.dla.mil/search/parts.aspx?qpl=1780 이렇게만 링크타고 들어가보자 M835..
[Eng/Kor] X path
If you run your code successfuly, you have to use many kinds of elements. (e.g id, name, class and so on) So you have to type your code case by case. However, X path makes you use one way. driver.find_element_by_xpath("//*[@id='Top1_TABSearchLink']").click() # go to link thourgh Xpath driver.find_element_by_id("Top1_TABSearchLink").click() #go to link through ID Element For example above, whatev..
If your selenium never move on ENTER PHASE after typed in browser, and terminal indicated below, 'NoneType' object has no attribute 'send_keys' The solution is put the 'Clear()' GoogleSearch = driver.find_element_by_name("q") GoogleSearch.clear() GoogleSearch.send_keys("Hello world") GoogleSearch.submit() Clear() makes your code work ENTER phase. Bonus - the ways you can enter after typed by sel..
I just run the below simple code but Chrome browser got auto-quite after it worked. import requests from bs4 import BeautifulSoup import os from selenium import webdriver from selenium.webdriver.common.keys import Keys driver = webdriver.Chrome("./chromedriver") url = "https://www.google.com/" driver.get(url) In summary, It was due to version discrepancy between Chrome and Chrome driver. 1. Inst..