Contents
1. 목ì � ¶
Pythonì�„ ì�´ìš©í•´ì„œ Web Crawler를 ì œìž‘í•˜ë©´ì„œ Pythonì�˜ 사용법ì�„ ì�µížˆê³ , ì›�하는 웹 페ì�´ì§€ë¥¼ ê¸�기 위한 Web Crawler를 ì œìž‘í•œë‹¤. (네ì�´ë²„웹툰(ë�Œì•„온 ëŸí‚¤ì§±, ì‹ ì�˜ 탑...), 네ì�´ë²„ ìº�스트, ê·¸ 외ì�˜ ê°�종 웹페ì�´ì§€..)
2. í•„ìš”ê¸°ìˆ ¶
- HTML
- CSS
- JavaScript
- Python
HTML, CSS, JavaScript - 웹 페�지 분�
Python
3.1. 필요한 문서 ¶
- http://docs.python.org/
- http://hyogeun.tistory.com/107 - try, except.
3.2.1. 웹 페ì�´ì§€ 소스 ê¸�어오기 ¶
import urllib
import urllib2
req = urllib2.Request('http://9632024.tistory.com/974')
try: urllib2.urlopen(req)
except URLError, e:
print e.reason
fo = open("test1.html","w")
for line in urllib2.urlopen(req).readlines():
fo.write(line)
fo.close()
3.2.2. 소스ì—�서 URLë§Œ 추출하기 ¶
import urllib
import urllib2
import string
fo1 = open("test1.html", "r")
fo2 = open("test2.html", "w")
for line in fo1.readlines() :
pos = string.find(line, '"http')
if pos is not -1 :
for c in range(pos+1, len(line)) :
if line[c] is '"' :
fo2.write("\n")
break
fo2.write(line[c])
fo1.close()
fo2.close()
3.2.3. 파ì�¼ 다운로드하기 ¶
import urllib
import urllib2
fo = open("test2.html", "r")
for line in fo.readlines():
urllib.urlretrieve(line,line.split('/')[-1])
fo.close()
- http://www.wellho.net/resources/ex.php4?item=y108/bejo.py
- split
line = 'http://cfile23.uf.tistory.com/original/2001D2044C945F80495C6F'
line.split('/')-1 == '2001D2044C945F80495C6F'
line.split('/')-2 == 'original'
say = "This is a line of text"
part = line.split(' ')
part == 'This', 'is', 'a', 'line', 'of', 'text'
- swap
Python 2.7.2+ (default, Oct 4 2011, 20:03:08) [GCC 4.6.1] on linux2 Type "help", "copyright", "credits" or "license" for more information. >>> first = 1 >>> second = 2 >>> first, second = second, first >>> print first 2 >>> print second 1 >>> first, second = second, first >>> third = 3 >>> first, second, third = third, first, second >>> print first, second, third 3 1 2
- retrieve
urllib.urlretrieve(url[, filename[, reporthook[, data]]])
http://docs.python.org/library/urllib.html
3.2.4. ë””ë ‰í† ë¦¬ 만들기 ¶
import os os.chdir(os.getcwd() + '/folder') def create_dir(folder): cdir = os.getcwd() mdir = cdir + folder print mdir; if os.path.isdir(mdir) is False : os.mkdir(mdir , 0755) type = ['/mp3', '/jpg', '/txt'] for t in type : create_dir(t)
- os.chdir(path) - Change the current working directory to path.
- os.getcwd() - Return a string representing the current working directory.
- os.path.isdir(path) - Return True if path is an existing directory.
- os.mkdir(path, mode) - Create a directory named path with numeric mode mode. If the directory already exists, OSError is raised.
http://docs.python.org/library/os.html
http://docs.python.org/library/os.path.html#module-os.path
- mode -
d - ë””ë ‰í† ë¦¬ 구분
r - �기 권한
w - 쓰기 권한
x - 실행 권한
d / rwx / r-x / r-x
3.2.5. 웹툰(네ì�´ë²„) ê¸�기 ¶
import prepare import os currentpath = os.getcwd() path = os.getcwd() + '/luckyzzang' if os.path.isdir(path) is False : os.mkdir(path, 0755) os.chdir(path) currentpath = os.getcwd() for i in range(1, 21): url = 'http://comic.naver.com/webtoon/detail.nhn?titleId=449854&no=' + str(i) + '&weekday=wed' path = currentpath + '/' + str(i) if os.path.isdir(path) is False : os.mkdir(path, 0755) os.chdir(path) prepare.readpage(url, str(i) + '.html') prepare.extractwt(str(i) + '.html', str(i) + 'file.html') prepare.download(str(i) + 'file.html')
- prepare는 앞ì�˜ 4ê°œ 소스를 ì �ì ˆížˆ 조합한 python 파ì�¼
- 'http://comic.naver.com/webtoon/detail.nhn?titleId=449854&no=***&weekday=wed'
titleId는 웹툰 ID, 여기서는 ë�Œì•„온 ëŸí‚¤ì§±
no는 그 페�지가 몇 편�지
weekday는 어떤 요� 웹툰�지.
'http://comic.naver.com/webtoon/detail.nhn?titleId=449854&no=20&weekday=wed' ì�´ë©´ ëŸí‚¤ì§± 20편ì�´ë�¼ëŠ” 소리.
3.2.6. wxPython ¶
ìž�꾸 글ìž�ë§Œ 나오는 환경ì�„ 보니까 질리기 시작했다. ì•„ì§� 완성ì�€ 멀었지만 GUI로 ë§Œë“¤ì–´ë³´ë ¤ê³ í•¨. 어차피 나중ì—� 해야ë�˜ë‹ˆê¹Œ...
- 설치 - apt-get install python-wxgtk2.8
- 문서 - http://wiki.wxpython.org/How to Learn wxPython
- Gtk-WARNING **: 모듈� module_path�서 찾� 수 없습니다: "pixmap"
sudo apt-get install libgtkglextmm-x11-1.2-dev
sudo apt-get install gtk2-engines-pixbuf
http://stackoverflow.com/questions/8046667/vpython-error-on-ubuntu-11-10
3.2.7. pywin32 ¶
- http://sourceforge.net/projects/pywin32/
- 2.7ë²„ì „.
3.2.8. Eclipse + PyDev + wxPython + pywin32 ¶
1. Eclipse 설치
2. Eclipse�서, Help > Install New Software > Add > PyDev, Http://pydev.org/updates
3. 중간ì—� Aptana ì‹ ë¢°í•˜ëŠ”ì§€ì—� ë�™ì�˜, 재시작
4. Window > Preference > PyDev > Interpreter - Python > Auto Config
5. New Folder > /usr/lib/python2.7/dist-packages/wx-2.8-gtk2-unicode
2. Eclipse�서, Help > Install New Software > Add > PyDev, Http://pydev.org/updates
3. 중간ì—� Aptana ì‹ ë¢°í•˜ëŠ”ì§€ì—� ë�™ì�˜, 재시작
4. Window > Preference > PyDev > Interpreter - Python > Auto Config
5. New Folder > /usr/lib/python2.7/dist-packages/wx-2.8-gtk2-unicode
- 사실 ë�¼ì�´ë¸ŒëŸ¬ë¦¬ 다 ê¹”ì•„ë†“ê³ Auto Config하면 ë�¨..
3.3. ê°œì„ í•´ì•¼ í• ì � ¶
- 파ì�¼ì�„ ì €ìž¥í• ë–„ 소스 파ì�¼ì—� ì €ìž¥ì�´ ë�˜ì„œ ì§€ì €ë¶„í•˜ë‹¤. o
- 필요하지 않ì�€ 파ì�¼ê¹Œì§€ ì „ë¶€ ê¸�어온다. o
- ë‚´ê°€ ì�¸í„°ë„·ì—�서 jpg 파ì�¼ ê¸�ì–´ì˜¤ë ¤ê³ ë§Œë“ íŒŒì�´ì�¬ 코드 있는ë�° 혹시 필요함? - 서민관
- 그러면 매우 ê°�ì‚¬í•˜ì£ ã…Žã…Ž - ê¶Œì˜�기
- 그러면 매우 ê°�ì‚¬í•˜ì£ ã…Žã…Ž - ê¶Œì˜�기
- http://git-scm.com/ - Git.
- http://wiki.kldp.org/wiki.php/SubversionBook/BranchingAndMerging
- http://www.crummy.com/software/BeautifulSoup/ - 웹분� ��브러리 BeautifulSoup �용해보기
- wxpython
- http://docs.python.org/tutorial/classes.html / 9.5까지.
- 로그ì�¸ì�´ 가능한 페ì�´ì§€ë¥¼ ê¸�ê³ ì‹¶ì�€ë�° 방법ì�„ ëª¨ë¥´ê² ë‹¤.
- êµ³ì�´ Python으로 GUI를 구현해야 í• í•„ìš”ì„±ì�„ ëª¨ë¥´ê² ë‹¤.
- �롤러�니.. �. -태진










