如何使用Python通过HTTP下载文件？

在wb中open('test.mp3','wb')打开一个文件(并清除所有现有文件)以二进制模式,所以你可以用它来代替刚才保存的文本数据.

Python 3

urllib.request.urlopen

import urllib.request
response = urllib.request.urlopen('http://www.example.com/')
html = response.read()

urllib.request.urlretrieve

import urllib.request
urllib.request.urlretrieve('http://www.example.com/songs/mp3.mp3', 'mp3.mp3')

Python 2

urllib2.urlopen(谢谢科里)

import urllib2
response = urllib2.urlopen('http://www.example.com/')
html = response.read()

urllib.urlretrieve(感谢PabloG)

import urllib
urllib.urlretrieve('http://www.example.com/songs/mp3.mp3', 'mp3.mp3')

使用wget模块:

import wget
wget.download('url')

Python 2/3的PabloG代码的改进版本:

#!/usr/bin/env python
# -*- coding: utf-8 -*-
from __future__ import ( division, absolute_import, print_function, unicode_literals )

import sys, os, tempfile, logging

if sys.version_info >= (3,):
    import urllib.request as urllib2
    import urllib.parse as urlparse
else:
    import urllib2
    import urlparse

def download_file(url, dest=None):
    """ 
    Download and save a file specified by url to dest directory,
    """
    u = urllib2.urlopen(url)

    scheme, netloc, path, query, fragment = urlparse.urlsplit(url)
    filename = os.path.basename(path)
    if not filename:
        filename = 'downloaded.file'
    if dest:
        filename = os.path.join(dest, filename)

    with open(filename, 'wb') as f:
        meta = u.info()
        meta_func = meta.getheaders if hasattr(meta, 'getheaders') else meta.get_all
        meta_length = meta_func("Content-Length")
        file_size = None
        if meta_length:
            file_size = int(meta_length[0])
        print("Downloading: {0} Bytes: {1}".format(url, file_size))

        file_size_dl = 0
        block_sz = 8192
        while True:
            buffer = u.read(block_sz)
            if not buffer:
                break

            file_size_dl += len(buffer)
            f.write(buffer)

            status = "{0:16}".format(file_size_dl)
            if file_size:
                status += "   [{0:6.2f}%]".format(file_size_dl * 100 / file_size)
            status += chr(13)
            print(status, end="")
        print()

    return filename

if __name__ == "__main__":  # Only run if this file is called directly
    print("Testing with 10MB download")
    url = "http://download.thinkbroadband.com/10MB.zip"
    filename = download_file(url)
    print(filename)

图书馆提供简单而Python 2 & Python 3兼容的方式six:

from six.moves import urllib
urllib.request.urlretrieve("http://www.example.com/songs/mp3.mp3", "mp3.mp3")

import os,requests
def download(url):
    get_response = requests.get(url,stream=True)
    file_name  = url.split("/")[-1]
    with open(file_name, 'wb') as f:
        for chunk in get_response.iter_content(chunk_size=1024):
            if chunk: # filter out keep-alive new chunks
                f.write(chunk)


download("https://example.com/example.jpg")

为此目的在纯Python中编写了wget库.从版本2.0 开始urlretrieve,它就具备了这些功能.

我同意Corey,urllib2比urllib更完整,如果你想做更复杂的事情,应该是使用的模块,但为了使答案更加完整,如果你只想要基础知识,urllib是一个更简单的模块:

import urllib
response = urllib.urlopen('http://www.example.com/sound.mp3')
mp3 = response.read()

会工作得很好.或者,如果您不想处理"响应"对象,可以直接调用read():

import urllib
mp3 = urllib.urlopen('http://www.example.com/sound.mp3').read()

以下是在python中下载文件最常用的调用:

urllib.urlretrieve ('url_to_file', file_name)

urllib2.urlopen('url_to_file')

requests.get(url)

wget.download('url', file_name)

注意:urlopen并且urlretrieve发现下载大文件(大小> 500 MB)时性能相对较差.requests.get将文件存储在内存中,直到下载完成.

您也可以通过urlretrieve获取进度反馈:

def report(blocknr, blocksize, size):
    current = blocknr*blocksize
    sys.stdout.write("\r{0:.2f}%".format(100.0*current/size))

def downloadFile(url):
    print "\n",url
    fname = url.split('/')[-1]
    print fname
    urllib.urlretrieve(url, fname, report)

如果安装了wget,则可以使用parallel_sync.

pip install parallel_sync

from parallel_sync import wget
urls = ['http://something.png', 'http://somthing.tar.gz', 'http://somthing.zip']
wget.download('/tmp', urls)
# or a single file:
wget.download('/tmp', urls[0], filenames='x.zip', extract=True)

Doc:https: //pythonhosted.org/parallel_sync/pages/examples.html

这非常强大.它可以并行下载文件,在失败时重试,甚至可以在远程机器上下载文件.

在python3中,您可以使用urllib3和shutil libraires.使用pip或pip3下载它们(取决于python3是否默认)

pip3 install urllib3 shutil

然后运行此代码

import urllib.request
import shutil

url = "http://www.somewebsite.com/something.pdf"
output_file = "save_this_name.pdf"
with urllib.request.urlopen(url) as response, open(output_file, 'wb') as out_file:
    shutil.copyfileobj(response, out_file)

请注意,您下载urllib3但urllib在代码中使用

仅出于完整性考虑，也可以使用该subprocess软件包调用任何程序来检索文件。专用于检索文件的程序比Python函数（如）强大urlretrieve。例如，wget可以递归下载目录（-R），可以处理FTP，重定向，HTTP代理，可以避免重新下载现有文件（-nc），并且aria2可以进行多连接下载，从而有可能加快下载速度。

import subprocess
subprocess.check_output(['wget', '-O', 'example_output_file.html', 'https://example.com'])

在Jupyter Notebook中，还可以使用以下!语法直接调用程序：

!wget -O example_output_file.html https://example.com