megfile.http module

megfile.http.get_http_session(timeout: int = 10, status_forcelist: Iterable[int] = (502, 503, 504)) → requests.sessions.Session[source]
megfile.http.is_http(path: Union[str, os.PathLike]) → bool[source]

http scheme definition: http(s)://domain/path

Parameters

path – Path to be tested

Returns

True if path is http url, else False

megfile.http.http_open(path: Union[str, os.PathLike], mode: str = 'rb', *, encoding: Optional[str] = None, errors: Optional[str] = None, max_concurrency: Optional[int] = None, max_buffer_size: int = 134217728, forward_ratio: Optional[float] = None, block_size: int = 8388608, **kwargs) → Union[_io.BufferedReader, megfile.lib.http_prefetch_reader.HttpPrefetchReader][source]

Open a BytesIO to read binary data of given http(s) url

Note

Essentially, it reads data of http(s) url to memory by requests, and then return BytesIO to user.

Parameters

  • path – Given path

  • mode – Only supports ‘rb’ mode now

  • encoding – encoding is the name of the encoding used to decode or encode the file. This should only be used in text mode.

  • errors – errors is an optional string that specifies how encoding and decoding errors are to be handled—this cannot be used in binary mode.

  • max_concurrency – Max download thread number, None by default

  • max_buffer_size – Max cached buffer size in memory, 128MB by default

  • block_size – Size of single block, 8MB by default. Each block will be uploaded or downloaded by single thread.

Returns

BytesIO initialized with http(s) data

megfile.http.http_stat(path: Union[str, os.PathLike], follow_symlinks=True) → megfile.pathlike.StatResult[source]

Get StatResult of http_url response, including size and mtime, referring to http_getsize and http_getmtime

Parameters

  • path – Given path

  • follow_symlinks – Ignore this parameter, just for compatibility

Returns

StatResult

Raises

HttpPermissionError, HttpFileNotFoundError

megfile.http.http_getsize(path: Union[str, os.PathLike], follow_symlinks: bool = False) → int[source]

Get file size on the given http_url path.

If http response header don’t support Content-Length, will return None

Parameters

  • path – Given path

  • follow_symlinks – Ignore this parameter, just for compatibility

Returns

File size (in bytes)

Raises

HttpPermissionError, HttpFileNotFoundError

megfile.http.http_getmtime(path: Union[str, os.PathLike], follow_symlinks: bool = False) → float[source]

Get Last-Modified time of the http request on the given http_url path.

If http response header don’t support Last-Modified, will return None

Parameters

  • path – Given path

  • follow_symlinks – Ignore this parameter, just for compatibility

Returns

Last-Modified time (in Unix timestamp format)

Raises

HttpPermissionError, HttpFileNotFoundError

megfile.http.http_exists(path: Union[str, os.PathLike], followlinks: bool = False) → bool[source]

Test if http path exists

Parameters

  • path – Given path

  • followlinks (bool, optional) – ignore this parameter, just for compatibility

Returns

return True if exists

Return type

bool