grab.document

The Document class is the result of network request made with Grab instance.

Module Contents

Classes

FormRequestParams	dict() -> new empty dictionary
Document	Network response.

Functions

normalize_pairs(→ collections.abc.Sequence[tuple[str, ...)

Attributes

THREAD_STORAGE
logger
UNDEFINED

grab.document.THREAD_STORAGE

grab.document.logger

grab.document.UNDEFINED

class grab.document.FormRequestParams

Bases: TypedDict

dict() -> new empty dictionary dict(mapping) -> new dictionary initialized from a mapping object’s

(key, value) pairs

dict(iterable) -> new dictionary initialized as if via:

d = {} for k, v in iterable:

d[k] = v

dict(**kwargs) -> new dictionary initialized with the name=value pairs

in the keyword argument list. For example: dict(one=1, two=2)

url: str

method: str

multipart: bool

fields: collections.abc.Sequence[tuple[str, Any]]

grab.document.normalize_pairs(inp: collections.abc.Sequence[tuple[str, Any]] | collections.abc.Mapping[str, Any]) → collections.abc.Sequence[tuple[str, Any]]

class grab.document.Document(body: bytes, *, document_type: None | str = 'html', head: None | bytes = None, headers: None | email.message.Message = None, encoding: None | str = None, code: None | int = None, url: None | str = None, cookies: None | collections.abc.Sequence[http.cookiejar.Cookie] = None)

Bases: grab.base.BaseResponse

Network response.

property status: None | int

property json: Any

Return response body deserialized into JSON object.

property pyquery: Any

Return pyquery handler.

property body: bytes

property tree: lxml.etree._Element

Return DOM tree of the document built with HTML DOM builder.

property form: lxml.html.FormElement

Return default document’s form.

If form was not selected manually then select the form which has the biggest number of input elements.

The form value is just an lxml.html form element.

Example:

g.request('some URL')
# Choose form automatically
print g.form

# And now choose form manually
g.choose_form(1)
print g.form

__slots__ = ('document_type', 'code', 'head', 'headers', 'url', 'cookies', 'encoding', '_bytes_body',...

__call__(query: str) → selection.SelectorList[lxml.etree._Element]

select(*args: Any, **kwargs: Any) → selection.SelectorList[lxml.etree._Element]

process_encoding(encoding: None | str = None) → str

Process explicitly defined encoding or auto-detect it.

If encoding is explicitly defined, ensure it is a valid encoding the python can deal with. If encoding is not specified, auto-detect it.

Raises unicodec.InvalidEncodingName if explicitly set encoding is invalid.

copy() → Document

save(path: str) → None

Save response body to file.

url_details() → urllib.parse.SplitResult

Return result of urlsplit function applied to response url.

query_param(key: str) → str

Return value of parameter in query string.

browse() → None

Save response in temporary file and open it in GUI browser.

__getstate__() → collections.abc.Mapping[str, Any]

Reset cached lxml objects which could not be pickled.

__setstate__(state: collections.abc.Mapping[str, Any]) → None

text_search(anchor: str | bytes) → bool

Search the substring in response body.

Parameters

• anchor – string to search

• byte – if False then anchor should be the unicode string, and search will be performed in response.unicode_body() else anchor should be the byte-string and search will be performed in response.body

If substring is found return True else False.

text_assert(anchor: str | bytes) → None

If anchor is not found then raise DataNotFound exception.

text_assert_any(anchors: collections.abc.Sequence[str | bytes]) → None

If no anchors were found then raise DataNotFound exception.

rex_text(regexp: str | bytes | re.Pattern[str] | re.Pattern[bytes], flags: int = 0, default: Any = UNDEFINED) → Any

Return content of first matching group of regexp found in response body.

rex_search(regexp: str | bytes | re.Pattern[str] | re.Pattern[bytes], flags: int = 0, default: Any = UNDEFINED) → Any

Search the regular expression in response body.

Return found match object or None

rex_assert(rex: str | bytes | re.Pattern[str] | re.Pattern[bytes]) → None

Raise DataNotFound exception if rex expression is not found.

get_body_chunk() → bytes

unicode_body() → str

Return response body as unicode string.

classmethod wrap_io(inp: bytes | str) → io.StringIO | io.BytesIO

classmethod _build_dom(content: bytes | str, mode: str, encoding: str) → lxml.etree._Element

build_html_tree() → lxml.etree._Element

build_xml_tree() → lxml.etree._Element

choose_form(number: None | int = None, xpath: None | str = None, name: None | str = None, **kwargs: Any) → None

Set the default form.

Parameters

• number – number of form (starting from zero)

• id – value of “id” attribute

• name – value of “name” attribute

• xpath – XPath query

Raises

DataNotFound if form not found

Raises

GrabMisuseError if method is called without parameters

Selected form will be available via form attribute of Grab instance. All form methods will work with default form.

Examples:

# Select second form
g.choose_form(1)

# Select by id
g.choose_form(id="register")

# Select by name
g.choose_form(name="signup")

# Select by xpath
g.choose_form(xpath='//form[contains(@action, "/submit")]')

get_cached_form() → lxml.html.FormElement

Get form which has been already selected.

Returns None if form has not been selected yet.

It is for testing mainly. To not trigger pylint warnings about accessing protected element.

set_input(name: str, value: Any) → None

Set the value of form element by its name attribute.

Parameters

• name – name of element

• value – value which should be set to element

To check/uncheck the checkbox pass boolean value.

Example:

g.set_input('sex', 'male')

# Check the checkbox
g.set_input('accept', True)

set_input_by_id(_id: str, value: Any) → None

Set the value of form element by its id attribute.

Parameters

• _id – id of element

• value – value which should be set to element

set_input_by_number(number: int, value: Any) → None

Set the value of form element by its number in the form.

Parameters

• number – number of element

• value – value which should be set to element

set_input_by_xpath(xpath: str, value: Any) → None

Set the value of form element by xpath.

Parameters

• xpath – xpath path

• value – value which should be set to element

process_extra_post(post_items: list[tuple[str, Any]], extra_post_items: collections.abc.Sequence[tuple[str, Any]]) → list[tuple[str, Any]]

clean_submit_controls(post: collections.abc.MutableMapping[str, Any], submit_name: None | str) → None

get_form_request(submit_name: None | str = None, url: None | str = None, extra_post: None | collections.abc.Mapping[str, Any] | collections.abc.Sequence[tuple[str, Any]] = None, remove_from_post: None | collections.abc.Sequence[str] = None) → FormRequestParams

Submit default form.

Parameters

• submit_name – name of button which should be “clicked” to submit form

• url – explicitly specify form action url

• extra_post – (dict or list of pairs) additional form data which will override data automatically extracted from the form.

• remove_from_post – list of keys to remove from the submitted data

Following input elements are automatically processed:

• input[type=”hidden”] - default value

• select: value of last option

• radio - ???

• checkbox - ???

Multipart forms are correctly recognized by grab library.

build_fields_to_remove(fields: collections.abc.Mapping[str, Any], form_inputs: collections.abc.Sequence[lxml.html.HtmlElement]) → set[str]

process_form_fields(fields: collections.abc.MutableMapping[str, Any]) → None

form_fields() → collections.abc.MutableMapping[str, lxml.html.HtmlElement]

Return fields of default form.

Fill some fields with reasonable values.

choose_form_by_element(xpath: str) → None