grab.document
¶
The Document class is the result of network request made with Grab instance.
Module Contents¶
Classes¶
dict() -> new empty dictionary |
|
Network response. |
Functions¶
|
Attributes¶
- class grab.document.FormRequestParams[source]¶
Bases:
TypedDict
dict() -> new empty dictionary dict(mapping) -> new dictionary initialized from a mapping object’s
(key, value) pairs
- dict(iterable) -> new dictionary initialized as if via:
d = {} for k, v in iterable:
d[k] = v
- dict(**kwargs) -> new dictionary initialized with the name=value pairs
in the keyword argument list. For example: dict(one=1, two=2)
- grab.document.normalize_pairs(inp: collections.abc.Sequence[tuple[str, Any]] | collections.abc.Mapping[str, Any]) collections.abc.Sequence[tuple[str, Any]] [source]¶
- class grab.document.Document(body: bytes, *, document_type: None | str = 'html', head: None | bytes = None, headers: None | email.message.Message = None, encoding: None | str = None, code: None | int = None, url: None | str = None, cookies: None | collections.abc.Sequence[http.cookiejar.Cookie] = None)[source]¶
Bases:
grab.base.BaseResponse
Network response.
- property tree: lxml.etree._Element[source]¶
Return DOM tree of the document built with HTML DOM builder.
- property form: lxml.html.FormElement[source]¶
Return default document’s form.
If form was not selected manually then select the form which has the biggest number of input elements.
The form value is just an lxml.html form element.
Example:
g.request('some URL') # Choose form automatically print g.form # And now choose form manually g.choose_form(1) print g.form
- __slots__ = ('document_type', 'code', 'head', 'headers', 'url', 'cookies', 'encoding', '_bytes_body',...[source]¶
- process_encoding(encoding: None | str = None) str [source]¶
Process explicitly defined encoding or auto-detect it.
If encoding is explicitly defined, ensure it is a valid encoding the python can deal with. If encoding is not specified, auto-detect it.
Raises unicodec.InvalidEncodingName if explicitly set encoding is invalid.
- url_details() urllib.parse.SplitResult [source]¶
Return result of urlsplit function applied to response url.
- __getstate__() collections.abc.Mapping[str, Any] [source]¶
Reset cached lxml objects which could not be pickled.
- text_search(anchor: str | bytes) bool [source]¶
Search the substring in response body.
- Parameters
anchor – string to search
byte – if False then anchor should be the unicode string, and search will be performed in response.unicode_body() else anchor should be the byte-string and search will be performed in response.body
If substring is found return True else False.
- text_assert(anchor: str | bytes) None [source]¶
If anchor is not found then raise DataNotFound exception.
- text_assert_any(anchors: collections.abc.Sequence[str | bytes]) None [source]¶
If no anchors were found then raise DataNotFound exception.
- rex_text(regexp: str | bytes | re.Pattern[str] | re.Pattern[bytes], flags: int = 0, default: Any = UNDEFINED) Any [source]¶
Return content of first matching group of regexp found in response body.
- rex_search(regexp: str | bytes | re.Pattern[str] | re.Pattern[bytes], flags: int = 0, default: Any = UNDEFINED) Any [source]¶
Search the regular expression in response body.
Return found match object or None
- rex_assert(rex: str | bytes | re.Pattern[str] | re.Pattern[bytes]) None [source]¶
Raise DataNotFound exception if rex expression is not found.
- classmethod _build_dom(content: bytes | str, mode: str, encoding: str) lxml.etree._Element [source]¶
- choose_form(number: None | int = None, xpath: None | str = None, name: None | str = None, **kwargs: Any) None [source]¶
Set the default form.
- Parameters
number – number of form (starting from zero)
id – value of “id” attribute
name – value of “name” attribute
xpath – XPath query
- Raises
DataNotFound
if form not found- Raises
GrabMisuseError
if method is called without parameters
Selected form will be available via form attribute of Grab instance. All form methods will work with default form.
Examples:
# Select second form g.choose_form(1) # Select by id g.choose_form(id="register") # Select by name g.choose_form(name="signup") # Select by xpath g.choose_form(xpath='//form[contains(@action, "/submit")]')
- get_cached_form() lxml.html.FormElement [source]¶
Get form which has been already selected.
Returns None if form has not been selected yet.
It is for testing mainly. To not trigger pylint warnings about accessing protected element.
- set_input(name: str, value: Any) None [source]¶
Set the value of form element by its name attribute.
- Parameters
name – name of element
value – value which should be set to element
To check/uncheck the checkbox pass boolean value.
Example:
g.set_input('sex', 'male') # Check the checkbox g.set_input('accept', True)
- set_input_by_id(_id: str, value: Any) None [source]¶
Set the value of form element by its id attribute.
- Parameters
_id – id of element
value – value which should be set to element
- set_input_by_number(number: int, value: Any) None [source]¶
Set the value of form element by its number in the form.
- Parameters
number – number of element
value – value which should be set to element
- set_input_by_xpath(xpath: str, value: Any) None [source]¶
Set the value of form element by xpath.
- Parameters
xpath – xpath path
value – value which should be set to element
- process_extra_post(post_items: list[tuple[str, Any]], extra_post_items: collections.abc.Sequence[tuple[str, Any]]) list[tuple[str, Any]] [source]¶
- clean_submit_controls(post: collections.abc.MutableMapping[str, Any], submit_name: None | str) None [source]¶
- get_form_request(submit_name: None | str = None, url: None | str = None, extra_post: None | collections.abc.Mapping[str, Any] | collections.abc.Sequence[tuple[str, Any]] = None, remove_from_post: None | collections.abc.Sequence[str] = None) FormRequestParams [source]¶
Submit default form.
- Parameters
submit_name – name of button which should be “clicked” to submit form
url – explicitly specify form action url
extra_post – (dict or list of pairs) additional form data which will override data automatically extracted from the form.
remove_from_post – list of keys to remove from the submitted data
Following input elements are automatically processed:
input[type=”hidden”] - default value
select: value of last option
radio - ???
checkbox - ???
Multipart forms are correctly recognized by grab library.
- build_fields_to_remove(fields: collections.abc.Mapping[str, Any], form_inputs: collections.abc.Sequence[lxml.html.HtmlElement]) set[str] [source]¶