Work with HTTP Headers¶
Custom headers¶
If you need to submit custom HTTP headers, you can specify any number of them via headers option. A common case is to emulate an AJAX request:
>>> g = Grab()
>>> g.setup(headers={'X-Requested-With': 'XMLHttpRequest'})
Bear in mind, that except headers in headers option (that is empty by default) Grab also generates a bunch of headers to emulate a typical web browser. At the moment of writing these docs these headers are:
- Accept
- Accept-Language
- Accept-Charset
- Keep-Alive
- Except
If you need to change one of these headers, you can override its value with the headers option. You can also subclass the Grab class and define your own common_headers method to completely override the logic of generating these extra headers.
User-Agent header¶
By default, for each request Grab randomly chooses one user agent from a builtin list of real user agents. You can specify the exact User-Agent value with the user_agent option. If you need to randomly choose user agents from your own list of user agents, then you can put your list into a text file and pass its location as user_agent_file.
Referer header¶
To specify the content of the Referer header, use the referer option. By default, Grab use the URL of previously request document as value of Referer header. If you do not like this behaviour, you can turn it off with reuse_referer option.