Debugging

Using the logging module

The easiest way to see what is going on is to enable DEBUG logging messages. Write the following code at every entry point to your program:

>>> import logging
>>> logging.basicConfig(level=logging.DEBUG)

That logging configuration will output all logging messages to console, not just from Grab but from other modules too. If you are interested only in Grab’s messages:

>>> import logging
>>> logger = logging.getLogger('grab')
>>> logger.addHandler(logging.StreamHandler())
>>> logger.setLevel(logging.DEBUG)

You can also use a default_logging function that configures logging as follows:

  • all messages of any level except from Grab modules are printed to console
  • all “grab*” messages with level INFO or higher are printed to console
  • all “grab*” messages of any level are saved to /tmp/grab.log
  • all “grab.network*” messages (usually these are URLs being requested) of any level are saved to /tmp/grab.network.log

Usage of default_logging function is simple:

>>> from weblib.logs import default_logging
>>> default_logging()

Logging messages about network request

For each network request, Grab generates the “grab.network” logging message with level DEBUG. Let’s look at an example:

[5864] GET http://www.kino-govno.com/movies/rusichi via 188.120.244.68:8080 proxy of type http with authorization

We can see the requested URL and also that request has ID 5864, that the HTTP method is GET, and that the request goes through a proxy with authorization. For each network request Grab uses the next ID value from the sequence that is shared by all Grab instances. That does mean that even different Grab instances will generates network logging messages with unique ID.

You can also turn on logging of POST request content. Use the debug_post option:

>>> g.setup(debug_post=True)

The output will be like this:

[01] POST http://yandex.ru
POST request:
foo                      : bar
name                     : Ivan

Saving the content of requests and responses

You can ask Grab to save the content of each network response to the file located at the path passed as the log_file option:

>>> g.setup(log_file='log.html')

Of course, each new response will overwrite the content of the previous response.

If you want to log all traffic, then consider using the log_dir option, which tells Grab to save the contents of all responses to files inside the specified directory. Note that each such file will contain a request ID in its filename. For each response, there will be two files: XXX.log and XXX.html. The file XXX.html contains the raw response. Even if you requested an image or large movie, you’ll get its raw content in that file. The file XXX.log contains headers of network response. If you configure Grab with debug=True, the file XXX.log will also contain request headers.