Python: how to get rid of BeautifulSoup user warning

Question

When I run the following 'BeautifulSoup' code, I get a warning message. How can I fix the warning message?

request = urllib.request.Request(url, None, headers)
response = urllib.request.urlopen(request)
soup = BeautifulSoup(response)

The UserWarning is as follows:

UserWarning: No parser was explicitly specified, so I'm using the best available HTML parser for this system ("lxml"). This usually isn't a problem, but if you run this code on another system, or in a different virtual environment, it may use a different parser and behave differently.

The code that caused this warning is on line 33 of the file test.py. To get rid of this warning, pass the additional argument 'features="lxml"' to the BeautifulSoup constructor.

Answer 1

You have to mention an HTML parser with BeautifulSoup. You can use either 'lxml' or 'html5lib'.

Make the following change in your code and it should work.

request = urllib.request.Request(url, None, headers)
response = urllib.request.urlopen(request)
soup = BeautifulSoup(response, 'html5lib')

If 'lxml' or 'html5lib' is not installed on your system, you can install using the following commands:

$ pip install html5lib
$ pip install lxml

You can find details about BeautifulSoup here.