20 - web scraping with python using beautiful soup & requests (Python tutorial for beginners 2019)
# Web Scraping with Python: A Step-by-Step Guide Using Beautiful Soup and Requests
In this video tutorial, we will guide you through the process of web scraping using Python. This is an essential skill for both beginners and professional developers. We will demonstrate how to scrape weather forecast data from a website and store it in a CSV file or a pandas DataFrame.
## Inspired by Dataquest.io: A Fun Project
This tutorial was inspired by a Dataquest.io tutorial on weather forecasting. Instead of focusing on general weather forecasts, we decided to make it more specific by scraping weather data for a lake where one of the creators resides. We will be using the website [forecast.weather.gov](https://www.weather.gov/) for our project.
## Getting Started: Setting Up the Environment
Before diving into the code, ensure you have the necessary libraries installed:
1. **Beautiful Soup**: A Python library for parsing HTML and XML documents.
2. **Requests**: A Python HTTP client library to make requests to web pages.
3. **Pandas**: An open-source data analysis tool that will help us store the scraped data in a structured format.
Run the following commands to install these libraries:
```bash
pip install beautifulsoup4
pip install requests
pip install pandas
```
## Inspecting the Web Page
Using Chrome DevTools, we can inspect the structure of the web page. The goal is to locate the part of the HTML that contains the weather forecast data.
1. Right-click on the webpage and select "Inspect" (or use the shortcut `Ctrl + Shift + C` for Windows or `Command + Shift + C` for Mac).
2. Hover over different elements to see their HTML structure.
3. Look for the container holding the 7-day weather forecast, which has an ID of `seven-day-forecast-body`.
## Writing the Code
### Importing Libraries
```python
import requests
from bs4 import BeautifulSoup
import pandas as pd
```
### Making HTTP Requests and Parsing HTML
```python
# Make a request to the website
page = requests.get("https://www.weather.gov/losangeles")
# Create a Beautiful Soup object
soup = BeautifulSoup(page.content, 'html.parser')
# Print out the soup object (HTML content)
print(soup)
```
### Extracting Specific Data
```python
# Find the container with the 7-day forecast
week = soup.find(id="seven-day-forecast-body")
# Get all items within the container
items = week.find_all(class_="tombstone-container")
```
### Accessing Individual Elements
```python
# Extract period names, descriptions, and temperatures
period_names = [item.find(class_="period-name").text for item in items]
short_descriptions = [item.find(class_="short-description").text for item in items]
temperatures = [item.find(class_="temp").text for item in items]
print(period_names)
print(short_descriptions)
print(temperatures)
```
## Using Pandas to Structure Data
```python
# Create a DataFrame using pandas
weather_data = {
"Period": period_names,
"Description": short_descriptions,
"Temperature": temperatures
}
df = pd.DataFrame(weather_data)
# Print the DataFrame
print(df)
```
### Exporting Data to CSV
```python
# Save the DataFrame to a CSV file
df.to_csv("weather.csv", index=False)
```
## Example: Scraping Weather Data for Chicago
If you want to scrape weather data for a different location, simply replace the URL with the city's forecast page (e.g., `https://www.weather.gov/chicago`).
## Conclusion
This tutorial demonstrates how powerful web scraping can be. By using Python libraries like Beautiful Soup and Requests, we can extract valuable data from websites and organize it into structured formats like CSV or pandas DataFrames.
Stay tuned for more tutorials in our series on automation with Python. If you're interested in learning more about automation and data science, check out our upcoming course: *How to Automate Stuff with Python*.
"WEBVTTKind: captionsLanguage: enhey you in this video we are going to be doing web scraping with Python using libraries like beautifulsoup and request this is something that is very useful whether you are a beginner or a professional developer these are essential skills that almost every person needs so let's get started it's gonna be a fun little project and this was inspired by data quest io it's not a sponsored video or anything but I looked at one of their tutorials and they're doing a weather forecast I was like hey we should also just do that weather forecast except we're gonna be doing it for a lake that's where I live so we're gonna go to forecast dot weather.gov so that's number one let me zoom in here so you guys can see very clearly and I'm gonna type in Los Angeles will hit go and this is what pops up so what we're trying to do here is we want to pull this data that we want to put it into an Excel spreadsheet we you could also put it in pandas dataframe if you know what that is but most people know what an Excel spreadsheet is or what a CSV file is so that's essentially what we're going to be doing and we're gonna be doing web scraping here it's a pretty cool thing to learn how to do so let's jump right into it first of all for really basic stuff you know you have to know basic HTML we're not gonna cover too much of HTML or CSS in this video but you know for example if I write an h2 tag I have a code pen open here so I can just show how HTML works if I do h2 and I say hey guys like that him I do this it'll type in hey guys knife I do h6 tag right it'll make it smaller so these are H a.m. this is HTML this is basics very very basic basic use of it and now what we're gonna do is because I'm on Chrome we're gonna hit inspect here and dive deep using the chrome dev tools it's a very powerful way to learn the structure of the website that you want to scrape you want to abuse this tool okay use this tool a lot the chrome dev tools so now I'm gonna hover over and essentially what I want to do is I want to be able to get this week's forecast so I want to be able to find out what the weather is tonight Sunday Sunday night Monday and you know I also want to get the descriptions of it and I want to get what's the low and what's the high be able to put it in my Excel spreadsheet so let's do it so first of all I want you to click this button over here okay the shortcut for this is command shift C on a Mac or you can just come over here click this allows you to highlight over each thing and then it tells you what element it is of HTML or CSS file so he goes okay this is an image element as you can see it says I am g dot you know something so that dot pull left is a CSV clasp or sorry CSS class so let's go over here and check this out so it looks like we have a container here but this has an ID we have a div here and this has the ID of seven-day forecast body which is very good which is exactly what we need to leverage here so let's go ahead and use this because we know that once we have access to this we will have access to essentially everything else that's within this div so let's go to our code now you're gonna need a few modules installed and I'm not gonna walk through the full process of installing them because again those could turn into their own videos so let's go here and first of all we you need to install pip install beautiful soup 4 so that's step one another thing you're gonna need is pip install requests that's how you're gonna make HTTP requests to the websites and then you want to install the last one which is pandas all right because I've already installed them for me it says requirement already satisfied for you it's gonna take you through the installation process if you run into errors with installation process look up how to overcome them then once you solve it come back to this tutorial so once this is installed let's start writing some code so first of all we're gonna do important requests and I'm gonna also say from bs4 import beautiful soup like that alright and now essentially what I want to do is request this URL so I'm just gonna copy this URL right here I'll go here and I'll say this page is equal to request dot get and then I'll pop in that URL right there we'll also create a soup object out of that page so that requests thing that we just got we're gonna use that to create a beautiful soup object and what's beautiful soup allows you to do is makes the web scraping of the page really really easy it gives you nice structures it's able to find classes it's able to find paragraph tags it's able to find headers and it understands CSS and it understands HTML so it makes your life a lot easier that's why we're using beautifulsoup so let's go here in soup and go beautiful soup alright and we'll say page dot content and then we will say and we'll give it HTML parser okay so it'll note that it's HTML and so it needs to parse it as an HTML page alright and if I print this we should see something pretty weird so I'm gonna run this piece of code I'm gonna go here hit run all right so it says couldn't find a tree builder the features you requested HTML parser do you need to install a parser library Oh yep HTML dot parser not a - parser so we have now let's try it again we will run the code ok so looks like it worked and as you can see the soup object is all of the HTML code that's on that page it's pretty much the source code of that page so if I go and I right-click and I go I go view page source it's essentially this whole thing alright so it has access to all of it but what's cool is that I can do things like that find all and I can go find all of like the a tags you know find all a tags for example and if I run it it's essentially finding all of the links so it's finding the n-o a a.gov link that's there it's finding the weather.gov link that's on that page it's finding the Commerce gov link that's on that page so these all these links is able to find them right if I do not find images or if I do not find all you know short descriptions it'll find those as well so that's what's cool about beautiful soup makes your life really easy so let's go ahead and what we're gonna do is we're gonna create a variable that has this entire div stored inside of it that way we'll have access to this this div is what I'm talking about we'll have access to that div at all times so let's go to our code here and I'll say week is equal to soup dot find and we'll give it the ID in the ID what is the ID here the ID is seven-day forecast body so because I don't want to make a mistake I'm gonna click here once I click here you can see at the bottom it's highlighted this line I'll go to the ID and I'll double-click I will copy this I'll come back in here paste it so I'll make sure that this way there are no spelling errors as a programmer I know sometimes you want to do things quick and just type it out but the reason why you want to paste is not so you're faster but the real reason why you want to copy paste as much as you can is so you're accurate and you don't make mistakes that's why I took the time to copy it even if it was small like it was three letters I would still copy and paste because it has happened to me so many times that I'll make like a tiny little mistake like the one I made up here with HTML dot parser where I wrote a - here alright so we have this access to this week let's first of all print it out let's see what it shows us so we'll run our code alright so basically now what it's doing is showing us everything that's inside of that div that's all it's doing that's all it's showing us okay now it's not showing us the entire HTML page anymore it's just showing us that that's all we want anyways we need we just need access to that one div so what do we do next well now what we want to do is we want to access all of those objects or items whatever you want to call it we'll just call them items so I'm calling so I'm calling I this item number one what I'm highlighting here this is gonna be item number two this is gonna be item number three item for item five six seven eight nine you get the idea okay so if you notice each of the each of these is a list item okay so in HTML a list item has a tag of Li and an unordered list item has a tag of UL so you can see it says li here and how we can access all of these by either using the tag li or we can use the class four last tombstone so let's say week dot find all and let's just try Li and let's see what it gives us maybe we'll run into an error maybe we won't let's find out so when I hit Li actually let's comment this line out all right so when I hit a lie I get a lot of things okay and now let's actually do instead of a lie what we'll do is we'll say that find all and we'll say look for a class instead of that Li tag so we're gonna go class and the tricky thing in beautifulsoup is you can't just say the argument s class because that's means something special in Python so you get a new class underscore like that so your class is equal to and then we'll say tombstone actually we'll copy it so we don't make a mistake so I'll click here and I'll go forecast tombstone and we'll paste it just like that all right actually maybe that there's a better thing that we can do so let's go alright so we have a div actually that contains all of these items I miss this div so this div actually contains it's a tombstone container so it actually contains the the blue part is what it contains right so you can see that it contains the where it says Sunday it also contains the image it also looks like it contains the description where it says chance showers and it also contains the temperature whether it's a low or a high so let's actually access the tombstone container class all right so we're gonna go class and instead we're gonna say we're gonna go tombstone container if it lets me highlight it it's kind of tricky to highlight it's pretty secret well let's just click here and then let's dig a little deeper and this is the tombstone container right there so we're gonna say get all of those guys okay and we're gonna say get all of those guys okay so let's go boom boom do that and close this out and this we will call items okay this will call items and then we'll also print items so let's run this code now all right so you can see we have a list and each item in that list is essentially one tombstone container so let's say that I get item zero in that list the first item index it by zero I get one tombstone container so what is that it has the period name which is tonight if I go here you can see it right there it says tonight the next thing I should see is this if I hover over here this is the notice the paragraph tag and the CSS class here is period name so when I go here I see class period name is tonight when I go here the class here is short description short - des C and it should say slight chance showers and when I go to my code here so this is the image alt tag let's ignore that let's look for the thing that we're searching for the short description one alright so the short description here says slight chance right that's what we were looking for and slight chance showers and then we also have the temperature class all right which is this one right over here so that's that's what each item is right so each item in that list will give you man it's been kind of annoying because you cancel it's hard to select that but each item is essentially that tombstone container class this guy right over here okay tombstone container boom boom tombstone container the second one all right so now that we have that what we're gonna do is now we'll find what we want to do is get all of the period names so where it's us tonight where it's a Sunday Sunday night we want to get all of those period names and be able to put it in one column then we want to be able to take all of the short descriptions and put it in a description column and lastly we want to be able to take all of the temperatures and put them in a temperature column so each of them is gonna have their own columns so let's go ahead and do that so what I will do here is I'll say for items 0 let's first of all just get that items period name so I'll do class equals I remember the class name was period so I'm gonna just put that and first of all let's just print this guy and let's see what we get we will comment this out so when I print this out I get back this P tag but what I really want is the text the text that's in between the tags so what I can actually do is because it's a beautiful soup object I can just go and do get tagged like that and run this and when I run it gives me the text that says tonight now I can copy this line paste it and essentially just go and change the class name here to the next class which is short description so I'll do short description like that and get its class I'll run it and now it should said tonight and slight chance showers which is what it says right there and then last thing I want is temp so I'm just gonna call it by the temp class not temp low because this is temp hi this is temp high so we're just gonna get by the more general name so it works for every single tombstone container and we'll go here and we will simply say temperature like that and when I run it you will see that I get back those three things so now that I'm able to access all of those three things from the first tombstone container now it's very simple I just have to collect all of them in a list and then just run a for loop right to be able to access any of them so let's just go ahead and put them on all oh sorry it's already in the items list so items list already has them right now I'm only accessing the zeroth one which is in human language the first one now all I need to do is write a for loop that goes through all of all of those items what we're gonna use is list comprehension because that's cool so here's what I'm gonna demonstrate period underscore names is equal to and I'm gonna say item for item in items except what I want to do to this item is I want to not find its class and make sure that the class says period and then I also want to get its text alright so I'm gonna find oh you made a mistake here just didn't write the equal sign that should work now alright so this is a list comprehension that essentially says hey for each item in items as you loop through it just do this to it and I'll essentially get me back the period name now what I'll do is I'll print period names so you can see what it is it's right here this line and right all it did was loop through and just collect period names that's what we want okay so that's done now the second one we're gonna write is gonna be short description so we'll do short descriptions and we're gonna say item dot fine so it's essentially gonna be the same thing right so what I can do is I can just copy this guy come over here and paste it paste it move it up and then about a Bambaataa boom and we'll come here and instead of that we'll say a short description and then I'll copy paste this and I'll come here and I'll say temperature and I'll change this to temperatures all right so all of these now get us everything that we want I'll print these out as well for you so you can see what's going on all right I'll run this and you can see now we have three lists with what we want and that's exactly what we needed and now I'm gonna show you something really cool initially what my instinct was was to write it to a CSV file well you have to do a little bit of coding for that it takes some time it takes some stack overflowing it's kind of obnoxious there is a beautifully elegant and easy way to actually write this to a CSV file and also to do data analytics with it using pandas so let's use pandas here and it makes it so easy to turn this data into a table that you can then actually use manipulate and do stuff with so let's do that let's go up at the top and remember at the start I told you install pandas so now what we're gonna do is we're gonna go here and install pandas I believe P becomes before R so let's do it like this pandas as PD and now what we're gonna say is we're gonna say weather stuff is equal to pandas dot data frame and what data frame takes is actually a dictionary right and we are going to say period so the column is gonna be called period and the values of it are gonna be period names so it's gonna be this list that we made here and then the idea is essentially the same we're just gonna repeat that same process and we're gonna say the next one is gonna be short descriptions are just going to be open comma goes here gonna be short descriptions okay and then lastly we have temperatures which are just going to be temperatures oops comma make sure you have all the commas in the right place all right that is essentially the dictionary that's a much better way to actually write the dictionary so yeah that's the dictionary we got our data frame we put our column names these are the values that go in that column so that entire list is just gonna go there the entire list of short descriptions is gonna go under the short descriptions column the entire list of temperatures is gonna go under the temperatures column and one little trick I'm going to show you guys is you can put a comma here it won't give you a syntax error and it's just a good habit to get into because then when if you ever go back to that dictionary add more things you'll never run into an error so just a good code hygiene and if we print this out now let's see what this might look like so let's print this out so you can see that pandas actually turns us into a really I'll comments all the other print statements out now because we don't need them anymore and you can see what it does is it turns it into this beautiful looking table which you can now take a look at and do stuff with and you know figure out the means and figure out the averages and all that kind of nerdy stuff but we're not gonna get too much into that because that will be a whole nother video with pandas our job is not to use do that we are just learning how to do web scraping and to make this data useful the last cool thing I want to show you that makes it all super useful really handy trick is this awesome feature that panda has so I can just do weather stuff dot to underscore CSV all right it has this beautiful thing you can do which is just called dot to CSV and then just give it a CSV name so we're gonna call it let's call it weather dot CSV and I'm gonna save I'm gonna run this and right when I run it I get the CSV file over here which looks like this but let's take a look at what it looks like in our computer so we will go here and I will hit that and boom look at that it's looking super nice super fresh and you can see the entire data table right there I can also open it up in Excel if I want to let's zoom in a little bit so you can see it right and you can see we have our column called period with all of this data we have short descriptions with all of this data temperatures all of this data so that's so freaking cool right we just created a CSV file from scraping the internet scraping something online now you can just take that URL and pop something else in and they'll also work so for example forget Los Angeles weather maybe we can look at the weather of say Chicago so let's type in Chicago here and we will hit go hazardous weather outlook Wow wind advisory in effect so that's pretty scary huh so let's go let's copy this and for you it might look different right because you might be doing this video at following tutorial at another time so I'm gonna go here I'm gonna paste the new URL alright i pasted the new URL here and now let's run this code and we're gonna go take a look at our csv file now don't save and view will be three hundred percent and take a look at this now the data is completely different right now it's the Chicago's temperature so if you were making a web app you could use web scraping to use somebody else's website to dynamically generate stuff on your on website pretty powerful stuff all right hopefully you have enjoyed this tutorial that is all I had for you today but I do want to say that we have something very special coming up so again if you are a beginner or you are a working professional developer we have a new program coming out for you called how to automate stuff with Python so whether you're looking to land your first job and ace your interview or you are looking to get promoted at your current job or maybe you're trying to become a freelancer who kicks ass this brand new course will give you all of the essential skills that you need to make that happen this is something new that we're working on is gonna be coming out soon but in the description below we have the link to the page for the course how to automate stuff with Python and it will go deep dive into these things and much more all you need to do is just pop in your email below I will also send you a free gift once you put in your email below as well as once the course is out you will be the first one to know so go ahead pop it in your email below thank you so much for watching this video if you liked the video give it a like if you enjoyed this and you want to keep seeing the new tutorials that are gonna be coming out and make sure you subscribe to the channel with that said thank you so much for watching as always this is Kazi I love your face and I'll see you in the next videohey you in this video we are going to be doing web scraping with Python using libraries like beautifulsoup and request this is something that is very useful whether you are a beginner or a professional developer these are essential skills that almost every person needs so let's get started it's gonna be a fun little project and this was inspired by data quest io it's not a sponsored video or anything but I looked at one of their tutorials and they're doing a weather forecast I was like hey we should also just do that weather forecast except we're gonna be doing it for a lake that's where I live so we're gonna go to forecast dot weather.gov so that's number one let me zoom in here so you guys can see very clearly and I'm gonna type in Los Angeles will hit go and this is what pops up so what we're trying to do here is we want to pull this data that we want to put it into an Excel spreadsheet we you could also put it in pandas dataframe if you know what that is but most people know what an Excel spreadsheet is or what a CSV file is so that's essentially what we're going to be doing and we're gonna be doing web scraping here it's a pretty cool thing to learn how to do so let's jump right into it first of all for really basic stuff you know you have to know basic HTML we're not gonna cover too much of HTML or CSS in this video but you know for example if I write an h2 tag I have a code pen open here so I can just show how HTML works if I do h2 and I say hey guys like that him I do this it'll type in hey guys knife I do h6 tag right it'll make it smaller so these are H a.m. this is HTML this is basics very very basic basic use of it and now what we're gonna do is because I'm on Chrome we're gonna hit inspect here and dive deep using the chrome dev tools it's a very powerful way to learn the structure of the website that you want to scrape you want to abuse this tool okay use this tool a lot the chrome dev tools so now I'm gonna hover over and essentially what I want to do is I want to be able to get this week's forecast so I want to be able to find out what the weather is tonight Sunday Sunday night Monday and you know I also want to get the descriptions of it and I want to get what's the low and what's the high be able to put it in my Excel spreadsheet so let's do it so first of all I want you to click this button over here okay the shortcut for this is command shift C on a Mac or you can just come over here click this allows you to highlight over each thing and then it tells you what element it is of HTML or CSS file so he goes okay this is an image element as you can see it says I am g dot you know something so that dot pull left is a CSV clasp or sorry CSS class so let's go over here and check this out so it looks like we have a container here but this has an ID we have a div here and this has the ID of seven-day forecast body which is very good which is exactly what we need to leverage here so let's go ahead and use this because we know that once we have access to this we will have access to essentially everything else that's within this div so let's go to our code now you're gonna need a few modules installed and I'm not gonna walk through the full process of installing them because again those could turn into their own videos so let's go here and first of all we you need to install pip install beautiful soup 4 so that's step one another thing you're gonna need is pip install requests that's how you're gonna make HTTP requests to the websites and then you want to install the last one which is pandas all right because I've already installed them for me it says requirement already satisfied for you it's gonna take you through the installation process if you run into errors with installation process look up how to overcome them then once you solve it come back to this tutorial so once this is installed let's start writing some code so first of all we're gonna do important requests and I'm gonna also say from bs4 import beautiful soup like that alright and now essentially what I want to do is request this URL so I'm just gonna copy this URL right here I'll go here and I'll say this page is equal to request dot get and then I'll pop in that URL right there we'll also create a soup object out of that page so that requests thing that we just got we're gonna use that to create a beautiful soup object and what's beautiful soup allows you to do is makes the web scraping of the page really really easy it gives you nice structures it's able to find classes it's able to find paragraph tags it's able to find headers and it understands CSS and it understands HTML so it makes your life a lot easier that's why we're using beautifulsoup so let's go here in soup and go beautiful soup alright and we'll say page dot content and then we will say and we'll give it HTML parser okay so it'll note that it's HTML and so it needs to parse it as an HTML page alright and if I print this we should see something pretty weird so I'm gonna run this piece of code I'm gonna go here hit run all right so it says couldn't find a tree builder the features you requested HTML parser do you need to install a parser library Oh yep HTML dot parser not a - parser so we have now let's try it again we will run the code ok so looks like it worked and as you can see the soup object is all of the HTML code that's on that page it's pretty much the source code of that page so if I go and I right-click and I go I go view page source it's essentially this whole thing alright so it has access to all of it but what's cool is that I can do things like that find all and I can go find all of like the a tags you know find all a tags for example and if I run it it's essentially finding all of the links so it's finding the n-o a a.gov link that's there it's finding the weather.gov link that's on that page it's finding the Commerce gov link that's on that page so these all these links is able to find them right if I do not find images or if I do not find all you know short descriptions it'll find those as well so that's what's cool about beautiful soup makes your life really easy so let's go ahead and what we're gonna do is we're gonna create a variable that has this entire div stored inside of it that way we'll have access to this this div is what I'm talking about we'll have access to that div at all times so let's go to our code here and I'll say week is equal to soup dot find and we'll give it the ID in the ID what is the ID here the ID is seven-day forecast body so because I don't want to make a mistake I'm gonna click here once I click here you can see at the bottom it's highlighted this line I'll go to the ID and I'll double-click I will copy this I'll come back in here paste it so I'll make sure that this way there are no spelling errors as a programmer I know sometimes you want to do things quick and just type it out but the reason why you want to paste is not so you're faster but the real reason why you want to copy paste as much as you can is so you're accurate and you don't make mistakes that's why I took the time to copy it even if it was small like it was three letters I would still copy and paste because it has happened to me so many times that I'll make like a tiny little mistake like the one I made up here with HTML dot parser where I wrote a - here alright so we have this access to this week let's first of all print it out let's see what it shows us so we'll run our code alright so basically now what it's doing is showing us everything that's inside of that div that's all it's doing that's all it's showing us okay now it's not showing us the entire HTML page anymore it's just showing us that that's all we want anyways we need we just need access to that one div so what do we do next well now what we want to do is we want to access all of those objects or items whatever you want to call it we'll just call them items so I'm calling so I'm calling I this item number one what I'm highlighting here this is gonna be item number two this is gonna be item number three item for item five six seven eight nine you get the idea okay so if you notice each of the each of these is a list item okay so in HTML a list item has a tag of Li and an unordered list item has a tag of UL so you can see it says li here and how we can access all of these by either using the tag li or we can use the class four last tombstone so let's say week dot find all and let's just try Li and let's see what it gives us maybe we'll run into an error maybe we won't let's find out so when I hit Li actually let's comment this line out all right so when I hit a lie I get a lot of things okay and now let's actually do instead of a lie what we'll do is we'll say that find all and we'll say look for a class instead of that Li tag so we're gonna go class and the tricky thing in beautifulsoup is you can't just say the argument s class because that's means something special in Python so you get a new class underscore like that so your class is equal to and then we'll say tombstone actually we'll copy it so we don't make a mistake so I'll click here and I'll go forecast tombstone and we'll paste it just like that all right actually maybe that there's a better thing that we can do so let's go alright so we have a div actually that contains all of these items I miss this div so this div actually contains it's a tombstone container so it actually contains the the blue part is what it contains right so you can see that it contains the where it says Sunday it also contains the image it also looks like it contains the description where it says chance showers and it also contains the temperature whether it's a low or a high so let's actually access the tombstone container class all right so we're gonna go class and instead we're gonna say we're gonna go tombstone container if it lets me highlight it it's kind of tricky to highlight it's pretty secret well let's just click here and then let's dig a little deeper and this is the tombstone container right there so we're gonna say get all of those guys okay and we're gonna say get all of those guys okay so let's go boom boom do that and close this out and this we will call items okay this will call items and then we'll also print items so let's run this code now all right so you can see we have a list and each item in that list is essentially one tombstone container so let's say that I get item zero in that list the first item index it by zero I get one tombstone container so what is that it has the period name which is tonight if I go here you can see it right there it says tonight the next thing I should see is this if I hover over here this is the notice the paragraph tag and the CSS class here is period name so when I go here I see class period name is tonight when I go here the class here is short description short - des C and it should say slight chance showers and when I go to my code here so this is the image alt tag let's ignore that let's look for the thing that we're searching for the short description one alright so the short description here says slight chance right that's what we were looking for and slight chance showers and then we also have the temperature class all right which is this one right over here so that's that's what each item is right so each item in that list will give you man it's been kind of annoying because you cancel it's hard to select that but each item is essentially that tombstone container class this guy right over here okay tombstone container boom boom tombstone container the second one all right so now that we have that what we're gonna do is now we'll find what we want to do is get all of the period names so where it's us tonight where it's a Sunday Sunday night we want to get all of those period names and be able to put it in one column then we want to be able to take all of the short descriptions and put it in a description column and lastly we want to be able to take all of the temperatures and put them in a temperature column so each of them is gonna have their own columns so let's go ahead and do that so what I will do here is I'll say for items 0 let's first of all just get that items period name so I'll do class equals I remember the class name was period so I'm gonna just put that and first of all let's just print this guy and let's see what we get we will comment this out so when I print this out I get back this P tag but what I really want is the text the text that's in between the tags so what I can actually do is because it's a beautiful soup object I can just go and do get tagged like that and run this and when I run it gives me the text that says tonight now I can copy this line paste it and essentially just go and change the class name here to the next class which is short description so I'll do short description like that and get its class I'll run it and now it should said tonight and slight chance showers which is what it says right there and then last thing I want is temp so I'm just gonna call it by the temp class not temp low because this is temp hi this is temp high so we're just gonna get by the more general name so it works for every single tombstone container and we'll go here and we will simply say temperature like that and when I run it you will see that I get back those three things so now that I'm able to access all of those three things from the first tombstone container now it's very simple I just have to collect all of them in a list and then just run a for loop right to be able to access any of them so let's just go ahead and put them on all oh sorry it's already in the items list so items list already has them right now I'm only accessing the zeroth one which is in human language the first one now all I need to do is write a for loop that goes through all of all of those items what we're gonna use is list comprehension because that's cool so here's what I'm gonna demonstrate period underscore names is equal to and I'm gonna say item for item in items except what I want to do to this item is I want to not find its class and make sure that the class says period and then I also want to get its text alright so I'm gonna find oh you made a mistake here just didn't write the equal sign that should work now alright so this is a list comprehension that essentially says hey for each item in items as you loop through it just do this to it and I'll essentially get me back the period name now what I'll do is I'll print period names so you can see what it is it's right here this line and right all it did was loop through and just collect period names that's what we want okay so that's done now the second one we're gonna write is gonna be short description so we'll do short descriptions and we're gonna say item dot fine so it's essentially gonna be the same thing right so what I can do is I can just copy this guy come over here and paste it paste it move it up and then about a Bambaataa boom and we'll come here and instead of that we'll say a short description and then I'll copy paste this and I'll come here and I'll say temperature and I'll change this to temperatures all right so all of these now get us everything that we want I'll print these out as well for you so you can see what's going on all right I'll run this and you can see now we have three lists with what we want and that's exactly what we needed and now I'm gonna show you something really cool initially what my instinct was was to write it to a CSV file well you have to do a little bit of coding for that it takes some time it takes some stack overflowing it's kind of obnoxious there is a beautifully elegant and easy way to actually write this to a CSV file and also to do data analytics with it using pandas so let's use pandas here and it makes it so easy to turn this data into a table that you can then actually use manipulate and do stuff with so let's do that let's go up at the top and remember at the start I told you install pandas so now what we're gonna do is we're gonna go here and install pandas I believe P becomes before R so let's do it like this pandas as PD and now what we're gonna say is we're gonna say weather stuff is equal to pandas dot data frame and what data frame takes is actually a dictionary right and we are going to say period so the column is gonna be called period and the values of it are gonna be period names so it's gonna be this list that we made here and then the idea is essentially the same we're just gonna repeat that same process and we're gonna say the next one is gonna be short descriptions are just going to be open comma goes here gonna be short descriptions okay and then lastly we have temperatures which are just going to be temperatures oops comma make sure you have all the commas in the right place all right that is essentially the dictionary that's a much better way to actually write the dictionary so yeah that's the dictionary we got our data frame we put our column names these are the values that go in that column so that entire list is just gonna go there the entire list of short descriptions is gonna go under the short descriptions column the entire list of temperatures is gonna go under the temperatures column and one little trick I'm going to show you guys is you can put a comma here it won't give you a syntax error and it's just a good habit to get into because then when if you ever go back to that dictionary add more things you'll never run into an error so just a good code hygiene and if we print this out now let's see what this might look like so let's print this out so you can see that pandas actually turns us into a really I'll comments all the other print statements out now because we don't need them anymore and you can see what it does is it turns it into this beautiful looking table which you can now take a look at and do stuff with and you know figure out the means and figure out the averages and all that kind of nerdy stuff but we're not gonna get too much into that because that will be a whole nother video with pandas our job is not to use do that we are just learning how to do web scraping and to make this data useful the last cool thing I want to show you that makes it all super useful really handy trick is this awesome feature that panda has so I can just do weather stuff dot to underscore CSV all right it has this beautiful thing you can do which is just called dot to CSV and then just give it a CSV name so we're gonna call it let's call it weather dot CSV and I'm gonna save I'm gonna run this and right when I run it I get the CSV file over here which looks like this but let's take a look at what it looks like in our computer so we will go here and I will hit that and boom look at that it's looking super nice super fresh and you can see the entire data table right there I can also open it up in Excel if I want to let's zoom in a little bit so you can see it right and you can see we have our column called period with all of this data we have short descriptions with all of this data temperatures all of this data so that's so freaking cool right we just created a CSV file from scraping the internet scraping something online now you can just take that URL and pop something else in and they'll also work so for example forget Los Angeles weather maybe we can look at the weather of say Chicago so let's type in Chicago here and we will hit go hazardous weather outlook Wow wind advisory in effect so that's pretty scary huh so let's go let's copy this and for you it might look different right because you might be doing this video at following tutorial at another time so I'm gonna go here I'm gonna paste the new URL alright i pasted the new URL here and now let's run this code and we're gonna go take a look at our csv file now don't save and view will be three hundred percent and take a look at this now the data is completely different right now it's the Chicago's temperature so if you were making a web app you could use web scraping to use somebody else's website to dynamically generate stuff on your on website pretty powerful stuff all right hopefully you have enjoyed this tutorial that is all I had for you today but I do want to say that we have something very special coming up so again if you are a beginner or you are a working professional developer we have a new program coming out for you called how to automate stuff with Python so whether you're looking to land your first job and ace your interview or you are looking to get promoted at your current job or maybe you're trying to become a freelancer who kicks ass this brand new course will give you all of the essential skills that you need to make that happen this is something new that we're working on is gonna be coming out soon but in the description below we have the link to the page for the course how to automate stuff with Python and it will go deep dive into these things and much more all you need to do is just pop in your email below I will also send you a free gift once you put in your email below as well as once the course is out you will be the first one to know so go ahead pop it in your email below thank you so much for watching this video if you liked the video give it a like if you enjoyed this and you want to keep seeing the new tutorials that are gonna be coming out and make sure you subscribe to the channel with that said thank you so much for watching as always this is Kazi I love your face and I'll see you in the next video\n"