Okay, so today I’m gonna walk you through how I pulled player stats for that San Antonio Spurs vs. 76ers game. It was a bit of a grind, but I got there in the end!
First off, I started by hitting up the usual sports stats sites. You know, the ones everyone uses. But the problem was, I needed the data in a format I could actually use, not just look at. Copy-pasting a million rows? No thanks!
So, I thought, “Right, time for some web scraping!” I’d done it before, so I wasn’t totally in the dark. I fired up Python. Yeah, I know, super original. But hey, it gets the job done.
Then, I grabbed BeautifulSoup and Requests. These are like the bread and butter of scraping. Requests to actually get the HTML, and BeautifulSoup to pick through it like a vulture on… well, you get the picture.
I spent a while just inspecting the webpage source. Figuring out what HTML tags held the player names, points, rebounds, all that jazz. This is always the most tedious bit, honestly. It’s like detective work, except the criminal is badly written HTML.
After that, I wrote a bunch of code to loop through the tables and extract the data. I ran into a few snags. The site’s HTML wasn’t super consistent (why would it be, right?), so I had to add some `if` statements to handle different table structures. It was messy, but functional.
Here’s the thing that almost tripped me up: pagination! The stats were split across multiple pages. Ugh. So, I had to modify my code to loop through each page, grab the stats, and combine them into one big pile.
Once I had all the data, it was a mess. Like, seriously messy. Strings where there should be numbers, missing values, weird characters. I had to clean it up big time. I used pandas for this. Pandas is a lifesaver, honestly. I cleaned it by converting the datatypes, filling NaN values and fixing the string formatting errors
Finally, after what felt like forever, I had a nice, clean CSV file with all the player stats. I could finally do some actual analysis! It was a slog, but seeing that data neatly organized made it all worthwhile. Plus, I learned a thing or two about web scraping in the process.
Moral of the story: web scraping is never as easy as it looks, but with a little Python and a lot of patience, you can wrangle almost any data you need.
