Please help contribute to the Reddit categorization project here

    lieutenant_lowercase

    + friends - friends
    17,909 link karma
    9,683 comment karma
    send message redditor for

    [–] What are some web scraping tricks everyone should know? lieutenant_lowercase 233 points ago * (lasted edited 11 days ago) in learnpython

    • If you can use an AJAX API call then use it. Don't waste time with Selenium unless you absolutely need it.
    • Sites that check for javascript can be scraped efficiently by loading the site in Selenium then passing the cookies/headers back to a requests session.
    • Quickly recreate requests in python by copying the request from chrome developer console into postman, then using that to generate python code. You can get python code in 10seconds using this.
    • If I'm scraping tabular data I prefer to create a dictionary for each entry and append them to a list. Once done you can load this straight into a dataframe with all columns labelled correctly.
    • Often a page will have all the data you need emebedded into the source as a JSON array. Strip that out instead of using beautifulsoup to pick apart the HTML. Use demjson library for a less-strict JSON loader.
    • If you have a json object you can easily flatten it into a dataframe using pd.io.json.json_normalize()
    • Create a @retry request decorator that you can re-use
    • If you are prototyping, cache your responses so you don't keep having to scrape the same page over and over whilst testing
    • Use an exponential backoff to retry

    [–] It just gets better and better. lieutenant_lowercase 201 points ago in PUBATTLEGROUNDS

    Your team-mate constantly telling you what to do would wind me up

    [–] Single male (28), looking for really good fish & chip shop in London. lieutenant_lowercase 111 points ago in london

    What on earth does your age, sex and relationship status have to do with fish & chip shops haha? Is this post code for something?

    [–] Free Talk Friday lieutenant_lowercase 93 points ago in soccer

    Check this out - A new restaurant has opened in Bolton (of all places) called Steaks on a Plane. It's an actual jumbo jet haha. Pic here -

    [–] Revealed: the $2bn offshore trail that leads to Vladimir Putin lieutenant_lowercase 85 points ago in worldnews

    There are 107 media organisations in 78 countries that have been analysing the documents

    [–] Please ALWAYS steal my goal! lieutenant_lowercase 74 points ago in RocketLeague

    Are you purposely all saying the exactly same thing just slightly reworded?