\n This is one paragraph.\n

\n This is another paragraph. HTML is cool!\n

\nThis is one paragraph.\n

\nThis is another paragraph. HTML is cool!\n

\n This is one paragraph.\n

\n This is another paragraph. HTML is cool!\n

\nThis is one paragraph.\n

\nThis is another paragraph. HTML is cool!\n

\nThis is one paragraph.\n

\n \n Data Exploration with Pandas Profiler and D-Tale\n \n

"},{"_key":"a7c4d26a8eb32","_type":"span","marks":[],"text":" tag which is an unordered list, and each entry is in a "},{"_key":"a7c4d26a8eb33","_type":"span","marks":["code"],"text":"
• "},{"_key":"a7c4d26a8eb34","_type":"span","marks":[],"text":" tag, which is a list item. The unordered list has no class or role (unlike other lists in the page):"}],"markDefs":[],"style":"normal"},{"_key":"8f3ab442d7a0","_type":"block.code","content":{"code":"blog_soup.find('ul', class_=None, role=None)","language":"python"}},{"_key":"c6118395d91b","_type":"block.code","content":{"code":"
  \n
  • Detecting erroneous data.
  \n
  • Determining how much missing data there is.
  \n
  • Understanding the structure of the data.
  \n
  • Identifying important variables in the data.
  \n
  • Sense-checking the validity of the data.
  \n
  ","language":"text"}},{"_key":"d8b0da30e298","_type":"block","children":[{"_key":"d8b0da30e2980","_type":"span","marks":[],"text":"OK, we can now extract the entries for the HTML list and put them in a Python list:"}],"markDefs":[],"style":"normal"},{"_key":"2b2768cb2162","_type":"block.code","content":{"code":"my_ul = blog_soup.find('ul', class_=None, role=None)\nli_goals =my_ul.find_all('li')\ngoals = []\nfor li_goal in li_goals: \n v goals.append(li_goal. string)\nprint(goals)","language":"python"}},{"_key":"40d534d03ae2","_type":"block.code","content":{"code":"['Detecting erroneous data.',\n'Determining how much missing data there is.',\n'Understanding the structure of the data.',\n'Identifying important variables in the data.',\n'Sense-checking the validity of the data.']","language":"text"}},{"_key":"8fbf34e8caf9","_type":"block","children":[{"_key":"8fbf34e8caf90","_type":"span","marks":[],"text":"As mentioned before, we could be interested in getting the text of the blog post to carry out some natural language processing. We can do that in one go with the "},{"_key":"8fbf34e8caf91","_type":"span","marks":["code"],"text":"get_text()"},{"_key":"8fbf34e8caf92","_type":"span","marks":[],"text":" method."}],"markDefs":[],"style":"normal"},{"_key":"acfad3b153d2","_type":"block.code","content":{"code":"blog_text = blog_soup.get_text()","language":"python"}},{"_key":"2192a9dbab49","_type":"block","children":[{"_key":"2192a9dbab490","_type":"span","marks":[],"text":"We can now use some of the techniques described in the earlier post on "},{"_key":"2192a9dbab491","_type":"span","marks":["cb1acf2f9445"],"text":"natural language with spaCy"},{"_key":"2192a9dbab492","_type":"span","marks":[],"text":". In this case we are showing each entry, its part-of-speech (POS), the explanation for POS and whether the entry is considered a stop word or not. For convenience we are only showing the first 10 entries."}],"markDefs":[{"_key":"cb1acf2f9445","_type":"link","deepLink":null,"eventName":null,"internalLink":{"_id":"bQvS44cO2Kqh2eurZ5pLts","_type":"blogPost","locale":"en","routePath":"blog/natural-language-in-python-using-spacy"},"linkType":"internal","newWindow":null,"url":null,"utmParameters":null,"videoPopup":null}],"style":"normal"},{"_key":"c23377336d62","_type":"block.code","content":{"code":"import spacy\n\nnlp = spacy.load(\"en_core_web_sm\")\ndoc = nlp(blog_text)\nfor entry in doc[:10]:\n print(entry.text, entry.pos_, \n spacy.explain(entry.pos_), \n entry.is_stop)","language":"python"}},{"_key":"1d6510b3b513","_type":"block.code","content":{"code":" SPACE space False\nData PROPN proper noun False\nExploration PROPN proper noun False\nwith ADP adposition True\nPandas PROPN proper noun False\nProfiler PROPN proper noun False\nand CCONJ coordinating conjunction True\nD PROPN proper noun False\n- PUNCT punctuation False\nTale PROPN proper noun False","language":"text"}},{"_key":"7b56b1ea690d","_type":"block","children":[{"_key":"7b56b1ea690d0","_type":"span","marks":[],"text":"Reading table data"}],"markDefs":[],"style":"h2"},{"_key":"bcf635a8ea6e","_type":"block","children":[{"_key":"bcf635a8ea6e0","_type":"span","marks":[],"text":"Finally, let us use what we have learned so far to get data that can be shown in the form of a table. We mentioned at the beginning of this post that we may want to see the number of gold medals obtained by different countries in the Olympic Games in Tokyo. We can read that information from the relevant entry in Wikipedia."}],"markDefs":[],"style":"normal"},{"_key":"db265a35c982","_type":"block.code","content":{"code":"url = 'https://en.wikipedia.org/wiki/2020_Summer_Olympics_medal_table'\n\nwiki_page = requests.get(url)\n\nmedal_soup = BeautifulSoup(wiki_page.content, 'html.parser')","language":"python"}},{"_key":"b2d85a6cd5f7","_type":"block","children":[{"_key":"b2d85a6cd5f70","_type":"span","marks":[],"text":"Using the inspect element functionality of my browser I can see that the table where the data is located has a class. See the screenshot from my browser:"}],"markDefs":[],"style":"normal"},{"_key":"zP6ApMxq1wlO","_type":"block.image","alt":"beautiful_soup_medals_table","asset":{"_ref":"image-7c57951c281cc41fe3f7ae371adf07cdc298eab6-1468x880-jpg","_type":"reference"}},{"_key":"0fc197a8d936","_type":"block.code","content":{"code":"medal_table = medal_soup.find('table', \n class_='wikitable sortable plainrowheaders jquery-tablesorter')","language":"python"}},{"_key":"2761a9620889","_type":"block","children":[{"_key":"2761a96208890","_type":"span","marks":[],"text":"In this case, we need to iterate through each row ("},{"_key":"2761a96208891","_type":"span","marks":["code"],"text":"tr"},{"_key":"2761a96208892","_type":"span","marks":[],"text":") and then assign each of its elements ("},{"_key":"2761a96208893","_type":"span","marks":["code"],"text":"td"},{"_key":"2761a96208894","_type":"span","marks":[],"text":") to a variable and append it to a list. One exception is the heading of the table which has"},{"_key":"2761a96208895","_type":"span","marks":["code"],"text":"th"},{"_key":"2761a96208896","_type":"span","marks":[],"text":"elements."}],"markDefs":[],"style":"normal"},{"_key":"6e6699a4f585","_type":"block","children":[{"_key":"6e6699a4f5850","_type":"span","marks":[],"text":"Let us find all the rows. We will single out the first one of those to extract the headers, and we'll store the medal information in a variable called "},{"_key":"6e6699a4f5851","_type":"span","marks":["code"],"text":"allRows"},{"_key":"6e6699a4f5852","_type":"span","marks":[],"text":":"}],"markDefs":[],"style":"normal"},{"_key":"638139979ff5","_type":"block.code","content":{"code":"tmp = medal_table.find_all('tr')\n\nfirst = tmp[0]\nallRows = tmp[1:-1]","language":"python"}},{"_key":"1d5ef6044099","_type":"block","children":[{"_key":"1d5ef60440990","_type":"span","marks":[],"text":"Let us look a the "},{"_key":"1d5ef60440991","_type":"span","marks":["code"],"text":"first"},{"_key":"1d5ef60440992","_type":"span","marks":[],"text":" row:"}],"markDefs":[],"style":"normal"},{"_key":"86b90ee893a9","_type":"block.code","content":{"code":" print(first)","language":"python"}},{"_key":"b2d8db8f0d8a","_type":"block.code","content":{"code":"RankTeamGoldSilverBronzeTotal","language":"text"}},{"_key":"d22e7088e5cc","_type":"block","children":[{"_key":"d22e7088e5cc0","_type":"span","marks":[],"text":"As you can see, we need to find all the "},{"_key":"d22e7088e5cc1","_type":"span","marks":["code"],"text":"th"},{"_key":"d22e7088e5cc2","_type":"span","marks":[],"text":" tags and get the text, and furthermore we will get rid of heading and trailing spaces with the "},{"_key":"d22e7088e5cc3","_type":"span","marks":["code"],"text":"strip()"},{"_key":"d22e7088e5cc4","_type":"span","marks":[],"text":" method. We do all this within a list comprehension syntax:"}],"markDefs":[],"style":"normal"},{"_key":"3b0adf219fbb","_type":"block.code","content":{"code":" headers = [header.get_text().strip() for \n header in first.find_all('th')]\n print(headers)","language":"python"}},{"_key":"fbaf8d9e92de","_type":"block.code","content":{"code":"['Rank', 'Team', 'Gold', 'Silver', 'Bronze', 'Total']","language":"text"}},{"_key":"af21e271b71f","_type":"block","children":[{"_key":"af21e271b71f0","_type":"span","marks":[],"text":"Cool! We now turn our attention to the medals:"}],"markDefs":[],"style":"normal"},{"_key":"5a5e1b499215","_type":"block.code","content":{"code":" results = [[data.get_text() for \n data in row.find_all('td')] \n for row in allRows]","language":"python"}},{"_key":"a4b735532152","_type":"block.code","content":{"code":" print(results[:10])","language":"python"}},{"_key":"3ed010217b41","_type":"block.code","content":{"code":"[['1', '39', '41', '33', '113'], ['2', '38', '32', '18', '88'], ['3', '27', '14', '17', '58'], ['4', '22', '21', '22', '65'], ['5', '20', '28', '23', '71'], ['6', '17', '7', '22', '46'], ['7', '10', '12', '14', '36'], ['8', '10', '12', '11', '33'], ['9', '10', '11', '16', '37'], ['10', '10', '10', '20', '40']]","language":"text"}},{"_key":"0c44e70f7682","_type":"block","children":[{"_key":"0c44e70f76820","_type":"span","marks":[],"text":"Hang on... that looks great but it does not have the names of the countries. Let us see the content of "},{"_key":"0c44e70f76821","_type":"span","marks":["code"],"text":"allRows"},{"_key":"0c44e70f76822","_type":"span","marks":[],"text":":"}],"markDefs":[],"style":"normal"},{"_key":"024b15ad3785","_type":"block.code","content":{"code":"allRows[0]","language":"python"}},{"_key":"dd2e1523ad0c","_type":"block.code","content":{"code":"1 United States (USA)394133113","language":"text"}},{"_key":"ae607675fc6c","_type":"block","children":[{"_key":"ae607675fc6c0","_type":"span","marks":[],"text":"Aha! The name of the country is in a "},{"_key":"ae607675fc6c1","_type":"span","marks":["code"],"text":"th"},{"_key":"ae607675fc6c2","_type":"span","marks":[],"text":" tag, and actually we can extract it from the string inside the hyperlink:"}],"markDefs":[],"style":"normal"},{"_key":"f854a0058855","_type":"block.code","content":{"code":" countries = [[countries.find(text=True) for countries in \n row.find_all('a')] \n for row in allRows ]\n \ncountries[:10]","language":"python"}},{"_key":"9f329042dec9","_type":"block.code","content":{"code":"[['United States'], ['China'], ['Japan'], ['Great Britain'], ['ROC'], ['Australia'], ['Netherlands'], ['France'], ['Germany'], ['Italy']]","language":"text"}},{"_key":"8955c551f90b","_type":"block","children":[{"_key":"8955c551f90b0","_type":"span","marks":[],"text":"You can see from the table in the website that some countries have the same number of gold, silver and bronze medals and thus are given the same rank. See for instance rank 36 given to both Greece and Uganda. This has some implications in our data scraping strategy, let look at the results for entries 35 to 44 :"}],"markDefs":[],"style":"normal"},{"_key":"f12b6c371756","_type":"block.code","content":{"code":"results[33:44]","language":"python"}},{"_key":"679128345125","_type":"block.code","content":{"code":"[['34', '2', '4', '6', '12'],\n['35', '2', '2', '9', '13'],\n['36', '2', '1', '1', '4'],\n['2', '1', '1', '4'],\n['38', '2', '1', '0', '3'],\n['39', '2', '0', '2', '4'],\n['2', '0', '2', '4'],\n['41', '2', '0', '1', '3'],\n['42', '2', '0', '0', '2'],\n['2', '0', '0', '2'],\n['44', '1', '6', '12', '19']]","language":"text"}},{"_key":"2a9b2b605236","_type":"block","children":[{"_key":"2a9b2b6052360","_type":"span","marks":[],"text":"Our rows have five entries, but those that have the same ranking actually have four entries. These entries have a "},{"_key":"2a9b2b6052361","_type":"span","marks":["code"],"text":"rowspan"},{"_key":"2a9b2b6052362","_type":"span","marks":[],"text":" attribute as shown in the screenshot for rank 36 below:"}],"markDefs":[],"style":"normal"},{"_key":"6rWBBQk4gIEM","_type":"block.image","alt":"beautiful_soup_rowspan","asset":{"_ref":"image-012456ebcc7ae414b41b0a7d9bbbca1b7adb9557-1218x822-jpg","_type":"reference"}},{"_key":"ZrhWZ9wXupmU","_type":"block.image","alt":"beautiful_soup_rowspan","asset":{"_ref":"image-012456ebcc7ae414b41b0a7d9bbbca1b7adb9557-1218x822-jpg","_type":"reference"}},{"_key":"7437ac280890","_type":"block","children":[{"_key":"7437ac2808900","_type":"span","marks":[],"text":"Let us find the entries that have a "},{"_key":"7437ac2808901","_type":"span","marks":["code"],"text":"rowspan"},{"_key":"7437ac2808902","_type":"span","marks":[],"text":" attribute and count the number of countries that share the same rank. We will keep track of the entry number, the "},{"_key":"7437ac2808903","_type":"span","marks":["code"],"text":"td"},{"_key":"7437ac2808904","_type":"span","marks":[],"text":" number, the number of countries that share the same rank and rank assigned:"}],"markDefs":[],"style":"normal"},{"_key":"27f75376cd17","_type":"block.code","content":{"code":" rowspan = []\n\nfor num, tr in enumerate(allRows):\n tmp = []\n for td_num, data in enumerate(tr.find_all('td')):\n if data.has_attr(\"rowspan\"):\n rowspan.append((num, td_num, int(data[\"rowspan\"]), data.get_text()))","language":"python"}},{"_key":"76f5412bf71d","_type":"block.code","content":{"code":"print(rowspan)","language":"python"}},{"_key":"66a356195c29","_type":"block.code","content":{"code":"[(35, 0, 2, '36'),\n(38, 0, 2, '39'),\n(41, 0, 2, '42'),\n(45, 0, 2, '46'),\n(49, 0, 2, '50'),\n(55, 0, 2, '56'),\n(58, 0, 4, '59'),\n(62, 0, 3, '63'),\n(71, 0, 2, '72'),\n(73, 0, 3, '74'),\n(76, 0, 6, '77'),\n(85, 0, 8, '86')]","language":"text"}},{"_key":"bf6238857b2f","_type":"block","children":[{"_key":"bf6238857b2f0","_type":"span","marks":[],"text":"We can now fix our "},{"_key":"bf6238857b2f1","_type":"span","marks":["code"],"text":"results"},{"_key":"bf6238857b2f2","_type":"span","marks":[],"text":" by inserting the correct rank in the rows that have missing values:"}],"markDefs":[],"style":"normal"},{"_key":"28765688a254","_type":"block.code","content":{"code":" for i in rowspan:\n # tr value of rowspan is in the 1st place in results\n for j in range(1, i[2]):\n # Add value in the next tr\n results[i[0]+j].insert(i[1], i[3])","language":"python"}},{"_key":"11fe7436abb9","_type":"block","children":[{"_key":"11fe7436abb90","_type":"span","marks":[],"text":"Let us check that this worked:"}],"markDefs":[],"style":"normal"},{"_key":"8ba0b42614bd","_type":"block.code","content":{"code":" print(results)[33:44]","language":"python"}},{"_key":"bed60705eb92","_type":"block.code","content":{"code":"[['34', '2', '4', '6', '12'],\n['35', '2', '2', '9', '13'],\n['36', '2', '1', '1', '4'],\n['36', '2', '1', '1', '4'],\n['38', '2', '1', '0', '3'],\n['39', '2', '0', '2', '4'],\n['39', '2', '0', '2', '4'],\n['41', '2', '0', '1', '3'],\n['42', '2', '0', '0', '2'],\n['42', '2', '0', '0', '2'],\n['44', '1', '6', '12', '19']]","language":"text"}},{"_key":"aac23220717a","_type":"block","children":[{"_key":"aac23220717a0","_type":"span","marks":[],"text":"We can now insert the names of the countries too:"}],"markDefs":[],"style":"normal"},{"_key":"e5d28c33d8d5","_type":"block.code","content":{"code":" for i, country in enumerate(countries):\n results[i].insert(1, country[0])","language":"python"}},{"_key":"7adfc437d1f2","_type":"block","children":[{"_key":"7adfc437d1f20","_type":"span","marks":[],"text":"Finally, we can use our data to create a Pandas dataframe:"}],"markDefs":[],"style":"normal"},{"_key":"0733e0c493d8","_type":"block.code","content":{"code":"import pandas as pd\ndf = pd.DataFrame(data = results, columns = headers)\ndf['Rank'] = df['Rank'].map(lambda x: x.replace('\\n',''))\ndf['Total'] = df['Total'].map(lambda x: x.replace('\\n',''))\n\ncols = ['Rank','Gold', 'Silver', 'Bronze', 'Total']\ndf[cols] = df[cols].apply(pd.to_numeric)\n\ndf.head()","language":"python"}},{"_key":"uUajw23F8sl5","_type":"block.table","rows":[{"_key":"o8Ck10JLIMnF","cells":[{"_key":"ZERJ2VCvyGA1","_type":"cell","content":[{"_key":"N5dxHBpvRe6c","_type":"block","children":[{"_key":"3PR1ckKNrddt","_type":"span","marks":[],"text":"1"}],"markDefs":[]}]},{"_key":"W7LV8085xEV2","_type":"cell","content":[{"_key":"5oCjGtHJcTUP","_type":"block","children":[{"_key":"fs8OU4soo0GO","_type":"span","marks":[],"text":"United States"}],"markDefs":[]}]},{"_key":"UOuC84c30mdw","_type":"cell","content":[{"_key":"EEGYvhiXRmum","_type":"block","children":[{"_key":"1VZnu84ba351","_type":"span","marks":[],"text":"39"}],"markDefs":[]}]},{"_key":"oRzkthXRtPMD","_type":"cell","content":[{"_key":"gwuPLebgJWZt","_type":"block","children":[{"_key":"xTMglDnlRwN5","_type":"span","marks":[],"text":"41"}],"markDefs":[]}]},{"_key":"AMSCxtuqeAFx","_type":"cell","content":[{"_key":"ARNo98iJXWik","_type":"block","children":[{"_key":"3PyM3MPbdpJ0","_type":"span","marks":[],"text":"33"}],"markDefs":[]}]},{"_key":"5bKByQRZYHHx","_type":"cell","content":[{"_key":"h6A2FU1gn7LY","_type":"block","children":[{"_key":"KZNzDmOjQLX1","_type":"span","marks":[],"text":"113"}],"markDefs":[]}]}]},{"_key":"aGS9N7ddSpRU","cells":[{"_key":"vnmiws5MyK6L","_type":"cell","content":[{"_key":"cKlMx6yCBz2u","_type":"block","children":[{"_key":"cPYiiL9wx9pl","_type":"span","marks":[],"text":"2"}],"markDefs":[]}]},{"_key":"lErELDP1f8dS","_type":"cell","content":[{"_key":"YH4zrd5M4cLi","_type":"block","children":[{"_key":"Gbc7DKPapDEr","_type":"span","marks":[],"text":"China"}],"markDefs":[]}]},{"_key":"lKuwCMyCrVoW","_type":"cell","content":[{"_key":"uOgCcNlwdiEj","_type":"block","children":[{"_key":"op43djQxZSBl","_type":"span","marks":[],"text":"38"}],"markDefs":[]}]},{"_key":"Yg8aARN1QE5t","_type":"cell","content":[{"_key":"XgpElq0uWhBh","_type":"block","children":[{"_key":"RjgPuEk1abax","_type":"span","marks":[],"text":"32"}],"markDefs":[]}]},{"_key":"81L6TWpfTT0Y","_type":"cell","content":[{"_key":"0WnFN9BDp09v","_type":"block","children":[{"_key":"x2Oe0bXJYfIq","_type":"span","marks":[],"text":"18"}],"markDefs":[]}]},{"_key":"j0mnSOvZ2nU2","_type":"cell","content":[{"_key":"yz7p0MvVezA1","_type":"block","children":[{"_key":"5NNZfJgZlMNK","_type":"span","marks":[],"text":"88"}],"markDefs":[]}]}]},{"_key":"YkAJB2BmREny","cells":[{"_key":"Mh8Ehv8v4u2O","_type":"cell","content":[{"_key":"07da3Ct6cgRj","_type":"block","children":[{"_key":"24XjgSVvzsbU","_type":"span","marks":[],"text":"3"}],"markDefs":[]}]},{"_key":"PcOtO7oHnvhu","_type":"cell","content":[{"_key":"W8DgQByTx4QE","_type":"block","children":[{"_key":"J48xq3CdZRLi","_type":"span","marks":[],"text":"Japan"}],"markDefs":[]}]},{"_key":"16KOVLCLmy67","_type":"cell","content":[{"_key":"FKeWId2UeDKk","_type":"block","children":[{"_key":"fYbI5OZWKcsi","_type":"span","marks":[],"text":"27"}],"markDefs":[]}]},{"_key":"VHCbTFgc0xSG","_type":"cell","content":[{"_key":"MP6K1FEBSUXk","_type":"block","children":[{"_key":"16zbIBVkE4n7","_type":"span","marks":[],"text":"14"}],"markDefs":[]}]},{"_key":"jdkHixyPwX4P","_type":"cell","content":[{"_key":"G20PFEyhSygq","_type":"block","children":[{"_key":"aUNro4hceapv","_type":"span","marks":[],"text":"17"}],"markDefs":[]}]},{"_key":"Rqto2kXRsPKe","_type":"cell","content":[{"_key":"87yERDiWKC39","_type":"block","children":[{"_key":"NRnpjLJAfx4z","_type":"span","marks":[],"text":"58"}],"markDefs":[]}]}]},{"_key":"x0hhvjPwPOqZ","cells":[{"_key":"DYQRrbOyT6Kt","_type":"cell","content":[{"_key":"IcNa2pjIx7AN","_type":"block","children":[{"_key":"DMUJYwDoVqgN","_type":"span","marks":[],"text":"4"}],"markDefs":[]}]},{"_key":"10cPeam1QVrC","_type":"cell","content":[{"_key":"YfUXbPdSLfpu","_type":"block","children":[{"_key":"CemhfIhEcqYj","_type":"span","marks":[],"text":"Great Britain"}],"markDefs":[]}]},{"_key":"keyMtpy3kMXX","_type":"cell","content":[{"_key":"DTaArbEatMCN","_type":"block","children":[{"_key":"Zi4UhXQAynUP","_type":"span","marks":[],"text":"22"}],"markDefs":[]}]},{"_key":"bENiu3QBAVl7","_type":"cell","content":[{"_key":"FdQ1ydAW4L7N","_type":"block","children":[{"_key":"7CW2HyEv6WPh","_type":"span","marks":[],"text":"21"}],"markDefs":[]}]},{"_key":"ZSK4v2h9QUmi","_type":"cell","content":[{"_key":"xDmk11ufvWz5","_type":"block","children":[{"_key":"2KsmRkZgkzj3","_type":"span","marks":[],"text":"22"}],"markDefs":[]}]},{"_key":"pVVzlsIAVOnv","_type":"cell","content":[{"_key":"rxT1lRnnUojq","_type":"block","children":[{"_key":"wXtMSNmHt7FY","_type":"span","marks":[],"text":"65"}],"markDefs":[]}]}]},{"_key":"G1xnSksvONCY","cells":[{"_key":"9cX2kFws72Ch","_type":"cell","content":[{"_key":"USvhFWddcrNh","_type":"block","children":[{"_key":"4P5kSfxqWYAp","_type":"span","marks":[],"text":"5"}],"markDefs":[]}]},{"_key":"ZJWn0dJ1nKSQ","_type":"cell","content":[{"_key":"F1EmFoFeSpFr","_type":"block","children":[{"_key":"5zDRuBTH2qGs","_type":"span","marks":[],"text":"ROC"}],"markDefs":[]}]},{"_key":"TmH7kfdERd9V","_type":"cell","content":[{"_key":"elWz3Z68toGn","_type":"block","children":[{"_key":"H94AZ0PVoJFm","_type":"span","marks":[],"text":"20"}],"markDefs":[]}]},{"_key":"qZlzxOgzIo9z","_type":"cell","content":[{"_key":"oPiCOhbdBIuT","_type":"block","children":[{"_key":"uwJIe66ayPM1","_type":"span","marks":[],"text":"28"}],"markDefs":[]}]},{"_key":"BOJEoZOTqiyW","_type":"cell","content":[{"_key":"Fbptqjvn78vr","_type":"block","children":[{"_key":"9VwkFl0mJU0Z","_type":"span","marks":[],"text":"23"}],"markDefs":[]}]},{"_key":"LxXMcv3jXHvm","_type":"cell","content":[{"_key":"rXYQNk2CXxMD","_type":"block","children":[{"_key":"5JqmlMCdFkBY","_type":"span","marks":[],"text":"71"}],"markDefs":[]}]}]}]},{"_key":"7a0623483d36","_type":"block.code","content":{"code":" df['Gold'].mean()","language":"python"}},{"_key":"7752323cfdd5","_type":"block.code","content":{"code":"3.6559139784946235","language":"text"}},{"_key":"6356fdab0209","_type":"block.code","content":{"code":" df['Total'].mean()","language":"python"}},{"_key":"e641eadc3dee","_type":"block.code","content":{"code":"11.612903225806452","language":"text"}},{"_key":"eef4a3eaa287","_type":"block","children":[{"_key":"eef4a3eaa2870","_type":"span","marks":[],"text":"Summary"}],"markDefs":[],"style":"h2"},{"_key":"37a326e000c8","_type":"block","children":[{"_key":"37a326e000c80","_type":"span","marks":[],"text":"We have seen how to parse an HTML document and make sense of the tags within it with the help of Beautiful Soup. You may want to use some of the things you have learned here to get some data that otherwise may be only available in a webpage. Please remember that you should be mindful of the rights of the material you are obtaining. Read the terms and conditions of the pages you are interested in, and if in doubt it is better to err on the side of caution. One last word - web scraping depends on the given structure of the webpages you are parsing. If the pages change, it is quite likely that your code will fail. In this case, be ready to roll up your sleeves, re-inspect the HTML tags, and fix your code accordingly."}],"markDefs":[],"style":"normal"}],"estimatedReadingTime":13,"featuredImage":{"_type":"image","alt":"","asset":{"_ref":"image-d6afcc6f162e247931befc97eb26f297b5f618dd-4985x2492-jpg","_type":"reference"}},"heroBackground":null,"locale":"en","newsletterCta":{"_type":"block.newsletterCta","buttonText":"Subscribe","darkMode":false,"disclaimer":"By submitting this form you agree to receive communications from Domino related to products and services in accordance with Domino's privacy policy and may opt-out at anytime.","emailInputPlaceholder":"Enter your email","errorButtonText":"Try again","errorSubtitle":"It seems that something went wrong. Please try again and check if your email address is correct.","errorTitle":"Something went wrong...","formId":"f468b5e4-92d6-4260-9d26-2dde6b786b96","subtitle":"Receive data science tips and tutorials from leading Data Science leaders, right to your inbox.","successSubtitle":"You successfully subscribed to the Domino Newsletter! Please check your email, we just sent a welcome message.","successTitle":"Congratulations!","title":"Subscribe to the Domino Newsletter","verticalPadding":"large"},"preamble":[{"_key":"3e7247f829bc","_type":"block","children":[{"_key":"3e7247f829bc0","_type":"span","marks":[],"text":"Data is all around us, from the spreadsheets we analyse on a daily basis, to the weather forecast we rely on every morning or the webpages we read. In many cases, the data we consume is simply given to us, and a simple glance is enough to make a decision. For example, knowing that the chance of rain today is 75% all day makes me take my umbrella with me. In many other cases, the data provided is so rich that we need to roll up our sleeves and we may use some exploratory analysis to get our heads around it. We have talked about some useful packages to do this exploration in "},{"_key":"3e7247f829bc1","_type":"span","marks":["0824b30ae3f0"],"text":"a previous post"},{"_key":"3e7247f829bc2","_type":"span","marks":[],"text":"."}],"markDefs":[{"_key":"0824b30ae3f0","_type":"link","deepLink":null,"eventName":null,"internalLink":{"_id":"o1Y7BUrMcHK7lLTrzH8DzU","_type":"blogPost","locale":"en","routePath":"blog/data-exploration-with-pandas-profiler-and-d-tale"},"linkType":"internal","newWindow":null,"url":null,"utmParameters":null,"videoPopup":null}],"style":"normal"}],"publishStatus":"public","publishedAt":"2021-09-30","recentPosts":[{"_id":"40b2bbe1-56cd-4083-b77d-a19d32d55da7","_type":"blogPost","author":{"_id":"77a6736f-2bd1-4248-adb1-c58f6d3369e4","_type":"blogPostAuthor","image":{"_type":"image","alt":"Nicholas Goble","asset":{"_ref":"image-eed0a5df1076c0f9d5e1af6061bd1c8bf851516e-1132x1374-png","_type":"reference"},"crop":{"_type":"sanity.imageCrop","bottom":0.16930746268656716,"left":0,"right":0,"top":0},"hotspot":{"_type":"sanity.imageHotspot","height":0.8306925373134328,"width":1,"x":0.5,"y":0.4153462686567164}},"locale":"en","name":"Nicholas Goble","routePath":"blog/author/nicholas-goble"},"estimatedReadingTime":5,"featuredImage":{"_type":"image","asset":{"_ref":"image-0f163cf16dab12f124ab363c1b928128d0ab310a-620x360-png","_type":"reference"}},"heroBackground":null,"internalLink":{"_id":"40b2bbe1-56cd-4083-b77d-a19d32d55da7","_type":"blogPost","locale":"en","routePath":"blog/what-to-do-about-sr-26-2"},"preamble":"SR 26-2 replaces checklist compliance with principles-based defensibility. That demands three things from an institution, in order:\n\nGet the organization right, because judgment-based supervision puts weight on people rather than checklists.\n\nGet the technology right, because principles only scale on a platform that can enforce them.","publishedAt":"2026-04-30","tag":"Finance","title":"What to do about SR 26-2"},{"_id":"49d8c9e5-045e-4ec1-ad2e-80953413cbf7","_type":"blogPost","author":{"_id":"04f92941-782f-48fd-8920-6d356c9680be","_type":"blogPostAuthor","image":{"_type":"image","alt":"Danny Stout","asset":{"_ref":"image-875700aedbd2cdd73a943e615f08cbf9e5484794-605x605-jpg","_type":"reference"}},"locale":"en","name":"Danny Stout","routePath":"blog/author/danny-stout"},"estimatedReadingTime":11,"featuredImage":{"_type":"image","alt":"Coding assistants","asset":{"_ref":"image-34b66a307c53469c6ff8643c807bb94276a3bf73-724x482-jpg","_type":"reference"}},"heroBackground":null,"internalLink":{"_id":"49d8c9e5-045e-4ec1-ad2e-80953413cbf7","_type":"blogPost","locale":"en","routePath":"blog/ai-coding-assistants-for-data-scientists"},"preamble":"There’s a moment most data science leaders run into, even if they don’t immediately recognize it. You invest in a platform to bring structure to how work gets done, standardizing development, improving collaboration, putting guardrails around deployment, and for the most part, it works. But over time, you start to notice that some of your strongest practitioners are doing their most effective work just slightly outside of it. Not abandoning the platform entirely, but bending around it in small ways to stay productive.\n\nKey Takeaways\n\nEnterprise data science platforms weren't designed for the coding assistant era. That friction is pushing practitioners outside the platform.","publishedAt":"2026-04-30","tag":"Agentic AI ","title":"Coding Assistants Are Reshaping How Data Science Gets Built. Your Platform Needs to Catch Up."},{"_id":"232127fd-679f-4039-a9da-52eaeebe1e17","_type":"blogPost","author":{"_id":"77a6736f-2bd1-4248-adb1-c58f6d3369e4","_type":"blogPostAuthor","image":{"_type":"image","alt":"Nicholas Goble","asset":{"_ref":"image-eed0a5df1076c0f9d5e1af6061bd1c8bf851516e-1132x1374-png","_type":"reference"},"crop":{"_type":"sanity.imageCrop","bottom":0.16930746268656716,"left":0,"right":0,"top":0},"hotspot":{"_type":"sanity.imageHotspot","height":0.8306925373134328,"width":1,"x":0.5,"y":0.4153462686567164}},"locale":"en","name":"Nicholas Goble","routePath":"blog/author/nicholas-goble"},"estimatedReadingTime":8,"featuredImage":{"_type":"image","asset":{"_ref":"image-5ab30a688a2abca03d9d22da91cb252fe1bfec4b-620x360-png","_type":"reference"}},"heroBackground":null,"internalLink":{"_id":"232127fd-679f-4039-a9da-52eaeebe1e17","_type":"blogPost","locale":"en","routePath":"blog/how-to-navigate-sr-26-2"},"preamble":"The regulator's job is to set expectations. The institution's job is to manage risk. SR 26-2 changed the first of these. It did not change the second. Exam findings still land where the risk is. Model failures still cause losses. Internal audit questions, board risk appetite reviews, conversations with regulators about decisions made years ago – all of these still happen, and the answers still have to be defensible on their own terms.\n\nA principles-based document is worth reading twice: once for what it says, and once for what it leaves unsaid. Everything in the second read is work sitting on the institution's side. Six takes cover where the regulation is silent, and how institutions will need to speak up.\n\nTake 1. The model vs. rules engine line is drawn by code type, not by risk","publishedAt":"2026-04-28","tag":"Finance","title":"How to navigate SR 26-2"}],"relatedCards":null,"relatedPosts":[{"_id":"bc7bbe5d-cb8e-4266-bb49-7b6b1b3e3698","_type":"blogPost","author":{"_id":"3JsSqb7vTVo8qxQw2sVR52","_type":"blogPostAuthor","image":{"_type":"image","alt":"Thomas Been","asset":{"_ref":"image-e4b05d5c8844bd7550e00d52b8a72da70af6ccd7-800x800-jpg","_type":"reference"}},"locale":"en","name":"Thomas Been","routePath":"blog/author/thomas-been"},"estimatedReadingTime":4,"featuredImage":{"_type":"image","asset":{"_ref":"image-146f32291cefd19de194f50e00a7a55f3fc1d069-770x460-png","_type":"reference"}},"heroBackground":null,"internalLink":{"_id":"bc7bbe5d-cb8e-4266-bb49-7b6b1b3e3698","_type":"blogPost","locale":"en","routePath":"blog/transform-ai-readiness-for-government-nvidia-gtc-2025"},"preamble":"According to MIT’s latest report, “The GenAI Divide: State of AI in Business 2025,” while only a small percentage of AI projects successfully reach operations, this creates an important opportunity for government agencies to focus on developing strategies that translate executive AI mandates, pilot programs, and innovation budgets into measurable, scalable results. Any deployment that doesn’t have the right operational strategy will waste both time and money — and AI is no different. At the same time, delays in AI deployments mean first responders and federal teams will endure capability gaps.\n\nThat’s where Domino and NVIDIA can help. Together, Domino and NVIDIA bridge the gaps on the missing layer in between experimentation and execution. It’s about governed, secure infrastructure that turns a model into a system, and ultimately a system into an operational asset. Leveraging both Domino and NVIDIA, your AI teams can build, deploy, and trust AI systems that are ready for the real world.\n\nInnovation with NVIDIA AI Factory for Government","publishedAt":"2025-10-28","tag":"Event","title":"Transform AI readiness into operational impact for government "},{"_id":"b90988ca-424f-455b-b7f4-34781dda991c","_type":"blogPost","author":{"_id":"bQvS44cO2Kqh2eurYwwIdo","_type":"blogPostAuthor","image":{"_type":"image","alt":"Domino","asset":{"_ref":"image-e26699206eea2085ee9ecf387d6093f58ff3dd91-200x200-png","_type":"reference"}},"locale":"en","name":"Domino","routePath":"blog/author/domino"},"estimatedReadingTime":10,"featuredImage":{"_type":"image","asset":{"_ref":"image-7d13c2d981af3f69440d41cd4c3caf7ab906de53-2957x1973-jpg","_type":"reference"}},"heroBackground":null,"internalLink":{"_id":"b90988ca-424f-455b-b7f4-34781dda991c","_type":"blogPost","locale":"en","routePath":"blog/the-white-house-ai-action-plan-a-data-science-roadmap-for-federal-agencies"},"preamble":"How the AI Action Plan helps solve federal data science’s biggest challenges\n\n\n\nFederal agencies face a familiar problem: it takes months to deploy AI models and even longer to update them. The U.S. Navy’s Project AMMO reduced its AI model deployment time from six months to six days, and retraining time from 12 months to just two weeks. This was made possible by leveraging cross-agency collaboration and shared infrastructure. This improved efficiency allows the system to adapt to evolving threats faster. Exactly what the White House AI Action Plan, released on July 23, 2025, wants to make standard across government.","publishedAt":"2025-08-21","tag":"Public Sector ","title":"The White House AI Action Plan: A data science roadmap for federal agencies "},{"_id":"a5a8c0ea-982f-42dd-91a4-22939ca68313","_type":"blogPost","author":{"_id":"66233a79-12ed-483d-9287-717ff7a4cd06","_type":"blogPostAuthor","image":{"_type":"image","asset":{"_ref":"image-fc6a2e5915217309b75c22e7d7d28ed1476245a9-300x300-jpg","_type":"reference"}},"locale":"en","name":"Etan Lightstone","routePath":"etan-lightstone"},"estimatedReadingTime":10,"featuredImage":{"_type":"image","asset":{"_ref":"image-fa28b4c0ba05c1535e6b0f1445d6bf0d465e2408-7132x4754-jpg","_type":"reference"}},"heroBackground":{"_type":"image","asset":{"_ref":"image-1548f7f3d56502568efd112707808e2d6585d9db-2400x1254-png","_type":"reference"}},"internalLink":{"_id":"a5a8c0ea-982f-42dd-91a4-22939ca68313","_type":"blogPost","locale":"en","routePath":"blog/what-is-vibe-modeling-code-less-analyze-more"},"preamble":"Vibe modeling: The future of high-impact data science\n\nIf you're a data scientist, you are likely already using AI coding tools to accelerate parts of your workflow. But as someone who has been focused on this for years, I’ve seen that writing the initial code is only a tiny part of the data science job. The real bottleneck, and where the most time is spent, is in the iterative process of experimentation. That process includes everything from finding, analyzing, and transforming data to running countless variations of a model. At Domino Data Lab, we’re passionate about serving as an accelerant to building AI models more quickly and safely, with over a decade of experience doing this for large enterprises.\n\nThis is where the paradigm shifts. Yes, we can use natural language to have a coding assistant generate a model architecture. But the real opportunity is in describing the predictive outcomes we want and involving an AI assistant in both the development and the experimentation and evaluation of our models. What if we could accelerate that entire process in a safe, reproducible, and enterprise-ready way? That is the vision behind vibe modeling, and it's what we are building in the open at Domino Data Lab.","publishedAt":"2025-07-10","tag":"Data Science","title":"What is vibe modeling? Code less, analyze more"}],"routePath":{"_type":"slug","current":"blog/getting-data-with-beautiful-soup"},"seo":{"description":"In this blog post, Dr J Rogel-Salazar walks us over the process of scraping websites for data using Python Beautiful Soup.","image":{"_type":"image","alt":"","asset":{"_ref":"image-d6afcc6f162e247931befc97eb26f297b5f618dd-4985x2492-jpg","_type":"reference"}},"title":"Web Scraping & Getting Data with Beautiful Soup | Domino"},"subtitle":null,"tags":[{"_id":"wN4plt0FLLjYOsFTYavDhL","_type":"tag","routePath":"data-science","title":"Data Science"},{"_id":"wN4plt0FLLjYOsFTYayoJv","_type":"tag","routePath":"code","title":"Code"},{"_id":"o1Y7BUrMcHK7lLTrz42u6J","_type":"tag","routePath":"data-engineering","title":"Data Engineering"},{"_id":"bQvS44cO2Kqh2eurYwpVyW","_type":"tag","routePath":"python","title":"Python"}],"title":"Getting Data with Beautiful Soup","translations":null,"typData":null},"settings":{"fallbackOgImage":{"_type":"ogImage","asset":{"_ref":"image-e4dceb2fb2141b67db834ca57ae439b9848ab51d-2400x1260-png","_type":"reference"}},"forms":{"disclaimer":"By submitting this form you agree to receive communications from Domino related to products and services in accordance with Domino's privacy policy and may opt-out at anytime.","emailInputPlaceholder":"Your email","errorButtonText":"Try again","errorSubtitle":"It seems that something went wrong. Please try again and check if your email address is correct. ","errorTitle":"Something went wrong... ","successSubtitle":"Please check your email, we just sent a welcome message!","successTitle":"Congratulations!"},"gtmContainerId":"GTM-N4Q7ZX4","labels":{"caseStudy":"Case Study","dominoNews":"Domino in the News","learnMore":"Learn more","pressReleases":"Press Releases","readLess":"Read Less","readMore":"Read More","readTheCaseStudy":"Read the Case Study"},"podcast":{"ctaTitle":"Get new episodes 
in your inbox","heroBackgroundImage":{"_type":"image","alt":"Podcast hero"},"heroHeader":"Podcast","summaryHeading":"About the Show"},"searchFilters":[{"_key":"f6f8004562a9","_type":"searchFilter","title":"Events","types":["event"]},{"_key":"4ebdedf9f834","_type":"searchFilter","title":"Resources","types":["resource"]},{"_key":"5523a11d3053","_type":"searchFilter","title":"Podcast","types":["podcastShow","podcastEpisode"]},{"_key":"ad4cffa70e4a","_type":"searchFilter","title":"Blog","types":["blogPost","blogPostAuthor"]}],"shareLinks":[{"_key":"ffcfedb6a012","_type":"shareLink","icon":{"_type":"image","asset":{"_ref":"image-3990faa9df52a6ca86fd82d22d91a31864d91991-29x29-svg","_type":"reference"}},"template":"https://www.facebook.com/share.php?u={CURRENT_URL}","title":"Facebook"},{"_key":"278e5e1c2924","_type":"shareLink","icon":{"_type":"image","asset":{"_ref":"image-7fdd5d0b59f965208ec834db1e73e11a7e884f98-29x29-svg","_type":"reference"}},"template":"https://twitter.com/intent/tweet?url={CURRENT_URL}","title":"Twitter"},{"_key":"e4a029518400","_type":"shareLink","icon":{"_type":"image","asset":{"_ref":"image-fbb9fe5b4586b8344f2ef2dd52f5b796919494a7-29x29-svg","_type":"reference"}},"template":"http://www.linkedin.com/shareArticle?mini=true&url={CURRENT_URL}","title":"LinkedIn"}],"structuredData":{"logo":{"_type":"image","asset":{"_ref":"image-946c04aaf7ec56b1a8e64fdbeb90751f8829330f-164x33-svg","_type":"reference"}}}}}},"actionData":null},"future":{"v2_dev":false,"unstable_postcss":false,"unstable_tailwind":false,"v2_errorBoundary":true,"v2_headers":true,"v2_meta":true,"v2_normalizeFormMethod":true,"v2_routeConvention":true}};