From a4767174e19630cf0411d18b8c30684b2449099c Mon Sep 17 00:00:00 2001 From: valeriapaksivatkina <157164927+valeriapaksivatkina@users.noreply.github.com> Date: Wed, 21 Feb 2024 22:53:33 +0300 Subject: [PATCH] Add files via upload --- lab_rss-main/README.md | 27 + lab_rss-main/your-code/Learning.ipynb | 839 ++++++++++++++++++ lab_rss-main/your-code/main.ipynb | 1140 +++++++++++++++++++++++++ 3 files changed, 2006 insertions(+) create mode 100644 lab_rss-main/README.md create mode 100644 lab_rss-main/your-code/Learning.ipynb create mode 100644 lab_rss-main/your-code/main.ipynb diff --git a/lab_rss-main/README.md b/lab_rss-main/README.md new file mode 100644 index 0000000..8b57970 --- /dev/null +++ b/lab_rss-main/README.md @@ -0,0 +1,27 @@ +![Ironhack logo](https://i.imgur.com/1QgrNNw.png) + +# Lab | Working with RSS Feeds + +## Introduction + +In the Working with RSS lesson, you learned how to parse an RSS feed with the feedparser library, explore and extract components of a feed, and perform some basic analyses on the information. + +In this lab, you will practice parsing RSS feeds with feedparser and extracting/analyzing the information contained within them. + +## Getting Started + +Open the `main.ipynb` file in the `your-code` directory. There are a bunch of questions to be solved. If you get stuck in one exercise you can skip to the next one. Read each instruction carefully and provide your answer beneath it. + +## Deliverables + +- `main.ipynb` with your responses to each of the exercises. + +## Submission + +Upon completion, add your deliverables to git. Then commit git and push your branch to the remote. + +## Resources + +- [Feedparser Documentation](https://pythonhosted.org/feedparser/) +- [Using Feedparser in Python | Python for Beginners](https://www.pythonforbeginners.com/feedparser/using-feedparser-in-python) +- [Python - Reading RSS Feed | TutorialsPoint](https://www.tutorialspoint.com/python/python_reading_rss_feed.htm) diff --git a/lab_rss-main/your-code/Learning.ipynb b/lab_rss-main/your-code/Learning.ipynb new file mode 100644 index 0000000..0b03387 --- /dev/null +++ b/lab_rss-main/your-code/Learning.ipynb @@ -0,0 +1,839 @@ +{ + "cells": [ + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "# Working with RSS Feeds\n", + "\n", + "\n", + "Lesson Goals\n", + "\n", + " Learn about the feedparser library\n", + " Use feedparser to parse RSS feeds\n", + " Explore the components of parsed RSS feeds\n", + " Convert results into data frames and conduct analysis\n", + "\n", + "Introduction\n", + "\n", + "In the previous lesson, we learned how to use Python to extract structured information from web APIs. In this lesson, we are going to take a look at another source of structured web content called RSS. RSS stands for Rich Site Summary or Really Simple Syndication, and it is a type of web feed which allows users and applications to access updates to online content in a standardized, computer-readable format (typically XML).\n", + "\n", + "Python has an excellent library called feedparser that is very useful for reading and parsing RSS feeds. We are going to be using this library throughout the lesson, so let's make sure it is installed and imported.\n", + "\n", + "$ pip install feedparser\n", + "\n" + ] + }, + { + "cell_type": "code", + "execution_count": 1, + "metadata": {}, + "outputs": [], + "source": [ + "import feedparser" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "# RSS Feed Versions Formats\n", + "\n", + "Due to the way web feeds evolved, there are various versions of RSS (0.9X, 1.0, 2.0, etc.) as well as alternate forms of feeds (Atom, CDF, etc.). We would have to worry about slight differences in formats if we were going to parse the feeds manually, but feedparser is able to handle all of them, so that is one less thing we need to worry about.\n", + "\n", + "\n", + "# Parsing RSS Feeds\n", + "\n", + "To parse an RSS feed with feedparser, you just need to call its parse method and pass it a URL. Let's take a look at an example using the RSS feed for the tech subreddit category. " + ] + }, + { + "cell_type": "code", + "execution_count": 2, + "metadata": {}, + "outputs": [], + "source": [ + "reddit = feedparser.parse('https://www.reddit.com/r/tech.rss')" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "If we take a look at the results, we will see a nested dictionary structure that contains a lot of information and looks something like the following." + ] + }, + { + "cell_type": "code", + "execution_count": 3, + "metadata": {}, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "{'tags': [{'term': 'tech', 'scheme': None, 'label': 'r/tech'}], 'updated': '2019-07-08T15:03:57+00:00', 'updated_parsed': time.struct_time(tm_year=2019, tm_mon=7, tm_mday=8, tm_hour=15, tm_min=3, tm_sec=57, tm_wday=0, tm_yday=189, tm_isdst=0), 'icon': 'https://www.redditstatic.com/icon.png/', 'id': 'https://www.reddit.com/r/tech.rss', 'guidislink': True, 'link': 'https://www.reddit.com/r/tech', 'links': [{'rel': 'self', 'href': 'https://www.reddit.com/r/tech.rss', 'type': 'application/atom+xml'}, {'rel': 'alternate', 'href': 'https://www.reddit.com/r/tech', 'type': 'text/html'}], 'logo': 'https://f.thumbs.redditmedia.com/kI7eGVG6kaObGTdM.png', 'subtitle': 'The goal of /r/tech is to provide a space dedicated to the intelligent discussion of innovations and changes to technology in our ever changing world. We focus on high quality news articles about technology and informative and thought provoking self posts.', 'subtitle_detail': {'type': 'text/plain', 'language': None, 'base': 'https://www.reddit.com/r/tech.rss', 'value': 'The goal of /r/tech is to provide a space dedicated to the intelligent discussion of innovations and changes to technology in our ever changing world. We focus on high quality news articles about technology and informative and thought provoking self posts.'}, 'title': '/r/tech: Technological innovations and changes.', 'title_detail': {'type': 'text/plain', 'language': None, 'base': 'https://www.reddit.com/r/tech.rss', 'value': '/r/tech: Technological innovations and changes.'}}\n" + ] + } + ], + "source": [ + "print(reddit['feed'])" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "This is great because we can now use what we learned earlier in the program about working with data structures to explore and extract the information we need from this.\n", + "\n", + "\n", + "\n", + "# Exploring the Parsed Feed\n", + "\n", + "Let's take a look at the first level of dictionary keys from the results and see what each of them looks like." + ] + }, + { + "cell_type": "code", + "execution_count": 4, + "metadata": {}, + "outputs": [ + { + "data": { + "text/plain": [ + "dict_keys(['feed', 'entries', 'bozo', 'headers', 'href', 'status', 'encoding', 'version', 'namespaces'])" + ] + }, + "execution_count": 4, + "metadata": {}, + "output_type": "execute_result" + } + ], + "source": [ + "reddit.keys()" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "These are the different components of the RSS feed, and each of them is going to contain information about something more specific. For example, feed is going to contain information about this Reddit RSS feed, entries is going to contain information about the specific entries in the feed, etc.\n", + "\n", + "Since the feed component is now structured as just a dictionary inside the larger dictionary, we can view its keys to get a sense of what type of information is available to us within the feed dictionary. " + ] + }, + { + "cell_type": "code", + "execution_count": 5, + "metadata": {}, + "outputs": [ + { + "data": { + "text/plain": [ + "dict_keys(['tags', 'updated', 'updated_parsed', 'icon', 'id', 'guidislink', 'link', 'links', 'logo', 'subtitle', 'subtitle_detail', 'title', 'title_detail'])" + ] + }, + "execution_count": 5, + "metadata": {}, + "output_type": "execute_result" + } + ], + "source": [ + "reddit.feed.keys()" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "Here, we can see that we would be able to extract any tags for the feed, when the feed was updated, and the icon image for the feed as well as the feed title, subtitle, and various other pieces of information about it. You can see what each of those looks like by calling each component from reddit.feed. " + ] + }, + { + "cell_type": "code", + "execution_count": 6, + "metadata": {}, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "[{'term': 'tech', 'scheme': None, 'label': 'r/tech'}]\n", + "\n", + "https://www.redditstatic.com/icon.png/\n", + "\n", + "/r/tech: Technological innovations and changes.\n", + "\n", + "The goal of /r/tech is to provide a space dedicated to the intelligent discussion of innovations and changes to technology in our ever changing world. We focus on high quality news articles about technology and informative and thought provoking self posts.\n" + ] + } + ], + "source": [ + "print (reddit.feed.tags)\n", + "print ('')\n", + "print (reddit.feed.icon)\n", + "print ('')\n", + "print (reddit.feed.title)\n", + "print ('')\n", + "print (reddit.feed.subtitle)" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "This is great, but the most interesting part of the feed is probably going to be the entries. We can access them as follows." + ] + }, + { + "cell_type": "code", + "execution_count": 7, + "metadata": {}, + "outputs": [ + { + "data": { + "text/plain": [ + "{'authors': [{'name': '/u/OriginalHoneyBadger',\n", + " 'href': 'https://www.reddit.com/user/OriginalHoneyBadger'}],\n", + " 'author_detail': {'name': '/u/OriginalHoneyBadger',\n", + " 'href': 'https://www.reddit.com/user/OriginalHoneyBadger'},\n", + " 'href': 'https://www.reddit.com/user/OriginalHoneyBadger',\n", + " 'author': '/u/OriginalHoneyBadger',\n", + " 'tags': [{'term': 'tech', 'scheme': None, 'label': 'r/tech'}],\n", + " 'content': [{'type': 'text/html',\n", + " 'language': None,\n", + " 'base': 'https://www.reddit.com/r/tech.rss',\n", + " 'value': '

Hey guys!

Thanks to /u/thonkerton, /r/tech has an official discord server.

The permanent join link is posted below, feel free to join and invite your friends.

Should you have any questions, concerns or suggestions do not hesitate to reach out!

Cheers!

https://discord.gg/tech

submitted by /u/OriginalHoneyBadger
[link] [comments]'}],\n", + " 'summary': '

Hey guys!

Thanks to /u/thonkerton, /r/tech has an official discord server.

The permanent join link is posted below, feel free to join and invite your friends.

Should you have any questions, concerns or suggestions do not hesitate to reach out!

Cheers!

https://discord.gg/tech

submitted by /u/OriginalHoneyBadger
[link] [comments]',\n", + " 'id': 'https://www.reddit.com/r/t3_7dx2ew',\n", + " 'guidislink': True,\n", + " 'link': 'https://www.reddit.com/r/tech/comments/7dx2ew/rtech_now_has_its_own_discord_server/',\n", + " 'links': [{'href': 'https://www.reddit.com/r/tech/comments/7dx2ew/rtech_now_has_its_own_discord_server/',\n", + " 'rel': 'alternate',\n", + " 'type': 'text/html'}],\n", + " 'updated': '2017-11-19T00:37:30+00:00',\n", + " 'updated_parsed': time.struct_time(tm_year=2017, tm_mon=11, tm_mday=19, tm_hour=0, tm_min=37, tm_sec=30, tm_wday=6, tm_yday=323, tm_isdst=0),\n", + " 'title': \"/r/Tech now has it's own Discord server!\",\n", + " 'title_detail': {'type': 'text/plain',\n", + " 'language': None,\n", + " 'base': 'https://www.reddit.com/r/tech.rss',\n", + " 'value': \"/r/Tech now has it's own Discord server!\"}}" + ] + }, + "execution_count": 7, + "metadata": {}, + "output_type": "execute_result" + } + ], + "source": [ + "reddit.entries[0]" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "We can see that the data structure within this seems to be a list where each entry is an element that contains a dictionary with the information for each entry. We can access the individual entries via indexing and then we can look at the keys available for the entry by calling the keys() method. " + ] + }, + { + "cell_type": "code", + "execution_count": 8, + "metadata": {}, + "outputs": [ + { + "data": { + "text/plain": [ + "dict_keys(['authors', 'author_detail', 'href', 'author', 'tags', 'content', 'summary', 'id', 'guidislink', 'link', 'links', 'updated', 'updated_parsed', 'title', 'title_detail'])" + ] + }, + "execution_count": 8, + "metadata": {}, + "output_type": "execute_result" + } + ], + "source": [ + "reddit.entries[0].keys()" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "If we wanted to obtain a particular piece of data for an entry, we could just index that entry and then call the key for the information we wanted. For example, if we wanted to get the title of the third entry, we would obtain it as follows.\n", + "\n" + ] + }, + { + "cell_type": "code", + "execution_count": 9, + "metadata": {}, + "outputs": [ + { + "data": { + "text/plain": [ + "'Nintendo president: ‘we must keep up’ with cloud gaming tech'" + ] + }, + "execution_count": 9, + "metadata": {}, + "output_type": "execute_result" + } + ], + "source": [ + "reddit.entries[2].title" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "To extract the titles for all the entries, we could use a list comprehension." + ] + }, + { + "cell_type": "code", + "execution_count": 10, + "metadata": {}, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "[\"/r/Tech now has it's own Discord server!\", \"FBI and ICE use DMV photos as 'gold mine' for facial recognition data.\", 'Nintendo president: ‘we must keep up’ with cloud gaming tech', \"Volkswagen's electric ID R race car broke its own Goodwood Hill Climb record\", 'Now you can check in on Bill Nye’s solar sail as it orbits Earth', 'Apple tests iCloud.com sign-ins with your face or finger', 'Huawei staff share deep links with Chinese military, new study claims-', 'Hype or hope? How important is blockchain?', \"Gmail Tracks Everything You Purchase and You Can't Delete It or Opt Out | Digital Trends\", 'is bandwidth(regarding wifi) inversely proportional to the effective range of network?', 'A portable Bluetooth cassette tape player brings retro audio into 2019', 'Idea for gaming laptop!', \"Twitter's Disinformation Data Dumps Are Helpful—to a Point\", 'Industrial Uses of Connectors And Custom Cable Assemblies', 'You can explore mazes of mirrors and light at teamLab Tokyo', '7-Eleven Japan shut down its mobile payment app after hackers stole $500,000 from users', 'AMD Launches 7nm Ryzen 3000 CPUs & Radeon RX 5700 GPUs', 'Formula E’s electric off-road SUV is an absolute unit', 'Wimbledon reworks AI tech to reduce bias in game highlights', 'Nio CEO cautiously eyes Europe entry for Chinese EV startup', 'Tesla shows off next-gen automated emergency braking stopping for pedestrians and cyclists', \"How Solar Panels Work (And Why They're Taking Over the World)\", 'DNA Data Storage Is Closer Than You Think', 'Google is testing a play button for Chrome’s toolbar', 'Google still keeps a list of everything you ever bought using Gmail, even if you delete all your emails.', 'Samsung files a new patent for a foldable display']\n" + ] + } + ], + "source": [ + "titles = [reddit.entries[i].title for i in range(len(reddit.entries))]\n", + "print(titles)" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "# Analyzing Data From an RSS Feed\n", + "\n", + "Thus far, feedparser has helped us obtain data from an RSS feed and structure in a way that makes it easy for us to explore it and extract the information we need. If we wanted to analyze the data further, we could leverage the Pandas library and create a data frame containing the information about entries in the feed." + ] + }, + { + "cell_type": "code", + "execution_count": 11, + "metadata": {}, + "outputs": [ + { + "data": { + "text/html": [ + "
\n", + "\n", + "\n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + "
authorauthor_detailauthorscontentguidislinkhrefidlinklinkssummarytagstitletitle_detailupdatedupdated_parsed
0/u/OriginalHoneyBadger{'name': '/u/OriginalHoneyBadger', 'href': 'ht...[{'name': '/u/OriginalHoneyBadger', 'href': 'h...[{'type': 'text/html', 'language': None, 'base...Truehttps://www.reddit.com/user/OriginalHoneyBadgerhttps://www.reddit.com/r/t3_7dx2ewhttps://www.reddit.com/r/tech/comments/7dx2ew/...[{'href': 'https://www.reddit.com/r/tech/comme...<!-- SC_OFF --><div class=\"md\"><p>Hey guys!</p...[{'term': 'tech', 'scheme': None, 'label': 'r/.../r/Tech now has it's own Discord server!{'type': 'text/plain', 'language': None, 'base...2017-11-19T00:37:30+00:00(2017, 11, 19, 0, 37, 30, 6, 323, 0)
1/u/dfc76{'name': '/u/dfc76', 'href': 'https://www.redd...[{'name': '/u/dfc76', 'href': 'https://www.red...[{'type': 'text/html', 'language': None, 'base...Truehttps://www.reddit.com/user/dfc76https://www.reddit.com/r/t3_cae8rbhttps://www.reddit.com/r/tech/comments/cae8rb/...[{'href': 'https://www.reddit.com/r/tech/comme...&#32; submitted by &#32; <a href=\"https://www....[{'term': 'tech', 'scheme': None, 'label': 'r/...FBI and ICE use DMV photos as 'gold mine' for ...{'type': 'text/plain', 'language': None, 'base...2019-07-08T00:38:26+00:00(2019, 7, 8, 0, 38, 26, 0, 189, 0)
2/u/sdblro{'name': '/u/sdblro', 'href': 'https://www.red...[{'name': '/u/sdblro', 'href': 'https://www.re...[{'type': 'text/html', 'language': None, 'base...Truehttps://www.reddit.com/user/sdblrohttps://www.reddit.com/r/t3_ca5870https://www.reddit.com/r/tech/comments/ca5870/...[{'href': 'https://www.reddit.com/r/tech/comme...&#32; submitted by &#32; <a href=\"https://www....[{'term': 'tech', 'scheme': None, 'label': 'r/...Nintendo president: ‘we must keep up’ with clo...{'type': 'text/plain', 'language': None, 'base...2019-07-07T09:56:42+00:00(2019, 7, 7, 9, 56, 42, 6, 188, 0)
3/u/ourlifeintoronto{'name': '/u/ourlifeintoronto', 'href': 'https...[{'name': '/u/ourlifeintoronto', 'href': 'http...[{'type': 'text/html', 'language': None, 'base...Truehttps://www.reddit.com/user/ourlifeintorontohttps://www.reddit.com/r/t3_ca7r1fhttps://www.reddit.com/r/tech/comments/ca7r1f/...[{'href': 'https://www.reddit.com/r/tech/comme...&#32; submitted by &#32; <a href=\"https://www....[{'term': 'tech', 'scheme': None, 'label': 'r/...Volkswagen's electric ID R race car broke its ...{'type': 'text/plain', 'language': None, 'base...2019-07-07T15:02:41+00:00(2019, 7, 7, 15, 2, 41, 6, 188, 0)
4/u/tozameer{'name': '/u/tozameer', 'href': 'https://www.r...[{'name': '/u/tozameer', 'href': 'https://www....[{'type': 'text/html', 'language': None, 'base...Truehttps://www.reddit.com/user/tozameerhttps://www.reddit.com/r/t3_ca7b38https://www.reddit.com/r/tech/comments/ca7b38/...[{'href': 'https://www.reddit.com/r/tech/comme...&#32; submitted by &#32; <a href=\"https://www....[{'term': 'tech', 'scheme': None, 'label': 'r/...Now you can check in on Bill Nye’s solar sail ...{'type': 'text/plain', 'language': None, 'base...2019-07-07T14:19:42+00:00(2019, 7, 7, 14, 19, 42, 6, 188, 0)
\n", + "
" + ], + "text/plain": [ + " author author_detail \\\n", + "0 /u/OriginalHoneyBadger {'name': '/u/OriginalHoneyBadger', 'href': 'ht... \n", + "1 /u/dfc76 {'name': '/u/dfc76', 'href': 'https://www.redd... \n", + "2 /u/sdblro {'name': '/u/sdblro', 'href': 'https://www.red... \n", + "3 /u/ourlifeintoronto {'name': '/u/ourlifeintoronto', 'href': 'https... \n", + "4 /u/tozameer {'name': '/u/tozameer', 'href': 'https://www.r... \n", + "\n", + " authors \\\n", + "0 [{'name': '/u/OriginalHoneyBadger', 'href': 'h... \n", + "1 [{'name': '/u/dfc76', 'href': 'https://www.red... \n", + "2 [{'name': '/u/sdblro', 'href': 'https://www.re... \n", + "3 [{'name': '/u/ourlifeintoronto', 'href': 'http... \n", + "4 [{'name': '/u/tozameer', 'href': 'https://www.... \n", + "\n", + " content guidislink \\\n", + "0 [{'type': 'text/html', 'language': None, 'base... True \n", + "1 [{'type': 'text/html', 'language': None, 'base... True \n", + "2 [{'type': 'text/html', 'language': None, 'base... True \n", + "3 [{'type': 'text/html', 'language': None, 'base... True \n", + "4 [{'type': 'text/html', 'language': None, 'base... True \n", + "\n", + " href \\\n", + "0 https://www.reddit.com/user/OriginalHoneyBadger \n", + "1 https://www.reddit.com/user/dfc76 \n", + "2 https://www.reddit.com/user/sdblro \n", + "3 https://www.reddit.com/user/ourlifeintoronto \n", + "4 https://www.reddit.com/user/tozameer \n", + "\n", + " id \\\n", + "0 https://www.reddit.com/r/t3_7dx2ew \n", + "1 https://www.reddit.com/r/t3_cae8rb \n", + "2 https://www.reddit.com/r/t3_ca5870 \n", + "3 https://www.reddit.com/r/t3_ca7r1f \n", + "4 https://www.reddit.com/r/t3_ca7b38 \n", + "\n", + " link \\\n", + "0 https://www.reddit.com/r/tech/comments/7dx2ew/... \n", + "1 https://www.reddit.com/r/tech/comments/cae8rb/... \n", + "2 https://www.reddit.com/r/tech/comments/ca5870/... \n", + "3 https://www.reddit.com/r/tech/comments/ca7r1f/... \n", + "4 https://www.reddit.com/r/tech/comments/ca7b38/... \n", + "\n", + " links \\\n", + "0 [{'href': 'https://www.reddit.com/r/tech/comme... \n", + "1 [{'href': 'https://www.reddit.com/r/tech/comme... \n", + "2 [{'href': 'https://www.reddit.com/r/tech/comme... \n", + "3 [{'href': 'https://www.reddit.com/r/tech/comme... \n", + "4 [{'href': 'https://www.reddit.com/r/tech/comme... \n", + "\n", + " summary \\\n", + "0

Hey guys!\n", + "\n", + "\n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + "
authorentries
10/u/ourlifeintoronto5
12/u/tozameer4
1/u/JackFisherBooks2
2/u/Md-Sabbir-Howlader2
6/u/bhaktbhagwan2
8/u/cryptoz2
0/u/FirebaseLearner1
3/u/OriginalHoneyBadger1
4/u/RenatoZX1
5/u/adam-tech1
7/u/chrisk20001
9/u/dfc761
11/u/sdblro1
13/u/trueslicky1
14/u/zinc--1
\n", + "

" + ], + "text/plain": [ + " author entries\n", + "10 /u/ourlifeintoronto 5\n", + "12 /u/tozameer 4\n", + "1 /u/JackFisherBooks 2\n", + "2 /u/Md-Sabbir-Howlader 2\n", + "6 /u/bhaktbhagwan 2\n", + "8 /u/cryptoz 2\n", + "0 /u/FirebaseLearner 1\n", + "3 /u/OriginalHoneyBadger 1\n", + "4 /u/RenatoZX 1\n", + "5 /u/adam-tech 1\n", + "7 /u/chrisk2000 1\n", + "9 /u/dfc76 1\n", + "11 /u/sdblro 1\n", + "13 /u/trueslicky 1\n", + "14 /u/zinc-- 1" + ] + }, + "execution_count": 12, + "metadata": {}, + "output_type": "execute_result" + } + ], + "source": [ + "authors = df.groupby('author', as_index=False).agg({'title':'count'})\n", + "authors.columns = ['author', 'entries']\n", + "authors.sort_values('entries', ascending=False)" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "Similarly, if we wanted to see which entries had the longest titles, we could create a new column called title_length that contains the number of characters in the title and then sort the data frame by that new column. " + ] + }, + { + "cell_type": "code", + "execution_count": 13, + "metadata": {}, + "outputs": [ + { + "data": { + "text/html": [ + "
\n", + "\n", + "\n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + "
titleauthortitle_length
24Google still keeps a list of everything you ev.../u/Md-Sabbir-Howlader104
20Tesla shows off next-gen automated emergency b.../u/cryptoz90
8Gmail Tracks Everything You Purchase and You C.../u/bhaktbhagwan88
157-Eleven Japan shut down its mobile payment ap.../u/trueslicky87
9is bandwidth(regarding wifi) inversely proport.../u/FirebaseLearner86
\n", + "
" + ], + "text/plain": [ + " title author \\\n", + "24 Google still keeps a list of everything you ev... /u/Md-Sabbir-Howlader \n", + "20 Tesla shows off next-gen automated emergency b... /u/cryptoz \n", + "8 Gmail Tracks Everything You Purchase and You C... /u/bhaktbhagwan \n", + "15 7-Eleven Japan shut down its mobile payment ap... /u/trueslicky \n", + "9 is bandwidth(regarding wifi) inversely proport... /u/FirebaseLearner \n", + "\n", + " title_length \n", + "24 104 \n", + "20 90 \n", + "8 88 \n", + "15 87 \n", + "9 86 " + ] + }, + "execution_count": 13, + "metadata": {}, + "output_type": "execute_result" + } + ], + "source": [ + "df['title_length'] = df['title'].apply(len)\n", + "df[['title', 'author', 'title_length']].sort_values('title_length', ascending=False).head()" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "These are just a couple of the things you can analyze about the entries using the information we were able to obtain." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": {}, + "outputs": [], + "source": [] + } + ], + "metadata": { + "kernelspec": { + "display_name": "Python 3", + "language": "python", + "name": "python3" + }, + "language_info": { + "codemirror_mode": { + "name": "ipython", + "version": 3 + }, + "file_extension": ".py", + "mimetype": "text/x-python", + "name": "python", + "nbconvert_exporter": "python", + "pygments_lexer": "ipython3", + "version": "3.7.2" + } + }, + "nbformat": 4, + "nbformat_minor": 2 +} diff --git a/lab_rss-main/your-code/main.ipynb b/lab_rss-main/your-code/main.ipynb new file mode 100644 index 0000000..31b5841 --- /dev/null +++ b/lab_rss-main/your-code/main.ipynb @@ -0,0 +1,1140 @@ +{ + "cells": [ + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "# Working with RSS Feeds Lab\n", + "\n", + "Complete the following set of exercises to solidify your knowledge of parsing RSS feeds and extracting information from them." + ] + }, + { + "cell_type": "code", + "execution_count": 2, + "metadata": {}, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "Requirement already satisfied: feedparser in /Users/paksivatkinavaleria/anaconda3/lib/python3.11/site-packages (6.0.11)\n", + "Requirement already satisfied: sgmllib3k in /Users/paksivatkinavaleria/anaconda3/lib/python3.11/site-packages (from feedparser) (1.0.0)\n", + "Note: you may need to restart the kernel to use updated packages.\n" + ] + } + ], + "source": [ + "pip install feedparser" + ] + }, + { + "cell_type": "code", + "execution_count": 3, + "metadata": {}, + "outputs": [], + "source": [ + "import feedparser" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "### 1. Use feedparser to parse the following RSS feed URL." + ] + }, + { + "cell_type": "code", + "execution_count": 4, + "metadata": {}, + "outputs": [], + "source": [ + "url = 'https://www.nasa.gov/news-release/feed/'" + ] + }, + { + "cell_type": "code", + "execution_count": 5, + "metadata": {}, + "outputs": [], + "source": [ + "nasa = feedparser.parse(url)" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "### 2. Obtain a list of components (keys) that are available for this feed." + ] + }, + { + "cell_type": "code", + "execution_count": 6, + "metadata": {}, + "outputs": [ + { + "data": { + "text/plain": [ + "dict_keys(['bozo', 'entries', 'feed', 'headers', 'etag', 'updated', 'updated_parsed', 'href', 'status', 'encoding', 'version', 'namespaces'])" + ] + }, + "execution_count": 6, + "metadata": {}, + "output_type": "execute_result" + } + ], + "source": [ + "nasa.keys()" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "### 3. Obtain a list of components (keys) that are available for the *feed* component of this RSS feed." + ] + }, + { + "cell_type": "code", + "execution_count": 7, + "metadata": {}, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "{'title': 'NASA', 'title_detail': {'type': 'text/plain', 'language': None, 'base': 'https://www.nasa.gov/news-release/feed/', 'value': 'NASA'}, 'links': [{'href': 'https://www.nasa.gov/news-release/feed/', 'rel': 'self', 'type': 'application/rss+xml'}, {'rel': 'alternate', 'type': 'text/html', 'href': 'https://www.nasa.gov'}], 'link': 'https://www.nasa.gov', 'subtitle': 'Official National Aeronautics and Space Administration Website', 'subtitle_detail': {'type': 'text/html', 'language': None, 'base': 'https://www.nasa.gov/news-release/feed/', 'value': 'Official National Aeronautics and Space Administration Website'}, 'updated': 'Wed, 21 Feb 2024 19:05:57 +0000', 'updated_parsed': time.struct_time(tm_year=2024, tm_mon=2, tm_mday=21, tm_hour=19, tm_min=5, tm_sec=57, tm_wday=2, tm_yday=52, tm_isdst=0), 'language': 'en-US', 'sy_updateperiod': 'hourly', 'sy_updatefrequency': '1', 'generator_detail': {'name': 'https://wordpress.org/?v=6.3.3'}, 'generator': 'https://wordpress.org/?v=6.3.3'}\n" + ] + } + ], + "source": [ + "print(nasa['feed'])" + ] + }, + { + "cell_type": "code", + "execution_count": 8, + "metadata": {}, + "outputs": [ + { + "data": { + "text/plain": [ + "dict_keys(['title', 'title_detail', 'links', 'link', 'subtitle', 'subtitle_detail', 'updated', 'updated_parsed', 'language', 'sy_updateperiod', 'sy_updatefrequency', 'generator_detail', 'generator'])" + ] + }, + "execution_count": 8, + "metadata": {}, + "output_type": "execute_result" + } + ], + "source": [ + "nasa.feed.keys()" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "### 4. Extract and print the feed title, subtitle, author, and link." + ] + }, + { + "cell_type": "code", + "execution_count": 9, + "metadata": {}, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "NASA\n", + "\n", + "Official National Aeronautics and Space Administration Website\n", + "\n", + "https://wordpress.org/?v=6.3.3\n", + "\n", + "https://www.nasa.gov\n" + ] + } + ], + "source": [ + "print (nasa.feed.title)\n", + "print ('')\n", + "print (nasa.feed.subtitle)\n", + "print ('')\n", + "print (nasa.feed.generator)\n", + "print ('')\n", + "print (nasa.feed.link)" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "### 5. Count the number of entries that are contained in this RSS feed." + ] + }, + { + "cell_type": "code", + "execution_count": 10, + "metadata": {}, + "outputs": [ + { + "data": { + "text/plain": [ + "10" + ] + }, + "execution_count": 10, + "metadata": {}, + "output_type": "execute_result" + } + ], + "source": [ + "len(nasa.entries)" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "### 6. Obtain a list of components (keys) available for an entry.\n", + "\n", + "*Hint: Remember to index first before requesting the keys*" + ] + }, + { + "cell_type": "code", + "execution_count": 11, + "metadata": {}, + "outputs": [ + { + "data": { + "text/plain": [ + "dict_keys(['title', 'title_detail', 'links', 'link', 'authors', 'author', 'author_detail', 'published', 'published_parsed', 'tags', 'id', 'guidislink', 'summary', 'summary_detail', 'content', 'media_content', 'media_player', 'media_thumbnail', 'href', 'media_rating', 'rating'])" + ] + }, + "execution_count": 11, + "metadata": {}, + "output_type": "execute_result" + } + ], + "source": [ + "nasa.entries[0].keys()" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "### 7. Extract a list of entry titles." + ] + }, + { + "cell_type": "code", + "execution_count": 12, + "metadata": {}, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "['Stellar Beads on a String', 'NASA Astronomer Sees Power in Community, Works to Build More', 'Ride the Wave of Radio Astronomy During the Solar Eclipse', 'NASA Selects University Teams to Explore Innovative Aeronautical Research', 'Become a SunSketcher, and Help Measure the Shape of the Sun!', 'NASA Sets Coverage of First US Uncrewed Commercial Moon Landing', 'Seeing is Communicating', 'Annual Highlights of Results 2023: Introduction and Analyses', '2023 Annual Highlights of Results from the International Space Station', 'Renee King: Ensuring Space for Everyone']\n" + ] + } + ], + "source": [ + "titles = [nasa.entries[i].title for i in range(len(nasa.entries))]\n", + "print(titles)" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "### 8. Calculate the percentage of \"Four short links\" entry titles." + ] + }, + { + "cell_type": "code", + "execution_count": 13, + "metadata": {}, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "Object `??` not found.\n" + ] + } + ], + "source": [ + "????" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "### 9. Create a Pandas data frame from the feed's entries." + ] + }, + { + "cell_type": "code", + "execution_count": 21, + "metadata": {}, + "outputs": [ + { + "data": { + "text/html": [ + "
\n", + "\n", + "\n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + "
titlelinkslinkauthorsauthorpublishedpublished_parsedtagsidguidislink...title_detail.valueauthor_detail.namesummary_detail.typesummary_detail.languagesummary_detail.basesummary_detail.valuemedia_player.urlmedia_player.contentmedia_rating.schememedia_rating.content
0Stellar Beads on a String[{'rel': 'alternate', 'type': 'text/html', 'hr...https://www.nasa.gov/image-article/stellar-bea...[{'name': 'Lee Mohon'}]Lee MohonWed, 21 Feb 2024 17:25:50 +0000(2024, 2, 21, 17, 25, 50, 2, 52, 0)[{'term': 'Astrophysics', 'scheme': None, 'lab...https://www.nasa.gov/?post_type=image-article&...False...Stellar Beads on a StringLee Mohontext/htmlNonehttps://www.nasa.gov/news-release/feed/Astronomers have discovered one of the most po...https://www.youtube.com/embed/bouQkHDXMKAurn:simplenonadult
1NASA Astronomer Sees Power in Community, Works...[{'rel': 'alternate', 'type': 'text/html', 'hr...https://www.nasa.gov/general/nasa-astronomer-s...[{'name': 'Abby Tabor'}]Abby TaborWed, 21 Feb 2024 17:22:18 +0000(2024, 2, 21, 17, 22, 18, 2, 52, 0)[{'term': 'General', 'scheme': None, 'label': ...https://www.nasa.gov/?p=616846False...NASA Astronomer Sees Power in Community, Works...Abby Tabortext/htmlNonehttps://www.nasa.gov/news-release/feed/Science is often portrayed as a solitary affai...NaNNaNNaNNaN
2Ride the Wave of Radio Astronomy During the So...[{'rel': 'alternate', 'type': 'text/html', 'hr...https://science.nasa.gov/solar-system/skywatch...[{}]Wed, 21 Feb 2024 16:24:06 +0000(2024, 2, 21, 16, 24, 6, 2, 52, 0)[{'term': '2024 Solar Eclipse', 'scheme': None...https://science.nasa.gov/solar-system/skywatch...False...Ride the Wave of Radio Astronomy During the So...NaNtext/htmlNonehttps://www.nasa.gov/news-release/feed/Students and science enthusiasts are invited t...NaNNaNNaNNaN
3NASA Selects University Teams to Explore Innov...[{'rel': 'alternate', 'type': 'text/html', 'hr...https://www.nasa.gov/aeronautics/university-te...[{'name': 'Jim Banke'}]Jim BankeWed, 21 Feb 2024 05:50:33 +0000(2024, 2, 21, 5, 50, 33, 2, 52, 0)[{'term': 'Aeronautics', 'scheme': None, 'labe...https://www.nasa.gov/?p=617878False...NASA Selects University Teams to Explore Innov...Jim Banketext/htmlNonehttps://www.nasa.gov/news-release/feed/NASA has selected another five university team...NaNNaNNaNNaN
4Become a SunSketcher, and Help Measure the Sha...[{'rel': 'alternate', 'type': 'text/html', 'hr...https://science.nasa.gov/get-involved/citizen-...[{}]Tue, 20 Feb 2024 21:35:49 +0000(2024, 2, 20, 21, 35, 49, 1, 51, 0)[{'term': 'Citizen Science', 'scheme': None, '...https://science.nasa.gov/get-involved/citizen-...False...Become a SunSketcher, and Help Measure the Sha...NaNtext/htmlNonehttps://www.nasa.gov/news-release/feed/What shape Is the Sun? Hint: it’s not perfectl...NaNNaNNaNNaN
\n", + "

5 rows × 29 columns

\n", + "
" + ], + "text/plain": [ + " title \\\n", + "0 Stellar Beads on a String \n", + "1 NASA Astronomer Sees Power in Community, Works... \n", + "2 Ride the Wave of Radio Astronomy During the So... \n", + "3 NASA Selects University Teams to Explore Innov... \n", + "4 Become a SunSketcher, and Help Measure the Sha... \n", + "\n", + " links \\\n", + "0 [{'rel': 'alternate', 'type': 'text/html', 'hr... \n", + "1 [{'rel': 'alternate', 'type': 'text/html', 'hr... \n", + "2 [{'rel': 'alternate', 'type': 'text/html', 'hr... \n", + "3 [{'rel': 'alternate', 'type': 'text/html', 'hr... \n", + "4 [{'rel': 'alternate', 'type': 'text/html', 'hr... \n", + "\n", + " link \\\n", + "0 https://www.nasa.gov/image-article/stellar-bea... \n", + "1 https://www.nasa.gov/general/nasa-astronomer-s... \n", + "2 https://science.nasa.gov/solar-system/skywatch... \n", + "3 https://www.nasa.gov/aeronautics/university-te... \n", + "4 https://science.nasa.gov/get-involved/citizen-... \n", + "\n", + " authors author published \\\n", + "0 [{'name': 'Lee Mohon'}] Lee Mohon Wed, 21 Feb 2024 17:25:50 +0000 \n", + "1 [{'name': 'Abby Tabor'}] Abby Tabor Wed, 21 Feb 2024 17:22:18 +0000 \n", + "2 [{}] Wed, 21 Feb 2024 16:24:06 +0000 \n", + "3 [{'name': 'Jim Banke'}] Jim Banke Wed, 21 Feb 2024 05:50:33 +0000 \n", + "4 [{}] Tue, 20 Feb 2024 21:35:49 +0000 \n", + "\n", + " published_parsed \\\n", + "0 (2024, 2, 21, 17, 25, 50, 2, 52, 0) \n", + "1 (2024, 2, 21, 17, 22, 18, 2, 52, 0) \n", + "2 (2024, 2, 21, 16, 24, 6, 2, 52, 0) \n", + "3 (2024, 2, 21, 5, 50, 33, 2, 52, 0) \n", + "4 (2024, 2, 20, 21, 35, 49, 1, 51, 0) \n", + "\n", + " tags \\\n", + "0 [{'term': 'Astrophysics', 'scheme': None, 'lab... \n", + "1 [{'term': 'General', 'scheme': None, 'label': ... \n", + "2 [{'term': '2024 Solar Eclipse', 'scheme': None... \n", + "3 [{'term': 'Aeronautics', 'scheme': None, 'labe... \n", + "4 [{'term': 'Citizen Science', 'scheme': None, '... \n", + "\n", + " id guidislink ... \\\n", + "0 https://www.nasa.gov/?post_type=image-article&... False ... \n", + "1 https://www.nasa.gov/?p=616846 False ... \n", + "2 https://science.nasa.gov/solar-system/skywatch... False ... \n", + "3 https://www.nasa.gov/?p=617878 False ... \n", + "4 https://science.nasa.gov/get-involved/citizen-... False ... \n", + "\n", + " title_detail.value author_detail.name \\\n", + "0 Stellar Beads on a String Lee Mohon \n", + "1 NASA Astronomer Sees Power in Community, Works... Abby Tabor \n", + "2 Ride the Wave of Radio Astronomy During the So... NaN \n", + "3 NASA Selects University Teams to Explore Innov... Jim Banke \n", + "4 Become a SunSketcher, and Help Measure the Sha... NaN \n", + "\n", + " summary_detail.type summary_detail.language \\\n", + "0 text/html None \n", + "1 text/html None \n", + "2 text/html None \n", + "3 text/html None \n", + "4 text/html None \n", + "\n", + " summary_detail.base \\\n", + "0 https://www.nasa.gov/news-release/feed/ \n", + "1 https://www.nasa.gov/news-release/feed/ \n", + "2 https://www.nasa.gov/news-release/feed/ \n", + "3 https://www.nasa.gov/news-release/feed/ \n", + "4 https://www.nasa.gov/news-release/feed/ \n", + "\n", + " summary_detail.value \\\n", + "0 Astronomers have discovered one of the most po... \n", + "1 Science is often portrayed as a solitary affai... \n", + "2 Students and science enthusiasts are invited t... \n", + "3 NASA has selected another five university team... \n", + "4 What shape Is the Sun? Hint: it’s not perfectl... \n", + "\n", + " media_player.url media_player.content \\\n", + "0 https://www.youtube.com/embed/bouQkHDXMKA \n", + "1 NaN NaN \n", + "2 NaN NaN \n", + "3 NaN NaN \n", + "4 NaN NaN \n", + "\n", + " media_rating.scheme media_rating.content \n", + "0 urn:simple nonadult \n", + "1 NaN NaN \n", + "2 NaN NaN \n", + "3 NaN NaN \n", + "4 NaN NaN \n", + "\n", + "[5 rows x 29 columns]" + ] + }, + "execution_count": 21, + "metadata": {}, + "output_type": "execute_result" + } + ], + "source": [ + "import pandas as pd\n", + "\n", + "df = pd.DataFrame(pd.json_normalize(nasa.entries))\n", + "df.head()" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": {}, + "outputs": [], + "source": [] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "### 10. Count the number of entries per author and sort them in descending order." + ] + }, + { + "cell_type": "code", + "execution_count": 24, + "metadata": {}, + "outputs": [ + { + "data": { + "text/html": [ + "
\n", + "\n", + "\n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + "
authorpublished
02
2Ana Guzman2
1Abby Tabor1
3Andrew Wagner1
4Jim Banke1
5Lee Mohon1
6Madison Olson1
7Tiernan P. Doyle1
\n", + "
" + ], + "text/plain": [ + " author published\n", + "0 2\n", + "2 Ana Guzman 2\n", + "1 Abby Tabor 1\n", + "3 Andrew Wagner 1\n", + "4 Jim Banke 1\n", + "5 Lee Mohon 1\n", + "6 Madison Olson 1\n", + "7 Tiernan P. Doyle 1" + ] + }, + "execution_count": 24, + "metadata": {}, + "output_type": "execute_result" + } + ], + "source": [ + "df.groupby('author', as_index = False).agg({'published':'count'}).sort_values('published', ascending = False)" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "### 11. Add a new column to the data frame that contains the length (number of characters) of each entry title. Return a data frame that contains the title, author, and title length of each entry in descending order (longest title length at the top)." + ] + }, + { + "cell_type": "code", + "execution_count": 25, + "metadata": {}, + "outputs": [ + { + "data": { + "text/html": [ + "
\n", + "\n", + "\n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + "
titlelinkslinkauthorsauthorpublishedpublished_parsedtagsidguidislink...author_detail.namesummary_detail.typesummary_detail.languagesummary_detail.basesummary_detail.valuemedia_player.urlmedia_player.contentmedia_rating.schememedia_rating.contenttitle_len
0Stellar Beads on a String[{'rel': 'alternate', 'type': 'text/html', 'hr...https://www.nasa.gov/image-article/stellar-bea...[{'name': 'Lee Mohon'}]Lee MohonWed, 21 Feb 2024 17:25:50 +0000(2024, 2, 21, 17, 25, 50, 2, 52, 0)[{'term': 'Astrophysics', 'scheme': None, 'lab...https://www.nasa.gov/?post_type=image-article&...False...Lee Mohontext/htmlNonehttps://www.nasa.gov/news-release/feed/Astronomers have discovered one of the most po...https://www.youtube.com/embed/bouQkHDXMKAurn:simplenonadult25
1NASA Astronomer Sees Power in Community, Works...[{'rel': 'alternate', 'type': 'text/html', 'hr...https://www.nasa.gov/general/nasa-astronomer-s...[{'name': 'Abby Tabor'}]Abby TaborWed, 21 Feb 2024 17:22:18 +0000(2024, 2, 21, 17, 22, 18, 2, 52, 0)[{'term': 'General', 'scheme': None, 'label': ...https://www.nasa.gov/?p=616846False...Abby Tabortext/htmlNonehttps://www.nasa.gov/news-release/feed/Science is often portrayed as a solitary affai...NaNNaNNaNNaN60
2Ride the Wave of Radio Astronomy During the So...[{'rel': 'alternate', 'type': 'text/html', 'hr...https://science.nasa.gov/solar-system/skywatch...[{}]Wed, 21 Feb 2024 16:24:06 +0000(2024, 2, 21, 16, 24, 6, 2, 52, 0)[{'term': '2024 Solar Eclipse', 'scheme': None...https://science.nasa.gov/solar-system/skywatch...False...NaNtext/htmlNonehttps://www.nasa.gov/news-release/feed/Students and science enthusiasts are invited t...NaNNaNNaNNaN57
3NASA Selects University Teams to Explore Innov...[{'rel': 'alternate', 'type': 'text/html', 'hr...https://www.nasa.gov/aeronautics/university-te...[{'name': 'Jim Banke'}]Jim BankeWed, 21 Feb 2024 05:50:33 +0000(2024, 2, 21, 5, 50, 33, 2, 52, 0)[{'term': 'Aeronautics', 'scheme': None, 'labe...https://www.nasa.gov/?p=617878False...Jim Banketext/htmlNonehttps://www.nasa.gov/news-release/feed/NASA has selected another five university team...NaNNaNNaNNaN73
4Become a SunSketcher, and Help Measure the Sha...[{'rel': 'alternate', 'type': 'text/html', 'hr...https://science.nasa.gov/get-involved/citizen-...[{}]Tue, 20 Feb 2024 21:35:49 +0000(2024, 2, 20, 21, 35, 49, 1, 51, 0)[{'term': 'Citizen Science', 'scheme': None, '...https://science.nasa.gov/get-involved/citizen-...False...NaNtext/htmlNonehttps://www.nasa.gov/news-release/feed/What shape Is the Sun? Hint: it’s not perfectl...NaNNaNNaNNaN60
\n", + "

5 rows × 30 columns

\n", + "
" + ], + "text/plain": [ + " title \\\n", + "0 Stellar Beads on a String \n", + "1 NASA Astronomer Sees Power in Community, Works... \n", + "2 Ride the Wave of Radio Astronomy During the So... \n", + "3 NASA Selects University Teams to Explore Innov... \n", + "4 Become a SunSketcher, and Help Measure the Sha... \n", + "\n", + " links \\\n", + "0 [{'rel': 'alternate', 'type': 'text/html', 'hr... \n", + "1 [{'rel': 'alternate', 'type': 'text/html', 'hr... \n", + "2 [{'rel': 'alternate', 'type': 'text/html', 'hr... \n", + "3 [{'rel': 'alternate', 'type': 'text/html', 'hr... \n", + "4 [{'rel': 'alternate', 'type': 'text/html', 'hr... \n", + "\n", + " link \\\n", + "0 https://www.nasa.gov/image-article/stellar-bea... \n", + "1 https://www.nasa.gov/general/nasa-astronomer-s... \n", + "2 https://science.nasa.gov/solar-system/skywatch... \n", + "3 https://www.nasa.gov/aeronautics/university-te... \n", + "4 https://science.nasa.gov/get-involved/citizen-... \n", + "\n", + " authors author published \\\n", + "0 [{'name': 'Lee Mohon'}] Lee Mohon Wed, 21 Feb 2024 17:25:50 +0000 \n", + "1 [{'name': 'Abby Tabor'}] Abby Tabor Wed, 21 Feb 2024 17:22:18 +0000 \n", + "2 [{}] Wed, 21 Feb 2024 16:24:06 +0000 \n", + "3 [{'name': 'Jim Banke'}] Jim Banke Wed, 21 Feb 2024 05:50:33 +0000 \n", + "4 [{}] Tue, 20 Feb 2024 21:35:49 +0000 \n", + "\n", + " published_parsed \\\n", + "0 (2024, 2, 21, 17, 25, 50, 2, 52, 0) \n", + "1 (2024, 2, 21, 17, 22, 18, 2, 52, 0) \n", + "2 (2024, 2, 21, 16, 24, 6, 2, 52, 0) \n", + "3 (2024, 2, 21, 5, 50, 33, 2, 52, 0) \n", + "4 (2024, 2, 20, 21, 35, 49, 1, 51, 0) \n", + "\n", + " tags \\\n", + "0 [{'term': 'Astrophysics', 'scheme': None, 'lab... \n", + "1 [{'term': 'General', 'scheme': None, 'label': ... \n", + "2 [{'term': '2024 Solar Eclipse', 'scheme': None... \n", + "3 [{'term': 'Aeronautics', 'scheme': None, 'labe... \n", + "4 [{'term': 'Citizen Science', 'scheme': None, '... \n", + "\n", + " id guidislink ... \\\n", + "0 https://www.nasa.gov/?post_type=image-article&... False ... \n", + "1 https://www.nasa.gov/?p=616846 False ... \n", + "2 https://science.nasa.gov/solar-system/skywatch... False ... \n", + "3 https://www.nasa.gov/?p=617878 False ... \n", + "4 https://science.nasa.gov/get-involved/citizen-... False ... \n", + "\n", + " author_detail.name summary_detail.type summary_detail.language \\\n", + "0 Lee Mohon text/html None \n", + "1 Abby Tabor text/html None \n", + "2 NaN text/html None \n", + "3 Jim Banke text/html None \n", + "4 NaN text/html None \n", + "\n", + " summary_detail.base \\\n", + "0 https://www.nasa.gov/news-release/feed/ \n", + "1 https://www.nasa.gov/news-release/feed/ \n", + "2 https://www.nasa.gov/news-release/feed/ \n", + "3 https://www.nasa.gov/news-release/feed/ \n", + "4 https://www.nasa.gov/news-release/feed/ \n", + "\n", + " summary_detail.value \\\n", + "0 Astronomers have discovered one of the most po... \n", + "1 Science is often portrayed as a solitary affai... \n", + "2 Students and science enthusiasts are invited t... \n", + "3 NASA has selected another five university team... \n", + "4 What shape Is the Sun? Hint: it’s not perfectl... \n", + "\n", + " media_player.url media_player.content \\\n", + "0 https://www.youtube.com/embed/bouQkHDXMKA \n", + "1 NaN NaN \n", + "2 NaN NaN \n", + "3 NaN NaN \n", + "4 NaN NaN \n", + "\n", + " media_rating.scheme media_rating.content title_len \n", + "0 urn:simple nonadult 25 \n", + "1 NaN NaN 60 \n", + "2 NaN NaN 57 \n", + "3 NaN NaN 73 \n", + "4 NaN NaN 60 \n", + "\n", + "[5 rows x 30 columns]" + ] + }, + "execution_count": 25, + "metadata": {}, + "output_type": "execute_result" + } + ], + "source": [ + "df['title_len'] = [len(nasa.entries[i].title) for i in range(len(nasa.entries))]\n", + "df.head()" + ] + }, + { + "cell_type": "code", + "execution_count": 26, + "metadata": {}, + "outputs": [ + { + "data": { + "text/html": [ + "
\n", + "\n", + "\n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + "
titleauthortitle_len
3NASA Selects University Teams to Explore Innov...Jim Banke73
82023 Annual Highlights of Results from the Int...Ana Guzman70
5NASA Sets Coverage of First US Uncrewed Commer...Tiernan P. Doyle63
1NASA Astronomer Sees Power in Community, Works...Abby Tabor60
4Become a SunSketcher, and Help Measure the Sha...60
\n", + "
" + ], + "text/plain": [ + " title author \\\n", + "3 NASA Selects University Teams to Explore Innov... Jim Banke \n", + "8 2023 Annual Highlights of Results from the Int... Ana Guzman \n", + "5 NASA Sets Coverage of First US Uncrewed Commer... Tiernan P. Doyle \n", + "1 NASA Astronomer Sees Power in Community, Works... Abby Tabor \n", + "4 Become a SunSketcher, and Help Measure the Sha... \n", + "\n", + " title_len \n", + "3 73 \n", + "8 70 \n", + "5 63 \n", + "1 60 \n", + "4 60 " + ] + }, + "execution_count": 26, + "metadata": {}, + "output_type": "execute_result" + } + ], + "source": [ + "df_new = df[['title', 'author', 'title_len']].sort_values('title_len', ascending = False)\n", + "df_new.head()\n", + "\n" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "### 12. Create a list of entry titles whose summary includes the phrase \"machine learning.\"" + ] + }, + { + "cell_type": "code", + "execution_count": 18, + "metadata": {}, + "outputs": [ + { + "data": { + "text/plain": [ + "'Stellar Beads on a String'" + ] + }, + "execution_count": 18, + "metadata": {}, + "output_type": "execute_result" + } + ], + "source": [ + "df.title[0]" + ] + }, + { + "cell_type": "code", + "execution_count": 19, + "metadata": {}, + "outputs": [ + { + "data": { + "text/plain": [ + "'NASA Astronomer Sees Power in Community, Works to Build More'" + ] + }, + "execution_count": 19, + "metadata": {}, + "output_type": "execute_result" + } + ], + "source": [ + "df.title[1]" + ] + }, + { + "cell_type": "code", + "execution_count": 20, + "metadata": {}, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "[]\n" + ] + } + ], + "source": [ + "lst = []\n", + "\n", + "for i in df.title:\n", + " if 'machine learning' in i.lower():\n", + " lst.append(i)\n", + "print(lst) " + ] + } + ], + "metadata": { + "kernelspec": { + "display_name": "Python 3 (ipykernel)", + "language": "python", + "name": "python3" + }, + "language_info": { + "codemirror_mode": { + "name": "ipython", + "version": 3 + }, + "file_extension": ".py", + "mimetype": "text/x-python", + "name": "python", + "nbconvert_exporter": "python", + "pygments_lexer": "ipython3", + "version": "3.11.5" + } + }, + "nbformat": 4, + "nbformat_minor": 2 +}