Exploring the Power of Detailed Visualizations
When it comes to understanding complex data, visualizations play a crucial role in uncovering trends and relationships. In this section, we'll delve into the world of detailed visualizations and explore how they can complement summary visualizations. By examining specific subsets of our data, we can gain deeper insights that might be lost in summarization.
For instance, let's take a look at the annual return for four stocks. At first glance, the summary visualization reveals that all four stocks had a similar year-over-year return of around 13%. However, upon closer inspection of detailed plots of daily prices, we see that each stock has a distinct performance. This highlights the importance of taking a more detailed approach when exploring complex data.
Visualizing Large Data in Detail
One of the challenges of working with large datasets is the sheer amount of information that can be overwhelming. To make sense of this data, it's essential to identify meaningful subsets that can help us focus our analysis. In our case, we've found that taking a subset of all data for one stock allows us to visualize and explore its characteristics more effectively.
For example, let's examine the distribution of tip amounts across various payment types. We know that credit card payments always result in zero tips, but what about cash payments? Does the taxi payment system distinguish between tips and fares, or does the total fare amount simply exclude the tipped portion? To investigate this question further, we'll turn to detailed visualizations of a subset of our data.
Pulling Out Meaningful Subsets
When working with large datasets, it's essential to identify subsets that have natural meaning. In our case, we've extracted a subset of data for the most popular route from the Upper East Side south to the Upper East Side north. By focusing on this specific route and only considering cash and credit transactions, we're able to analyze a significant amount of data while still maintaining focus.
We'll take a closer look at the relationship between total fare and trip duration to ensure that our subset is well-behaved. Even with a relatively small dataset, there's still the potential for overplotting, which can make it difficult to discern meaningful patterns. To alleviate this issue, we'll use the Alpha parameter to add transparency to the points.
Comparing Distributions
Now that we have a manageable subset of data, we can move on to comparing the distributions of fare and tip amounts using card versus cash payments. A quantile plot is an excellent choice for this task, as it displays the ordered values of the data against the quantiles of a uniform distribution. This type of visualization is often more informative than histograms when comparing distributions.
By creating a quantile plot using GGPlot, we can see that the card and cash distributions have similar shapes but are shifted. Moreover, we notice that cash payments consist of discrete values, while card payments are more continuous due to binning in histograms. From this analysis, it's clear that tips are not included in the total reported fare amount for cash payments.
Drawing Conclusions
Our detailed visualization has revealed an important insight: the taxi payment system does indeed distinguish between tips and fares when reporting cash transactions. By examining a subset of our data, we've been able to uncover this phenomenon and gain a deeper understanding of how the payment system works. This approach highlights the importance of taking a more detailed approach when exploring complex data.
In the next section, we'll explore whether removing tips from both card and cash payments would result in similar distributions. The answer is no, as the two types of payments have distinct characteristics that are influenced by different factors. By using detailed visualizations to analyze our data, we can gain a more nuanced understanding of these complexities and uncover new insights that might otherwise remain hidden.
"WEBVTTKind: captionsLanguage: enwe have seen that visualizing summaries can help us discover and describe overall trends and relationships in the data in this section we will explore some examples of how we can complement summary visualizations with detailed visualizations of smaller subsets of our data while summary visualizations can be very revealing sometimes important insights are covered up in the summarization and we need to look at the data in more detail to discover them for example here we have a summary visualization of the annual return for four stocks from the summary it appears all four stocks had a similar year a return of around 13% however looking at detailed plots of the daily prices we see a very different year for each stock we'll have more fun with this stock data in chapter 3 visualizing large data in detail is challenging because there's too much data to look at a useful technique in this case is to take a manageable subset of the data that has some natural meaning such as for example all data for one stock and visualize and explore as we saw in one of our previous exercises the distribution of the tip amount is zero for all payment types but credit card this is an interesting phenomenon that we want to get to the bottom of with cash payments does the taxi payment system not distinguish between tips and fare where does the total fare amount just not include the amount that was tipped to investigate this question we turn to detailed visualization of a subset of our data we expect rides of the same nature to have similar fare and tip amounts therefore if we can pull out a subset of our data for similar routes we can compare the distributions of fare and tip amount to investigate our question we expect the distributions of total fare for rides paid with cash and card to look similar if both cases include tips here we have extracted a subset of the data for the most popular route from the Upper East Side south to the Upper East Side north of man looking only at these trips and only at cash and credit transactions we have about 5,000 observations let's do a check to ensure that this subset is well-behaved looking at the relationship between total fare versus trip duration we expect the relationship to be cleaner since we are focusing on one simple route even with the data this small we are still over plotting many points and we can alleviate this to a degree using the Alpha parameter to add transparency to the points this looks much cleaner than what we saw for all routes to compare the distribution of payments using card versus cash we can use a quantile plot this displays the ordered values of the data against the quantiles of a uniform distribution and is often more useful than a histogram for comparing distributions we create a quantile plot using GG plot to use geum specifying that the data should be plotted against the uniform distribution in this plot we see that the card and cash distributions have a similar shape but are shifted we also see that the cash payments are made up of several discrete values while card payments are more continuous which we wouldn't be able to see in a histogram due to binning from this we can reasonably conclude that tips are not included in the total reported fare amount for cash payments in the exercise we will see if the two distributions are similar if we remove tips from both let's gowe have seen that visualizing summaries can help us discover and describe overall trends and relationships in the data in this section we will explore some examples of how we can complement summary visualizations with detailed visualizations of smaller subsets of our data while summary visualizations can be very revealing sometimes important insights are covered up in the summarization and we need to look at the data in more detail to discover them for example here we have a summary visualization of the annual return for four stocks from the summary it appears all four stocks had a similar year a return of around 13% however looking at detailed plots of the daily prices we see a very different year for each stock we'll have more fun with this stock data in chapter 3 visualizing large data in detail is challenging because there's too much data to look at a useful technique in this case is to take a manageable subset of the data that has some natural meaning such as for example all data for one stock and visualize and explore as we saw in one of our previous exercises the distribution of the tip amount is zero for all payment types but credit card this is an interesting phenomenon that we want to get to the bottom of with cash payments does the taxi payment system not distinguish between tips and fare where does the total fare amount just not include the amount that was tipped to investigate this question we turn to detailed visualization of a subset of our data we expect rides of the same nature to have similar fare and tip amounts therefore if we can pull out a subset of our data for similar routes we can compare the distributions of fare and tip amount to investigate our question we expect the distributions of total fare for rides paid with cash and card to look similar if both cases include tips here we have extracted a subset of the data for the most popular route from the Upper East Side south to the Upper East Side north of man looking only at these trips and only at cash and credit transactions we have about 5,000 observations let's do a check to ensure that this subset is well-behaved looking at the relationship between total fare versus trip duration we expect the relationship to be cleaner since we are focusing on one simple route even with the data this small we are still over plotting many points and we can alleviate this to a degree using the Alpha parameter to add transparency to the points this looks much cleaner than what we saw for all routes to compare the distribution of payments using card versus cash we can use a quantile plot this displays the ordered values of the data against the quantiles of a uniform distribution and is often more useful than a histogram for comparing distributions we create a quantile plot using GG plot to use geum specifying that the data should be plotted against the uniform distribution in this plot we see that the card and cash distributions have a similar shape but are shifted we also see that the cash payments are made up of several discrete values while card payments are more continuous which we wouldn't be able to see in a histogram due to binning from this we can reasonably conclude that tips are not included in the total reported fare amount for cash payments in the exercise we will see if the two distributions are similar if we remove tips from both let's go\n"
