First Enhancement in Pandas

First Enhancement in Pandas


After spending the last few months fixing bugs in Pandas and writing tests, I decided that I wanted to try something a little bit different. I went on a hunt for enhancement issues to see if I can add another type of contribution to Pandas. A lot of enhancement requests that I found were hard and technical, so I stayed away from them for now. However, I found one that didn't require a whole new functionality to be added to the code, but asked to add an extra calculation in an existing function, and I proceeded to work on it.

Issue

The issue deals with the pd.DataFrame.describe() function in Pandas. Currently, this function provides 8 summary statistics that are performed on the DataFrame that calls this function. These summary statistics include: count, mean, std, min, 25%, 50%, 75%, max. All of these summary statistics are useful in data analysis and provide a summary of the central tendency, dispersion and shape of a dataset's distribution. 




However, the creator of the issue stated that there is one more summary statistic that could be added to the pd.DataFrame.describe() function that is also very useful for data analysis. He requested for a "missing" summary statistic to be added to the function, which would return a count of missing values in the DataFrame. Currently, one of the statistics includes "count", which returns a count of values excluding missing values. So, "missing" would do the opposite and return a count of missing values. 


Once I spotted this issue, it immediately drew my attention because just a couple of weeks ago, when I was performing data analysis in the machine learning research project that I'm currently working on, one of the statistics that we needed to calculate for our evaluation phase was the number of missing values. So from my own experience in data analytics, I could tell that this was a nice enhancement to the describe() function.

Pull Request


As always I started by writing a test that would help me add this enhancement and ensure that it works as expected in the pd.DataFrame.describe() function.

def test_missing_describe(self):
df = pd.DataFrame(data={'col1': [1, np.nan],
'col2': [3, 4]})
result = df.describe()
expected = pd.DataFrame({'col1': [1, 1, np.nan, 1, 1, 1, 1, 1, 1],
'col2': [2, 3.5, 0.707107, 3, 3.25, 3.5,
3.75, 4, 0]},
index=['count', 'mean', 'std', 'min', '25%',
'50%', '75%', 'max', 'missing'])
tm.assert_frame_equal(result, expected)

I then started to explore the describe() function and figured out how it works and where it calls the calculations for the summary statistics. I added the "missing" summary statistic that would calculate the amount of missing values with series.isna().sum().


This allowed the describe() function to also include a summary of missing values and produced the DataFrame that the creator of the issue expected. I then had to go and change a few tests to also include this new addition. However, one of the moderators commented on the issue, stating that instead of adding the "missing" summary statistic, a "length" statistic should be added. The "length" statistic would show the total amount of values, including missing values. So in order to calculate missing values, user would have to subtract "count" statistic from the "length" statistic. I thought that having "missing" statistic would be better and am waiting for feedback on my Pull Request.

Once we agree to have "missing" or "length", I will have to change the documentation of the function to reflect the new change and change the rest of the tests that are currently failing due to this new addition.

Overall, this was a great experience and I'm glad that I chose to work on an enhancement this time, because it will give me extra experience with working on enhancements instead of bugs. In addition, I will also get some experience with changing documentation in Pandas based on the new functionality that was added.




Comments

Post a Comment

Popular posts from this blog

Another Dropna Bug in Pandas

Image
Another Dropna Bug in Pandas This year I fixed a few bugs that dealt with NaN values and more specifically the Dropna function in Pandas. Every bug like this made me more and more interested in fixing behaviour that is not consistent across the code base, specifically how the functions deal with non existent values. Some functions have a dropna argument. When set to true, the function should drop columns/rows that contain missing values and when set to false, the function should keep columns/rows that contain missing values. Currently, some functions deal well with NaN values, however there are functions that either produce buggy behaviour with NaN or don't deal with it consistently across the code base. This week I found another bug like this and I will be working on it throughout next week. Issue https://github.com/pandas-dev/pandas/issues/10772 The creator of the issue noticed inconsistent behaviour when using the dropna argument inside the crosstab function...
Read more

Unit Tests in Pandas

Image
Contributing Unit Tests to Pandas The issue that I decided to work on this week was one of the easier issues discussed in my last blog, however it gave me valuable experience with unit testing in Python and Pandas specifically. I decided to start my 2019 open-source contributions with unit testing because I haven't had much experience with unit testing previously in Python.  Overview of the Issue https://github.com/pandas-dev/pandas/issues/21205 As I previously stated in my last blog, this issue shed light on an issue with using NaT when converting a Series to a Period using the dt.to_period('D') function. Other datetime64[ns] objects with values have no problem converting into a Period, however when a NaT was passed in, it stayed as datetime64[ns]. NaT represents a missing value for datetime objects. Since the issue was posted, it has been resolved with current updates to the master branch. However, one of the moderators asked for unit tests that would ens...
Read more

Progress in Open Source

Image
Progress in Open Source In this blog I will talk about my journey and progress in open source development so far. I've been really enjoying working on real world projects that are used by many people every day. It is not only challenging as every Pull Request required me to perform research in order to understand the issue and solve it, but it is also rewarding because I acquire hands-on experience that is much needed and also contribute to the open source community that I am very fond of now. Open Source Timeline Summary I started exploring open source development during September of 2018. I had to learn a lot of useful tools that Git and Github offers, which now come in handy almost every day. I now know how to create meaningful issues that follow the specifications of the project, as well as create and structure my Pull Requests to the standards of the project that I'm working on. Furthermore learning how to properly manipulate version history using ...
Read more