Solving the Next Bug in Pandas

Solving the Next Bug in Pandas

This week I went on another search for an issue in Pandas. I used the same tactic that I started using this year, I've been finding and solving older bugs that nobody fixed yet. There are over 2,800 issues in Pandas and it takes a while to find a bug that you're interested in working on, but when you fix the bug, you get a big feeling of satisfaction and accomplishment. But before we dive into the new bug, I will talk about a quick update from the bug I was working on last week.

Update on Last Week's Bug


The feedback on the issue relating ordering of regular dictionaries for Python 3.6 and 3.7 that I previously observed from the moderators was outdated. Previously, I stated that the ordering of regular dictionaries in Python 3.6 is an implementation detail of CPython, and users should not rely on it. However, in Python 3.7 ordering of regular dictionaries becomes a language feature. Although this statement is still correct, Pandas ensured that users can rely on ordering of regular dictionaries in Python 3.6.

I received this update on my Pull Request, along with a request to add a line of documentation in the change log to reflect my bug. I fixed my code and ensured that the ordering of dictionaries and OrderedDicts followed the correct order in Python 3.6 and above. The following are my changed:

Changed the dict_keys_to_ordered_list function to use >= Python 3.6:


Ensured that the tests that previously used sorted now use the new function:


Added a line of documentation to the change log:


Included a check for version and adjusted expected result in one of the test:

In addition to ensuring that everything is functioning with Python 3.6 and above, I also had to work on one of the tests that was failing due to my change. In this test, one of the functions uses concat, meaning that the expected result will be different based on the version of Python. However, there is only one expected result generated. I had to manually reverse the order of the columns in the test and ensure that the test produces the correct expected result based on the version of Python.


New Issue

I found this issue by looking through older Pandas issues and I got interested in solving it. It was posted on October 2, 2017 and since then nobody has worked on it. I started working on it this week and will continue my work into the next week.

The creator of the issue brought to attention a bug that does not allow MultiIndex keys to be incomplete in the subset parameter of the "dropna()" function. 

DataFrame.dropna(...) function removes missing values.
If you pass in a "subset=..." parameter into it, it will remove rows based on the specified list of columns that are listed in the subset.

I created some sample code, following the examples that the moderator gave and here are my observations:

Sample Code:
_____________________________________________



Running this produces an error:
_____________________________________________


Expected Output:
_____________________________________________



Progress

Apart from creating sample code to run in order to further explore the issue, I also created a test that will test the bug. For now, it is a very basic test, but I will parametrize it to include more test cases next week.


Now that I have explored the issue and I understand what the bug is and I have a test to test the work that I will be doing, it is time to jump in and actually fix the bug. Next week I will be jumping into the code base and attempting to produce a Pull Requests that solves this issue.





Comments

Post a Comment

Popular posts from this blog

First Enhancement in Pandas

Image
First Enhancement in Pandas After spending the last few months fixing bugs in Pandas and writing tests, I decided that I wanted to try something a little bit different. I went on a hunt for enhancement issues to see if I can add another type of contribution to Pandas. A lot of enhancement requests that I found were hard and technical, so I stayed away from them for now. However, I found one that didn't require a whole new functionality to be added to the code, but asked to add an extra calculation in an existing function, and I proceeded to work on it. Issue https://github.com/pandas-dev/pandas/issues/21689 The issue deals with the pd.DataFrame.describe() function in Pandas. Currently, this function provides 8 summary statistics that are performed on the DataFrame that calls this function. These summary statistics include: count, mean, std, min, 25%, 50%, 75%, max. All of these summary statistics are useful in data analysis and provide a summary of the central t
Read more

Working with Incomplete MultiIndex keys in Pandas

Image
Incomplete MultiIndex keys in Dropna This week I produced a Pull Request for the bug in Pandas that I discussed last week. Although I think the solution that I provided can be further improved, it does fix the immediate issue of incomplete MultiIndex keys in the subset parameter of the "dropna()" function. I submitted a Pull Request to gather some feedback on my solution and improve it based on the moderator's discretion. Overview of the Issue https://github.com/pandas-dev/pandas/issues/17737 The creator of the issue brought to attention a bug that does not allow MultiIndex keys to be incomplete in the subset parameter of the "dropna()" function.  DataFrame.dropna(...) function removes missing values. If you pass in a "subset=..." parameter into it, it will remove rows based on the specified list of columns that are listed in the subset. I created some sample code, following the examples that the moderator gave and here a
Read more

Progress in Open Source

Image
Progress in Open Source In this blog I will talk about my journey and progress in open source development so far. I've been really enjoying working on real world projects that are used by many people every day. It is not only challenging as every Pull Request required me to perform research in order to understand the issue and solve it, but it is also rewarding because I acquire hands-on experience that is much needed and also contribute to the open source community that I am very fond of now. Open Source Timeline Summary I started exploring open source development during September of 2018. I had to learn a lot of useful tools that Git and Github offers, which now come in handy almost every day. I now know how to create meaningful issues that follow the specifications of the project, as well as create and structure my Pull Requests to the standards of the project that I'm working on. Furthermore learning how to properly manipulate version history using
Read more