Solving the Next Bug in Pandas

Solving the Next Bug in Pandas

This week I went on another search for an issue in Pandas. I used the same tactic that I started using this year, I've been finding and solving older bugs that nobody fixed yet. There are over 2,800 issues in Pandas and it takes a while to find a bug that you're interested in working on, but when you fix the bug, you get a big feeling of satisfaction and accomplishment. But before we dive into the new bug, I will talk about a quick update from the bug I was working on last week.

Update on Last Week's Bug


The feedback on the issue relating ordering of regular dictionaries for Python 3.6 and 3.7 that I previously observed from the moderators was outdated. Previously, I stated that the ordering of regular dictionaries in Python 3.6 is an implementation detail of CPython, and users should not rely on it. However, in Python 3.7 ordering of regular dictionaries becomes a language feature. Although this statement is still correct, Pandas ensured that users can rely on ordering of regular dictionaries in Python 3.6.

I received this update on my Pull Request, along with a request to add a line of documentation in the change log to reflect my bug. I fixed my code and ensured that the ordering of dictionaries and OrderedDicts followed the correct order in Python 3.6 and above. The following are my changed:

Changed the dict_keys_to_ordered_list function to use >= Python 3.6:


Ensured that the tests that previously used sorted now use the new function:


Added a line of documentation to the change log:


Included a check for version and adjusted expected result in one of the test:

In addition to ensuring that everything is functioning with Python 3.6 and above, I also had to work on one of the tests that was failing due to my change. In this test, one of the functions uses concat, meaning that the expected result will be different based on the version of Python. However, there is only one expected result generated. I had to manually reverse the order of the columns in the test and ensure that the test produces the correct expected result based on the version of Python.


New Issue

I found this issue by looking through older Pandas issues and I got interested in solving it. It was posted on October 2, 2017 and since then nobody has worked on it. I started working on it this week and will continue my work into the next week.

The creator of the issue brought to attention a bug that does not allow MultiIndex keys to be incomplete in the subset parameter of the "dropna()" function. 

DataFrame.dropna(...) function removes missing values.
If you pass in a "subset=..." parameter into it, it will remove rows based on the specified list of columns that are listed in the subset.

I created some sample code, following the examples that the moderator gave and here are my observations:

Sample Code:
_____________________________________________



Running this produces an error:
_____________________________________________


Expected Output:
_____________________________________________



Progress

Apart from creating sample code to run in order to further explore the issue, I also created a test that will test the bug. For now, it is a very basic test, but I will parametrize it to include more test cases next week.


Now that I have explored the issue and I understand what the bug is and I have a test to test the work that I will be doing, it is time to jump in and actually fix the bug. Next week I will be jumping into the code base and attempting to produce a Pull Requests that solves this issue.





Comments

Post a Comment

Popular posts from this blog

Another Dropna Bug in Pandas

Image
Another Dropna Bug in Pandas This year I fixed a few bugs that dealt with NaN values and more specifically the Dropna function in Pandas. Every bug like this made me more and more interested in fixing behaviour that is not consistent across the code base, specifically how the functions deal with non existent values. Some functions have a dropna argument. When set to true, the function should drop columns/rows that contain missing values and when set to false, the function should keep columns/rows that contain missing values. Currently, some functions deal well with NaN values, however there are functions that either produce buggy behaviour with NaN or don't deal with it consistently across the code base. This week I found another bug like this and I will be working on it throughout next week. Issue https://github.com/pandas-dev/pandas/issues/10772 The creator of the issue noticed inconsistent behaviour when using the dropna argument inside the crosstab function...
Read more

Unit Tests in Pandas

Image
Contributing Unit Tests to Pandas The issue that I decided to work on this week was one of the easier issues discussed in my last blog, however it gave me valuable experience with unit testing in Python and Pandas specifically. I decided to start my 2019 open-source contributions with unit testing because I haven't had much experience with unit testing previously in Python.  Overview of the Issue https://github.com/pandas-dev/pandas/issues/21205 As I previously stated in my last blog, this issue shed light on an issue with using NaT when converting a Series to a Period using the dt.to_period('D') function. Other datetime64[ns] objects with values have no problem converting into a Period, however when a NaT was passed in, it stayed as datetime64[ns]. NaT represents a missing value for datetime objects. Since the issue was posted, it has been resolved with current updates to the master branch. However, one of the moderators asked for unit tests that would ens...
Read more

Progress in Open Source

Image
Progress in Open Source In this blog I will talk about my journey and progress in open source development so far. I've been really enjoying working on real world projects that are used by many people every day. It is not only challenging as every Pull Request required me to perform research in order to understand the issue and solve it, but it is also rewarding because I acquire hands-on experience that is much needed and also contribute to the open source community that I am very fond of now. Open Source Timeline Summary I started exploring open source development during September of 2018. I had to learn a lot of useful tools that Git and Github offers, which now come in handy almost every day. I now know how to create meaningful issues that follow the specifications of the project, as well as create and structure my Pull Requests to the standards of the project that I'm working on. Furthermore learning how to properly manipulate version history using ...
Read more