Open Source Contribution - Release 0.4 Part 3

Linting Errors in Pandas


Building onto the previous blog where I explored style and linting errors in the pySearch project and the tools used to find these errors, last week I was dealing with similar issues in Pandas. On top of the Flake8 and PEP8 Speaks that the Pandas project uses, they also integrated their own script file that checks for their custom made errors. Since I already explored Flake8 and PEP8 Speaks in-depth, I decided to focus on custom errors within Pandas and to find ways to solve the errors which are difficult to find the origin for.

GL07 Error

The error that I focused on fixing last week was: GL07 - Sections are in the wrong order. As I stated before, Pandas is a massive project and in order to help developers remember the purpose and use of all the functions, as well as for new contributors to understand the functionalities of the functions within the project, Pandas does a very great job of documenting their code through docstrings within each function. These docstrings often include large sections of documentation with different headings. Some examples of these headings are: Parameters, Returns, Raises, See Also, Notes, and Examples. There are even more headings that are utilized throughout the project, however the ones that I listed are the most commonly used. 

Parameters section focuses on listing and explaining all the parameters that the function takes.
Returns section focuses on listing and explaining what the function returns.
Raises section focuses on talking about the different exceptions that can be raised in the function.
See Also section lists similar functions or ones that are used within the current function.
Notes section includes extra notes that a developer may want to know about the function.
Examples section provides thorough examples of how the function can be used.

As you can see, the order in which I listed the sections makes a lot of sense and seems like a natural way to order the sections. The developers of Pandas also thought that this would be the best way to order these headings. In order to ensure that all docstrings within the project are consistent and follow the natural order, they created the GL07 error which ensures that all docstrings are ordered in this manner.

However, before they created this error, many people made the mistake of putting some of the docstrings out of order. I ensured that all docstrings will have ordered sections, fixing almost all GL07 errors within Pandas throughout two pull requests. There were total of 286 errors, and by the end of the second pull request, there were only 14 errors left (I will talk about why I couldn't fix the remaining errors later in this blog). This was a large task and took a lot of time to fix such a large amount of errors.

Fixing the GL07 Error

Running "./scripts/validate_docstrings.py --errors=GL07" will give you all the GL07 errors in Pandas. Adding an additional format parameter to this line ("--format=azure") will allow you to see files and line numbers where the error occurs. With a combination of these parameters, some errors were easily found as the script told me exactly where the error occurs. However, most of the time the task wasn't so easy. Since Pandas is a massive project, a lot of the times the functions are similar or repeated throughout the project. So in order to save on space and work, developers of Pandas created templates and structures for docstrings, which were then utilized in different files. In order to find and fix the origin of the issue, I had to look through where these docstrings originate from, which was sometimes very difficult to find. In addition, there were some cases where the file and line numbers were not displayed and I had to search through the whole project based on which function is used, and find where this function originates from.

Throughout successfully fixing 272 GL07 errors, I became very good at navigating throughout the Pandas project and learned a lot about their structure and the project as a whole. This will be extremely beneficial for my future contributions to Pandas because I am already very familiar with their project now.



Non-Resolved GL07 Errors

There were 14 errors left that I could not currently resolve. I explained in detail the reason behind this to the moderators of Pandas and we agreed that we will find a way to resolve them in the near future. Some of the errors could not be fixed because sometimes there are two different docstrings appended together that come from different files. While I could sort the docstrings independently, once they are appended together the overall order was not sorted.






Comments

Post a Comment

Popular posts from this blog

Another Dropna Bug in Pandas

Image
Another Dropna Bug in Pandas This year I fixed a few bugs that dealt with NaN values and more specifically the Dropna function in Pandas. Every bug like this made me more and more interested in fixing behaviour that is not consistent across the code base, specifically how the functions deal with non existent values. Some functions have a dropna argument. When set to true, the function should drop columns/rows that contain missing values and when set to false, the function should keep columns/rows that contain missing values. Currently, some functions deal well with NaN values, however there are functions that either produce buggy behaviour with NaN or don't deal with it consistently across the code base. This week I found another bug like this and I will be working on it throughout next week. Issue https://github.com/pandas-dev/pandas/issues/10772 The creator of the issue noticed inconsistent behaviour when using the dropna argument inside the crosstab function...
Read more

Unit Tests in Pandas

Image
Contributing Unit Tests to Pandas The issue that I decided to work on this week was one of the easier issues discussed in my last blog, however it gave me valuable experience with unit testing in Python and Pandas specifically. I decided to start my 2019 open-source contributions with unit testing because I haven't had much experience with unit testing previously in Python.  Overview of the Issue https://github.com/pandas-dev/pandas/issues/21205 As I previously stated in my last blog, this issue shed light on an issue with using NaT when converting a Series to a Period using the dt.to_period('D') function. Other datetime64[ns] objects with values have no problem converting into a Period, however when a NaT was passed in, it stayed as datetime64[ns]. NaT represents a missing value for datetime objects. Since the issue was posted, it has been resolved with current updates to the master branch. However, one of the moderators asked for unit tests that would ens...
Read more

Progress in Open Source

Image
Progress in Open Source In this blog I will talk about my journey and progress in open source development so far. I've been really enjoying working on real world projects that are used by many people every day. It is not only challenging as every Pull Request required me to perform research in order to understand the issue and solve it, but it is also rewarding because I acquire hands-on experience that is much needed and also contribute to the open source community that I am very fond of now. Open Source Timeline Summary I started exploring open source development during September of 2018. I had to learn a lot of useful tools that Git and Github offers, which now come in handy almost every day. I now know how to create meaningful issues that follow the specifications of the project, as well as create and structure my Pull Requests to the standards of the project that I'm working on. Furthermore learning how to properly manipulate version history using ...
Read more