Testing

This description of Tax-Calculator testing procedures is written for a person who wants to contribute changes to Tax-Calculator source code.

It assumes that you have read Contributor guide and Policy parameter naming and placing conventions, have forked the central GitHub Tax-Calculator repository to your GitHub account, and have cloned that forked copy to your local computer.

This document also assumes that you have read Pull request workflow so that you understand where the testing procedures fit into the broader workflow of preparing a pull request that changes Tax-Calculator source code.

Currently there are two phases of testing.

Testing with pycodestyle (the program formerly known as pep8)

The first phase of testing checks the formatting of the Python code against a PEP8-like standard. Assuming you are in the top-level directory of the repository, run these tests by doing:

pycodestyle taxcalc

No messages indicate the tests pass. Fix any errors. When you pass all these PEP8-like tests, proceed to the second phase of testing.

If you are proposing changes in a JSON file, then you should also run the following test:

pycodestyle --ignore=E501,E121 PATH_TO_JSON_FILE

where you replace PATH_TO_JSON_FILE with the relative path to the JSON file you changed. So, for example, if you edited the policy_current_law.json file in the taxcalc subdirectory, you would replace PATH_TO_JSON_FILE with taxcalc/policy_current_law.json.

Testing with pytest

There are two variants of this second testing phase depending on whether or not you have access to a file called puf.csv that contains a representative sample of tax filing units derived from the IRS-SOI PUF data.

A brief description of the puf.csv file is followed by instructions on how to run the two variants of the second-phase tests.

The Tax-Calculator puf.csv file has been constructed by the core development team by merging information from the most recent publicly available IRS-SOI PUF file and from the Census CPS file for the corresponding year. If you have acquired from IRS the most recent SOI PUF file and want to execute the tests that require the puf.csv file, contact the core development team to discuss your options.

If you have access to the Tax-Calculator puf.csv file, you should have access to the private GitHub repository taxpuf, from which updates of the puf.csv are distributed. When you receive a private email announcing a new version of the puf.csv file, be sure to execute a conda update ... taxpuf command (as described in the taxpuf repository’s README file) before executing the tests described below.

NO PUF.CSV: If you do not have access to the puf.csv file (or if you want to do just a quick test), run the second-phase of testing as follows at the command prompt in the tax-calculator directory at the top of the repository directory tree:

cd taxcalc
pytest -m "not requires_pufcsv and not pre_release" -n4
cd ..

This will start executing a pytest suite containing hundreds of tests, but will skip the tests that require the puf.csv file as input and the tests that are executed only just before a new release is being prepared. Depending on your computer, the execution time for this incomplete suite of tests is roughly one minute. The -n4 option calls for using as many as four CPU cores for parallel execution of the tests. If you want sequential execution of the tests (which will take at least twice as long to execute), simply omit the -n4 option.

HAVE PUF.CSV: If you do have access to the puf.csv file, copy it into the Tax-Calculator directory at the top of the repository directory tree (but never add it to your repository) and run the second-phase of testing as follows at the command prompt in the top-level directory:

cd taxcalc
pytest -m "not pre_release" -n4
cd ..

This will start executing a pytest suite containing hundreds of tests, including the tests that require the puf.csv file as input but excluding the tests that are executed only just before a new release is being prepared. Depending on your computer, the execution time for this suite of unit tests is roughly two minutes. The -n4 option calls for using as many as four CPU cores for parallel execution of the tests. If you want sequential execution of the tests (which will take at least twice as long to execute), simply omit the -n4 option.

Just before releasing a new version of Tax-Calculator or just after adding a new parameter to policy_current_law.json, you should also execute the pre-release tests using this command:

cd taxcalc
pytest -m pre_release -n4
cd ..

But if you execute the pre_release tests well before releasing a new version of Tax-Calculator, be sure not to include the updated docs/index.html file in your commit. After checking that you can make a docs/index.html file that is correct, revert to the old version of docs/index.html by executing this command:

git checkout -- docs/index.html

Following this procedure will ensure that the documentation is not updated before the Tax-Calculator release is available.

Interpreting test results

If you are adding an enhancement that expands the capabilities of the Tax-Calculator, then all the tests you can run should pass before you submit a pull request containing the enhancement. In addition, it would be highly desirable to add a test to the pytest suite, which is located in the taxcalc/tests directory, that somehow checks that your enhancement is working as you expect it to work.

On the other hand, if you think you have found a bug in the Tax-Calculator source code, the first thing to do is add a test to the pytest suite that demonstrates how the source code produces an incorrect result (that is, the test fails because the result is incorrect). Then change the source code to fix the bug and demonstrate that the newly-added test, which used to fail, now passes.

Updating test results

After an enhancement or bug fix, you may be convinced that the new and different second-phase test results are, in fact, correct. How do you eliminate the test failures? For all but the few tests that require the puf.csv file or the cps.csv file as input, simply edit the appropriate taxcalc/tests/test_*.py file so that the test passes when you rerun pytest. If there are failures for the tests that write results files, read the test error message for instructions about how to update the test results.

Optional coding style testing with pylint

There is another tool, pylint, that warns about deviations from a broader set coding styles than does pycodestyle. The use of pylint, while being the number one recommendation in the Google Python Style Guide, is strictly-speaking optional for Tax-Calculator work. But several important files in the repository are maintained in a way that their coding style does not generate any pylint warnings. You can determine which files these are by looking for the comments near the top of the file that begin # CODING-STYLE CHECKS:. If there is a line in these comments that mentions pylint, then this file is being tested with pylint. It is highly recommended that, if you are proposing changes in one these files, you check your work by running the pylint command listed in that file’s coding-style comment.

Make sure you have an up-to-date version of pylint installed on your computer by entering at the operating system command prompt:

pylint --version

If you get a no-such-command error, install pylint as follows:

conda install pylint

If you do have pylint installed, but the version is before 1.8.4, then get a more recent version as follows:

conda update pylint