Integration tests ================= Integration tests for |showyourwork| are located in the `tests/integration` subdirectory of the ``showyourwork`` package. Each test subclasses the ``TemporaryShowyourworkRepository`` class in `tests/integration/helpers/temp_repo.py`, which defines an asynchronous method called ``test_repo`` to test the workflow on a given repository by running a series of steps. Each step is defined by a method within the class, such as ``create_remote`` to create the remote repository on GitHub, ``create_local`` to create and initialize the repository locally, ``customize`` to add/edit files in the repository, ``build_local`` to run the workflow locally, and ``run_github_action`` to push the changes to the remote and trigger the ``showyourwork-action`` run. There are a few other methods as well -- see `tests/integration/helpers/temp_repo.py` for details. To create a new test, subclass ``TemporaryShowyourworkRepository`` and override any of the aforementioned methods. As an example, consider the test that checks the ability of ``showyourwork`` to dynamically generate text in a TeX file: .. code-block:: python from helpers import TemporaryShowyourworkRepository from showyourwork.config import edit_yaml # A script that computes the age of the universe variable_script = r""" import paths import numpy as np # Compute the age of the universe np.random.seed(42) age = np.random.normal(14.0, 1.0) # Write it to disk with open(paths.output / "age_of_universe.txt", "w") as f: f.write(f"{age:.3f}") """ class TestLatexVariable(TemporaryShowyourworkRepository): """Test a workflow with dynamic quantities imported into the tex file.""" def customize(self): """Create and edit all the necessary files for the workflow.""" # Create the script with open( self.cwd / "src" / "scripts" / "age_of_universe.py", "w" ) as f: print(variable_script, file=f) # Import the variable into the tex file ms = self.cwd / "src" / "tex" / "ms.tex" with open(ms, "r") as f: ms_orig = f.read() with open(ms, "w") as f: ms_new = ms_orig.replace( r"\end{document}", r"Based on a detailed analysis of Planck observations of the cosmic " r"microwave background, we have determined the age of the universe " r"to be \variable{output/age_of_universe.txt} Gyr." "\n" r"\end{document}", ) print(ms_new, file=f) # Add a Snakemake rule to run the script with open(self.cwd / "Snakefile", "r") as f: contents = f.read() with open(self.cwd / "Snakefile", "w") as f: print(contents, file=f) print("\n", file=f) print( "\n".join( [ "rule age_of_universe:", " output:", " 'src/tex/output/age_of_universe.txt'", " script:", " 'src/scripts/age_of_universe.py'", ] ), file=f, ) This test subclasses ``TemporaryShowyourworkRepository`` and overrides a single method: ``customize``, which makes local changes to the repository before attempting to run the workflow. In the code above, we create a Python script (`src/scripts/age_of_universe.py`), which outputs the age of the universe (purportedly from Planck cosmic microwave background data) to the file `src/tex/output/age_of_universe.txt`. We then edit the default `ms.tex` file (which is generated during the ``create_local`` step of the test) to include a call to ``\variable{output/age_of_universe.txt}``, which imports the value generated by the script into the manuscript. Finally, we add a Snakemake rule telling the workflow how to generate the output from the Python script. That's it! The parent class (``TemporaryShowyourworkRepository``) takes care of the rest. Debugging within the tests -------------------------- By default, the integration tests capture the output when running the ``showyourwork`` command. This will cause problems when trying to debug parts of the main ``showyourwork`` code within the tests. To do so, you will need to enable debug mode to let the output pass through: .. code-block:: bash DEBUG=1 python -m pytest tests/integration/test_default.py -s Remote tests ------------ Note that several integration tests are marked with the ``remote`` mark (using ``pytest.mark``). This means that they will create a temporary remote repository on GitHub with the same name as the test (but in ``snake_case`` instead of ``camelCase``); if a repository already exists, the test will force-push new commits to it, mimicking the behavior of a newly created repository. Running locally ^^^^^^^^^^^^^^^ By default, the tests create the repository under the `github.com/showyourwork` organization, which means they require push access to the ``showyourwork`` organization in order to test the entire workflow, including the ``showyourwork-action`` and the generation of the PDF when running on the remote. Since most contributors and users do not have push access to the ``showyourowrk`` organization, there is also a flag to create the test repositories under a regular user. Users first need to `create a personal access token `_ with read access to your public repositories, and **read and write** access to actions, administration, code, and workflows. The token should then be stored under the ``GH_API_KEY`` environment variable. Once this is done, users should be able to run the integration tests with: .. code-block:: bash python -m pytest tests/integration --remote --action-spec git+https://github.com/showyourwork/showyourwork.git --no-org Running via GitHub actions ^^^^^^^^^^^^^^^^^^^^^^^^^^ Remote tests are spawned from the `remote_integration_tests.yml` workflow in `.github/workflows` of the ``showyourwork/showyourwork`` repository on ``push`` events. These do not get run on forks, since they do not have write access to the ``showyourwork`` organization. This means pull request tests can only check unit tests and local integration tests. In order to run remote tests on pull requests, maintainers may label them with the ``safe to test`` label, in which case the `remote_integration_tests.yml` workflow is executed with the ``pull_request_target`` trigger. Maintainers should carefully review the proposed changes to check for malicious code before marking PRs as ``safe to test``, since the workflow will have full write privileges to the organization!