Re-Engineering CI/CD pipelines for Scipy
Harsh Mishra
This talk on Github: HarshCasper/quansight-labs-presentation
Agenda
- Introduction to the Project
- Project implementation
- Learnings out of the Project
- Future plan for the project
- Acknowledgement
- Q&A
About the project!
- Re-engineer the CI pipelines for Scipy to be compatible with Meson.
- Enable local debugging of CI Actions using Act.
- Implement GitHub Actions for building and testing on macOS.
- Plan the CI matrix for the upcoming Scipy build setup using Meson.
- Implement CI benchmarking using GitHub Actions to be compatible with ASV
Project implementation: PR#57
- Configured `act` to run the GitHub Action workflows locally.
- Refactored the `meson.build` to generate `__config__.py` and `version.py` dynamically.
- Wrote the documentation on usage of `act` with GitHub Actions.
Project implementation: PR#59
- Implemented build caching for Meson to restore compiler targets for faster builds.
- Reduced the build time by 70% on the GitHub Actions CI (takes ~3 minutes for a build)
- Integrated timestamp-based caching to cache and restore the latest compiler targets.
Project implementation: PR#65
- Implemented the Scipy build and test CI action for macOS using GitHub Actions.
- Reduced the build time by 30% on the GitHub Actions CI using ccache compiler caching.
- Integrated conda caching and Mamba for faster installs and build time.
Project implementation: PR#67
- Implemented the automated benchmarking on GitHub Actions CI using ASV.
- Automatically builds Scipy and runs a benchmark action to compare the base commit with the PR commit.
- Uploads the ASV results, generated HTML files and the benchmark logs to be verified on demand.
Miscellaneous
Learnings out of the Project
- Learnt about Meson build system and custom targets.
- Increased my familiarity with GitHub Actions and CI toolings.
- Understood caching and benchmarking through working on an actual project.
- Real-time collaboration and communication with my mentor and stakeholders.
Future plan for the project
- Start working on the CI matrix and remove CI-deadweight.
- Build and test the new Meson build on various architectures through GitHub Actions.
- Implement GHA release action for automating the release of source archives.
Acknowledgement
- Thanks to my mentor, John Lee for providing the necessary guidance and feedback on every step.
- Thanks Ralf Gommers for helping with feedback, suggestions and reviews.
- Thanks Anirudh Dagar for providing necessary motivation and reviews.
- Thanks Jaime RodrÃguez-Guerra for helping with CI benchmarking and other stuff.
