We would like to demonstrate that Pi can be efficiently calculated to 62.8 trillion decimal places with limited hardware, personnel and budgetary resources. The current world record for the most accurate number of Pi will serve as a reference for our performance.
Calculating the most accurate number of Pi is an unofficial benchmark in the area of high-performance computing. Although there is no practical application in knowing trillions of decimal places of a transcendental number, this is however, a useful test of our abilities to run extremely resource-demanding computations. Such an endeavour not only requires the selection of the appropriate hardware but also needs expertise in assembling, configuring and tuning hardware and software for maximum performance. Furthermore, we would like to test our ability to reliably manage a several-months-long running computation. The challenge for this record attempt is taken up by project leader Thomas Keller and by the head of DAViS, Prof. Dr Heiko Rölke.
DAViS as part of the University of Applied Sciences of the Grisons is consolidating expertise in the area of high-performance computing (HPC) and HPC infrastructure for a mandate of the canton of Grisons. This also includes implementing a new degree programme in the field of 'Computational and Data Science'. Prospective students will benefit from access to powerful hardware and will be able to practice newly acquired theoretical knowledge using state-of-the-art hardware and software.
University of Applied Sciences of the Grisons (Challenger)
Decimal Places: 62'831'853'071'796
Number of days calculating Pi: less than 140 days
Start date: 28/04/2021
Timothy Mullican (current record holder)
Decimal Places: 50'000'000'000'000
Number of days calculating Pi: 303 days
Start date: 1/4/2019
Google (former record holder)
Decimal places: 31'415'926'535'897
Number of days calculating Pi: 121 days
Start date: 22/9/2018
Peter Trüb (former record holder)
Decimal places: 22'459'157'718'361
Number of days calculating Pi: 105 days
Start date: 11/11/2016
18.01.2022 - Independent verification
Sergey, a Pi enthusiast from Russia, has informed us that the first 50 trillion digits of Timothy Mullicans computation correspond exactly with our digits at http://pi.fhgr.ch. To this end, he had to split the Pi number caluclated by Mullican into 40 GB size files, in order to get the same file size as the files we generated during the record attempt. With the help of a hash function (cryptographic method) he could show, that our Pi digits match with the calculation made by Timothy Mullican.
30.08.2021 - Pi verification
One option to verify a calculation of Pi is to use the BBP algorithm. We used this algorithm to check a few of the last digits we previously calculated with the Chudnovsky formula (See first image below):
Our Pi calculation has also been confirmed by numberworld.org.
14.08.2021 - End of Computation
This Saturday morning at 9:30 am, our High Performance Computer successfully completed its calculation to 62'831'853'071'750 decimal places, hence an additional 12.8 trillion decimal digits of Pi have been discovered. Therefore a new world record for the most precise number of Pi has been set by team DAViS of Fachhochschule Graubünden.
The ten last known digits of Pi are: 7817924264
11.08.2021 - Decimal Version of Pi created
At around 7:30 this morning, our computer finalized the conversion of 62.8 trillion digits from hexadecimal to decimal. The number is split into 63 compressed files. As a next step, the software is verifying if the base conversion process has been done correctly.
04.08.2021 - hexadecimal version of Pi calculated
After calculating for a little bit more than 92 days, Y-cruncher wrote its hexadecimal version of Pi onto the disks of our computer. The number is compressed and therefore uses only about 24 TB of disk space (instead of 48 TB if it were uncompressed). Currently our computer is converting the calculated number of Pi into the decimal number system.
01.08.2021 - Calculation «Inverse Square Root»
01.08.2021: Y-cruncher finished calculating the core of the Chudnovsky formula on the 25th of July (this is marked with a green line in the formula below). As a next step, the programm is currently calculating the so called «Inverse Square Root»* part of the algorithm.
The previously calculated denominator of the fraction above (inside green line), is a gigantic number of trillion of decimal places. We anticipate that this part of the calculation will take about 5-6 days to complete.
* We can only guess, why this section was named «Inverse Square Root» by the programmer. We assume that he preferes multiplying the inverse of a value over doing a division, as multiplying the inverse of a number seems to be faster.
26.07.2021 - Backup and calculation second part of the Pi-algorithm
Our backup finished successfully a few hours ago and y-cruncher is now in the last stages of our calculation, where it verifies the 62.8 trillion digits. We estimate that the last leg will conclude in 3 - 4 weeks time.
25.07.2021- Calculation Chudnovsky serie completed
Our computer finally finished its calculation of 62.8 trillion digits today, after running for 88 days and 14 hours! Net calculation time (without downtime) took 83 days and 20 hours. Next, we will start backing up the data generated by the program (approx. 51 TB). After that, Y-Cruncher will start cross-checking its calculated result.
Pi computation progress: 100%
Pi verification progress: 0%
Downtime (backup or maintenance): 18.57hrs (backup) / 27.55hrs (backup) / 41.64hrs (backup) / 25.51hrs (backup) / 26.47hrs (backup) - 139.74hrs (Total downtime)
Calculating Pi to the trillion digit range demands extreme amounts of RAM and swap space as well as high memory bandwidth. For this type of computation processor performance is of secondary importance.
2 AMD EPYC 7542 CPUs, 32 cores each
Disk space total
38 hard disk drives running at 7,200 rpm and with 16 TB each. 34 of these disks are used for swap space and 4 disks will hold the final Pi digits
2 SSDs in a RAID configuration for storing the operating system
Although our installed RAM is very large compared to a regular desktop PC, it would only be enough to calculate Pi to a few 100 billion decimal places. To work around this limitation, the Pi calculation program stores a large portion of its memory on hard disk drives. This technique is called ‘swapping’ and is a method also employed by regular PCs, for example with Windows or Mac. Performance-wise, SSDs (very fast hard disks based on semiconductor technologies) would be of great advantage for this type of calculation. However, for our Pi computation and also future computations, we expect a large amount of disk write access, which would result in high wear and early failure of an SSD. Since SSD storage prices are 5 to 10 times higher compared to conventional HDDs, we opted for HDDs in the design process. In our case the risk of early failure applies primarily to swap storage space. We use SSDs for storing the operating system because we anticipate fewer write cycles. Additionally, storing the operating system requires only a few GBs of space.
To somewhat compensate for the speed disadvantage using HDDs compared to SSDs, we configured y-cruncher for highly parallel access to the swap area. This configuration allows us an aggregated read/write speed to the swap memory area of about 8.5 GB/s using 34 HDDs. The same result could be achieved by using only three SSDs. However, with just three SSDs, we would have missed the Pi storage requirement of about 310 TB by several factors.
The final Pi digit will need 63 terabytes of storage space. An additional 91 TB of hard disk space is assigned for the storage of intermediate calculation, called checkpoints. As mentioned before, up to 310 TB of storage space is used for the swap area, and for the occasional backup we have earmarked another 180 TB of storage space.
The system used in this calculation is made up of two units:
The system is hosted in a standard server rack and receives cooling and emergency power. Cooling is imperative to keep all components below the maximum allowed temperature recommended by the manufacturer.
The computing unit with CPUs, memory and SSDs consumes approx. 300 watts of power. The JBOD with 38 HDDs requires approx. 430 watts.
Since the computer and the disks need to be continuously cooled during operation (the CPU cores can heat up to 80° C and the disks up to 40° C), this heat has to be dissipated with air cooling. According to American Power Conversion, we need an additional 950 watts for cooling. We estimate our power demand to total at around 1,700 watts.
- a conventional hair dryer draws about 2,000 watts of power
- 1.75 GFlops/Watt would put us in 153rd place on the Green500 list of supercomputers
For calculating Pi we use the software y-cruncher. This software has been employed by previous world record runs of Pi and is characterised by the fact that it takes advantage of modern processor features. y-cruncher has been developed by the American programmer Alexander Yee and can be run on any standard PC with Linux or Windows to calculate mathematical constants such as Pi.
We use the operating system Ubuntu 20.04 on the computation unit. This system allowed us to tune various parameters to decrease calculation time. Among other things, we configured the following settings to improve execution time:
- disabled various security features
- activated an IO scheduler optimised for the computation
- disabled power-saving settings
- changed to a faster file system
- made BIOS optimisations
Some of these optimisations have serious disadvantages. For example less reliability or higher power consumption. Others only improve execution time for a specific software or calculation type.
The number Pi is an irrational and transcendental number. This entails that the number Pi can neither be represented as a fraction (irrational) nor as polynomial (transcendental).
In 1988, the Chudnovsky brothers developed the following algorithm for calculating the number Pi:
A noteworthy property of the algorithm is that the sum (sigma symbol) has a limit of infinity (lying eight). The reason is that Pi is an irrational number, which converges but can never be represented with final precision. Having only limited computing resources at hand, we will never be able to calculate Pi precisely. The algorithm belongs to the family of hypergeometric series and relies for its computation mainly on integer numbers. Thanks to these properties we can calculate Pi without rounding or approximations. This avoids rounding errors, which can lead to precision errors. Rounding errors have led to spectacular failures in the past.
Currently, the Chudnovsky formula is still the most efficient method of calculating Pi to an arbitrary number of digits. Using this method, the number Pi is calculated to an average precision of 14.81 digits per iteration. All recent world records have been calculated using this method. This algorithm was implemented in y-cruncher using C/C++ and and is currently used by DAViS and previously by Timothy Mullican and Google in their respective world record calculations.
There are two possibilities of checking the newly calculated digits for their correctness: one could recalculate the number of digits with another algorithm (for example with the Gauss-Legendre algorithm). However, this would again take at least as much time (and probably a lot more time) than the original calculation. Fortunately, the Bailey-Borwein-Plouffe formula was discovered in 1995 by Simon Plouffe. With this formula an arbitrary digit of the number Pi can be calculated without needing to calculate previous digits! For example, one could calculate the digit 10,581 without having to calculate all the preceding digits. However, calculating a single digit of Pi is very computationally complex (exponential), so only some of the last digits of Pi are verified for a given calculation.
Two years later, the French mathematician Fabrice Bellard found another and faster way to calculate random digits of Pi. This method is used by y-cruncher to verify the calculated number.
High Performance Computing (HPC) is by no means restricted to the area of calculating mathematical constants. HPC is in fact used in many areas of applied research and development. As ever increasing amounts of data require ever more powerful hardware and software, HPC is for example used in the process of DNA sequence analysis or machine learning.
Among other research projects, DAViS provides computing power for the Translaturia project (machine learning) or supports the allergy and asthma research of the SIAF in Davos with DNA sequence analysis.
DAViS is a regional competence centre for research, industry and education in the areas of data analysis and scientific computing. We collaborate with the national high-performance computing centre (CSCS) in Lugano operated by ETH Zurich and cooperate with the Swiss Institute for Allergy and Asthma Research (SIAF) at the University of Zurich.
As DAViS is part of the University of Applied Sciences of the Grisons, we can draw on a broad range of domain-specific expertise and run an advanced IT infrastructure to support our research. In the case of the world record attempt, we are working closely with the IT services department of the University of Applied Sciences during the hardware setup and the commissioning of the new server.
Further team member
Swiss Institute of Allergy and Astma Research (SIAF)
PD, Ph.D. Katja Bärenfaller
+41 81 410 08 49