I finally published my Masters thesis this year! I now have a Masters of Science in Computer Science!! That means in my official title, the word “science” appears 3 times now. Dr. Spock would be proud… if he could feel emotions that is.
This is incredibly exciting, however I doubt many people will be interested in reading the entire 49ish page document. It’s available here if you prefer, but I will break my thesis down into a series of blog posts. I will do my best to unpack the formal academic presentation of the full paper into plain language.
My thesis topic officially is I/O Throughput Prediction for HPC Applications Using Darshan Logs. This is a very dense title. Basically, I made a system that predicts the input and output (I/O) rate of files in a supercomputer, or High Performance Computer (HPC). I used machine learning to analyze log files. The model I produced is able to predict the I/O based on the logs, better than any previous model.
Now, why do we care about predicting I/O in an HPC system? This particular system has 49,000 central processing units (CPU’s). The system is used for scientific research into a wide number of applications. The compute power is very optimized, and scaled horizontally, taking a large number of CPU’s and networking them together into one very powerful system. One of the major challenges with operating an HPC system like this is getting the data to the compute cores fast enough to feed the incredible processing capacity of this system.
It is very difficult to analyze the performance of an HPC system, because of the scale of the operations that are happening are simply too much for any person or group of people to manually analyze the hundreds of simultaneous jobs running and very very large amounts of data moving through the network. This requires a very in depth analysis of log files which themselves are hundreds of Terabytes in size.
So in summary, I programmed a custom tool set to help me analyze and build a predictive model to study the rate of I/O in a super computer. More on this soon!