\label{results}
In this section, we will take a look at how outer solver FGMRES preconditioned with either different implementations of \cg{} method solvers or exact subdomain solutions perform in different setups and also show a comparison between solvers using evaluation metrics mentioned in evaluation subsection \ref{evaluation}.

\begin{figure}[h]
    \centering
    \includegraphics[width=1\linewidth]{images/Figure_wave.png}
    \caption{3D Visualization of the Average Solution Showcasing the Propagation of a Frontal Wave Across the Domain}
    \label{fig:frontalwave}
\end{figure}

\subsection{Time complexity of the solvers}
As time complexity is one of the most useful indicators of performance, we have performed a series of experiments on the Falcon HPC cluster nodes to measure the execution times of various solvers. The graphs will display the comparison of execution times in different subdomain setups. These execution times are recorded in seconds. In the following experiments, we solve the Helmholtz problem with the FGMRES method as an outer iterative method that is preconditioned either by exact subdomain solutions in the RAAS method or different implementations of the CG method. The stopping criteria for the FGMRES method is the initial residual norm decreased by 6 orders of magnitude; we perform $4*subdomain\_width$ CG iterations in subdomain solves. If not stated otherwise, we fix $k$ and $eps$ 20. The subdomain overlap, if not stated otherwise, is chosen to be $(subdomain\_width/2-1)$. The initial guess of the CG method is chosen to be 0s for the experiments. Moreover, calculations are done for special RHS that produce a frontal wave moving across a medium. This wavefront starts at one corner and moves diagonally across to the opposite corner of the domain; see Figure \ref{fig:frontalwave}. The graph also illustrates how the wave amplitude varies in a square domain.

\subsubsection{Time complexity comparison in different subdomain setups}
The first element in each tuple shows the number of subdomains, and the second element shows the width of each subdomain, which is the number of subdomain nodes in each direction. As the width of subdomains increases, particularly from 64 to 128, there is a substantial increase in the computational demands across all methods, see Figure \ref{fig:cg-time-comparision}.

The Scipy exact solver shows a steep exponential increase as the subdomain width increases. This dramatic rise suggests that the exact subsolves solution method’s computational demands grow significantly with larger and more complex subdomain configurations. It starts with the lowest value but ends with the 2nd highest, indicating that it may not scale well with increasing problem size. A similar pattern is also observed in the CG solver written in Numpy. Although it shows good performance in small subdomain setups, the performance of the solver is decreased in bigger subdomains compared to other implementations.

The Performance curve for the PyOpenCL implementation with a single right-hand side starts with the worst performance in the case of 9 subdomains, having 27 subdomain nodes in each direction.
However, it performs better as the subdomain number and size increase. That shows the scalability and efficiency of the solver in bigger subdomains which is typical for Helmholtz problems.

The best performance is obtained in PyOpenCL implementation of \bcg{} method for multiple RHSs. We can see from the chart that multiple RHS implementations have always outperformed all the solvers. 

\begin{figure}[h]
    \centering
    \includegraphics[width=1\linewidth]{images/Figure_1.png}
    \caption{Comparison of time complexity of different CG solvers}
    \label{fig:cg-time-comparision}
\end{figure}


\subsubsection{Time complexity in a big subdomain setup}
A similar experiment is performed in a bigger subdomain setup, having 256 subdomain nodes in each direction and 16 total subdomains, see Figure \ref{fig:ex-t-big}. The number of iterations for the CG method is set at 1000. This setting can impact performance, as too many iterations may waste resources. On the other hand, setting too few iterations might slow down convergence or, in some cases, prevent convergence completely. For example, while running the experiment within the same subdomain setup, having 500 CG iteration didn't converge to the solution because of a few iterations. The size of submatrices obtained in this setup was 260100, which means that the size of the original matrix is 4161600.

The blue bar in the chart represents the execution time of the Scipy exact solver. In total, the exact solver took 5223 seconds (87 mins) to calculate the $Ax=b$ linear equation for 16 subdomains, which was 12 minutes faster than NumPy subsolves. Higher numbers are indicative of its computationally intensive nature. Exact subsolves often involve direct methods, which, while accurate, can be significantly slower, especially in complex or large-scale problems where approximation methods might be more efficient.

Numpy Solver solved the problem in 100 minutes, being the last in the ranking. It also took 21 iterations of the outer solver to converge to the true solution, while the exact subsolver only had 20 iterations to converge. Considering the Numpy CG solver is using approximate solutions, having almost the same number of outer iterations is quite impressive.

PyOpenCL solution for CG for multiple RHSs shows a significant reduction in execution time when solving for multiple RHS vectors simultaneously. With 21 outer iterations, it solved the problem in approximately 16 min that is more than 6 times faster than the Numpy solver.

PyOpenCL CG for a single RHS also performs considerably faster than exact subsolves and NumPy CG subsolves because of it is efficient usage of GPUs.
\begin{figure}[h]
    \centering
    \includegraphics[width=1\linewidth]{images/Figure_2.png}
    \caption{Execution times for CG solvers in bigger domain with 16 subdomains, each having the size of 256}
    \label{fig:ex-t-big}
\end{figure}

\subsection{Subdomain overlap size}
We have conducted 2 experiments with the size of subdomain overlaps. The first experiment illustrates how the number of outer iterations of the FGMRES method changes as subdomain overlap size increases. This will allow us to see how many iterations it takes to converge to the true solution. The second experiment shows how the execution time varies with different sizes of subdomain overlap for each solver. It will help us see if more overlaps lead to longer or shorter execution times for different solvers. 

For these experiments, we have used 16 subdomains, each having a subdomain size of 64. The number of CG iterations was fixed to 256, wavenumber $k$ and $eps$ to 20.

\subsubsection{The impact of subdomain overlaps on the convergence of solution}
In figure \ref{fig:it-ol}, the number of outer iterations needed to converge across different subdomain overlaps is illustrated.
The number of iterations needed to converge decreases as the size of subdomain overlap increases, stabilizing at 19 iterations from 21 subdomain overlap sizes onward. This suggests that having a higher subdomain overlap size can initially speed up the convergence. However, a bigger number of subdomain overlaps doesn't bring any benefit but increases the computational time to solve extended subdomains. Moreover, it is worth mentioning that none of the solvers converged when the number of overlap sizes was bigger than the subdomain size. 
\begin{figure}[h]
    \centering
    \includegraphics[width=1\linewidth]{images/Figure_4.png}
    \caption{Convergence Iterations vs. Subdomain Overlap Size}
    \label{fig:it-ol}
\end{figure}

\subsubsection{The impact of subdomain overlaps on the performance of solvers}

The execution time for four different solvers was compared with the number of subdomain overlap sizes ranging from 1 to 60. The line plot displays the execution times for each solver as the number of subdomain layers increases; see Figure \ref{fig:ex-t-ol}. 

Exact subsolves exhibited a relatively linear increase in execution time with increasing subdomain overlaps. The execution time was moderate at lower layer counts but escalated notably beyond 20 layers. For exact subsolves, one may expect that the computational overhead would grow in larger domains and, hence, be one of the reasons some efficiency gets lost with a larger overlap.

The execution time of PyOpenCL CG for multiple RHSs initially decreases when the number of subdomain overlap sizes changes from 1 to 13, indicating improved efficiency of the solver with moderate overlaps. As the width of the overlap increases, the number of outer iterations decreases, but the size of the subdomain problem grows. Beyond 13 layers, the execution time increased gradually, reflecting the diminishing returns in efficiency at higher overlaps.

\begin{figure}[h]
    \centering
    \includegraphics[width=1\linewidth]{images/Figure_3.png}
    \caption{Execution Time vs. Subdomain Overlap Size}
    \label{fig:ex-t-ol}
\end{figure}

Starting with execution times comparable to the Exact subsolves, Numpy CG's execution time increased significantly with more overlaps. The substantial rise in time after 20 layers was the steepest among the solvers, suggesting that while Numpy CG can handle lower overlaps efficiently, its performance degrades under larger computational loads typical of higher overlaps.

Overall, the data suggests that increasing the subdomain overlap size can initially reduce the computational complexity. Especially up to the point where the overlap doesn't overlap with the third subdomain overlap. In order to avoid such situations, we have a formula $(subdomain\_width/2-1)$ to calculate the maximum subdomain overlap without extending over the second subsequent subdomain. Also, excessive overlaps lead to a general increase in execution time across all solvers. This effect is most pronounced in solvers like Numpy CG, which do not inherently parallelize operations as effectively as PyOpenCL-based methods.

From the graphs, we can come to the conclusion that PyOpenCL CG implementation for multiple RHSs showed the best performance when the subdomain overlap size was 13, which is approximately one-fifth of the subdomain size.

\subsection{Influence of the wave number to the convergence}
As it was mentioned in earlier sections, we have a function to generate Helmholtz problems. One of the parameters of this function is wave number $k$, which influences the complexity of the domain. In this section, we are going to check the convergence of solvers while using different wave numbers. PyOpenCL-CG for multiple RHSs is the inner solver for this particular testing, FGMRES being the outer solver. For the experiment we have set the total number of subdomains to 16 and the size of each subdomain to 128. Moreover, the subdomain overlap size was set to 31, and the damping number $\epsilon$ to 20. The wavenumber varied from 5 to 150.

\begin{figure}[h]
    \centering
    \includegraphics[width=1\linewidth]{images/Figure_wavenumber.png}
    \caption{Wavenumber vs. Outer iterations and Execution Time}
    \label{fig:wavenumber}
\end{figure}

The execution time increases as the wavenumber increases from 5 to 50, which is expected due to the increasing complexity of the problem. Higher wavenumbers in Helmholtz equations often make the problems more oscillatory and harder to solve due to their impact on the eigenvalue distribution of the matrix. However, it is surprising that the solver computes the problem in less time when the wavenumber increases to 100 and 150. This behavior of the solver can be investigated further.

The trend for the number of outer iterations follows the same pattern as the execution time. Obviously, it takes more to solve the problem as the number of iterations increases.

\subsection{Weak scaling testing for PyOpenCL-CG for a single RHS}
In preparation for weak scaling tests we have performed a series of experiments with keeping the subdomain size fixed while changing the number of subdomains. Note that here we still use a single node with a single GPU. Real weak scaling tests will be performed as a later contribution.

The weak scaling testing is performed on the implementation using the inner solver of PyOpenCL CG for the single RHS where the workload per subdomain remains constant, but the total size of the problem increases. For the testing, the subdomain size is fixed to 64, the number of CG iterations to 256, the number to 20, and the subdomain overlap size to the max, which is 31. You can also see the example of how the number of subdomains increases during the testing; see Figure \ref{fig:domain-dist}. Total number of subdomains used for the testing are 16, 25, 36, 49, 64, 81, 100.

\begin{figure}[h]
    \centering
    \includegraphics[width=1\linewidth]{images/Figure_dom_dist.jpg}
    \caption{Example of subdomain size increase in weak scaling testing}
    \label{fig:domain-dist}
\end{figure}

From the graph, we can see that execution time increases significantly as the number of subdomains increases. This increase appears to be more than linear, suggesting that adding more subdomains leads to disproportionately higher computation times. Some of the possible reasons for that increase are increased communication overhead, synchronization costs, or inefficient parallelization could contribute to this trend. As more subdomains are added, the complexity of managing them and the data exchange between them might increase. Also, it was surprising to see some of the iterations took significantly more time compared to others. For example, although the average time for iterations was 23 seconds while solving a 9x9 domain, some iterations took more than 100 seconds to compute.

Expectedly, the number of outer iterations (FGMRES iterations) also increases as the number of subdomains increases (Figure \ref{fig:wst}). This suggests that the problem becomes more challenging to solve iteratively as it is divided into more subdomains. It indicates that the preconditioner becomes less efficient as the domain is split into more parts. At the same time, of course, the potential for further parallelism is increasing, making it possible to add more nodes with GPUs to contribute to the solution process.

\begin{figure}[h]
    \centering
    \includegraphics[width=1\linewidth]{images/Figure_wst.png}
    \caption{Number of Subdomains vs. Outer Iterations/ Execution time}
    \label{fig:wst}
\end{figure}

