\documentclass{article}

\usepackage{hyperref}

\usepackage{graphicx}
\usepackage{amsmath}
\usepackage{bm}

\usepackage{listings}
\lstset{language=Haskell
       ,captionpos=b}

\title{Parallel Cellular Automata Fluid Simulation}
\author{Rongcui Dong}

\begin{document}
\maketitle

\section{Introduction}\label{sec:introduction}

This project implements a simple fixed time step cellular automata fluid simulator using finite difference method.
Two programs are included, \verb|cellularfluid-sim| and \verb|cellularfluid-view|.
The former implements a command line program, 
reading initial state from an input file and outputting states of each time step to an output file.
The latter reads the output file generated by the simulator,
and plays back the simulation using an SDL2 window at nominal frame rate of 60 FPS.
Detailed usage is included in Appendix \ref{app:usage}.

\section{Algorithm}\label{sec:algoithm}

\subsection{Cellular Automata}\label{ssec:cell}

The project's algorithm is based on a grid cellular automata where the state of each grid depends only on the state of its eight adjacent grids.
The grid is a \verb|Data.Vector| of cells, stored in row major format.
Each cell may be a fluid cell, a wall, or an edge.
As will be described in the Simulation section, the wall and edge distinction was originally useful, 
but in final implementation they behave the same.

The grid's origin is at top-left, with $x$ going right and $y$ going down.
This grid structure yields the following type for next state function, which is mapped over the grid:

\begin{verbatim}
stepCell :: AdjCells -> Cell -> Cell
\end{verbatim}

The adjacent cells themselves are created by a vector \verb|imap| operation.
An adjacent cell located outside the grid is assumed to be an edge.

In essense, the next state function computes the next value of the center cell given a local $3\times3$ grid.
I built parallism out of this map operation.

\subsection{Simulation: Navier-Stokes Equation}

Originally, I attempted to simulate the an incompressible Navier-Stokes equation
without external force \cite{wiki:nv, stam1999stable}, as shown in Equation \ref{eq:nv}.
However, this resulted in extremely numerically unstable simulation, 
oscillating within a few time steps and blowing up to NaN quickly.
Therefore, the final implementation does not use this simulator.
This simulation method is described for completeness and for show of effort.

\begin{equation}\label{eq:nv}
    \frac{\partial\boldsymbol{u}}{\partial t} = 
    - (\boldsymbol{u}\cdot\nabla)\boldsymbol{u}
    - \frac{1}{\rho}\nabla p
    + \nu\nabla^2 u
\end{equation}

I discretized the equation combining methods described in \cite{stam1999stable, foster1996realistic}.
I split the integrator into stages like \cite{stam1999stable}, 
but in places where solving linear or Poisson systems were required, 
I used relaxation method described in \cite{foster1996realistic}.
This was done to avoid treating the simulation grid as a large matrix
and defeating the purpose to use cellular automata in the first place.

I used staggered grid discretization, as showin in Figure \ref{fig:stagger}.
Therefore, for the 9 cells available for state updating, 
there are 9 defined points of velocity $\boldsymbol{u}$ (blue dots),
6 defined points for each component of gradient velocity $\boldsymbol{u}_x$ (red) and $\boldsymbol{u}_y$ (green).
Pressure in staggered grid is also similar.

\begin{figure}[htpb]
    \includegraphics[width=\textwidth]{stagger}
    \caption{Staggared grid. Blue: $a$; red: $a_x$; green: $a_y$}\label{fig:stagger}
\end{figure}

Quantities at all points other than defined ones were linearly interpolated from the closest 4 valid points forming a square.
Any quantity outside the rectangle bounded by all valid points were ``clamped'' to the boundary.  
Therefore, time step must be sufficiently small for discretization to be valid.

The following are discretization methods I used on defined points,

\begin{align}
    \boldsymbol{u}_x(x, y) &= \frac{u(w/2+x, y) - u(-w/2+x, y)}{w}\\
    \boldsymbol{u}_x(x, y) &= \frac{u(x, w/2+y) - u(x, -w/2+y)}{w}\\
    \nabla\cdot\boldsymbol{u}(x, y) &= \frac{u_x(w/2+x,y) - u_x(-w/2+x,y) + u_y(x,w/2+y) - u_y(x,-w/2+y)}{w}
\end{align}
where $w$ is grid width.

A wall is a boundary condition where velocity is always $0$, and pressure gradient is $0$.
An edge is a boundary condition where both velocity gradient and pressure gradient are $0$

The first step in integration was advection step, corresponding to the second step in \cite{stam1999stable},

\begin{equation}
    \boldsymbol{u}_1 = \boldsymbol{u}(\boldsymbol{p}(\vec{x}, -\Delta t))
\end{equation}
where $\boldsymbol{p}(\vec{x}, t)$ is the velocity at $\vec{x} + \boldsymbol{u}(\vec{x})t$.
In other words, it is the velocity obtained by tracing backwards in the velocity field for $\Delta t$.
This step was straightforward, by sampling $\boldsymbol{u}(x - \Delta t \boldsymbol{u})$.

The second step was diffusion step, corresponding to the third step in \cite{stam1999stable}.

\begin{equation}
    \frac{\partial \boldsymbol{u}_2}{\partial t} = \nu\nabla^2\boldsymbol{u}_1
\end{equation}

In that paper, the diffusion step is solved using an implicit method.
For simplicity, I used explicit method for integration:

\begin{equation}
    \boldsymbol{u}_2 = \boldsymbol{u}_1 + \Delta t\frac{\partial \boldsymbol{u}_2}{\partial t}
\end{equation}

In the final step, divergence-free component of vector field $\boldsymbol u$ was solved to fullfill the incompressibility condition:

\begin{equation}
    \nabla\cdot\boldsymbol{u} = 0
\end{equation}

Since I tried to avoid transforming the problem into matrix operations,
I chose the velocity-pressure relaxation described in \cite{foster1996realistic}.
Note that due to the simulator's architecture (Section \ref{ssec:cell}),
the next state function could only update the center cell, 
the relaxation was done locally on the $3\times3$ grid and only the center cell's value is kept.

The relaxation started with updating pressure, a process modified from \cite{foster1996realistic}:

\begin{align}
    \Delta p &= -\beta \nabla\cdot\boldsymbol{u} \\
    \beta &= \frac{\beta_{0} w^2}{4\Delta t}
\end{align}
where $\beta_0$ is a relaxation coefficient chosen for numerical stability, in range $[1, 2]$\cite{foster1996realistic}.

Then, local $3x3$ grid velocities were updated as following:

\begin{align}
    \Delta\boldsymbol{U} = 
    \begin{pmatrix}
        0 & (0, -\Delta t \Delta p / w) & 0 \\
        (-\Delta t \Delta p / w, 0) & 0 & (\Delta t \Delta p / w, 0) \\
        0 & (0, \Delta t \Delta p / w) & 0
    \end{pmatrix}
\end{align}

Then, center cell pressure was updated by $\Delta p$.
The relaxation process was repeated until $\nabla\cdot \boldsymbol{u}$ was sufficiently small or until maximum iteration.

Finally, center cell velocity was updated by
\begin{equation}
    \Delta\boldsymbol{u} = \alpha \Delta p
\end{equation}
where $\alpha$ is a coefficient to guess the relaxation results for adjacent cells.

As the entire simulator used explicit integration, relaxation, and heuristics, 
the parameters were extremely difficult to tune for stability,
and simulation diverged numerically even with small and smooth inputs. 
Therefore, the final implementation did not use the Navier-Stokes simulator.


\subsection{Simulation: Divergence Flow}

Due to the failure of the Navier-Stokes simlator, 
I designed a simple and stable fluid simulation algorithm that models only diffusion.

The central equation is:

\begin{align}
    \tilde{p} &= p + (\nabla\cdot p) \Delta t / \alpha \\
    \alpha &= \alpha_{0}\nu
\end{align}
where $\alpha_0$ is a coefficient on the order of $10^9$, 
used to dampen diffusion and provide numerical stability.

$\nabla\cdot p$ was discretized so that it used information on all adjacent cells:

\begin{equation}
    \nabla\cdot p = \frac{4p - \sum_{p'\in edge} p'}{w} + \frac{4p - \sum_{p'\in corner} p'}{\sqrt{2}w}
\end{equation}

Walls and edges are both treated with pressure gradient of $0$.

This solver was stable, and allowed visually verification, such as in Figure \ref{fig:div-flow-sim}.

\begin{figure}[htpb]
    \begin{minipage}{.5\textwidth}
        \includegraphics[width=.9\textwidth]{div-flow-sim-0}
    \end{minipage}
    \begin{minipage}{.5\textwidth}
        \includegraphics[width=.9\textwidth]{div-flow-sim-1}
    \end{minipage}
    \caption{Small input simulation screenshots showing diffusion over time}\label{fig:div-flow-sim}
\end{figure}


\section{Parallelization}\label{sec:parallel}

Multiple parallization methods were attempted.
All benchmarks were performed on a i9-9900K CPU with 8 physical core and 16 threads, with 64GB of memory.
I identified, through profiling, two main area of performance bottleneck:
grid stepping and grid outputting.
The stepper was accelerated via parallization, while grid output acceleration was attempted using a dedicated output thread.

For the stepper, multiple data structures were tested, and all were based on the Vector type.
The first method I tried was encapsulating chunks of vectors in a datatype \verb|ParVector a|,
providing \verb|map| and \verb|imap| interface so semantically it functioned like \verb|Vector a|.
The second method I tried was to keep the \verb|Vector a| structure, 
and to provide parallel versions of \verb|map| and \verb|imap|.
In each method, I tested chunking as \verb|Vector (Vector a)| and as \verb|[Vector a]|.
In addition, I tested sparking with \verb|par|, \verb|parMap| or \verb|parTraversable|, and the Par monad.
In each chunking method, I tested using Vector \verb|Bundle| in hopes it is more efficient in concatenation.

For the output thread, I tested blocking IO in the computation thread
and sending the grid through a \verb|TBMQueue| of various sizes to an output thread.
In each case, I tested cloning the vector and using the vector as is.
In addition, I compared performance of storing as text format and serializing into binary format.
In the same-thread case, I also tried various buffer sizes.


\section{Results}\label{sec:result}

Table \ref{tab:results} compares performance tests using final parallel implementation 
and sequential implementation (not included in source code, but easy to change by substituting all \verb|parMapV| and \verb|parMapV| with \verb|map| and \verb|imap|).

\begin{table}[htpb]
    \begin{tabular}{|c|c|c|c|c|c|c|}
        \hline
        Grid Size & Time Steps & N1 & N2 & N4 & N8 & N16 \\
        \hline
        $16\times16$ & 600 & $0.348s$ & $0.357s$ & $0.372s$ & $0.410s$ & $0.619s$ \\
        \hline
        $256\times256$ & 600 & $40.5s$ & $22.0s$ & $19.5s$ & $27.6s$ & $44.1s$ \\
        \hline
        $512\times512$ & 600 & $2m50s$ & $2m16s$ & $1m58s$ & $2m14s$ & $4m58s$ \\
        \hline
    \end{tabular}
    \caption{Benchmark Results}\label{tab:results}
\end{table}

I found that \verb|ParVector| and parallelized \verb|Vector| had nearly identical performance,
given identical chunk size.
Therefore, for readability I kept \verb|Vector| in the final implementation.
In either case, chunking did not provide any advantages until they were at least 1024 cells each,
and until the simulation grid was at least $256\times256$ in size.
In addition, chunking as \verb|[Vector a]| provided significant speed up over \verb|Vector (Vector a)|,
possibly due to how they were manipulated when they were split and concatenated.
Using \verb|Bundle| provided no speedup.
In all cases, the Par monad gave best performance, but \verb|[Vector a]| chunking with \verb|parMap| came close.

Single thread output writing with large buffer size worked better than sending to another output thread.
Cloning did not help making output and computation concurrent, and had no positive effects.
In all cases, binary output performed better than text output.

In final implementation, profiling showed that IO was not the performance bottleneck:

\tiny\begin{verbatim}
COST CENTRE          MODULE                                SRC                                                    %time %alloc

divP'.f              CellularFluid.Grid.FD.FDSimpleStepper src/CellularFluid/Grid/FD/FDSimpleStepper.hs:21:5-20    21.7    7.7
>>=                  Data.Vector.Fusion.Util               Data/Vector/Fusion/Util.hs:36:3-18                      11.2   10.6
adjCells.f.adjEdge   CellularFluid.Grid                    src/CellularFluid/Grid.hs:60:11-51                       8.3    4.4
primitive            Control.Monad.Primitive               Control/Monad/Primitive.hs:195:3-16                      7.7    9.2
adjCells.f.adjCorner CellularFluid.Grid                    src/CellularFluid/Grid.hs:61:11-57                       6.9    4.5
fmap                 Data.Vector.Fusion.Stream.Monadic     Data/Vector/Fusion/Stream/Monadic.hs:(133,3)-(135,20)    6.2    7.6
sum                  Data.Vector                           Data/Vector.hs:425:3-11                                  4.9    3.8
basicUnsafeWrite     Data.Vector.Mutable                   Data/Vector/Mutable.hs:118:3-65                          3.6    5.1
basicUnsafeGrow      Data.Vector.Generic.Mutable.Base      Data/Vector/Generic/Mutable/Base.hs:(138,3)-(144,23)     3.4    1.4
basicUnsafeIndexM    Data.Vector                           Data/Vector.hs:277:3-62                                  3.0    2.2
adjCells.f           CellularFluid.Grid                    src/CellularFluid/Grid.hs:(50,5)-(63,19)                 2.4    3.6
cellAt               CellularFluid.Grid                    src/CellularFluid/Grid.hs:(34,1)-(42,20)                 2.3    2.4
cellAt.idx           CellularFluid.Grid                    src/CellularFluid/Grid.hs:39:5-24                        2.2    0.0
basicUnsafeNew       Data.Vector.Mutable                   Data/Vector/Mutable.hs:(99,3)-(102,32)                   2.1    3.5
rnf.rnfAll           Data.Vector                           Data/Vector.hs:(225,11)-(226,36)                         1.9    0.5
basicUnsafeSlice     Data.Vector.Mutable                   Data/Vector/Mutable.hs:89:3-62                           1.2    4.1
parMapV.f'           Data.Vector.Parallel                  src/Data/Vector/Parallel.hs:38:5-27                      1.1    0.9
basicUnsafeFreeze    Data.Vector                           Data/Vector.hs:(263,3)-(264,47)                          1.0    3.2
basicUnsafeCopy      Data.Vector.Mutable                   Data/Vector/Mutable.hs:(121,3)-(122,36)                  0.5    1.8
>>=                  Data.Serialize.Put                    src/Data/Serialize/Put.hs:(173,5)-(176,37)               0.4    6.1
>>=.(...)            Data.Serialize.Put                    src/Data/Serialize/Put.hs:175:13-36                      0.3    2.4
*>                   Data.Serialize.Put                    src/Data/Serialize/Put.hs:(162,9)-(165,41)               0.1    1.7
\end{verbatim}\normalsize

Using threadscope, I identified sequential regions between two time steps, as shown in Figure \ref{fig:threadscope}.
This region was present in all implementations, including the sequential one.
I could not identify its origin from threadscope, and ghc-events-analyze seemed to suggest an internal synchronization of the \verb|Data.Vector| implementation.
The \verb|ParVector| and cloning was an attempt to avoid this overhead, but they both had no effect.
The first sequential region in the figure is reading grid input.

\begin{figure}[htpb]
    \includegraphics[width=\textwidth]{threadscope}
    \caption{Threadscope output of a typical simulation.}\label{fig:threadscope}
\end{figure}

\section{Conclusion}

In summary, the stepper of the simulator could be parallized.
However, an unidentified sequential region limited the amount of speed up achievable.


\bibliographystyle{plain}
\bibliography{refs}

\appendix

\section{Command Line Usage}\label{app:usage}

\begin{verbatim}
Usage: cellularfluid-sim [--version] [--help] [-v|--verbose] (-i|--input ARG)
                         (-o|--output ARG) --time ARG

Available options:
  --version                Show version
  --help                   Show this help text
  -v,--verbose             Verbose logging?
  -i,--input ARG           Input file
  -o,--output ARG          Output file
  --time ARG               Simultaion time
\end{verbatim}

\begin{verbatim}
Views fluid simulation

Usage: cellularfluid-view [--version] [--help] [-v|--verbose] [--width ARG]
                          [--height ARG] [--hidpi] (-i|--input ARG)

Available options:
  --version                Show version
  --help                   Show this help text
  -v,--verbose             Verbose logging?
  --width ARG              Window width
  --height ARG             Window height
  --hidpi                  HiDPI support
  -i,--input ARG           Input file
\end{verbatim}

\section{Grid Input Format}\label{app:grid}

First line is a header storing grid metadata, separated by space:

\scriptsize
\verb|Width(# Columns)::Int Height(# Rows)::Int Grid_Size::Double Density::Double Viscosity::Double|
\normalsize

After the header, each entry has its own format, while entries are separated by spaces.

Walls are specified with \verb|W|; 
fluid cells using divergence-flow simulation are specified with \verb|FD p| where \verb|p| is a \verb|Double| value for pressure.
Edges cannot be specified in grid input.

The number of cells must match $width \times height$ specified in header,
or the program fails a sanity check and exits without starting simulation.

\section{Code Listing}\label{app:code}

\scriptsize
\lstinputlisting[caption={app/Main.hs}]{../app/Main.hs}
\lstinputlisting[caption={src/CellularFluid.hs}]{../src/CellularFluid.hs}
\lstinputlisting[caption={src/Import.hs}]{../src/Import.hs}
\lstinputlisting[caption={src/Run.hs}]{../src/Run.hs}
\lstinputlisting[caption={src/Sim.hs}]{../src/Sim.hs}
\lstinputlisting[caption={src/Types.hs}]{../src/Types.hs}
\lstinputlisting[caption={src/CellularFluid/Grid.hs}]{../src/CellularFluid/Grid.hs}
\lstinputlisting[caption={src/CellularFluid/Grid/Types.hs}]{../src/CellularFluid/Grid/Types.hs}
\lstinputlisting[caption={src/CellularFluid/Grid/Parse.hs}]{../src/CellularFluid/Grid/Parse.hs}
\lstinputlisting[caption={src/CellularFluid/Grid/FD/FDSimpleStepper.hs}]{../src/CellularFluid/Grid/FD/FDSimpleStepper.hs}
\lstinputlisting[caption={src/Data/Vector/Parallel.hs}]{../src/Data/Vector/Parallel.hs}
\normalsize

\end{document}
