\documentclass[11pt]{article}

\usepackage[right=0.8in, top=1in, bottom=1.2in, left=0.8in]{geometry}
\usepackage{amsmath,amsfonts,amssymb,amsthm}
\usepackage{mathtools}
\usepackage{graphicx}
\usepackage[colorlinks,linkcolor=black]{hyperref}
\usepackage{cleveref}
\usepackage{tikz}
\usepackage{url}
\usepackage{fixme}
\usepackage[roman]{complexity}
\usepackage{setspace}
\spacing{1.06}

\newtheorem{theorem}{Theorem}
\newtheorem{example}[theorem]{Example}
\theoremstyle{definition}
\newtheorem{definition}[theorem]{Definition}
\theoremstyle{remark}
\newtheorem{remark}[theorem]{Remark}

\newcommand{\handout}[5]{
    \noindent
    \begin{center}
        \framebox{
            \vbox{\vspace{0.25cm}
                \hbox to 5.78in { {CS Theory (Spring '25)} \hfill #2 }
                \vspace{0.48cm}
                \hbox to 5.78in { {\Large \hfill #5  \hfill} }
                \vspace{0.42cm}
                \hbox to 5.78in { {#3 \hfill #4} }\vspace{0.25cm}
            }
        }
    \end{center}
    \vspace*{4mm}
}

\newcommand{\lecture}[4]{\handout{#1}{#2}{#3}{#4}{#1}}

\begin{document}

\lecture{Lecture Note: Streaming Algorithms}{February 20, 2025}{Instructor:\hspace{0.08cm}\emph{Josh Alman }}{}

Until now, we introduced several computation models, including DFA, NFA, and regular expressions, but all of them only recognize regular languages. Today we introduce streaming algorithms, which can recognize more than just regular languages. In fact, we will prove that streaming algorithms can recognize all languages.

For streaming algorithms, in each step the algorithm can read a piece of input (usually a constant number of bits), and after reading a piece, the algorithm cannot see it again. Obviously, we can always store all the data we read, and then do what we want on the data. However, a typical scenario is that the input size is so big that we do not want to store the whole inputs, but we still want to figure out interesting things about the data. We want to use as little space as possible (while ensuring that we still have the correct results), and we will mainly measure the performance of the streaming algorithms based on its space usage.
%DFA, NFA and regular expressions allow us to differentiate languages based on whether they are regular, while streaming algorithms can differentiate languages based on the space usage of the algorithms.

We first give some real-life examples as motivation.

\begin{example}
    On \url{https://trends.google.com} (\Cref{fig: google trends}), we can see the recent trending searches. There are billions of searches on Google every day, and Google wants to process them to show the trending searches, while not storing all the searches.
\end{example}

\begin{figure}[ht]
    \centering\includegraphics[width=0.5\textwidth]{google\_trend.png}\caption{A screenshot of Google Trends}\label{fig: google trends}
\end{figure}

Some other examples include processing of experimental data from measurement devices: the device may generate lots of data in every second, but we only want to extract certain useful information.

%1011001 $\leftarrow$ \boxed{Alg} $\leftarrow$ 11000011110101100111$\cdots$

%so much that the alg does not want to have to store all of it but we want to still figure out interesting things about the data

%trends.google.com Hundreds of millions of search coming all time. Want to process this but not storing all the searches.

%financial transactions at a bank

%experimental date from hardware, measuring physical phenomenon.

\section{Streaming Algorithms}
\begin{example}\label{example: 2}
    Let $L_2 \coloneq \{w \in \{0, 1\}^* \mid w\text{ contains more $0$'s than $1$'s}\}$. A streaming algorithm that computes it is as follows:
    \begin{itemize}
        \item Variables: the algorithm maintains two variables $a, b$.
        \item Initialization: at the beginning it sets $a \coloneq 0$, and sets $b \coloneq 0$.
        \item Update rule: when the algorithm reads the next piece of data (in this case a symbol) $\sigma \in \{0, 1\}$, if $\sigma = 0$, then it lets $a \coloneq a + 1$, otherwise it lets $b \coloneq b + 1$.
        \item Stopping rule: after the algorithm reads the last piece of data, it accepts if $a > b$, and rejects otherwise.
    \end{itemize}
\end{example}

We now consider the space usage of the above algorithm. We will consider the usage in terms of $n = |w|$, which is the number of input bits, since we are concerned with how the space usage grows when the input size grows. 

The algorithm above stores only $2$ variables, but its space usage is not constant. This is because the variables $a$ and $b$ may be as large as $n$. We can store a non-negative integer that is at most $n$ using $\lceil \log_2(n + 1) \rceil = O(\log n)$ bits by storing it in its binary form,\footnote{Here $\lceil x\rceil$ means the smallest integer that is at least $x$. For example, $\lceil 2.2\rceil = 3$, $\lceil 2\rceil = 2$.} so the space usage of the above algorithm is $O(\log n)$.

\begin{remark}
    To store a non-negative integer that is smaller than $n$, we need at least $\log_2n$ bits of space. This is because there are $2^t$ possible configurations for $t$ bits of storage, and each possible value of the integer must correspond to a distinct configuration. Therefore, $n$, the number of possible values, is at most $2^t$.
\end{remark}


%Variables: $a$, $b$

%Initialization: set $a = 0$, set $b = 0$.

%Update rule: what do you do when you see the next datapoint. On input $\sigma \in \{0, 1\}$, if $\sigma = 0$, then let $a = a + 1$. if $\sigma = 1$, then let $b = b + 1$.

%Stopping rule: if $a > b$ then accept, else reject.

%Question: How much space does this streaming algorithm use? 

%$\infty$ But this is not the most useful answer.

%Better question: how much space does it use in terms of $n$, where $n = |w| = $ how many input bits we've seen.

%Storing just two variables! But each variable may need lots of space to store.

%$O(\log n)$ bits of space $a, b$ might each be integers from $O$ up to $n$. so we can store each with $O(\log n)$ bits.

%If you have $t$ bits, there are $2^t$ different settings for the bits, so you can use them to store up to $2^t$ options.

%$\log_2(n)$ bits $\to$ $2^{\log_2(n)} = n$ options.

%Is there an even better algorithm for this that uses even less space?    

%Store only the difference of the number of $0$'s and the number of $1$'s

Next time, we will prove we cannot do much better in terms of space usage. The intuition is as follows. Suppose $w$ is first a lot of $0$'s and then about the same number of $1$'s. Intuitively, you need to keep track of the exact number of $0$'s to see whether there are more $0$'s or more $1$'s.

Now we give the formal definition of streaming algorithms.

\begin{definition}[Streaming Algorithms]
    A streaming algorithm $\mathcal{A}$ over the alphabet $\Sigma$ has a working memory $M \in \Pi^*$ for some set $\Pi$. The memory can be viewed as a set of variables $M=\{X_1, \ldots, X_\ell\}$, and $\Pi$ is the set of possible values of these variables. These variables can store different kinds of information, and their values might change during the execution of the algorithm. The algorithm has the following three components: 
    \begin{itemize}
        \item An initialization rule $\mathcal{I} \in \Pi^*$. This tells the algorithm what $M$ should be before receiving any data.
        \item An update rule $\delta : \Pi^* \times \Sigma \to \Pi^*$. This tells the algorithm how to update the memory after reading a piece of data.
        \item A stopping rule $\gamma : \Pi^* \to O$. This tells the algorithm what to output at the end of the execution (after reading all data), depending on the content of the memory (i.e.~the value of the variables) at that time. Here $O$ is the set of possible outputs of the algorithm, and this would depend on the task we are trying to solve, and for us $O$ will usually be $\{\textsf{Accept}, \textsf{Reject} \}$.
    \end{itemize}
\end{definition}

% Streaming algorithm over alphabet $\Sigma$ has $3$ components:

% 1) Initialization: initialize all the variables to be used

% 2) Update rule: says for each input $\sigma \in \Sigma$ how to update the variables

% 3) Stopping rule: Whether to accept or reject when the stream ends

% Focus here on space usage of algorithm. E.g. we don't care (for now) about time usage.

We focus on space usage of the streaming algorithms (and don't card about time usage, at least for now). We now define the space usage.

\begin{definition}[Space usage]
    The space usage of a streaming algorithm $A$ is a function $S: \mathbb N_{\ge 0} \to \mathbb N_{\ge 0}$ where $S(n)$ is the maximum number of bits used to store $M$, the variables of $A$, over all possible inputs of length at most $n$.\footnote{$\mathbb N_{\ge 0}$ is the set of non-negative integers.}
\end{definition}

%Definition: Space usage of streaming algorithm $A$ is a function $s: \mathbb N_{\ge 0} \to \mathbb N_{\ge 0}$ ($\mathbb N$ is the set of non-negative integers) where $S(n) = $ the max number of bits used to store the variables of $A$ over all possible inputs of length $\le n$.

We now see another example.

\begin{example}
    Let $L_6 \coloneq \{w \in \{0, 1\}^* \mid \text{number of $1$'s in $w$ is divisible by $4$}\}$. A streaming algorithm that computes it is as follows:
    \begin{itemize}
        \item Variable: $a$.
        \item Initialization: set $a \coloneq 0$.
        \item Update rule: on input $\sigma \in \{0, 1\}$, if $\sigma = 0$, then let $a \coloneq a$, otherwise then let \[a \coloneq (a + 1) \bmod 4 = \begin{cases}
        a + 1 & a = 0, 1, 2\\
        0 & a = 3
        \end{cases}.\]
        Formally,
        \[\delta(a, \sigma) \coloneq \begin{cases}
            a & \sigma = 0\\
            (a + 1) \bmod 4 & \sigma = 1
        \end{cases}.\]
        \item Stopping rule: if $a = 0$, accept, else reject. Formally,
        \[\gamma(a) \coloneq \begin{cases}
            \mathsf{Accept} & a = 0\\
            \mathsf{Reject} & a \ne 0
        \end{cases}.\]
    \end{itemize}
\end{example}

In the algorithm above, $a$ always keep track of the number of $1$'s in the portion it has already read modulo $4$. Since the possible values of $a$ are $0, 1, 2, 3$, storing $a$ needs $\log_2 4 = 2$ bits, so the space usage of the algorithm is constant.\footnote{We usually call a function \emph{constant} if the function is $O(1)$ (thus bounded by a constant).}

%Example: $\{w \in \{0, 1\}^* \mid number of 1's in w is divisible by 4\}$

%Variable: $a$

%Initialization: set $a = 0$

%Update rule: on input $\sigma \in \{0, 1\}$, if $\sigma = 0$, then let $a = a$. If $\sigma = 1$, then let $a = (a + 1) \bmod 4$.

%Stopping rule: if $a = 0$, accept, else reject.

It is not hard to see that $L_6$ is a regular language. Below we will prove that regular languages are exactly those languages that can be computed by streaming algorithm with constant space usage.

\begin{theorem}
    Every regular language has a streaming algorithm with constant space usage.
\end{theorem}

\begin{proof}
    Since the language is regular, it is recognized by a DFA. Suppose the DFA is $\mathcal D = (Q, \Sigma, \delta, q_0, F)$. We construct the following streaming algorithm $\mathcal A$:
    \begin{itemize}
        \item Variable: $q \in Q$.
        \item Initialization: set $q \coloneq q_0$.
        \item Update rule: set $q \coloneq \delta(q, \sigma)$ where $\sigma$ is the current input. Formally, the update rule is exactly $\delta$.
        \item Stopping rule: if $q \in F$, accept, else reject. Formally,
        \[\gamma(q) \coloneq \begin{cases}
            \mathsf{Accept} & q \in F\\
            \mathsf{Reject} & q \notin F
        \end{cases}.\]
    \end{itemize}
    We can see that the algorithm $\mathcal A$ just simulates the DFA $\mathcal D$ on $\mathcal A$'s input, and the value of $q$ and the current state of $\mathcal D$ is the same after reading the same input. Suppose the final value of $q$ is $q^*$, since $\mathcal D$ recognizes the language, $q \in F$ if the input is in the language and $q \notin F$ otherwise. Therefore, $\mathcal A$ accepts if and only if the input is in the language. 
    
    Since $Q$ is finite, the space usage of $\mathcal A$ is $\log_2|Q| = O(1)$.
\end{proof}

\begin{theorem}
    If a language $L$ is computed by a streaming algorithm with constant space usage, then $L$ is regular.
\end{theorem}

\begin{proof}
    Denote by $\mathcal A$ the streaming algorithm. Suppose $Q$ is the set of all possible settings of $\mathcal A$. Since $\mathcal A$ uses constant space, $Q$ is finite. Suppose the initial setting of $\mathcal A$ is $q_0$, its update rule is $\delta$ and its stopping rule is $\gamma$. We define a DFA $\mathcal D = (Q, \Sigma, \delta, q_0, F)$ as follows.
    \begin{itemize}
        \item The set of state is just $Q$.
        \item The transition function is just $\delta$, that is, when the current input symbol is $\sigma$ and the previous state is $q$, the current state will be $\delta(q, \sigma)$.
        \item The start state is just $q_0$.
        \item The set of accepting states $F \coloneq \{q \in Q \mid \gamma(q) = \mathsf{Accept}\}$.
    \end{itemize}
    We can see (and formally prove by induction on $t$) that if the algorithm $\mathcal A$ and the DFA $\mathcal D$ read the same input $x_1, x_2, \dots, x_t$, the current setting of the memory of $\mathcal A$ will be the same as the current state of $\mathcal D$. Suppose the final setting of the memory of $\mathcal A$ is $q^*$, then the final state of $\mathcal D$ is also $q^*$. Therefore, $\mathcal A$ accepts if and only if $\gamma(q) = \mathsf{Accept}$, which is equivalent to $q \in F$. Thus, $\mathcal D$ recognizes the same language as $\mathcal A$, so $\mathcal D$ recognizes $L$ and $L$ is regular.
\end{proof}

%Theorem: Every regular language has an $O(1)$ space streaming algorithm. 

%Proof: Since the language is regular, it has a DFA $D = (Q, \Sigma, \delta, q_0, F)$.

%Here is out streaming alg:

%vars: $q \in Q$
%init: set $q = q_0$
%update: on input $\sigma \in \Sigma$, set $q = \delta(q, \sigma)$.
%stop: accept if $q \in F$, reject otherwise.

%Is it true that every language with an $O(1)$ space streaming algorithm is a regular language? 

%Go into more detail here

%Proof sketch: Suppose the language has an $O(1)$ space streaming algorithm $A$. We'll make a DFA that basically runs $A$.

%$Q = $ set of all assignments to the variables of $A$.

%$q_0 = $ assignment you got agter running initialization of $A$

%$F = $ set of assignments such that the stopping would accept given that assignment of the vars

%$\delta = $ given state $q$ (an assignment of vars of $A$) and char $\sigma \in \Sigma$ run the update rule with that assignment and $\sigma$, move to the new resulting assignment of vars.

%A language is regular if and only if it has an $O(1)$ space streaming algorithm.

Now we see yet another example.

\begin{example}
    Let $L_9 \coloneq \{w \in \{0, 1\}^* \mid w\text{ is a palindrome}\}.$ Here we call a string a palindrome if it reads the same forwards and backwards. (For example, $111010111$ is a palindrome.) We construct a streaming algorithm for $L_9$ as follows.
    \begin{itemize}
        \item Variable: $a$.
        \item Initialization: let $a \coloneq \varepsilon$.
        \item Update rule: on input $\sigma \in \Sigma$, set $a = a \circ \sigma$.
        \item Stop rule: accept if $a$ is a palindrome, otherwise reject.
    \end{itemize}
\end{example}

We can see that after reading the whole input, $a$ is just the whole input, so the algorithm works. The space usage of the above algorithm is $O(n)$ since we will store the whole input string in $a$. We will show later that we cannot do better than this. 

Besides, we can see that the algorithm above is not specific to this language, except for the stop rule. In fact, for every algorithm there is a similar algorithm.

%$\{w \in \{0, 1\}^* \mid w\text{ is a palindrome}\}$. (reads the same forwards and backwards) e.g. $111010111$

%var: $a$

%init: $a = \varepsilon$

%update on input $\sigma \in \Sigma$, set $a = a \circ \sigma$

%stop: accept if $a$ is a palindrome, otherwise reject.

%$O(n)$ space usage

%can't do better than this!

\begin{theorem}
    Every language has a streaming algorithm that takes $O(n)$ space.
\end{theorem}

\begin{proof}
    Let $L$ be an arbitrary language. We construct the following algorithm.
    \begin{itemize}
        \item Variable: $w$.
        \item Initialization: set $w = \varepsilon$.
        \item Update rule: on input $\sigma \in \{0, 1\}$, set $w = w \circ \sigma$.
        \item Stop rule: accept if $w \in L$, reject otherwise.
    \end{itemize}
    After reading the whole input, $w$ will become the input string, so the algorithm accepts if and only if $w \in L$. During the execution of the algorithm, $w$ is a string of length at most $n$, so the algorithm takes $O(n\log|\Sigma|) = O(n)$ space where $\Sigma$ is the alphabet of $L$ and is finite.
\end{proof}
%var $w$

%init set $w = \varepsilon$

%update: on input $\sigma \in \{0, 1\}$, set $w = w \circ \sigma$.

%stop: if $w$ is in the language then accept, else reject.

To summarize, in this lecture we see three types of streaming algorithms in terms of space usage:
\begin{itemize}
    \item $O(1)$-space streaming algorithm for any regular language.
    \item $O(\log n)$-space streaming algorithm for $L_2$ defined in \Cref{example: 2}.
    \item $O(n)$-space streaming algorithm for any language.
\end{itemize}

%Summary: $O(1)$ space $\leftarrow$ regular languages

%$O(\log n)$ space $\leftarrow$ e.g. number of $0$'s $>$ number of $1$'s

%$O(n)$ space $\leftarrow$ all languages
\end{document}