Bartosz Milewski's Programming Cafe (original) (raw)

June 20, 2025

Previously: Fibrations and Cofibrations.

In topology, we say that two shapes are the same if there is a homeomorphism– an invertible continuous map– between them. Continuity means that nothing is broken and nothing is glued together. This is how we can turn a coffe cup into a torus. A homeomorphism, however, won’t let us shrink a torus to a circle. So if we are only interested in how many holes the shapes have, we have to relax our notion of equivalence.

Let’s go back to the definition of homeomorphism. It is defined as a pair of continuous functions between two topological spaces, $f \colon X \to Y$ and $g \colon Y \to X$ , such that their compositions are identity maps, $f \circ g = id_Y$ and $g \circ f = id_X$ , respectively.

If we want to allow for extreme shrinkage, as with a torus shrinking down to a circle, we have to relax the identity conditions.

A homotopy equivalence is a pair of continuous functions, but their compositions don’t have to be equal to identities. It’s enough that they are homotopic to identities. In other words, it’s possible to create an animation that transforms one to another.

Take the example of a line $\mathbb R$ and a point. The point is a trivial topological space where only the whole space (here, the singleton $\{*\}$ ) and the empty set are open. $\mathbb R$ and $\{*\}$ are obviously not homeomorphic, but they don’t have any holes, so they should be homotopy equivalent.

Indeed, let’s construct the equivalence as a pair of constant functions: $f \colon x \mapsto *$ and $g \colon * \mapsto 0$ (the origin of $\mathbb R$ ). Both are continuous: The pre-image $f^{-1} \{ * \}$ is the whole real line, which is open. The pre-image $g^{-1}$ of any open set in $\mathbb R$ is $\{*\}$ , which is also open.

The composition $f \circ g \colon * \mapsto *$ is equal to the identity on $\{*\}$ , so it’s automatically homotopic to it. The interesting part is the composition $g \circ f \colon x \mapsto 0$ , which is emphatically not equal to identity on $\mathbb R$ . We can however construct a homotpy between the identity and it. It’s a function that interpolates between them:

$h \colon \mathbb R \times I \to \mathbb R$

$h (x, 0) = id_{\mathbb R} x = x$

$h(x, 1) = (g \circ f) x = 0$

( $I$ is the unit interval $[0, 1]$ .) Such a function exists:

$h (x, t) = (1 - t) \,x$

When a space is homotopy equivalent to a point, we say that it’s contractible. Thus $\mathbb R$ is contractible. Similarly, n-dimensional spaces, $\mathbb R^n$ , as well as n-dimensional balls are all contractible. A circle, however, is not contractible, because it has a hole.

Another way of looking for holes in spaces is by trying to shrink loops. If there is a hole inside a loop, it cannot be continuously shrunk to a point. Imagine a loop going around a circle. There is no way you can “unwind” it without breaking something. In fact, you can have loops that wind n times around a circle. They can’t be homotopied into each other if their winding numbers are different.

In general, paths in a topological space $X$ , i.e. continuous mappings $\gamma \colon I \to X$ , naturally split into equivalence classes with respect to homotopy. Two paths, $\gamma$ and $\gamma'$ , sharing the same endpoints are in the same class if there is a homotopy between them:

$h \colon I \times I \to X$

$h (0, t) = \gamma t$

$h (1, t) = \gamma' t$

Moreover, two paths can be composed, as long as the endpoint of one coincides with the start point of the other (after performing appropriate reparametrizations).

There is a unit of such composition, a constant path; and every path has its inverse, a path that traces the original but goes in the opposite direction.

It’s easy to see that path composition induces a groupoid structure on the set of equivalence classes of paths. A groupoid is a category in which every morphism (here, path) is invertible. This particular groupoid is called the fundamental groupoid of the topological space $X$ .

If we pick a base point $x_0$ in $X$ , the paths that start and end at this point form closed loops. These loops then form a fundamental group $\pi_1(X, x_0)$ .

Notice that, as long as there is a path $\gamma$ from $x_0$ to $x_1$ , the fundamental groups at both points are isomorphic. This is because every loop $l$ at $x_1$ induces a loop at $x_0$ given by the concatenation $l' = \gamma^{-1} \circ l \circ \gamma$ .

So there is essentially a single fundamental group for the whole space, as long as $X$ is path-connected, i.e., it doesn’t split into multiple disconnected chunks.

Going back to our example of a circle, its fundamental group is the group of integers $\mathbb Z$ with addition. All loops that wind n-times around the circle in one direction correspond to the integer n.

Negative integers correspond to loops winding in the opposite direction. If you follow an n-loop with an m-loop, you get an (m+n)-loop. Zero corresponds to loops that can be shrunk to a point.

To tie these two notions of hole-counting together, it can be shown that two spaces that are homotopy equivalent also have the same fundamental groups. This makes sense, since equivalent spaces have the same holes.

Not only that, they also have the same higher homotopy groups (as long as both are path-connected).

We define the n-th homotopy group by replacing simple loops (which are homotopic to a circle, or a 1-dimensional sphere) with n-dimensional spheres. Attempts at shrinking those may detect higher-dimensional holes.

For instance, imagine an Earth-like ball with it’s inner core scooped out. Any 1-loop inside its bulk can be shrunk to a point (it will glide off the core). But a 2-sphere that envelops the core cannot be shrunk.

In fact such a 2-sphere can be wrapped around the core an arbitrary number of times. In math notation, we say:

$\pi_2 (B^3 \setminus \{ 0 \}) = \mathbb Z$

Corresponding to these higher homotopy groups there is also a higher homotopy groupoid, in which there are invertible paths, surfaces between paths, volumes between surfaces, etc. Taken together, these form an infinity groupoid. It is exactly the inspiration behind the infinity groupoid in homotopy type theory, HoTT.

Spaces that are homotopy equivalent have the same homotopy groups, but the converse is not true. There are spaces that are not homotopy equivalent even though they have the same homotopy groups. This is why a weaker version of equivalence was introduced.

A map between topological spaces is called a weak homotopy equivalence if it induces isomorphisms between all homotopy groups.

There is a subtle difference between strong and weak equivalences. Strong equivalence can be broken by a local anomaly, like in the following example: Take $X = \{0, 1, 2, ... \}$ , the set of natural numbers in which every number is considered an open set. Take $Y = \{ 0 \} \cup \{ 1, \frac{1}{2}, \frac{1}{3}, ... \}$ with the topology inherited from the real line.

The singleton set $\{ 0 \}$ in $Y$ is the obstacle to constructing a homeomorphism or a homotopy equivalence between $X$ and $Y$ . That’s because it is not open (there is no open set in $\mathbb R$ that contains it and nothing else). However, it’s impossible to have a non-trivial loop that would contain it, so $X$ and $Y$ are weakly equivalent.

Grothendieck conjectured that the infinity groupoid captures all information about a topological space up to weak homotopy equivalence.

Weak equivalences, together with fibrations and cofibrations, form the foundation of weak factorization systems and Quillen model categories.

May 30, 2025

Previously: Subfunctor Classifier.

We are used to thinking of a mapping as either being invertible or not. It’s a yes or no question. A mapping between sets is invertible if it’s both injective and surjective. It means that it never merges two elements into one, and it covers the whole target set.

But if you are willing to look closer, the failures of invertibility are a whole fascinating area of research. Things get a lot more interesting if you consider mapping between topological spaces, which have to skillfully navigate around various holes and tears in the target space, and shrink or glue together parts of the source space. I will be glossing over some of the topological considerations, concentrating on the big-picture intuitions (both culinary and cinematographic).

Fibrations

In what follows, we’ll be considering a function $p \colon E \to B$ . We’ll call $p$ a projection from the total set $E$ to the base set $B$ .

Let’s start by considering the first reason for a failure of invertibility: multiple elements being mapped into one. Even though we can’t invert such a mapping, we can use it to fibrate the source $E$ .

To each element $y \in B$ we’ll assign a fiber, the set of elements of $E$ that are mapped to $y$ . By abuse of notation, we call such a fiber $p^{-1} y$ :

$p^{-1} y = \{ x |\; p x = y \}$

For some $y$ ‘s, this set may be empty $\emptyset$ ; for others, it may contain lots elements.

Notice that $p$ is an isomorphism if and only if every fiber is a singleton set. This property gives rise to a very useful definition of equivalence in homotopy type theory, where we ask for every fiber to be contractible.

A set-theoretic union of all fibers reconstructs the total set $E$ .

Things get more interesting when we move to topological spaces and continuous functions. To begin with, we can define a path in $B$ as a continuous mapping from the unit interval $I = [0, 1]$ to $B$ .

$\gamma \colon I \to B$

We can then ask if it’s possible to lift this path to $E$ , that is to construct $\tilde{\gamma} \colon I \to E$ that lies above $\gamma$ . To do this, we pick the starting point $e \in E$ that lies above the starting point of $\gamma$ , that is $p \,e = \gamma \,0$ .

This setup can be summarized in the commuting square:

The lifting $\tilde{\gamma}$ is then the diagonal arrow that makes both triangles commute:

It’s helpful to imagine a path as a trajectory of a point moving through a topological space. The parameter $t \in I$ is then interpreted as time.

In homotopy theory we generalize this idea to the movement, or deformation, of arbitrary shapes, not just points.

If we describe such a shape as the image of some topological space $X$ , its deformation is a mapping $h \colon X \times I \to B$ . Such potentially “fat” paths are called homotopies. In particular, if we replace $X$ by a single point, we recover our “thin” paths.

A homotopy lifting property is expressed as the existence of the diagonal function $\tilde h$ in this commuting square, such that the resulting triangles commute:

In other words, given a homotopy $h$ of the shape $X$ in the base, and an embedding $e$ of this shape in $E$ above $h \, 0$ , we can construct a homotopy in $E$ whose projection is $h$ .

Cofibrations

In category theory, every construction has its dual, which is obtained by reversing all the arrows. In topology, there is an analogous relation called the Eckmann-Hilton duality. Besides reversing the arrows, it also dualizes products to exponentials (using the currying adjunction), and injections to surjections.

When dualizing the homotopy lifting diagram, we replace the trivial injection of $X \times \{0\} \cong X$ into $X \times I$ by a surjection $\varepsilon_0$ of the exponential $X^I$ onto $X$ . Here, $X^I$ is the set of functions $I \to X$ , or the path space of $X$ . The surjection $\varepsilon_0$ maps every path $\gamma \colon I \to X$ to its starting point by evaluating it at zero. (The evaluation map $\varepsilon_t \colon \gamma \mapsto \gamma \,t$ is continuous in the compact-open topology of $X^I$ .)

The homotopy lifting property is thus dualized to the homotopy extension property:

Corresponding to the fibration that deals with the failure of the injectivity of $p$ , the homotopy extension property deals with the failure of the surjectivity of $i$ .

If the mapping $i$ has the extension property for all topological spaces $X$ , it is called a cofibration.

Let’s deconstruct the meaning of the outer commuting square.

The commuting condition for this square means that the starting points of all these paths, the result of applying $\varepsilon_0$ to each of them, coincide with the embedding of $B$ into $X$ given by $e \circ i$ . It’s like extruding a bunch of spaghetti, each strand corresponding to a point in $B$ .

With this setup, we postulate the existence of the diagonal mapping $\tilde h \colon E \to X^I$ that makes the two triangles commute. In other words, $E$ is mapped to a family of paths in $X$ which, when restricted to the image of $B$ coincide with the original mapping $h$ .

The spaghetti extruded through the $E$ shape contain the spaghetti extruded through the $B$ shape.

Another intuitive description of this situation uses the idea of homotopy as animation. The strands of spaghetti become trajectories of particles.

We start by setting up the initial scene. We embed $E$ in the big space $X$ using $e \colon E \to X$ .

We have the embedding $i \colon B \to E$ , which induces the embedding of $B$ into $X$ :

$b = e \circ i$

Then we animate the embedding of $B$ using the homotopy

$h \colon B \times I \to X$

The initial frame of this animation is given by $b$ :

$h(x, 0) = b \, x$

We say that $i$ is a cofibration if every such animation can be extended to the bigger animation:

$\tilde h \colon E \times I \to X$

whose starting frame is given by $e$ :

$\tilde h (x, 0) = e \, x$ .

The commuting condition (for the lower triangle) means that the two animations coincide for all $x \in B$ and $t \in I$ :

$h (x, t) = \tilde h (i \,x, t)$

Just like a fiber is an epitome of non-injectiveness, one can define a cofiber as an epitome of non-surjectiveness. It’s essentially the part of $E$ that is not covered by $i$ .

As a topological space it’s the result of shrinking the image of $B$ inside $E$ to a point (the resulting topology is called the quotient topology).

Notice that, unlike with fibers, there is just one cofiber for a given cofibration (up to homotopy equivalence).

Lifting Property

A category theorist looking at the two diagrams that define, respectively, homotopy lifting and extension, will immediately ignore all the superfluous details. She will turn the second diagram upside down and merge it with the first diagram to get:

This is how we read the new diagram: If for any morphisms $f$ and $g$ that make the outer square commute, there exist a diagonal morphism $h$ that makes the two triangles commute, then we say that $i$ has the left lifting property with respect to $p$ . Equivalently, $p$ has the right lifting property with respect to $i$ .

Or we can say that the two morphisms are orthogonal to each other.

The motivation for this nomenclature is interesting. In the category of sets, the archetypal non-surjective function is the “absurd” $0 \to 1$ (or, in set notation, $\emptyset \to \{*\}$ ). It turns out that all surjective functions are its right liftings. In other words, all surjections are right-orthogonal to the simplest non-surjective function.

Indeed, the function at the bottom $y \colon 1 \to Y$ picks an element $y \in Y$ . Similarly, the diagonal function picks an element $x \in X$ . The commuting triangle tells us that for every $y$ there exists an $x$ such that $y = p \, x$ .

Similarly, we can show that all injective functions are orthogonal to the archetypal non-injective function $2 \to 1$ (or $\{x_1, x_2\} \to \{*\}$ ).

Indeed, assume that $i$ maps two different elements $a_1, a_2 \in A$ to a single element $b \in B$ . We could then pick $f$ such that $f \, a_1 = x_1$ and $f \, a_2 = x_2$ . The diagonal $h$ can map $b$ to either $x1$ or $x2$ but not both, so we couldn’t make the upper triangle commute.

Incidentally, injections are also _right_-orthogonal to the archetypal non-injection.

Next: (Weak) Homotopy Equivalences.

April 21, 2025

Previously: Subobject Classifier.

In category theory, objects are devoid of internal structure. We’ve seen however that in certain categories we can define relationships between objects that mimic the set-theoretic idea of one set being the subset of another. We do this using the subobject classifier.

We would like to define a subobject classifier in the category of presheaves, so we could easily characterize subfunctors of a given presheaf. This will help us work with sieves, which are subfunctors of the hom-functor $C(-, a)$ ; and coverages, which are special kinds of sieves.

Recall that a presheaf $S$ is a subfunctor of another presheaf $P \colon C^{op} \to Set$ if it satisfies two conditions.

As category theory goes, this is a very low-level definition. We need something more abstract: We need to construct a subobject classifier in the category of presheaves. Recall that a subobject classifier is defined by the following pullback diagram:

This time, however, the objects are presheaves and the arrows are natural transformations.

To begin with we have to define a terminal presheaf, $1 \colon C^{op} \to Set$ that satisfies the condition that, for any presheaf $P$ , there is a unique natural transformation $! \colon P \to 1$ . This will work if every component $!_a \colon P a \to 1 a$ of this natural transformation is unique, which is true if we choose $1 a$ to be the terminal singleton set $\{ * \}$ . Thus the terminal presheaf maps all objects to the terminal set, and all morphisms to the identity on $\{ * \}$ .

Next, let’s instantiate the subobject classifier diagram at a particular object $a$ .

Here, the component $true_a$ picks a special “True” element in the set $\Omega_a$ . If the presheaf $S$ is a subfunctor of $P$ , the set $S a$ is a subset of $P a$ . The function $\chi_a$ must therefore map the whole subset $S a$ to “True”. This is consistent with our definition of the subobject classifier for sets.

The second condition in the definition of a subfunctor is more interesting. It involves the mapping of morphisms.

The restriction condition

We have to consider all morphisms converging on our object of interest $a$ . For instance, lets take $f \colon c \to a$ . The presheaf $P$ lifts it to a function $P f \colon P a \to P c$ . If $S$ is a subfunctor of $P$ , $S f$ is a restriction of $P f$ .

Specifically the restriction condition tells us that, if we pick an element $x \in S a$ , then both $P f$ and $S f$ will map it to the same element of $S c$ . In fact, when defining a subobject, we only care if the embedding of $S c$ in $P c$ is injective (monomorphic). It’s okay if it permutes the elements of $S c$ . So it’s enough that, for all $x \in S a$ , the following condition is satisfied:

$(P f) x \in S c$

Now consider an arbitrary $x \in P a$ (not necessarily an element of $S a$ ). We can gather all arrows $f$ converging on $a$ for which the subset-mapping condition is satisfied:

$(P f) x \in S c$

If $S$ is a subfunctor of $P$ , these arrows form a sieve on $a$ , as any composition $f \circ g$ also satisfies the subset-mapping condition:

Moreover, if $x$ is in fact an element of $S a$ , this sieve is the maximal sieve. A maximal sieve on $a$ is a collection of all arrows converging on $a$ .

We can now define a function $\chi_a$ that assigns to each $x \in P a$ the sieve of arrows that satisfy the subset-mapping condition.

$\chi_a x = \{f \colon c \to a \, | \, (P f) x \in S c\}$

The function $\chi_a$ has the property that, if $x$ is an element of $S a$ , the result is the maximal sieve on $a$ .

It makes sense then to define $\Omega_a$ as a set of sieves on $a$ , and “True” as the maximal sieve on $a$ . (Thus $\Omega_a$ is a set whose elements are sets.)

The mapping $\Omega \colon a \to \Omega_a$ can be made into a presheaf by defining its action on morphisms. The lifting of $f \colon c \to a$ takes a sieve $s_a \in \Omega_a$ to a sieve $s'_{c} \in \Omega c$ , defined as a set of arrows $h \colon c' \to c$ , such that $f \circ h \in s_a$ .

Notice that the resulting sieve $s_c'$ is maximal if and only if $f \in \Omega_a$ . (Hint: If a sieve is maximal, then it contains identity.)

It can be shown that the the functions $\chi_a$ combine to form a natural transformation $\chi \colon P \to \Omega$ .

What remains to be shown is that this $\chi$ is a unique such natural transformation that completes the pullback:

To show that, let’s assume that there is another natural transformation $\theta \colon P \to \Omega$ making this diagram into a pullback. Let’s redraw the subfunctor condition for arrows, replacing $\chi$ with $\theta$ :

Let’s pick an $x \in P a$ and call $y = (P f) x$ . We’ll follow a set of equivalences.

The pullback condition:

tells us that $y \in S c$ is equivalent to $\theta_c y = true_c$ . In other words:

$\theta_c ((P f) x) = true_c$

Using naturality of $\theta$ :

we can rewrite it as:

$(\Omega f) (\theta_a x) = true_c$ .

Both sides of this equation are sieves. By definition, the lifting of $f$ , $\Omega f$ , acting on $\theta_a x$ is a sieve defined by the following set of arrows:

$(\Omega f) (\theta_a x) = \{ h \colon c' \to c \, | \, f \circ h \in \theta_a x \}$

Since $true_c$ is a maximal sieve, it must be that $f \in \theta_a x$ .

We have shown that the condition $(P f) x \in S c$ is equivalent to $f \in \theta_a x$ . But the first condition is exactly the one we used to define $\chi_a x$ . Therefore $\chi$ is the only function that makes the subobject classifier diagram into a pullback.

Subfunctor classifier

The subobject classifier in the category of presheaves is thus a presheaf $\Omega$ that maps objects to sieves, together with the natural transformation $true \colon 1 \to \Omega$ that picks maximal sieves.

Every natural transformation $\chi \colon P \to \Omega$ defines a subfunctor of the presheaf $P$ . The components of this natural transformation serve as characteristic functions for the sets $P a$ . A given element $x$ is in the subset $S a$ iff $\chi_a$ maps it to the maximal sieve on $a$ .

But there’s not one but many different ways of failing the subset test. They are given by non-maximal sieves. We may think of them as satisfying the Anna Karenina principle, “All happy families are alike; each unhappy family is unhappy in its own way.”

Why sieves? Because once an element of a set $P a$ is mapped by $P f$ to an element of a subset $S c$ , it will continue to be mapped into consecutive subsets $S c'$ , etc. The network of “happy” morphisms keeps growing outward. By contrast, the “unhappy” elements of $x \in P a$ have at least one morphism $f \colon c \to a$ , whose lifting maps it outside the subset $S c$ . That’s the morphism that’s absent from the non-maximal sieve $\chi_a$ . Finally, naturality of $\chi$ ensures that subset conditions propagate coherently from object to object.

Next: Fibrations and Cofibrations.

March 20, 2025

Proviously Sieves and Sheaves.

We have seen how topology can be defined by working with sets of continuous functions over coverages. Categorically speaking, a coverage is a special case of a sieve, which is defined as a subfunctor of the hom-functor $C(-, a)$ .

We’d like to characterize the relationship between a functor and its subfunctor by looking at them as objects in the category of presheaves. For that we need to introduce the idea of a subobject.

We’ll start by defining subobjects in the category of sets in a way that avoids talking about elements. Here we have two options.

The first one uses a characteristic function. It’s a predicate that answers the question: Is some element $x$ a member of a given subset or not? Notice that any Boolean-valued function uniquely defines a subset of its domain, so we don’t really need to talk about elements, just a function.

But we still have to define a Boolean set. Let’s call this set $\Omega$ , and designate one of its element as “True.” Selecting “True” can be done by defining a function $true \colon 1 \to \Omega$ , where $1$ is the terminal object (here, a singleton set). In principle we should insist that $\Omega$ contains two elements, “True” and “False,” but that would make it more difficult to generalize.

The second way to define a subset $S \subseteq P$ is to provide an injective function $m \colon S \rightarrowtail P$ that embeds $S$ in $P$ . Injectivity guarantees that no two elements are mapped to the same element. The image of $m$ then defines the subset of $P$ . In a general category, injective functions are replaced by monics (monomorphisms).

Notice that there can be many injections that define the same subset. It’s okay for them to permute the image of $m$ as long as it covers exactly the same subset of $P$ . (These injections form an equivalence class.)

The fact that the two definitions coincide can be summarized by one commuting diagram. In the category of sets, given a characteristic function $\chi$ , the subset $S$ and the monic $m$ are uniquely (up to isomorphism) defined as a pullback of this diagram.

We can now turn the tables and use this diagram to define the object $\Omega$ called the subobject classifier, together with the monic $true \colon 1 \rightarrowtail \Omega$ . We do it by means of a universal construction. We postulate that: For every monic $S \rightarrowtail P$ between two arbitrary objects there exist a unique arrow $\chi \colon P \to \Omega$ such that the above diagram constitutes a pullback.

The question is: What happens to the other elements of $P$ ? They cannot be mapped to $\text{True}$ , so $\Omega$ must contain at least one more element (in case $m$ is not an isomorphism). Can it contain more?

This is where the universal construction comes into play. Any monic $m$ (here, an injective function) must uniquely determine a $\chi$ that completes the pullback. In particular, we can pick $S$ to be a singleton set and $P$ to be a two-element set. We see that if $\Omega$ contained only $\text{True}$ and nothing else, no $\chi$ would complete the pullback. And if $\Omega$ contained more than two elements, there would be not one but at least two such $\chi$ ‘s. So, by the Goldilock principle, $\Omega$ must have exactly two elements.

We’ll see later that this is not necessarily true in a more general category.

Next: Subfunctor Classifier.

March 6, 2025

There are many excellent AI papers and tutorials that explain the attention pattern in Large Language Models. But this essentially simple pattern is often obscured by implementation details and optimizations. In this post I will try to cut to the essentials.

In a nutshell, the attention machinery tries to get at a meaning of a word (more precisely, a token). This should be easy in principle: we could just look it up in the dictionary. For instance, the word “triskaidekaphobia” means “extreme superstition regarding the number thirteen.” Simple enough. But consider the question: What does “it” mean in the sentence “look it up in the dictionary”? You could look up the word “it” in the dictionary, but that wouldn’t help much. More ofthen than not, we guess the meaning of words from their context, sometimes based on a whole conversation.

The attention mechanism is a way to train a language model to be able to derive a meaning of a word from its context.

The first step is the rough encoding of words (tokens) as multi-dimensional vectors. We’re talking about 12,288 dimensions for GPT-3. That’s an enormous semantic space. It means that you can adjust the meaning of a word by nudging it in thousands of different directions by varying amounts (usually 32-bit floating point numbers). This “nudging” is like adding little footnotes to a word, explaining what was its precise meaning in a particular situation.

(Note: In practice, working with such huge vectors would be prohibitively expensive, so they are split between multiple heads of attention and processed in parallel. For instance, GPT-3 has 96 heads of attention, and each head works within a 128-dimensional vector space.)

We are embedding the input word as a 12,288-dimensional vector $\vec E$ , and we are embedding $N$ words of context as 12,288-dimensional vectors, $\vec E_i$ , $i = 1..N$ (for GPT-3, $N=2048$ tokens). Initially, the mapping from words to embeddings is purely random, but as the training progresses, it is gradually refined using backpropagation.

(Note: In most descriptions, you’ll see a whole batch of words begin embedded all at once, and the context often being identical with that same batch–but this is just an optimization.)

The goal is to refine the embedding $\vec E$ by adding to it a small delta $\Delta \vec E$ that is derived from the context $\vec E_i$ .

This is usually described as the vector $\vec E$ querying the context, and the context responding to this query.

First, we apply a gigantic trainable 12,288 by 12,288 matrix $W_Q$ to the embedding $\vec E$ , to get the query vector:

$\vec Q = W_Q \vec E$

You may think of $\vec Q$ as the question: “Who am I with respect to the Universe?” The entries in the matrix $W_Q$ are the weights that are learned by running the usual backpropagation.

We then apply another trainable matrix $W_K$ to every vector $\vec E_i$ of the context to get a batch of key vectors:

$\vec K_i = W_K \vec E_i$

You may think of $\vec K_i$ as a possible response from the $i$ ‘th component of the context to all kinds of questions.

The next step is crucial: We calculate how relevant each element of the context is to our word. In other words, we are focusing our attention on particular elements of the context. We do it by taking $N$ scalar products $\vec K_i \cdot \vec Q$ .

A scalar product can vary widely. A large positive number means that the $i$ ‘th element of the context is highly relevant to the meaning of $\vec E$ . A large negative number, on the other hand, means that it has very little to contribute.

What we really need is to normalize these numbers to a range between zero (not relevant) and one (extremely relevant) and make sure that they all add up to one. This is normally done using the softmax procedure: we first raise $e$ to the power of the given number to make the result non-negative:

$a_i = exp (\vec K_i \cdot \vec Q)$

We then divide it by the total sum, to normalize it:

$A_i = \dfrac {a_i}{ \sum_{i = 1}^N a_i}$

These are our attention weights. (Note: For efficiency, before performing the softmax, we divide all numbers by the square root of the dimension of the vector space $\sqrt{d_k}$ .)

Attention weights tell us how much each element of the context can contribute to the meaning of $\vec E$ , but we still don’t know what it contributes. We figure this out by multiplying each element of the context by yet another trainable matrix, $W_V$ . The result is a batch of value vectors $\vec V_i$ :

$\vec V_i = W_V \vec E_i$

We now accumulate all these contribution, weighing them by their attention weights $A_i$ . We get the adjusted meaning of our original embedding $\vec E$ (this step is called the residual connection):

$\vec E' = \vec E + \sum_{i = 1}^N A_i \vec V_i$

The result $\vec E'$ is infused with the additional information gained from a larger context. It’s closer to the actual meaning. For instance, the word “it” would be nudged towards the noun that it stands for.

And that’s essentially the basic block of the attention system. The rest is just optimization and composition.

One major optimization is gained by processing a whole window of tokens at once (2048 tokens for GPT-3). In particular, in the self-attention pattern we use the same batch of tokens for both the input and the context. In general, though, the context can be distinct from the input. It could, for instance, be a sentence in a different language.

Another optimization is the partitioning of all three vectors, $\vec Q$ , $\vec K_i$ , and $\vec V_i$ between the heads of attention. Each of these heads operates inside a smaller subspace of the vector space. For GPT-3 these are 128-dimensional subspaces. The heads produce smaller-dimensional deltas, $\Delta \vec E$ , which are concatenated into larger vectors, which are then added to the original embeddings through residual connection.

In GPT-3, each multi-headed attention block is followed by a multi-layer perceptron, MLP, and this transformation is repeated 96 times. Most steps in an LLM are just linear algebra; except for softmax and the activation function in the MLP, which are non-linear.

All this work is done just to produce the next word in a sentence.

January 4, 2025

The yearly Advent of Code is always a source of interesting coding challenges. You can often solve them the easy way, or spend days trying to solve them “the right way.” I personally prefer the latter. This year I decided to do some yak shaving with a puzzle that involved looking for patterns in a grid. The pattern was the string XMAS, and it could start at any location and go in any direction whatsoever.

My immediate impulse was to elevate the grid to a comonad. The idea is that a comonad describes a data structure in which every location is a center of some neighborhood, and it lets you apply an algorithm to all neighborhoods in one fell swoop. Common examples of comonads are infinite streams and infinite grids.

Why would anyone use an infinite grid to solve a problem on a finite grid? Imagine you’re walking through a neighborhood. At every step you may hit the boundary of a grid. So a function that retrieves the current state is allowed to fail. You may implement it as returning a Maybe value. So why not pre-fill the infinite grid with Maybe values, padding it with Nothing outside of bounds. This might sound crazy, but in a lazy language it makes perfect sense to trade code for data.

I won’t bore you with the details, they are available at my GitHub repository. Instead, I will discuss a similar program, one that I worked out some time ago, but wasn’t satisfied with the solution: the famous Conway’s Game of Life. This one actually uses an infinite grid, and I did implement it previously using a comonad. But this time I was more ambitious: I wanted to generate this two-dimensional comonad by composing a pair of one-dimensional ones.

The idea is simple. Each row of the grid is an infinite bidirectional stream. Since it has a specific “current position,” we’ll call it a cursor. Such a cursor can be easily made into a comonad. You can extract the current value; and you can duplicate a cursor by creating a cursor of cursors, each shifted by the appropriate offset (increasing in one direction, decreasing in the other).

A two-dimensional grid can then be implemented as a cursor of cursors–the inner one extending horizontally, and the outer one vertically.

It should be a piece of cake to define a comonad instance for it: extract should be a composition of (extract . extract) and duplicate a composition of (duplicate . fmap duplicate), right? It typechecks, so it must be right. But, just in case, like every good Haskell programmer, I decided to check the comonad laws. There are three of them:

extract . duplicate = id fmap extract . duplicate = id duplicate . duplicate = fmap duplicate . duplicate

And they failed! I must have done something illegal, but what?

In cases like this, it’s best to turn to basics–which means category theory. Compared to Haskell, category theory is much less verbose. A comonad is a functor $W$ equipped with two natural transformations:

$\varepsilon \colon W \to \text{Id}$

$\delta \colon W \to W \circ W$

In Haskell, we write the components of these transformations as:

extract :: w a -> a duplicate :: w a -> w (w a)

The comonad laws are illustrated by the following commuting diagrams. Here are the two counit laws:

and one associativity law:

These are the same laws we’ve seen above, but the categorical notation makes them look more symmetric.

So the problem is: Given a comonad $W$ , is the composition $W \circ W$ also a comonad? Can we implement the two natural transformations for it?

$\varepsilon_c \colon W \circ W \to \text{Id}$

$\delta_c \colon W \circ W \to W \circ W \circ W \circ W$

The straightforward implementation would be:

$W \circ W \xrightarrow{\varepsilon \circ W} W \xrightarrow{\varepsilon} \text{Id}$

corresponding to (extract . extract), and:

$W \circ W \xrightarrow{W \circ \delta} W \circ W \circ W \xrightarrow{\delta \circ W \circ W} W \circ W \circ W \circ W$

corresponding to (duplicate . fmap duplicate).

To see why this doesn’t work, let’s ask a more general question: When is a composition of two comonads, say $W_2 \circ W_1$ , again a comonad? We can easily define a counit:

$W_2 \circ W_1 \xrightarrow{\varepsilon_2 \circ W_1} W \xrightarrow{\varepsilon_1} \text{Id}$

The comultiplication, though, is tricky:

$W_2 \circ W_1 \xrightarrow{W_2 \circ \delta_1} W_2 \circ W_1 \circ W_1 \xrightarrow{\delta_2 \circ W} W_2 \circ W_2 \circ W_1 \circ W_1$

Do you see the problem? The result is $W_2^2 \circ W_1^2$ but it should be $(W_2 \circ W_1)^2$ . To make it a comonad, we have to be able to push $W_2$ through $W_1$ in the middle. We need $W_2$ to distribute over $W_1$ through a natural transformation:

$\lambda \colon W_2 \circ W_1 \to W_1 \circ W_2$

But isn’t that only relevant when we compose two different comonads–surely any functor distributes over itself! And there’s the rub: Not every comonad distributes over itself. Because a distributive comonad must preserve the comonad laws. In particular, to restore the the counit law we need this diagram to commute:

and for the comultiplication law, we require:

Even if the two comonad are the same, the counit condition is still non-trivial:

The two whiskerings of $\varepsilon$ are in general not equal. All we can get from the original comonad laws is that they are only equal when applied to the result of comultiplication:

$(\varepsilon \circ W) \cdot \delta = (W \circ \varepsilon) \cdot \delta$ .

Equipped with the distributive mapping $\lambda$ we can complete our definition of comultiplication for a composition of two comonads:

$W_2 \circ W_1 \xrightarrow{W_2 \circ \delta_1} W_2 \circ W_1^2 \xrightarrow{\delta_2 \circ W} W_2^2 \circ W_1^2 \xrightarrow{W_2 \circ \lambda \circ W_1} (W_2 \circ W_1)^2$

Going back to our Haskell code, we need to impose the distributivity condition on our comonad. There is a type class for it defined in Data.Distributive:

class Functor w => Distributive w where distribute :: Functor f => f (w a) -> w (f a)

Thus the general formula for composing two comonads is:

instance (Comonad w2, Comonad w1, Distributive w1) =>
Comonad (Compose w2 w1) where extract = extract . extract . getCompose duplicate = fmap Compose . Compose . fmap distribute . duplicate . fmap duplicate . getCompose

In particular, it works for composing a comonad with itself, as long as the comonad distributes over itself.

Equipped with these new tools, let’s go back to implementing a two-dimensional infinite grid. We start with an infinite stream:

data Stream a = (:>) { headS :: a , tailS :: Stream a} deriving Functor

infixr 5 :>

What does it mean for a stream to be distributive? It means that we can transpose a “matrix” whose rows are streams. The functor f is used to organize these rows. It could, for instance, be a list functor, in which case you’d have a list of (infinite) streams.

[ 1 :> 2 :> 3 .. , 10 :> 20 :> 30 .. , 100 :> 200 :> 300 .. ]

Transposing a list of streams means creating a stream of lists. The first row is a list of heads of all the streams, the second row is a list of second elements of all the streams, and so on.

[1, 10, 100] :> [2, 20, 200] :> [3, 30, 300] :> ..

Because streams are infinite, we end up with an infinite stream of lists. For a general functor, we use a recursive formula:

instance Distributive Stream where distribute :: Functor f => f (Stream a) -> Stream (f a) distribute stms = (headS stms) :> distribute (tailS stms)

(Notice that, if we wanted to transpose a list of lists, this procedure would fail. Interestingly, the list monad is not distributive. We really need either fixed size or infinity in the picture.)

We can build a cursor from two streams, one going backward to infinity, and one going forward to infinity. The head of the forward stream will serve as our “current position.”

data Cursor a = Cur { bwStm :: Stream a , fwStm :: Stream a } deriving Functor

Because streams are distributive, so are cursors. We just flip them about the diagonal:

instance Distributive Cursor where distribute :: Functor f => f (Cursor a) -> Cursor (f a) distribute fCur = Cur (distribute (bwStm fCur)) (distribute (fwStm fCur))

A cursor is also a comonad:

instance Comonad Cursor where extract (Cur _ (a :> _)) = a duplicate bi = Cur (iterateS moveBwd (moveBwd bi)) (iterateS moveFwd bi)

duplicate creates a cursor of cursors that are progressively shifted backward and forward. The forward shift is implemented as:

moveFwd :: Cursor a -> Cursor a moveFwd (Cur bw (a :> as)) = Cur (a :> bw) as

and similarly for the backward shift.

Finally, the grid is defined as a cursor of cursors:

type Grid a = Compose Cursor Cursor a

And because Cursor is a distributive comonad, Grid is automatically a lawful comonad. We can now use the comonadic extend to advance the state of the whole grid:

generations :: Grid Cell -> [Grid Cell] generations = iterate $ extend nextGen

using a local function:

nextGen :: Grid Cell -> Cell nextGen grid | cnt == 3 = Full | cnt == 2 = extract grid | otherwise = Empty where cnt = countNeighbors grid

You can find the full implementation of the Game of Life and the solution of the Advent of Code puzzle, both using comonad composition, on my GitHub.

November 16, 2024

Previously: Covering Sieves.

We’ve seen an intuitive description of presheaves as virtual objects. We can use the same trick to visualize natural transformations.

A natural transformation can be drawn as a virtual arrow $\alpha$ between two virtual objects corresponding to two presheaves $S$ and $P$ . Indeed, for every $s_a \in S a$ , seen as an arrow $a \to S$ , we get an arrow $a \to P$ simply by composition $\alpha \circ s_a$ . Notice that we are thus defining the composition with $\alpha$ , because we are outside of the original category. A component $\alpha_a$ of a natural transformation is a mapping between two arrows.

Untitled Artwork

This composition must be associative and, indeed, associativity is guaranteed by the naturality condition. For any arrow $f \colon a \to b$ , consider a zigzag path from $a$ to $P$ given by $\alpha \circ s_b \circ f$ . The two ways of associating this composition give us $\alpha_a \circ S f = P f \circ \alpha_b$ .

Untitled Artwork

Let’s now recap our previous definitions: A cover of $u$ is a bunch of arrows converging on $u$ satisfying certain conditions. These conditions are defined in terms of a coverage. For every object $u$ we define a whole family of covers, and then combine them into one big collection that we call the coverage.

A sheaf is a presheaf that is compatible with a coverage. It means that for every cover $\{u_i\}$ , if we pick a compatible family of $x_i \in P u_i$ that agrees on all overlaps, then this uniquely determines the element (virtual arrow) $x \in P u$ .

Untitled Artwork

A covering sieve of $u$ is a presheaf that extends a cover $\{u_i\}$ . It assigns a singleton set to each $u_i$ and all its open subsets (that is objects that have arrows pointing to $u_i$ ); and an empty set otherwise. In particular, the sieve includes all the overlaps, like $u_i \cap u_j$ , even if they are not present in the original cover.

The key observation here is that a sieve can serve as a blueprint for, or a skeleton of, a compatible family $\{ x_i \}$ . Indeed, $S_u$ maps all objects either to singletons or to empty sets. In terms of virtual arrows, there is at most one arrow going to $S_u$ from any object. This is why a natural transformation from $S_u$ to any presheaf $P$ produces a family of arrows $x_i \in P u_i$ . It picks a single arrow from each of the hom-sets $u_i \to P$ .

Untitled Artwork

The sieve includes all intersections, and all diagrams involving those intersections necessarily commute. They commute because the category we’re working with is thin, and so is the category extended by adding the virtual object $S_u$ . Thus a family generated by a natural transformation $\alpha \in Nat (S_u, P)$ is automatically a compatible family. Therefore, if $P$ is a sheaf, it determines a unique element $x \in P u$ .

This lets us define a sheaf in terms of sieves, rather than coverages.

A presheaf $P$ is a sheaf if and only if, for every covering sieve $S_u$ of every $u$ , there is a one-to-one correspondence between the set of natural transformations $Nat (S_u, P)$ and the set $P u$ .

In terms of virtual arrows, this means that there is a one-to-one correspondence between arrows $\alpha \colon S_u \to P$ and $x \colon u \to P$ .

Untitled Artwork
Next: Subobject Classifier

October 31, 2024

Previously: Sheaves as Virtual Objects.

In order to define a sheaf, we have to start with coverage. A coverage defines, for every object $u$ , a family of covers that satisfy the sub-coverage conditions. Granted, we can express coverage using objects and arrows, but it would be much nicer if we could use the language of functors and natural transformations.

Let’s start with the idea that, categorically, a cover of $u$ is a bunch of arrows converging on $u$ . Each arrow $p_i \colon u_i \to u$ is a member of the hom-set $\mathcal C (u_i, u)$ . Now consider the fact that $\mathcal C (-, u)$ is a presheaf, $\mathcal C^{op} \to \mathbf{Set}$ , and ask the question: Is a cover a “subfunctor” of $\mathcal C (-, u)$ ?

A subfunctor of a presheaf $P$ is defined as a functor $S$ such that, for each object $v$ , $S v$ is a subset of $P v$ and, for each arrow $f \colon v \to w$ , the function $S f \colon S w \to S v$ is a restriction of $P f$ .

Untitled Artwork

In general, a cover does not correspond to a subfunctor of the hom-functor. Let’s see why, and how we can fix it.

Let’s try to define $S$ , such that $S u_i$ is non-empty for any object $u_i$ that’s in the cover of $u$ , and empty otherwise. As a presheaf, we could represent it as a virtual object with arrows coming from all $\{ u_i \}$ ‘s.

Untitled Artwork

Now consider an object $v$ that is not in the cover, but it has an arrow $f \colon v \to u_k$ connecting it to some element $u_k$ of the cover. Functoriality requires the (virtual) composition $s_k \circ f$ to exist. Untitled Artwork

Thus $v$ must be included in the cover–if we want $S$ to be a functor.

In particular, if we are looking at a category of open sets with inclusions, this condition means that all (open) _sub_-sets of the covering sets must also be included in the cover. Such a “downward closed” family of sets is called a sieve.

Imagine sets in the cover as holes in a sieve. Smaller sets that can “pass through” these holes must also be parts of the sieve.

If you start with a cover, you can always extend it to a covering sieve by adding more arrows. It’s as if you started with a few black holes, and everything that could fall into them, would fall.

We have previously defined sheaves in terms of coverings. In the next installment we’ll see that they can equally well be defined using covering sieves.

Next Sieves and Sheaves.

October 24, 2024

Previously: Coverages and Sites

The definition of a sheaf is rather complex and involves several layers of abstraction. To help us navigate this maze we can use some useful intuitions. One such intuition is to view objects in our category as some kind of sets (in particular, open sets, when we talk about topology), and arrows as set inclusions. An arrow from $v$ to $u$ means that $v$ is a subset of $u$ .

A cover of $u$ is a family of arrows $\{ p_i \colon u_i \to u \}$ . A coverage assigns a collection of covers to every object, satisfying the sub-coverage conditions described in the previous post. A category with coverage is called a site.

The next layer of abstraction deals with presheaves, which are set-valued contravariant functors. Interestingly, there is a way to interpret a presheaf as an extension of the original category. I learned this trick from Paolo Perrone.

We may represent a presheaf $P$ using virtual hom-sets. First we add one virtual object, let’s call it $\bullet$ , to our category. The set $P u$ is then interpreted as the set of arrows from $u$ to $\bullet$ .

Untitled Artwork

Moreover, we can represent the action of $P$ on arrows as simple composition. Take an arrow $f \colon v \to u$ . The presheaf lifts it to a function between sets: $P f \colon P u \to P v$ (contravariance means that the arrow is reversed). For any $h \in P u$ we can define the composition $h \circ f$ to be $(P f) h$ .

Untitled Artwork

Incidentally, if the functor $P$ is representable, it means that we can replace the virtual object $\bullet$ with an actual object in our category.

Notice that, even though the category of open sets with inclusions is a poset (hom-sets are either singletons or empty, and all diagrams automatically commute), the added virtual hom-sets usually contain lots of arrows. In topology these hom-sets are supposed to represent sets of continuous functions over open sets.

We can interpret the virtual object $\bullet$ as representing an imaginary open set that “includes” all the objects $u$ for which $P u$ is non-empty, but we have to imagine that it’s possible to include an object in more than one way, to account for multiple arrows. In fact, in what follows we won’t be assuming that the underlying category is a poset, so virtual hom-sets are nothing special.

To express the idea of intersections of open sets, we use commuting diagrams. For every pair of objects $u_i$ and $u_j$ that are in the cover of $u$ , an object $v$ is in their intersection if the following diagram commutes:

Untitled Artwork

Note that in a poset all diagrams commute, but here we’re generalizing this condition to an arbitrary category. We could say that $v$ is in the intersection of $u_i$ and $u_j$ seen as covers of $u$ .

Equipped with this new insight, we can now express the sheaf condition. We assume that there is a coverage defined in our category. We are adding one more virtual object $\bullet$ for the presheaf $P$ , with bunches of virtual arrows pointing to it.

For every cover $\{ p_i \colon u_i \to u \}$ we try to select a family of virtual arrows, $s_i \colon u_i \to \bullet$ . It’s as if the objects $u_i$ , besides covering $u$ , also covered the virtual object $\bullet$ .

We call the family $\{s_i \}$ a matching family, if this new covering respects the existing intersections. If $v$ is in the intersection of $u_i$ and $u_j$ (as covers of $u$ , see the previous diagram), then we want the following diagram to also commute:
Untitled Artwork
In other words, the $\{u_i\}$ ‘s intersect as covers of $\bullet$ .

A presheaf $P$ is a sheaf if, for every covering family $p_i$ and every matching family $s_i$ there exists a unique $s \colon u \to \bullet$ that factorizes those $s_i$ ‘s:
Untitled Artwork
Translating it back to the language of topology: There is a unique global function $s$ defined over $u$ whose restrictions are $s_i$ ‘s.

The advantage of this approach is that it’s easy to imagine the sheafification of an arbitrary presheaf by freely adding virtual arrows (the $s$ ‘s and their compositions with $p_i$ ‘s in the above diagram) to all intersection diagrams.

Next: Covering Sieves

October 7, 2024

Coverages and Sites

Posted by Bartosz Milewski under Topology | Tags: math |
Leave a Comment

Previously: Sheaves and Topology.

In our quest to rewrite topology using the language of category theory we introduced the category of open sets with set inclusions as morphisms. But when we needed to describe open covers, we sort of cheated: we chose to talk about set unions. Granted, set unions can be defined as coproducts in this category (not to be confused with coproducts in the category of sets and functions, where they correspond to disjoint unions). This poset of open sets with finite products and infinite coproducts is called a frame. There is however a more general definition of coverage that is applicable to categories that are not necessarily posets.

A cover of an open set $u$ is a family of open sets $u_i$ that doesn’t leave any part of $u$ uncovered. In terms of inclusions, we can rephrase this as having a family of of morphisms $p_i \colon u_i \to u$ . But then how do we ensure that these sets indeed cover the whole of $u$ ?
Cover
The familiar trick of category theory is to look at the totality of all covers. Suppose that for every open set $u$ we have not one, but a whole collection of covers: that is a collection of families $\{u_i\}$ , each indexed by a potentially different set $I$ .

Now consider an open subset $v \subseteq u$ . If the family $u_i$ is a cover of $u$ than the family of intersections of $v_i = v \cup u_i$ should form a cover of $v$ .

Indeed, an intersection of two open sets is again an open set, so all $v_i$ ‘s are automatically open. And if no part of $u$ was left uncovered by $u_i$ ‘s, then no part of $v$ is left uncovered by $v_i$ ‘s.

Conversely, imagine that we had an incomplete cover, with a hole large enough to stick an open set $v$ in it. The empty intersection of that “cover” with $v$ would then produce no cover of $v$ . So the requirement that each cover produces a smaller sub-cover eliminates the possibility of substantial holes in coverage. This is good enough for our generalization.

We are now ready to define coverage using categorical language. We just have to make sure that this definition reproduces the set-theoretic picture when we replace objects with open sets, and arrows with inclusions.

A coverage $J$ on a category $\mathcal C$ assigns to each object $u$ a collection of families of arrows $p_i \colon u_i \to u$ .

CatCover3

For every such family $\{ p_i\}$ , and every object $v$ equipped with an arrow $g \colon v \to u$ , there exist a covering family $q_j \colon v_j \to v$ that is a sub-family of $u_i$ .

CatCover

This means that for every $v_j$ we can find its “parent” $u_i$ , i.e., every inclusion $g \circ q_j \colon v_j \to u$ can be factored through some $p_i$ :

$g \circ q_j = p_i \circ k_{i j}$

CatCover

A category with a coverage is called a site. As we’ve seen before, the definition of a sheaf uses a coverage, so a site provides the minimum of structure for a sheaf.

For completeness, here’s the definition of a sheaf on a site $(\mathcal C , J)$ as explained in detail in the previous post:

Our starting point is a presheaf $P \colon \mathcal C^{op} \to Set$ , abstracting the idea of assigning a set of functions to every open set (an object of $\mathcal C$ ). $P$ maps arrows in $\mathcal C$ (inclusions of open sets) to restrictions of said functions. This presheaf is a sheaf if:

Untitled Artwork

As you’d expect in topology, this definition doesn’t mention sizes or distances. More interestingly, we don’t talk about points. Normally a topological space is defined as a set of points, and so are open sets. The categorical language lets us talk about point-free topologies.

There is a further generalization of sheaves, where the target of the functor $P$ is not $\mathbf{Set}$ but a category with some additional structure. This makes sense, because the set of functions defined over an open set has usually more structure. It’s the structure induced by the target of these functions. For instance real- or complex-valued functions can be added, subtracted, and multiplied–point-wise. Division of functions is not well defined because of zeros, so they only form a ring.

There is an intriguing possibility that the definition of a coverage could be used to generalize convolutional neural networks (CNN). For instance, voice or image recognition involves applying a sliding window to the input. This is usually done using fixed-sized (square) window, but what really matters is that the input is covered by overlapping continuous areas that capture the 2-dimensional (topological) nature of the input. The same idea can be applied to higher dimensional data. In particular, we can treat time as an additional dimension, capturing the idea of continuous motion.

Next Sheaves as Virtual Objects.

Subscribe Subscribed
- Bartosz Milewski's Programming Cafe
- Already have a WordPress.com account? Log in now.
- Bartosz Milewski's Programming Cafe
- Subscribe Subscribed
- Sign up
- Log in
- Report this content
- View site in Reader
- Manage subscriptions
- Collapse this bar