Skip to the Main Content

Note:These pages make extensive use of the latest XHTML and CSS Standards. They ought to look great in any standards-compliant modern browser. Unfortunately, they will probably look horrible in older browsers, like Netscape 4.x and IE 4.x. Moreover, many posts use MathML, which is, currently only supported in Mozilla. My best suggestion (and you will thank me when surfing an ever-increasing number of sites on the web which have been crafted to use the new standards) is to upgrade to the latest version of your browser. If that's not possible, consider moving to the Standards-compliant and open-source Mozilla browser.

April 10, 2024

Machine Learning Jobs for Category Theorists

Posted by John Baez

Former Tesla engineer George Morgan has started a company called Symbolica to improve machine learning using category theory.

When Musk and his AI head Andrej Karpathy didn’t listen to Morgan’s worry that current techniques in deep learning couldn’t “scale to infinity and solve all problems,” Morgan left Tesla and started Symbolica. The billionaire Vinod Khosla gave him $2 million to prove that ideas from category theory could help.

Khosla later said “He delivered that, very credibly. So we said, ‘Go hire the best people in this field of category theory.’ ” He says that while he still believes in OpenAI’s continued success building large language models, he is “relatively bullish” on Morgan’s idea and that it will be a “significant contribution” to AI if it works as expected. So he’s invested $30 million more.

Needless to say, this raises lots of issues. Some category theorists are worried about hype. I’m more worried about what happens if this technology actually works!

But perhaps the most immediate issue is that Symbolica is hiring category theorists. It’s already hired some, and here are job ads for 6 more. They are hiring in the UK and Australia.

If category theorists are getting jobs in this field, they might as well be readers of the n-Category Café. Here are those 6 positions — you can see more detailed descriptions at the links:

For more on the math, check out these papers.

Posted at April 10, 2024 4:29 PM UTC

TrackBack URL for this Entry:   https://golem.ph.utexas.edu/cgi-bin/MT-3.0/dxy-tb.fcgi/3524

7 Comments & 0 Trackbacks

Re: Machine Learning Jobs for Category Theorists

do they realize how stupid this sounds ?

“that current techniques in deep learning couldn’t “scale to infinity and solve all problems,”

Khosla later said “He delivered that, very credibly. So we said, ‘Go hire the best people in this field of category theory.’ ”

Posted by: zzz on April 10, 2024 9:19 PM | Permalink | Reply to this

Re: Machine Learning Jobs for Category Theorists

Just in case it was unclear, “He delivered that, very credibly” refers not to how Morgan said that “current techniques in deep learning couldn’t scale to infinity and solve all problems”, but how Morgan and his team of researchers wrote up a plan to use category theory in AI.

Posted by: John Baez on April 10, 2024 11:35 PM | Permalink | Reply to this

Re: Machine Learning Jobs for Category Theorists

Maybe you need to explain what the stupid part is

Posted by: Mitchell Porter on April 11, 2024 12:00 AM | Permalink | Reply to this

Re: Machine Learning Jobs for Category Theorists

anything real world does not scale to infinity.

‘hire the best people in this field of…’ is MBA speak for i dont know what the project really is.

Posted by: zzz on April 11, 2024 1:05 AM | Permalink | Reply to this

Re: Machine Learning Jobs for Category Theorists

A lot of nuance does not come across in the quote, but sample complexity of neural network architectures is a well-studied problem which effectively answers: how many datapoints are needed to effectively learn a task.

When it comes to some kinds of tasks (i.e. group actions), architectures like Transformers manage to learn the appropriate equivariance (see https://arxiv.org/abs/2210.02984). For others (i.e. initial/final coalgebras of endofunctors which are categorical semantics of structured (co)recursion), they fail (see https://arxiv.org/abs/2305.14699 or https://arxiv.org/abs/2402.05785).

So really, scaling neural networks is a big issue, and a lot of funding/resources is being used trying to do that well, as evident by the number of LLMs today. Almost without exception, algorithmic/structured learning tasks are attempted without rigorous definitions of these algorithms or structures in the first place.

Posted by: Bruno Gavranovic on April 11, 2024 6:35 PM | Permalink | Reply to this

Re: Machine Learning Jobs for Category Theorists

Hi John,

thank you for sharing the job postings. We’re starting something really exciting, and as research leads on the team, we - Paul Lessard and Bruno Gavranović - thought we’d provide clarifications.

Symbolica was not started to improve ML using category theory. Instead, Symbolica was founded ~2 years ago, with its 2M seed funding round aimed at tackling the problem of symbolic reasoning, but at the time, its path to getting there wasn’t via categorical deep learning (CDL). The original plan was to use hypergraph rewriting as means of doing learning more efficiently. That approach however was eventually shown unviable.

Symbolica’s pivot to CDL started about five months ago. Bruno had just finished his Ph.D. thesis laying the foundations for the topic and we reoriented much of the organization towards this research direction. In particular, we began: a) refining a roadmap to develop and apply CDL, and b) writing a position paper, in collaboration with with researchers at Google DeepMind which you’ve cited below.

Over these last few months, it has become clear that our hunches about applicability are actually exciting and viable research directions. We’ve made fantastic progress, even doing some of the research we planned to advocate for in the aforementioned position paper. Really, we discovered just how much Taking Categories Seriously gives you in the field of Deep Learning.

Many advances in DL are about creating models which identify robust and general patterns in data (see the Transformers/Attention mechanism, for instance). In many ways this is exactly what CT is about: it is an indispensable tool for many scientists, including ourselves, to understand the world around us: to find robust patterns in data, but also to communicate, verify, and explain our reasoning.

At the same time, the research engineering team of Symbolica has made significant, independent, and concrete progress implementing a particular deep learning model that operates on text data, but not in an autoregressive manner as most GPT-style models do.

These developments were key signals to Vinod and other investors, leading to the closing of the 31M funding round.

We are now developing a research programme merging the two, leveraging insights from theories of structure, e.g. categorical algebra, as means of formalising the process by which we find structure in data. This has twofold consequence: pushing models to identify more robust patterns in data, but also interpretable and verifiable ones.

In summary:

a) The push to apply category theory was not based on a singular whim, as the the post might suggest,

but that instead

b) Symbolica is developing a serious research programme devoted to applying category theory to deep learning, not merely hiring category theorists

All of this is to add extra context for evaluating the company, its team, and our direction, which does not come across in the recently published tech articles.

We strongly encourage interested parties to look at all of the job ads, which we’ve tailored to particular roles. Roughly, in the CDL team, we’re looking for either

1) expertise in category theory, and a strong interest in deep learning, or

2) expertise in deep learning, and a strong interest in category theory.

at all levels of seniority.

Happy to answer any other questions/thoughts.

Bruno Gavranović,

Paul Lessard

Posted by: Bruno Gavranovic on April 11, 2024 4:57 PM | Permalink | Reply to this

Re: Machine Learning Jobs for Category Theorists

Excellent - if someone’s willing to pay me to do category theory, I’m all in!

Posted by: Christopher Townsend on May 19, 2024 10:40 AM | Permalink | Reply to this

Post a New Comment