Preliminary Objectives and Ideas

Let me put it clearly: although I am knowledgeable in deep learning and algorithm theory, I don’t know much about MANNs yet (by the way I’m not a big fan of acronyms, but memory-augmented neural networks is very long to write, so MANN it will be). In fact this blog will accompany me along my journey of discovering MANNs, as it will be the main subject of my PhD research.

I am one year into my PhD; in the first twelve months I worked mostly on classical deep learning, focusing on binary activated neural networks and PAC-Bayes statistical guarantees, mostly because of my advisor’s research interests. It led to a paper which hopefully will be accepted at AIStats. But I insisted on working on my own stuff after the submission of the paper, because I don’t really believe PAC-Bayes will revolutionize the world, nor am I interested in spending my life doing statistics stuff, as I am a computer scientist, not an actuary.

On the other hand, my master’s degree was about algorithmic complexity issues in post-quantum cryptography. I enjoyed the complexity part a lot, but did not feel at my place in the cryptography world, because I don’t like the rest of computer security and do not care too much about online privacy in general. Except my bank account. Don’t go there.

I have about three years to go, maybe a little more, to complete my PhD, and I wanted to do my thing. I like algorithms, I like complexity, I like artificial intelligence, I like neural networks… is there something for me out there?

Well, actually if there were already something, I’d be sad. Remember, I want to do my own thing. While MANNs are indeed at the crossroads of artificial intelligence, neural networks and algorithms, it does not take complexity into account. I did come across one paper claiming they could learn to sort with operations, which is great as sorting provably cannot get any better, but this is still very far from the complexity landscape we have with classical algorithms.

Me? I want to define the landscape of the complexity of algorithms learned with MANNs. I’ll give more details on MANNs in a subsequent post, but for now I can quickly state that MANNs are neural networks that are optimized not just to classifier, but to become whole algorithms, with loops, memory accesses, etc. The obtained algorithms should have a complexity, just as classical ones do. Can we understand what complexity is possible to achieve for a given problem? Can we force the optimization process to find efficient solutions, by trying to force it to find a solution within a given complexity class?

Why did I choose to write a blog? Firstly, my wife suggested I do so, and I do everything she says because I love her a lot. But mostly, after writing papers and grant submissions in the cold, length-limited, so-called scientific style, I want to feel the freedom of writing however I feel like. I want to be able to write standard deviation instead of std because I have the space to do so. I want to be able to write a paragraph-long analogy to make my readers really understand what I am talking about. I want to be able to repeat « I want to be able » at every sentence, not combine it into one long, painful, sentence.

What I have read so far in the MANN community was disastrously unclear, even for blog posts, an exception being the foundation paper on the subject, Neural Turing Machines, and maybe this blog post which seems promisingly clear. Furthermore, it seems most papers in the field of MANNs have only been published on Arxiv, not in any peer-reviewed conference. This makes me think the field really lacks theoretical foundations and I’d be pleased to contribute to that.