EPFL AppliedMLdays summary

Posted on by Standard Reply

True Artificial Intelligence will Change Everything

Jürgen Schmidhuber, explained why: “True Artificial Intelligence will Change Everything” and why this technology is inevitable.

The current crop of state-of-the-art products (eg. google voice, google translate, Alexa’s voice) are built on a technology called Long Short Term Memory (LSTM). The clever thing about this technology is that you feed it very raw training data (eg. for voice you don’t have to sync the sound envelope with its phrase). and it can learn from this.

The LSTM software algorithms for this have been there for many (~10) years – we’ve just been waiting for today’s faster hardware.

Scientists think that the brain computes at 10^20 operations per second. Computers are doing 10^15 operations per second today. 1kg matter can in theory do 10 ^51. We will be able to build real artificial brains. What will come soon is small animal brain-like AI. Nature tool a long time to get there and very little more time percentage wise to get to human intelligence – it will probably be similar for machines.

The biggest LSTM network (google translate) is 1 billion nodes. There are 100,000 billion neurons in the human brain cortex – will have this in a machine in 25 years.

Google was a hack

Emmanuel Mogenet head of Google Research Europe

First 10 years of Google was a “hack”: information retrieval without understanding the meaning of the question or the answer.Then in 2010 came the knowledge graph (factual information about the world) with natural language querying – essentially recognising patterns in questions & converting them to database queries.

Natural language so hard because it has constant implicit references to the world we live in. Humans use it to have efficient human to human communication – eg: “will it be dark by the time I get home?” contains no entities: to answer it you need to understand the question which has obvious things to us but computers are completely blind to.

However learning the world by rote doesn’t scale. AI needs to solve “common sense”. If children can learn about the world why can’t computers? This probably implies building/using robots.

A human life is 10 billion pictures (10 frames a second over a life). We already have 100 billion images on the internet today so this should be enough to train “common sense”.

The Idea would be to learn a “world model” by counting occurrences. Eg. AI has to learn that cows and fields go together by counting occurrences of them together images.

The current state of machine/deep learning is that it needs labelled datasets. We don’t have these labels on the Internet images so researchers are trying to build unsupervised machine learning which avoids the need for labels. A major help is that data in the human world is hierarchical (overlapping hierarchies). They can start with high level concepts from human knowledge. Researchers are making lots of progress and think they should be able to solve this soon.

It involves a combination of Computer vision + Unsupervised Hierarchical ML + Common sense DB (logical scaffolding) + Natural Language understanding. Its a loop where the learning is fed back into the computer vision: Natural Language knowledge helps Image recognition.

He thinks we are 10 to 15 years away from all this.

Amazon Web Services
Amazon has an easy to deploy machine learning framework for AWS called MXnet https://aws.amazon.com/mxnet/

Sensors used to be the bottleneck.
Key tool: compression

Panel on ML and society
Ed Bugnon: Todays AI can do anything that a human can do in 1/10th of a second.
Eg. react to a situation driving a car, analyse a radiology image to recognise a broken bone. Data science requires data. Data is concentrated.

Emmanuel Mogenet (Google): There’s a lot of public data – today it’s more a question of infrastructure. Google plans to make it’s datasets available if you use our cloud.

Nuria Oliver, Vodafone: The open movement is that it’s better to bring the algorithms to the data.

AI replacing jobs?

Ed: Generation transition problem – Just like with the fall of the Iron curtain older people found it much harder to transform than the younger generation.

Em: ML will be an exoskeleton for the brain. Will it empower people rather than rend obsolete?

The gap between ML experts and general population is becoming big. Very difficult for society (government) to make decisions. Education is important. We need more informed conversation not sensational (apocalyptical) articles.

Swiss gov. recognises there’s massive change coming. Eg. Digital Switzerland initiative. EPFL has computation thinking at the core of it’s curriculum.
Need to educate people to understand what’s possible – not how to do things.

A quick show of hands from the audience showed ¾ positive on AI & society.

Algorithms and bias.
Nu: it’s a real problem – with complex data it’s very difficult to understand the bias in it. Corollary: humans are full of bias, selfish and make biased decisions.
Transparency of algorithms is a problem, accountability.

Ed: Computers analyse us as unique not as equal. Society isn’t ready to deal with this.
Em: Not worried about explanability – researchers will figure this out. Issue will be the legal framework (eg. Contesting an AI based decision). Humanities fields need to understand AI tech – Nu: Homo Deus is an example.

Em: change isn’t worrying – it’s the rate of change.
Machines don’t have creativity, intent and purpose.
Ed: learning how to interact with other human beings takes a lifetime.
The jobs we have today are an artefact of the limitations of our machines.
Nu: what does it mean to be human? – this will change over time.

Em: in 5 years – want to allow anyone with a spreadsheet press a button and get a predictive model

Generic AI
There were many presentations about implementing & hacking what I’d call “current generic AI software running on generic GPU based hardware”

The recipe is
1) get a dataset (eg. A set of categorised images or texts) to train the AI
2) try training AI frameworks with this dataset
3) use the resulting AI to recognise patterns etc.

The training step involves a lot of fiddling with parameters etc. BUT the next generation of AI should be able to train itself – I’m assuming this means that it will be able to figure out which are the best algorithms and parameters, so it will replace all this fiddling. Amazon is some way towards making this easier.

What’s missing is a real discussion about data in the real world. Eg. what’s needed take research and to build real-world products.

Thanks to Marcel Salathe and team for organising a great 2 day conference.

Update: videos and slides of most presentations of the Applied Machine Learning Days (AMLD) are now online

The Universal Basic Income debate

Posted on by Standard Reply

In Switzerland we are going to vote on a Universal Basic Income later this year.
It’s a polarising debate and from what I’ve observed people fall into 3 camps:

  1. For: those who think its a fair and simpler system
  2. For: those who think the world is changing fast and that new technology, robots etc. will make full-time employment for all an obsolete idea
  3. Against: those who think it’s a distraction and we need to make our current system work.

The against camp is also skeptical of some of the vocal support for UBI coming from silicon valley and the world economic forum – as these are groups which support massive wealth creation benefiting a small elite.

The for camp is citing some similar arguments to Silicon Valley and is imagining a campaign by robots concerned for humanity:

Robots demand for universal basic income as a humanistic response to technological progress
We – the robots – call for an universal basic income for humans. We want to work for the humans to relieve them from the struggle for income. We are really good in working. But we do not want to take away people’s jobs and thereby bring them into existential difficulties.

Robot beings – we could have them now

Posted on by Standard Reply

Autonomous cars are coming soon(ish) probably along with delivery drones and many others. Nobody can guess how humans and robots will co-exist in the future.

Today pretty much the only autonomous robots are lawn-mowing robots & vacuum cleaning robots (& weeding robots coming soon). Humans buy them and then own them (often giving them a like like a pet). But here’s another scenario: Instead someone plays robot-creator-god and orders lawn-mowing robot and connects it up to the Internet and gives it some money to start out. ie. The robot is not treated as a slave but bought it’s freedom, so it can become economically active and in return over time it pays back it’s benefactor.

The robot needs an email address, a bitcoin wallet (no bank would give it a bank account), and access to freelance job sites (which will pay for it’s services in bitcoins) and bids on jobs. It gets there and back (to where the job is) by hailing a ride service which accepts payment in bitcoins. It pays rent for somewhere to live (where it can plug-in and recharge it’s batteries), and if it has any error messages it calls it’s manufacturer’s service centre.

It needs these support services to exist, but then so do we humans to live in the modern world (we need banks, doctors, supermarkets etc.).

If it earns enough cash it could even order another robot to be manufactured, eventually building a robot family.

There’s plenty of ways it can fail to survive:
– an accident or breakdown where it can’t notify the service centre
– it can’t adapt to a change on one of it’s Internet services (they shutdown or change the API).

.. but this scenario demonstrates that from an economic and work point of view we may have to view robots as more equal workers than we think today.

Update: the latest humanoid robot from Google’s Boston Dynamics

Most new markets have matured – growing startups has become a serious business

Posted on by Standard Reply

Most new markets are ten years old, if not older. Ecommerce, Online apps, disintermediation platforms, social, mobile, games, enterprise SaaS, etc. are all mature or maturing markets. What this means to new startups is when they try to scale user acquisition they are up against serious competition:

  • Mature startups with a full product suite and a sales & marketing machine staffed with teams with a track record of growing, maintaining and monetising user bases. They are either already profitable, or have raised tens of millions of growth capital.
  • Mature global companies who, when it’s clear there’s a sizeable market opportunity, they move into this market.

Gone is the day where customers will mix together products from dozens of new startups. Apart from a few early adopters who will buy the best tool for each task, most see this as too complicated. Instead it’s easier, less risky, to by a product suite, one tool which does everything OK, works together and has enough customer references/online reviews to say it works.

These mature startups and global companies have mature sales and marketing operations – to them it doesn’t mater if it takes a couple of years before a new customer is generating revenue for them. Their products are bigger than the “minimum viable product” from a young startup – they can sell it for more so they can outbid the young startups in all user acquisition channels. This dynamic has always existed – it’s about scale. However in the past there’s been several paradigm shifts which have shifted the power from mature products towards startups:

  • Exponential growth of internet users starting in the 90s – a rising tide raises all boats meant success for many who were one of the first to see an opportunity
  • Disintermediation (many players in a value-chain were no-longer adding value in an online world and they could be easily disrupted by startups starting off small – eg. Ebay)
  • The first wave of agile product development (in 2000 mature products were built on Oracle & coded in C or Java, and along came Linux, Apache, PHP and Mysql which was free and easy to update your product continuously)
  • The techcrunch effect circa 2006 was strong enough to launch new products
  • The rise of social
  • The second wave of agile product development – the move to the cloud, SaaS, cheap and scalable
  • Exponetial growth of smart phones (Mature products were slow to adapt to building mobile versions)

Maybe there will be new paradigm shifts which will create opportunities again. However the main shift I see is working against startups. It’s the convergence of hardware, software and services with the internet of things, sensors, wearables, increasingly complex smart phones. It’s difficult to see what a “minimum viable product” is in a world where everything is converging and the tech giants (Google, Apple, etc.) are sucking in more and more data.

As a tech seed investor, it’s become increasingly difficult to judge what’s going to succeed. For a startup to see off the mature competitors it will need a VC to back it all the way and a team which is motivated to build out the product until it and the team have matured – its a long and uncertain journey. In this interconnected world, the halfway solution (that we’ve used in Europe where startups raise a tenth of what they raise in the US and find a niche) is going to be increasingly difficult to make work. Even if the IP (intellectual property) is compelling, it’s risky to try and build an industry from scratch unless you have access to the full US scale VC capital (tens of millions) and a team with a track record to scale startups big. The default European option will continue to be to sell the startup early to avoid the risks of going it alone.

My guess is somehow this maturing will play into the hands of the multinationals. They have the scale, and somehow the combination of them, their corporate VC teams, kickstarter/indigogo (as a means of testing product/market interest), accelerators/incubators (to process ideas in large batches), VCs and angels will have work together to get new products to scale.

But (except for the rare exceptions) it’s not going to be about startups going it alone. As markets mature, its brutal out there for startups.

P.S. Here’s an example of Apple’s software maturing and competing with many startups https://www.cbinsights.com/blog/apples-wwdc-startup-losers/

The global hardware startup ecosystem in depth

Posted on by Standard 1

The Silicon Valley/Shenzen based hardware accelerator haxlr8r has just published a 200 slide presentation “hardware trends 2015” on slideshare.

Masses of info showing trends, successes and failures – every slide has some key information that anyone involved in hardware startups needs to know. For example here’s an extensive list of traps to avoid:

hardware risks

Hardware innovation is happening very fast. You can see that there’s lots of elements driving this. Behind all this is also the commoditization of hardware components such as sensors, platforms such as Arduino, and a lot of software components (frameworks and algorithms) which are mature enough and modular enough to use as building blocks.

In Switzerland we’re doing well in intellectual property for components, but we are struggling to capture more of the value with complete products or even just subsystems. The only example of a Swiss product I can find in the presentation is Gimball (due later this year) – I think the other Swiss drones are still research projects.

The Silicon Valley seed-investing bubble deflates

Posted on by Standard 2

Mattermark has just published a report showing that Amid Pre-IPO Mega Deals, Overall Q4 2014 Startup Deal Volume Returns to Late 2011 Levels with Seed Rounds Slowing Dramatically – here’s a couple of graphs from that report (linked from their site) which show it clearly.

I think this lower volume makes sense when you consider the following:

For low-funded startups its difficult impossible to retain rockstar engineers in Silicon Valley – they all want to work at somewhere like Uber where they think that their stock options are more valuable. Note: this is a silicon Valley specific problem – elsewhere it’s not the case.

As sectors mature, it’s difficult impossible to launch a startup with a tiny team of 2 or 3 founders – a hacker (programmer), a hustler (deal maker, biz dev) and a hipster (designer). Eg. in order to get visibility on the app store you also need a marketing budget of $50k/month to build traction and in a lot of sectors it’s now extremely difficult to launch an app which is really just a feature solving one pain-point – users are expecting something more fully-featured from day one. For many sectors to build,test and support a minimum viable product and to create enough mindshare amongst users to get growth needs a team of 10 or more.

There’s serious momentum investing (I’ve written about this before) where one startup in each sector gets $ tens of millions and, unless they screw up big time, this cash gives them enough power to crush all competition and dominate their market. Seed investing in this environment is high risk unless you’re sure you’re backing the one which will attract the momentum investors.

Why raising money for Hardware startups is hard

Posted on by Standard Reply

Here’s my attempt to summarising the investment ecosystem for hardware startups (with my European bias, but it does seem to be a similar position in the US). It’s based on what I saw and heard at the Invest in Photonics (photonics is lasers, displays, LEDs, etc.) conference recently in France, plus my own experience as an angel investor in several Swiss hardware startups.

On the face of it, it’s all very contradictory. For example, in the same presentation at Invest in Photonics a US VC said:

  • Optics + IT will save more lives than doctors.
  • VCs won’t invest in the hard part (the optics).

The discussions, presentations & roundtables at the conference did provide some logic to this. VCs get a better return on investing money in software, so they focus on that. Hard stuff is just too hard for VCs to finance and to get a return comparable with software. If they do invest in hardware then it’s startups who are mostly just packaging up components and building a brand – eg. Occulus Rift, Nest, GoPro, etc. or, occasionally materials.

In addition to the lack of proven returns is another less-obvious issue: traditional VCs don’t have the network to do proper due diligence on most hardware startups. Doing DD involves getting actionable data on the market, how it is going to evolve, what the completion will look like and which tech is going to win out. Only industry insiders have this. A VC can’t act on the limited data that he can get hold of.

Even European success stories like Novaled’s recent exit (to Samsung) for 260M EUR hasn’t change the VCs’ lack of interest. The general consensus from speakers & panels (including Novaled’s outgoing CEO) is that it’s got even more difficult to finance hardware startups the last few years.

Most of the advice was pointing to this strategy for startups:

  1. raise initial funding from a combination of government grants & angels.
  2. be very capital efficient
  3. raise follow on from Corporate VCs and/or family offices.

Wellington Partners said they see all photonics fundraising post-series-A coming from family offices (what they call “unconventional sources”). But it’s not that simple – matching a family office to a deal can’t be done systematically – it has to be opportunistic networking on a case by case basis.

Corporate VCs also seem to be getting involved at a late stage and filing some of the void of traditional VCs (my guess is at this later stage they can do the due diligence that traditional VCs can’t). The general feeling is that getting a corporate VC onboard works well unless there’s a conflict over strategy between the startup and the corporate VC, therefore a corporate VC who operates independently of the mother ship should be best.

All this uncertainty in the investment ecosystem means that the initial investors have to expect that there is a high probability that they will have to lead the follow-on investing all the way until the startup reaches break even. And the new normal is the hardware CEO is always in full-time fundraising mode. Although for really exceptional startups it can be a lot easier than all this.

And finally: although financing hardware innovation is a bit pessimistic, the technology continues to advance. In the near future Airbus thinks we’ll see drones powered from laser beams on the ground and other fun stuff!