Facebook finishes its move to neural machine translation

Facebook announced this morning that it had completed its move to neural machine translation a complicated way of saying that Facebook is now using convolutional neural networks (CNNs) and recurrent neural networks (RNNs) to automatically translate content across Facebook.

Google, Microsoft and Facebook have been making the move to neural machine translation for some time now, rapidly leaving old-school phrase-based statistical machine translation behind. There are a lot of reasons why neural approaches show more promise than phrase-based approaches, but the bottom line is that they produce more accurate translations.

Traditional machine translation is a fairly explicit process. Relying on key phrases, phrase-based systems translate sentences then probabilistically determine a final translation. You can think of this in a similar light as using theRosetta Stone (identical phrases in multiple languages) to translate text.

In contrast, neural models deal in a higher level of abstraction. The interpretation of a sentence becomes part of a multi-dimensional vector representation, which really just means were trying to translate based on some semblance of context rather than phrases.

Facebook Status update translation

Its not a perfect process, and researchers are still tinkering with how to deal with long-term dependencies (i.e. retaining understanding and accuracy throughout a long text), but the approach is incredibly promising and has produced great results, thus far, for those implementing it.

Google announced the first stage of its move to neural machine translationin September 2016 and Microsoft made a similar announcement two months later. Facebook has been working on its conversion efforts for about a year and its now at full deployment. Facebook AI Research (FAIR) published its own research on the topic back in May and open sourced its CNN models on GitHub.

Our problem is different than that of most of the standard places,mostly because of the type of language we see at Facebook, Necip Fazil Ayan, engineering manager in Facebooks language technologies group,explained to me in an interview. We see a lot ofinformal language and slang acronyms. The style of language is very different.

Facebook has seen about a 10 percent jump in translation quality. You can read more into the improvement in FAIRs research. The results are particularly striking for languages that lack a lot of data in the form of comparative translation pairs.

Read more: https://techcrunch.com/2017/08/03/facebook-finishes-its-move-to-neural-machine-translation/

Googles Tensor2Tensor makes it easier to conduct deep learning experiments

Googles brain team is open sourcing Tensor2Tensor, a new deep learning library designed to help researchers replicate results from recent papers in the field and push the boundaries of whats possible by trying new combinations of models, datasets and other parameters. The sheer number of variables in AI research combined with the fast pace of new developments makes it difficult for experiments run in two distinct settings to match. This is a pain for researchers and a drag on research progress.

The Tensor2Tensor library makes it easier to maintain best practices while conducting AI research. It comes equipped with key ingredients including hyperparameters, data-sets, model architectures and learning rate decay schemes.

The best part is that any of these components can be swapped in and out in a modular fashion without completely destroying everything. From a training perspective, this means that with Tensor2Tensor you can bring in new models and data sets at any time a much simpler process than would ordinarily be possible.

Google isnt alone in its pursuits to help make research more reproducible outside the lab. Facebook recently open sourced ParlAI, its tool to facilitate dialog research that comes prepackaged with commonly used datasets.

Similarly, Googles Tensor2Tensor comes with models from recent Google research projects like Attention Is All You Need and One Model to Learn Them All. Everything is available now on Github so you can start training your own deep learning-powered tools.

Read more: https://techcrunch.com/2017/06/19/tensor2tensor/

If AI Can Fix Peer Review in Science, AI Can Do Anything

Here’s how science works: You have a question about some infinitesimal sliver of the universe. You form a hypothesis, test it, and eventually gather enough data to support or disprove what you thought was going on. That’s the fun part. The next bit is less glamorous: You write a manuscript, submit it to an academic journal, and endure the gauntlet of peer review, where a small group of anonymous experts in your field scrutinize the quality of your work.

Peer review has its flaws. Human beings (even scientists) are biased, lazy, and self-interested. Sometimes they suck at math (even scientists). So, perhaps inevitably, some people want to remove humans from the process—and replace them with artificial intelligence. Computers are, after all, unbiased, sedulous, and lack a sense of identity. They are also, by definition, good at math. And scientists aren’t just waiting around for some binary brain to manifest a set of protocols for identifying experimental excellence. Journal publishers are already building this stuff, piecemeal.

Recently, a competition called ScienceIE challenged teams to create programs that could extract the basic facts out of sentences in scientific papers, and compare those to the basic facts from sentences in other papers. “The broad goal of my project is to help scientists and practitioners gain more knowledge about a research area more quickly,” says Isabelle Augenstein, a post-doctoral AI researcher at University College of London, who devised the challenge.

That’s a tiny part of artificial intelligence’s biggest challenge: processing natural human language. Competitors designed programs to tackle three subtasks: reading each paper and identifying its key concepts, organizing key words by type, and identifying relationships between different key phrases. And it’s not just an academic exercise: Augenstein is on a two-year contract with Elsevier, one of the world’s largest publishers of scientific research, to develop computational tools for their massive library of manuscripts.

She has her work cut out for her. Elsevier publishes over 2,5001 different journals. Each has an editor, who has to find the right reviewer for each manuscript. (In 2015, 700,000 peer reviewers reviewed over 1.8 million manuscripts across Elsevier’s journals; 400,000 were eventually published.) “The number of humans capable of reviewing a proposal is generally limited to the specialists in that field,” says Mike Warren, AI veteran and CTO/co-founder of Descartes Labs, a digital mapping company that uses AI to parse satellite images. “So, you’ve got this small set of people with PhDs, and you keep dividing them into disciplines and sub-disciplines, and when you’re done there might only be 100 people on the planet qualified to review a certain manuscript.” Augenstein’s work is part of Elseviers work to automatically suggest the right reviewers for each manuscript.

Elsevier has developed a suite of automated tools, called Evise, to aid in peer review. The program checks for plagiarism (although that’s not really AI, just a search and match function), clears potential reviewers for things like conflicts of interest, and handles workflow between authors, editors, and reviewers. Several other major publishers have automated software to aid peer review—Springer-Nature, for instance, is currently trialing an independently-developed software package called StatReviewer that ensures that each submitted paper has complete and accurate statistical data.

But none seem as open about their capabilities or aspirations as Elsevier. “We are investigating more ambitious tasks,” says Augenstein. “Say you have a question about a paper: A machine learning model reads the paper and answers your question.”

Thank you very much, Dr. Roboto, but no thanks

Not everyone is charmed by the prospect of Dr. Roboto, PhD. Last month, Janne Hukkinen, professor of environmental policy at University of Helsinki, Finland, and editor of the Elsevier journal Ecological Economics wrote a cautionary op-ed for WIRED, premised on a future where AI peer review became fully autonomous:

I dont see why learning algorithms couldnt manage the entire review from submission to decision by drawing on publishers databases of reviewer profiles, analyzing past streams of comments by reviewers and editors, and recognizing the patterns of change in a manuscript from submission to final editorial decision. Whats more, disconnecting humans from peer review would ease the tension between the academics who want open access and the commercial publishers who are resisting it.

By Hukkinens logic, an AI that could do peer review could also write manuscripts. Eventually, people become a legacy system within the scientific method—redundant, inefficient, obsolete. His final argument: “New knowledge which humans no longer experience as something they themselves have produced would shake the foundations of human culture.”

But Hukkinens dark vision of machines capable of outthinking human scientists is, at the very least, decades away. “AI, despite its big successes in games like chess, Go, and poker, still cant understand most normal English sentences, let alone scientific text,” says Oren Etzioni, CEO of the Allen Institute for Artificial Intelligence. Consider this: The winning team from Augensteins ScienceIE competition scored 43 percent across the three subtasks.

And even non-computer brains have a hard time comprehending the passive-voiced mumbo jumbo common in scientific manuscripts; it is not uncommon for inscriptions within the literature to be structured such that the phenomenon being discussed is often described, after layers of prepositional preamble, and in vernacular that is vague, esoteric, and exorbitant, as being acted upon by causative factors. Linguists call anything written by humans, for humans, natural language. Computer scientists call natural language a hot mess.

“One large category of problems in natural language for AI is ambiguity,” says Ernest Davis, a computer scientist at NYU who studies common sense processing. Lets take a classic example of ambiguity, illustrated in this sentence by Stanford University emeritus computer scientist Terry Winograd:

The city councilmen refused the demonstrators a permit because they [feared/advocated] violence.

To you and me, the verbs give away who they refers to: the city council fears; the demonstrators advocate. But a computer brain would have a hell of a time figuring out which verb indicates which pronoun. And that type of ambiguity is just one thread in the tangled knot of natural language—from simple things like understanding homographs to unraveling the logic of narratives.

That’s not even touching on the specific issues in scientific papers, like connecting a written argument to some pattern in the data. This is even the case in pure mathematics papers. “Going from English to the formal logic of mathematics is not something we can automate,” says Davis. “And that would be one of the easiest things to work on because it’s highly restrictive and we understand the targets.” Disciplines that aren’t rooted in mathematics, like psychology, will be even more difficult. “In psychology papers, were nowhere near being able to check the reasonableness of arguments, says Davis. We dont know how to express the experiment in a way that a computer could use it.

And of course, a fully autonomous AI peer reviewer doesnt just have to outread humans, it has to outthink them. “When you think about AI problems, peer review is probably among the very hardest you can come up with, since the most important part of peer review is determining that research is novel, its something that has not been done before by someone else,” says Warren. A computer program might be able to survey the literature and figure out which questions remain, but would it be able to pick out research of Einsteinian proportions—some new theory that completely upends previous assumptions about how the world works?

Then again, what if everyone—AI advocates and critics alike—are looking at the problem backwards? “Maybe we just need to change the way we do scientific publishing,” says Tom Dietterich, AI researcher at Oregon State University. So, rather than writing our research as a story in English, we link our claims and evidence into a formalized structure, like a database, containing all the things that are known about a problem people are working on. Computerize the process of peer review, in other words, rather than its solution. But at that point its not computers youre reprogramming: Youre reprogramming human behavior.

1 UPDATE: 2/22/2017 — Originally, this said Elsevier published 7,500 journals. This was due to either a typo or just poorly-transcribed information. Either way, it’s fixed now.

Read more: https://www.wired.com/2017/02/ai-can-solve-peer-review-ai-can-solve-anything/

Move Over, CodersPhysicists Will Soon Rule Silicon Valley

It’s a bad time to be a physicist.

At least, thats what Oscar Boykin says. He majored in physics at the Georgia Institute of Technology and in 2002 he finished a physics PhD at UCLA. But four years ago, physicists at the Large Hadron Collider in Switzerland discovered the Higgs boson, a subatomic particle first predicted in the 1960s. As Boykin points out, everyone expected it. The Higgs didn’t mess with the theoretical models of the universe. It didn’t change anything or give physcists anything new to strive for. “Physicists are excited when there’s something wrong with physics, and we’re in a situation now where there’s not a lot that’s wrong,” he says. “It’s a disheartening place for a physicist to be in.” Plus, the pay isn’t too good.

Boykin is no longer a physicist. He’s a Silicon Valley software engineer. And it’s a very good time to be one of those.

Boykin works at Stripe, a $9-billion startup that helps businesses accept payments online. He helps build and operate software systems that collect data from across the company’s services, and he works to predict the future of these services, including when, where, and how the fraudulent transactions will come. As a physicist, he’s ideally suited to the job, which requires both extreme math and abstract thought. And yet, unlike a physicist, he’s working in a field that now offers endless challenges and possibilities. Plus, the pay is great.

If physics and software engineering were subatomic particles, Silicon Valley has turned into the place where the fields collide. Boykin works with three other physicists at Stripe. In December, when General Electric acquired the machine learning startup Wise.io, CEO Jeff Immelt boasted that he had just grabbed a company packed with physicists, most notably UC Berkeley astrophysicist Joshua Bloom. The open source machine learning software H20, used by 70,000 data scientists across the globe, was built with help from Swiss physicist Arno Candel, who once worked at the SLAC National Accelerator Laboratory. Vijay Narayanan, Microsoft’s head of data science, is an astrophysicist, and several other physicists work under him.

Its not on purpose, exactly. “We didn’t go into the physics kindergarten and steal a basket of children,” says Stripe president and co-founder John Collison. “It just happened.” And it’s happening across Silicon Valley. Because structurally and technologically, the things that just about every internet company needs to do are more and more suited to the skill set of a physicist.

The Naturals

Of course, physicists have played a role in computer technology since its earliest days, just as they’ve played a role in so many other fields. John Mauchly, who helped design the ENIAC, one of the earliest computers, was a physicist. Dennis Ritchie, the father of the C programming language, was too.

But this is a particularly ripe moment for physicists in computer tech, thanks to the rise of machine learning, where machines learn tasks by analyzing vast amounts of data. This new wave of data science and AI is something that suits physicists right down to their socks.

Among other things, the industry has embraced neural networks, software that aims to mimic the structure of the human brain. But these neural networks are really just math on an enormous scale, mostly linear algebra and probability theory. Computer scientists aren’t necessarily trained in these areas, but physicists are. “The only thing that is really new to physicists is learning how to optimize these neural networks, training them, but that’s relatively straightforward,” Boykin says. “One technique is called Newton’s method. Newton the physicist, not some other Newton.”

Chris Bishop, who heads Microsoft’s Cambridge research lab, felt the same way thirty years ago, when deep neural networks first started to show promise in the academic world. That’s what led him from physics into machine learning. “There is something very natural about a physicist going into machine learning,” he says, “more natural than a computer scientist.”

The Challenge Space

Ten years ago, Boykin says, so many of his old physics pals were moving into the financial world. That same flavor of mathematics was also enormously useful on Wall Street as a way of predicting where the markets would go. One key method was The Black-Scholes Equation, a means of determining the value of a financial derivative. But Black-Scholes helped foment the great crash of 2008, and now, Boykin and others physicists say that far more of their colleagues are moving into data science and other kinds of computer tech.

Earlier this decade, physicists arrived at the top tech companies to help build so-called Big Data software, systems that juggle data across hundreds or even thousands of machines. At Twitter, Boykin helped build one called Summingbird, and three guys who met in the physics department at MIT built similar software at a startup called Cloudant. Physicists know how to handle data—at MIT, Cloudant’s founders handled massive datasets from the the Large Hadron Collider—and building these enormously complex systems requires its own breed of abstract thought. Then, once these systems were built, so many physicists have helped use the data they harnessed.

In the early days of Google, one of the key people building the massively distributed systems in the companys engine room was Yonatan Zunger, who has a PhD in string theory from Stanford. And when Kevin Scott joined the Google’s ads team, charged with grabbing data from across Google and using it to predict which ads were most likely to get the most clicks, he hired countless physicists. Unlike many computer scientists, they were suited to the very experimental nature of machine learning. “It was almost like lab science,” says Scott, now chief technology officer at LinkedIn.

Now that Big Data software is commonplace—Stripe uses an open source version of what Boykin helped build at Twitter—its helping machine learning models drive predictions inside so many other companies. That provides physicists with any even wider avenue into the Silicon Valley. At Stripe, Boykin’s team also includes Roban Kramer (physics PhD, Columbia), Christian Anderson (physics master’s, Harvard), and Kelley Rivoire (physics bachelor’s, MIT). They come because they’re suited to the work. And they come because of the money. As Boykin says: “The salaries in tech are arguably absurd.” But they also come because there are so many hard problems to solve.

Anderson left Harvard before getting his PhD because he came to view the field much as Boykin does—as an intellectual pursuit of diminishing returns. But that’s not the case on the internet. “Implicit in ‘the internet’ is the scope, the coverage of it,” Anderson says. “It makes opportunities much greater, but it also enriches the challenge space, the problem space. There is intellectual upside.”

The Future

Today, physicists are moving into Silicon Valley companies. But in the years come, a similar phenomenon will spread much further. Machine learning will change not only how the world analyzes data but how it builds software. Neural networks are already reinventing image recognition, speech recognition, machine translation, and the very nature of software interfaces. As Microsofts Chris Bishop says, software engineering is moving from handcrafted code based on logic to machine learning models based on probability and uncertainty. Companies like Google and Facebook are beginning to retrain their engineers in this new way of thinking. Eventually, the rest of the computing world will follow suit.

In other words, all the physicists pushing into the realm of the Silicon Valley engineer is a sign of a much bigger change to come. Soon, all the Silicon Valley engineers will push into the realm of the physicist.

Read more: https://www.wired.com/2017/01/move-coders-physicists-will-soon-rule-silicon-valley/