7 min read.
Ifit’sdifficult for a person to find the information they are looking for, imagine how complex it is to teachartificialintelligence algorithms to identify relevant information and deliver it when a user needs it! This is precisely the challenge we encountered when developing FindIt: an intelligent platform that, much like Netflix,brings knowledge generated by the IDB Group to its staff and external audiences.
We facedtwomainchallenges creating this platform:
- Offeringcontent recommendations to our usersproactively, requiring minimal effort and delivering information even before they ask for it.
- Recognizingthe level of experience of colleagues based on data that the organization already had, to be able to recommendwho to talk toabout a certain topic.
We are proud that we managed to overcome both challenges,and in thisarticlewe’lltell you how.
Howdo youteach algorithms to simulate intelligence?
To start, think like a person! Whenyouneed to find information on a specific topic, you think about it in words and concepts, using natural language. So,for an algorithm, which is just a finite set of instructions, tohave any chance of responding to a request withrelevant suggestions,ithas tolearn to understand human language.Even more difficult in this case,it alsohas tolearnto understand the jargonthatwe use at the IDB.
There aremultiplecurrenttrends inimplementingartificial intelligence,andnaturallanguageprocessing (NLP) is perhaps the most dynamic.NLPfocuses precisely on understanding,interpreting,and manipulating human language,and in thatvein, we applied2 NLP methodologies to design 2algorithmschools: one focused on Ontology, and another on Deep Learning.
TheOntologyFocusedSchool
The first lesson for our algorithms is Taxonomy which, in essence, is a hierarchical list of termsused toclassifyinformation into categories. What does a taxonomy look like? Imaginea treestructure where the main branches represent categories and the secondary branches subcategories. Here you can see an example:
Although complex, a taxonomy isinsufficient for our algorithms to understand human natural language andhave the ability torecommend relevant content to our audiences.Therefore, we createdan Ontology, which is a sophisticatedlanguagemodel that contains a set of taxonomies, called classes,whichrepresent families of concepts, andtheirrelationshipto each other.
This diagram canbeunderstoodin multiple ways, here is one interpretation as an example:
Author - Works for institutions
Author - Writescontent
Content -Has one or moretopics
Topic - It is related tosectors
Institutions - Belong to one or moresectors
Content - Is related to countries and consequently toregions
In the case of our algorithms, the advantage of an ontological model is that it teaches them language through concepts. These concepts belong to one or more well-establishedcategories, andhave attributes that describe their characteristics. When reviewingtext, our algorithms can identify the language, know the definition of the terms they recognize, understand the synonyms of those terms, and consistentlyinterpret jargon,dialectsand languages. But more importantly, they understand the relationships between conceptsallowing themto produce recommendations and answer complex questions such as:
- What content has the IDB recently publishedondigital transformation inLatin America andtheCaribbean?
- Are we working on projects that use Drones?
- What policies or actions have been proposed for the economic recovery of Small and Medium Enterprises in LAC after the pandemic?
We scaledthis process to the level of 80,000 digital resourcesby creatinga Knowledge Graphcontainingtheconnections that allow our algorithms to makecontentrecommendations and learn from their own experience.
Interesting, but what is the result? JustlikeAmazon recommends products to you,FindItand its algorithms infer that if a user visits a publicationabout initiatives to increase gender equality, they will surely be interested in otherrelatedresources and deliver them in the same interaction.Click on the following imageto see an example liveand start living theFindItexperience.
The SchoolFocused onDeep Learning
In the field of artificial intelligence, deep learning is one of the areas that hastrulyincreased our ability to create intelligent machines. At its core, deep learning is about using algorithms inspired by the structure and function of the human brain. These neural networks, as they are called,iterativelyprocess large amounts of data to discover and infer connections between the data.In seconds, deep learningcan performa volume of analysis that would take a human being several monthsoreven years.
To apply thismethodologyinthe school focused on deep learning, wegatheredmore than2.1 billion wordswritten in English and Spanish about theIDB Group’swork. These words came from sources asvariedas publications, job descriptions, strategies, and project proposals. We analyzedthat largeamountof words with an algorithm that generates word embeddings, creatinga model that reveals the relationships between concepts inmultipledimensions. It is important to emphasize that these associations reflect our jargon, ourparticular wayof speaking in the institution, and not simply standard Spanish or English. As an example, the image belowpresentssome interesting connections that the model returned:
The word "agriculture" is related to "livestock"," forestry "and" mining ", which is understandable, but the model also shows that the word "econometrics" is closely connected to" agriculture ", which makes sense in the context of the work we do."Agriculture" is also related to "agricultural", which is close to the term "El Salvador", where we support agricultural projects, which we call "operations"."Operations" in turn, is connected to terms that weuse internally torefer to our operational work at the IDB, terms such as "loans", "TCs", and "non-reimbursable funds". This is an unsupervised process, which means that alltheconnections between terms are mapped by an algorithm, with no need for human curation, unlike the ontology-focused school that requires regular manual supervision. Although the graph above shows three examples, remember that thefullmodelwascreatedona scale of more than 2 billion words.
There are manypotentialuses for this language model that our jargon map reveals. In the case ofFindIt, we used it to bring a new perspective toanalyzetext that the organization already had on itspersonnelto revealevidence oftheir skills and experiences.The end resultis a tacit knowledgelocator, so to speak, that allows colleagues to easily and quickly connectwith each otherto answer a question, share relevant experience, or bring specific skills to a project or team.And it is alldriven by that language model.
Check outone of the results when yousearchfornatural language processing. Good jobFindIt!
Complementarity: two models is better than one
FindItgraduated from both schools: the one focused on ontology and the one focused on deep learning. The learning obtained has been applied to understand, classify, and organizedigital resources, as well as to infera person’sknowledgeprofile.As a result, now,whenfaced with a specific requestmadein words,FindItcontextualizes and suggests relevant information from the IDB Group's universe of knowledge. This ability to connect userswithknowledge increases our capacity for collaboration, andgenerates greater reuse of knowledge, which takes us one step further on the path of digital transformation.
By Kyle Strand, Senior Knowledge Management Specialist, and Monica Hernandez, consultant in the Knowledge and Learning Department of the IDB