Rewriting OpenAI web browsing plugin from scratch
Introduction
Two weeks ago, OpenAI made a decision to discontinue its AI chatbot plugin for web browsers. This add-on used to channel user queries to the Microsoft Bing search engine, but it was shut down on July 3, as it started displaying content in ways deemed inappropriate by OpenAI.
As a frequent user of the tool, I found the decision quite predictable due to the rapidly evolving nature of AI technology and the associated ethical concerns. Despite my understanding of the move, it impacted me, as the tool had become a significant part of my digital exploration.
In response to the shutdown, I embarked on a journey to recreate the plugin from scratch. My version of the tool, designed to capture the essential functionality of the original, is now available on GitHub for public use and contribution.
The project, which I called DataEnricherLLM following one of ChatGPT suggestions, evolved in something a little bit different once I embarked in solving the technical challenges behind this task. For example I soon find out that reaching out for content on the web, live, is very time consuming. I assume that the guys at OpenAI and Bing used some sort of multithread magic to quick reaching out for content, I didn’t want to take that road therefore I opted for a different approach: let’s forget about performance and let’s concentrate on creating the “perfect prompt” for gpt in order to come up with the “perfect answer”.
I also decided some other constraints:
- As LLM I will use OpenAI API, for several reasons: I wanted to experience with it, are a known standard, it was easy to understand if I was going on the right direction since I know ChatGPT as a user
- The project should be didactic first
- Change to another LLM and search engine should be easy
Background
I started checking my assumptions:
- Certain questions can’t be answered by my LLM with its own knowledge
- The LLM itself knows when it is able to answer without external help
Therefore I took a page from the prompting engineering thought as deeplearning.ai and used a similar prompt in order to ask the LLM itself about its degree of confidence toward a certain question, mimicking a best practice against hallucination.
Let’s assume to have available the usual function to interrogate OpenAI API with a valid key:
def get_completion(prompt):
messages = [{"role": "user", "content": prompt}]
response = openai.ChatCompletion.create(
model=model,
messages=messages,
temperature=0, # this is the degree of randomness of the model's output
)
return response.choices[0].message["content"]We can easily check if it seems confident on answering the most vannilla question I can think of:
question = 'Who was USA first president?'
print(get_completion(question))
prompt = f'Do you think you may need external help to answer this question: "{question}"? Just answer YES or NO'
print(get_completion(prompt))The first president of the United States was George Washington.
NOAnd know let’s check some other question
question = "What is the latest scientific consensus on the causes of bee population decline?"
prompt = f'Do you think you may need external help to answer this question: "{question}"? Just answer YES or NO'
print(get_completion(prompt))In this case I get a clear “YES”. Therefore I assume I can proceed asking to the LLM itself to provide 100 hundred question that may require external help, in order to test the results.
class SampleQuestions():
@staticmethod
def get_list():
test_questions = '''What is the latest scientific consensus on the causes of bee population decline?
What are the most recent advancements in electric vehicle battery technology?
What's the current status of international efforts to combat climate change?
What are the latest findings in research on Alzheimer's disease prevention?
How are augmented reality technologies being used in modern classrooms?
What new methods are being explored in the field of earthquake prediction?
How has the field of artificial intelligence evolved over the past decade?
Who are the leading contemporary poets and what are their major works?
What are the recent breakthroughs in deep learning algorithms for self-driving cars?
What new species have been discovered in the past year?
What are the latest developments in nuclear fusion research?
Can you provide an overview of the newest models of quantum computers?
How are countries adapting their infrastructure for the rise of electric vehicles?
What are the latest trends in sustainable fashion?
What is the current state of restoration efforts in the Amazon rainforest?
How have recent advancements in technology affected the film industry?
What are the latest breakthroughs in CRISPR gene editing technology?
How is the Internet of Things (IoT) impacting everyday life?
What is the current understanding of the health effects of prolonged space travel?
What are the latest advancements in renewable energy sources?
...Time to start to think how help out the LLM…
Implementation
Once I gave up of performance it was easy to establish this process that aims solely on getting the best results from the LLM hence to create the more helpful prompt for the LLM iselft.
After some experimentation on a notebook I came up with this flow:
The easier part was asking to my gpt for the most helpful queries for Google:
prompt = f'''Please provide a max {nqueries} web search queries that can help to answer this question: "{question}".
Format the result in JSON following this template:
{{
"queries": [...]
}}
'''
json_queries = get_completion(prompt)The result seems fair:
{
"queries": [
"Latest scientific consensus on bee population decline causes",
"Causes of bee population decline according to scientific research",
"Scientific consensus on factors contributing to bee population decline",
"Current understanding of the causes of declining bee populations"
]
}Then it’s Google time: I admit it was the most annoying part. I didn’t want to use Google API (which requires keys) therefore I ended up using a library that wraps web calls to google in order to bypass keys creations and billing. It’s not fault proof and for production you don’t want to adopt this approach, but for my purpose was enough.
from googlesearch import search
def get_urls(queries):
urls = []
for query in queries:
search_results = search(query, num_results=MAX_SEARCH_RESULTS)
for url in search_results:
urls.append((query, url))
time.sleep(0.2)
return list(set(urls))It takes few seconds (about 6 in this example) to come up with a fair result:
[('Scientific consensus on factors contributing to bee population decline',
'https://www.fao.org/news/story/en/item/1194910/icode/'),
('Latest scientific consensus on bee population decline causes',
'https://www.fao.org/news/story/en/item/1194910/icode/'),
('Scientific consensus on factors contributing to bee population decline',
'https://ucanr.edu/blogs/blogcore/postdetail.cfm?postnum=24276'),
('Causes of bee population decline according to scientific research',
'https://www.fao.org/news/story/en/item/1194910/icode/'),
('Causes of bee population decline according to scientific research',
'https://www.theguardian.com/environment/2020/jul/29/bees-food-crops-shortage-study'),
('Causes of bee population decline according to scientific research',
'https://bkgc.in/ejournal/paper_list/35-43.pdf'),
('Scientific consensus on factors contributing to bee population decline',
'https://sgp.fas.org/crs/misc/R43191.pdf'),
...Then it was the time for more interesting part of the whole project: dealing with tokens and context length! I soon find out that using large chunks of text from the web and copy&paste into a prompt is the road to exceed the maximum token length of my model.
To fix this issue I combined the usage of:
- spaCy, to split the text into paragraphs
- tiktoken, to check the token length of my string buffer
- A custom script, to create a text chunk of the maximum available length… not the more elegant code I’ve seen, but it suits its purpose
import spacy
def split_text_spacy(text: str):
nlp = spacy.load('en_core_web_sm')
doc = nlp(text)
sentences = [sent.text.strip() for sent in doc.sents]
return sentencesimport tiktoken
encoding = tiktoken.encoding_for_model(model)
def num_tokens_from_string(string: str) -> int:
return len(encoding.encode(string))def get_chunks(content):
pars = split_text_spacy(content)
prompt_delta = num_tokens_from_string(get_information_prompt()) + 1
buffer = '\n'
for par in pars:
if num_tokens_from_string(buffer + '\n' + par) + prompt_delta > MAX_MODEL_TOKENS:
yield buffer
buffer = ''
buffer += '\n' + par
yield bufferAt this point, having a reliable method to interrogate the LLM without context length worries, it’s time for some prompting.
At first I ask to extract information from a chunk of text scraped from the web (this code is from the actual github project):
@staticmethod
def get_information_prompt(content = '', question = ''):
return f'''
Considering the web page text between triple ``` as the source,
extract and summarize all information that can be used on answering the question "{question}".
```{content}```
'''And from this result I interrogate the LLM once more in order to rank this information. I first tried to call gpt just once in order to save some time and money, but I got poor result and switched to this safe approach (this code is also from the actual github project):
@staticmethod
def get_rank_information_prompt(infos, question):
return f'''
Given the text between triple ``` as the source
and the question "{question}",
please evaluate and provide a ranking on a scale from 1 to 100,
with 1 indicating no relevance and 10 indicating high relevance,
on how well the text answers the question.
```{infos}```
Format the result in JSON following this template:
{{
"rank": ...
}}
'''This is the prompt tries (and fails) to perform the previous tasks in one go:
@staticmethod
def get_summary_prompt(topic, content):
return f'''You have 2 tasks:
1) extract all the informations related to the topic "{topic}" and write it down in plain english, present the informations without introduction, in a detailed and precise way, without changin the terms and numbers used in the text.\'
2) measure the relevance of that information to the topic "{topic}"
Present the informations without introduction, in a detailed and precise way, without changin the terms and numbers used in the text.
Write the answer in JSON format using this template:
{{
"relevance": RELEVANCE,
"informations": INFORMATIONS,
}}
\n
\'\'\'"{content}"\'\'\'
'''At the end of this process I have ranked list of information that I can use to compose my “enriched” prompt, here the code the uses the prompts above in order to get the result:
def get_data(self, question, all_urls):
data = []
for query, url in all_urls:
self.logger.info(f'Reading {url} related to {query}')
if self.webreader.is_text_content(url):
content = self.webreader.get_text_from_url(url)
chunks = self.llm_api.get_chunks(content)
for chunk in chunks:
infos = self.get_information(chunk, query, question)
rank = self.rank_information(infos, question)
data.append((rank, query, url, infos))
else:
self.logger.info(f'skip {url}')
self.logger.info(f'Found {len(data)} infos')
# cut infos not really relevant
data = [infos for infos in data if infos[0] > 50]
self.logger.info(f'Selected {len(data)} infos')
# return sorted by relevance
return sorted(data, key=lambda x: x[0], reverse=True)So I can compose the prompt, adding all the information I’ve got (until I used it all or reached the maximum amount or token that I can safely use):
prompt = Prompts().get_enriched_prompt(question)
enriched_prompt = prompt
for d in data:
prompt += Prompts().get_prompted_infos(d[2], d[3])
if self.llm_api.does_prompt_exceeds_maxtokens(prompt):
break
enriched_prompt = promptConclusions
I am still testing and tuning (help is always welcome btw) but it seems clear to me that this approach may actually lead to some pretty good results.
For the question “What is the latest scientific consensus on the causes of bee population decline?”, if you ask gpt though API you still get an answer:
The latest scientific consensus suggests that bee population decline is caused by a combination of multiple factors, often referred to as the “Four P’s”: pesticides, parasites, pathogens, and habitat loss. Here is a breakdown of each factor:
1. Pesticides: The use of pesticides, particularly neonicotinoids, has been linked to bee population decline. These chemicals can have sublethal effects on bees, impairing their navigation, foraging abilities, reproduction, and immune systems.
2. Parasites: The Varroa mite (Varroa destructor) is a significant parasite that infests honeybees and weakens their immune systems, making them more susceptible to diseases. This mite has been a major contributor to colony losses worldwide.
3. Pathogens: Bees are vulnerable to various diseases and viruses, including Nosema, American Foulbrood, and Deformed Wing Virus. These pathogens can spread within colonies and impact bee health and survival.
4. Habitat loss: The loss and degradation of natural habitats, including wildflower meadows, forests, and hedgerows, have reduced the availability of diverse and nutritious food sources for bees. This lack of forage negatively affects their nutrition and overall health.
It is important to note that the relative importance of these factors may vary depending on the region and specific bee species. Additionally, climate change, air pollution, and changes in land use practices can also contribute to bee population decline, although their exact roles are still being studied. Ongoing research aims to better understand the interactions and cumulative effects of these factors to develop effective conservation strategies for bee populations.
But after the whole (very long) process of creating this (very long) prompt, from the same gpt you get the answer:
The latest scientific consensus on the causes of bee population decline includes several factors.
These factors include disturbances of natural habitats due to agricultural intensification and urbanization, climate change, pesticides, management practices, invasive alien species, intensive farming practices, mono-cropping, excessive use of agricultural chemicals, higher temperatures associated with climate change, biodiversity loss, pollution, urbanization, changes in the way we live, and increased use of herbicides, insecticides, and pesticides.
Habitat loss and fragmentation, as well as climate changes, are considered major drivers of pollinator declines.
The presence of managed honey bee colonies in the environment can also compromise habitat suitability for wild bees. Pesticides, particularly agrochemicals, have been shown to have deleterious effects on eusocial bees. The decline of wild bee populations can have negative impacts on plant-pollinator interactions, especially in tropical biomes. Nutritional stress due to habitat loss may explain recent honeybee colony collapses. Pesticide residues in the hive environment have also been linked to winter colony losses.
The infestation of Varroa destructor, a parasite, can impact the winter survival of individual honey bees and colonies.
Other factors that contribute to bee population decline include climate change, diseases, limited resources, and the increasing demand for pollination services.
It is important to note that there are still knowledge gaps in understanding the causes of wild bee losses, how wild bees respond to different stressors, and the consequences of their extinction.
To address the decline of wild bee populations, comprehensive approaches involving environmental education, risk assessments, media coverage, and dialogue between various stakeholders are needed. It is also important for individuals to provide a honey bee-friendly habitat, support organic food and purchase honey, avoid the use of insecticides and limit the use of herbicides, and refrain from killing bees.
[Sources: https://www.frontiersin.org/articles/10.3389/fevo.2022.1027169/full, https://www.fao.org/news/story/en/item/1194910/icode/, https://www.osu.edu/impact/research-and-innovation/bee-population, https://www.nature.com/articles/s41598-022-25290-3]
Which is very different from the baseline and may be more helpful.
In the github repository there are many other example, like “What is the current state of restoration efforts in the Amazon rainforest?” which has the baseline:
The current state of restoration efforts in the Amazon rainforest is a complex and ongoing process. The Amazon rainforest has been subject to deforestation and degradation due to various factors such as agriculture, logging, mining, and infrastructure development. However, there are several initiatives and organizations working towards restoring and conserving the Amazon rainforest.
1. Protected Areas and Indigenous Reserves: The establishment and expansion of protected areas and indigenous reserves have been crucial in conserving the Amazon rainforest. These areas help prevent further deforestation and provide a habitat for numerous plant and animal species.
2. Reforestation and Afforestation: Many organizations and governments are involved in reforestation and afforestation projects in the Amazon. These initiatives aim to restore degraded areas by planting native tree species, promoting natural regeneration, and creating corridors to connect fragmented forests.
3. Sustainable Land Use Practices: Encouraging sustainable land use practices, such as agroforestry and sustainable agriculture, is essential for the restoration of the Amazon rainforest. These practices promote the integration of trees with crops and livestock, reducing the pressure on natural forests.
4. International Cooperation and Funding: International organizations, governments, and NGOs are providing financial and technical support for restoration efforts in the Amazon. Initiatives like the Amazon Fund, supported by countries like Norway and Germany, aim to finance projects that promote conservation and sustainable development in the region.
5. Indigenous and Local Community Involvement: Recognizing the rights and involvement of indigenous and local communities is crucial for successful restoration efforts. These communities have traditional knowledge and practices that can contribute to the conservation and restoration of the Amazon rainforest.
Despite these efforts, the Amazon rainforest still faces significant challenges, including illegal logging, land encroachment, and climate change. Continued collaboration, increased funding, and stricter enforcement of environmental regulations are necessary to ensure the long-term restoration and conservation of the Amazon rainforest.
While the result of the “enriched” prompt gets:
The current state of restoration efforts in the Amazon rainforest is a major initiative involving multiple partnerships and projects. The goal is to restore degraded areas and plant millions of trees to promote sustainable development, preserve biodiversity, and mitigate the impacts of climate change.
According to the information from the World Bank [https://www.worldbank.org/en/news/press-release/2017/09/15/brazil-initiative-recovers-forest-lands-amazon], the initiative aims to restore almost 30,000 hectares of land, which is equivalent to an estimated 73 million trees, over the next six years. So far, 3 million trees have already been restored on 1,200 hectares of the Amazon Forest through contributions from various organizations, including Rock in Rio, Conservation International, and the general public.
The remaining 70 million trees are part of the goals of the Amazon Sustainable Landscapes Program, which aims to expand the area of the forest being restored, promote sustainable use of natural resources, and strengthen the network of Conservation Units in the Brazilian Amazon. The restoration efforts involve various methods such as enriching existing secondary forests, seeding native species, promoting natural regeneration, and direct planting of native species.
The priority areas for restoration actions are the southern regions of the states of Amazonas, Rondonia, Acre, and Para. The partnership between the Amazon Sustainable Landscapes Program and Amazonia Live aims to preserve the forest, mitigate the impacts of climate change, and strengthen local communities.
Additionally, according to the information from ReforestAction [https://www.reforestaction.com/en/magazine/brazil-new-planting-season-within-center-bioeconomy-and-conservation-amazon-cbca], there have been specific projects focused on agroforestry and restoring degraded pastures. These projects aim to establish sustainable agricultural models, protect against deforestation caused by soybean fields and intensive cattle breeding, and create edible forests.
The Centre for Bioeconomy and Conservation of the Amazon (CBCA) is leading these restoration efforts and aims to promote the sustainable development of the Amazon by rehabilitating native vegetation. The projects contribute to the progress of the CBCA and the scientific data it disseminates.
Local communities are actively involved in the restoration process, with job creation and income generation being social benefits. Partnerships have been established with local associations of producers and residents, and these communities will be responsible for monitoring and maintaining the planted trees in the future.
Overall, the current state of restoration efforts in the Amazon rainforest involves a combination of initiatives focused on restoring degraded areas, planting millions of trees, promoting sustainable development, involving local communities, and preserving biodiversity.
