แท็ก: AI For Business
2 ผลลัพธ์ที่เกี่ยวข้อง
Get to know the Reranking model technique: A key tool for enterprise information retrieval system
17 Dec 2024
We are now in the age of Large Language Models (LLMs). Since OpenAI released the GPT model, we've seen many applications emerge, such as smart search systems, knowledge management systems, chatbots, and machine translation tools. Each of these applications is powered by LLMs at their core. These applications rely heavily on data, requiring us to find similarities and rank results before processing. While the basic and commonly used method is to find similarity based on cosine similarity between two vectors, today AIGEN will introduce a new method: the reranking model. Before we delve into the reranking model that AIGEN uses to improve the accuracy and performance of many AI services, we need to understand the history of ranking. We'll explore the evolution from traditional full-text search using BM25, to vector search, and finally to reranking models. We'll conclude by comparing the metrics of each method, highlighting their pros and cons. How many types of ranking method? Ranking techniques play a crucial role in information retrieval and natural language processing tasks. Some of the most widely used methods include: Full text search BM25 This is a probabilistic ranking function used to estimate the relevance of documents to a given search query. BM25 (Best Matching 25) is an improvement over earlier models and is widely used in search engines due to its effectiveness and simplicity. Vector Similarity This method represents documents and queries as vectors in a high-dimensional space. The similarity between a document and a query is then computed using metrics like cosine similarity or dot product. This approach is particularly useful when dealing with semantic meaning rather than just keyword matching. Methods like TF-IDF vectorization or more advanced techniques like word embeddings (e.g., Word2Vec, GloVe) are often used to create these vector representations. Reranking method
Diary of AI CEO [EP.1] : Get to know with “Artificial AGI”
4 Dec 2024
Given the excitement of Generative AI Large-language model (LLM) intelligence and how it can now solve problems, answer questions naturally like humans, words on the street have started to talk about whether we are near AGI or Artificial General Intelligence. AGI can be defined as the AI that is generalized such that it can deduce new knowledge from old ones through reasoning even on the things it has never seen before. However, despite the incredible smartness we have witnessed of LLMs in recent years, at best these LLMs seem to be able to appear intelligent in so many subject domains because of the way it memorizes patterns from the enormous data it has been trained with. And it does not really use reasoning in creating responses. It is argued here that even if we reached what appears to be AGI through current LLM technology, at best it is only artificial AGI hence the term artificial Artificial General Intelligence in the title (it’s not an editing mistake!) What is Artificial AGI? Seemingly, these LLM’s appear to be almost like AGI. But in fact, at best it can only be artificial AGI or Artificial Artificial General Intelligence. That is, the true AGI should be able to reason and infer new domain knowledge from existing domain knowledge it was trained with just like humans do when it is still a small child. The way humans learn just by seeing a few examples of new information is still out of reach of AGI for the same computing resource and memory. It is true that GenAI LLM can seemingly infer correct answers for absolutely new things it has never seen before (zero-shot inference), but the mechanism that it does was simply from seeing many things similar before, not from “reasoning”.
