My Work

The OpenAI Moderation API has always been the essential endpoint you wire into every production surface that accepts user-generated content. The big difference today is that moderation is no longer limited to text: The omni-moderation-latest model is a next-generation multimodal content moderation system built on GPT-4o which can classify  text and images  with a single request, and it gives you better tools for understanding  why  something was flagged. This post is a follow-up to my earlier deep dive on the OpenAI moderation classifier: https://blog1.neuralengineer.org/llm-moderation-classifer-openai-moderation-api-fdb124c4536a What moderation is (and is not) Moderation answers one question: “Does this content appear to fall into one of the policy categories I care about?” It is: - A fast, structured classifier for  routing  (allow, block, review, rate-limit, redact). - A complement to your product policy (not a replacement). It is not: - A substitute for...

Sentence similarity involves determining the likeness between two texts. The idea behind semantic search is to embed all entries in your corpus, whether sentences, paragraphs, or documents, into a vector space. The query is embedded into the same vector space at search time, and the closest embeddings from your corpus are found. Some applications of sentence similarity include question answering, passage retrieval, paraphrase matching, duplicate question retrieval, and semantic search. In this article, we will explore semantic search. The application uses sentence similarity to implement a document search on a Medium blog article. The input to the application will be a question/sentence, and the output will be a set of sentences containing semantically similar content to the input sentence. The Sentence Transformers library The  Sentence Transformers  library is open-source for creating state-of-the-art embeddings from text and computing sentence similarity. Hugging Face offer...