[{"data":1,"prerenderedAt":-1},["ShallowReactive",2],{"similar-JohnSnowLabs--nlu":3,"tool-JohnSnowLabs--nlu":64},[4,17,27,35,43,56],{"id":5,"name":6,"github_repo":7,"description_zh":8,"stars":9,"difficulty_score":10,"last_commit_at":11,"category_tags":12,"status":16},3808,"stable-diffusion-webui","AUTOMATIC1111\u002Fstable-diffusion-webui","stable-diffusion-webui 是一个基于 Gradio 构建的网页版操作界面，旨在让用户能够轻松地在本地运行和使用强大的 Stable Diffusion 图像生成模型。它解决了原始模型依赖命令行、操作门槛高且功能分散的痛点，将复杂的 AI 绘图流程整合进一个直观易用的图形化平台。\n\n无论是希望快速上手的普通创作者、需要精细控制画面细节的设计师，还是想要深入探索模型潜力的开发者与研究人员，都能从中获益。其核心亮点在于极高的功能丰富度：不仅支持文生图、图生图、局部重绘（Inpainting）和外绘（Outpainting）等基础模式，还独创了注意力机制调整、提示词矩阵、负向提示词以及“高清修复”等高级功能。此外，它内置了 GFPGAN 和 CodeFormer 等人脸修复工具，支持多种神经网络放大算法，并允许用户通过插件系统无限扩展能力。即使是显存有限的设备，stable-diffusion-webui 也提供了相应的优化选项，让高质量的 AI 艺术创作变得触手可及。",162132,3,"2026-04-05T11:01:52",[13,14,15],"开发框架","图像","Agent","ready",{"id":18,"name":19,"github_repo":20,"description_zh":21,"stars":22,"difficulty_score":23,"last_commit_at":24,"category_tags":25,"status":16},1381,"everything-claude-code","affaan-m\u002Feverything-claude-code","everything-claude-code 是一套专为 AI 编程助手（如 Claude Code、Codex、Cursor 等）打造的高性能优化系统。它不仅仅是一组配置文件，而是一个经过长期实战打磨的完整框架，旨在解决 AI 代理在实际开发中面临的效率低下、记忆丢失、安全隐患及缺乏持续学习能力等核心痛点。\n\n通过引入技能模块化、直觉增强、记忆持久化机制以及内置的安全扫描功能，everything-claude-code 能显著提升 AI 在复杂任务中的表现，帮助开发者构建更稳定、更智能的生产级 AI 代理。其独特的“研究优先”开发理念和针对 Token 消耗的优化策略，使得模型响应更快、成本更低，同时有效防御潜在的攻击向量。\n\n这套工具特别适合软件开发者、AI 研究人员以及希望深度定制 AI 工作流的技术团队使用。无论您是在构建大型代码库，还是需要 AI 协助进行安全审计与自动化测试，everything-claude-code 都能提供强大的底层支持。作为一个曾荣获 Anthropic 黑客大奖的开源项目，它融合了多语言支持与丰富的实战钩子（hooks），让 AI 真正成长为懂上",138956,2,"2026-04-05T11:33:21",[13,15,26],"语言模型",{"id":28,"name":29,"github_repo":30,"description_zh":31,"stars":32,"difficulty_score":23,"last_commit_at":33,"category_tags":34,"status":16},2271,"ComfyUI","Comfy-Org\u002FComfyUI","ComfyUI 是一款功能强大且高度模块化的视觉 AI 引擎，专为设计和执行复杂的 Stable Diffusion 图像生成流程而打造。它摒弃了传统的代码编写模式，采用直观的节点式流程图界面，让用户通过连接不同的功能模块即可构建个性化的生成管线。\n\n这一设计巧妙解决了高级 AI 绘图工作流配置复杂、灵活性不足的痛点。用户无需具备编程背景，也能自由组合模型、调整参数并实时预览效果，轻松实现从基础文生图到多步骤高清修复等各类复杂任务。ComfyUI 拥有极佳的兼容性，不仅支持 Windows、macOS 和 Linux 全平台，还广泛适配 NVIDIA、AMD、Intel 及苹果 Silicon 等多种硬件架构，并率先支持 SDXL、Flux、SD3 等前沿模型。\n\n无论是希望深入探索算法潜力的研究人员和开发者，还是追求极致创作自由度的设计师与资深 AI 绘画爱好者，ComfyUI 都能提供强大的支持。其独特的模块化架构允许社区不断扩展新功能，使其成为当前最灵活、生态最丰富的开源扩散模型工具之一，帮助用户将创意高效转化为现实。",107662,"2026-04-03T11:11:01",[13,14,15],{"id":36,"name":37,"github_repo":38,"description_zh":39,"stars":40,"difficulty_score":23,"last_commit_at":41,"category_tags":42,"status":16},3704,"NextChat","ChatGPTNextWeb\u002FNextChat","NextChat 是一款轻量且极速的 AI 助手，旨在为用户提供流畅、跨平台的大模型交互体验。它完美解决了用户在多设备间切换时难以保持对话连续性，以及面对众多 AI 模型不知如何统一管理的痛点。无论是日常办公、学习辅助还是创意激发，NextChat 都能让用户随时随地通过网页、iOS、Android、Windows、MacOS 或 Linux 端无缝接入智能服务。\n\n这款工具非常适合普通用户、学生、职场人士以及需要私有化部署的企业团队使用。对于开发者而言，它也提供了便捷的自托管方案，支持一键部署到 Vercel 或 Zeabur 等平台。\n\nNextChat 的核心亮点在于其广泛的模型兼容性，原生支持 Claude、DeepSeek、GPT-4 及 Gemini Pro 等主流大模型，让用户在一个界面即可自由切换不同 AI 能力。此外，它还率先支持 MCP（Model Context Protocol）协议，增强了上下文处理能力。针对企业用户，NextChat 提供专业版解决方案，具备品牌定制、细粒度权限控制、内部知识库整合及安全审计等功能，满足公司对数据隐私和个性化管理的高标准要求。",87618,"2026-04-05T07:20:52",[13,26],{"id":44,"name":45,"github_repo":46,"description_zh":47,"stars":48,"difficulty_score":23,"last_commit_at":49,"category_tags":50,"status":16},2268,"ML-For-Beginners","microsoft\u002FML-For-Beginners","ML-For-Beginners 是由微软推出的一套系统化机器学习入门课程，旨在帮助零基础用户轻松掌握经典机器学习知识。这套课程将学习路径规划为 12 周，包含 26 节精炼课程和 52 道配套测验，内容涵盖从基础概念到实际应用的完整流程，有效解决了初学者面对庞大知识体系时无从下手、缺乏结构化指导的痛点。\n\n无论是希望转型的开发者、需要补充算法背景的研究人员，还是对人工智能充满好奇的普通爱好者，都能从中受益。课程不仅提供了清晰的理论讲解，还强调动手实践，让用户在循序渐进中建立扎实的技能基础。其独特的亮点在于强大的多语言支持，通过自动化机制提供了包括简体中文在内的 50 多种语言版本，极大地降低了全球不同背景用户的学习门槛。此外，项目采用开源协作模式，社区活跃且内容持续更新，确保学习者能获取前沿且准确的技术资讯。如果你正寻找一条清晰、友好且专业的机器学习入门之路，ML-For-Beginners 将是理想的起点。",84991,"2026-04-05T10:45:23",[14,51,52,53,15,54,26,13,55],"数据工具","视频","插件","其他","音频",{"id":57,"name":58,"github_repo":59,"description_zh":60,"stars":61,"difficulty_score":10,"last_commit_at":62,"category_tags":63,"status":16},3128,"ragflow","infiniflow\u002Fragflow","RAGFlow 是一款领先的开源检索增强生成（RAG）引擎，旨在为大语言模型构建更精准、可靠的上下文层。它巧妙地将前沿的 RAG 技术与智能体（Agent）能力相结合，不仅支持从各类文档中高效提取知识，还能让模型基于这些知识进行逻辑推理和任务执行。\n\n在大模型应用中，幻觉问题和知识滞后是常见痛点。RAGFlow 通过深度解析复杂文档结构（如表格、图表及混合排版），显著提升了信息检索的准确度，从而有效减少模型“胡编乱造”的现象，确保回答既有据可依又具备时效性。其内置的智能体机制更进一步，使系统不仅能回答问题，还能自主规划步骤解决复杂问题。\n\n这款工具特别适合开发者、企业技术团队以及 AI 研究人员使用。无论是希望快速搭建私有知识库问答系统，还是致力于探索大模型在垂直领域落地的创新者，都能从中受益。RAGFlow 提供了可视化的工作流编排界面和灵活的 API 接口，既降低了非算法背景用户的上手门槛，也满足了专业开发者对系统深度定制的需求。作为基于 Apache 2.0 协议开源的项目，它正成为连接通用大模型与行业专有知识之间的重要桥梁。",77062,"2026-04-04T04:44:48",[15,14,13,26,54],{"id":65,"github_repo":66,"name":67,"description_en":68,"description_zh":69,"ai_summary_zh":69,"readme_en":70,"readme_zh":71,"quickstart_zh":72,"use_case_zh":73,"hero_image_url":74,"owner_login":75,"owner_name":76,"owner_avatar_url":77,"owner_bio":78,"owner_company":79,"owner_location":79,"owner_email":79,"owner_twitter":79,"owner_website":80,"owner_url":81,"languages":82,"stars":91,"forks":92,"last_commit_at":93,"license":94,"difficulty_score":10,"env_os":95,"env_gpu":96,"env_ram":96,"env_deps":97,"category_tags":101,"github_topics":102,"view_count":23,"oss_zip_url":79,"oss_zip_packed_at":79,"status":16,"created_at":122,"updated_at":123,"faqs":124,"releases":155},1992,"JohnSnowLabs\u002Fnlu","nlu","1 line for thousands of State of The Art NLP models in hundreds of languages The fastest and most accurate way to solve text problems.","NLU 是一个简洁强大的 Python 库，让你只需一行代码就能调用上千个顶尖的自然语言处理模型，支持超过百种语言。无论是情感分析、实体识别、词性标注，还是多语言文本分类，它都能直接在 Pandas、Spark 等数据框上运行，无需复杂配置。它解决了传统 NLP 模型部署繁琐、跨语言支持困难、代码冗长的问题，让开发者和研究人员能快速验证想法、构建原型或集成到生产流程中。特别适合数据科学家、NLP 工程师和需要快速测试文本分析效果的开发者使用。NLU 基于获奖的 Spark NLP 构建，支持模型训练、可视化和一键生成交互式 Web 界面（通过 Streamlit），并兼容多种数据格式。其核心亮点是“一行代码调用 SOTA 模型”，将复杂的技术封装成极简接口，大幅降低使用门槛，同时保留了工业级的性能与可扩展性。"," \n# NLU: The Power of Spark NLP, the Simplicity of Python\nJohn Snow Labs' NLU is a Python library for applying state-of-the-art text mining, directly on any dataframe, with a single line of code.\nAs a facade of the award-winning Spark NLP library, it comes with **1000+** of pretrained models in **100+**, all production-grade, scalable, and trainable, with **everything in 1 line of code.**\n\n\n\n## NLU in Action \nSee how easy it is to use any of the **thousands** of models in 1 line of code, there are hundreds of [tutorials](https:\u002F\u002Fnlu.johnsnowlabs.com\u002Fdocs\u002Fen\u002Fnotebooks) and [simple examples](https:\u002F\u002Fgithub.com\u002FJohnSnowLabs\u002Fnlu\u002Ftree\u002Fmaster\u002Fexamples) you can copy and paste into your projects to achieve State Of The Art easily.\n\u003Cimg src=\"http:\u002F\u002Fckl-it.de\u002Fwp-content\u002Fuploads\u002F2020\u002F08\u002FMy-Video6.gif\" width=\"1800\" height=\"500\"\u002F>\n\n## NLU & Streamlit in Action \nThis 1 line let's you visualize and play with **1000+ SOTA NLU & NLP models** in **200** languages \n\n```shell\nstreamlit run https:\u002F\u002Fraw.githubusercontent.com\u002FJohnSnowLabs\u002Fnlu\u002Fmaster\u002Fexamples\u002Fstreamlit\u002F01_dashboard.py \n```\n\u003Cimg src=\"https:\u002F\u002Foss.gittoolsai.com\u002Fimages\u002FJohnSnowLabs_nlu_readme_f2d54c29650a.gif\">\n\nNLU provides tight and simple integration into Streamlit, which enables building powerful webapps in just 1 line of code which showcase the.\nView the [NLU&Streamlit documentation](https:\u002F\u002Fnlu.johnsnowlabs.com\u002Fdocs\u002Fen\u002Fstreamlit_viz_examples) or [NLU & Streamlit examples section](https:\u002F\u002Fgithub.com\u002FJohnSnowLabs\u002Fnlu\u002Ftree\u002Fmaster\u002Fexamples\u002Fstreamlit). \nThe entire GIF demo and \n\n\n## All NLU resources overview\nTake a look at our official NLU page: [https:\u002F\u002Fnlu.johnsnowlabs.com\u002F](https:\u002F\u002Fnlu.johnsnowlabs.com\u002F) for user documentation and examples\n\n| Ressource | Description|\n|-----------------------------------------------------------------------|-------------------------------------------|\n| [Install NLU](https:\u002F\u002Fnlu.johnsnowlabs.com\u002Fdocs\u002Fen\u002Finstall) | Just run `pip install nlu pyspark==3.0.2` \n| [The NLU Namespace](https:\u002F\u002Fnlu.johnsnowlabs.com\u002Fdocs\u002Fen\u002Fnamespace) | Find all the names of models you can load with `nlu.load()`\n| [The `nlu.load(\u003CModel>)` function](https:\u002F\u002Fnlu.johnsnowlabs.com\u002Fdocs\u002Fen\u002Fload_api) | Load any of the **1000+ models in 1 line**\n| [The `nlu.load(\u003CModel>).predict(data)` function](https:\u002F\u002Fnlu.johnsnowlabs.com\u002Fdocs\u002Fen\u002Fpredict_api) | Predict on `Strings`, `List of Strings`, `Numpy Arrays`, `Pandas`, `Modin` and `Spark Dataframes`\n| [The `nlu.load(\u003Ctrain.Model>).fit(data)` function](https:\u002F\u002Fnlu.johnsnowlabs.com\u002Fdocs\u002Fen\u002Ftraining) | Train a text classifier for `2-Class`, `N-Classes` `Multi-N-Classes`, `Named-Entitiy-Recognition` or `Parts of Speech Tagging`\n| [The `nlu.load(\u003CModel>).viz(data)` function](https:\u002F\u002Fnlu.johnsnowlabs.com\u002Fdocs\u002Fen\u002Fviz_examples) | Visualize the results of `Word Embedding Similarity Matrix`, `Named Entity Recognizers`, `Dependency Trees & Parts of Speech`, `Entity Resolution`,`Entity Linking` or `Entity Status Assertion` \n| [The `nlu.load(\u003CModel>).viz_streamlit(data)` function](https:\u002F\u002Fnlu.johnsnowlabs.com\u002Fdocs\u002Fen\u002Fstreamlit_viz_examples) | Display an interactive GUI which lets you explore and test every model and feature in NLU in 1 click.\n| [General Concepts](https:\u002F\u002Fnlu.johnsnowlabs.com\u002Fdocs\u002Fen\u002Fconcepts) | General concepts in NLU\n| [The latest release notes](https:\u002F\u002Fnlu.johnsnowlabs.com\u002Fdocs\u002Fen\u002Frelease_notes) | Newest features added to NLU\n| [Overview NLU 1-liners examples](https:\u002F\u002Fnlu.johnsnowlabs.com\u002Fdocs\u002Fen\u002Fexamples) | Most common used models and their results\n| [Overview NLU 1-liners examples for healthcare models](https:\u002F\u002Fnlu.johnsnowlabs.com\u002Fdocs\u002Fen\u002Fexamples_hc) | Most common used healthcare models and their results \n| [Overview of all NLU tutorials and Examples](https:\u002F\u002Fnlu.johnsnowlabs.com\u002Fdocs\u002Fen\u002Fnotebooks) | 100+ tutorials on how to use NLU on text datasets for various problems and from various sources like Twitter, Chinese News, Crypto News Headlines, Airline Traffic communication, Product review classifier training,\n| [Connect with us on Slack](https:\u002F\u002Fjoin.slack.com\u002Ft\u002Fspark-nlp\u002Fshared_invite\u002Fzt-lutct9gm-kuUazcyFKhuGY3_0AMkxqA) | Problems, questions or suggestions? We have a very active and helpful community of over 2000+ AI enthusiasts putting NLU, Spark NLP & Spark OCR to good use \n| [Discussion Forum](https:\u002F\u002Fgithub.com\u002FJohnSnowLabs\u002Fspark-nlp\u002Fdiscussions) | More indepth discussion with the community? Post a thread in our discussion Forum\n| [John Snow Labs Medium](https:\u002F\u002Fmedium.com\u002Fspark-nlp) | Articles and Tutorials on the NLU, Spark NLP and Spark OCR\n| [John Snow Labs Youtube](https:\u002F\u002Fwww.youtube.com\u002Fchannel\u002FUCmFOjlpYEhxf_wJUDuz6xxQ\u002Fvideos) | Videos and Tutorials on the NLU, Spark NLP and Spark OCR\n| [NLU Website](https:\u002F\u002Fnlu.johnsnowlabs.com\u002F) | The official NLU website\n|[Github Issues](https:\u002F\u002Fgithub.com\u002FJohnSnowLabs\u002Fnlu\u002Fissues) | Report a bug\n\n\n\n\n\n\n## Getting Started with NLU \nTo get your hands on the power of NLU, you just need to install it via pip and ensure Java 8 is installed and properly configured. Checkout [Quickstart for more infos](https:\u002F\u002Fnlu.johnsnowlabs.com\u002Fdocs\u002Fen\u002Finstall)\n```bash \npip install nlu pyspark==3.0.2\n``` \n\n## Loading and predicting with any model in 1 line python \n```python\nimport nlu \nnlu.load('sentiment').predict('I love NLU! \u003C3') \n``` \n\n## Loading and predicting with multiple models in 1 line \n\nGet 6 different embeddings in 1 line and use them for downstream data science tasks! \n\n```python \nnlu.load('bert elmo albert xlnet glove use').predict('I love NLU! \u003C3') \n``` \n\n## What kind of models does NLU provide? \nNLU provides everything a data scientist might want to wish for in one line of code! \n - NLU provides everything a data scientist might want to wish for in one line of code!\n - 1000 + pre-trained models\n - 100+ of the latest NLP word embeddings ( BERT, ELMO, ALBERT, XLNET, GLOVE, BIOBERT, ELECTRA, COVIDBERT) and different variations of them\n - 50+ of the latest NLP sentence embeddings ( BERT, ELECTRA, USE) and different variations of them\n - 100+ Classifiers (NER, POS, Emotion, Sarcasm, Questions, Spam)\n - 300+ Supported Languages\n- Summarize Text and Answer Questions with T5\n- Labeled and Unlabeled Dependency parsing\n - Various Text Cleaning and Pre-Processing methods like Stemming, Lemmatizing, Normalizing, Filtering, Cleaning pipelines and more\n\n\n## Classifiers trained on many different datasets \nChoose the right tool for the right task! Whether you analyze movies or twitter, NLU has the right model for you! \n\n- trec6 classifier \n- trec10 classifier \n- spam classifier \n- fake news classifier \n- emotion classifier \n- cyberbullying classifier \n- sarcasm classifier \n- sentiment classifier for movies \n- IMDB Movie Sentiment classifier \n- Twitter sentiment classifier \n- NER pretrained on ONTO notes \n- NER trainer on CONLL \n- Language classifier for 20 languages on the wiki 20 lang dataset. \n\n## Utilities for the Data Science NLU applications \nWorking with text data can sometimes be quite a dirty job. NLU helps you keep your hands clean by providing components that take away from data engineering intensive tasks. \n\n- Datetime Matcher\n- Pattern Matcher\n- Chunk Matcher\n- Phrases Matcher\n- Stopword Cleaners\n- Pattern Cleaners\n- Slang Cleaner \n\n## Where can I see all models available in NLU? \nFor NLU models to load, see [the NLU Namespace](https:\u002F\u002Fnlu.johnsnowlabs.com\u002Fdocs\u002Fen\u002Fnamespace) or the [John Snow Labs Modelshub](https:\u002F\u002Fmodelshub.johnsnowlabs.com\u002Fmodels) or go [straight to the source](https:\u002F\u002Fgithub.com\u002FJohnSnowLabs\u002Fnlu\u002Fblob\u002Fmaster\u002Fnlu\u002Fnamespace.py).\n\n## Supported Data Types\n- Pandas DataFrame and Series\n- Spark DataFrames\n- Modin with Ray backend\n- Modin with Dask backend\n- Numpy arrays\n- Strings and lists of strings \n\n## Overview of all tutorials using the NLU-Library\n\nIn the following tabular, all available tutorials using NLU are listed. These tutorials will help you learn the \nusage of the NLU library and on how to use it for your own tasks. Some of the tasks NLU does are\ntranslating from any language to the english language, lemmatizing, tokenizing, cleaning text from \nSymbol or unwanted syntax, spellchecking, detecting entities, analyzing sentiments and many more!\n\n{:.table2}\n\n| Tutorial Description | NLU Spells Used |Open In Colab | Dataset and Paper References |\n|-----------------------------------------------------------------------------------------------------|---------------------------------------------------------------------------------------------------------------------------|----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|\n| Albert Word Embeddings with NLU | `albert`, `sentiment pos albert emotion` |[![Open In Colab](https:\u002F\u002Fcolab.research.google.com\u002Fassets\u002Fcolab-badge.svg)](https:\u002F\u002Fcolab.research.google.com\u002Fgithub\u002FJohnSnowLabs\u002Fnlu\u002Fblob\u002Fmaster\u002Fexamples\u002Fcolab\u002Fcomponent_examples\u002Fword_embeddings\u002FNLU_ALBERT_word_embeddings_and_t-SNE_visualization_example.ipynb) | [Albert-Paper](https:\u002F\u002Farxiv.org\u002Fpdf\u002F1909.11942.pdf), [Albert on Github](https:\u002F\u002Fgithub.com\u002Fgoogle-research\u002FALBERT), [Albert on TensorFlow](https:\u002F\u002Ftfhub.dev\u002Fs?q=albert), [T-SNE](https:\u002F\u002Fwww.jmlr.org\u002Fpapers\u002Fvolume9\u002Fvandermaaten08a\u002Fvandermaaten08a.pdf?fbclid=IwA), [T-SNE-Albert](https:\u002F\u002Fmedium.com\u002Fspark-nlp\u002F1-line-to-albert-word-embeddings-with-nlu-in-python-1691bc048ed1), [Albert_Embedding](https:\u002F\u002Fnlp.johnsnowlabs.com\u002F2021\u002F06\u002F23\u002Falbert_base_uncased_en.html) | \n| Bert Word Embeddings with NLU | `bert`, `pos sentiment emotion bert` |[![Open In Colab](https:\u002F\u002Fcolab.research.google.com\u002Fassets\u002Fcolab-badge.svg)](https:\u002F\u002Fcolab.research.google.com\u002Fgithub\u002FJohnSnowLabs\u002Fnlu\u002Fblob\u002Fmaster\u002Fexamples\u002Fcolab\u002Fcomponent_examples\u002Fword_embeddings\u002FNLU_BERT_word_embeddings_and_t-SNE_visualization_example.ipynb) | [Bert-Paper](https:\u002F\u002Farxiv.org\u002Fabs\u002F1810.04805), [Bert Github](https:\u002F\u002Fgithub.com\u002Fgoogle-research\u002Fbert), [T-SNE](https:\u002F\u002Fwww.jmlr.org\u002Fpapers\u002Fvolume9\u002Fvandermaaten08a\u002Fvandermaaten08a.pdf?fbclid=IwA), [T-SNE-Bert](https:\u002F\u002Fmedium.com\u002Fspark-nlp\u002F1-line-to-bert-word-embeddings-with-nlu-f50d2b08cddc), [Bert_Embedding](https:\u002F\u002Fnlp.johnsnowlabs.com\u002F2020\u002F08\u002F25\u002Fsent_small_bert_L2_128.html) |\n| BIOBERT Word Embeddings with NLU | `biobert` , `sentiment pos biobert emotion` |[![Open In Colab](https:\u002F\u002Fcolab.research.google.com\u002Fassets\u002Fcolab-badge.svg)](https:\u002F\u002Fcolab.research.google.com\u002Fgithub\u002FJohnSnowLabs\u002Fnlu\u002Fblob\u002Fmaster\u002Fexamples\u002Fcolab\u002Fcomponent_examples\u002Fword_embeddings\u002FNLU_BIOBERT_word_embeddings_and_t-SNE_visualization_example.ipynb) | [BioBert-Paper](https:\u002F\u002Farxiv.org\u002Fabs\u002F1901.08746), [Bert Github](https:\u002F\u002Fgithub.com\u002Fgoogle-research\u002Fbert) , [BERT: Deep Bidirectional Transformers](https:\u002F\u002Farxiv.org\u002Fabs\u002F1810.04805), [Bert Github](https:\u002F\u002Fgithub.com\u002Fgoogle-research\u002Fbert), [T-SNE](https:\u002F\u002Fwww.jmlr.org\u002Fpapers\u002Fvolume9\u002Fvandermaaten08a\u002Fvandermaaten08a.pdf?fbclid=IwA), [T-SNE-Biobert](https:\u002F\u002Fmedium.com\u002Fspark-nlp\u002F1-line-to-biobert-word-embeddings-with-nlu-in-python-7224ab52e131), [Biobert_Embedding](https:\u002F\u002Fnlp.johnsnowlabs.com\u002F2020\u002F08\u002F25\u002Fbiobert_pubmed_base_cased.html) |\n| COVIDBERT Word Embeddings with NLU | `covidbert`, `sentiment covidbert pos` |[![Open In Colab](https:\u002F\u002Fcolab.research.google.com\u002Fassets\u002Fcolab-badge.svg)](https:\u002F\u002Fcolab.research.google.com\u002Fgithub\u002FJohnSnowLabs\u002Fnlu\u002Fblob\u002Fmaster\u002Fexamples\u002Fcolab\u002Fcomponent_examples\u002Fword_embeddings\u002FNLU_COVIDBERT_word_embeddings_and_t-SNE_visualization_example.ipynb) | [CovidBert-Paper](https:\u002F\u002Fjournals.flvc.org\u002FFLAIRS\u002Farticle\u002Fview\u002F128488), [Bert Github](https:\u002F\u002Fgithub.com\u002Fgoogle-research\u002Fbert), [T-SNE](https:\u002F\u002Fwww.jmlr.org\u002Fpapers\u002Fvolume9\u002Fvandermaaten08a\u002Fvandermaaten08a.pdf?fbclid=IwA), [T-SNE-CovidBert](https:\u002F\u002Fmedium.com\u002Fspark-nlp\u002F1-line-to-covidbert-word-embeddings-with-nlu-in-python-e67396da2f78), [Covidbert_Embedding](https:\u002F\u002Fnlp.johnsnowlabs.com\u002F2020\u002F08\u002F27\u002Fcovidbert_large_uncased.html) |\n| ELECTRA Word Embeddings with NLU | `electra`, `sentiment pos en.embed.electra emotion` |[![Open In Colab](https:\u002F\u002Fcolab.research.google.com\u002Fassets\u002Fcolab-badge.svg)](https:\u002F\u002Fcolab.research.google.com\u002Fgithub\u002FJohnSnowLabs\u002Fnlu\u002Fblob\u002Fmaster\u002Fexamples\u002Fcolab\u002Fcomponent_examples\u002Fword_embeddings\u002FNLU_ELECTRA_word_embeddings_and_t-SNE_visualization_example.ipynb) | [Electra-Paper](https:\u002F\u002Farxiv.org\u002Fabs\u002F2003.10555), [T-SNE](https:\u002F\u002Fwww.jmlr.org\u002Fpapers\u002Fvolume9\u002Fvandermaaten08a\u002Fvandermaaten08a.pdf?fbclid=IwA), [T-SNE-Electra](https:\u002F\u002Fmedium.com\u002Fspark-nlp\u002F1-line-to-electra-word-embeddings-with-nlu-in-python-25f749bf3e92), [Electra_Embedding](https:\u002F\u002Fnlp.johnsnowlabs.com\u002F2020\u002F08\u002F27\u002Felectra_small_uncased.html) |\n| ELMO Word Embeddings with NLU | `elmo`, `sentiment pos elmo emotion` |[![Open In Colab](https:\u002F\u002Fcolab.research.google.com\u002Fassets\u002Fcolab-badge.svg)](https:\u002F\u002Fcolab.research.google.com\u002Fgithub\u002FJohnSnowLabs\u002Fnlu\u002Fblob\u002Fmaster\u002Fexamples\u002Fcolab\u002Fcomponent_examples\u002Fword_embeddings\u002FNLU_ELMo_word_embeddings_and_t-SNE_visualization_example.ipynb) | [ELMO-Paper](https:\u002F\u002Farxiv.org\u002Fabs\u002F1802.05365), [Elmo-TensorFlow](https:\u002F\u002Fgithub.com\u002Fgoogle-research\u002Fbert), [T-SNE](https:\u002F\u002Fwww.jmlr.org\u002Fpapers\u002Fvolume9\u002Fvandermaaten08a\u002Fvandermaaten08a.pdf?fbclid=IwA), [T-SNE-Elmo](https:\u002F\u002Fmedium.com\u002Fspark-nlp\u002F1-python-line-for-elmo-word-embeddings-with-john-snow-labs-nlu-628e9b924a3), [Elmo-Embedding](https:\u002F\u002Fnlp.johnsnowlabs.com\u002F2020\u002F01\u002F31\u002Felmo.html) |\n| GLOVE Word Embeddings with NLU | `glove`, `sentiment pos glove emotion` |[![Open In Colab](https:\u002F\u002Fcolab.research.google.com\u002Fassets\u002Fcolab-badge.svg)](https:\u002F\u002Fcolab.research.google.com\u002Fgithub\u002FJohnSnowLabs\u002Fnlu\u002Fblob\u002Fmaster\u002Fexamples\u002Fcolab\u002Fcomponent_examples\u002Fword_embeddings\u002FNLU_GLOVE_word_embeddings_and_t-SNE_visualization_example.ipynb) | [Glove-Paper](https:\u002F\u002Fnlp.stanford.edu\u002Fpubs\u002Fglove.pdf), [T-SNE](https:\u002F\u002Fwww.jmlr.org\u002Fpapers\u002Fvolume9\u002Fvandermaaten08a\u002Fvandermaaten08a.pdf?fbclid=IwA), [T-SNE-Glove](https:\u002F\u002Fmedium.com\u002Fspark-nlp\u002F1-line-to-glove-word-embeddings-with-nlu-in-python-baed152fff4d) , [Glove_Embedding](https:\u002F\u002Fnlp.johnsnowlabs.com\u002F2020\u002F01\u002F22\u002Fglove_100d.html) |\n| XLNET Word Embeddings with NLU | `xlnet`, `sentiment pos xlnet emotion` |[![Open In Colab](https:\u002F\u002Fcolab.research.google.com\u002Fassets\u002Fcolab-badge.svg)](https:\u002F\u002Fcolab.research.google.com\u002Fgithub\u002FJohnSnowLabs\u002Fnlu\u002Fblob\u002Fmaster\u002Fexamples\u002Fcolab\u002Fcomponent_examples\u002Fword_embeddings\u002FNLU_XLNET_word_embeddings_and_t-SNE_visualization_example.ipynb) | [XLNet-Paper](https:\u002F\u002Farxiv.org\u002Fabs\u002F1906.08237), [Bert Github](https:\u002F\u002Fgithub.com\u002Fzihangdai\u002Fxlnet), [T-SNE](https:\u002F\u002Fwww.jmlr.org\u002Fpapers\u002Fvolume9\u002Fvandermaaten08a\u002Fvandermaaten08a.pdf?fbclid=IwA), [T-SNE-XLNet](https:\u002F\u002Fmedium.com\u002Fspark-nlp\u002F1-line-to-xlnet-word-embeddings-with-nlu-in-python-5efc57d7ac79), [Xlnet_Embedding](https:\u002F\u002Fnlp.johnsnowlabs.com\u002F2021\u002F07\u002F07\u002Fxlnet_base_cased_en.html) |\n| Multiple Word-Embeddings and Part of Speech in 1 Line of code | `bert electra elmo glove xlnet albert pos` |[![Open In Colab](https:\u002F\u002Fcolab.research.google.com\u002Fassets\u002Fcolab-badge.svg)](https:\u002F\u002Fcolab.research.google.com\u002Fgithub\u002FJohnSnowLabs\u002Fnlu\u002Fblob\u002Fmaster\u002Fexamples\u002Fcolab\u002Fcomponent_examples\u002Fword_embeddings\u002FNLU_multiple_word_embeddings_and_t-SNE_visualization_example.ipynb) | [Bert-Paper](https:\u002F\u002Farxiv.org\u002Fpdf\u002F1810.04805.pdf), [Albert-Paper](https:\u002F\u002Fopenreview.net\u002Fforum?id=H1eA7AEtvS), [ELMO-Paper](https:\u002F\u002Farxiv.org\u002Fabs\u002F1802.05365), [Electra-Paper](https:\u002F\u002Farxiv.org\u002Fabs\u002F2003.10555), [XLNet-Paper](https:\u002F\u002Farxiv.org\u002Fpdf\u002F1906.08237.pdf), [Glove-Paper](https:\u002F\u002Fnlp.stanford.edu\u002Fpubs\u002Fglove.pdf) |\n| Normalzing with NLU | `norm` |[![Open In Colab](https:\u002F\u002Fcolab.research.google.com\u002Fassets\u002Fcolab-badge.svg)](https:\u002F\u002Fcolab.research.google.com\u002Fgithub\u002FJohnSnowLabs\u002Fnlu\u002Fblob\u002Fmaster\u002Fexamples\u002Fcolab\u002Fcomponent_examples\u002Ftext_pre_processing_and_cleaning\u002FNLU_normalizer_example.ipynb) | - |\n| Detect sentences with NLU | `sentence_detector.deep`, `sentence_detector.pragmatic`, `xx.sentence_detector` |[![Open In Colab](https:\u002F\u002Fcolab.research.google.com\u002Fassets\u002Fcolab-badge.svg)](https:\u002F\u002Fcolab.research.google.com\u002Fgithub\u002FJohnSnowLabs\u002Fnlu\u002Fblob\u002Fmaster\u002Fexamples\u002Fcolab\u002Fcomponent_examples\u002Ftext_pre_processing_and_cleaning\u002FNLU_sentence_detection_example.ipynb) | [Sentence Detector](https:\u002F\u002Fnlp.johnsnowlabs.com\u002F2020\u002F09\u002F13\u002Fsentence_detector_dl_en.html) |\n| Spellchecking with NLU | n.a. | n.a. | - |\n| Stemming with NLU | `en.stem`, `de.stem` |[![Open In Colab](https:\u002F\u002Fcolab.research.google.com\u002Fassets\u002Fcolab-badge.svg)](https:\u002F\u002Fcolab.research.google.com\u002Fgithub\u002FJohnSnowLabs\u002Fnlu\u002Fblob\u002Fmaster\u002Fexamples\u002Fcolab\u002Fcomponent_examples\u002Ftext_pre_processing_and_cleaning\u002FNLU_stemmer_example.ipynb) | - |\n| Stopwords removal with NLU | `stopwords` |[![Open In Colab](https:\u002F\u002Fcolab.research.google.com\u002Fassets\u002Fcolab-badge.svg)](https:\u002F\u002Fcolab.research.google.com\u002Fgithub\u002FJohnSnowLabs\u002Fnlu\u002Fblob\u002Fmaster\u002Fexamples\u002Fcolab\u002Fcomponent_examples\u002Ftext_pre_processing_and_cleaning\u002FNLU_stopwords_removal_example.ipynb) | [Stopwords](https:\u002F\u002Fnlp.johnsnowlabs.com\u002F2020\u002F07\u002F14\u002Fstopwords_en.html) |\n| Tokenization with NLU | `tokenize` |[![Open In Colab](https:\u002F\u002Fcolab.research.google.com\u002Fassets\u002Fcolab-badge.svg)](https:\u002F\u002Fcolab.research.google.com\u002Fgithub\u002FJohnSnowLabs\u002Fnlu\u002Fblob\u002Fmaster\u002Fexamples\u002Fcolab\u002Fcomponent_examples\u002Ftext_pre_processing_and_cleaning\u002FNLU_tokenization_example.ipynb) | - |\n| Normalization of Documents | `norm_document` |[![Open In Colab](https:\u002F\u002Fcolab.research.google.com\u002Fassets\u002Fcolab-badge.svg)](https:\u002F\u002Fcolab.research.google.com\u002Fgithub\u002FJohnSnowLabs\u002Fnlu\u002Fblob\u002Fmaster\u002Fexamples\u002Fcolab\u002Fcomponent_examples\u002Ftext_pre_processing_and_cleaning\u002Fdocument_normalizer_demo.ipynb) | - |\n| Open and Closed book question answering with Google's T5 | `en.t5` , `answer_question` |[![Open In Colab](https:\u002F\u002Fcolab.research.google.com\u002Fassets\u002Fcolab-badge.svg)](https:\u002F\u002Fcolab.research.google.com\u002Fgithub\u002FJohnSnowLabs\u002Fnlu\u002Fblob\u002Fmaster\u002Fexamples\u002Fcolab\u002Fcomponent_examples\u002Fsequence2sequence\u002FT5_question_answering.ipynb) | [T5-Paper](https:\u002F\u002Farxiv.org\u002Fpdf\u002F1910.10683.pdf), [T5-Model](https:\u002F\u002Fnlp.johnsnowlabs.com\u002F2021\u002F01\u002F08\u002Ft5_base_en.html) | \n| Overview of every task available with T5 | `en.t5.base` |[![Open In Colab](https:\u002F\u002Fcolab.research.google.com\u002Fassets\u002Fcolab-badge.svg)](https:\u002F\u002Fcolab.research.google.com\u002Fgithub\u002FJohnSnowLabs\u002Fnlu\u002Fblob\u002Fmaster\u002Fexamples\u002Fcolab\u002Fcomponent_examples\u002Fsequence2sequence\u002FT5_tasks_summarize_question_answering_and_more.ipynb) | [T5-Paper](https:\u002F\u002Farxiv.org\u002Fpdf\u002F1910.10683.pdf), [T5-Model](https:\u002F\u002Fnlp.johnsnowlabs.com\u002F2021\u002F01\u002F08\u002Ft5_base_en.html) | \n| Translate between more than 200 Languages in 1 line of code with Marian Models | `tr.translate_to.fr`, `en.translate_to.fr` ,`fr.translate_to.he` , `en.translate_to.de` |[![Open In Colab](https:\u002F\u002Fcolab.research.google.com\u002Fassets\u002Fcolab-badge.svg)](https:\u002F\u002Fcolab.research.google.com\u002Fgithub\u002FJohnSnowLabs\u002Fnlu\u002Fblob\u002Fmaster\u002Fexamples\u002Fcolab\u002Fcomponent_examples\u002Fsequence2sequence\u002Ftranslation_demo.ipynb) | [Marian-Papers](https:\u002F\u002Fmarian-nmt.github.io\u002Fpublications\u002F), [Translation-Pipeline (En to Fr)](https:\u002F\u002Fnlp.johnsnowlabs.com\u002F2021\u002F06\u002F04\u002Ftranslate_en_fr_xx.html), [Translation-Pipeline (En to Ger)](https:\u002F\u002Fnlp.johnsnowlabs.com\u002F2021\u002F06\u002F04\u002Ftranslate_en_de_xx.html) |\n| BERT Sentence Embeddings with NLU | `embed_sentence.bert`, `pos sentiment embed_sentence.bert` |[![Open In Colab](https:\u002F\u002Fcolab.research.google.com\u002Fassets\u002Fcolab-badge.svg)](https:\u002F\u002Fcolab.research.google.com\u002Fgithub\u002FJohnSnowLabs\u002Fnlu\u002Fblob\u002Fmaster\u002Fexamples\u002Fcolab\u002Fcomponent_examples\u002Fsentence_embeddings\u002FNLU_BERT_sentence_embeddings_and_t-SNE_visualization_Example.ipynb) | [Bert-Paper](https:\u002F\u002Farxiv.org\u002Fabs\u002F1810.04805), [Bert Github](https:\u002F\u002Fgithub.com\u002Fgoogle-research\u002Fbert), [Bert-Sentence_Embedding](https:\u002F\u002Fnlp.johnsnowlabs.com\u002F2020\u002F08\u002F25\u002Fsent_small_bert_L2_128.html) | \n| ELECTRA Sentence Embeddings with NLU | `embed_sentence.electra`, `pos sentiment embed_sentence.electra` |[![Open In Colab](https:\u002F\u002Fcolab.research.google.com\u002Fassets\u002Fcolab-badge.svg)](https:\u002F\u002Fcolab.research.google.com\u002Fgithub\u002FJohnSnowLabs\u002Fnlu\u002Fblob\u002Fmaster\u002Fexamples\u002Fcolab\u002Fcomponent_examples\u002Fsentence_embeddings\u002FNLU_ELECTRA_sentence_embeddings_and_t-SNE_visualization_example.ipynb) | [Electra Paper](https:\u002F\u002Farxiv.org\u002Fabs\u002F2003.10555), [Sentence-Electra-Embedding](https:\u002F\u002Fnlp.johnsnowlabs.com\u002F2020\u002F08\u002F27\u002Fsent_electra_small_uncased.html) |\n| USE Sentence Embeddings with NLU | `use`, `pos sentiment use emotion` |[![Open In Colab](https:\u002F\u002Fcolab.research.google.com\u002Fassets\u002Fcolab-badge.svg)](https:\u002F\u002Fcolab.research.google.com\u002Fgithub\u002FJohnSnowLabs\u002Fnlu\u002Fblob\u002Fmaster\u002Fexamples\u002Fcolab\u002Fcomponent_examples\u002Fsentence_embeddings\u002FNLU_USE_sentence_embeddings_and_t-SNE_visualization_example.ipynb) | [Universal Sentence Encoder](https:\u002F\u002Farxiv.org\u002Fabs\u002F1803.11175), [USE-TensorFlow](https:\u002F\u002Ftfhub.dev\u002Fgoogle\u002Funiversal-sentence-encoder\u002F2), [Sentence-USE-Embedding](https:\u002F\u002Fnlp.johnsnowlabs.com\u002F2020\u002F04\u002F17\u002Ftfhub_use_lg.html) |\n| Sentence similarity with NLU using BERT embeddings | `embed_sentence.bert`, `use en.embed_sentence.electra embed_sentence.bert` |[![Open In Colab](https:\u002F\u002Fcolab.research.google.com\u002Fassets\u002Fcolab-badge.svg)](https:\u002F\u002Fcolab.research.google.com\u002Fgithub\u002FJohnSnowLabs\u002Fnlu\u002Fblob\u002Fmaster\u002Fexamples\u002Fcolab\u002Fcomponent_examples\u002Fsentence_embeddings\u002Fsentence_similarirty_stack_overflow_questions.ipynb) | [Bert-Paper](https:\u002F\u002Farxiv.org\u002Fabs\u002F1810.04805), [Bert Github](https:\u002F\u002Fgithub.com\u002Fgoogle-research\u002Fbert), [Bert-Sentence_Embedding](https:\u002F\u002Fnlp.johnsnowlabs.com\u002F2020\u002F08\u002F25\u002Fsent_small_bert_L2_128.html) |\n| Part of Speech tagging with NLU | `pos` |[![Open In Colab](https:\u002F\u002Fcolab.research.google.com\u002Fassets\u002Fcolab-badge.svg)](https:\u002F\u002Fcolab.research.google.com\u002Fgithub\u002FMurat-Karadag\u002Fnlu\u002Fblob\u002Ftutorial_docs\u002Fexamples\u002Fcolab\u002Fcomponent_examples\u002Fpart_of_speechPOS\u002FNLU_part_of_speech_ANC_example.ipynb) | [Part of Speech](https:\u002F\u002Fnlp.johnsnowlabs.com\u002F2021\u002F03\u002F05\u002Fpos_anc.html) | \n| NER Aspect Airline ATIS | `en.ner.aspect.airline` |[![Open In Colab](https:\u002F\u002Fcolab.research.google.com\u002Fassets\u002Fcolab-badge.svg)](https:\u002F\u002Fcolab.research.google.com\u002Fgithub\u002FMurat-Karadag\u002Fnlu\u002Fblob\u002Ftutorial_docs\u002Fexamples\u002Fcolab\u002Fcomponent_examples\u002Fnamed_entity_recognition_NER\u002FNER_aspect_airline_ATIS.ipynb) | [NER Airline Model](https:\u002F\u002Fnlp.johnsnowlabs.com\u002F2021\u002F01\u002F25\u002Fnerdl_atis_840b_300d_en.html), [Atis intent Dataset](https:\u002F\u002Fwww.kaggle.com\u002Fhassanamin\u002Fatis-airlinetravelinformationsystem) | \n| NLU-NER_CONLL_2003_5class_example | `ner` |[![Open In Colab](https:\u002F\u002Fcolab.research.google.com\u002Fassets\u002Fcolab-badge.svg)](https:\u002F\u002Fcolab.research.google.com\u002Fgithub\u002FMurat-Karadag\u002Fnlu\u002Fblob\u002Ftutorial_docs\u002Fexamples\u002Fcolab\u002Fcomponent_examples\u002Fnamed_entity_recognition_NER\u002FNLU_ner_CONLL_2003_5class_example.ipynb) | [NER-Piple](https:\u002F\u002Fnlp.johnsnowlabs.com\u002F2021\u002F03\u002F22\u002Fonto_recognize_entities_sm_en.html) |\n| Named-entity recognition with Deep Learning ONTO NOTES | `ner.onto` |[![Open In Colab](https:\u002F\u002Fcolab.research.google.com\u002Fassets\u002Fcolab-badge.svg)](https:\u002F\u002Fcolab.research.google.com\u002Fgithub\u002FMurat-Karadag\u002Fnlu\u002Fblob\u002Ftutorial_docs\u002Fexamples\u002Fcolab\u002Fcomponent_examples\u002Fnamed_entity_recognition_NER\u002FNLU_ner_ONTO_18class_example.ipynb) | [NER_Onto](https:\u002F\u002Fnlp.johnsnowlabs.com\u002F2021\u002F03\u002F22\u002Fonto_recognize_entities_sm_en.html) |\n| Aspect based NER-Sentiment-Restaurants | `en.ner.aspect_sentiment` |[![Open In Colab](https:\u002F\u002Fcolab.research.google.com\u002Fassets\u002Fcolab-badge.svg)](https:\u002F\u002Fcolab.research.google.com\u002Fgithub\u002FMurat-Karadag\u002Fnlu\u002Fblob\u002Ftutorial_docs\u002Fexamples\u002Fcolab\u002Fcomponent_examples\u002Fnamed_entity_recognition_NER\u002Faspect_based_ner_sentiment_restaurants.ipynb) | - |\n| Detect Named Entities (NER), Part of Speech Tags (POS) and Tokenize in Chinese | `zh.segment_words`, `zh.pos`, `zh.ner`, `zh.translate_to.en` | [![Open In Colab](https:\u002F\u002Fcolab.research.google.com\u002Fassets\u002Fcolab-badge.svg)](https:\u002F\u002Fcolab.research.google.com\u002Fgithub\u002FJohnSnowLabs\u002Fnlu\u002Fblob\u002Fmaster\u002Fexamples\u002Fcolab\u002Fcomponent_examples\u002Fmultilingual\u002Fchinese_ner_pos_and_tokenization.ipynb) | [Translation-Pipeline (Zh to En)](https:\u002F\u002Fnlp.johnsnowlabs.com\u002F2021\u002F06\u002F04\u002Ftranslate_zh_en_xx.html) |\n| Detect Named Entities (NER), Part of Speech Tags (POS) and Tokenize in Japanese | `ja.segment_words`, `ja.pos`, `ja.ner`, `ja.translate_to.en` | [![Open In Colab](https:\u002F\u002Fcolab.research.google.com\u002Fassets\u002Fcolab-badge.svg)](https:\u002F\u002Fcolab.research.google.com\u002Fgithub\u002FJohnSnowLabs\u002Fnlu\u002Fblob\u002Fmaster\u002Fexamples\u002Fcolab\u002Fcomponent_examples\u002Fmultilingual\u002Fjapanese_ner_pos_and_tokenization.ipynb) | [Translation-Pipeline (Ja to En)](https:\u002F\u002Fnlp.johnsnowlabs.com\u002F2021\u002F06\u002F04\u002Ftranslate_ja_en_xx.html) |\n| Detect Named Entities (NER), Part of Speech Tags (POS) and Tokenize in Korean | `ko.segment_words`, `ko.pos`, `ko.ner.kmou.glove_840B_300d`, `ko.translate_to.en` | [![Open In Colab](https:\u002F\u002Fcolab.research.google.com\u002Fassets\u002Fcolab-badge.svg)](https:\u002F\u002Fcolab.research.google.com\u002Fgithub\u002FMurat-Karadag\u002Fnlu\u002Fblob\u002Fmaster\u002Fexamples\u002Fcolab\u002Fcomponent_examples\u002Fmultilingual\u002Fkorean_ner_pos_and_tokenization.ipynb) | - |\n| Date Matching | `match.datetime` | [![Open In Colab](https:\u002F\u002Fcolab.research.google.com\u002Fassets\u002Fcolab-badge.svg)](https:\u002F\u002Fcolab.research.google.com\u002Fgithub\u002FJohnSnowLabs\u002Fnlu\u002Fblob\u002Fmaster\u002Fexamples\u002Fcolab\u002Fcomponent_examples\u002Fmatchers\u002FNLU_date_matching.ipynb) | - |\n| Typed Dependency Parsing with NLU | `dep` | [![Open In Colab](https:\u002F\u002Fcolab.research.google.com\u002Fassets\u002Fcolab-badge.svg)](https:\u002F\u002Fcolab.research.google.com\u002Fgithub\u002FJohnSnowLabs\u002Fnlu\u002Fblob\u002Fmaster\u002Fexamples\u002Fcolab\u002Fcomponent_examples\u002Fdependency_parsing\u002FNLU_typed_dependency_parsing_example.ipynb) | [Dependency Parsing ](https:\u002F\u002Fnlp.johnsnowlabs.com\u002F2021\u002F03\u002F27\u002FTyped_Dependency_Parsing_en.html) |\n| Untyped Dependency Parsing with NLU | `dep.untyped` | [![Open In Colab](https:\u002F\u002Fcolab.research.google.com\u002Fassets\u002Fcolab-badge.svg)](https:\u002F\u002Fcolab.research.google.com\u002Fgithub\u002FJohnSnowLabs\u002Fnlu\u002Fblob\u002Fmaster\u002Fexamples\u002Fcolab\u002Fcomponent_examples\u002Fdependency_parsing\u002FNLU_untyped_dependency_parsing_example.ipynb) | - |\n| E2E Classification with NLU | `e2e` | [![Open In Colab](https:\u002F\u002Fcolab.research.google.com\u002Fassets\u002Fcolab-badge.svg)](https:\u002F\u002Fcolab.research.google.com\u002Fgithub\u002FJohnSnowLabs\u002Fnlu\u002Fblob\u002Fmaster\u002Fexamples\u002Fcolab\u002Fcomponent_examples\u002Fclassifiers\u002FE2E_classification.ipynb) | [e2e-Model](https:\u002F\u002Fnlp.johnsnowlabs.com\u002F2021\u002F01\u002F21\u002Fmulticlassifierdl_use_e2e_en.html) |\n| Language Classification with NLU | `lang` | [![Open In Colab](https:\u002F\u002Fcolab.research.google.com\u002Fassets\u002Fcolab-badge.svg)](https:\u002F\u002Fcolab.research.google.com\u002Fgithub\u002FJohnSnowLabs\u002Fnlu\u002Fblob\u002Fmaster\u002Fexamples\u002Fcolab\u002Fcomponent_examples\u002Fclassifiers\u002FNLU_language_classification.ipynb) | - |\n| Cyberbullying Classification with NLU | `classify.cyberbullying` | [![Open In Colab](https:\u002F\u002Fcolab.research.google.com\u002Fassets\u002Fcolab-badge.svg)](https:\u002F\u002Fcolab.research.google.com\u002Fgithub\u002FJohnSnowLabs\u002Fnlu\u002Fblob\u002Fmaster\u002Fexamples\u002Fcolab\u002Fcomponent_examples\u002Fclassifiers\u002Fcyberbullying_cassification_for_racism_and_sexism.ipynb) | [Cyberbullying-Classifier](https:\u002F\u002Fnlp.johnsnowlabs.com\u002F2021\u002F01\u002F09\u002Fclassifierdl_use_cyberbullying_en.html) |\n| Sentiment Classification with NLU for Twitter | `emotion` | [![Open In Colab](https:\u002F\u002Fcolab.research.google.com\u002Fassets\u002Fcolab-badge.svg)](https:\u002F\u002Fcolab.research.google.com\u002Fgithub\u002FJohnSnowLabs\u002Fnlu\u002Fblob\u002Fmaster\u002Fexamples\u002Fcolab\u002Fcomponent_examples\u002Fclassifiers\u002Femotion_classification.ipynb) | [Emotion detection](https:\u002F\u002Fnlp.johnsnowlabs.com\u002F2021\u002F01\u002F09\u002Fclassifierdl_use_emotion_en.html) |\n| Fake News Classification with NLU | `en.classify.fakenews` | [![Open In Colab](https:\u002F\u002Fcolab.research.google.com\u002Fassets\u002Fcolab-badge.svg)](https:\u002F\u002Fcolab.research.google.com\u002Fgithub\u002FJohnSnowLabs\u002Fnlu\u002Fblob\u002Fmaster\u002Fexamples\u002Fcolab\u002Fcomponent_examples\u002Fclassifiers\u002Ffake_news_classification.ipynb) | [Fakenews-Classifier](https:\u002F\u002Fnlp.johnsnowlabs.com\u002F2021\u002F01\u002F09\u002Fclassifierdl_use_fakenews_en.html) |\n| Intent Classification with NLU | `en.classify.intent.airline` | [![Open In Colab](https:\u002F\u002Fcolab.research.google.com\u002Fassets\u002Fcolab-badge.svg)](https:\u002F\u002Fcolab.research.google.com\u002Fgithub\u002FJohnSnowLabs\u002Fnlu\u002Fblob\u002Fmaster\u002Fexamples\u002Fcolab\u002Fcomponent_examples\u002Fclassifiers\u002Fintent_classification_airlines_ATIS.ipynb) | [Airline-Intention classifier](https:\u002F\u002Fnlp.johnsnowlabs.com\u002F2021\u002F01\u002F25\u002Fclassifierdl_use_atis_en.html), [Atis-Dataset](https:\u002F\u002Fwww.kaggle.com\u002Fhassanamin\u002Fatis-airlinetravelinformationsystem?select=atis_intents.csv) |\n| Question classification based on the TREC dataset | `en.classify.questions` | [![Open In Colab](https:\u002F\u002Fcolab.research.google.com\u002Fassets\u002Fcolab-badge.svg)](https:\u002F\u002Fcolab.research.google.com\u002Fgithub\u002FJohnSnowLabs\u002Fnlu\u002Fblob\u002Fmaster\u002Fexamples\u002Fcolab\u002Fcomponent_examples\u002Fclassifiers\u002Fquestion_classification.ipynb) | [Question-Classifier](https:\u002F\u002Fnlp.johnsnowlabs.com\u002F2021\u002F01\u002F08\u002Fclassifierdl_use_trec50_en.html) |\n| Sarcasm Classification with NLU | `en.classify.sarcasm` | [![Open In Colab](https:\u002F\u002Fcolab.research.google.com\u002Fassets\u002Fcolab-badge.svg)](https:\u002F\u002Fcolab.research.google.com\u002Fgithub\u002FJohnSnowLabs\u002Fnlu\u002Fblob\u002Fmaster\u002Fexamples\u002Fcolab\u002Fcomponent_examples\u002Fclassifiers\u002Fsarcasm_classification.ipynb) | [Sarcasm-Classifier](https:\u002F\u002Fnlp.johnsnowlabs.com\u002F2021\u002F01\u002F09\u002Fclassifierdl_use_sarcasm_en.html) |\n| Sentiment Classification with NLU for Twitter | `en.sentiment.twitter` | [![Open In Colab](https:\u002F\u002Fcolab.research.google.com\u002Fassets\u002Fcolab-badge.svg)](https:\u002F\u002Fcolab.research.google.com\u002Fgithub\u002FJohnSnowLabs\u002Fnlu\u002Fblob\u002Fmaster\u002Fexamples\u002Fcolab\u002Fcomponent_examples\u002Fclassifiers\u002Fsentiment_classification.ipynb) | [Sentiment_Twitter-Classifier](https:\u002F\u002Fnlp.johnsnowlabs.com\u002F2021\u002F01\u002F18\u002Fsentimentdl_use_twitter_en.html) |\n| Sentiment Classification with NLU for Movies | `en.sentiment.imdb` | [![Open In Colab](https:\u002F\u002Fcolab.research.google.com\u002Fassets\u002Fcolab-badge.svg)](https:\u002F\u002Fcolab.research.google.com\u002Fgithub\u002FJohnSnowLabs\u002Fnlu\u002Fblob\u002Fmaster\u002Fexamples\u002Fcolab\u002Fcomponent_examples\u002Fclassifiers\u002Fsentiment_classification_movies.ipynb) | [Sentiment_imdb-Classifier](https:\u002F\u002Fnlp.johnsnowlabs.com\u002F2021\u002F01\u002F15\u002Fanalyze_sentimentdl_use_imdb_en.html) |\n| Spam Classification with NLU | `en.classify.spam` | [![Open In Colab](https:\u002F\u002Fcolab.research.google.com\u002Fassets\u002Fcolab-badge.svg)](https:\u002F\u002Fcolab.research.google.com\u002Fgithub\u002FJohnSnowLabs\u002Fnlu\u002Fblob\u002Fmaster\u002Fexamples\u002Fcolab\u002Fcomponent_examples\u002Fclassifiers\u002Fspam_classification.ipynb) | [Spam-Classifier](https:\u002F\u002Fnlp.johnsnowlabs.com\u002F2021\u002F01\u002F09\u002Fclassifierdl_use_spam_en.html) |\n| Toxic text classification with NLU | `en.classify.toxic` | [![Open In Colab](https:\u002F\u002Fcolab.research.google.com\u002Fassets\u002Fcolab-badge.svg)](https:\u002F\u002Fcolab.research.google.com\u002Fgithub\u002FJohnSnowLabs\u002Fnlu\u002Fblob\u002Fmaster\u002Fexamples\u002Fcolab\u002Fcomponent_examples\u002Fclassifiers\u002Ftoxic_classification.ipynb) | [Toxic-Classifier](https:\u002F\u002Fnlp.johnsnowlabs.com\u002F2021\u002F01\u002F21\u002Fmulticlassifierdl_use_toxic_en.html) |\n| Unsupervised keyword extraction with NLU using the YAKE algorithm | `yake` | [![Open In Colab](https:\u002F\u002Fcolab.research.google.com\u002Fassets\u002Fcolab-badge.svg)](https:\u002F\u002Fcolab.research.google.com\u002Fgithub\u002FJohnSnowLabs\u002Fnlu\u002Fblob\u002Fmaster\u002Fexamples\u002Fcolab\u002Fcomponent_examples\u002Fclassifiers\u002Funsupervised_keyword_extraction_with_YAKE.ipynb) | - |\n| Grammatical Chunk Matching with NLU | `match.chunks` | [![Open In Colab](https:\u002F\u002Fcolab.research.google.com\u002Fassets\u002Fcolab-badge.svg)](https:\u002F\u002Fcolab.research.google.com\u002Fgithub\u002FJohnSnowLabs\u002Fnlu\u002Fblob\u002Fmaster\u002Fexamples\u002Fcolab\u002Fcomponent_examples\u002Fchunkers\u002FNLU_chunking_example.ipynb) | - |\n| Getting n-Grams with NLU | `ngram` | [![Open In Colab](https:\u002F\u002Fcolab.research.google.com\u002Fassets\u002Fcolab-badge.svg)](https:\u002F\u002Fcolab.research.google.com\u002Fgithub\u002FJohnSnowLabs\u002Fnlu\u002Fblob\u002Fmaster\u002Fexamples\u002Fcolab\u002Fcomponent_examples\u002Fchunkers\u002FNLU_n-gram.ipynb) | - |\n| Assertion | `en.med_ner.clinical en.assert`, `en.med_ner.clinical.biobert en.assert.biobert`, ... | [![Open In Colab](https:\u002F\u002Fcolab.research.google.com\u002Fassets\u002Fcolab-badge.svg)](https:\u002F\u002Fcolab.research.google.com\u002Fgithub\u002FJohnSnowLabs\u002Fnlu\u002Fblob\u002Fmaster\u002Fexamples\u002Fcolab\u002Fhealthcare\u002Fassertion\u002Fassertion_overview.ipynb) | [Healthcare-NER](https:\u002F\u002Fnlp.johnsnowlabs.com\u002F2021\u002F01\u002F27\u002Fner_clinical_en.html), [NER_Clinical-Classifier]( https:\u002F\u002Fnlp.johnsnowlabs.com\u002F2021\u002F11\u002F03\u002Fner_profiling_biobert_en.html), [Toxic-Classifier](https:\u002F\u002Fnlp.johnsnowlabs.com\u002F2021\u002F01\u002F26\u002Fassertion_dl_biobert_en.html) |\n| De-Identification Model overview | `med_ner.jsl.wip.clinical en.de_identify`, `med_ner.jsl.wip.clinical en.de_identify.clinical`, ... | [![Open In Colab](https:\u002F\u002Fcolab.research.google.com\u002Fassets\u002Fcolab-badge.svg)](https:\u002F\u002Fcolab.research.google.com\u002Fgithub\u002FJohnSnowLabs\u002Fnlu\u002Fblob\u002Fmaster\u002Fexamples\u002Fcolab\u002Fhealthcare\u002Fde_identification\u002FDeIdentification_model_overview.ipynb) | [NER-Clinical](https:\u002F\u002Fnlp.johnsnowlabs.com\u002F2021\u002F11\u002F03\u002Fner_profiling_clinical_en.html) |\n| Drug Normalization | `norm_drugs` | [![Open In Colab](https:\u002F\u002Fcolab.research.google.com\u002Fassets\u002Fcolab-badge.svg)](https:\u002F\u002Fcolab.research.google.com\u002Fgithub\u002FJohnSnowLabs\u002Fnlu\u002Fblob\u002Fmaster\u002Fexamples\u002Fcolab\u002Fhealthcare\u002F\u002Fdrug_normalization\u002Fdrug_norm.ipynb) | - | \n| Entity Resolution | `med_ner.jsl.wip.clinical en.resolve_chunk.cpt_clinical`, `med_ner.jsl.wip.clinical en.resolve.icd10cm`, ... | [![Open In Colab](https:\u002F\u002Fcolab.research.google.com\u002Fassets\u002Fcolab-badge.svg)](https:\u002F\u002Fcolab.research.google.com\u002Fgithub\u002FJohnSnowLabs\u002Fnlu\u002Fblob\u002Fmaster\u002Fexamples\u002Fcolab\u002Fhealthcare\u002Fentity_resolution\u002Fentity_resolvers_overview.ipynb) | [NER-Clinical](https:\u002F\u002Fnlp.johnsnowlabs.com\u002F2021\u002F11\u002F03\u002Fner_profiling_clinical_en.html), [Entity-Resolver clinical](https:\u002F\u002Fnlp.johnsnowlabs.com\u002F2021\u002F11\u002F01\u002Fsbiobertresolve_icd10cm_augmented_billable_hcc_en.html) |\n| Medical Named Entity Recognition | `en.med_ner.ade.clinical`, `en.med_ner.ade.clinical_bert`, `en.med_ner.anatomy`,`en.med_ner.anatomy.biobert`, ... | [![Open In Colab](https:\u002F\u002Fcolab.research.google.com\u002Fassets\u002Fcolab-badge.svg)](https:\u002F\u002Fcolab.research.google.com\u002Fgithub\u002FJohnSnowLabs\u002Fnlu\u002Fblob\u002Fmaster\u002Fexamples\u002Fcolab\u002Fhealthcare\u002Fmedical_named_entity_recognition\u002Foverview_medical_entity_recognizers.ipynb) | - |\n| Relation Extraction | `en.med_ner.jsl.wip.clinical.greedy en.relation`, `en.med_ner.jsl.wip.clinical.greedy en.relation.bodypart.problem`, ... | [![Open In Colab](https:\u002F\u002Fcolab.research.google.com\u002Fassets\u002Fcolab-badge.svg)](https:\u002F\u002Fcolab.research.google.com\u002Fgithub\u002FJohnSnowLabs\u002Fnlu\u002Fblob\u002Fmaster\u002Fexamples\u002Fcolab\u002Fhealthcare\u002Frelation_extraction\u002Foverview_relation.ipynb) | - |\n| Visualization of NLP-Models with Spark-NLP and NLU | `ner`, `dep.typed`, `med_ner.jsl.wip.clinical resolve_chunk.rxnorm.in`, `med_ner.jsl.wip.clinical resolve.icd10cm` | [![Open In Colab](https:\u002F\u002Fcolab.research.google.com\u002Fassets\u002Fcolab-badge.svg)](https:\u002F\u002Fcolab.research.google.com\u002Fgithub\u002FJohnSnowLabs\u002Fnlu\u002Fblob\u002Fmaster\u002Fexamples\u002Fcolab\u002Fvisualization\u002FNLU_visualizations_tutorial.ipynb) | [NER-Piple](https:\u002F\u002Fnlp.johnsnowlabs.com\u002F2021\u002F03\u002F22\u002Fonto_recognize_entities_sm_en.html), [Dependency Parsing](https:\u002F\u002Fnlp.johnsnowlabs.com\u002F2021\u002F03\u002F27\u002FTyped_Dependency_Parsing_en.html), [NER-Clinical](https:\u002F\u002Fnlp.johnsnowlabs.com\u002F2021\u002F11\u002F03\u002Fner_profiling_clinical_en.html), [Entity-Resolver (Chunks) clinical](https:\u002F\u002Fnlp.johnsnowlabs.com\u002F2021\u002F04\u002F16\u002Fchunkresolve_rxnorm_in_clinical_en.html) |\n| NLU Covid-19 Emotion Showcase | `emotion` | [![Open In GitHub]()](https:\u002F\u002Fgithub.com\u002FJohnSnowLabs\u002Fnlu\u002Fblob\u002Fmaster\u002Fexamples\u002Fkaggle\u002Fnlu_covid_emotion_showcase.ipynb) | [Emotion detection](https:\u002F\u002Fnlp.johnsnowlabs.com\u002F2021\u002F01\u002F09\u002Fclassifierdl_use_emotion_en.html) |\n| NLU Covid-19 Sentiment Showcase | `sentiment` | [![Open In GitHub]()](https:\u002F\u002Fgithub.com\u002FJohnSnowLabs\u002Fnlu\u002Fblob\u002Fmaster\u002Fexamples\u002Fkaggle\u002Fnlu_covid_sentiment_showcase.ipynb) | [Sentiment classification](https:\u002F\u002Fnlp.johnsnowlabs.com\u002F2021\u002F03\u002F24\u002Fanalyze_sentiment_en.html) |\n| NLU Airline Emotion Demo | `emotion` | [![Open In GitHub]()](https:\u002F\u002Fgithub.com\u002FJohnSnowLabs\u002Fnlu\u002Fblob\u002Fmaster\u002Fexamples\u002Fkaggle\u002Fnlu_emotion_airline_demo.ipynb) | [Emotion detection](https:\u002F\u002Fnlp.johnsnowlabs.com\u002F2021\u002F01\u002F09\u002Fclassifierdl_use_emotion_en.html) |\n| NLU Airline Sentiment Demo | `sentiment` | [![Open In GitHub]()](https:\u002F\u002Fgithub.com\u002FJohnSnowLabs\u002Fnlu\u002Fblob\u002Fmaster\u002Fexamples\u002Fkaggle\u002Fnlu_sentiment_airline_demo.ipynb) | [Sentiment classification](https:\u002F\u002Fnlp.johnsnowlabs.com\u002F2021\u002F03\u002F24\u002Fanalyze_sentiment_en.html) |\n| Bengali NER Hindi Embeddings for 30 Models | `bn.ner`, `bn.lemma`, `ja.lemma`, `am.lemma`, `bh.lemma`,` en.ner.onto.bert.small_l2_128`,.. | [![Open In Colab](https:\u002F\u002Fcolab.research.google.com\u002Fassets\u002Fcolab-badge.svg)](https:\u002F\u002Fcolab.research.google.com\u002Fgithub\u002FMurat-Karadag\u002Fnlu\u002Fblob\u002Fmaster\u002Fexamples\u002Frelease_notebooks\u002FNLU1.1.2_Bengali_ner_Hindi_Embeddings_30_new_models.ipynb) | [Bengali-NER](https:\u002F\u002Fnlp.johnsnowlabs.com\u002F2021\u002F01\u002F27\u002Fner_jifs_glove_840B_300d_bn.html), [Bengali-Lemmatizer](https:\u002F\u002Fnlp.johnsnowlabs.com\u002F2021\u002F01\u002F20\u002Flemma_bn.html), [Japanese-Lemmatizer](https:\u002F\u002Fnlp.johnsnowlabs.com\u002F2021\u002F01\u002F15\u002Flemma_ja.html), [Amharic-Lemmatizer](https:\u002F\u002Fnlp.johnsnowlabs.com\u002F2021\u002F01\u002F20\u002Flemma_am.html) |\n| Entity Resolution | `med_ner.jsl.wip.clinical en.resolve.umls`, `med_ner.jsl.wip.clinical en.resolve.loinc`, `med_ner.jsl.wip.clinical en.resolve.loinc.biobert` | [![Open In Colab](https:\u002F\u002Fcolab.research.google.com\u002Fassets\u002Fcolab-badge.svg)](https:\u002F\u002Fcolab.research.google.com\u002Fgithub\u002FJohnSnowLabs\u002Fnlu\u002Fblob\u002Fmaster\u002Fexamples\u002Frelease_notebooks\u002FNLU_3_0_2_release_notebook.ipynb) | - |\n| NLU 20 Minutes Crashcourse - the fast Data Science route | `spell`, `sentiment`, `pos`, `ner`, `yake`, `en.t5`, `emotion`, `answer_question`, `en.t5.base` ... | [![Open In Colab](https:\u002F\u002Fcolab.research.google.com\u002Fassets\u002Fcolab-badge.svg)](https:\u002F\u002Fcolab.research.google.com\u002Fgithub\u002FJohnSnowLabs\u002Fnlu\u002Fblob\u002Fmaster\u002Fexamples\u002Fwebinars_conferences_etc\u002FAI4_2021\u002FNLU_crash_course_AI4.ipynb) | [T5-Model](https:\u002F\u002Fnlp.johnsnowlabs.com\u002F2021\u002F01\u002F08\u002Ft5_base_en.html), [Part of Speech](https:\u002F\u002Fnlp.johnsnowlabs.com\u002F2021\u002F03\u002F05\u002Fpos_anc.html), [NER-Piple](https:\u002F\u002Fnlp.johnsnowlabs.com\u002F2021\u002F03\u002F22\u002Fonto_recognize_entities_sm_en.html), [Emotion detection](https:\u002F\u002Fnlp.johnsnowlabs.com\u002F2021\u002F01\u002F09\u002Fclassifierdl_use_emotion_en.html) , [Spellchecker](https:\u002F\u002Fnlp.johnsnowlabs.com\u002F2021\u002F03\u002F28\u002Fspellcheck_dl_en.html), [Sentiment classification](https:\u002F\u002Fnlp.johnsnowlabs.com\u002F2021\u002F03\u002F24\u002Fanalyze_sentiment_en.html) |\n| Chapter 0: Intro: 1-liners | `sentiment`, `pos`, `ner`, `bert`, `elmo`, `embed_sentence.bert` | [![Open In Colab](https:\u002F\u002Fcolab.research.google.com\u002Fassets\u002Fcolab-badge.svg)](https:\u002F\u002Fcolab.research.google.com\u002Fgithub\u002FJohnSnowLabs\u002Fnlu\u002Fblob\u002Fmaster\u002Fexamples\u002Fwebinars_conferences_etc\u002FNYC_DC_NLP_MEETUP\u002F0_liners_intro.ipynb) | [Part of Speech](https:\u002F\u002Fnlp.johnsnowlabs.com\u002F2021\u002F03\u002F05\u002Fpos_anc.html), [NER-Piple](https:\u002F\u002Fnlp.johnsnowlabs.com\u002F2021\u002F03\u002F22\u002Fonto_recognize_entities_sm_en.html), [Sentiment classification](https:\u002F\u002Fnlp.johnsnowlabs.com\u002F2021\u002F03\u002F24\u002Fanalyze_sentiment_en.html), [Elmo-Embedding](https:\u002F\u002Fnlp.johnsnowlabs.com\u002F2020\u002F01\u002F31\u002Felmo.html), [Bert-Sentence_Embedding](https:\u002F\u002Fnlp.johnsnowlabs.com\u002F2020\u002F08\u002F25\u002Fsent_small_bert_L2_128.html) |\n| Chapter 1: NLU base-features with some classifiers on testdata | `emotion`, `yake`, `stem` | [![Open In Colab](https:\u002F\u002Fcolab.research.google.com\u002Fassets\u002Fcolab-badge.svg)](https:\u002F\u002Fcolab.research.google.com\u002Fgithub\u002FJohnSnowLabs\u002Fnlu\u002Fblob\u002Fmaster\u002Fexamples\u002Fwebinars_conferences_etc\u002FNYC_DC_NLP_MEETUP\u002F1_NLU_base_features_on_dataset_with_YAKE_Lemma_Stemm_classifiers_NER_.ipynb) | [Emotion detection](https:\u002F\u002Fnlp.johnsnowlabs.com\u002F2021\u002F01\u002F09\u002Fclassifierdl_use_emotion_en.html) |\n| Chapter 2: Translation between 300+ languages with Marian | `tr.translate_to.en`, `en.translate_to.fr`, `en.translate_to.he` | [![Open In Colab](https:\u002F\u002Fcolab.research.google.com\u002Fassets\u002Fcolab-badge.svg)](https:\u002F\u002Fcolab.research.google.com\u002Fgithub\u002FJohnSnowLabs\u002Fnlu\u002Fblob\u002Fmaster\u002Fexamples\u002Fcolab\u002Fcomponent_examples\u002Fsequence2sequence\u002Ftranslation_demo.ipynb) | [Translation-Pipeline (En to Fr)](https:\u002F\u002Fnlp.johnsnowlabs.com\u002F2021\u002F06\u002F04\u002Ftranslate_en_fr_xx.html), [Translation (En to He)](https:\u002F\u002Fnlp.johnsnowlabs.com\u002F2021\u002F06\u002F04\u002Ftranslate_en_he_xx.html) |\n| Chapter 3: Answer questions and summarize Texts with T5 | `answer_question`, `en.t5`, `en.t5.base` | [![Open In Colab](https:\u002F\u002Fcolab.research.google.com\u002Fassets\u002Fcolab-badge.svg)](https:\u002F\u002Fcolab.research.google.com\u002Fgithub\u002FJohnSnowLabs\u002Fnlu\u002Fblob\u002Fmaster\u002Fexamples\u002Fcolab\u002Fcomponent_examples\u002Fsequence2sequence\u002FT5_question_answering.ipynb) | [T5-Model](https:\u002F\u002Fnlp.johnsnowlabs.com\u002F2021\u002F01\u002F08\u002Ft5_base_en.html) |\n| Chapter 4: Overview of T5-Tasks | `en.t5.base` | [![Open In Colab](https:\u002F\u002Fcolab.research.google.com\u002Fassets\u002Fcolab-badge.svg)](https:\u002F\u002Fcolab.research.google.com\u002Fgithub\u002FJohnSnowLabs\u002Fnlu\u002Fblob\u002Fmaster\u002Fexamples\u002Fcolab\u002Fcomponent_examples\u002Fsequence2sequence\u002FT5_tasks_summarize_question_answering_and_more.ipynb) | [T5-Model](https:\u002F\u002Fnlp.johnsnowlabs.com\u002F2021\u002F01\u002F08\u002Ft5_base_en.html) |\n| Graph NLU 20 Minutes Crashcourse - State of the Art Text Mining for Graphs | `spell`, `sentiment`, `pos`, `ner`, `yake`, `emotion`, `med_ner.jsl.wip.clinical`, ... | [![Open In Colab](https:\u002F\u002Fcolab.research.google.com\u002Fassets\u002Fcolab-badge.svg)](https:\u002F\u002Fcolab.research.google.com\u002Fgithub\u002FJohnSnowLabs\u002Fnlu\u002Fblob\u002Fmaster\u002Fexamples\u002Fwebinars_conferences_etc\u002Fgraph_ai_summit\u002FHealthcare_Graph_NLU_COVID_Tigergraph.ipynb) | [Part of Speech](https:\u002F\u002Fnlp.johnsnowlabs.com\u002F2021\u002F03\u002F05\u002Fpos_anc.html), [NER-Piple](https:\u002F\u002Fnlp.johnsnowlabs.com\u002F2021\u002F03\u002F22\u002Fonto_recognize_entities_sm_en.html), [Emotion detection](https:\u002F\u002Fnlp.johnsnowlabs.com\u002F2021\u002F01\u002F09\u002Fclassifierdl_use_emotion_en.html), [Spellchecker](https:\u002F\u002Fnlp.johnsnowlabs.com\u002F2021\u002F03\u002F28\u002Fspellcheck_dl_en.html), [Sentiment classification](https:\u002F\u002Fnlp.johnsnowlabs.com\u002F2021\u002F03\u002F24\u002Fanalyze_sentiment_en.html) |\n| Healthcare with NLU | `med_ner.human_phenotype.gene_biobert`, `med_ner.ade_biobert`, `med_ner.anatomy`, `med_ner.bacterial_species`,... | [![Open In Colab](https:\u002F\u002Fcolab.research.google.com\u002Fassets\u002Fcolab-badge.svg)](https:\u002F\u002Fcolab.research.google.com\u002Fgithub\u002FJohnSnowLabs\u002Fnlu\u002Fblob\u002Fmaster\u002Fexamples\u002Fwebinars_conferences_etc\u002Fhealthcare_webinar\u002FNLU_healthcare_webinar.ipynb) | - |\n| Part 0: Intro: 1-liners | `spell`, `sentiment`, `pos`, `ner`, `bert`, `elmo`, `embed_sentence.bert` | [![Open In Colab](https:\u002F\u002Fcolab.research.google.com\u002Fassets\u002Fcolab-badge.svg)](https:\u002F\u002Fcolab.research.google.com\u002Fgithub\u002FJohnSnowLabs\u002Fnlu\u002Fblob\u002Fmaster\u002Fexamples\u002Fwebinars_conferences_etc\u002Fmulti_lingual_webinar\u002F0_liners_intro.ipynb) | [Bert-Paper](https:\u002F\u002Farxiv.org\u002Fabs\u002F1810.04805), [Bert Github](https:\u002F\u002Fgithub.com\u002Fgoogle-research\u002Fbert), [T-SNE](https:\u002F\u002Fwww.jmlr.org\u002Fpapers\u002Fvolume9\u002Fvandermaaten08a\u002Fvandermaaten08a.pdf?fbclid=IwA), [T-SNE-Bert](https:\u002F\u002Fmedium.com\u002Fspark-nlp\u002F1-line-to-bert-word-embeddings-with-nlu-f50d2b08cddc) , [Part of Speech](https:\u002F\u002Fnlp.johnsnowlabs.com\u002F2021\u002F03\u002F05\u002Fpos_anc.html), [NER-Piple](https:\u002F\u002Fnlp.johnsnowlabs.com\u002F2021\u002F03\u002F22\u002Fonto_recognize_entities_sm_en.html), [Spellchecker](https:\u002F\u002Fnlp.johnsnowlabs.com\u002F2021\u002F03\u002F28\u002Fspellcheck_dl_en.html), [Sentiment classification](https:\u002F\u002Fnlp.johnsnowlabs.com\u002F2021\u002F03\u002F24\u002Fanalyze_sentiment_en.html), [Elmo-Embedding](https:\u002F\u002Fnlp.johnsnowlabs.com\u002F2020\u002F01\u002F31\u002Felmo.html) , [Bert-Sentence_Embedding](https:\u002F\u002Fnlp.johnsnowlabs.com\u002F2020\u002F08\u002F25\u002Fsent_small_bert_L2_128.html) | \n| Part 1: NLU base-features with some classifiers on Testdata | `yake`, `stem`, `ner`, `emotion` | [![Open In Colab](https:\u002F\u002Fcolab.research.google.com\u002Fassets\u002Fcolab-badge.svg)](https:\u002F\u002Fcolab.research.google.com\u002Fgithub\u002FJohnSnowLabs\u002Fnlu\u002Fblob\u002Fmaster\u002Fexamples\u002Fwebinars_conferences_etc\u002Fmulti_lingual_webinar\u002F1_NLU_base_features_on_dataset_with_YAKE_Lemma_Stemm_classifiers_NER_.ipynb) | [NER-Piple](https:\u002F\u002Fnlp.johnsnowlabs.com\u002F2021\u002F03\u002F22\u002Fonto_recognize_entities_sm_en.html), [Emotion detection](https:\u002F\u002Fnlp.johnsnowlabs.com\u002F2021\u002F01\u002F09\u002Fclassifierdl_use_emotion_en.html) | \n| Part 2: Translate between 200+ Languages in 1 line of code with Marian-Models | `en.translate_to.de`, `en.translate_to.fr`, `en.translate_to.he` | [![Open In Colab](https:\u002F\u002Fcolab.research.google.com\u002Fassets\u002Fcolab-badge.svg)](https:\u002F\u002Fcolab.research.google.com\u002Fgithub\u002FJohnSnowLabs\u002Fnlu\u002Fblob\u002Fmaster\u002Fexamples\u002Fwebinars_conferences_etc\u002Fmulti_lingual_webinar\u002F2_multilingual_translation_with_marian_intro.ipynb) | [Translation-Pipeline (En to Fr)](https:\u002F\u002Fnlp.johnsnowlabs.com\u002F2021\u002F06\u002F04\u002Ftranslate_en_fr_xx.html), [Translation-Pipeline (En to Ger)](https:\u002F\u002Fnlp.johnsnowlabs.com\u002F2021\u002F06\u002F04\u002Ftranslate_en_de_xx.html), [Translation (En to He)](https:\u002F\u002Fnlp.johnsnowlabs.com\u002F2021\u002F06\u002F04\u002Ftranslate_en_he_xx.html) | \n| Part 3: More Multilingual NLP-translations for Asian Languages with Marian | `en.translate_to.hi`, `en.translate_to.ru`, `en.translate_to.zh` | [![Open In Colab](https:\u002F\u002Fcolab.research.google.com\u002Fassets\u002Fcolab-badge.svg)](https:\u002F\u002Fcolab.research.google.com\u002Fgithub\u002FJohnSnowLabs\u002Fnlu\u002Fblob\u002Fmaster\u002Fexamples\u002Fwebinars_conferences_etc\u002Fmulti_lingual_webinar\u002F3_more_multi_lingual_NLP_translation_Asian_languages_with_Marian.ipynb) | [Translation (En to Hi)](https:\u002F\u002Fnlp.johnsnowlabs.com\u002F2021\u002F06\u002F04\u002Ftranslate_en_hil_xx.html), [Translation (En to Ru)](https:\u002F\u002Fnlp.johnsnowlabs.com\u002F2021\u002F06\u002F04\u002Ftranslate_en_run_xx.html), [Translation (En to Zh)](https:\u002F\u002Fnlp.johnsnowlabs.com\u002F2021\u002F06\u002F04\u002Ftranslate_en_zh_xx.html) | \n| Part 4: Unsupervise Chinese Keyword Extraction, NER and Translation from chinese news | `zh.translate_to.en`, `zh.segment_words`, `yake`, `zh.lemma`, `zh.ner` | [![Open In Colab](https:\u002F\u002Fcolab.research.google.com\u002Fassets\u002Fcolab-badge.svg)](https:\u002F\u002Fcolab.research.google.com\u002Fgithub\u002FJohnSnowLabs\u002Fnlu\u002Fblob\u002Fmaster\u002Fexamples\u002Fwebinars_conferences_etc\u002Fmulti_lingual_webinar\u002F4_Unsupervise_Chinese_Keyword_Extraction_NER_and_Translation_from_Chinese_News.ipynb) | [Translation-Pipeline (Zh to En)](https:\u002F\u002Fnlp.johnsnowlabs.com\u002F2021\u002F06\u002F04\u002Ftranslate_zh_en_xx.html), [Zh-Lemmatizer](https:\u002F\u002Fnlp.johnsnowlabs.com\u002F2020\u002F03\u002F19\u002Fexplain_document_dl.html) | \n| Part 5: Multilingual sentiment classifier training for 100+ languages | `train.sentiment`, `xx.embed_sentence.labse train.sentiment` | n.a. | [Sentence_Embedding.Labse](https:\u002F\u002Fnlp.johnsnowlabs.com\u002F2020\u002F09\u002F23\u002Flabse.html) | \n| Part 6: Question-answering and Text-summarization with T5-Modell | `answer_question`, `en.t5`, `en.t5.base` | [![Open In Colab](https:\u002F\u002Fcolab.research.google.com\u002Fassets\u002Fcolab-badge.svg)](https:\u002F\u002Fcolab.research.google.com\u002Fgithub\u002FJohnSnowLabs\u002Fnlu\u002Fblob\u002Fmaster\u002Fexamples\u002Fwebinars_conferences_etc\u002Fmulti_lingual_webinar\u002F6_T5_question_answering_and_Text_summarization.ipynb) | [T5-Paper](https:\u002F\u002Farxiv.org\u002Fpdf\u002F1910.10683.pdf) | \n| Part 7: Overview of all tasks available with T5 | `en.t5.base` | [![Open In Colab](https:\u002F\u002Fcolab.research.google.com\u002Fassets\u002Fcolab-badge.svg)](https:\u002F\u002Fcolab.research.google.com\u002Fgithub\u002FJohnSnowLabs\u002Fnlu\u002Fblob\u002Fmaster\u002Fexamples\u002Fwebinars_conferences_etc\u002Fmulti_lingual_webinar\u002F7_T5_SQUAD_GLUE_SUPER_GLUE_TASKS.ipynb) | [T5-Paper](https:\u002F\u002Farxiv.org\u002Fpdf\u002F1910.10683.pdf) | \n| Part 8: Overview of some of the Multilingual modes with State Of the Art accuracy (1-liner) | `bn.lemma`, `ja.lemma`, `am.lemma`, `bh.lemma`, `zh.segment_words`, ... | [![Open In Colab](https:\u002F\u002Fcolab.research.google.com\u002Fassets\u002Fcolab-badge.svg)](https:\u002F\u002Fcolab.research.google.com\u002Fgithub\u002FJohnSnowLabs\u002Fnlu\u002Fblob\u002Fmaster\u002Fexamples\u002Fwebinars_conferences_etc\u002Fmulti_lingual_webinar\u002F8_Multi_lingual_ner_pos_stop_words_senti) | [Bengali-Lemmatizer](https:\u002F\u002Fnlp.johnsnowlabs.com\u002F2021\u002F01\u002F20\u002Flemma_bn.html), [Japanese-Lemmatizer](https:\u002F\u002Fnlp.johnsnowlabs.com\u002F2021\u002F01\u002F15\u002Flemma_ja.html) , [Amharic-Lemmatizer](https:\u002F\u002Fnlp.johnsnowlabs.com\u002F2021\u002F01\u002F20\u002Flemma_am.html) |\n| Overview of some Multilingual modes avaiable with State Of the Art accuracy (1-liner) | `bn.ner.cc_300d`, `ja.ner`, `zh.ner`, `th.ner.lst20.glove_840B_300D`, `ar.ner` | [![Open In Colab](https:\u002F\u002Fcolab.research.google.com\u002Fassets\u002Fcolab-badge.svg)](https:\u002F\u002Fcolab.research.google.com\u002Fgithub\u002FJohnSnowLabs\u002Fnlu\u002Fblob\u002Fmaster\u002Fexamples\u002Fwebinars_conferences_etc\u002Fpython_web_conf\u002FMulti_Linigual_examples.ipynb) | [Bengali-NER](https:\u002F\u002Fnlp.johnsnowlabs.com\u002F2021\u002F01\u002F27\u002Fner_jifs_glove_840B_300d_bn.html) \n| NLU 20 Minutes Crashcourse - the fast Data Science route | - | [![Open In Colab](https:\u002F\u002Fcolab.research.google.com\u002Fassets\u002Fcolab-badge.svg)](https:\u002F\u002Fcolab.research.google.com\u002Fgithub\u002FJohnSnowLabs\u002Fnlu\u002Fblob\u002Fmaster\u002Fexamples\u002Fwebinars_conferences_etc\u002Fpython_web_conf\u002FNLU_crashcourse_py_web.ipynb) | - | \n \n\n# Need help? \n- [Ping us on Slack](https:\u002F\u002Fspark-nlp.slack.com\u002Farchives\u002FC0196BQCDPY) \n- [Post an issue on Github](https:\u002F\u002Fgithub.com\u002FJohnSnowLabs\u002Fnlu\u002Fissues)\n\n# Simple NLU Demos\n- [NLU different output levels Demo](https:\u002F\u002Fcolab.research.google.com\u002Fdrive\u002F1C4N3wpC17YzZf9fXHDNAJ5JvSmfbq7zT?usp=sharing)\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n# Features in NLU Overview\n* Tokenization\n* Trainable Word Segmentation\n* Stop Words Removal\n* Token Normalizer\n* Document Normalizer\n* Stemmer\n* Lemmatizer\n* NGrams\n* Regex Matching\n* Text Matching,\n* Chunking\n* Date Matcher\n* Sentence Detector\n* Deep Sentence Detector (Deep learning)\n* Dependency parsing (Labeled\u002Funlabeled)\n* Part-of-speech tagging\n* Sentiment Detection (ML models)\n* Spell Checker (ML and DL models)\n* Word Embeddings (GloVe and Word2Vec)\n* BERT Embeddings (TF Hub models)\n* ELMO Embeddings (TF Hub models)\n* ALBERT Embeddings (TF Hub models)\n* XLNet Embeddings\n* Universal Sentence Encoder (TF Hub models)\n* BERT Sentence Embeddings (42 TF Hub models)\n* Sentence Embeddings\n* Chunk Embeddings\n* Unsupervised keywords extraction\n* Language Detection & Identification (up to 375 languages)\n* Multi-class Sentiment analysis (Deep learning)\n* Multi-label Sentiment analysis (Deep learning)\n* Multi-class Text Classification (Deep learning)\n* Neural Machine Translation\n* Text-To-Text Transfer Transformer (Google T5)\n* Named entity recognition (Deep learning)\n* Easy TensorFlow integration\n* GPU Support\n* Full integration with Spark ML functions\n* 1000 pre-trained models in +200 languages!\n* Multi-lingual NER models: Arabic, Chinese, Danish, Dutch, English, Finnish, French, German, Hewbrew, Italian, Japanese, Korean, Norwegian, Persian, Polish, Portuguese, Russian, Spanish, Swedish, Urdu and more\n* Natural Language inference\n* Coreference resolution\n* Sentence Completion\n* Word sense disambiguation\n* Clinical entity recognition\n* Clinical Entity Linking\n* Entity normalization\n* Assertion Status Detection\n* De-identification\n* Relation Extraction\n* Clinical Entity Resolution\n\n\n## Citation\n\nWe have published a [paper](https:\u002F\u002Fwww.sciencedirect.com\u002Fscience\u002Farticle\u002Fpii\u002FS2665963821000063) that you can cite for the NLU library:\n\n```bibtex\n@article{KOCAMAN2021100058,\n title = {Spark NLP: Natural language understanding at scale},\n journal = {Software Impacts},\n pages = {100058},\n year = {2021},\n issn = {2665-9638},\n doi = {https:\u002F\u002Fdoi.org\u002F10.1016\u002Fj.simpa.2021.100058},\n url = {https:\u002F\u002Fwww.sciencedirect.com\u002Fscience\u002Farticle\u002Fpii\u002FS2665963821000063},\n author = {Veysel Kocaman and David Talby},\n keywords = {Spark, Natural language processing, Deep learning, Tensorflow, Cluster},\n abstract = {Spark NLP is a Natural Language Processing (NLP) library built on top of Apache Spark ML. It provides simple, performant & accurate NLP annotations for machine learning pipelines that can scale easily in a distributed environment. Spark NLP comes with 1100+ pretrained pipelines and models in more than 192+ languages. It supports nearly all the NLP tasks and modules that can be used seamlessly in a cluster. Downloaded more than 2.7 million times and experiencing 9x growth since January 2020, Spark NLP is used by 54% of healthcare organizations as the world’s most widely used NLP library in the enterprise.}\n }\n}\n```\n","# NLU：Spark NLP的强大功能，Python的简洁易用\nJohn Snow Labs 的 NLU 是一个 Python 库，可直接在任何数据框上应用最先进的文本挖掘技术，只需一行代码即可实现。\n作为屡获殊荣的 Spark NLP 库的前端接口，它内置了 **1000 多个** 预训练模型，涵盖 **100 多种语言**，全部具备生产级质量、可扩展性和可训练性，且**所有操作仅需一行代码**。\n\n\n\n## NLU 实战演示 \n看看如何轻松地用一行代码调用数千种模型中的任意一种。我们提供了数百个[教程](https:\u002F\u002Fnlu.johnsnowlabs.com\u002Fdocs\u002Fen\u002Fnotebooks)和[简单示例](https:\u002F\u002Fgithub.com\u002FJohnSnowLabs\u002Fnlu\u002Ftree\u002Fmaster\u002Fexamples)，你可以直接复制粘贴到自己的项目中，轻松实现最先进水平的自然语言处理效果。 \n\u003Cimg src=\"http:\u002F\u002Fckl-it.de\u002Fwp-content\u002Fuploads\u002F2020\u002F08\u002FMy-Video6.gif\" width=\"1800\" height=\"500\"\u002F>\n\n## NLU 与 Streamlit 实战演示 \n只需一行代码，你就能可视化并玩转 **1000 多种 SOTA NLU 和 NLP 模型**，支持 **200 种语言**。 \n\n```shell\nstreamlit run https:\u002F\u002Fraw.githubusercontent.com\u002FJohnSnowLabs\u002Fnlu\u002Fmaster\u002Fexamples\u002Fstreamlit\u002F01_dashboard.py \n```\n\u003Cimg src=\"https:\u002F\u002Foss.gittoolsai.com\u002Fimages\u002FJohnSnowLabs_nlu_readme_f2d54c29650a.gif\">\n\nNLU 提供了与 Streamlit 的紧密而简单的集成，让你只需一行代码就能构建功能强大的 Web 应用程序，充分展示其强大能力。 \n查看[NLU 与 Streamlit 文档](https:\u002F\u002Fnlu.johnsnowlabs.com\u002Fdocs\u002Fen\u002Fstreamlit_viz_examples)或[NLU 与 Streamlit 示例部分](https:\u002F\u002Fgithub.com\u002FJohnSnowLabs\u002Fnlu\u002Ftree\u002Fmaster\u002Fexamples\u002Fstreamlit)。 \n整个 GIF 演示和 \n\n\n## NLU 全部资源概览 \n请访问我们的官方 NLU 页面：[https:\u002F\u002Fnlu.johnsnowlabs.com\u002F](https:\u002F\u002Fnlu.johnsnowlabs.com\u002F) 获取用户文档和示例。\n\n| 资源 | 描述|\n|-----------------------------------------------------------------------|-------------------------------------------|\n| [安装 NLU](https:\u002F\u002Fnlu.johnsnowlabs.com\u002Fdocs\u002Fen\u002Finstall) | 只需运行 `pip install nlu pyspark==3.0.2` \n| [NLU 命名空间](https:\u002F\u002Fnlu.johnsnowlabs.com\u002Fdocs\u002Fen\u002Fnamespace) | 查找所有可通过 `nlu.load()` 加载的模型名称\n| [nlu.load(\u003CModel>) 函数](https:\u002F\u002Fnlu.johnsnowlabs.com\u002Fdocs\u002Fen\u002Fload_api) | 用一行代码加载 **1000 多种模型中的任意一种**\n| [nlu.load(\u003CModel>).predict(data) 函数](https:\u002F\u002Fnlu.johnsnowlabs.com\u002Fdocs\u002Fen\u002Fpredict_api) | 对 `字符串`、`字符串列表`、`NumPy 数组`、`Pandas`、`Modin` 和 `Spark 数据框`进行预测\n| [nlu.load(\u003Ctrain.Model>).fit(data) 函数](https:\u002F\u002Fnlu.johnsnowlabs.com\u002Fdocs\u002Fen\u002Ftraining) | 训练文本分类器，适用于 `二分类`、`多分类`、`多标签分类`、`命名实体识别` 或 `词性标注`\n| [nlu.load(\u003CModel>).viz(data) 函数](https:\u002F\u002Fnlu.johnsnowlabs.com\u002Fdocs\u002Fen\u002Fviz_examples) | 可视化 `词嵌入相似度矩阵`、`命名实体识别器`、`依存句法树与词性`、`实体消歧`、`实体链接` 或 `实体状态断言` 的结果\n| [nlu.load(\u003CModel>).viz_streamlit(data) 函数](https:\u002F\u002Fnlu.johnsnowlabs.com\u002Fdocs\u002Fen\u002Fstreamlit_viz_examples) | 展示交互式 GUI，让你只需一键就能探索和测试 NLU 中的每种模型和功能。\n| [通用概念](https:\u002F\u002Fnlu.johnsnowlabs.com\u002Fdocs\u002Fen\u002Fconcepts) | NLU 中的通用概念\n| [最新版本说明](https:\u002F\u002Fnlu.johnsnowlabs.com\u002Fdocs\u002Fen\u002Frelease_notes) | NLU 新增的最新增功能\n| [NLU 1 行代码示例概览](https:\u002F\u002Fnlu.johnsnowlabs.com\u002Fdocs\u002Fen\u002Fexamples) | 最常用的模型及其结果\n| [医疗健康领域 NLU 1 行代码示例概览](https:\u002F\u002Fnlu.johnsnowlabs.com\u002Fdocs\u002Fen\u002Fexamples_hc) | 医疗健康领域最常用的模型及其结果 \n| [NLU 所有教程与示例概览](https:\u002F\u002Fnlu.johnsnowlabs.com\u002Fdocs\u002Fen\u002Fnotebooks) | 100 多个关于如何在各种问题上使用 NLU 处理文本数据集的教程，涵盖 Twitter、中文新闻、加密货币新闻标题、航空交通通信、产品评论分类器训练等不同来源的数据\n| [加入 Slack 与我们交流](https:\u002F\u002Fjoin.slack.com\u002Ft\u002Fspark-nlp\u002Fshared_invite\u002Fzt-lutct9gm-kuUazcyFKhuGY3_0AMkxqA) | 遇到问题、疑问或建议？我们拥有一个非常活跃且乐于助人的社区，超过 2000 名 AI 爱好者正积极利用 NLU、Spark NLP 和 Spark OCR\n| [讨论论坛](https:\u002F\u002Fgithub.com\u002FJohnSnowLabs\u002Fspark-nlp\u002Fdiscussions) | 想与社区进行更深入的讨论？在我们的讨论论坛发帖\n| [John Snow Labs Medium](https:\u002F\u002Fmedium.com\u002Fspark-nlp) | 关于 NLU、Spark NLP 和 Spark OCR 的文章与教程\n| [John Snow Labs Youtube](https:\u002F\u002Fwww.youtube.com\u002Fchannel\u002FUCmFOjlpYEhxf_wJUDuz6xxQ\u002Fvideos) | 关于 NLU、Spark NLP 和 Spark OCR 的视频与教程\n| [NLU 官网](https:\u002F\u002Fnlu.johnsnowlabs.com\u002F) | NLU 官方网站\n|[Github 问题](https:\u002F\u002Fgithub.com\u002FJohnSnowLabs\u002Fnlu\u002Fissues) | 报告 Bug\n\n\n\n\n\n\n## 开始使用 NLU \n要体验 NLU 的强大功能，只需通过 pip 安装它，并确保已正确安装和配置 Java 8。查看[快速入门指南](https:\u002F\u002Fnlu.johnsnowlabs.com\u002Fdocs\u002Fen\u002Finstall)获取更多信息。 \n```bash \npip install nlu pyspark==3.0.2\n``` \n\n## 用一行 Python 代码加载并预测任意模型 \n```python\nimport nlu \nnlu.load('sentiment').predict('我爱 NLU！\u003C3') \n``` \n\n## 用一行代码加载并预测多个模型 \n\n只需一行代码即可获得 6 种不同的词嵌入，并将其用于下游数据科学任务！ \n\n```python \nnlu.load('bert elmo albert xlnet glove use').predict('我爱 NLU！\u003C3') \n```\n\n## NLU 提供哪些类型的模型？ \nNLU 用一行代码就能满足数据科学家的所有需求！ \n- NLU 用一行代码就能满足数据科学家的所有需求！ \n- 1000 多个预训练模型 \n- 100 多种最新的自然语言处理词嵌入（BERT、ELMO、ALBERT、XLNET、GLOVE、BIOBERT、ELECTRA、COVIDBERT）及其不同变体 \n- 50 多种最新的自然语言处理句子嵌入（BERT、ELECTRA、USE）及其不同变体 \n- 100 多种分类器（命名实体识别、词性标注、情感分析、讽刺识别、问题检测、垃圾邮件检测） \n- 支持 300 多种语言 \n- 使用 T5 总结文本并回答问题 \n- 标注与未标注的依存句法分析 \n- 各种文本清洗与预处理方法，如词干提取、词形还原、规范化、过滤、清洗流水线等 \n\n\n## 基于多种不同数据集训练的分类器 \n为合适的任务选择合适的工具！无论你是分析电影还是推特，NLU 都有适合你的模型！ \n\n- trec6 分类器 \n- trec10 分类器 \n- 垃圾邮件分类器 \n- 假新闻分类器 \n- 情感分类器 \n- 网络欺凌分类器 \n- 讽刺分类器 \n- 电影情感分类器 \n- IMDB 电影情感分类器 \n- 推特情感分类器 \n- 在 ONTO notes 上预训练的命名实体识别 \n- 在 CONLL 上训练的命名实体识别 \n- 在 wiki 20 lang 数据集上针对 20 种语言的语言分类器 \n\n## 数据科学 NLU 应用的实用工具 \n处理文本数据有时会是一项相当繁琐的工作。NLU 通过提供一系列组件，帮你摆脱数据工程中繁重的任务，让你的手保持干净。 \n\n- 日期时间匹配器 \n- 模式匹配器 \n- 片段匹配器 \n- 短语匹配器 \n- 停用词清理器 \n- 模式清理器 \n- 俚语清理器 \n\n## 我可以在哪里查看 NLU 中所有可用的模型？ \n要加载 NLU 模型，请参阅 [NLU 命名空间](https:\u002F\u002Fnlu.johnsnowlabs.com\u002Fdocs\u002Fen\u002Fnamespace) 或 [John Snow Labs Modelshub](https:\u002F\u002Fmodelshub.johnsnowlabs.com\u002Fmodels)，或者直接访问 [源文件](https:\u002F\u002Fgithub.com\u002FJohnSnowLabs\u002Fnlu\u002Fblob\u002Fmaster\u002Fnlu\u002Fnamespace.py)。\n\n## 支持的数据类型 \n- Pandas DataFrame 和 Series \n- Spark DataFrames \n- 使用 Ray 后端的 Modin \n- 使用 Dask 后端的 Modin \n- NumPy 数组 \n- 字符串和字符串列表 \n\n## 使用 NLU 库的所有教程概览 \n\n以下表格列出了所有使用 NLU 的可用教程。这些教程将帮助你了解 NLU 库的用法，以及如何将其应用于自己的任务。NLU 可以完成的一些任务包括：将任何语言翻译成英语、词形还原、分词、清除符号或不需要的语法、拼写检查、实体识别、情感分析等等！\n\n{:.table2}\n\n| Tutorial Description | NLU Spells Used |Open In Colab | Dataset and Paper References |\n|-----------------------------------------------------------------------------------------------------|---------------------------------------------------------------------------------------------------------------------------|----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|\n| Albert Word Embeddings with NLU | `albert`, `sentiment pos albert emotion` |[![Open In Colab](https:\u002F\u002Fcolab.research.google.com\u002Fassets\u002Fcolab-badge.svg)](https:\u002F\u002Fcolab.research.google.com\u002Fgithub\u002FJohnSnowLabs\u002Fnlu\u002Fblob\u002Fmaster\u002Fexamples\u002Fcolab\u002Fcomponent_examples\u002Fword_embeddings\u002FNLU_ALBERT_word_embeddings_and_t-SNE_visualization_example.ipynb) | [Albert-Paper](https:\u002F\u002Farxiv.org\u002Fpdf\u002F1909.11942.pdf), [Albert on Github](https:\u002F\u002Fgithub.com\u002Fgoogle-research\u002FALBERT), [Albert on TensorFlow](https:\u002F\u002Ftfhub.dev\u002Fs?q=albert), [T-SNE](https:\u002F\u002Fwww.jmlr.org\u002Fpapers\u002Fvolume9\u002Fvandermaaten08a\u002Fvandermaaten08a.pdf?fbclid=IwA), [T-SNE-Albert](https:\u002F\u002Fmedium.com\u002Fspark-nlp\u002F1-line-to-albert-word-embeddings-with-nlu-in-python-1691bc048ed1), [Albert_Embedding](https:\u002F\u002Fnlp.johnsnowlabs.com\u002F2021\u002F06\u002F23\u002Falbert_base_uncased_en.html) | \n| Bert Word Embeddings with NLU | `bert`, `pos sentiment emotion bert` |[![Open In Colab](https:\u002F\u002Fcolab.research.google.com\u002Fassets\u002Fcolab-badge.svg)](https:\u002F\u002Fcolab.research.google.com\u002Fgithub\u002FJohnSnowLabs\u002Fnlu\u002Fblob\u002Fmaster\u002Fexamples\u002Fcolab\u002Fcomponent_examples\u002Fword_embeddings\u002FNLU_BERT_word_embeddings_and_t-SNE_visualization_example.ipynb) | [Bert-Paper](https:\u002F\u002Farxiv.org\u002Fabs\u002F1810.04805), [Bert Github](https:\u002F\u002Fgithub.com\u002Fgoogle-research\u002Fbert), [T-SNE](https:\u002F\u002Fwww.jmlr.org\u002Fpapers\u002Fvolume9\u002Fvandermaaten08a\u002Fvandermaaten08a.pdf?fbclid=IwA), [T-SNE-Bert](https:\u002F\u002Fmedium.com\u002Fspark-nlp\u002F1-line-to-bert-word-embeddings-with-nlu-f50d2b08cddc), [Bert_Embedding](https:\u002F\u002Fnlp.johnsnowlabs.com\u002F2020\u002F08\u002F25\u002Fsent_small_bert_L2_128.html) |\n| BIOBERT Word Embeddings with NLU | `biobert` , `sentiment pos biobert emotion` |[![Open In Colab](https:\u002F\u002Fcolab.research.google.com\u002Fassets\u002Fcolab-badge.svg)](https:\u002F\u002Fcolab.research.google.com\u002Fgithub\u002FJohnSnowLabs\u002Fnlu\u002Fblob\u002Fmaster\u002Fexamples\u002Fcolab\u002Fcomponent_examples\u002Fword_embeddings\u002FNLU_BIOBERT_word_embeddings_and_t-SNE_visualization_example.ipynb) | [BioBert-Paper](https:\u002F\u002Farxiv.org\u002Fabs\u002F1901.08746), [Bert Github](https:\u002F\u002Fgithub.com\u002Fgoogle-research\u002Fbert) , [BERT: Deep Bidirectional Transformers](https:\u002F\u002Farxiv.org\u002Fabs\u002F1810.04805), [Bert Github](https:\u002F\u002Fgithub.com\u002Fgoogle-research\u002Fbert), [T-SNE](https:\u002F\u002Fwww.jmlr.org\u002Fpapers\u002Fvolume9\u002Fvandermaaten08a\u002Fvandermaaten08a.pdf?fbclid=IwA), [T-SNE-Biobert](https:\u002F\u002Fmedium.com\u002Fspark-nlp\u002F1-line-to-biobert-word-embeddings-with-nlu-in-python-7224ab52e131), [Biobert_Embedding](https:\u002F\u002Fnlp.johnsnowlabs.com\u002F2020\u002F08\u002F25\u002Fbiobert_pubmed_base_cased.html) |\n| COVIDBERT Word Embeddings with NLU | `covidbert`, `sentiment covidbert pos` |[![Open In Colab](https:\u002F\u002Fcolab.research.google.com\u002Fassets\u002Fcolab-badge.svg)](https:\u002F\u002Fcolab.research.google.com\u002Fgithub\u002FJohnSnowLabs\u002Fnlu\u002Fblob\u002Fmaster\u002Fexamples\u002Fcolab\u002Fcomponent_examples\u002Fword_embeddings\u002FNLU_COVIDBERT_word_embeddings_and_t-SNE_visualization_example.ipynb) | [CovidBert-Paper](https:\u002F\u002Fjournals.flvc.org\u002FFLAIRS\u002Farticle\u002Fview\u002F128488), [Bert Github](https:\u002F\u002Fgithub.com\u002Fgoogle-research\u002Fbert), [T-SNE](https:\u002F\u002Fwww.jmlr.org\u002Fpapers\u002Fvolume9\u002Fvandermaaten08a\u002Fvandermaaten08a.pdf?fbclid=IwA), [T-SNE-CovidBert](https:\u002F\u002Fmedium.com\u002Fspark-nlp\u002F1-line-to-covidbert-word-embeddings-with-nlu-in-python-e67396da2f78), [Covidbert_Embedding](https:\u002F\u002Fnlp.johnsnowlabs.com\u002F2020\u002F08\u002F27\u002Fcovidbert_large_uncased.html) |\n| ELECTRA Word Embeddings with NLU | `electra`, `sentiment pos en.embed.electra emotion` |[![Open In Colab](https:\u002F\u002Fcolab.research.google.com\u002Fassets\u002Fcolab-badge.svg)](https:\u002F\u002Fcolab.research.google.com\u002Fgithub\u002FJohnSnowLabs\u002Fnlu\u002Fblob\u002Fmaster\u002Fexamples\u002Fcolab\u002Fcomponent_examples\u002Fword_embeddings\u002FNLU_ELECTRA_word_embeddings_and_t-SNE_visualization_example.ipynb) | [Electra-Paper](https:\u002F\u002Farxiv.org\u002Fabs\u002F2003.10555), [T-SNE](https:\u002F\u002Fwww.jmlr.org\u002Fpapers\u002Fvolume9\u002Fvandermaaten08a\u002Fvandermaaten08a.pdf?fbclid=IwA), [T-SNE-Electra](https:\u002F\u002Fmedium.com\u002Fspark-nlp\u002F1-line-to-electra-word-embeddings-with-nlu-in-python-25f749bf3e92), [Electra_Embedding](https:\u002F\u002Fnlp.johnsnowlabs.com\u002F2020\u002F08\u002F27\u002Felectra_small_uncased.html) |\n| ELMO Word Embeddings with NLU | `elmo`, `sentiment pos elmo emotion` |[![Open In Colab](https:\u002F\u002Fcolab.research.google.com\u002Fassets\u002Fcolab-badge.svg)](https:\u002F\u002Fcolab.research.google.com\u002Fgithub\u002FJohnSnowLabs\u002Fnlu\u002Fblob\u002Fmaster\u002Fexamples\u002Fcolab\u002Fcomponent_examples\u002Fword_embeddings\u002FNLU_ELMo_word_embeddings_and_t-SNE_visualization_example.ipynb) | [ELMO-Paper](https:\u002F\u002Farxiv.org\u002Fabs\u002F1802.05365), [Elmo-TensorFlow](https:\u002F\u002Fgithub.com\u002Fgoogle-research\u002Fbert), [T-SNE](https:\u002F\u002Fwww.jmlr.org\u002Fpapers\u002Fvolume9\u002Fvandermaaten08a\u002Fvandermaaten08a.pdf?fbclid=IwA), [T-SNE-Elmo](https:\u002F\u002Fmedium.com\u002Fspark-nlp\u002F1-python-line-for-elmo-word-embeddings-with-john-snow-labs-nlu-628e9b924a3), [Elmo-Embedding](https:\u002F\u002Fnlp.johnsnowlabs.com\u002F2020\u002F01\u002F31\u002Felmo.html) |\n| GLOVE Word Embeddings with NLU | `glove`, `sentiment pos glove emotion` |[![Open In Colab](https:\u002F\u002Fcolab.research.google.com\u002Fassets\u002Fcolab-badge.svg)](https:\u002F\u002Fcolab.research.google.com\u002Fgithub\u002FJohnSnowLabs\u002Fnlu\u002Fblob\u002Fmaster\u002Fexamples\u002Fcolab\u002Fcomponent_examples\u002Fword_embeddings\u002FNLU_GLOVE_word_embeddings_and_t-SNE_visualization_example.ipynb) | [Glove-Paper](https:\u002F\u002Fnlp.stanford.edu\u002Fpubs\u002Fglove.pdf), [T-SNE](https:\u002F\u002Fwww.jmlr.org\u002Fpapers\u002Fvolume9\u002Fvandermaaten08a\u002Fvandermaaten08a.pdf?fbclid=IwA), [T-SNE-Glove](https:\u002F\u002Fmedium.com\u002Fspark-nlp\u002F1-line-to-glove-word-embeddings-with-nlu-in-python-baed152fff4d) , [Glove_Embedding](https:\u002F\u002Fnlp.johnsnowlabs.com\u002F2020\u002F01\u002F22\u002Fglove_100d.html) |\n| XLNET Word Embeddings with NLU | `xlnet`, `sentiment pos xlnet emotion` |[![Open In Colab](https:\u002F\u002Fcolab.research.google.com\u002Fassets\u002Fcolab-badge.svg)](https:\u002F\u002Fcolab.research.google.com\u002Fgithub\u002FJohnSnowLabs\u002Fnlu\u002Fblob\u002Fmaster\u002Fexamples\u002Fcolab\u002Fcomponent_examples\u002Fword_embeddings\u002FNLU_XLNET_word_embeddings_and_t-SNE_visualization_example.ipynb) | [XLNet-Paper](https:\u002F\u002Farxiv.org\u002Fabs\u002F1906.08237), [Bert Github](https:\u002F\u002Fgithub.com\u002Fzihangdai\u002Fxlnet), [T-SNE](https:\u002F\u002Fwww.jmlr.org\u002Fpapers\u002Fvolume9\u002Fvandermaaten08a\u002Fvandermaaten08a.pdf?fbclid=IwA), [T-SNE-XLNet](https:\u002F\u002Fmedium.com\u002Fspark-nlp\u002F1-line-to-xlnet-word-embeddings-with-nlu-in-python-5efc57d7ac79), [Xlnet_Embedding](https:\u002F\u002Fnlp.johnsnowlabs.com\u002F2021\u002F07\u002F07\u002Fxlnet_base_cased_en.html) |\n| Multiple Word-Embeddings and Part of Speech in 1 Line of code | `bert electra elmo glove xlnet albert pos` |[![Open In Colab](https:\u002F\u002Fcolab.research.google.com\u002Fassets\u002Fcolab-badge.svg)](https:\u002F\u002Fcolab.research.google.com\u002Fgithub\u002FJohnSnowLabs\u002Fnlu\u002Fblob\u002Fmaster\u002Fexamples\u002Fcolab\u002Fcomponent_examples\u002Fword_embeddings\u002FNLU_multiple_word_embeddings_and_t-SNE_visualization_example.ipynb) | [Bert-Paper](https:\u002F\u002Farxiv.org\u002Fpdf\u002F1810.04805.pdf), [Albert-Paper](https:\u002F\u002Fopenreview.net\u002Fforum?id=H1eA7AEtvS), [ELMO-Paper](https:\u002F\u002Farxiv.org\u002Fabs\u002F1802.05365), [Electra-Paper](https:\u002F\u002Farxiv.org\u002Fabs\u002F2003.10555), [XLNet-Paper](https:\u002F\u002Farxiv.org\u002Fpdf\u002F1906.08237.pdf), [Glove-Paper](https:\u002F\u002Fnlp.stanford.edu\u002Fpubs\u002Fglove.pdf) |\n| Normalzing with NLU | `norm` |[![Open In Colab](https:\u002F\u002Fcolab.research.google.com\u002Fassets\u002Fcolab-badge.svg)](https:\u002F\u002Fcolab.research.google.com\u002Fgithub\u002FJohnSnowLabs\u002Fnlu\u002Fblob\u002Fmaster\u002Fexamples\u002Fcolab\u002Fcomponent_examples\u002Ftext_pre_processing_and_cleaning\u002FNLU_normalizer_example.ipynb) | - |\n| Detect sentences with NLU | `sentence_detector.deep`, `sentence_detector.pragmatic`, `xx.sentence_detector` |[![Open In Colab](https:\u002F\u002Fcolab.research.google.com\u002Fassets\u002Fcolab-badge.svg)](https:\u002F\u002Fcolab.research.google.com\u002Fgithub\u002FJohnSnowLabs\u002Fnlu\u002Fblob\u002Fmaster\u002Fexamples\u002Fcolab\u002Fcomponent_examples\u002Ftext_pre_processing_and_cleaning\u002FNLU_sentence_detection_example.ipynb) | [Sentence Detector](https:\u002F\u002Fnlp.johnsnowlabs.com\u002F2020\u002F09\u002F13\u002Fsentence_detector_dl_en.html) |\n| Spellchecking with NLU | n.a. | n.a. | - |\n| Stemming with NLU | `en.stem`, `de.stem` |[![Open In Colab](https:\u002F\u002Fcolab.research.google.com\u002Fassets\u002Fcolab-badge.svg)](https:\u002F\u002Fcolab.research.google.com\u002Fgithub\u002FJohnSnowLabs\u002Fnlu\u002Fblob\u002Fmaster\u002Fexamples\u002Fcolab\u002Fcomponent_examples\u002Ftext_pre_processing_and_cleaning\u002FNLU_stemmer_example.ipynb) | - |\n| Stopwords removal with NLU | `stopwords` |[![Open In Colab](https:\u002F\u002Fcolab.research.google.com\u002Fassets\u002Fcolab-badge.svg)](https:\u002F\u002Fcolab.research.google.com\u002Fgithub\u002FJohnSnowLabs\u002Fnlu\u002Fblob\u002Fmaster\u002Fexamples\u002Fcolab\u002Fcomponent_examples\u002Ftext_pre_processing_and_cleaning\u002FNLU_stopwords_removal_example.ipynb) | [Stopwords](https:\u002F\u002Fnlp.johnsnowlabs.com\u002F2020\u002F07\u002F14\u002Fstopwords_en.html) |\n| Tokenization with NLU | `tokenize` |[![Open In Colab](https:\u002F\u002Fcolab.research.google.com\u002Fassets\u002Fcolab-badge.svg)](https:\u002F\u002Fcolab.research.google.com\u002Fgithub\u002FJohnSnowLabs\u002Fnlu\u002Fblob\u002Fmaster\u002Fexamples\u002Fcolab\u002Fcomponent_examples\u002Ftext_pre_processing_and_cleaning\u002FNLU_tokenization_example.ipynb) | - |\n| Normalization of Documents | `norm_document` |[![Open In Colab](https:\u002F\u002Fcolab.research.google.com\u002Fassets\u002Fcolab-badge.svg)](https:\u002F\u002Fcolab.research.google.com\u002Fgithub\u002FJohnSnowLabs\u002Fnlu\u002Fblob\u002Fmaster\u002Fexamples\u002Fcolab\u002Fcomponent_examples\u002Ftext_pre_processing_and_cleaning\u002Fdocument_normalizer_demo.ipynb) | - |\n| Open and Closed book question answering with Google's T5 | `en.t5` , `answer_question` |[![Open In Colab](https:\u002F\u002Fcolab.research.google.com\u002Fassets\u002Fcolab-badge.svg)](https:\u002F\u002Fcolab.research.google.com\u002Fgithub\u002FJohnSnowLabs\u002Fnlu\u002Fblob\u002Fmaster\u002Fexamples\u002Fcolab\u002Fcomponent_examples\u002Fsequence2sequence\u002FT5_question_answering.ipynb) | [T5-Paper](https:\u002F\u002Farxiv.org\u002Fpdf\u002F1910.10683.pdf), [T5-Model](https:\u002F\u002Fnlp.johnsnowlabs.com\u002F2021\u002F01\u002F08\u002Ft5_base_en.html) | \n| Overview of every task available with T5 | `en.t5.base` |[![Open In Colab](https:\u002F\u002Fcolab.research.google.com\u002Fassets\u002Fcolab-badge.svg)](https:\u002F\u002Fcolab.research.google.com\u002Fgithub\u002FJohnSnowLabs\u002Fnlu\u002Fblob\u002Fmaster\u002Fexamples\u002Fcolab\u002Fcomponent_examples\u002Fsequence2sequence\u002FT5_tasks_summarize_question_answering_and_more.ipynb) | [T5-Paper](https:\u002F\u002Farxiv.org\u002Fpdf\u002F1910.10683.pdf), [T5-Model](https:\u002F\u002Fnlp.johnsnowlabs.com\u002F2021\u002F01\u002F08\u002Ft5_base_en.html) | \n| Translate between more than 200 Languages in 1 line of code with Marian Models | `tr.translate_to.fr`, `en.translate_to.fr` ,`fr.translate_to.he` , `en.translate_to.de` |[![Open In Colab](https:\u002F\u002Fcolab.research.google.com\u002Fassets\u002Fcolab-badge.svg)](https:\u002F\u002Fcolab.research.google.com\u002Fgithub\u002FJohnSnowLabs\u002Fnlu\u002Fblob\u002Fmaster\u002Fexamples\u002Fcolab\u002Fcomponent_examples\u002Fsequence2sequence\u002Ftranslation_demo.ipynb) | [Marian-Papers](https:\u002F\u002Fmarian-nmt.github.io\u002Fpublications\u002F), [Translation-Pipeline (En to Fr)](https:\u002F\u002Fnlp.johnsnowlabs.com\u002F2021\u002F06\u002F04\u002Ftranslate_en_fr_xx.html), [Translation-Pipeline (En to Ger)](https:\u002F\u002Fnlp.johnsnowlabs.com\u002F2021\u002F06\u002F04\u002Ftranslate_en_de_xx.html) |\n| BERT Sentence Embeddings with NLU | `embed_sentence.bert`, `pos sentiment embed_sentence.bert` |[![Open In Colab](https:\u002F\u002Fcolab.research.google.com\u002Fassets\u002Fcolab-badge.svg)](https:\u002F\u002Fcolab.research.google.com\u002Fgithub\u002FJohnSnowLabs\u002Fnlu\u002Fblob\u002Fmaster\u002Fexamples\u002Fcolab\u002Fcomponent_examples\u002Fsentence_embeddings\u002FNLU_BERT_sentence_embeddings_and_t-SNE_visualization_Example.ipynb) | [Bert-Paper](https:\u002F\u002Farxiv.org\u002Fabs\u002F1810.04805), [Bert Github](https:\u002F\u002Fgithub.com\u002Fgoogle-research\u002Fbert), [Bert-Sentence_Embedding](https:\u002F\u002Fnlp.johnsnowlabs.com\u002F2020\u002F08\u002F25\u002Fsent_small_bert_L2_128.html) | \n| ELECTRA Sentence Embeddings with NLU | `embed_sentence.electra`, `pos sentiment embed_sentence.electra` |[![Open In Colab](https:\u002F\u002Fcolab.research.google.com\u002Fassets\u002Fcolab-badge.svg)](https:\u002F\u002Fcolab.research.google.com\u002Fgithub\u002FJohnSnowLabs\u002Fnlu\u002Fblob\u002Fmaster\u002Fexamples\u002Fcolab\u002Fcomponent_examples\u002Fsentence_embeddings\u002FNLU_ELECTRA_sentence_embeddings_and_t-SNE_visualization_example.ipynb) | [Electra Paper](https:\u002F\u002Farxiv.org\u002Fabs\u002F2003.10555), [Sentence-Electra-Embedding](https:\u002F\u002Fnlp.johnsnowlabs.com\u002F2020\u002F08\u002F27\u002Fsent_electra_small_uncased.html) |\n| USE Sentence Embeddings with NLU | `use`, `pos sentiment use emotion` |[![Open In Colab](https:\u002F\u002Fcolab.research.google.com\u002Fassets\u002Fcolab-badge.svg)](https:\u002F\u002Fcolab.research.google.com\u002Fgithub\u002FJohnSnowLabs\u002Fnlu\u002Fblob\u002Fmaster\u002Fexamples\u002Fcolab\u002Fcomponent_examples\u002Fsentence_embeddings\u002FNLU_USE_sentence_embeddings_and_t-SNE_visualization_example.ipynb) | [Universal Sentence Encoder](https:\u002F\u002Farxiv.org\u002Fabs\u002F1803.11175), [USE-TensorFlow](https:\u002F\u002Ftfhub.dev\u002Fgoogle\u002Funiversal-sentence-encoder\u002F2), [Sentence-USE-Embedding](https:\u002F\u002Fnlp.johnsnowlabs.com\u002F2020\u002F04\u002F17\u002Ftfhub_use_lg.html) |\n| Sentence similarity with NLU using BERT embeddings | `embed_sentence.bert`, `use en.embed_sentence.electra embed_sentence.bert` |[![Open In Colab](https:\u002F\u002Fcolab.research.google.com\u002Fassets\u002Fcolab-badge.svg)](https:\u002F\u002Fcolab.research.google.com\u002Fgithub\u002FJohnSnowLabs\u002Fnlu\u002Fblob\u002Fmaster\u002Fexamples\u002Fcolab\u002Fcomponent_examples\u002Fsentence_embeddings\u002Fsentence_similarirty_stack_overflow_questions.ipynb) | [Bert-Paper](https:\u002F\u002Farxiv.org\u002Fabs\u002F1810.04805), [Bert Github](https:\u002F\u002Fgithub.com\u002Fgoogle-research\u002Fbert), [Bert-Sentence_Embedding](https:\u002F\u002Fnlp.johnsnowlabs.com\u002F2020\u002F08\u002F25\u002Fsent_small_bert_L2_128.html) |\n| Part of Speech tagging with NLU | `pos` |[![Open In Colab](https:\u002F\u002Fcolab.research.google.com\u002Fassets\u002Fcolab-badge.svg)](https:\u002F\u002Fcolab.research.google.com\u002Fgithub\u002FMurat-Karadag\u002Fnlu\u002Fblob\u002Ftutorial_docs\u002Fexamples\u002Fcolab\u002Fcomponent_examples\u002Fpart_of_speechPOS\u002FNLU_part_of_speech_ANC_example.ipynb) | [Part of Speech](https:\u002F\u002Fnlp.johnsnowlabs.com\u002F2021\u002F03\u002F05\u002Fpos_anc.html) | \n| NER Aspect Airline ATIS | `en.ner.aspect.airline` |[![Open In Colab](https:\u002F\u002Fcolab.research.google.com\u002Fassets\u002Fcolab-badge.svg)](https:\u002F\u002Fcolab.research.google.com\u002Fgithub\u002FMurat-Karadag\u002Fnlu\u002Fblob\u002Ftutorial_docs\u002Fexamples\u002Fcolab\u002Fcomponent_examples\u002Fnamed_entity_recognition_NER\u002FNER_aspect_airline_ATIS.ipynb) | [NER Airline Model](https:\u002F\u002Fnlp.johnsnowlabs.com\u002F2021\u002F01\u002F25\u002Fnerdl_atis_840b_300d_en.html), [Atis intent Dataset](https:\u002F\u002Fwww.kaggle.com\u002Fhassanamin\u002Fatis-airlinetravelinformationsystem) | \n| NLU-NER_CONLL_2003_5class_example | `ner` |[![Open In Colab](https:\u002F\u002Fcolab.research.google.com\u002Fassets\u002Fcolab-badge.svg)](https:\u002F\u002Fcolab.research.google.com\u002Fgithub\u002FMurat-Karadag\u002Fnlu\u002Fblob\u002Ftutorial_docs\u002Fexamples\u002Fcolab\u002Fcomponent_examples\u002Fnamed_entity_recognition_NER\u002FNLU_ner_CONLL_2003_5class_example.ipynb) | [NER-Piple](https:\u002F\u002Fnlp.johnsnowlabs.com\u002F2021\u002F03\u002F22\u002Fonto_recognize_entities_sm_en.html) |\n| Named-entity recognition with Deep Learning ONTO NOTES | `ner.onto` |[![Open In Colab](https:\u002F\u002Fcolab.research.google.com\u002Fassets\u002Fcolab-badge.svg)](https:\u002F\u002Fcolab.research.google.com\u002Fgithub\u002FMurat-Karadag\u002Fnlu\u002Fblob\u002Ftutorial_docs\u002Fexamples\u002Fcolab\u002Fcomponent_examples\u002Fnamed_entity_recognition_NER\u002FNLU_ner_ONTO_18class_example.ipynb) | [NER_Onto](https:\u002F\u002Fnlp.johnsnowlabs.com\u002F2021\u002F03\u002F22\u002Fonto_recognize_entities_sm_en.html) |\n| Aspect based NER-Sentiment-Restaurants | `en.ner.aspect_sentiment` |[![Open In Colab](https:\u002F\u002Fcolab.research.google.com\u002Fassets\u002Fcolab-badge.svg)](https:\u002F\u002Fcolab.research.google.com\u002Fgithub\u002FMurat-Karadag\u002Fnlu\u002Fblob\u002Ftutorial_docs\u002Fexamples\u002Fcolab\u002Fcomponent_examples\u002Fnamed_entity_recognition_NER\u002Faspect_based_ner_sentiment_restaurants.ipynb) | - |\n| Detect Named Entities (NER), Part of Speech Tags (POS) and Tokenize in Chinese | `zh.segment_words`, `zh.pos`, `zh.ner`, `zh.translate_to.en` | [![Open In Colab](https:\u002F\u002Fcolab.research.google.com\u002Fassets\u002Fcolab-badge.svg)](https:\u002F\u002Fcolab.research.google.com\u002Fgithub\u002FJohnSnowLabs\u002Fnlu\u002Fblob\u002Fmaster\u002Fexamples\u002Fcolab\u002Fcomponent_examples\u002Fmultilingual\u002Fchinese_ner_pos_and_tokenization.ipynb) | [Translation-Pipeline (Zh to En)](https:\u002F\u002Fnlp.johnsnowlabs.com\u002F2021\u002F06\u002F04\u002Ftranslate_zh_en_xx.html) |\n| Detect Named Entities (NER), Part of Speech Tags (POS) and Tokenize in Japanese | `ja.segment_words`, `ja.pos`, `ja.ner`, `ja.translate_to.en` | [![Open In Colab](https:\u002F\u002Fcolab.research.google.com\u002Fassets\u002Fcolab-badge.svg)](https:\u002F\u002Fcolab.research.google.com\u002Fgithub\u002FJohnSnowLabs\u002Fnlu\u002Fblob\u002Fmaster\u002Fexamples\u002Fcolab\u002Fcomponent_examples\u002Fmultilingual\u002Fjapanese_ner_pos_and_tokenization.ipynb) | [Translation-Pipeline (Ja to En)](https:\u002F\u002Fnlp.johnsnowlabs.com\u002F2021\u002F06\u002F04\u002Ftranslate_ja_en_xx.html) |\n| Detect Named Entities (NER), Part of Speech Tags (POS) and Tokenize in Korean | `ko.segment_words`, `ko.pos`, `ko.ner.kmou.glove_840B_300d`, `ko.translate_to.en` | [![Open In Colab](https:\u002F\u002Fcolab.research.google.com\u002Fassets\u002Fcolab-badge.svg)](https:\u002F\u002Fcolab.research.google.com\u002Fgithub\u002FMurat-Karadag\u002Fnlu\u002Fblob\u002Fmaster\u002Fexamples\u002Fcolab\u002Fcomponent_examples\u002Fmultilingual\u002Fkorean_ner_pos_and_tokenization.ipynb) | - |\n| Date Matching | `match.datetime` | [![Open In Colab](https:\u002F\u002Fcolab.research.google.com\u002Fassets\u002Fcolab-badge.svg)](https:\u002F\u002Fcolab.research.google.com\u002Fgithub\u002FJohnSnowLabs\u002Fnlu\u002Fblob\u002Fmaster\u002Fexamples\u002Fcolab\u002Fcomponent_examples\u002Fmatchers\u002FNLU_date_matching.ipynb) | - |\n| Typed Dependency Parsing with NLU | `dep` | [![Open In Colab](https:\u002F\u002Fcolab.research.google.com\u002Fassets\u002Fcolab-badge.svg)](https:\u002F\u002Fcolab.research.google.com\u002Fgithub\u002FJohnSnowLabs\u002Fnlu\u002Fblob\u002Fmaster\u002Fexamples\u002Fcolab\u002Fcomponent_examples\u002Fdependency_parsing\u002FNLU_typed_dependency_parsing_example.ipynb) | [Dependency Parsing ](https:\u002F\u002Fnlp.johnsnowlabs.com\u002F2021\u002F03\u002F27\u002FTyped_Dependency_Parsing_en.html) |\n| Untyped Dependency Parsing with NLU | `dep.untyped` | [![Open In Colab](https:\u002F\u002Fcolab.research.google.com\u002Fassets\u002Fcolab-badge.svg)](https:\u002F\u002Fcolab.research.google.com\u002Fgithub\u002FJohnSnowLabs\u002Fnlu\u002Fblob\u002Fmaster\u002Fexamples\u002Fcolab\u002Fcomponent_examples\u002Fdependency_parsing\u002FNLU_untyped_dependency_parsing_example.ipynb) | - |\n| E2E Classification with NLU | `e2e` | [![Open In Colab](https:\u002F\u002Fcolab.research.google.com\u002Fassets\u002Fcolab-badge.svg)](https:\u002F\u002Fcolab.research.google.com\u002Fgithub\u002FJohnSnowLabs\u002Fnlu\u002Fblob\u002Fmaster\u002Fexamples\u002Fcolab\u002Fcomponent_examples\u002Fclassifiers\u002FE2E_classification.ipynb) | [e2e-Model](https:\u002F\u002Fnlp.johnsnowlabs.com\u002F2021\u002F01\u002F21\u002Fmulticlassifierdl_use_e2e_en.html) |\n| Language Classification with NLU | `lang` | [![Open In Colab](https:\u002F\u002Fcolab.research.google.com\u002Fassets\u002Fcolab-badge.svg)](https:\u002F\u002Fcolab.research.google.com\u002Fgithub\u002FJohnSnowLabs\u002Fnlu\u002Fblob\u002Fmaster\u002Fexamples\u002Fcolab\u002Fcomponent_examples\u002Fclassifiers\u002FNLU_language_classification.ipynb) | - |\n| Cyberbullying Classification with NLU | `classify.cyberbullying` | [![Open In Colab](https:\u002F\u002Fcolab.research.google.com\u002Fassets\u002Fcolab-badge.svg)](https:\u002F\u002Fcolab.research.google.com\u002Fgithub\u002FJohnSnowLabs\u002Fnlu\u002Fblob\u002Fmaster\u002Fexamples\u002Fcolab\u002Fcomponent_examples\u002Fclassifiers\u002Fcyberbullying_cassification_for_racism_and_sexism.ipynb) | [Cyberbullying-Classifier](https:\u002F\u002Fnlp.johnsnowlabs.com\u002F2021\u002F01\u002F09\u002Fclassifierdl_use_cyberbullying_en.html) |\n| Sentiment Classification with NLU for Twitter | `emotion` | [![Open In Colab](https:\u002F\u002Fcolab.research.google.com\u002Fassets\u002Fcolab-badge.svg)](https:\u002F\u002Fcolab.research.google.com\u002Fgithub\u002FJohnSnowLabs\u002Fnlu\u002Fblob\u002Fmaster\u002Fexamples\u002Fcolab\u002Fcomponent_examples\u002Fclassifiers\u002Femotion_classification.ipynb) | [Emotion detection](https:\u002F\u002Fnlp.johnsnowlabs.com\u002F2021\u002F01\u002F09\u002Fclassifierdl_use_emotion_en.html) |\n| Fake News Classification with NLU | `en.classify.fakenews` | [![Open In Colab](https:\u002F\u002Fcolab.research.google.com\u002Fassets\u002Fcolab-badge.svg)](https:\u002F\u002Fcolab.research.google.com\u002Fgithub\u002FJohnSnowLabs\u002Fnlu\u002Fblob\u002Fmaster\u002Fexamples\u002Fcolab\u002Fcomponent_examples\u002Fclassifiers\u002Ffake_news_classification.ipynb) | [Fakenews-Classifier](https:\u002F\u002Fnlp.johnsnowlabs.com\u002F2021\u002F01\u002F09\u002Fclassifierdl_use_fakenews_en.html) |\n| Intent Classification with NLU | `en.classify.intent.airline` | [![Open In Colab](https:\u002F\u002Fcolab.research.google.com\u002Fassets\u002Fcolab-badge.svg)](https:\u002F\u002Fcolab.research.google.com\u002Fgithub\u002FJohnSnowLabs\u002Fnlu\u002Fblob\u002Fmaster\u002Fexamples\u002Fcolab\u002Fcomponent_examples\u002Fclassifiers\u002Fintent_classification_airlines_ATIS.ipynb) | [Airline-Intention classifier](https:\u002F\u002Fnlp.johnsnowlabs.com\u002F2021\u002F01\u002F25\u002Fclassifierdl_use_atis_en.html), [Atis-Dataset](https:\u002F\u002Fwww.kaggle.com\u002Fhassanamin\u002Fatis-airlinetravelinformationsystem?select=atis_intents.csv) |\n| Question classification based on the TREC dataset | `en.classify.questions` | [![Open In Colab](https:\u002F\u002Fcolab.research.google.com\u002Fassets\u002Fcolab-badge.svg)](https:\u002F\u002Fcolab.research.google.com\u002Fgithub\u002FJohnSnowLabs\u002Fnlu\u002Fblob\u002Fmaster\u002Fexamples\u002Fcolab\u002Fcomponent_examples\u002Fclassifiers\u002Fquestion_classification.ipynb) | [Question-Classifier](https:\u002F\u002Fnlp.johnsnowlabs.com\u002F2021\u002F01\u002F08\u002Fclassifierdl_use_trec50_en.html) |\n| Sarcasm Classification with NLU | `en.classify.sarcasm` | [![Open In Colab](https:\u002F\u002Fcolab.research.google.com\u002Fassets\u002Fcolab-badge.svg)](https:\u002F\u002Fcolab.research.google.com\u002Fgithub\u002FJohnSnowLabs\u002Fnlu\u002Fblob\u002Fmaster\u002Fexamples\u002Fcolab\u002Fcomponent_examples\u002Fclassifiers\u002Fsarcasm_classification.ipynb) | [Sarcasm-Classifier](https:\u002F\u002Fnlp.johnsnowlabs.com\u002F2021\u002F01\u002F09\u002Fclassifierdl_use_sarcasm_en.html) |\n| Sentiment Classification with NLU for Twitter | `en.sentiment.twitter` | [![Open In Colab](https:\u002F\u002Fcolab.research.google.com\u002Fassets\u002Fcolab-badge.svg)](https:\u002F\u002Fcolab.research.google.com\u002Fgithub\u002FJohnSnowLabs\u002Fnlu\u002Fblob\u002Fmaster\u002Fexamples\u002Fcolab\u002Fcomponent_examples\u002Fclassifiers\u002Fsentiment_classification.ipynb) | [Sentiment_Twitter-Classifier](https:\u002F\u002Fnlp.johnsnowlabs.com\u002F2021\u002F01\u002F18\u002Fsentimentdl_use_twitter_en.html) |\n| Sentiment Classification with NLU for Movies | `en.sentiment.imdb` | [![Open In Colab](https:\u002F\u002Fcolab.research.google.com\u002Fassets\u002Fcolab-badge.svg)](https:\u002F\u002Fcolab.research.google.com\u002Fgithub\u002FJohnSnowLabs\u002Fnlu\u002Fblob\u002Fmaster\u002Fexamples\u002Fcolab\u002Fcomponent_examples\u002Fclassifiers\u002Fsentiment_classification_movies.ipynb) | [Sentiment_imdb-Classifier](https:\u002F\u002Fnlp.johnsnowlabs.com\u002F2021\u002F01\u002F15\u002Fanalyze_sentimentdl_use_imdb_en.html) |\n| Spam Classification with NLU | `en.classify.spam` | [![Open In Colab](https:\u002F\u002Fcolab.research.google.com\u002Fassets\u002Fcolab-badge.svg)](https:\u002F\u002Fcolab.research.google.com\u002Fgithub\u002FJohnSnowLabs\u002Fnlu\u002Fblob\u002Fmaster\u002Fexamples\u002Fcolab\u002Fcomponent_examples\u002Fclassifiers\u002Fspam_classification.ipynb) | [Spam-Classifier](https:\u002F\u002Fnlp.johnsnowlabs.com\u002F2021\u002F01\u002F09\u002Fclassifierdl_use_spam_en.html) |\n| Toxic text classification with NLU | `en.classify.toxic` | [![Open In Colab](https:\u002F\u002Fcolab.research.google.com\u002Fassets\u002Fcolab-badge.svg)](https:\u002F\u002Fcolab.research.google.com\u002Fgithub\u002FJohnSnowLabs\u002Fnlu\u002Fblob\u002Fmaster\u002Fexamples\u002Fcolab\u002Fcomponent_examples\u002Fclassifiers\u002Ftoxic_classification.ipynb) | [Toxic-Classifier](https:\u002F\u002Fnlp.johnsnowlabs.com\u002F2021\u002F01\u002F21\u002Fmulticlassifierdl_use_toxic_en.html) |\n| Unsupervised keyword extraction with NLU using the YAKE algorithm | `yake` | [![Open In Colab](https:\u002F\u002Fcolab.research.google.com\u002Fassets\u002Fcolab-badge.svg)](https:\u002F\u002Fcolab.research.google.com\u002Fgithub\u002FJohnSnowLabs\u002Fnlu\u002Fblob\u002Fmaster\u002Fexamples\u002Fcolab\u002Fcomponent_examples\u002Fclassifiers\u002Funsupervised_keyword_extraction_with_YAKE.ipynb) | - |\n| Grammatical Chunk Matching with NLU | `match.chunks` | [![Open In Colab](https:\u002F\u002Fcolab.research.google.com\u002Fassets\u002Fcolab-badge.svg)](https:\u002F\u002Fcolab.research.google.com\u002Fgithub\u002FJohnSnowLabs\u002Fnlu\u002Fblob\u002Fmaster\u002Fexamples\u002Fcolab\u002Fcomponent_examples\u002Fchunkers\u002FNLU_chunking_example.ipynb) | - |\n| Getting n-Grams with NLU | `ngram` | [![Open In Colab](https:\u002F\u002Fcolab.research.google.com\u002Fassets\u002Fcolab-badge.svg)](https:\u002F\u002Fcolab.research.google.com\u002Fgithub\u002FJohnSnowLabs\u002Fnlu\u002Fblob\u002Fmaster\u002Fexamples\u002Fcolab\u002Fcomponent_examples\u002Fchunkers\u002FNLU_n-gram.ipynb) | - |\n| Assertion | `en.med_ner.clinical en.assert`, `en.med_ner.clinical.biobert en.assert.biobert`, ... | [![Open In Colab](https:\u002F\u002Fcolab.research.google.com\u002Fassets\u002Fcolab-badge.svg)](https:\u002F\u002Fcolab.research.google.com\u002Fgithub\u002FJohnSnowLabs\u002Fnlu\u002Fblob\u002Fmaster\u002Fexamples\u002Fcolab\u002Fhealthcare\u002Fassertion\u002Fassertion_overview.ipynb) | [Healthcare-NER](https:\u002F\u002Fnlp.johnsnowlabs.com\u002F2021\u002F01\u002F27\u002Fner_clinical_en.html), [NER_Clinical-Classifier]( https:\u002F\u002Fnlp.johnsnowlabs.com\u002F2021\u002F11\u002F03\u002Fner_profiling_biobert_en.html), [Toxic-Classifier](https:\u002F\u002Fnlp.johnsnowlabs.com\u002F2021\u002F01\u002F26\u002Fassertion_dl_biobert_en.html) |\n| De-Identification Model overview | `med_ner.jsl.wip.clinical en.de_identify`, `med_ner.jsl.wip.clinical en.de_identify.clinical`, ... | [![Open In Colab](https:\u002F\u002Fcolab.research.google.com\u002Fassets\u002Fcolab-badge.svg)](https:\u002F\u002Fcolab.research.google.com\u002Fgithub\u002FJohnSnowLabs\u002Fnlu\u002Fblob\u002Fmaster\u002Fexamples\u002Fcolab\u002Fhealthcare\u002Fde_identification\u002FDeIdentification_model_overview.ipynb) | [NER-Clinical](https:\u002F\u002Fnlp.johnsnowlabs.com\u002F2021\u002F11\u002F03\u002Fner_profiling_clinical_en.html) |\n| Drug Normalization | `norm_drugs` | [![Open In Colab](https:\u002F\u002Fcolab.research.google.com\u002Fassets\u002Fcolab-badge.svg)](https:\u002F\u002Fcolab.research.google.com\u002Fgithub\u002FJohnSnowLabs\u002Fnlu\u002Fblob\u002Fmaster\u002Fexamples\u002Fcolab\u002Fhealthcare\u002F\u002Fdrug_normalization\u002Fdrug_norm.ipynb) | - | \n| Entity Resolution | `med_ner.jsl.wip.clinical en.resolve_chunk.cpt_clinical`, `med_ner.jsl.wip.clinical en.resolve.icd10cm`, ... | [![Open In Colab](https:\u002F\u002Fcolab.research.google.com\u002Fassets\u002Fcolab-badge.svg)](https:\u002F\u002Fcolab.research.google.com\u002Fgithub\u002FJohnSnowLabs\u002Fnlu\u002Fblob\u002Fmaster\u002Fexamples\u002Fcolab\u002Fhealthcare\u002Fentity_resolution\u002Fentity_resolvers_overview.ipynb) | [NER-Clinical](https:\u002F\u002Fnlp.johnsnowlabs.com\u002F2021\u002F11\u002F03\u002Fner_profiling_clinical_en.html), [Entity-Resolver clinical](https:\u002F\u002Fnlp.johnsnowlabs.com\u002F2021\u002F11\u002F01\u002Fsbiobertresolve_icd10cm_augmented_billable_hcc_en.html) |\n| Medical Named Entity Recognition | `en.med_ner.ade.clinical`, `en.med_ner.ade.clinical_bert`, `en.med_ner.anatomy`,`en.med_ner.anatomy.biobert`, ... | [![Open In Colab](https:\u002F\u002Fcolab.research.google.com\u002Fassets\u002Fcolab-badge.svg)](https:\u002F\u002Fcolab.research.google.com\u002Fgithub\u002FJohnSnowLabs\u002Fnlu\u002Fblob\u002Fmaster\u002Fexamples\u002Fcolab\u002Fhealthcare\u002Fmedical_named_entity_recognition\u002Foverview_medical_entity_recognizers.ipynb) | - |\n| Relation Extraction | `en.med_ner.jsl.wip.clinical.greedy en.relation`, `en.med_ner.jsl.wip.clinical.greedy en.relation.bodypart.problem`, ... | [![Open In Colab](https:\u002F\u002Fcolab.research.google.com\u002Fassets\u002Fcolab-badge.svg)](https:\u002F\u002Fcolab.research.google.com\u002Fgithub\u002FJohnSnowLabs\u002Fnlu\u002Fblob\u002Fmaster\u002Fexamples\u002Fcolab\u002Fhealthcare\u002Frelation_extraction\u002Foverview_relation.ipynb) | - |\n| Visualization of NLP-Models with Spark-NLP and NLU | `ner`, `dep.typed`, `med_ner.jsl.wip.clinical resolve_chunk.rxnorm.in`, `med_ner.jsl.wip.clinical resolve.icd10cm` | [![Open In Colab](https:\u002F\u002Fcolab.research.google.com\u002Fassets\u002Fcolab-badge.svg)](https:\u002F\u002Fcolab.research.google.com\u002Fgithub\u002FJohnSnowLabs\u002Fnlu\u002Fblob\u002Fmaster\u002Fexamples\u002Fcolab\u002Fvisualization\u002FNLU_visualizations_tutorial.ipynb) | [NER-Piple](https:\u002F\u002Fnlp.johnsnowlabs.com\u002F2021\u002F03\u002F22\u002Fonto_recognize_entities_sm_en.html), [Dependency Parsing](https:\u002F\u002Fnlp.johnsnowlabs.com\u002F2021\u002F03\u002F27\u002FTyped_Dependency_Parsing_en.html), [NER-Clinical](https:\u002F\u002Fnlp.johnsnowlabs.com\u002F2021\u002F11\u002F03\u002Fner_profiling_clinical_en.html), [Entity-Resolver (Chunks) clinical](https:\u002F\u002Fnlp.johnsnowlabs.com\u002F2021\u002F04\u002F16\u002Fchunkresolve_rxnorm_in_clinical_en.html) |\n| NLU Covid-19 Emotion Showcase | `emotion` | [![Open In GitHub]()](https:\u002F\u002Fgithub.com\u002FJohnSnowLabs\u002Fnlu\u002Fblob\u002Fmaster\u002Fexamples\u002Fkaggle\u002Fnlu_covid_emotion_showcase.ipynb) | [Emotion detection](https:\u002F\u002Fnlp.johnsnowlabs.com\u002F2021\u002F01\u002F09\u002Fclassifierdl_use_emotion_en.html) |\n| NLU Covid-19 Sentiment Showcase | `sentiment` | [![Open In GitHub]()](https:\u002F\u002Fgithub.com\u002FJohnSnowLabs\u002Fnlu\u002Fblob\u002Fmaster\u002Fexamples\u002Fkaggle\u002Fnlu_covid_sentiment_showcase.ipynb) | [Sentiment classification](https:\u002F\u002Fnlp.johnsnowlabs.com\u002F2021\u002F03\u002F24\u002Fanalyze_sentiment_en.html) |\n| NLU Airline Emotion Demo | `emotion` | [![Open In GitHub]()](https:\u002F\u002Fgithub.com\u002FJohnSnowLabs\u002Fnlu\u002Fblob\u002Fmaster\u002Fexamples\u002Fkaggle\u002Fnlu_emotion_airline_demo.ipynb) | [Emotion detection](https:\u002F\u002Fnlp.johnsnowlabs.com\u002F2021\u002F01\u002F09\u002Fclassifierdl_use_emotion_en.html) |\n| NLU Airline Sentiment Demo | `sentiment` | [![Open In GitHub]()](https:\u002F\u002Fgithub.com\u002FJohnSnowLabs\u002Fnlu\u002Fblob\u002Fmaster\u002Fexamples\u002Fkaggle\u002Fnlu_sentiment_airline_demo.ipynb) | [Sentiment classification](https:\u002F\u002Fnlp.johnsnowlabs.com\u002F2021\u002F03\u002F24\u002Fanalyze_sentiment_en.html) |\n| Bengali NER Hindi Embeddings for 30 Models | `bn.ner`, `bn.lemma`, `ja.lemma`, `am.lemma`, `bh.lemma`,` en.ner.onto.bert.small_l2_128`,.. | [![Open In Colab](https:\u002F\u002Fcolab.research.google.com\u002Fassets\u002Fcolab-badge.svg)](https:\u002F\u002Fcolab.research.google.com\u002Fgithub\u002FMurat-Karadag\u002Fnlu\u002Fblob\u002Fmaster\u002Fexamples\u002Frelease_notebooks\u002FNLU1.1.2_Bengali_ner_Hindi_Embeddings_30_new_models.ipynb) | [Bengali-NER](https:\u002F\u002Fnlp.johnsnowlabs.com\u002F2021\u002F01\u002F27\u002Fner_jifs_glove_840B_300d_bn.html), [Bengali-Lemmatizer](https:\u002F\u002Fnlp.johnsnowlabs.com\u002F2021\u002F01\u002F20\u002Flemma_bn.html), [Japanese-Lemmatizer](https:\u002F\u002Fnlp.johnsnowlabs.com\u002F2021\u002F01\u002F15\u002Flemma_ja.html), [Amharic-Lemmatizer](https:\u002F\u002Fnlp.johnsnowlabs.com\u002F2021\u002F01\u002F20\u002Flemma_am.html) |\n| Entity Resolution | `med_ner.jsl.wip.clinical en.resolve.umls`, `med_ner.jsl.wip.clinical en.resolve.loinc`, `med_ner.jsl.wip.clinical en.resolve.loinc.biobert` | [![Open In Colab](https:\u002F\u002Fcolab.research.google.com\u002Fassets\u002Fcolab-badge.svg)](https:\u002F\u002Fcolab.research.google.com\u002Fgithub\u002FJohnSnowLabs\u002Fnlu\u002Fblob\u002Fmaster\u002Fexamples\u002Frelease_notebooks\u002FNLU_3_0_2_release_notebook.ipynb) | - |\n| NLU 20 Minutes Crashcourse - the fast Data Science route | `spell`, `sentiment`, `pos`, `ner`, `yake`, `en.t5`, `emotion`, `answer_question`, `en.t5.base` ... | [![Open In Colab](https:\u002F\u002Fcolab.research.google.com\u002Fassets\u002Fcolab-badge.svg)](https:\u002F\u002Fcolab.research.google.com\u002Fgithub\u002FJohnSnowLabs\u002Fnlu\u002Fblob\u002Fmaster\u002Fexamples\u002Fwebinars_conferences_etc\u002FAI4_2021\u002FNLU_crash_course_AI4.ipynb) | [T5-Model](https:\u002F\u002Fnlp.johnsnowlabs.com\u002F2021\u002F01\u002F08\u002Ft5_base_en.html), [Part of Speech](https:\u002F\u002Fnlp.johnsnowlabs.com\u002F2021\u002F03\u002F05\u002Fpos_anc.html), [NER-Piple](https:\u002F\u002Fnlp.johnsnowlabs.com\u002F2021\u002F03\u002F22\u002Fonto_recognize_entities_sm_en.html), [Emotion detection](https:\u002F\u002Fnlp.johnsnowlabs.com\u002F2021\u002F01\u002F09\u002Fclassifierdl_use_emotion_en.html) , [Spellchecker](https:\u002F\u002Fnlp.johnsnowlabs.com\u002F2021\u002F03\u002F28\u002Fspellcheck_dl_en.html), [Sentiment classification](https:\u002F\u002Fnlp.johnsnowlabs.com\u002F2021\u002F03\u002F24\u002Fanalyze_sentiment_en.html) |\n| Chapter 0: Intro: 1-liners | `sentiment`, `pos`, `ner`, `bert`, `elmo`, `embed_sentence.bert` | [![Open In Colab](https:\u002F\u002Fcolab.research.google.com\u002Fassets\u002Fcolab-badge.svg)](https:\u002F\u002Fcolab.research.google.com\u002Fgithub\u002FJohnSnowLabs\u002Fnlu\u002Fblob\u002Fmaster\u002Fexamples\u002Fwebinars_conferences_etc\u002FNYC_DC_NLP_MEETUP\u002F0_liners_intro.ipynb) | [Part of Speech](https:\u002F\u002Fnlp.johnsnowlabs.com\u002F2021\u002F03\u002F05\u002Fpos_anc.html), [NER-Piple](https:\u002F\u002Fnlp.johnsnowlabs.com\u002F2021\u002F03\u002F22\u002Fonto_recognize_entities_sm_en.html), [Sentiment classification](https:\u002F\u002Fnlp.johnsnowlabs.com\u002F2021\u002F03\u002F24\u002Fanalyze_sentiment_en.html), [Elmo-Embedding](https:\u002F\u002Fnlp.johnsnowlabs.com\u002F2020\u002F01\u002F31\u002Felmo.html), [Bert-Sentence_Embedding](https:\u002F\u002Fnlp.johnsnowlabs.com\u002F2020\u002F08\u002F25\u002Fsent_small_bert_L2_128.html) |\n| Chapter 1: NLU base-features with some classifiers on testdata | `emotion`, `yake`, `stem` | [![Open In Colab](https:\u002F\u002Fcolab.research.google.com\u002Fassets\u002Fcolab-badge.svg)](https:\u002F\u002Fcolab.research.google.com\u002Fgithub\u002FJohnSnowLabs\u002Fnlu\u002Fblob\u002Fmaster\u002Fexamples\u002Fwebinars_conferences_etc\u002FNYC_DC_NLP_MEETUP\u002F1_NLU_base_features_on_dataset_with_YAKE_Lemma_Stemm_classifiers_NER_.ipynb) | [Emotion detection](https:\u002F\u002Fnlp.johnsnowlabs.com\u002F2021\u002F01\u002F09\u002Fclassifierdl_use_emotion_en.html) |\n| Chapter 2: Translation between 300+ languages with Marian | `tr.translate_to.en`, `en.translate_to.fr`, `en.translate_to.he` | [![Open In Colab](https:\u002F\u002Fcolab.research.google.com\u002Fassets\u002Fcolab-badge.svg)](https:\u002F\u002Fcolab.research.google.com\u002Fgithub\u002FJohnSnowLabs\u002Fnlu\u002Fblob\u002Fmaster\u002Fexamples\u002Fcolab\u002Fcomponent_examples\u002Fsequence2sequence\u002Ftranslation_demo.ipynb) | [Translation-Pipeline (En to Fr)](https:\u002F\u002Fnlp.johnsnowlabs.com\u002F2021\u002F06\u002F04\u002Ftranslate_en_fr_xx.html), [Translation (En to He)](https:\u002F\u002Fnlp.johnsnowlabs.com\u002F2021\u002F06\u002F04\u002Ftranslate_en_he_xx.html) |\n| Chapter 3: Answer questions and summarize Texts with T5 | `answer_question`, `en.t5`, `en.t5.base` | [![Open In Colab](https:\u002F\u002Fcolab.research.google.com\u002Fassets\u002Fcolab-badge.svg)](https:\u002F\u002Fcolab.research.google.com\u002Fgithub\u002FJohnSnowLabs\u002Fnlu\u002Fblob\u002Fmaster\u002Fexamples\u002Fcolab\u002Fcomponent_examples\u002Fsequence2sequence\u002FT5_question_answering.ipynb) | [T5-Model](https:\u002F\u002Fnlp.johnsnowlabs.com\u002F2021\u002F01\u002F08\u002Ft5_base_en.html) |\n| Chapter 4: Overview of T5-Tasks | `en.t5.base` | [![Open In Colab](https:\u002F\u002Fcolab.research.google.com\u002Fassets\u002Fcolab-badge.svg)](https:\u002F\u002Fcolab.research.google.com\u002Fgithub\u002FJohnSnowLabs\u002Fnlu\u002Fblob\u002Fmaster\u002Fexamples\u002Fcolab\u002Fcomponent_examples\u002Fsequence2sequence\u002FT5_tasks_summarize_question_answering_and_more.ipynb) | [T5-Model](https:\u002F\u002Fnlp.johnsnowlabs.com\u002F2021\u002F01\u002F08\u002Ft5_base_en.html) |\n| Graph NLU 20 Minutes Crashcourse - State of the Art Text Mining for Graphs | `spell`, `sentiment`, `pos`, `ner`, `yake`, `emotion`, `med_ner.jsl.wip.clinical`, ... | [![Open In Colab](https:\u002F\u002Fcolab.research.google.com\u002Fassets\u002Fcolab-badge.svg)](https:\u002F\u002Fcolab.research.google.com\u002Fgithub\u002FJohnSnowLabs\u002Fnlu\u002Fblob\u002Fmaster\u002Fexamples\u002Fwebinars_conferences_etc\u002Fgraph_ai_summit\u002FHealthcare_Graph_NLU_COVID_Tigergraph.ipynb) | [Part of Speech](https:\u002F\u002Fnlp.johnsnowlabs.com\u002F2021\u002F03\u002F05\u002Fpos_anc.html), [NER-Piple](https:\u002F\u002Fnlp.johnsnowlabs.com\u002F2021\u002F03\u002F22\u002Fonto_recognize_entities_sm_en.html), [Emotion detection](https:\u002F\u002Fnlp.johnsnowlabs.com\u002F2021\u002F01\u002F09\u002Fclassifierdl_use_emotion_en.html), [Spellchecker](https:\u002F\u002Fnlp.johnsnowlabs.com\u002F2021\u002F03\u002F28\u002Fspellcheck_dl_en.html), [Sentiment classification](https:\u002F\u002Fnlp.johnsnowlabs.com\u002F2021\u002F03\u002F24\u002Fanalyze_sentiment_en.html) |\n| Healthcare with NLU | `med_ner.human_phenotype.gene_biobert`, `med_ner.ade_biobert`, `med_ner.anatomy`, `med_ner.bacterial_species`,... | [![Open In Colab](https:\u002F\u002Fcolab.research.google.com\u002Fassets\u002Fcolab-badge.svg)](https:\u002F\u002Fcolab.research.google.com\u002Fgithub\u002FJohnSnowLabs\u002Fnlu\u002Fblob\u002Fmaster\u002Fexamples\u002Fwebinars_conferences_etc\u002Fhealthcare_webinar\u002FNLU_healthcare_webinar.ipynb) | - |\n| Part 0: Intro: 1-liners | `spell`, `sentiment`, `pos`, `ner`, `bert`, `elmo`, `embed_sentence.bert` | [![Open In Colab](https:\u002F\u002Fcolab.research.google.com\u002Fassets\u002Fcolab-badge.svg)](https:\u002F\u002Fcolab.research.google.com\u002Fgithub\u002FJohnSnowLabs\u002Fnlu\u002Fblob\u002Fmaster\u002Fexamples\u002Fwebinars_conferences_etc\u002Fmulti_lingual_webinar\u002F0_liners_intro.ipynb) | [Bert-Paper](https:\u002F\u002Farxiv.org\u002Fabs\u002F1810.04805), [Bert Github](https:\u002F\u002Fgithub.com\u002Fgoogle-research\u002Fbert), [T-SNE](https:\u002F\u002Fwww.jmlr.org\u002Fpapers\u002Fvolume9\u002Fvandermaaten08a\u002Fvandermaaten08a.pdf?fbclid=IwA), [T-SNE-Bert](https:\u002F\u002Fmedium.com\u002Fspark-nlp\u002F1-line-to-bert-word-embeddings-with-nlu-f50d2b08cddc) , [Part of Speech](https:\u002F\u002Fnlp.johnsnowlabs.com\u002F2021\u002F03\u002F05\u002Fpos_anc.html), [NER-Piple](https:\u002F\u002Fnlp.johnsnowlabs.com\u002F2021\u002F03\u002F22\u002Fonto_recognize_entities_sm_en.html), [Spellchecker](https:\u002F\u002Fnlp.johnsnowlabs.com\u002F2021\u002F03\u002F28\u002Fspellcheck_dl_en.html), [Sentiment classification](https:\u002F\u002Fnlp.johnsnowlabs.com\u002F2021\u002F03\u002F24\u002Fanalyze_sentiment_en.html), [Elmo-Embedding](https:\u002F\u002Fnlp.johnsnowlabs.com\u002F2020\u002F01\u002F31\u002Felmo.html) , [Bert-Sentence_Embedding](https:\u002F\u002Fnlp.johnsnowlabs.com\u002F2020\u002F08\u002F25\u002Fsent_small_bert_L2_128.html) | \n| Part 1: NLU base-features with some classifiers on Testdata | `yake`, `stem`, `ner`, `emotion` | [![Open In Colab](https:\u002F\u002Fcolab.research.google.com\u002Fassets\u002Fcolab-badge.svg)](https:\u002F\u002Fcolab.research.google.com\u002Fgithub\u002FJohnSnowLabs\u002Fnlu\u002Fblob\u002Fmaster\u002Fexamples\u002Fwebinars_conferences_etc\u002Fmulti_lingual_webinar\u002F1_NLU_base_features_on_dataset_with_YAKE_Lemma_Stemm_classifiers_NER_.ipynb) | [NER-Piple](https:\u002F\u002Fnlp.johnsnowlabs.com\u002F2021\u002F03\u002F22\u002Fonto_recognize_entities_sm_en.html), [Emotion detection](https:\u002F\u002Fnlp.johnsnowlabs.com\u002F2021\u002F01\u002F09\u002Fclassifierdl_use_emotion_en.html) | \n| Part 2: Translate between 200+ Languages in 1 line of code with Marian-Models | `en.translate_to.de`, `en.translate_to.fr`, `en.translate_to.he` | [![Open In Colab](https:\u002F\u002Fcolab.research.google.com\u002Fassets\u002Fcolab-badge.svg)](https:\u002F\u002Fcolab.research.google.com\u002Fgithub\u002FJohnSnowLabs\u002Fnlu\u002Fblob\u002Fmaster\u002Fexamples\u002Fwebinars_conferences_etc\u002Fmulti_lingual_webinar\u002F2_multilingual_translation_with_marian_intro.ipynb) | [Translation-Pipeline (En to Fr)](https:\u002F\u002Fnlp.johnsnowlabs.com\u002F2021\u002F06\u002F04\u002Ftranslate_en_fr_xx.html), [Translation-Pipeline (En to Ger)](https:\u002F\u002Fnlp.johnsnowlabs.com\u002F2021\u002F06\u002F04\u002Ftranslate_en_de_xx.html), [Translation (En to He)](https:\u002F\u002Fnlp.johnsnowlabs.com\u002F2021\u002F06\u002F04\u002Ftranslate_en_he_xx.html) | \n| Part 3: More Multilingual NLP-translations for Asian Languages with Marian | `en.translate_to.hi`, `en.translate_to.ru`, `en.translate_to.zh` | [![Open In Colab](https:\u002F\u002Fcolab.research.google.com\u002Fassets\u002Fcolab-badge.svg)](https:\u002F\u002Fcolab.research.google.com\u002Fgithub\u002FJohnSnowLabs\u002Fnlu\u002Fblob\u002Fmaster\u002Fexamples\u002Fwebinars_conferences_etc\u002Fmulti_lingual_webinar\u002F3_more_multi_lingual_NLP_translation_Asian_languages_with_Marian.ipynb) | [Translation (En to Hi)](https:\u002F\u002Fnlp.johnsnowlabs.com\u002F2021\u002F06\u002F04\u002Ftranslate_en_hil_xx.html), [Translation (En to Ru)](https:\u002F\u002Fnlp.johnsnowlabs.com\u002F2021\u002F06\u002F04\u002Ftranslate_en_run_xx.html), [Translation (En to Zh)](https:\u002F\u002Fnlp.johnsnowlabs.com\u002F2021\u002F06\u002F04\u002Ftranslate_en_zh_xx.html) | \n| Part 4: Unsupervise Chinese Keyword Extraction, NER and Translation from chinese news | `zh.translate_to.en`, `zh.segment_words`, `yake`, `zh.lemma`, `zh.ner` | [![Open In Colab](https:\u002F\u002Fcolab.research.google.com\u002Fassets\u002Fcolab-badge.svg)](https:\u002F\u002Fcolab.research.google.com\u002Fgithub\u002FJohnSnowLabs\u002Fnlu\u002Fblob\u002Fmaster\u002Fexamples\u002Fwebinars_conferences_etc\u002Fmulti_lingual_webinar\u002F4_Unsupervise_Chinese_Keyword_Extraction_NER_and_Translation_from_Chinese_News.ipynb) | [Translation-Pipeline (Zh to En)](https:\u002F\u002Fnlp.johnsnowlabs.com\u002F2021\u002F06\u002F04\u002Ftranslate_zh_en_xx.html), [Zh-Lemmatizer](https:\u002F\u002Fnlp.johnsnowlabs.com\u002F2020\u002F03\u002F19\u002Fexplain_document_dl.html) | \n| Part 5: Multilingual sentiment classifier training for 100+ languages | `train.sentiment`, `xx.embed_sentence.labse train.sentiment` | n.a. | [Sentence_Embedding.Labse](https:\u002F\u002Fnlp.johnsnowlabs.com\u002F2020\u002F09\u002F23\u002Flabse.html) | \n| Part 6: Question-answering and Text-summarization with T5-Modell | `answer_question`, `en.t5`, `en.t5.base` | [![Open In Colab](https:\u002F\u002Fcolab.research.google.com\u002Fassets\u002Fcolab-badge.svg)](https:\u002F\u002Fcolab.research.google.com\u002Fgithub\u002FJohnSnowLabs\u002Fnlu\u002Fblob\u002Fmaster\u002Fexamples\u002Fwebinars_conferences_etc\u002Fmulti_lingual_webinar\u002F6_T5_question_answering_and_Text_summarization.ipynb) | [T5-Paper](https:\u002F\u002Farxiv.org\u002Fpdf\u002F1910.10683.pdf) | \n| Part 7: Overview of all tasks available with T5 | `en.t5.base` | [![Open In Colab](https:\u002F\u002Fcolab.research.google.com\u002Fassets\u002Fcolab-badge.svg)](https:\u002F\u002Fcolab.research.google.com\u002Fgithub\u002FJohnSnowLabs\u002Fnlu\u002Fblob\u002Fmaster\u002Fexamples\u002Fwebinars_conferences_etc\u002Fmulti_lingual_webinar\u002F7_T5_SQUAD_GLUE_SUPER_GLUE_TASKS.ipynb) | [T5-Paper](https:\u002F\u002Farxiv.org\u002Fpdf\u002F1910.10683.pdf) | \n| Part 8: Overview of some of the Multilingual modes with State Of the Art accuracy (1-liner) | `bn.lemma`, `ja.lemma`, `am.lemma`, `bh.lemma`, `zh.segment_words`, ... | [![Open In Colab](https:\u002F\u002Fcolab.research.google.com\u002Fassets\u002Fcolab-badge.svg)](https:\u002F\u002Fcolab.research.google.com\u002Fgithub\u002FJohnSnowLabs\u002Fnlu\u002Fblob\u002Fmaster\u002Fexamples\u002Fwebinars_conferences_etc\u002Fmulti_lingual_webinar\u002F8_Multi_lingual_ner_pos_stop_words_senti) | [Bengali-Lemmatizer](https:\u002F\u002Fnlp.johnsnowlabs.com\u002F2021\u002F01\u002F20\u002Flemma_bn.html), [Japanese-Lemmatizer](https:\u002F\u002Fnlp.johnsnowlabs.com\u002F2021\u002F01\u002F15\u002Flemma_ja.html) , [Amharic-Lemmatizer](https:\u002F\u002Fnlp.johnsnowlabs.com\u002F2021\u002F01\u002F20\u002Flemma_am.html) |\n| Overview of some Multilingual modes avaiable with State Of the Art accuracy (1-liner) | `bn.ner.cc_300d`, `ja.ner`, `zh.ner`, `th.ner.lst20.glove_840B_300D`, `ar.ner` | [![Open In Colab](https:\u002F\u002Fcolab.research.google.com\u002Fassets\u002Fcolab-badge.svg)](https:\u002F\u002Fcolab.research.google.com\u002Fgithub\u002FJohnSnowLabs\u002Fnlu\u002Fblob\u002Fmaster\u002Fexamples\u002Fwebinars_conferences_etc\u002Fpython_web_conf\u002FMulti_Linigual_examples.ipynb) | [Bengali-NER](https:\u002F\u002Fnlp.johnsnowlabs.com\u002F2021\u002F01\u002F27\u002Fner_jifs_glove_840B_300d_bn.html) \n| NLU 20 Minutes Crashcourse - the fast Data Science route | - | [![Open In Colab](https:\u002F\u002Fcolab.research.google.com\u002Fassets\u002Fcolab-badge.svg)](https:\u002F\u002Fcolab.research.google.com\u002Fgithub\u002FJohnSnowLabs\u002Fnlu\u002Fblob\u002Fmaster\u002Fexamples\u002Fwebinars_conferences_etc\u002Fpython_web_conf\u002FNLU_crashcourse_py_web.ipynb) | - |\n\n# 需要帮助吗？ \n- [在 Slack 上联系我们](https:\u002F\u002Fspark-nlp.slack.com\u002Farchives\u002FC0196BQCDPY) \n- [在 Github 上提交问题](https:\u002F\u002Fgithub.com\u002FJohnSnowLabs\u002Fnlu\u002Fissues)\n\n# 简单的 NLU 演示\n- [NLU 不同输出级别的演示](https:\u002F\u002Fcolab.research.google.com\u002Fdrive\u002F1C4N3wpC17YzZf9fXHDNAJ5JvSmfbq7zT?usp=sharing)\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n# NLU 功能概览\n* 分词\n* 可训练的分词\n* 停用词去除\n* 令牌规范化\n* 文档规范化\n* 词干提取器\n* 词形还原器\n* n-gram\n* 正则表达式匹配\n* 文本匹配\n* 分块\n* 日期匹配器\n* 句子检测器\n* 深度句子检测器（深度学习）\n* 依存句法分析（带标签\u002F不带标签）\n* 词性标注\n* 情感分析（机器学习模型）\n* 拼写检查器（机器学习和深度学习模型）\n* 词嵌入（GloVe 和 Word2Vec）\n* BERT 嵌入（TF Hub 模型）\n* ELMO 嵌入（TF Hub 模型）\n* ALBERT 嵌入（TF Hub 模型）\n* XLNet 嵌入\n* 通用句子编码器（TF Hub 模型）\n* BERT 句子嵌入（42 个 TF Hub 模型）\n* 句子嵌入\n* 分块嵌入\n* 无监督关键词提取\n* 语言检测与识别（支持多达 375 种语言）\n* 多分类情感分析（深度学习）\n* 多标签情感分析（深度学习）\n* 多分类文本分类（深度学习）\n* 神经机器翻译\n* 文本到文本转换变压器（Google T5）\n* 命名实体识别（深度学习）\n* 易于 TensorFlow 集成\n* GPU 支持\n* 与 Spark ML 函数完全集成\n* 超过 200 种语言的 1000 个预训练模型！\n* 多语言 NER 模型：阿拉伯语、中文、丹麦语、荷兰语、英语、芬兰语、法语、德语、希伯来语、意大利语、日语、韩语、挪威语、波斯语、波兰语、葡萄牙语、俄语、西班牙语、瑞典语、乌尔都语等\n* 自然语言推理\n* 共指消解\n* 句子补全\n* 词义消歧\n* 临床实体识别\n* 临床实体链接\n* 实体规范化\n* 断言状态检测\n* 去标识化\n* 关系抽取\n* 临床实体消解\n\n\n## 引用\n\n我们发表了一篇关于 NLU 库的论文，您可以引用：\n\n```bibtex\n@article{KOCAMAN2021100058,\n title = {Spark NLP: 大规模自然语言理解},\n journal = {Software Impacts},\n pages = {100058},\n year = {2021},\n issn = {2665-9638},\n doi = {https:\u002F\u002Fdoi.org\u002F10.1016\u002Fj.simpa.2021.100058},\n url = {https:\u002F\u002Fwww.sciencedirect.com\u002Fscience\u002Farticle\u002Fpii\u002FS2665963821000063},\n author = {Veysel Kocaman 和 David Talby},\n keywords = {Spark, 自然语言处理, 深度学习, Tensorflow, 集群},\n abstract = {Spark NLP 是一个基于 Apache Spark ML 构建的自然语言处理（NLP）库。它为机器学习流水线提供了简单、高效且准确的 NLP 注释，可在分布式环境中轻松扩展。Spark NLP 拥有超过 1100 个预训练流水线和模型，涵盖 192 种以上语言。它几乎支持所有 NLP 任务和模块，可无缝地应用于集群环境。自 2020 年 1 月以来，Spark NLP 下载量已超过 270 万次，增长了 9 倍，目前已有 54% 的医疗保健机构将其作为全球企业中最广泛使用的 NLP 库。}\n }\n}\n```","# NLU 快速上手指南\n\n## 环境准备\n\n- **系统要求**：Linux \u002F macOS \u002F Windows（推荐 Linux）\n- **前置依赖**：Java 8（需正确配置 `JAVA_HOME` 环境变量）\n- **Python 版本**：建议 3.7+\n\n> ⚠️ 注意：NLU 基于 Spark NLP，依赖 Apache Spark，需确保 Java 8 可用。如未安装 Java，请从 [Oracle JDK 8](https:\u002F\u002Fwww.oracle.com\u002Fjava\u002Ftechnologies\u002Fjavase\u002Fjavase8-archive-downloads.html) 或 [OpenJDK 8](https:\u002F\u002Fadoptopenjdk.net\u002F) 下载。\n\n## 安装步骤\n\n使用 pip 安装 NLU 及其依赖：\n\n```bash\npip install nlu pyspark==3.0.2\n```\n\n> 🚀 **国内加速推荐**：使用清华镜像源加速安装 \n> ```bash\n> pip install nlu pyspark==3.0.2 -i https:\u002F\u002Fpypi.tuna.tsinghua.edu.cn\u002Fsimple\n> ```\n\n## 基本使用\n\n### 1. 单行加载模型并预测（情感分析）\n\n```python\nimport nlu\nnlu.load('sentiment').predict('I love NLU! \u003C3')\n```\n\n### 2. 单行加载多个模型（多嵌入融合）\n\n```python\nnlu.load('bert elmo albert xlnet glove use').predict('I love NLU! \u003C3')\n```\n\n### 3. 快速启动交互式可视化面板（Streamlit）\n\n```bash\nstreamlit run https:\u002F\u002Fraw.githubusercontent.com\u002FJohnSnowLabs\u002Fnlu\u002Fmaster\u002Fexamples\u002Fstreamlit\u002F01_dashboard.py\n```\n\n> ✅ 支持中文、英文等 200+ 语言模型，直接在浏览器中交互测试。\n\n---\n\n**更多模型与示例**：[NLU 官方文档](https:\u002F\u002Fnlu.johnsnowlabs.com\u002F) | [GitHub 示例](https:\u002F\u002Fgithub.com\u002FJohnSnowLabs\u002Fnlu\u002Ftree\u002Fmaster\u002Fexamples)","一家跨国电商公司正在分析来自日本、德国、巴西等10余国的用户评论，以自动识别产品负面反馈中的具体问题（如“电池续航短”“物流太慢”），并生成中文报告供运营团队快速响应。团队需在一周内完成对20万条多语言评论的语义分析，但缺乏NLP专家和跨语言模型部署经验。\n\n### 没有 nlu 时\n- 需要为每种语言单独寻找、下载并部署不同的预训练模型（如spaCy、Transformers），耗时超过两周。\n- 每个模型的输入格式不统一，需为Pandas、Spark DataFrame分别写适配代码，开发复杂度高。\n- 模型训练需手动标注数百条样本，再用PyTorch从零搭建分类器，周期长达10天以上。\n- 无法可视化实体识别结果，分析师只能看原始JSON，难以快速定位“电池”“物流”等关键实体。\n- 部署后无法快速验证效果，测试新模型需重启服务，缺乏交互式调试能力。\n\n### 使用 nlu 后\n- 一行代码 `nlu.load('sentiment.multi')` 即可同时处理日语、德语、葡萄牙语等100+语言的评论，3小时内完成全量分析。\n- 直接对Pandas DataFrame调用 `.predict()`，无需转换数据格式，代码量减少90%。\n- 用 `nlu.load('ner').fit(comments)` 仅用50条标注样例训练出精准的“产品问题识别器”，2天内上线。\n- 通过 `.viz(data)` 一键生成实体高亮图，清晰看到“电池续航短”被正确识别为产品缺陷实体。\n- 用 `.viz_streamlit(data)` 启动交互式仪表盘，运营团队可实时点击评论测试模型效果，即时调整策略。\n\nnlu 让非NLP团队在一周内完成原本需要数月的多语言文本分析任务，真正实现“一行代码，全球洞察”。","https:\u002F\u002Foss.gittoolsai.com\u002Fimages\u002FJohnSnowLabs_nlu_d5a49c95.png","JohnSnowLabs","John Snow Labs","https:\u002F\u002Foss.gittoolsai.com\u002Favatars\u002FJohnSnowLabs_3a40ac8f.jpg","",null,"http:\u002F\u002Fwww.JohnSnowLabs.com","https:\u002F\u002Fgithub.com\u002FJohnSnowLabs",[83,87],{"name":84,"color":85,"percentage":86},"Python","#3572A5",99.9,{"name":88,"color":89,"percentage":90},"Shell","#89e051",0.1,962,139,"2026-04-02T09:26:38","NOASSERTION","Linux, macOS, Windows","未说明",{"notes":98,"python":96,"dependencies":99},"需安装 Java 8 并正确配置环境；首次运行会自动下载大量预训练模型，建议预留至少 10GB 磁盘空间；支持在 Pandas、Spark 等数据框上直接使用，推荐使用 pip 安装 nlu 和 pyspark；可通过 Streamlit 快速构建交互式界面",[100],"pyspark==3.0.2",[26,13],[67,103,104,105,106,107,108,109,110,111,112,113,114,115,116,117,118,119,120,121],"natural-language-understanding","sentiment-classifier","text-classification","transformers","language-detection","named-entity-recognition","seq2seq","t5","lemmatizer","spell-checker","sentence-embeddings","sentiment-analysis","bert-embedding","text-summarization","text-translation","streamlit","pandas","dependency-parsing","entity-resolution","2026-03-27T02:49:30.150509","2026-04-06T05:36:46.969151",[125,130,135,140,145,150],{"id":126,"question_zh":127,"answer_zh":128,"source_url":129},8996,"NLU 是否支持 Python 3.8？","NLU 3.0.0 及以上版本支持 Python 3.8，但仅当使用 Spark 3.0+ 时才兼容。若使用 Spark 2.x，则必须使用 Python \u003C 3.8。推荐安装命令：pip install pyspark==3.0.1 spark-nlp==3.0.1 nlu==3.0.0rc3。","https:\u002F\u002Fgithub.com\u002FJohnSnowLabs\u002Fnlu\u002Fissues\u002F11",{"id":131,"question_zh":132,"answer_zh":133,"source_url":134},8997,"在 Apple M1 芯片上运行 NLU 报错如何解决？","当前 NLU 仅支持 Apple M1 原生架构，不支持 M1 Pro、M1 Max 或 M2 芯片。由于 TensorFlow for Java 等依赖缺乏 M1 支持，建议在 Google Colab 或 Databricks 等远程环境中运行，避免本地环境兼容性问题。","https:\u002F\u002Fgithub.com\u002FJohnSnowLabs\u002Fnlu\u002Fissues\u002F140",{"id":136,"question_zh":137,"answer_zh":138,"source_url":139},8998,"如何在 NLU 中使用自己微调的 Electra 模型？","目前无法直接加载 Hugging Face 微调的 TensorFlow v2 模型。需先将模型导出为 TensorFlow v1 格式，并确保张量名称符合 TensorflowBert.scala 的规范（如 input_ids:0、sequence_output:0 等），再通过 Spark-NLP 加载并导出为 Spark 模型，最后才能在 NLU 中使用。未来版本将支持直接导入 Hugging Face 模型。","https:\u002F\u002Fgithub.com\u002FJohnSnowLabs\u002Fnlu\u002Fissues\u002F46",{"id":141,"question_zh":142,"answer_zh":143,"source_url":144},8999,"加载 en.ner.dl.bert 模型时报错 'NoneType' 对象无属性 '__set_missing_model_attributes__' 如何解决？","该问题是 NLU 早期版本的命名空间 Bug，已在 v1.0.6 版本中修复。请升级 NLU 至最新版本：pip install --upgrade nlu。","https:\u002F\u002Fgithub.com\u002FJohnSnowLabs\u002Fnlu\u002Fissues\u002F23",{"id":146,"question_zh":147,"answer_zh":148,"source_url":149},9000,"如何移除 NLU 对 pyspark 的硬依赖以适配 Databricks？","NLU v1.0.5 已移除对 pyspark 的硬依赖，改用 findspark 自动发现现有 Spark 环境。无需安装 pyspark，只需确保环境中已配置 Spark（如 Databricks Runtime），即可直接使用 NLU。","https:\u002F\u002Fgithub.com\u002FJohnSnowLabs\u002Fnlu\u002Fissues\u002F19",{"id":151,"question_zh":152,"answer_zh":153,"source_url":154},9001,"在 Windows 上运行 NLU 报错 'Could not locate executable null\\bin\\winutils.exe' 如何解决？","需手动下载 hadoop-common 的 winutils.exe 文件（如从 https:\u002F\u002Fgithub.com\u002Fcdarlint\u002Fwinutils），将其放入 HADOOP_HOME\\bin 目录，并设置环境变量 HADOOP_HOME 指向该目录（如 C:\\hadoop）。同时建议以管理员身份运行 Python。","https:\u002F\u002Fgithub.com\u002FJohnSnowLabs\u002Fnlu\u002Fissues\u002F9",[156,161,166,171,176,181,186,191,196,201,206,211,216,221,226,231,236,241,246,251],{"id":157,"version":158,"summary_zh":159,"released_at":160},106437,"541","## Few-Shot Assertion Classifier\r\n\r\nFewShotAssertionClassifier Model is an advanced annotator designed to get higher accuracy with fewer data samples inspired by the SetFit framework. Few-Shot Assertion models consist of a sentence embedding component paired with a classifier (or head). While current support is focused on MPNet-based Few-Shot Assertion models, future updates will extend compatibility to include other popular models like Bert, DistillBert, and Roberta.\r\nThis classifier model supports various classifier types, including sklearn’s LogisticRegression and custom PyTorch models, providing flexibility for different model setups.\r\n\r\nPowered by FewShotAssertionClassifier\r\n \r\n| Language | nlp.load() reference | Spark NLP Model reference |\r\n|----------|-------------------------------|-----------------------------|\r\n| en | en.few_assert_shot_classifier | assertion_fewshotclassifier |\r\n\r\n\r\n## Partitioning Spark-DFs\r\nSupport for configuring partitioning of Spark-DFs via `pipe.predict(data, partitioning=1000)`\r\nIn Spark ML pipelines, which are the backbone of NLU, effective partitioning optimizes parallelism, reduces shuffling and ensuring even data distribution, which is crucial for high-performance machine learning tasks. \r\n\r\n## Bugfixes\r\n\r\n- Fixed bug causing DB endpoint environments to fail predicting on data \r\n","2024-10-24T16:12:40",{"id":162,"version":163,"summary_zh":164,"released_at":165},106438,"540","We are excited to announce NLU 5.4.0 has been released! \r\nIt comes with support for deidentifying PDFs leveraging a combination of OCR and Medical NLP models.\r\nAdditionally you can leverage MPnet for sequence classifcation and Pipeline Tracer is now supported\r\n\r\n\r\n\r\n\r\n----\r\n\r\n## Visual PDF Deidentifcation\r\n\r\n[Tutorial Notebook](https:\u002F\u002Fcolab.research.google.com\u002Fgithub\u002FJohnSnowLabs\u002Fnlu\u002Fblob\u002Fmaster\u002Fexamples\u002Fcolab\u002Fhealthcare\u002Fpipeline_parser\u002FParser.ipynb)\r\n\r\nIntroducing our advanced healthcare deidentification model, effortlessly deployable with a single line of code. This powerful solution integrates state-of-the-art algorithms like ner_deid_subentity_augmented, ContextualParser, RegexMatcher, and TextMatcher, alongside a streamlined de-identification stage. It efficiently masks sensitive entities such as names, locations, and medical records, ensuring compliance and data security in medical texts. Utilizing OCR capabilities, it also redacts detected information before saving the processed file to the specified location.\r\n\r\nPowered By: [PdfToImage](https:\u002F\u002Fnlp.johnsnowlabs.com\u002Fdocs\u002Fen\u002Focr_pipeline_components#pdftoimage), [ImageDrawRegions](https:\u002F\u002Fnlp.johnsnowlabs.com\u002Fdocs\u002Fen\u002Focr_pipeline_components#imagedrawregions), [ImageToPdf](https:\u002F\u002Fnlp.johnsnowlabs.com\u002Fdocs\u002Fen\u002Focr_pipeline_components#imagetopdf), [PositionFinder](https:\u002F\u002Fnlp.johnsnowlabs.com\u002Fdocs\u002Fen\u002Focr_pipeline_components#positionfinder)\r\n\r\n| **nlu.load() reference** | **Spark NLP Model Reference** |\r\n| ------------------------ | ----------------------------- |\r\n| en.image_deid | pdf_deid_pdf_output |\r\n\r\n\r\n```python\r\n! wget https:\u002F\u002Fgithub.com\u002FJohnSnowLabs\u002Fnlu\u002Fraw\u002Frelease\u002F540\u002Ftests\u002Fdatasets\u002Focr\u002Fdeid\u002Fdeid2.pdf \r\n! wget https:\u002F\u002Fgithub.com\u002FJohnSnowLabs\u002Fnlu\u002Fraw\u002Frelease\u002F540\u002Ftests\u002Fdatasets\u002Focr\u002Fdeid\u002Fdownload.pdf \r\n \r\n#provide the input and the output path \r\ninput_path,output_path = ['download.pdf',' deid2.pdf'], ['download_deidentified.pdf',' deid2_deidentified.pdf'] \r\n \r\n#predict and save the deidentified pdf's. \r\ndfs = model.predict(input_path, output_path=output_path)\r\n```\r\n\r\n\r\n![Pasted image 20240713173840](https:\u002F\u002Fgithub.com\u002Fuser-attachments\u002Fassets\u002F99acd529-f662-4b9f-ac18-04cacc5ef5c2)\r\n\r\n\r\n---\r\n## MPNetForSequenceClassification\r\n\r\n[Tutorial Notebook](https:\u002F\u002Fcolab.research.google.com\u002Fgithub\u002FJohnSnowLabs\u002Fnlu\u002Fblob\u002Fmaster\u002Fexamples\u002Fcolab\u002Fcomponent_examples\u002Fclassifiers\u002FNLU_MPNetForSequenceClassification.ipynb)\r\n\r\nMPNetForSequenceClassification is a state-of-the-art annotator in Spark NLP, designed for sequence classification tasks. It uses the MPNet architecture, which combines the strengths of BERT and XLNet, addressing their limitations.\r\n\r\nMPNet, or Masked and Permuted Pre-training for Language Understanding, improves token dependency understanding and sentence position information. This enhances sentence structure comprehension and reduces position discrepancies seen in XLNet.\r\n\r\nThe annotator excels in tasks like document classification and sentiment analysis, offering superior performance due to its innovative pre-training and fine-tuning on large datasets. Integrated into Spark NLP, it ensures scalable, efficient, and high-accuracy sequence classification.\r\n\r\n**Read More**: [Paper](https:\u002F\u002Farxiv.org\u002Fabs\u002F2004.09297)\r\n\r\nPowered by [MPNet](https:\u002F\u002Fsparknlp.org\u002Fdocs\u002Fen\u002Ftransformers#mpnetforsequenceclassification)\r\n\r\n|Language|nlp.load() reference|Spark NLP Model reference|\r\n|---|---|---|\r\n|en|en.classify.mpnet.ukr_message|[mpnet_sequence_classifier_ukr_message](https:\u002F\u002Fsparknlp.org\u002F2024\u002F01\u002F10\u002Fmpnet_sequence_classifier_ukr_message_en.html)|\r\n\r\n---\r\n## Pipeline Tracer\r\n\r\n[Tutorial Notebook](https:\u002F\u002Fcolab.research.google.com\u002Fgithub\u002FJohnSnowLabs\u002Fnlu\u002Fblob\u002Fmaster\u002Fexamples\u002Fcolab\u002Fhealthcare\u002Fpipeline_parser\u002FParser.ipynb)\r\n\r\nThe PipelineTracer is now accessible on NLU pipelines which is a versatile class designed to trace and analyze the stages of a pipeline, offering in-depth insights into entities, assertions, deidentification, classification, and relationships. It also facilitates the creation of parser dictionaries for building a PipelineOutputParser. Key functions include printing the pipeline schema, creating parser dictionaries, and retrieving possible assertions, relations, and entities. Also, provide direct access to parser dictionaries and available pipeline schemas\r\n\r\n\r\nLoad a pipe\r\n```python\r\npipe = nlp.load(\"en.explain_doc.clinical_oncology.pipeline\")\r\n```\r\n\r\n\r\nGet all assertions predictable with pipe \r\n```python \r\npipe.getPossibleAssertions()\r\n>>> ['Past', 'Family', 'Absent', 'Hypothetical', 'Possible', 'Present']\r\n```\r\n\r\nGet all entities predictable with pipe \r\n```python \r\npipe.getPossibleEntities()\r\n>>> ['Cycle_Number','Direction','Histological_Type', .... ] \r\n```\r\n\r\nGet all relation predictable with pipe \r\n```python\r\npipe.getPossibleRelations()\r\n>>> ['is_size_of', 'is_date_of', 'is_location_of', 'is_finding_of']\r\n```\r\n\r\nPredict parsed with configs\r\n```python\r\ncolumn_maps = pipe.createParserDictionary() \r\ncolumn_maps.upda","2024-07-13T16:15:57",{"id":167,"version":168,"summary_zh":169,"released_at":170},106439,"5.3.2","Hotfix for Databricks Endpoints https:\u002F\u002Fgithub.com\u002FJohnSnowLabs\u002Fnlu\u002Fpull\u002F264","2024-05-21T22:42:28",{"id":172,"version":173,"summary_zh":174,"released_at":175},106440,"531","We are excited to announce NLU 5.3.1 has been released! It comes with Visual Document NER, enabling you to extract entities from image files like JPGs.\r\nAdditionally 5 Healthcare Pipelines have been added for domains like Therapeutic Chemicals, HPO Resolvers, Voice of Patient, Oncology and Generic Clinical . \r\nAdditionally TextMatcherInternal based pipelines are now supported \r\n\r\n---\r\n## Visual NER\r\n\r\n- [Tutorial Notebook](https:\u002F\u002Fcolab.research.google.com\u002Fgithub\u002FJohnSnowLabs\u002Fnlu\u002Fblob\u002Fmaster\u002Fexamples\u002Fcolab\u002Focr\u002Focr_visual_document_ner.ipynb \"https:\u002F\u002Fcolab.research.google.com\u002Fgithub\u002FJohnSnowLabs\u002Fnlu\u002Fblob\u002Fmaster\u002Fexamples\u002Fcolab\u002Focr\u002Focr_visual_document_ner.ipynb\")\r\n- [Medium: Named Entity Recognition in Documents with Transformer Models using Visual-NLP: Part 1](https:\u002F\u002Fgaddesaishailesh.medium.com\u002F44f8c65df8d3)\r\n- [Medium: One-Liner Magic with Spark NLP: Deep Learning for NER in Documents — Part 2](https:\u002F\u002Fgaddesaishailesh.medium.com\u002Faa9168a6febd)\r\n\r\nVisualDocumentNER is a transformer-based model designed for Named Entity Recognition (NER) in documents. It serves as the primary interface for tasks such as detecting keys and values in datasets like FUNSD, representing the structure of a form. These keys and values are typically interconnected using a FormRelationExtractor model.\r\n\r\nHowever, some VisualDocumentNER models are trained with a different approach, considering entities in isolation. These entities could be names, places, or medications, and the goal is not to connect these entities to others, but to utilize them individually.\r\n\r\nPowered by Spark OCR's [VisualDocumentNER](https:\u002F\u002Fnlp.johnsnowlabs.com\u002Fdocs\u002Fen\u002Focr_visual_document_understanding#visualdocumentner-1 \"https:\u002F\u002Fnlp.johnsnowlabs.com\u002Fdocs\u002Fen\u002Focr_visual_document_understanding#visualdocumentner-1\")\r\n\r\n----\r\n## New Healthcare Models \r\n\r\n| NLU ref | Model |\r\n| ----------------------------------------- | -------------------------------------------------------------------------------------------------------------- |\r\n| en.resolve.atc_pipeline | [atc_resolver_pipeline](https:\u002F\u002Fnlp.johnsnowlabs.com\u002F2024\u002F01\u002F17\u002Fatc_resolver_pipeline_en.html) |\r\n| en.map_entity.hpo_resolver_pipe | [hpo_resolver_pipeline](https:\u002F\u002Fnlp.johnsnowlabs.com\u002F2024\u002F01\u002F18\u002Fhpo_resolver_pipeline_en.html) |\r\n| en.explain_doc.pipeline_vop | [explain_clinical_doc_vop](https:\u002F\u002Fnlp.johnsnowlabs.com\u002F2024\u002F01\u002F16\u002Fexplain_clinical_doc_vop_en.html) |\r\n| en.explain_doc.clinical_generic.pipeline | [explain_clinical_doc_generic](https:\u002F\u002Fnlp.johnsnowlabs.com\u002F2024\u002F01\u002F17\u002Fexplain_clinical_doc_generic_en.html) |\r\n| en.explain_doc.clinical_oncology.pipeline | [explain_clinical_doc_oncology](https:\u002F\u002Fnlp.johnsnowlabs.com\u002F2024\u002F01\u002F29\u002Fexplain_clinical_doc_oncology_en.html) |\r\n\r\n-----\r\n\r\n\r\n## New Medium Articles \r\nTutotirals on how to leverage Visual NLPs table extraction and Visual NER in 1 line and with custom pipelines:\r\n\r\n- [Deep Learning based Table Extraction using Visual NLP: Part 1](https:\u002F\u002Fgaddesaishailesh.medium.com\u002Fd81c6ff776a3)\r\n- [One-Liner Magic with Spark NLP: Deep Learning for Table Extraction — Part 2](https:\u002F\u002Fgaddesaishailesh.medium.com\u002F1a41f0ff6522)\r\n- [Named Entity Recognition in Documents with Transformer Models using Visual-NLP: Part 1](https:\u002F\u002Fgaddesaishailesh.medium.com\u002F44f8c65df8d3)\r\n- [One-Liner Magic with Spark NLP: Deep Learning for NER in Documents — Part 2](https:\u002F\u002Fgaddesaishailesh.medium.com\u002Faa9168a6febd)\r\n\r\n----\r\n\r\n## 📖Additional NLU resources\r\n\r\n* [140+ NLU Tutorials](https:\u002F\u002Fnlp.johnsnowlabs.com\u002Fdocs\u002Fen\u002Fjsl\u002Fnotebooks)\r\n* [Streamlit visualizations docs](https:\u002F\u002Fnlp.johnsnowlabs.com\u002Fdocs\u002Fen\u002Fjsl\u002Fstreamlit_viz_examples)\r\n* The complete list of all 20000+ models & pipelines in 300+ languages is available on [Models Hub](https:\u002F\u002Fnlp.johnsnowlabs.com\u002Fmodels)\r\n* [Spark NLP publications](https:\u002F\u002Fmedium.com\u002Fspark-nlp)\r\n* [NLU documentation](https:\u002F\u002Fnlp.johnsnowlabs.com\u002Fdocs\u002Fen\u002Fjsl\u002Finstall)\r\n* [Discussions](https:\u002F\u002Fgithub.com\u002FJohnSnowLabs\u002Fspark-nlp\u002Fdiscussions) Engage with other community members, share ideas, and show off how you use Spark NLP and NLU!\r\n\r\n\r\n---\r\n\r\n## Installation\r\n```shell\r\n#PyPI\r\npip install nlu pyspark\r\n```\r\n\r\n\r\n","2024-04-30T22:34:05",{"id":177,"version":178,"summary_zh":179,"released_at":180},106441,"530","We are very excited to announce NLU 5.3.0 has been released!\r\nIt features support for Open AI's Completion and Word Embeddings, alongside visual document classification, Bart and XLM RoBerta for Zero Shot Classification. \r\n\r\n---\r\n## Open AI Completion\r\n\r\n[Tutorial Notebook](https:\u002F\u002Fcolab.research.google.com\u002Fgithub\u002FJohnSnowLabs\u002Fnlu\u002Fblob\u002Fmaster\u002Fexamples\u002Fcolab\u002Fcomponent_examples\u002Fsequence2sequence\u002FOpenAI_completion.ipynb)\r\n**OpenAICompletion** combines powers of OpenAI’s completion models with the robust NLP processing capabilities of Spark NLP. This integration not only ensures the utilization of OpenAI's capabilities but also capitalizes on Spark's inherent scalability advantages.\r\nThis annotator makes direct API calls to OpenAI’s Completion endpoint right from datasets. This enhancement promises to elevate the efficiency and versatility of data processing workflows within Spark NLP pipelines.\r\nPowered by [OpenAICompletion](https:\u002F\u002Fsparknlp.org\u002Fdocs\u002Fen\u002Ftransformers#openaicompletion)\r\nReference: [OpenAI API Doc](https:\u002F\u002Fplatform.openai.com\u002Fdocs\u002Fapi-reference\u002Fcompletions\u002Fcreate)\r\nReference: [OpenAICompletion Doc](https:\u002F\u002Fsparknlp.org\u002Fapi\u002Fpython\u002Freference\u002Fautosummary\u002Fsparknlp\u002Fannotator\u002Fopenai\u002Fopenai_completion\u002Findex.html#sparknlp.annotator.openai.openai_completion.OpenAICompletion)\r\n\r\n| nlu.load() reference | Spark NLP Model reference |\r\n| -------------------- | ------------------------------------------------------------------------------ |\r\n| openai.completion | [OpenAICompletion](https:\u002F\u002Fsparknlp.org\u002Fdocs\u002Fen\u002Ftransformers#openaicompletion) |\r\n\r\n----\r\n## Open AI Embeddings\r\n\r\n[Tutorial Notebook](https:\u002F\u002Fcolab.research.google.com\u002Fgithub\u002FJohnSnowLabs\u002Fnlu\u002Fblob\u002Fmaster\u002Fexamples\u002Fcolab\u002Fcomponent_examples\u002Fsentence_embeddings\u002FNLU_OpenAI_embeddings.ipynb)\r\n**OpenAIEmbeddings** combines powers of OpenAI’s embeddings model with the robust NLP processing capabilities of Spark NLP. This integration not only ensures the utilization of OpenAI's capabilities but also capitalizes on Spark's inherent scalability advantages.\r\n This annotator makes direct API calls to OpenAI’s Embeddings endpoint right from datasets. This enhancement promises to elevate the efficiency and versatility of data processing workflows within Spark NLP pipelines.\r\nPowered by [OpenAIEmbeddings](https:\u002F\u002Fsparknlp.org\u002Fapi\u002Fpython\u002Freference\u002Fautosummary\u002Fsparknlp\u002Fannotator\u002Fopenai\u002Fopenai_embeddings\u002Findex.html)\r\n\r\n| nlu.load() reference| Spark NLP Model reference |\r\n|---------------------------------|-----------------------------------------------------------------------------------------------|\r\n| openai.embeddings \t | [OpenAIEmbeddings](https:\u002F\u002Fsparknlp.org\u002Fapi\u002Fpython\u002Freference\u002Fautosummary\u002Fsparknlp\u002Fannotator\u002Fopenai\u002Fopenai_embeddings\u002Findex.html) |\r\n\r\n----\r\n\r\n## Visual Document Classifier\r\n\r\n[Tutorial Notebook](https:\u002F\u002Fcolab.research.google.com\u002Fgithub\u002FJohnSnowLabs\u002Fnlu\u002Fblob\u002Fmaster\u002Fexamples\u002Fcolab\u002Focr\u002Focr_visual_document_classifier.ipynb) \r\n \r\nThe **VisualDocumentClassifier** is a DL model for document classification using text and layout data. The currently available pre-trained model on the Tobacco3482 dataset contains 3482 images belonging to 10 different classes (Resume, News, Note, Advertisement, Scientific, Report, Form, Letter, Email and Memo)\r\n\r\nPowered By \r\n[VisualDocumentClassifier](https:\u002F\u002Fnlp.johnsnowlabs.com\u002Fdocs\u002Fen\u002Focr_visual_document_understanding)\r\n\r\n| Language | nlu.load() reference | Spark NLP Model reference |\r\n| -------- | ------------------------- | -------------------------------------- |\r\n| xx | en.classify_image.tabacco | visual_document_classifier_tobacco3482 |\r\n\r\n\r\n\r\n---\r\n## Bart for Zero Shot Classificaiton\r\n\r\n[Tutorial Notebook](https:\u002F\u002Fcolab.research.google.com\u002Fhttps:\u002F\u002Fgithub.com\u002FJohnSnowLabs\u002Fnlu\u002Ftree\u002Fmaster\u002Fexamples\u002Fcolab\u002Fcomponent_examples\u002Fclassifiers\u002FBart_Zero_Shot_Classifiers.ipynb)\r\n\r\nBartForZeroShotClassification using a ModelForSequenceClassification trained on NLI (natural language inference) tasks.\r\nThe equivalent of BartForSequenceClassification models, but these models don’t require a hardcoded number of potential classes, they can be chosen at runtime. It usually means it’s slower but it is much more flexible.\r\nWe used TFBartForSequenceClassification to train this model and used BartForZeroShotClassification annotator in Spark NLP 🚀 for prediction at scale\r\nPowered by [BartForZeroShotClassification](https:\u002F\u002Fsparknlp.org\u002Fdocs\u002Fen\u002Ftransformers#bartforzeroshotclassification)\r\n\r\n| Language | nlu.load() reference | Spark NLP Model reference |\r\n| -------- | ---------------------------- | -------------------------------------------------------------------------------------------------------------------- |\r\n| English | en.bart.zero_shot_classifier | [bart_large_zero_shot_classifier_mnli](h","2024-04-30T22:26:53",{"id":182,"version":183,"summary_zh":184,"released_at":185},106442,"514","various minor bugfixes which fix various pre-trained pipelines\r\n\r\n----------------\r\n:book: Additional NLU resources\r\n----------------\r\n* [140+ NLU Tutorials](https:\u002F\u002Fnlp.johnsnowlabs.com\u002Fdocs\u002Fen\u002Fjsl\u002Fnotebooks)\r\n* [Streamlit visualizations docs](https:\u002F\u002Fnlp.johnsnowlabs.com\u002Fdocs\u002Fen\u002Fjsl\u002Fstreamlit_viz_examples)\r\n* The complete list of all 20000+ models & pipelines in 300+ languages is available on [Models Hub](https:\u002F\u002Fnlp.johnsnowlabs.com\u002Fmodels)\r\n* [Spark NLP publications](https:\u002F\u002Fmedium.com\u002Fspark-nlp)\r\n* [NLU documentation](https:\u002F\u002Fnlp.johnsnowlabs.com\u002Fdocs\u002Fen\u002Fjsl\u002Finstall)\r\n* [Discussions](https:\u002F\u002Fgithub.com\u002FJohnSnowLabs\u002Fspark-nlp\u002Fdiscussions) Engage with other community members, share ideas, and show off how you use Spark NLP and NLU!\r\n---------------\r\nInstallation\r\n---------------\r\n```shell\r\n#PyPI\r\npip install nlu pyspark \r\n```","2024-02-08T06:25:00",{"id":187,"version":188,"summary_zh":189,"released_at":190},106443,"513","various minor bugfixes which fix various pre-trained pipelines\r\n\r\n- proper handling for finisher\r\n- light pipe bugfix\r\n- missing metadata handling\r\n\r\n## Bugfixes\r\n- Fixed a bug that caused some Chunk Mapper based pretrained pipelines to throw exceptions\r\n- Fixed bug that caused pretrained some pipes with sentence embed converters to crash\r\n\r\n----------------\r\n:book: Additional NLU resources\r\n----------------\r\n* [140+ NLU Tutorials](https:\u002F\u002Fnlp.johnsnowlabs.com\u002Fdocs\u002Fen\u002Fjsl\u002Fnotebooks)\r\n* [Streamlit visualizations docs](https:\u002F\u002Fnlp.johnsnowlabs.com\u002Fdocs\u002Fen\u002Fjsl\u002Fstreamlit_viz_examples)\r\n* The complete list of all 20000+ models & pipelines in 300+ languages is available on [Models Hub](https:\u002F\u002Fnlp.johnsnowlabs.com\u002Fmodels)\r\n* [Spark NLP publications](https:\u002F\u002Fmedium.com\u002Fspark-nlp)\r\n* [NLU documentation](https:\u002F\u002Fnlp.johnsnowlabs.com\u002Fdocs\u002Fen\u002Fjsl\u002Finstall)\r\n* [Discussions](https:\u002F\u002Fgithub.com\u002FJohnSnowLabs\u002Fspark-nlp\u002Fdiscussions) Engage with other community members, share ideas, and show off how you use Spark NLP and NLU!\r\n---------------\r\nInstallation\r\n---------------\r\n```shell\r\n#PyPI\r\npip install nlu pyspark \r\n```\r\n","2024-01-22T22:49:52",{"id":192,"version":193,"summary_zh":194,"released_at":195},106444,"512","We are announcing NLU 5.1.2 with new pipelines and bug fixes.\r\n10+ new medical ner, summarization, classification, mapper, deidentification healthcare pipelines has been added!\r\n## New Healthcare Pipelines\r\n|Language|nlu.load() reference |Spark NLP reference |\r\n|--------|------------------------------|-------------------------------------------------|\r\n|Arabic |ar.deid.clinical|[clinical_deidentification](https:\u002F\u002Fnlp.johnsnowlabs.com\u002F2023\u002F06\u002F22\u002Fclinical_deidentification_ar.html)|\r\n|English |en.summarize.biomedical_pubmed.pipeline|[summarizer_biomedical_pubmed_pipeline](https:\u002F\u002Fnlp.johnsnowlabs.com\u002F2023\u002F06\u002F22\u002Fsummarizer_biomedical_pubmed_pipeline_en.html)|\r\n|English |en.ner.oncology.pipeline|[ner_oncology_pipeline](https:\u002F\u002Fnlp.johnsnowlabs.com\u002F2023\u002F06\u002F26\u002Fner_oncology_pipeline_en.html)|\r\n|English |en.ner.oncology_response_to_treatment.pipeline|[ner_oncology_response_to_treatment_pipeline](https:\u002F\u002Fnlp.johnsnowlabs.com\u002F2023\u002F06\u002F17\u002Fner_oncology_response_to_treatment_pipeline_en.html)|\r\n|English |en.med_ner.vop.pipeline|[ner_vop_pipeline](https:\u002F\u002Fnlp.johnsnowlabs.com\u002F2023\u002F06\u002F22\u002Fner_vop_pipeline_en.html)|\r\n|English |en.med_ner.vop_demographic.pipeline|[ner_vop_demographic_pipeline](https:\u002F\u002Fnlp.johnsnowlabs.com\u002F2023\u002F06\u002F22\u002Fner_vop_demographic_pipeline_en.html)|\r\n|English |en.med_ner.vop_treatment.pipeline|[ner_vop_treatment_pipeline](https:\u002F\u002Fnlp.johnsnowlabs.com\u002F2023\u002F06\u002F22\u002Fner_vop_treatment_pipeline_en.html)|\r\n|English |en.med_ner.vop_problem.pipeline|[ner_vop_problem_pipeline](https:\u002F\u002Fnlp.johnsnowlabs.com\u002F2023\u002F06\u002F22\u002Fner_vop_problem_pipeline_en.html)|\r\n|English |en.classify.bert_sequence.vop_hcp_consult.pipeline|[bert_sequence_classifier_vop_hcp_consult_pipeline](https:\u002F\u002Fnlp.johnsnowlabs.com\u002F2023\u002F06\u002F21\u002Fbert_sequence_classifier_vop_hcp_consult_pipeline_en.html)|\r\n|English |en.classify.bert_sequence.vop_drug_side_effect.pipeline|[bert_sequence_classifier_vop_drug_side_effect_pipeline](https:\u002F\u002Fnlp.johnsnowlabs.com\u002F2023\u002F06\u002F22\u002Fbert_sequence_classifier_vop_drug_side_effect_pipeline_en.html)|\r\n|English |en.map_entity.rxnorm_resolver.pipe|[rxnorm_resolver_pipeline](https:\u002F\u002Fnlp.johnsnowlabs.com\u002F2023\u002F08\u002F16\u002Frxnorm_resolver_pipeline_en.html))|\r\n\r\n## Bugfixes\r\n- Fixed a bug that caused some Chunk Mapper based pretrained pipelines to throw exceptions\r\n- Fixed bug that caused pretrained some pipes with sentence embed converters to crash\r\n\r\n----------------\r\n:book: Additional NLU resources\r\n----------------\r\n* [140+ NLU Tutorials](https:\u002F\u002Fnlp.johnsnowlabs.com\u002Fdocs\u002Fen\u002Fjsl\u002Fnotebooks)\r\n* [Streamlit visualizations docs](https:\u002F\u002Fnlp.johnsnowlabs.com\u002Fdocs\u002Fen\u002Fjsl\u002Fstreamlit_viz_examples)\r\n* The complete list of all 20000+ models & pipelines in 300+ languages is available on [Models Hub](https:\u002F\u002Fnlp.johnsnowlabs.com\u002Fmodels)\r\n* [Spark NLP publications](https:\u002F\u002Fmedium.com\u002Fspark-nlp)\r\n* [NLU documentation](https:\u002F\u002Fnlp.johnsnowlabs.com\u002Fdocs\u002Fen\u002Fjsl\u002Finstall)\r\n* [Discussions](https:\u002F\u002Fgithub.com\u002FJohnSnowLabs\u002Fspark-nlp\u002Fdiscussions) Engage with other community members, share ideas, and show off how you use Spark NLP and NLU!\r\n---------------\r\nInstallation\r\n---------------\r\n```shell\r\n#PyPI\r\npip install nlu pyspark \r\n```\r\n","2024-01-20T17:40:42",{"id":197,"version":198,"summary_zh":199,"released_at":200},106445,"511","\r\nWe are incredibly excited to announce NLU 5.1.1 has been released with over 130+ models in 36+ languages including new models based on [Whisper](https:\u002F\u002Farxiv.org\u002Fabs\u002F2212.04356) for multilingual automatic speech recognition and [Deep Learning based Visual Table Recogition using cascade R-CNN](https:\u002F\u002Farxiv.org\u002Fpdf\u002F1906.09756.pdf \"https:\u002F\u002Farxiv.org\u002Fpdf\u002F1906.09756.pdf\")\r\n\r\nYou can now transcribe speech to text with `Whispe` with 85+ models across 36 languages for Automatic Speech Recognition (ASR). \r\nAdditionally, `Deep Learning based Visual Table Recogition` based on an Cascade mask R-CNN HRNet that features detection of tables within images is now available in NLU 🌟.\r\n\r\nFinally, 40+ new models for existing model classes has been added!\r\n\r\n\r\n## Deep Learning based Visual Table Recogition \r\n![Cascade R-CNN](https:\u002F\u002Fproduction-media.paperswithcode.com\u002Fmethods\u002FScreen_Shot_2020-06-13_at_11.36.42_AM.png)\r\n[Tutorial Notebook](https:\u002F\u002Fcolab.research.google.com\u002Fgithub\u002FJohnSnowLabs\u002Fnlu\u002Fblob\u002Fmaster\u002Fexamples\u002Fcolab\u002Focr\u002Focr_table_recognition_dl.ipynb \"https:\u002F\u002Fcolab.research.google.com\u002Fgithub\u002FJohnSnowLabs\u002Fnlu\u002Fblob\u002Fmaster\u002Fexamples\u002Fcolab\u002Focr\u002Focr_table_recognition_dl.ipynb\") \r\n \r\nYou can now extract tables from images as pandas dataframe in 1 line of code, leveraging Spark OCR's ImageTableDetector, ImageTableCellDetector and ImageCellsToTextTable classes. \r\n\r\n\r\nThe ImageTableDetector is a deep-learning model designed to identify tables within images. It utilizes the CascadeTabNet architecture, which incorporates the Cascade mask Region-based Convolutional Neural Network High-Resolution Network (Cascade mask R-CNN HRNet).\r\n\r\nThe ImageTableCellDetector, on the other hand, is engineered to pinpoint cells within a table image. Its foundation is an image processing algorithm that identifies both horizontal and vertical lines.\r\n\r\nThe ImageCellsToTextTable applies Optical Character Recognition (OCR) to regions of cells within an image and returns the recognized text to the outputCol as a TableContainer structure.\r\n\r\nIt’s important to note that these annotators do not need to be invoked individually in NLU. Instead, you can simply load the `image_table_cell2text_table` model using the command `nlp.load('image_table_cell2text_table')`, and then use `nlp.predict` to make predictions on any document.\r\n\r\n \r\nPowered by Spark OCR's [ImageTableDetector](https:\u002F\u002Fnlp.johnsnowlabs.com\u002Fmodels?task=OCR+Table+Detection \"https:\u002F\u002Fnlp.johnsnowlabs.com\u002Fmodels?task=OCR+Table+Detection\"), [ImageTableCellDetector](https:\u002F\u002Fnlp.johnsnowlabs.com\u002Fdocs\u002Fen\u002Focr_table_recognition#imagetablecelldetector \"https:\u002F\u002Fnlp.johnsnowlabs.com\u002Fdocs\u002Fen\u002Focr_table_recognition#imagetablecelldetector\"), [ImageCellsToTextTable](https:\u002F\u002Fnlp.johnsnowlabs.com\u002Fdocs\u002Fen\u002Focr_table_recognition#imagecellstotexttable \"https:\u002F\u002Fnlp.johnsnowlabs.com\u002Fdocs\u002Fen\u002Focr_table_recognition#imagecellstotexttable\") \r\nReference: [Cascade R-CNN: High Quality Object Detection and Instance Segmentation](https:\u002F\u002Farxiv.org\u002Fpdf\u002F1906.09756.pdf \"https:\u002F\u002Farxiv.org\u002Fpdf\u002F1906.09756.pdf\")\r\n\r\n|**language**|**nlu.load() reference**|**Spark NLP Model Reference**|\r\n|---|---|---|\r\n|en|en.image_table_detector|[General Model for Table Detection](https:\u002F\u002Fnlp.johnsnowlabs.com\u002F2023\u002F01\u002F10\u002Fgeneral_model_table_detection_v2_en_3_2.html \"https:\u002F\u002Fnlp.johnsnowlabs.com\u002F2023\u002F01\u002F10\u002Fgeneral_model_table_detection_v2_en_3_2.html\")|\r\n\r\n\r\n\r\n## Whisper for CTC\r\n\r\n\r\n![Whisper](https:\u002F\u002Fraw.githubusercontent.com\u002Fopenai\u002Fwhisper\u002Fmain\u002Fapproach.png)\r\n\r\n[Tutorial Notebook](https:\u002F\u002Fgithub.com\u002FJohnSnowLabs\u002Fnlu\u002Fblob\u002Fmaster\u002Fexamples\u002Fcolab\u002Fcomponent_examples\u002Fautomatic_speech_recognition\u002Fautomatic_speech_recognition_overview_ASR.ipynb)\r\n\r\nWhisper Model with a language modeling head on top for Connectionist Temporal Classification (CTC). Whisper is an automatic speech recognition (ASR) system trained on 680,000 hours of multilingual and multitask supervised data collected from the web. It transcribe in multiple languages, as well as translate from those languages into English. Whisper was trained and open-sourced that approaches human level robustness and accuracy on English speech recognition.\r\n\r\nPowered by Spark-NLP's [WhisperForCTC Annotator](https:\u002F\u002Fsparknlp.org\u002Fdocs\u002Fen\u002Ftransformers#whisperforctc)\r\nReference: [OpenAI Whisper: Robust Speech Recognition via Large-Scale Weak Supervision](https:\u002F\u002Farxiv.org\u002Fabs\u002F2212.04356)\r\nNote that at the moment only Spark Versions 3.4 and up are supported.\r\n\r\n\r\n\r\n\r\n|Language|NLU Reference|Spark NLP Reference|Annotator Class|\r\n|---|---|---|---|\r\n|bg|bg.speech2text.whisper.tiny_bulgarian_l|[asr_whisper_tiny_bulgarian_l](https:\u002F\u002Fnlp.johnsnowlabs.com\u002F2023\u002F10\u002F17\u002Fasr_whisper_tiny_bulgarian_l_bg.html)|WhisperForCTC|\r\n|cs|cs.speech2text.whisper.small_czech_cv11|[asr_whisper_small_czech_cv11](https:\u002F\u002Fnlp.johnsnowlabs.com\u002F2023\u002F10\u002F17\u002Fasr_whisper_small_czech_cv11_cs.html)|WhisperForCTC|\r\n|da|da.speech2text.whisper.danish_small_augmented|[asr_whisper_danish_small_augmented](https:\u002F\u002Fnlp.johnsnowlabs.com\u002F2023\u002F10\u002F20\u002Fasr_w","2024-01-08T02:37:04",{"id":202,"version":203,"summary_zh":204,"released_at":205},106446,"504","We are very excited to announce John Snow Labs NLU 5.1.0 has been released!\r\nIt features 350+ new models with 3 new Sentence Embeddings Architectures: [Instructor](https:\u002F\u002Farxiv.org\u002Fabs\u002F2212.09741), [E5](https:\u002F\u002Farxiv.org\u002Fpdf\u002F2212.03533.pdf) and [MPNET](https:\u002F\u002Farxiv.org\u002Fpdf\u002F2004.09297.pdf) in English, French and Spanish.\r\n\r\n\r\n## Instructor Sentence Embeddings\r\n\r\n[Tutorial Notebook](https:\u002F\u002Fgithub.com\u002FJohnSnowLabs\u002Fnlu\u002Ftree\u002Fmaster\u002Fexamples\u002Fcolab\u002Fcomponent_examples\u002Fsentence_embeddings\u002FNLU_INSTRUCTOR_sentence_embeddings.ipynb) \r\n\r\nInstructor👨‍🏫, an instruction-finetuned text embedding model that can generate text embeddings tailored to any task (e.g., classification, retrieval, clustering, text evaluation, etc.) and domains (e.g., science, finance, etc.) by simply providing the task instruction, without any finetuning. Instructor👨‍ achieves sota on 70 diverse embedding tasks.\r\n\r\nInstructor was proposed in One Embedder, Any Task: Instruction-Finetuned Text Embeddings by Hongjin Su, Weijia Shi, Jungo Kasai, Yizhong Wang, Yushi Hu, Mari Ostendorf, Wen-tau Yih, Noah A. Smith, Luke Zettlemoyer, Tao Yu. Analysis of the writers suggests that INSTRUCTOR is robust to changes in instructions, and that instruction finetuning mitigates the challenge of training a single model on diverse datasets\r\n\r\nPowered by [InstructorEmbeddings](https:\u002F\u002Fsparknlp.org\u002Fdocs\u002Fen\u002Ftransformers#instructorembeddings) \r\nReference: [One Embedder, Any Task: Instruction-Finetuned Text Embeddings](https:\u002F\u002Farxiv.org\u002Fabs\u002F2212.09741)\r\nReference: [InstructorEmbeddings Github Repo](https:\u002F\u002Fgithub.com\u002Fxlang-ai\u002Finstructor-embedding)\r\n\r\n\r\n |Language|NLU Reference|Spark NLP Reference|\r\n|---|---|---|\r\n|English|en.embed_sentence.instructor_base|[instructor_base](https:\u002F\u002Fsparknlp.org\u002F2023\u002F06\u002F08\u002Finstructor_base_en.html)|\r\n|English|en.embed_sentence.instructor_large|[instructor_large](https:\u002F\u002Fsparknlp.org\u002F2023\u002F06\u002F21\u002Finstructor_large_en.html)|\r\n\r\n\r\n-------\r\n\r\n\r\n## E5 Sentence Embeddings\r\n\r\n\r\n[Tutorial Notebook](https:\u002F\u002Fgithub.com\u002FJohnSnowLabs\u002Fnlu\u002Ftree\u002Fmaster\u002Fexamples\u002Fcolab\u002Fcomponent_examples\u002Fsentence_embeddings\u002FNLU_E5_sentence_embeddings.ipynb) \r\n\r\nE5 is a weakly supervised text embedding model that can generate text embeddings tailored to any task (e.g., classification, retrieval, clustering, text evaluation, etc).\r\n\r\nE5 was proposed in Text Embeddings by Weakly-Supervised Contrastive Pre-training by Liang Wang, Nan Yang, Xiaolong Huang, Binxing Jiao, Linjun Yang, Daxin Jiang, Rangan Majumder, Furu Wei. The model is trained in a contrastive manner with weak supervision signals from our curated large-scale text pair dataset (called CCPairs). E5 can be readily used as a general-purpose embedding model for any tasks requiring a single-vector representation of texts such as retrieval, clustering, and classification, achieving strong performance in both zero-shot and fine-tuned settings. \r\n\r\nPowered by [E5Embeddings](https:\u002F\u002Fsparknlp.org\u002Fdocs\u002Fen\u002Ftransformers#e5embeddings) \r\nReference: [Text Embeddings by Weakly-Supervised Contrastive Pre-training](https:\u002F\u002Farxiv.org\u002Fpdf\u002F2212.03533.pdf)\r\nReference: [E5Embeddings Github Repo](https:\u002F\u002Fgithub.com\u002Fmicrosoft\u002Funilm\u002Ftree\u002Fmaster\u002Fe5)\r\n\r\n\r\n\r\n|Language|NLU Reference|Spark NLP Reference|\r\n|---|---|---|\r\n|English|en.embed_sentence.e5_small|[e5_small](https:\u002F\u002Fsparknlp.org\u002F2023\u002F08\u002F25\u002Fe5_small_en.html)|\r\n|English|en.embed_sentence.e5_small_opt|[e5_small_opt](https:\u002F\u002Fsparknlp.org\u002F2023\u002F08\u002F25\u002Fe5_small_opt_en.html)|\r\n|English|en.embed_sentence.e5_small_quantized|[e5_small_quantized](https:\u002F\u002Fsparknlp.org\u002F2023\u002F08\u002F25\u002Fe5_small_quantized_en.html)|\r\n|English|en.embed_sentence.e5_small_v2|[e5_small_v2](https:\u002F\u002Fsparknlp.org\u002F2023\u002F08\u002F25\u002Fe5_small_v2_en.html)|\r\n|English|en.embed_sentence.e5_small_v2_opt|[e5_small_v2_opt](https:\u002F\u002Fsparknlp.org\u002F2023\u002F08\u002F25\u002Fe5_small_v2_opt_en.html)|\r\n|English|en.embed_sentence.e5_small_v2_quantized|[e5_small_v2_quantized](https:\u002F\u002Fsparknlp.org\u002F2023\u002F08\u002F25\u002Fe5_small_v2_quantized_en.html)|\r\n|English|en.embed_sentence.e5_base|[e5_base](https:\u002F\u002Fsparknlp.org\u002F2023\u002F08\u002F25\u002Fe5_base_en.html)|\r\n|English|en.embed_sentence.e5_base_opt|[e5_base_opt](https:\u002F\u002Fsparknlp.org\u002F2023\u002F08\u002F25\u002Fe5_base_opt_en.html)|\r\n|English|en.embed_sentence.e5_base_quantized|[e5_base_quantized](https:\u002F\u002Fsparknlp.org\u002F2023\u002F08\u002F25\u002Fe5_base_quantized_en.html)|\r\n|English|en.embed_sentence.e5_base_v2|[e5_base_v2](https:\u002F\u002Fsparknlp.org\u002F2023\u002F08\u002F25\u002Fe5_base_v2_en.html)|\r\n|English|en.embed_sentence.e5_base_v2_opt|[e5_base_v2_opt](https:\u002F\u002Fsparknlp.org\u002F2023\u002F08\u002F25\u002Fe5_base_v2_opt_en.html)|\r\n|English|en.embed_sentence.e5_base_v2_quantized|[e5_base_v2_quantized](https:\u002F\u002Fsparknlp.org\u002F2023\u002F08\u002F25\u002Fe5_base_v2_quantized_en.html)|\r\n|English|en.embed_sentence.e5_large|[e5_large](https:\u002F\u002Fsparknlp.org\u002F2023\u002F06\u002F21\u002Fe5_large_en.html)|\r\n|English|en.embed_sentence.e5_large_v2|[e5_large_v2](https:\u002F\u002Fsparknlp.org\u002F2023\u002F08\u002F25\u002Fe5_large_v2_en.html)|\r\n|English|en.embed_sentence.e5_large_v2_opt|[e5_large_v2_opt](https:\u002F\u002Fsparknlp.org\u002F2023\u002F08\u002F25\u002Fe5_large_v2_opt_en.html)|\r\n|English|en.embed_sentence.e5_large_v2_q","2023-11-09T00:00:26",{"id":207,"version":208,"summary_zh":209,"released_at":210},106447,"503","disable verbose logs by default","2023-10-09T14:39:52",{"id":212,"version":213,"summary_zh":214,"released_at":215},106448,"v502","This is a hotfix release, making NLU compatible with `pandas>=2.` \r\nNLU is now compatible with any `pandas>=1.3.5`","2023-10-08T12:55:25",{"id":217,"version":218,"summary_zh":219,"released_at":220},106449,"501","\r\n- fix bug that caused predicted column names to change when saving\u002Freloading a pipe\r\n- fix bug causing some Visual based nlu components to use wrong data types\r\n- New Databricks-Endpoint based inference mode. It is enabled if the env variable `DB_ENDPOINT_ENV` is present. When enabled, the first row of every pandas dataframe passed to `pipe.predict()` is checked for parameters. If your dataframe contains of `output_level`,`positions`,`keep_stranger_features`,`metadata`,`multithread`,`drop_irrelevant_cols`,`return_spark_df`,`get_embeddings`, the first row of your dataframe is mapped to the corrosponding parameter and used to call `pipeline.predict()`","2023-09-11T00:59:45",{"id":222,"version":223,"summary_zh":224,"released_at":225},106450,"v5.0.0","We are very excited to announce NLU 5.0.0 has been released! \r\n\r\nIt comes with `ZeroShotClassification` models based on `Bert`, `DistilBert`, and `Roberta` architectures.\r\nAdditionally Medical Text Generator based on `Bio-GPT` as-well as a `Bart` based General Text Generator are now available in NLU.\r\nFinally, `ConvNextForImageClassification` is an image classifier based on ConvNet models.\r\n\r\n\r\n\r\n\r\n------\r\n\r\n## ConvNextForImageClassification\r\n[Tutorial Notebook](https:\u002F\u002Fgithub.com\u002FJohnSnowLabs\u002Fnlu\u002Fblob\u002Fmaster\u002Fexamples\u002Fcolab\u002Fcomponent_examples\u002Fimage_classification\u002Fconvnext_image_classification_overview.ipynb) \r\n`ConvNextForImageClassification` is an image classifier based on ConvNet models.\r\nThe ConvNeXT model was proposed in A ConvNet for the 2020s by Zhuang Liu, Hanzi Mao, Chao-Yuan Wu, Christoph Feichtenhofer, Trevor Darrell, Saining Xie. ConvNeXT is a pure convolutional model (ConvNet), inspired by the design of Vision Transformers, that claims to outperform them. \r\nPowered by [ConvNextForImageClassification](https:\u002F\u002Fsparknlp.org\u002Fdocs\u002Fen\u002Ftransformers#convnextforimageclassification) \r\nReference: [A ConvNet for the 2020s](https:\u002F\u002Farxiv.org\u002Fabs\u002F2201.03545) \r\n\r\nNew NLU Models:\r\n\r\n| Language | NLU Reference | Spark NLP Reference | Task | Annotator Class |\r\n|:-----------|:----------------------------------------------------------------------------------------------------------------------------|:-------------------------------------------------------------------------------------------------------------------------------------|:---------------------|:------------------------|\r\n| en | [en.classify_image.convnext.tiny](https:\u002F\u002Fnlp.johnsnowlabs.com\u002F2023\u002F03\u002F28\u002Fimage_classifier_convnext_tiny_224_local_en.html) | [image_classifier_convnext_tiny_224_local](https:\u002F\u002Fnlp.johnsnowlabs.com\u002F2023\u002F03\u002F28\u002Fimage_classifier_convnext_tiny_224_local_en.html) | Image Classification | ConvNextImageClassifier |\r\n| en | [en.classify_image.convnext.tiny](https:\u002F\u002Fnlp.johnsnowlabs.com\u002F2023\u002F07\u002F05\u002Fimage_classifier_convnext_tiny_224_local_en.html) | [image_classifier_convnext_tiny_224_local](https:\u002F\u002Fnlp.johnsnowlabs.com\u002F2023\u002F07\u002F05\u002Fimage_classifier_convnext_tiny_224_local_en.html) | Image Classification | ConvNextImageClassifier |\r\n\r\n------\r\n\r\n\r\n## DistilBertForZeroShotClassification\r\n[Tutorial Notebook](https:\u002F\u002Fgithub.com\u002FJohnSnowLabs\u002Fnlu\u002Fblob\u002Fmaster\u002Fexamples\u002Fcolab\u002Fcomponent_examples\u002Fclassifiers\u002FDistilbert_Zero_Shot_Classifier.ipynb) \r\n\r\n`DistilBertForZeroShotClassification` using a ModelForSequenceClassification trained on NLI (natural language inference) tasks.\r\nAny combination of sequences and labels can be passed and each combination will be posed as a premise\u002Fhypothesis pair and passed to the pretrained model.\r\nPowered by [DistilBertForZeroShotClassification](https:\u002F\u002Fsparknlp.org\u002Fdocs\u002Fen\u002Ftransformers#distilbertforzeroshotclassification) \r\n\r\nNew NLU Models:\r\n\r\n| Language | NLU Reference | Spark NLP Reference | Task | Annotator Class |\r\n|:-----------|:-----------------------------------------------------------------------------------------------------------------------------------------------------------|:---------------------------------------------------------------------------------------------------------------------------------------------------------------------------|:-------------------------|:------------------------------------|\r\n| en | [en.distilbert.zero_shot_classifier](https:\u002F\u002Fnlp.johnsnowlabs.com\u002F2023\u002F04\u002F20\u002Fdistilbert_base_zero_shot_classifier_uncased_mnli_en.html) | [distilbert_base_zero_shot_classifier_uncased_mnli](https:\u002F\u002Fnlp.johnsnowlabs.com\u002F2023\u002F04\u002F20\u002Fdistilbert_base_zero_shot_classifier_uncased_mnli_en.html) | Zero-Shot Classification | DistilBertForZeroShotClassification |\r\n| tr | [tr.distilbert.zero_shot_classifier.multinli](https:\u002F\u002Fnlp.johnsnowlabs.com\u002F2023\u002F04\u002F20\u002Fdistilbert_base_zero_shot_classifier_turkish_cased_multinli_tr.html) | [distilbert_base_zero_shot_classifier_turkish_cased_multinli](https:\u002F\u002Fnlp.johnsnowlabs.com\u002F2023\u002F04\u002F20\u002Fdistilbert_base_zero_shot_classifier_turkish_cased_multinli_tr.html) | Zero-Shot Classification | DistilBertForZeroShotClassification |\r\n| tr | [tr.distilbert.zero_shot_classifier.allnli](https:\u002F\u002Fnlp.johnsnowlabs.com\u002F2023\u002F04\u002F20\u002Fdistilbert_base_zero_shot_classifier_turkish_cased_allnli_tr.","2023-08-11T11:42:43",{"id":227,"version":228,"summary_zh":229,"released_at":230},106451,"422","New Medical Summarizers:\r\n- 'en.summarize.clinical_jsl'\r\n- 'en.summarize.clinical_jsl_augmented'\r\n- 'en.summarize.biomedical_pubmed'\r\n- 'en.summarize.generic_jsl'\r\n- 'en.summarize.clinical_questions'\r\n- 'en.summarize.radiology'\r\n- 'en.summarize.clinical_guidelines_large'\r\n- 'en.summarize.clinical_laymen'","2023-06-14T20:57:18",{"id":232,"version":233,"summary_zh":234,"released_at":235},106452,"421","Bugfixes for saving and reloading pipelines on databricks \r\n","2023-06-07T00:41:10",{"id":237,"version":238,"summary_zh":239,"released_at":240},106453,"v4.2.0","## Support for Speech2Text, Images-Classification, Tabular Data, Zero-Shot-NER, via Wav2Vec2, Tapas, VIT , 4000+ New Models, 90+ Languages, in John Snow Labs NLU 4.2.0\r\n\r\n\r\nWe are incredibly excited to announce NLU 4.2.0 has been released with new 4000+ models in 90+ languages and support for new 8 Deep Learning Architectures.\r\n4 new tasks are included for the very first time, \r\n**Zero-Shot-NER**, **Automatic Speech Recognition**, **Image Classification** and **Table Question Answering** powered \r\nby [Wav2Vec 2.0](https:\u002F\u002Farxiv.org\u002Fpdf\u002F2006.11477.pdf), [HuBERT](https:\u002F\u002Farxiv.org\u002Fabs\u002F2106.07447), [TAPAS](https:\u002F\u002Faclanthology.org\u002F2020.acl-main.398.pdf), [VIT](https:\u002F\u002Farxiv.org\u002Fpdf\u002F2010.11929.pdf), [SWIN](https:\u002F\u002Farxiv.org\u002Fabs\u002F2103.14030), [Zero-Shot-NER](https:\u002F\u002Fnlp.johnsnowlabs.com\u002Fdocs\u002Fen\u002Flicensed_annotators#zeroshotnermodel).\r\n\r\nAdditionally, [CamemBERT](https:\u002F\u002Farxiv.org\u002Fabs\u002F1911.03894) based architectures are available for Sequence and Token Classification powered by Spark-NLPs\r\n[CamemBertForSequenceClassification](https:\u002F\u002Fnlp.johnsnowlabs.com\u002Fdocs\u002Fen\u002Ftransformers#camembertforsequenceclassification) and [CamemBertForTokenClassification](https:\u002F\u002Fnlp.johnsnowlabs.com\u002Fdocs\u002Fen\u002Ftransformers#camembertfortokenclassification)\r\n\r\n# Automatic Speech Recognition (ASR)\r\n[Demo Notebook](https:\u002F\u002Fgithub.com\u002FJohnSnowLabs\u002Fnlu\u002Fblob\u002Fmaster\u002Fexamples\u002Fcolab\u002Fcomponent_examples\u002Fautomatic_speech_recognition\u002Fautomatic_speech_recognition_overview_ASR.ipynb)\r\n[Wav2Vec 2.0](https:\u002F\u002Farxiv.org\u002Fpdf\u002F2006.11477.pdf) and [HuBERT](https:\u002F\u002Farxiv.org\u002Fabs\u002F2106.07447) enable ASR for the very first time in NLU.\r\n**Wav2Vec2** is a transformer model for speech recognition that uses unsupervised pre-training on large amounts of unlabeled speech data to improve the accuracy of automatic speech recognition (ASR) systems. It is based on a self-supervised learning approach that learns to predict masked portions of speech signal, and has shown promising results in reducing the amount of labeled training data required for ASR tasks.\r\n\r\nThese Models are powered by Spark-NLP's [Wav2Vec2ForCTC Annotator](https:\u002F\u002Fnlp.johnsnowlabs.com\u002Fdocs\u002Fen\u002Ftransformers#wav2vec2forctc)\r\n![Wav2Vec2](https:\u002F\u002Fuser-images.githubusercontent.com\u002F5762953\u002F192140859-f165317e-4a8f-4b32-9d11-6063db19c503.png)\r\n\r\n**HuBERT** models match or surpass the SOTA approaches for speech representation learning for speech recognition, generation, and compression. The Hidden-Unit BERT (HuBERT) approach was proposed for self-supervised speech representation learning, which utilizes an offline clustering step to provide aligned target labels for a BERT-like prediction loss.\r\n\r\nThese Models is powered by Spark-NLP's [HubertForCTC Annotator](https:\u002F\u002Fnlp.johnsnowlabs.com\u002Fdocs\u002Fen\u002Ftransformers#hubertforctc)\r\n\r\n![HUBERT](https:\u002F\u002Fuser-images.githubusercontent.com\u002F5762953\u002F217865459-375756c3-a110-4917-8319-1deecb55304d.png)\r\n\r\n**Usage**\r\n\r\nYou just need an audio-file on disk and pass the path to it or a folder of audio-files.\r\n\r\n```python\r\nimport nlu\r\n# Let's download an audio file \r\n!wget https:\u002F\u002Fs3.amazonaws.com\u002Fauxdata.johnsnowlabs.com\u002Fpublic\u002Fresources\u002Fen\u002Faudio\u002Fsamples\u002Fwavs\u002Fngm_12484_01067234848.wav\r\n# Let's listen to it \r\nfrom IPython.display import Audio\r\nFILE_PATH = \"ngm_12484_01067234848.wav\"\r\nasr_df = nlu.load('en.speech2text.wav2vec2.v2_base_960h').predict('ngm_12484_01067234848.wav')\r\nasr_df\r\n```\r\n\r\n| text |\r\n|:---------------------------------------------|\r\n| PEOPLE WHO DIED WHILE LIVING IN OTHER PLACES |\r\n\r\n\r\n\r\nTo test out **HuBERT** you just need to update the parameter for `load()`\r\n```python\r\nasr_df = nlu.load('en.speech2text.hubert').predict('ngm_12484_01067234848.wav')\r\nasr_df\r\n```\r\n\r\n\r\n# Image Classification\r\n[Demo Notebook](https:\u002F\u002Fgithub.com\u002FJohnSnowLabs\u002Fnlu\u002Fblob\u002Fmaster\u002Fexamples\u002Fcolab\u002Fcomponent_examples\u002Fimage_classification\u002Fimage_classification_overview.ipynb)\r\n\r\nFor the first time ever NLU introduces state-of-the-art image classifiers based on \r\n[VIT](https:\u002F\u002Farxiv.org\u002Fpdf\u002F2010.11929.pdf) and [Swin](https:\u002F\u002Farxiv.org\u002Fabs\u002F2103.14030) giving you access to hundreds of image classifiers for various domains.\r\n\r\nInspired by the Transformer scaling successes in NLP, the researchers experimented with applying a standard Transformer directly to images, with the fewest possible modifications. To do so, images are split into patches and the sequence of linear embeddings of these patches were provided as an input to a Transformer. Image patches were actually treated the same way as tokens (words) in an NLP application. Image classification models were trained in supervised fashion.\r\n\r\nYou can check [Scale Vision Transformers (ViT) Beyond Hugging Face](https:\u002F\u002Fhackernoon.com\u002Fscale-vision-transformers-vit-beyond-hugging-face) article to learn deeper how ViT works and how it is implemeted in Spark NLP.\r\nThis is Powerd by Spark-NLP's [VitForImageClassification Annotator](https:\u002F\u002Fnlp.johnsnowlabs.com\u002Fdocs\u002Fen\u002Ftransformers#vitforimageclassification)\r\n\r\n![","2023-03-20T23:15:43",{"id":242,"version":243,"summary_zh":244,"released_at":245},106454,"v4.0.0","OCR Visual Tables into Pandas DataFrames from PDF\u002FDOC(X)\u002FPPT files, 1000+ new state-of-the-art transformer models for Question Answering (QA) for over 30 languages, up to 700% speedup on GPU, 20 Biomedical models for over 8 languages, 50+ Terminology Code Mappers between RXNORM, NDC, UMLS,ICD10, ICDO, UMLS, SNOMED and MESH, Deidentification in Romanian, various Spark NLP helper methods and much more in 1 line of code with John Snow Labs NLU 4.0.0\r\n\r\n______________________\r\n\r\n## NLU 4.0 for OCR Overview\r\n\r\nOn the OCR side, we now support extracting tables from PDF\u002FDOC(X)\u002FPPT files into structured pandas dataframe, making it easier than ever before to analyze bulks of files visually!\r\n\r\nCheckout the [OCR Tutorial for extracting `Tables` from Image\u002FPDF\u002FDOC(X) files](https:\u002F\u002Fcolab.research.google.com\u002Fgithub\u002FJohnSnowLabs\u002Fnlu\u002Fblob\u002Fmaster\u002Fexamples\u002Fcolab\u002Focr\u002Ftable_extraction.ipynb.ipynb) [![Open In Colab](https:\u002F\u002Fcolab.research.google.com\u002Fassets\u002Fcolab-badge.svg)](https:\u002F\u002Fcolab.research.google.com\u002Fgithub\u002FJohnSnowLabs\u002Fnlu\u002Fblob\u002Fmaster\u002Fexamples\u002Fcolab\u002Focr\u002Ftable_extraction.ipynb) to see this in action\r\n\r\nThese models grab all Table data from the files detected and return a `list of Pandas DataFrames`, \r\ncontaining Pandas DataFrame for every table detected \r\n \r\n| NLU Spell | Transformer Class | \r\n|----------------------|-----------------------------------------------------------------------------------------| \r\n| nlu.load(`pdf2table`) | [PdfToTextTable](https:\u002F\u002Fnlp.johnsnowlabs.com\u002Fdocs\u002Fen\u002Focr_pipeline_components#pdftotexttable) | \r\n| nlu.load(`ppt2table`) | [PptToTextTable](https:\u002F\u002Fnlp.johnsnowlabs.com\u002Fdocs\u002Fen\u002Focr_pipeline_components#ppttotexttable) | \r\n| nlu.load(`doc2table`) | [DocToTextTable](https:\u002F\u002Fnlp.johnsnowlabs.com\u002Fdocs\u002Fen\u002Focr_pipeline_components#doctotexttable) |\r\n\r\n\r\nThis is powerd by [John Snow Labs Spark OCR](https:\u002F\u002Fwww.johnsnowlabs.com\u002Fspark-ocr\u002F) Annotataors for [PdfToTextTable](https:\u002F\u002Fnlp.johnsnowlabs.com\u002Fdocs\u002Fen\u002Focr_pipeline_components#pdftotexttable), [DocToTextTable](https:\u002F\u002Fnlp.johnsnowlabs.com\u002Fdocs\u002Fen\u002Focr_pipeline_components#doctotexttable), [PptToTextTable](https:\u002F\u002Fnlp.johnsnowlabs.com\u002Fdocs\u002Fen\u002Focr_pipeline_components#ppttotexttable) \r\n\r\n---------------------------------------- \r\n## NLU 4.0 Core Overview\r\n\r\n- On the NLU core side we have over 1000+ new state-of-the-art models in over 30 languages for modern extractive transformer-based Question Answering problems powerd by the ALBERT\u002FBERT\u002FDistilBERT\u002FDeBERTa\u002FRoBERTa\u002FLongformer Spark NLP Annotators trained on various SQUAD-like QA datasets for domains like Twitter, Tech, News, Biomedical COVID-19 and in various model subflavors like [sci_bert](https:\u002F\u002Fwww.aclweb.org\u002Fanthology\u002FD19-1371\u002F), [electra](https:\u002F\u002Farxiv.org\u002Fabs\u002F2003.10555), [mini_lm](https:\u002F\u002Farxiv.org\u002Fabs\u002F2002.10957), [covid_bert](https:\u002F\u002Farxiv.org\u002Fabs\u002F2005.07503), [bio_bert](https:\u002F\u002Farxiv.org\u002Fabs\u002F1901.08746), [indo_bert](https:\u002F\u002Farxiv.org\u002Fabs\u002F2011.00677), [muril](https:\u002F\u002Farxiv.org\u002Fabs\u002F2103.10730), [sapbert](https:\u002F\u002Fgithub.com\u002Fcambridgeltl\u002Fsapbert), [bioformer](https:\u002F\u002Fgithub.com\u002FWGLab\u002FBioformer), [link_bert](https:\u002F\u002Farxiv.org\u002Fabs\u002F2203.15827), [mac_bert](https:\u002F\u002Faclanthology.org\u002F2020.findings-emnlp.58\u002F)\r\n\r\n- Additionally up to 700% speedup **transformer-based Word Embeddings** on **GPU** and up to **97% speedup on CPU** for **tensorflow operations**, support for Apple M1 chips, Pyspark 3.2 and 3.3 support.\r\nOntop of this, we are now supporting Apple M1 based architectures and every Pyspark 3.X version, while deprecating support for Pyspark 2.X. \r\n\r\n- Finally, NLU-Core features various new helper methods for working with Spark NLP and embellishes now the entire universe of Annotators defined by Spark NLP and Spark NLP for healthcare.\r\n\r\n\r\n-----------------------------\r\n\r\n\r\n## NLU 4.0 for Healthcare Overview\r\n\r\n- On the healthcare side NLU features 20 Biomedical models for over 8 languages (English, French, Italian, Portuguese, Romanian, Catalan and Galician) detect entities like `HUMAN` and `SPECIES` based on [LivingNER corpus](https:\u002F\u002Ftemu.bsc.es\u002Flivingner\u002F2022\u002F05\u002F03\u002Fmultilingual-corpus\u002F) \r\n\r\n- Romanian models for Deidentification and extracting Medical entities like `Measurements`, `Form`, `Symptom`, `Route`, `Procedure`, `Disease_Syndrome_Disorder`, `Score`, `Drug_Ingredient`, `Pulse`, `Frequency`, `Date`, `Body_Part`, `Drug_Brand_Name`, `Time`, `Direction`, `Dosage`, `Medical_Device`, `Imaging_Technique`, `Test`, `Imaging_Findings`, `Imaging_Test`, `Test_Result`, `Weight`, `Clinical_Dept` and `Units` with SPELL and SPELL respectively\r\n\r\n- English NER models for parsing entities in Clinical Trial Abstracts like `Age`, `AllocationRatio`, `Author`, `BioAndMedicalUnit`, ``CTAnalysisApproach``, `CTDesign`, `Confidence`, `Country`, `DisorderOrSyndrome`, `DoseValue`, `Drug`, `DrugTime`, `Duration`, `Journal`, `NumberPatients`, `PMID`, `PValue`, `PercentagePatients`, `","2022-07-17T03:58:20",{"id":247,"version":248,"summary_zh":249,"released_at":250},106455,"v3.4.4","We are very excited to announce NLU 3.4.4 has been released with over 600 new models, over 75 new languages, and 155 languages covered in total,\r\n400% speedup for tokenizers and 18x speedup of UniversalSentenceEncoder on GPU.\r\n\r\nOn the general NLP side, we have transformer-based Embeddings and Token Classifiers powered by state of the art [CamemBertEmbeddings](https:\u002F\u002Fcamembert-model.fr\u002F) and [DeBertaForTokenClassification](https:\u002F\u002Farxiv.org\u002Fabs\u002F2006.03654) based\r\narchitectures as well as various new models for \r\n`Historical`, `Ancient`,`Dead`, `Extinct`, `Genetic`, and `Constructed` languages like\r\n`Old Church Slavonic`, `Latin`, `Sanskrit`, `Esperanto`, `Volapük`, `Coptic`, `Nahuatl`, `Ancient Greek (to 1453)`, `Old Russian`.\r\nOn the healthcare side, we have `Portuguese De-identification Models`, have `NER` models for Gene detection and finally RxNorm Sentence resolution model for mapping and extracting pharmaceutical actions (e.g. analgesic, hypoglycemic)\r\nas well as treatments (e.g. backache, diabetes).\r\n\r\nFor full release notes with all models see\r\n[here](https:\u002F\u002Fgithub.com\u002FJohnSnowLabs\u002Fnlu\u002Fpull\u002F119)\r\nor [here](https:\u002F\u002Fnlu.johnsnowlabs.com\u002Fdocs\u002Fen\u002Frelease_notes) ,\r\n\r\n\r\n\r\n## First-time language models covered\r\nThe languages for these models are covered for the very first time ever by NLU.\r\n\r\n\r\n\r\n| Number | Language Name(s) | NLU Reference | Spark NLP Reference | Task | Annotator Class | Scope | Language Type |\r\n|---------:|:----------------------------------------------------------------------------|:----------------------------------------------------------------------------------------------------------------------------------------------------|:-----------------------------------------------------------------------------------------------------------------------------------------------------------------|:-----------------------|:---------------------|:--------------|:----------------|\r\n|0 | [Sanskrit](https:\u002F\u002Fiso639-3.sil.org\u002Fcode\u002Fsan)| [sa.embed.w2v_cc_300d](https:\u002F\u002Fnlp.johnsnowlabs.com\u002F2022\u002F03\u002F16\u002Fw2v_cc_300d_sa_3_0.html) | [w2v_cc_300d](https:\u002F\u002Fnlp.johnsnowlabs.com\u002F2022\u002F03\u002F16\u002Fw2v_cc_300d_sa_3_0.html)| Embeddings | WordEmbeddingsModel| Individual| Ancient|\r\n|1 | [Sanskrit](https:\u002F\u002Fiso639-3.sil.org\u002Fcode\u002Fsan)| [sa.lemma](https:\u002F\u002Fnlp.johnsnowlabs.com\u002F2022\u002F05\u002F01\u002Flemma_vedic_sa_3_0.html) | [lemma_vedic](https:\u002F\u002Fnlp.johnsnowlabs.com\u002F2022\u002F05\u002F01\u002Flemma_vedic_sa_3_0.html)| Lemmatization | LemmatizerModel| Individual| Ancient|\r\n|2 | [Sanskrit](https:\u002F\u002Fiso639-3.sil.org\u002Fcode\u002Fsan)| [sa.pos](https:\u002F\u002Fnlp.johnsnowlabs.com\u002F2022\u002F05\u002F01\u002Fpos_vedic_sa_3_0.html) | [pos_vedic](https:\u002F\u002Fnlp.johnsnowlabs.com\u002F2022\u002F05\u002F01\u002Fpos_vedic_sa_3_0.html)| Part of Speech Tagging | PerceptronModel| Individual| Ancient|\r\n|3 | [Sanskrit](https:\u002F\u002Fiso639-3.sil.org\u002Fcode\u002Fsan)| [sa.stopwords](https:\u002F\u002Fnlp.johnsnowlabs.com\u002F2022\u002F03\u002F07\u002Fstopwords_iso_sa_3_0.html) | [stopwords_iso](https:\u002F\u002Fnlp.johnsnowlabs.com\u002F2022\u002F03\u002F07\u002Fstopwords_iso_sa_3_0.html)| Stop Words Removal | StopWordsCleaner| Individual| Ancient|\r\n|4 | [Volapük](https:\u002F\u002Fiso639-3.sil.org\u002Fcode\u002Fvol)| [vo.embed.w2v_cc_300d](https:\u002F\u002Fnlp.johnsnowlabs.com\u002F2022\u002F03\u002F16\u002Fw2v_cc_300d_vo_3_0.html) | [w2v_cc_300d](https:\u002F\u002Fnlp.johnsnowlabs.com\u002F2022\u002F03\u002F16\u002Fw2v_cc_300d_vo_3_0.html)| Embeddings | WordEmbeddingsModel| Individual| Constructed|\r\n|5 | [Nahuatl languages](https:\u002F\u002Fiso639-3.sil.org\u002Fcode\u002Fnah)| [nah.embed.w2v_cc_300d](https:\u002F\u002Fnlp.johnsnowlabs.com\u002F2022\u002F03\u002F16\u002Fw2v_cc_300d_nah_3_0.html) | [w2v_cc_300d](https:\u002F\u002Fnlp.johnsnowlabs.com\u002F2022\u002F03\u002F16\u002Fw2v_cc_300d_nah_3_0.html)| Embeddings | WordEmbeddingsModel| Collective| Genetic|\r\n|6 | [Aragonese](https:\u002F\u002Fiso639-3.sil.org\u002Fcode\u002Farg)| [an.embed.w2v_cc_300d](https:\u002F\u002Fnlp.johnsnowlabs.com\u002F2022\u002F03\u002F14\u002Fw2v_cc_300d_an_3_0.html) | [w2v_cc_300d](https:\u002F\u002Fnlp.johnsnowlabs.com\u002F2022\u002F03\u002F14\u002Fw2v_cc_300d_an_3_0.html)| Embeddings | WordEmbeddingsModel| Individual| Living|\r\n|7 | [Assamese](https:\u002F\u002Fiso639-3.sil.org\u002Fcode\u002Fasm)| [as.embed.w2v_cc_300d](https:\u002F\u002Fnlp.johnsnowlabs.com\u002F2022\u002F03\u002F14\u002Fw2v_cc_300d_as_3_0.html) | [w2v_cc_300d](https:\u002F\u002Fnlp.johnsnowlabs.com\u002F2022\u002F03\u002F14\u002Fw2v_cc_300d_as_3_0.html)| Embeddings | WordEmbeddingsModel| Individual| Living|\r\n|8 | [Asturian, Asturleonese, Bable, Leonese](https:\u002F\u002Fiso639-3.sil.org\u002Fcode\u002Fast)| [ast.embed.w2v_cc_300d](https:\u002F\u002Fnlp.johnsnowlabs.com\u002F2022\u002F03\u002F14\u002Fw2v_cc_300d_ast_3_0.html) | [w2v_cc_300d](https:\u002F\u002Fnlp.johnsnowlabs.com\u002F2022\u002F03\u002F14\u002Fw2v_cc_300d_ast_3_0.html)| Embeddings | WordEmbeddingsModel| Individual| Living|\r\n|9 | [Bashkir](https:\u002F\u002Fiso639-3.sil.org\u002Fcode\u002Fbak)| [ba.embed.w2v_cc_300d](https:\u002F\u002Fnlp.johnsnowlabs.com\u002F2022\u002F03\u002F14\u002Fw2v_cc_300d_ba_3_0.html) | [w2v_cc_300d](https:\u002F\u002Fnlp.johnsnowlabs.com\u002F2022\u002F03\u002F14\u002Fw2v_cc_300d_ba_3_0.html)| Embeddings | WordEmbeddingsModel| Individual| Living|\r\n|10 | [Bavarian](https:\u002F\u002Fiso639-3.sil.org\u002Fcode\u002Fbar) | [bar.embed.w2v_cc_300d](https:\u002F\u002Fnlp.johnsnowlabs.com\u002F2022\u002F03\u002F14\u002Fw2v_cc_300d_bar_3_0.html) | [w2v_cc_300d](https:\u002F\u002Fnlp.johnsnowlabs.com\u002F2022\u002F03\u002F14\u002Fw2v_cc_300d_bar_3_0.html)| Embeddings | WordEmbeddingsModel| Individu","2022-05-20T13:01:44",{"id":252,"version":253,"summary_zh":254,"released_at":255},106456,"v3.4.3","We are very excited to announce NLU 3.4.3 has been released!\r\n\r\nThis release features new models for `Zero-Shot-Relation-Extraction`, [DeBERTa for Sequence Classification](https:\u002F\u002Farxiv.org\u002Fabs\u002F2006.03654),\r\n`Deidentification` in `French` and `Italian` and \r\nLemmatizers, Parts of Speech Taggers, and Word2Vec Embeddings for over `66 languages`, with 20 languages being covered\r\nfor the first time by NLU, including ancient and exotic languages like `Ancient Greek`, `Old Russian`, \r\n`Old French` and much more. Once again we would like to thank our community to make this release possible.\r\n\r\n\r\n\r\n## NLU for Healthcare\r\n\r\n\r\nOn the healthcare NLP side, a new `ZeroShotRelationExtractionModel` is available, which can extract relations between\r\nclinical entities in an unsupervised fashion, no training required!\r\nAdditionally, New French and Italian Deidentification models are available for clinical and healthcare domains.\r\nPowerd by the fantastic [ Spark NLP for helathcare 3.5.0 release](https:\u002F\u002Fnlp.johnsnowlabs.com\u002Fdocs\u002Fen\u002Flicensed_release_notes)\r\n\r\n### Zero-Shot Relation Extraction\r\n\r\nZero-shot Relation Extraction to extract relations between clinical entities with no training dataset\r\n\r\n```python\r\nimport nlu\r\n\r\npipe = nlu.load('med_ner.clinical relation.zeroshot_biobert')\r\n# Configure relations to extract\r\npipe['zero_shot_relation_extraction'].setRelationalCategories({\r\n \"CURE\": [\"{{TREATMENT}} cures {{PROBLEM}}.\"],\r\n \"IMPROVE\": [\"{{TREATMENT}} improves {{PROBLEM}}.\", \"{{TREATMENT}} cures {{PROBLEM}}.\"],\r\n \"REVEAL\": [\"{{TEST}} reveals {{PROBLEM}}.\"]})\r\n.setMultiLabel(False)\r\ndf = pipe.predict(\"Paracetamol can alleviate headache or sickness. An MRI test can be used to find cancer.\")\r\ndf[\r\n 'relation', 'relation_confidence', 'relation_entity1', 'relation_entity1_class', 'relation_entity2', 'relation_entity2_class',]\r\n# Results in following table :\r\n```\r\n\r\n| relation | relation_confidence | relation_entity1 | relation_entity1_class | relation_entity2 | relation_entity2_class |\r\n|:-----------|----------------------:|:-------------------|:-------------------------|:-------------------|:-------------------------|\r\n| REVEAL | 0.976004 | An MRI test | TEST | cancer | PROBLEM |\r\n| IMPROVE | 0.988195 | Paracetamol | TREATMENT | sickness | PROBLEM |\r\n| IMPROVE | 0.992962 | Paracetamol | TREATMENT | headache | PROBLEM |\r\n\r\n### New Healthcare Models overview\r\n\r\n| Language | NLU Reference | Spark NLP Reference | Task | Annotator Class |\r\n|:-----------|:--------------------------------------------------------------------------------------------------------|:-----------------------------------------------------------------------------------------------|:-------------------------|:--------------------------------|\r\n| en | [en.relation.zeroshot_biobert](https:\u002F\u002Fnlp.johnsnowlabs.com\u002F2022\u002F04\u002F05\u002Fre_zeroshot_biobert_en_3_0.html) | [re_zeroshot_biobert](https:\u002F\u002Fnlp.johnsnowlabs.com\u002F2022\u002F04\u002F05\u002Fre_zeroshot_biobert_en_3_0.html) | Relation Extraction | ZeroShotRelationExtractionModel |\r\n| fr | [fr.med_ner.deid_generic](https:\u002F\u002Fnlp.johnsnowlabs.com\u002F2022\u002F02\u002F11\u002Fner_deid_generic_fr.html) | [ner_deid_generic](https:\u002F\u002Fnlp.johnsnowlabs.com\u002F2022\u002F02\u002F11\u002Fner_deid_generic_fr.html) | De-identification | MedicalNerModel |\r\n| fr | [fr.med_ner.deid_subentity](https:\u002F\u002Fnlp.johnsnowlabs.com\u002F2022\u002F02\u002F14\u002Fner_deid_subentity_fr.html) | [ner_deid_subentity](https:\u002F\u002Fnlp.johnsnowlabs.com\u002F2022\u002F02\u002F14\u002Fner_deid_subentity_fr.html) | De-identification | MedicalNerModel |\r\n| it | [it.med_ner.deid_generic](https:\u002F\u002Fnlp.johnsnowlabs.com\u002F2022\u002F03\u002F25\u002Fner_deid_generic_it_3_0.html) | [ner_deid_generic](https:\u002F\u002Fnlp.johnsnowlabs.com\u002F2022\u002F03\u002F25\u002Fner_deid_generic_it_3_0.html) | Named Entity Recognition | MedicalNerModel |\r\n| it | [it.med_ner.deid_subentity](https:\u002F\u002Fnlp.johnsnowlabs.com\u002F2022\u002F03\u002F22\u002Fner_deid_subentity_it_3_0.html) | [ner_deid_subentity](https:\u002F\u002Fnlp.johnsnowlabs.com\u002F2022\u002F03\u002F22\u002Fner_deid_subentity_it_3_0.html) | Named Entity Recognition | MedicalNerModel |\r\n\r\n## NLU general\r\n\r\nOn the general NLP side we have new transformer based `DeBERTa v3 sequence classifiers` models fine-tuned in Urdu, French and English for\r\nSentiment and News classification. Additionally, 100+ Part Of Speech Taggers and Lemmatizers for 66 Languages and for 7\r\nlanguages new word2vec embeddings, including `hi`,`azb`,`bo`,`diq`,`cy`,`es`,`it`, \r\npowered by the amazing [Spark NLP 3.4.3 release](https:\u002F\u002Fgit","2022-04-22T08:36:01"]