[{"data":1,"prerenderedAt":-1},["ShallowReactive",2],{"tool-kingyiusuen--image-to-latex":3,"similar-kingyiusuen--image-to-latex":104},{"id":4,"github_repo":5,"name":6,"description_en":7,"description_zh":8,"ai_summary_zh":8,"readme_en":9,"readme_zh":10,"quickstart_zh":11,"use_case_zh":12,"hero_image_url":13,"owner_login":14,"owner_name":15,"owner_avatar_url":16,"owner_bio":17,"owner_company":17,"owner_location":18,"owner_email":17,"owner_twitter":17,"owner_website":17,"owner_url":19,"languages":20,"stars":37,"forks":38,"last_commit_at":39,"license":40,"difficulty_score":41,"env_os":42,"env_gpu":43,"env_ram":42,"env_deps":44,"category_tags":57,"github_topics":61,"view_count":41,"oss_zip_url":17,"oss_zip_packed_at":17,"status":70,"created_at":71,"updated_at":72,"faqs":73,"releases":103},391,"kingyiusuen\u002Fimage-to-latex","image-to-latex","Convert images of LaTex math equations into LaTex code.","image-to-latex 是一款强大的开源应用，能够直接将包含数学公式的图片转换为标准的 LaTeX 代码。在科研写作或技术文档编辑中，手动录入复杂的数学表达式往往耗时且易错，传统 OCR 也难以精准识别特殊符号，而这款工具正是为了解决这一痛点而生。\n\n它主要面向科研人员、学生以及需要处理科学内容的开发者。无论是从论文截图中提取公式，还是将扫描文档数字化，都能显著提升工作效率。\n\n技术层面，它采用了 ResNet-18 编码器搭配 Transformer 解码器的先进架构，比早期基于 RNN 的方案更具优势。为了克服通用性问题，项目引入了图像增强技术并使用原始数据集训练，使模型能适应不同尺寸和质量的图片，而非仅局限于特定格式。此外，它还智能处理了视觉上相同但代码不同的符号规范化问题。如果你需要在图片和 LaTeX 源码间快速切换，image-to-latex 是个值得信赖的选择。","# Image to LaTeX\n\n[![Code style: black](https:\u002F\u002Fimg.shields.io\u002Fbadge\u002Fcode%20style-black-000000.svg)](https:\u002F\u002Fgithub.com\u002Fpsf\u002Fblack)\n[![pre-commit](https:\u002F\u002Fimg.shields.io\u002Fbadge\u002Fpre--commit-enabled-brightgreen?logo=pre-commit&logoColor=white)](https:\u002F\u002Fgithub.com\u002Fkingyiusuen\u002Fimage-to-latex\u002Fblob\u002Fmain\u002F.pre-commit-config.yaml)\n[![License](https:\u002F\u002Fimg.shields.io\u002Fgithub\u002Flicense\u002Fkingyiusuen\u002Fimage-to-latex)](https:\u002F\u002Fgithub.com\u002Fkingyiusuen\u002Fimage-to-latex\u002Fblob\u002Fmain\u002FLICENSE)\n\nAn application that maps an image of a LaTeX math equation to LaTeX code.\n\n\u003Cimg src=\"https:\u002F\u002Foss.gittoolsai.com\u002Fimages\u002Fkingyiusuen_image-to-latex_readme_e6c71919a6b3.gif\" alt=\"Image to Latex streamlit app\" width=\"512\">\n\n## Introduction\n\nThe problem of image-to-markup generation was attempted by [Deng et al. (2016)](https:\u002F\u002Farxiv.org\u002Fpdf\u002F1609.04938v1.pdf). They extracted about 100K formulas by parsing LaTeX sources of papers from the arXiv. They rendered the formulas using pdflatex and converted the rendered PDF files to PNG format. The raw and preprocessed versions of their dataset are available [online](http:\u002F\u002Flstm.seas.harvard.edu\u002Flatex\u002Fdata\u002F). In their model, a CNN is first used to extract image features. The rows of the features are then encoded using a RNN. Finally, the encoded features are used by an RNN decoder with an attention mechanism. The model has 9.48 million parameters in total. Recently, Transformer has overtaken RNN for many language tasks, so I thought I might give it try in this problem.\n\n## Methods\n\nUsing their dataset, I trained a model that uses ResNet-18 as encoder with 2D positional encoding and a Transformer as decoder with cross-entropy loss. (Similar to the one described in [Singh et al. (2021)](https:\u002F\u002Farxiv.org\u002Fpdf\u002F2103.06450.pdf), except that I used ResNet only up to block 3 to reduce computational costs, and I excluded the line number encoding as it doesn't apply to this problem.) The model has about 3 million parameters.\n\n\u003Cimg src=\"https:\u002F\u002Foss.gittoolsai.com\u002Fimages\u002Fkingyiusuen_image-to-latex_readme_7052886317bc.png\" alt=\"Model Architecture\" width=\"384\">\n\n\u003Csmall>Model architecture. Taken from Singh et al. (2021).\u003C\u002Fsmall>\n\nInitially, I used the preprocessed dataset to train my model, because the preprocessed images are downsampled to half of their original sizes for efficiency, and are grouped and padded into similar sizes to facilitate batching. However, this rigid preprocessing turned out to be a huge limitation. Although the model could achieve a reasonable performance on the test set (which was preprocessed the same way as the training set), it did not generalize well to images outside the dataset, most likely because the image quality, padding, and font size are so different from the images in the dataset. This phenomenon has also been observed by others who have attempted the same problem using the same dataset (e.g., [this project](https:\u002F\u002Fwandb.ai\u002Fsite\u002Farticles\u002Fimage-to-latex), [this issue](https:\u002F\u002Fgithub.com\u002Fharvardnlp\u002Fim2markup\u002Fissues\u002F12) and [this issue](https:\u002F\u002Fgithub.com\u002Fharvardnlp\u002Fim2markup\u002Fissues\u002F21)).\n\nTo this end, I used the raw dataset and included image augmentation (e.g. random scaling, gaussian noise) in my data processing pipeline to increase the diversity of the samples. Moreover, unlike Deng et al. (2016), I did not group images by size. Rather, I sampled them uniformly and padded them to the size of the largest image in the batch, so that the model must learn how to adapt to different padding sizes.\n\nAdditional problems that I faced in the dataset:\n- Some latex code produces visually identical outputs (e.g. `\\left(` and `\\right)` look the same as `(` and `)`), so I normalized them.\n- Some latex code is used to add space (e.g. `\\vspace{2px}` and `\\hspace{0.3mm}`). However, the length of the space is diffcult to judge even for humans. Also, there are many ways to express the same spacing (e.g. 1 cm = 10 mm). Finally, I don't want the model to generate code on blank images, so I removed them. (I only removed `\\vspace` and `\\hspace`, but turns out there are a lot of commands for horizontal spacing. I only realized that during error analysis. See below.)\n\n## Results\n\nThe [best run](https:\u002F\u002Fwandb.ai\u002Fkingyiusuen\u002Fimage-to-latex\u002Fruns\u002F1w1abmg1\u002F) has a character error rate (CER) of 0.17 in test set. Here is an example from the test dataset:\n\n\u003Cimg width=\"480\" src=\"https:\u002F\u002Foss.gittoolsai.com\u002Fimages\u002Fkingyiusuen_image-to-latex_readme_2a0e1472d43e.png\">\n\n- The input image and the model prediction look identical. But in the ground truth label, the horizontal spacing was created using `~`, whereas the model used `\\,`, so this was still counted as an error.\n\nI also took some screenshots in some random Wikipedia articles to see whether the model generalizes to images outside of the dataset:\n\n\u003Cimg width=\"480\" alt=\"Screen Shot 2021-08-27 at 8 06 54 AM\" src=\"https:\u002F\u002Foss.gittoolsai.com\u002Fimages\u002Fkingyiusuen_image-to-latex_readme_bb305ad2aef9.png\">\n\n- The model output is actually correct but for some reason Streamlit can't render code with `\\cal`.\n\n\u003Cimg width=\"480\" src=\"https:\u002F\u002Foss.gittoolsai.com\u002Fimages\u002Fkingyiusuen_image-to-latex_readme_f7cd0891b325.png\">\n\n- Incorrectly bolded some of the symbols.\n\nThe model also seems to have some trouble when the image is larger than what those in the dataset. Perhaps I should have increased the range of rescaling factor in the data augmentation process.\n\n## Discussion\n\nI think I should have defined the scope of the project better:\n\n- Do I want the model to tell the difference between regular-sized and large parentheses (e.g. `(`, `\\big(`, `\\Big(`, `\\bigg(`, `\\Bigg(`)?\n- Do I want the model to recognize horizontal and vertical spacing? (There are [over 40 commands for horizontal spacing](https:\u002F\u002Ftex.stackexchange.com\u002Fa\u002F74354).)\n- Do I want to the model to recognize different font styles? (Here is [a list of available font style in LaTex](https:\u002F\u002Ftex.stackexchange.com\u002Fa\u002F58124).)\n- etc.\n\nThese questions should be used to guide the data cleaning process.\n\nI found a pretty established tool called [Mathpix Snip](https:\u002F\u002Fmathpix.com\u002F) that converts handwritten formulas into LaTex code. Its [vocabulary size](https:\u002F\u002Fdocs.mathpix.com\u002F#vocabulary) is around 200. Excluding numbers and English letters, the number of LaTex commands it can produce is actually just above 100. (The vocabulary size of im2latex-100k is almost 500). It only includes two horizontal spacing commands (`\\quad` and `\\qquad`), and it doesn't recognize different sizes of parentheses. Perphas confining to a limited set of vocabulary is what I should have done, since there are so many ambiguities in real-world LaTeX.\n\nObvious possible improvements of this work include (1) training the model for more epochs (for the sake of time, I only trained the model for 15 epochs, but the validation loss is still going down), (2) using beam search (I only implemented greedy search), (3) using a larger model (e.g., use ResNet-34 instead of ResNet-18) and doing some hyperparameter tuning. I didn't do any of these, because I had limited computational resources (I was using Google Colab). But ultimately, I believe having data that don't have ambiguous labels and doing more data augmentation are the keys to the success of this problem.\n\nThe model performacne is not as good as I want to be, but I hope the lessons I learned from this project are useful to someone wants to tackle similar problems in the future.\n\n## How To Use\n\n### Setup\n\nClone the repository to your computer and position your command line inside the repository folder:\n\n```\ngit clone https:\u002F\u002Fgithub.com\u002Fkingyiusuen\u002Fimage-to-latex.git\ncd image-to-latex\n```\n\nThen, create a virtual environment named `venv` and install required packages:\n\n```\nmake venv\nmake install-dev\n```\n\n### Data Preprocessing\n\nRun the following command to download the im2latex-100k dataset and do all the preprocessing. (The image cropping step may take over an hour.)\n\n```\npython scripts\u002Fprepare_data.py\n```\n\n### Model Training and Experiment Tracking\n\n#### Model Training\n\nAn example command to start a training session:\n\n```\npython scripts\u002Frun_experiment.py trainer.gpus=1 data.batch_size=32\n```\n\nConfigurations can be modified in `conf\u002Fconfig.yaml` or in command line. See [Hydra's documentation](https:\u002F\u002Fhydra.cc\u002Fdocs\u002Fintro) to learn more.\n\n#### Experiment Tracking using Weights & Biases\n\nThe best model checkpoint will be uploaded to Weights & Biases (W&B) automatically (you will be asked to register or login to W&B before the training starts). Here is an example command to download a trained model checkpoint from W&B:\n\n```\npython scripts\u002Fdownload_checkpoint.py RUN_PATH\n```\n\nReplace RUN_PATH with the path of your run. The run path should be in the format of `\u003Centity>\u002F\u003Cproject>\u002F\u003Crun_id>`. To find the run path for a particular experiment run, go to the Overview tab in the dashboard.\n\nFor example, you can use the following command to download my best run\n\n```\npython scripts\u002Fdownload_checkpoint.py kingyiusuen\u002Fimage-to-latex\u002F1w1abmg1\n```\n\nThe checkpoint will be downloaded to a folder named `artifacts` under the project directory.\n\n### Testing and Continuous Integration\n\nThe following tools are used to lint the codebase:\n\n`isort`: Sorts and formats import statements in Python scripts.\n\n`black`: A code formatter that adheres to PEP8.\n\n`flake8`: A code linter that reports stylistic problems in Python scripts.\n\n`mypy`: Performs static type checking in Python scripts.\n\nUse the following command to run all the checkers and formatters:\n\n```\nmake lint\n```\n\nSee `pyproject.toml` and `setup.cfg` at the root directory for their configurations.\n\nSimilar checks are done automatically by the pre-commit framework when a commit is made. Check out `.pre-commit-config.yaml` for the configurations.\n\n### Deployment\n\nAn API is created to make predictions using the trained model. Use the following command to get the server up and running:\n\n```\nmake api\n```\n\nYou can explore the API via the generated documentation at http:\u002F\u002F0.0.0.0:8000\u002Fdocs.\n\nTo run the Streamlit app, create a new terminal window and use the following command:\n\n```\nmake streamlit\n```\n\nThe app should be opened in your browser automatically. You can also open it by visiting [http:\u002F\u002Flocalhost:8501](http:\u002F\u002Flocalhost:8501). For the app to work, you need to download the artifacts of an experiment run (see above) and have the API up and running.\n\nTo create a Docker image for the API:\n\n```\nmake docker\n```\n\n## Acknowledgement\n\n- This project is inspired by the project ideas section in the [final project guidelines](https:\u002F\u002Fdocs.google.com\u002Fdocument\u002Fd\u002F1pXPJ79cQeyDk3WdlYipA6YbbcoUhIVURqan_INdjjG4\u002Fedit) of the course [Full Stack Deep Learning](https:\u002F\u002Ffullstackdeeplearning.com\u002F) at UC Berkely. Some of the code is adopted from its [labs](https:\u002F\u002Fgithub.com\u002Ffull-stack-deep-learning\u002Ffsdl-text-recognizer-2021-labs\u002Ftree\u002Fmain).\n\n- [MLOps - Made with ML](https:\u002F\u002Fmadewithml.com\u002Fcourses\u002Fmlops\u002F) for introducing Makefile, pre-commit, Github Actions and Python packaging.\n\n- [harvardnlp\u002Fim2markup](https:\u002F\u002Fgithub.com\u002Fharvardnlp\u002Fim2markup) for the im2latex-100k dataset.","# 图像转 LaTeX\n\n[![Code style: black](https:\u002F\u002Fimg.shields.io\u002Fbadge\u002Fcode%20style-black-000000.svg)](https:\u002F\u002Fgithub.com\u002Fpsf\u002Fblack)\n[![pre-commit](https:\u002F\u002Fimg.shields.io\u002Fbadge\u002Fpre--commit-enabled-brightgreen?logo=pre-commit&logoColor=white)](https:\u002F\u002Fgithub.com\u002Fkingyiusuen\u002Fimage-to-latex\u002Fblob\u002Fmain\u002F.pre-commit-config.yaml)\n[![License](https:\u002F\u002Fimg.shields.io\u002Fgithub\u002Flicense\u002Fkingyiusuen\u002Fimage-to-latex)](https:\u002F\u002Fgithub.com\u002Fkingyiusuen\u002Fimage-to-latex\u002Fblob\u002Fmain\u002FLICENSE)\n\n一个将 LaTeX 数学公式图像映射为 LaTeX 代码的应用程序。\n\n\u003Cimg src=\"https:\u002F\u002Foss.gittoolsai.com\u002Fimages\u002Fkingyiusuen_image-to-latex_readme_e6c71919a6b3.gif\" alt=\"Image to Latex streamlit app\" width=\"512\">\n\n## 简介\n\n图像转标记生成问题曾由 [Deng et al. (2016)](https:\u002F\u002Farxiv.org\u002Fpdf\u002F1609.04938v1.pdf) 尝试解决。他们通过解析来自 arXiv 的论文 LaTeX 源文件提取了约 10 万个公式。他们使用 pdflatex 渲染公式，并将渲染后的 PDF 文件转换为 PNG 格式。他们数据集的原始和预处理版本可在线获取 [online](http:\u002F\u002Flstm.seas.harvard.edu\u002Flatex\u002Fdata\u002F)。在他们的模型中，首先使用 CNN（卷积神经网络）提取图像特征。随后，使用 RNN（循环神经网络）对特征的行进行编码。最后，带有注意力机制的 RNN 解码器使用这些编码后的特征。该模型总共有 948 万个参数。最近，Transformer（Transformer 架构）在许多语言任务上超越了 RNN，所以我想到在这个问题上试一试。\n\n## 方法\n\n使用该数据集，我训练了一个使用 ResNet-18 作为编码器（带有 2D 位置编码）和 Transformer 作为解码器（带有交叉熵损失）的模型。（类似于 [Singh et al. (2021)](https:\u002F\u002Farxiv.org\u002Fpdf\u002F2103.06450.pdf) 中描述的那个，除了我只使用了 ResNet 直到 block 3 以减少计算成本，并且我排除了行号编码，因为它不适用于此问题。）该模型约有 300 万个参数。\n\n\u003Cimg src=\"https:\u002F\u002Foss.gittoolsai.com\u002Fimages\u002Fkingyiusuen_image-to-latex_readme_7052886317bc.png\" alt=\"Model Architecture\" width=\"384\">\n\n\u003Csmall>模型架构。取自 Singh et al. (2021)。\u003C\u002Fsmall>\n\n最初，我使用预处理过的数据集来训练我的模型，因为预处理后的图像为了效率被下采样到原始大小的一半，并且被分组和填充到相似的大小以方便批处理。然而，这种僵化的预处理最终成为了巨大的限制。虽然模型在测试集上能达到合理的性能（测试集的预处理方式与训练集相同），但它对数据集之外的图像泛化能力不佳，最可能是因为图像质量、填充和字体大小与数据集中的图像差异太大。其他尝试使用相同数据集解决同一问题的人也观察到了这一现象（例如，[this project](https:\u002F\u002Fwandb.ai\u002Fsite\u002Farticles\u002Fimage-to-latex), [this issue](https:\u002F\u002Fgithub.com\u002Fharvardnlp\u002Fim2markup\u002Fissues\u002F12) 和 [this issue](https:\u002F\u002Fgithub.com\u002Fharvardnlp\u002Fim2markup\u002Fissues\u002F21)）。\n\n为此，我使用了原始数据集，并在数据处理流程中包含了图像增强（例如随机缩放、高斯噪声），以增加样本的多样性。此外，与 Deng et al. (2016) 不同，我没有按大小对图像进行分组。相反，我均匀地采样它们并将它们填充到批次中最大图像的大小，以便模型必须学习如何适应不同的填充大小。\n\n我在数据集中遇到的额外问题：\n- 一些 LaTeX 代码会产生视觉上相同的输出（例如 `\\left(` 和 `\\right)` 看起来与 `(` 和 `)` 相同），所以我进行了归一化。\n- 一些 LaTeX 代码用于添加空格（例如 `\\vspace{2px}` 和 `\\hspace{0.3mm}`）。然而，即使对于人类来说，空格的长度也难以判断。此外，表达相同间距的方式有很多（例如 1 cm = 10 mm）。最后，我不希望模型在空白图像上生成代码，所以我删除了它们。（我只删除了 `\\vspace` 和 `\\hspace`，但事实证明有很多用于水平间距的命令。我只是在错误分析期间才意识到这一点。见下文。）\n\n## 结果\n\n[最佳运行](https:\u002F\u002Fwandb.ai\u002Fkingyiusuen\u002Fimage-to-latex\u002Fruns\u002F1w1abmg1\u002F) 在测试集上的字符错误率 (CER) 为 0.17。这是测试数据集中的一例：\n\n\u003Cimg width=\"480\" src=\"https:\u002F\u002Foss.gittoolsai.com\u002Fimages\u002Fkingyiusuen_image-to-latex_readme_2a0e1472d43e.png\">\n\n- 输入图像和模型预测看起来完全相同。但在真实标签中，水平间距是使用 `~` 创建的，而模型使用了 `\\,`，因此这仍被计为错误。\n\n我还在一些随机的维基百科文章中截了一些图，看看模型是否能泛化到数据集之外的图像：\n\n\u003Cimg width=\"480\" alt=\"Screen Shot 2021-08-27 at 8 06 54 AM\" src=\"https:\u002F\u002Foss.gittoolsai.com\u002Fimages\u002Fkingyiusuen_image-to-latex_readme_bb305ad2aef9.png\">\n\n- 模型输出实际上是正确的，但由于某种原因 Streamlit 无法渲染包含 `\\cal` 的代码。\n\n\u003Cimg width=\"480\" src=\"https:\u002F\u002Fuser-images.githubusercontent.com\u002F14181114\u002F131130008-867e7373-67cb-44fb-abdb-b2eb2b6d6dd9.png\">\n\n- 错误地将一些符号加粗了。\n\n当图像比数据集中的图像大时，模型似乎也有一些困难。也许我应该增加数据增强过程中重缩放因子的范围。\n\n## 讨论\n\n我认为我应该更好地定义项目的范围：\n\n- 我是否希望模型能够区分常规大小和大型括号（例如 `(`, `\\big(`, `\\Big(`, `\\bigg(`, `\\Bigg(`）？\n- 我是否希望模型能够识别水平和垂直间距？（有 [超过 40 个水平间距命令](https:\u002F\u002Ftex.stackexchange.com\u002Fa\u002F74354)。）\n- 我是否希望模型能够识别不同的字体样式？（这里是 [LaTeX 中可用的字体样式列表](https:\u002F\u002Ftex.stackexchange.com\u002Fa\u002F58124)。）\n- 等等。\n\n这些问题应被用来指导数据清洗过程。\n\n我发现了一个相当成熟的工具，叫做 [Mathpix Snip](https:\u002F\u002Fmathpix.com\u002F)，它可以将手写公式转换为 LaTeX 代码。它的 [词汇量 (vocabulary size)](https:\u002F\u002Fdocs.mathpix.com\u002F#vocabulary) 约为 200。排除数字和英文字母后，它能生成的 LaTeX 命令数量实际上刚刚超过 100 个。（im2latex-100k 的词汇量接近 500。）它仅包含两个水平间距命令（`\\quad` 和 `\\qquad`），并且不识别不同大小的括号。也许限制在有限的词汇集是我应该做的，因为现实世界的 LaTeX 中存在太多歧义。\n\n这项工作明显的潜在改进包括：(1) 训练模型更多轮次 (epochs)（为了节省时间，我只训练了 15 轮，但验证损失仍在下降），(2) 使用束搜索 (beam search)（我只实现了贪婪搜索 (greedy search)），(3) 使用更大的模型（例如，使用 ResNet-34 而不是 ResNet-18）并进行一些超参数调整。我没有做这些，因为我的计算资源有限（我当时使用的是 Google Colab）。但最终，我相信拥有没有歧义标签的数据以及进行更多的数据增强是解决这个问题的关键。\n\n模型性能不如我预期的那么好，但我希望我从这个项目中学到的经验对将来想要解决类似问题的人有用。\n\n## 如何使用\n\n### 设置\n\n将仓库克隆到你的电脑上，并将命令行定位到仓库文件夹内：\n\n```\ngit clone https:\u002F\u002Fgithub.com\u002Fkingyiusuen\u002Fimage-to-latex.git\ncd image-to-latex\n```\n\n然后，创建一个名为 `venv` 的虚拟环境并安装所需的包：\n\n```\nmake venv\nmake install-dev\n```\n\n### 数据预处理\n\n运行以下命令下载 im2latex-100k 数据集并进行所有预处理。（图像裁剪步骤可能需要超过一小时。）\n\n```\npython scripts\u002Fprepare_data.py\n```\n\n### 模型训练与实验跟踪\n\n#### 模型训练\n\n开始训练会话的一个示例命令：\n\n```\npython scripts\u002Frun_experiment.py trainer.gpus=1 data.batch_size=32\n```\n\n配置可以在 `conf\u002Fconfig.yaml` 或命令行中修改。参见 [Hydra 的文档](https:\u002F\u002Fhydra.cc\u002Fdocs\u002Fintro) 了解更多。\n\n#### 使用 Weights & Biases 进行实验跟踪\n\n最佳模型检查点 (checkpoint) 将自动上传到 Weights & Biases (W&B)（在训练开始前，你会被要求注册或登录 W&B）。这里有一个从 W&B 下载已训练模型检查点的示例命令：\n\n```\npython scripts\u002Fdownload_checkpoint.py RUN_PATH\n```\n\n将 RUN_PATH 替换为你的运行路径。运行路径应为 `\u003Centity>\u002F\u003Cproject>\u002F\u003Crun_id>` 格式。要查找特定实验运行的运行路径，请前往仪表板的概览选项卡。\n\n例如，你可以使用以下命令下载我最好的运行结果\n\n```\npython scripts\u002Fdownload_checkpoint.py kingyiusuen\u002Fimage-to-latex\u002F1w1abmg1\n```\n\n检查点将被下载到项目目录下的一个名为 `artifacts` 的文件夹中。\n\n### 测试与持续集成\n\n以下工具用于检查代码库：\n\n`isort`：排序和格式化 Python 脚本中的导入语句。\n\n`black`：遵循 PEP8 的代码格式化器。\n\n`flake8`：报告 Python 脚本中风格问题的代码检查器。\n\n`mypy`：执行 Python 脚本中的静态类型检查。\n\n使用以下命令运行所有检查器和格式化器：\n\n```\nmake lint\n```\n\n有关它们的配置，请参阅根目录下的 `pyproject.toml` 和 `setup.cfg`。\n\n类似的检查会在提交时由 pre-commit (预提交) 框架自动完成。查看 `.pre-commit-config.yaml` 获取配置。\n\n### 部署\n\n创建了一个 API (应用程序编程接口) 来使用训练好的模型进行预测。使用以下命令启动服务器：\n\n```\nmake api\n```\n\n你可以通过生成的文档 http:\u002F\u002F0.0.0.0:8000\u002Fdocs 探索该 API。\n\n要运行 Streamlit 应用，请打开一个新的终端窗口并使用以下命令：\n\n```\nmake streamlit\n```\n\n应用应该会自动在你的浏览器中打开。你也可以通过访问 [http:\u002F\u002Flocalhost:8501](http:\u002F\u002Flocalhost:8501) 打开它。为了让应用工作，你需要下载实验运行的产物（见上文）并确保 API 正在运行。\n\n为 API 创建 Docker (容器镜像技术) 镜像：\n\n```\nmake docker\n```\n\n## 致谢\n\n- 本项目灵感来源于 UC Berkeley 课程 [Full Stack Deep Learning](https:\u002F\u002Ffullstackdeeplearning.com\u002F) 的 [最终项目指南](https:\u002F\u002Fdocs.google.com\u002Fdocument\u002Fd\u002F1pXPJ79cQeyDk3WdlYipA6YbbcoUhIVURqan_INdjjG4\u002Fedit) 中的项目创意部分。部分代码采用了其 [实验室](https:\u002F\u002Fgithub.com\u002Ffull-stack-deep-learning\u002Ffsdl-text-recognizer-2021-labs\u002Ftree\u002Fmain) 的内容。\n\n- [MLOps - Made with ML](https:\u002F\u002Fmadewithml.com\u002Fcourses\u002Fmlops\u002F) 介绍了 Makefile、pre-commit、Github Actions 和 Python 打包。\n\n- [harvardnlp\u002Fim2markup](https:\u002F\u002Fgithub.com\u002Fharvardnlp\u002Fim2markup) 提供了 im2latex-100k 数据集。","# image-to-latex 快速上手指南\n\n**image-to-latex** 是一个开源工具，能够将包含 LaTeX 数学公式的图片自动转换为对应的 LaTeX 代码。本项目基于 ResNet-18 和 Transformer 架构，支持通过 Web 界面进行交互。\n\n## 环境准备\n\n在开始之前，请确保您的开发环境满足以下要求：\n\n- **操作系统**：Linux \u002F macOS \u002F Windows (推荐使用 WSL)\n- **编程语言**：Python 3.x\n- **必备工具**：Git, Make\n- **硬件建议**：NVIDIA GPU (CUDA 兼容) 可加速模型训练与推理；CPU 也可运行但速度较慢。\n- **网络环境**：由于涉及 GitHub 克隆及 Weights & Biases (W&B) 下载，国内用户可能需要配置代理或镜像源以确保连接稳定。\n\n## 安装步骤\n\n### 1. 克隆仓库\n将项目代码下载到本地并进入目录：\n```bash\ngit clone https:\u002F\u002Fgithub.com\u002Fkingyiusuen\u002Fimage-to-latex.git\ncd image-to-latex\n```\n\n### 2. 创建虚拟环境并安装依赖\n创建名为 `venv` 的虚拟环境并安装开发所需包：\n```bash\nmake venv\nmake install-dev\n```\n\n### 3. 获取预训练模型\n为了快速体验，无需从头训练模型，可直接下载作者提供的最佳检查点（Checkpoint）：\n```bash\npython scripts\u002Fdownload_checkpoint.py kingyiusuen\u002Fimage-to-latex\u002F1w1abmg1\n```\n*注：该命令会将模型文件下载至项目目录下的 `artifacts` 文件夹中。*\n\n### 4. 数据预处理（可选）\n如果您计划自行训练模型，需先下载并处理数据集（耗时约 1 小时以上）：\n```bash\npython scripts\u002Fprepare_data.py\n```\n\n## 基本使用\n\n本工具提供 API 服务和 Streamlit 网页应用两种方式。最简单的方式是直接启动 Streamlit 应用。\n\n### 启动服务与应用\n\n1. **启动后端 API 服务**\n   在终端运行以下命令以启动预测服务器：\n   ```bash\n   make api\n   ```\n   服务默认运行在 `http:\u002F\u002F0.0.0.0:8000`，API 文档可在 `http:\u002F\u002F0.0.0.0:8000\u002Fdocs` 查看。\n\n2. **启动 Streamlit 前端应用**\n   打开新的终端窗口，运行以下命令：\n   ```bash\n   make streamlit\n   ```\n   浏览器将自动打开应用页面（或访问 `http:\u002F\u002Flocalhost:8501`）。\n\n3. **上传图片转换**\n   在网页界面中上传包含数学公式的图片，系统将自动识别并生成对应的 LaTeX 代码。\n\n---\n*提示：若遇到网络问题导致无法下载模型或启动服务，请检查网络连接或配置相关代理。*","研究生小李在撰写机器学习方向的学术论文时，需要复现一篇经典文献中带有复杂分式与求和符号的推导公式，但手头只有扫描版的 PDF 截图。\n\n### 没有 image-to-latex 时\n- 手动逐字输入 LaTeX 代码极其耗时，一个中等难度的公式往往需要十几分钟。\n- 极易出现括号不匹配、下标位置错误或特殊符号遗漏等低级语法问题。\n- 遇到生僻的数学环境（如多行对齐）时，需频繁查阅文档确认语法细节。\n- 反复调整编译后的排版效果，严重拖慢论文定稿的整体进度。\n\n### 使用 image-to-latex 后\n- 只需将公式截图拖入界面，image-to-latex 即可快速输出对应的源代码。\n- 模型基于 Transformer 架构，能准确还原复杂的嵌套结构与特殊符号组合。\n- 生成的代码经过标准化处理，直接复制粘贴到编辑器中即可正常编译显示。\n- 大幅减少人工校对时间，让研究者能将精力集中在核心逻辑验证而非格式上。\n\n通过图像转代码的自动化能力，image-to-latex 彻底解决了从视觉素材到可编辑文本的转换瓶颈。","https:\u002F\u002Foss.gittoolsai.com\u002Fimages\u002Fkingyiusuen_image-to-latex_e6c71919.gif","kingyiusuen","King Yiu Suen","https:\u002F\u002Foss.gittoolsai.com\u002Favatars\u002Fkingyiusuen_408b7c73.png",null,"Hong Kong","https:\u002F\u002Fgithub.com\u002Fkingyiusuen",[21,25,29,33],{"name":22,"color":23,"percentage":24},"Python","#3572A5",93.4,{"name":26,"color":27,"percentage":28},"Makefile","#427819",4.1,{"name":30,"color":31,"percentage":32},"Dockerfile","#384d54",1.3,{"name":34,"color":35,"percentage":36},"Shell","#89e051",1.2,2163,313,"2026-04-05T13:53:44","MIT",3,"未说明","需要 GPU（训练场景），具体型号和显存未说明",{"notes":45,"python":42,"dependencies":46},"首次运行需下载 im2latex-100k 数据集并预处理（耗时超 1 小时）；训练建议使用 Google Colab 或配备 GPU 的机器；使用 Streamlit 应用前需下载模型检查点；需配置 Weights & Biases 账号以跟踪实验。",[47,48,49,50,51,52,53,54,55,56],"torch","torchvision","hydra-core","wandb","streamlit","black","pre-commit","isort","flake8","mypy",[58,59,60],"开发框架","语言模型","图像",[62,63,64,65,66,67,68,69],"pytorch","deep-learning","transformer","encoder-decoder","streamlit-webapp","latex","end-to-end-machine-learning","computer-vision","ready","2026-03-27T02:49:30.150509","2026-04-06T07:23:07.099550",[74,79,83,88,93,98],{"id":75,"question_zh":76,"answer_zh":77,"source_url":78},1437,"为什么使用作者提供的最佳模型效果不理想？","即使最佳运行的字符错误率 (CER) 也有 17%。该项目原为学生项目，开发周期较短。建议尝试调整图像尺寸以接近训练数据集的分布，可能有助于提升效果。","https:\u002F\u002Fgithub.com\u002Fkingyiusuen\u002Fimage-to-latex\u002Fissues\u002F7",{"id":80,"question_zh":81,"answer_zh":82,"source_url":78},1438,"使用模型时输入图像应该调整到什么尺寸？","建议将输入图像调整大小，使其尺寸更接近训练数据集中的图像。有用户在转换多项式公式时遇到同样问题，反馈减小测试图像尺寸后 API 推理效果显著改善。",{"id":84,"question_zh":85,"answer_zh":86,"source_url":87},1439,"Windows 环境下安装 editdistance 库报错如何解决？","编译错误可能与源码中的非 ASCII 字符有关。尝试删除 `editdistance-master\\editdistance\\_editdistance.cpp` 文件中的日文字符。也可参考相关配置教程链接进行环境配置。","https:\u002F\u002Fgithub.com\u002Fkingyiusuen\u002Fimage-to-latex\u002Fissues\u002F5",{"id":89,"question_zh":90,"answer_zh":91,"source_url":92},1440,"在 im2latex-100K 数据集上能达到的最佳性能指标是什么？","测试集字符错误率 (CER) 为 0.17。通过超参数调优和更好的数据清洗可以获得更好的结果。","https:\u002F\u002Fgithub.com\u002Fkingyiusuen\u002Fimage-to-latex\u002Fissues\u002F4",{"id":94,"question_zh":95,"answer_zh":96,"source_url":97},1441,"本地训练模型表现良好，但通过 API 部署后结果很差，原因是什么？","可能是图像尺寸不匹配导致。有用户反馈通过减小测试图像的尺寸解决了 API 推理效果差的问题，建议检查输入图像的分辨率设置。","https:\u002F\u002Fgithub.com\u002Fkingyiusuen\u002Fimage-to-latex\u002Fissues\u002F18",{"id":99,"question_zh":100,"answer_zh":101,"source_url":102},1442,"使用自定义数据训练时出现 `indexSelectLargeIndex` 断言错误如何解决？","该错误表明索引越界（`srcIndex \u003C srcSelectDimSize`）。通常是因为自定义数据中的 token ID 超过了 `vocab.json` 中定义的词汇表大小，请检查并确保所有 token 都在词汇表内。","https:\u002F\u002Fgithub.com\u002Fkingyiusuen\u002Fimage-to-latex\u002Fissues\u002F6",[],[105,114,123,131,139,152],{"id":106,"name":107,"github_repo":108,"description_zh":109,"stars":110,"difficulty_score":41,"last_commit_at":111,"category_tags":112,"status":70},3808,"stable-diffusion-webui","AUTOMATIC1111\u002Fstable-diffusion-webui","stable-diffusion-webui 是一个基于 Gradio 构建的网页版操作界面，旨在让用户能够轻松地在本地运行和使用强大的 Stable Diffusion 图像生成模型。它解决了原始模型依赖命令行、操作门槛高且功能分散的痛点，将复杂的 AI 绘图流程整合进一个直观易用的图形化平台。\n\n无论是希望快速上手的普通创作者、需要精细控制画面细节的设计师，还是想要深入探索模型潜力的开发者与研究人员，都能从中获益。其核心亮点在于极高的功能丰富度：不仅支持文生图、图生图、局部重绘（Inpainting）和外绘（Outpainting）等基础模式，还独创了注意力机制调整、提示词矩阵、负向提示词以及“高清修复”等高级功能。此外，它内置了 GFPGAN 和 CodeFormer 等人脸修复工具，支持多种神经网络放大算法，并允许用户通过插件系统无限扩展能力。即使是显存有限的设备，stable-diffusion-webui 也提供了相应的优化选项，让高质量的 AI 艺术创作变得触手可及。",162132,"2026-04-05T11:01:52",[58,60,113],"Agent",{"id":115,"name":116,"github_repo":117,"description_zh":118,"stars":119,"difficulty_score":120,"last_commit_at":121,"category_tags":122,"status":70},1381,"everything-claude-code","affaan-m\u002Feverything-claude-code","everything-claude-code 是一套专为 AI 编程助手（如 Claude Code、Codex、Cursor 等）打造的高性能优化系统。它不仅仅是一组配置文件，而是一个经过长期实战打磨的完整框架，旨在解决 AI 代理在实际开发中面临的效率低下、记忆丢失、安全隐患及缺乏持续学习能力等核心痛点。\n\n通过引入技能模块化、直觉增强、记忆持久化机制以及内置的安全扫描功能，everything-claude-code 能显著提升 AI 在复杂任务中的表现，帮助开发者构建更稳定、更智能的生产级 AI 代理。其独特的“研究优先”开发理念和针对 Token 消耗的优化策略，使得模型响应更快、成本更低，同时有效防御潜在的攻击向量。\n\n这套工具特别适合软件开发者、AI 研究人员以及希望深度定制 AI 工作流的技术团队使用。无论您是在构建大型代码库，还是需要 AI 协助进行安全审计与自动化测试，everything-claude-code 都能提供强大的底层支持。作为一个曾荣获 Anthropic 黑客大奖的开源项目，它融合了多语言支持与丰富的实战钩子（hooks），让 AI 真正成长为懂上",138956,2,"2026-04-05T11:33:21",[58,113,59],{"id":124,"name":125,"github_repo":126,"description_zh":127,"stars":128,"difficulty_score":120,"last_commit_at":129,"category_tags":130,"status":70},2271,"ComfyUI","Comfy-Org\u002FComfyUI","ComfyUI 是一款功能强大且高度模块化的视觉 AI 引擎，专为设计和执行复杂的 Stable Diffusion 图像生成流程而打造。它摒弃了传统的代码编写模式，采用直观的节点式流程图界面，让用户通过连接不同的功能模块即可构建个性化的生成管线。\n\n这一设计巧妙解决了高级 AI 绘图工作流配置复杂、灵活性不足的痛点。用户无需具备编程背景，也能自由组合模型、调整参数并实时预览效果，轻松实现从基础文生图到多步骤高清修复等各类复杂任务。ComfyUI 拥有极佳的兼容性，不仅支持 Windows、macOS 和 Linux 全平台，还广泛适配 NVIDIA、AMD、Intel 及苹果 Silicon 等多种硬件架构，并率先支持 SDXL、Flux、SD3 等前沿模型。\n\n无论是希望深入探索算法潜力的研究人员和开发者，还是追求极致创作自由度的设计师与资深 AI 绘画爱好者，ComfyUI 都能提供强大的支持。其独特的模块化架构允许社区不断扩展新功能，使其成为当前最灵活、生态最丰富的开源扩散模型工具之一，帮助用户将创意高效转化为现实。",107662,"2026-04-03T11:11:01",[58,60,113],{"id":132,"name":133,"github_repo":134,"description_zh":135,"stars":136,"difficulty_score":120,"last_commit_at":137,"category_tags":138,"status":70},3704,"NextChat","ChatGPTNextWeb\u002FNextChat","NextChat 是一款轻量且极速的 AI 助手，旨在为用户提供流畅、跨平台的大模型交互体验。它完美解决了用户在多设备间切换时难以保持对话连续性，以及面对众多 AI 模型不知如何统一管理的痛点。无论是日常办公、学习辅助还是创意激发，NextChat 都能让用户随时随地通过网页、iOS、Android、Windows、MacOS 或 Linux 端无缝接入智能服务。\n\n这款工具非常适合普通用户、学生、职场人士以及需要私有化部署的企业团队使用。对于开发者而言，它也提供了便捷的自托管方案，支持一键部署到 Vercel 或 Zeabur 等平台。\n\nNextChat 的核心亮点在于其广泛的模型兼容性，原生支持 Claude、DeepSeek、GPT-4 及 Gemini Pro 等主流大模型，让用户在一个界面即可自由切换不同 AI 能力。此外，它还率先支持 MCP（Model Context Protocol）协议，增强了上下文处理能力。针对企业用户，NextChat 提供专业版解决方案，具备品牌定制、细粒度权限控制、内部知识库整合及安全审计等功能，满足公司对数据隐私和个性化管理的高标准要求。",87618,"2026-04-05T07:20:52",[58,59],{"id":140,"name":141,"github_repo":142,"description_zh":143,"stars":144,"difficulty_score":120,"last_commit_at":145,"category_tags":146,"status":70},2268,"ML-For-Beginners","microsoft\u002FML-For-Beginners","ML-For-Beginners 是由微软推出的一套系统化机器学习入门课程，旨在帮助零基础用户轻松掌握经典机器学习知识。这套课程将学习路径规划为 12 周，包含 26 节精炼课程和 52 道配套测验，内容涵盖从基础概念到实际应用的完整流程，有效解决了初学者面对庞大知识体系时无从下手、缺乏结构化指导的痛点。\n\n无论是希望转型的开发者、需要补充算法背景的研究人员，还是对人工智能充满好奇的普通爱好者，都能从中受益。课程不仅提供了清晰的理论讲解，还强调动手实践，让用户在循序渐进中建立扎实的技能基础。其独特的亮点在于强大的多语言支持，通过自动化机制提供了包括简体中文在内的 50 多种语言版本，极大地降低了全球不同背景用户的学习门槛。此外，项目采用开源协作模式，社区活跃且内容持续更新，确保学习者能获取前沿且准确的技术资讯。如果你正寻找一条清晰、友好且专业的机器学习入门之路，ML-For-Beginners 将是理想的起点。",84991,"2026-04-05T10:45:23",[60,147,148,149,113,150,59,58,151],"数据工具","视频","插件","其他","音频",{"id":153,"name":154,"github_repo":155,"description_zh":156,"stars":157,"difficulty_score":41,"last_commit_at":158,"category_tags":159,"status":70},3128,"ragflow","infiniflow\u002Fragflow","RAGFlow 是一款领先的开源检索增强生成（RAG）引擎，旨在为大语言模型构建更精准、可靠的上下文层。它巧妙地将前沿的 RAG 技术与智能体（Agent）能力相结合，不仅支持从各类文档中高效提取知识，还能让模型基于这些知识进行逻辑推理和任务执行。\n\n在大模型应用中，幻觉问题和知识滞后是常见痛点。RAGFlow 通过深度解析复杂文档结构（如表格、图表及混合排版），显著提升了信息检索的准确度，从而有效减少模型“胡编乱造”的现象，确保回答既有据可依又具备时效性。其内置的智能体机制更进一步，使系统不仅能回答问题，还能自主规划步骤解决复杂问题。\n\n这款工具特别适合开发者、企业技术团队以及 AI 研究人员使用。无论是希望快速搭建私有知识库问答系统，还是致力于探索大模型在垂直领域落地的创新者，都能从中受益。RAGFlow 提供了可视化的工作流编排界面和灵活的 API 接口，既降低了非算法背景用户的上手门槛，也满足了专业开发者对系统深度定制的需求。作为基于 Apache 2.0 协议开源的项目，它正成为连接通用大模型与行业专有知识之间的重要桥梁。",77062,"2026-04-04T04:44:48",[113,60,58,59,150]]