[{"data":1,"prerenderedAt":-1},["ShallowReactive",2],{"similar-utkuozbulak--pytorch-custom-dataset-examples":3,"tool-utkuozbulak--pytorch-custom-dataset-examples":61},[4,18,26,36,44,53],{"id":5,"name":6,"github_repo":7,"description_zh":8,"stars":9,"difficulty_score":10,"last_commit_at":11,"category_tags":12,"status":17},4358,"openclaw","openclaw\u002Fopenclaw","OpenClaw 是一款专为个人打造的本地化 AI 助手，旨在让你在自己的设备上拥有完全可控的智能伙伴。它打破了传统 AI 助手局限于特定网页或应用的束缚，能够直接接入你日常使用的各类通讯渠道，包括微信、WhatsApp、Telegram、Discord、iMessage 等数十种平台。无论你在哪个聊天软件中发送消息，OpenClaw 都能即时响应，甚至支持在 macOS、iOS 和 Android 设备上进行语音交互，并提供实时的画布渲染功能供你操控。\n\n这款工具主要解决了用户对数据隐私、响应速度以及“始终在线”体验的需求。通过将 AI 部署在本地，用户无需依赖云端服务即可享受快速、私密的智能辅助，真正实现了“你的数据，你做主”。其独特的技术亮点在于强大的网关架构，将控制平面与核心助手分离，确保跨平台通信的流畅性与扩展性。\n\nOpenClaw 非常适合希望构建个性化工作流的技术爱好者、开发者，以及注重隐私保护且不愿被单一生态绑定的普通用户。只要具备基础的终端操作能力（支持 macOS、Linux 及 Windows WSL2），即可通过简单的命令行引导完成部署。如果你渴望拥有一个懂你",349277,3,"2026-04-06T06:32:30",[13,14,15,16],"Agent","开发框架","图像","数据工具","ready",{"id":19,"name":20,"github_repo":21,"description_zh":22,"stars":23,"difficulty_score":10,"last_commit_at":24,"category_tags":25,"status":17},3808,"stable-diffusion-webui","AUTOMATIC1111\u002Fstable-diffusion-webui","stable-diffusion-webui 是一个基于 Gradio 构建的网页版操作界面，旨在让用户能够轻松地在本地运行和使用强大的 Stable Diffusion 图像生成模型。它解决了原始模型依赖命令行、操作门槛高且功能分散的痛点，将复杂的 AI 绘图流程整合进一个直观易用的图形化平台。\n\n无论是希望快速上手的普通创作者、需要精细控制画面细节的设计师，还是想要深入探索模型潜力的开发者与研究人员，都能从中获益。其核心亮点在于极高的功能丰富度：不仅支持文生图、图生图、局部重绘（Inpainting）和外绘（Outpainting）等基础模式，还独创了注意力机制调整、提示词矩阵、负向提示词以及“高清修复”等高级功能。此外，它内置了 GFPGAN 和 CodeFormer 等人脸修复工具，支持多种神经网络放大算法，并允许用户通过插件系统无限扩展能力。即使是显存有限的设备，stable-diffusion-webui 也提供了相应的优化选项，让高质量的 AI 艺术创作变得触手可及。",162132,"2026-04-05T11:01:52",[14,15,13],{"id":27,"name":28,"github_repo":29,"description_zh":30,"stars":31,"difficulty_score":32,"last_commit_at":33,"category_tags":34,"status":17},1381,"everything-claude-code","affaan-m\u002Feverything-claude-code","everything-claude-code 是一套专为 AI 编程助手（如 Claude Code、Codex、Cursor 等）打造的高性能优化系统。它不仅仅是一组配置文件，而是一个经过长期实战打磨的完整框架，旨在解决 AI 代理在实际开发中面临的效率低下、记忆丢失、安全隐患及缺乏持续学习能力等核心痛点。\n\n通过引入技能模块化、直觉增强、记忆持久化机制以及内置的安全扫描功能，everything-claude-code 能显著提升 AI 在复杂任务中的表现，帮助开发者构建更稳定、更智能的生产级 AI 代理。其独特的“研究优先”开发理念和针对 Token 消耗的优化策略，使得模型响应更快、成本更低，同时有效防御潜在的攻击向量。\n\n这套工具特别适合软件开发者、AI 研究人员以及希望深度定制 AI 工作流的技术团队使用。无论您是在构建大型代码库，还是需要 AI 协助进行安全审计与自动化测试，everything-claude-code 都能提供强大的底层支持。作为一个曾荣获 Anthropic 黑客大奖的开源项目，它融合了多语言支持与丰富的实战钩子（hooks），让 AI 真正成长为懂上",160411,2,"2026-04-18T23:33:24",[14,13,35],"语言模型",{"id":37,"name":38,"github_repo":39,"description_zh":40,"stars":41,"difficulty_score":32,"last_commit_at":42,"category_tags":43,"status":17},2271,"ComfyUI","Comfy-Org\u002FComfyUI","ComfyUI 是一款功能强大且高度模块化的视觉 AI 引擎，专为设计和执行复杂的 Stable Diffusion 图像生成流程而打造。它摒弃了传统的代码编写模式，采用直观的节点式流程图界面，让用户通过连接不同的功能模块即可构建个性化的生成管线。\n\n这一设计巧妙解决了高级 AI 绘图工作流配置复杂、灵活性不足的痛点。用户无需具备编程背景，也能自由组合模型、调整参数并实时预览效果，轻松实现从基础文生图到多步骤高清修复等各类复杂任务。ComfyUI 拥有极佳的兼容性，不仅支持 Windows、macOS 和 Linux 全平台，还广泛适配 NVIDIA、AMD、Intel 及苹果 Silicon 等多种硬件架构，并率先支持 SDXL、Flux、SD3 等前沿模型。\n\n无论是希望深入探索算法潜力的研究人员和开发者，还是追求极致创作自由度的设计师与资深 AI 绘画爱好者，ComfyUI 都能提供强大的支持。其独特的模块化架构允许社区不断扩展新功能，使其成为当前最灵活、生态最丰富的开源扩散模型工具之一，帮助用户将创意高效转化为现实。",109154,"2026-04-18T11:18:24",[14,15,13],{"id":45,"name":46,"github_repo":47,"description_zh":48,"stars":49,"difficulty_score":32,"last_commit_at":50,"category_tags":51,"status":17},6121,"gemini-cli","google-gemini\u002Fgemini-cli","gemini-cli 是一款由谷歌推出的开源 AI 命令行工具，它将强大的 Gemini 大模型能力直接集成到用户的终端环境中。对于习惯在命令行工作的开发者而言，它提供了一条从输入提示词到获取模型响应的最短路径，无需切换窗口即可享受智能辅助。\n\n这款工具主要解决了开发过程中频繁上下文切换的痛点，让用户能在熟悉的终端界面内直接完成代码理解、生成、调试以及自动化运维任务。无论是查询大型代码库、根据草图生成应用，还是执行复杂的 Git 操作，gemini-cli 都能通过自然语言指令高效处理。\n\n它特别适合广大软件工程师、DevOps 人员及技术研究人员使用。其核心亮点包括支持高达 100 万 token 的超长上下文窗口，具备出色的逻辑推理能力；内置 Google 搜索、文件操作及 Shell 命令执行等实用工具；更独特的是，它支持 MCP（模型上下文协议），允许用户灵活扩展自定义集成，连接如图像生成等外部能力。此外，个人谷歌账号即可享受免费的额度支持，且项目基于 Apache 2.0 协议完全开源，是提升终端工作效率的理想助手。",100752,"2026-04-10T01:20:03",[52,13,15,14],"插件",{"id":54,"name":55,"github_repo":56,"description_zh":57,"stars":58,"difficulty_score":32,"last_commit_at":59,"category_tags":60,"status":17},4721,"markitdown","microsoft\u002Fmarkitdown","MarkItDown 是一款由微软 AutoGen 团队打造的轻量级 Python 工具，专为将各类文件高效转换为 Markdown 格式而设计。它支持 PDF、Word、Excel、PPT、图片（含 OCR）、音频（含语音转录）、HTML 乃至 YouTube 链接等多种格式的解析，能够精准提取文档中的标题、列表、表格和链接等关键结构信息。\n\n在人工智能应用日益普及的今天，大语言模型（LLM）虽擅长处理文本，却难以直接读取复杂的二进制办公文档。MarkItDown 恰好解决了这一痛点，它将非结构化或半结构化的文件转化为模型“原生理解”且 Token 效率极高的 Markdown 格式，成为连接本地文件与 AI 分析 pipeline 的理想桥梁。此外，它还提供了 MCP（模型上下文协议）服务器，可无缝集成到 Claude Desktop 等 LLM 应用中。\n\n这款工具特别适合开发者、数据科学家及 AI 研究人员使用，尤其是那些需要构建文档检索增强生成（RAG）系统、进行批量文本分析或希望让 AI 助手直接“阅读”本地文件的用户。虽然生成的内容也具备一定可读性，但其核心优势在于为机器",93400,"2026-04-06T19:52:38",[52,14],{"id":62,"github_repo":63,"name":64,"description_en":65,"description_zh":66,"ai_summary_zh":67,"readme_en":68,"readme_zh":69,"quickstart_zh":70,"use_case_zh":71,"hero_image_url":72,"owner_login":73,"owner_name":74,"owner_avatar_url":75,"owner_bio":76,"owner_company":77,"owner_location":78,"owner_email":79,"owner_twitter":80,"owner_website":81,"owner_url":82,"languages":83,"stars":88,"forks":89,"last_commit_at":90,"license":91,"difficulty_score":92,"env_os":93,"env_gpu":94,"env_ram":94,"env_deps":95,"category_tags":103,"github_topics":80,"view_count":32,"oss_zip_url":80,"oss_zip_packed_at":80,"status":17,"created_at":104,"updated_at":105,"faqs":106,"releases":142},9521,"utkuozbulak\u002Fpytorch-custom-dataset-examples","pytorch-custom-dataset-examples","Some custom dataset examples for PyTorch","pytorch-custom-dataset-examples 是一个专为 PyTorch 初学者和开发者打造的开源教程仓库，旨在通过丰富的实战案例，帮助用户轻松掌握自定义数据集（Custom Dataset）的构建方法。许多新手在使用 PyTorch 官方文档时，常因示例过于抽象而难以理解如何将本地数据（如图片、CSV 文件）转化为模型可训练的格式。本工具直击这一痛点，系统地拆解了从基础骨架到高级应用的完整流程。\n\n它详细演示了如何编写核心的 `__init__`、`__getitem__` 和 `__len__` 函数，并展示了多种实用场景：包括结合 torchvision 进行图像增强、利用 pandas 读取表格数据、在文件名中嵌入类别标签以及灵活设计数据加载逻辑等。其独特的技术亮点在于不仅提供了标准写法，还分享了作者历经多年项目迭代后的“极简”实现思路，强调了自定义数据集的灵活性——用户完全可以根据需求自由塑造数据结构。\n\n无论是刚入门深度学习的学生、需要快速验证想法的研究人员，还是正在构建复杂数据管道的工程师，都能从中找到适合自己的参考方案。通过阅读这些清晰的代码示例，用户","pytorch-custom-dataset-examples 是一个专为 PyTorch 初学者和开发者打造的开源教程仓库，旨在通过丰富的实战案例，帮助用户轻松掌握自定义数据集（Custom Dataset）的构建方法。许多新手在使用 PyTorch 官方文档时，常因示例过于抽象而难以理解如何将本地数据（如图片、CSV 文件）转化为模型可训练的格式。本工具直击这一痛点，系统地拆解了从基础骨架到高级应用的完整流程。\n\n它详细演示了如何编写核心的 `__init__`、`__getitem__` 和 `__len__` 函数，并展示了多种实用场景：包括结合 torchvision 进行图像增强、利用 pandas 读取表格数据、在文件名中嵌入类别标签以及灵活设计数据加载逻辑等。其独特的技术亮点在于不仅提供了标准写法，还分享了作者历经多年项目迭代后的“极简”实现思路，强调了自定义数据集的灵活性——用户完全可以根据需求自由塑造数据结构。\n\n无论是刚入门深度学习的学生、需要快速验证想法的研究人员，还是正在构建复杂数据管道的工程师，都能从中找到适合自己的参考方案。通过阅读这些清晰的代码示例，用户可以省去查阅海量文档的时间，快速建立起对 PyTorch 数据加载机制的直观理解，从而更高效地推进自己的 AI 项目。","\u003Cp align=\"center\">\u003Cimg width=\"40%\" src=\"https:\u002F\u002Foss.gittoolsai.com\u002Fimages\u002Futkuozbulak_pytorch-custom-dataset-examples_readme_28a00358044f.png\" \u002F>\u003C\u002Fp>\n\n--------------------------------------------------------------------------------\n\n# PyTorch Custom Dataset Examples\n\n*Update after two years:* It has been a long time since I have created this repository to guide people who are getting started with pytorch (like myself back then). However, over the course of years and various projects, the way I create my datasets changed many times. I included an additional bare bone dataset [here](#a-custom-custom-custom-dataset) to show what I am currently using. I would like to note that the reason why custom datasets are called custom is because you can shape it in anyway you desire. So, it is only natural that you (the reader) will develop your way of creating custom datasets after working on different projects. Examples presented in this project are not there as the ultimate way of creating them but instead, there to show the flexibility and the possiblity of pytorch datasets. I hope this repository is\u002Fwas useful in your understanding of pytorch datasets. \n\nThere are some official custom dataset examples on PyTorch repo like [this](https:\u002F\u002Fgithub.com\u002Fpytorch\u002Ftutorials\u002Fblob\u002Fmaster\u002Fbeginner_source\u002Fdata_loading_tutorial.py) but they still seemed a bit obscure to a beginner (like me, back then) so I had to spend some time understanding what exactly I needed to have a fully customized dataset. To save you the trouble of going through bajillions of pages, here, I decided to write down the basics of Pytorch datasets. The topics are as follows.\n\n# Topics\n- [Custom Dataset fundamentals](#custom-dataset-fundamentals)\n- [Using torchvision transforms](#using-torchvision-transforms)\n- [Another way to use torchvision transforms](#another-way-to-use-torchvision-transforms)\n- [Incorporating pandas (reading csv)](#incorporating-pandas)\n- [Incorporating pandas with more logic in `__getitem__()`](#incorporating-pandas-with-more-logic)\n- [Embedding classes into file names](#a-custom-custom-custom-dataset)\n- [Torchvision transforms: to use or not to use?](#a-custom-custom-custom-dataset)\n- [Using data loader with custom datasets](#using-data-loader)\n- [Future updates (hopefully)](#future-updates)\n\n## Custom Dataset Fundamentals\nThe first and foremost part is creating a dataset class.\n\n```python\nfrom torch.utils.data.dataset import Dataset\n\nclass MyCustomDataset(Dataset):\n    def __init__(self, ...):\n        # stuff\n        \n    def __getitem__(self, index):\n        # stuff\n        return (img, label)\n\n    def __len__(self):\n        return count # of how many examples(images?) you have\n```\n\nThis is the skeleton that you have to fill to have a custom dataset. A dataset must contain following functions to be used by data loader later on. \n\n* `__init__()` function is where the initial logic happens like reading a csv, assigning transforms, filtering data, etc.\n* `__getitem__()` function returns the data and labels. This function is called from dataloader like this:\n```python\nimg, label = MyCustomDataset.__getitem__(99)  # For 99th item\n```\nHere, ```MyCustomDataset``` returns two things, an image and a label but that does not mean that ```__getitem__()``` is only restricted to return those. Depending on your application you can return many things.\n\nAn important thing to note is that `__getitem__()` returns a specific type for a single data point (like a tensor, numpy array etc.), otherwise, in the data loader you will get an error like:\n\nTypeError: batch must contain tensors, numbers, dicts or lists; found \u003Cclass 'PIL.PngImagePlugin.PngImageFile'>\n\n* `__len__()` returns count of samples you have.\n\n## Using Torchvision Transforms\nIn most of the examples you see `transforms = None` in the `__init__()`, this is used to apply torchvision transforms to your data\u002Fimage. You can find the extensive list of the transforms [here](http:\u002F\u002Fpytorch.org\u002Fdocs\u002F0.2.0\u002Ftorchvision\u002Ftransforms.html) and [here](https:\u002F\u002Fgithub.com\u002Fpytorch\u002Fvision\u002Fblob\u002Fmaster\u002Ftorchvision\u002Ftransforms\u002Ftransforms.py). The most common usage of transforms is like this:\n\n```python\nfrom torch.utils.data.dataset import Dataset\nfrom torchvision import transforms\n\nclass MyCustomDataset(Dataset):\n    def __init__(self, ..., transforms=None):\n        # stuff\n        ...\n        self.transforms = transforms\n        \n    def __getitem__(self, index):\n        # stuff\n        ...\n        data = # Some data read from a file or image\n        if self.transforms is not None:\n            data = self.transforms(data)\n        # If the transform variable is not empty\n        # then it applies the operations in the transforms with the order that it is created.\n        return (img, label)\n\n    def __len__(self):\n        return count # of how many data(images?) you have\n        \nif __name__ == '__main__':\n    # Define transforms (1)\n    transformations = transforms.Compose([transforms.CenterCrop(100), transforms.ToTensor()])\n    # Call the dataset\n    custom_dataset = MyCustomDataset(..., transformations)\n    \n```\n\n## Another Way to Use Torchvision Transforms\n\nPersonally, I don't like having dataset transforms outside the dataset class (see (1) above). So, instead of using transforms like the example above, you can also use it like:\n\n```python\nfrom torch.utils.data.dataset import Dataset\nfrom torchvision import transforms\n\nclass MyCustomDataset(Dataset):\n    def __init__(self, ...):\n        # stuff\n        ...\n        # (2) One way to do it is define transforms individually\n        # When you define the transforms it calls __init__() of the transform\n        self.center_crop = transforms.CenterCrop(100)\n        self.to_tensor = transforms.ToTensor()\n        \n        # (3) Or you can still compose them like \n        self.transformations = \\\n            transforms.Compose([transforms.CenterCrop(100),\n                                transforms.ToTensor()])\n        \n    def __getitem__(self, index):\n        # stuff\n        ...\n        data = # Some data read from a file or image\n        \n        # When you call the transform for the second time it calls __call__() and applies the transform \n        data = self.center_crop(data)  # (2)\n        data = self.to_tensor(data)  # (2)\n        \n        # Or you can call the composed version\n        data = self.transformations(data)  # (3)\n        \n        # Note that you only need one of the implementations, (2) or (3)\n        return (img, label)\n\n    def __len__(self):\n        return count # of how many data(images?) you have\n        \nif __name__ == '__main__':\n    # Call the dataset\n    custom_dataset = MyCustomDataset(...)\n    \n```\n\n\n## Incorporating Pandas\n\nLet's say we want to read some data from a csv with pandas. The first example is of having a csv file like following (without the headers, even though it really doesn't matter), that contains file name, label(class) and an extra operation indicator and depending on this extra operation flag we do some operation on the image.\n\n File Name      | Label           | Extra Operation  |\n| ------------- |:-------------:| :-----:|\n| tr_0.png      | 5 | TRUE |\n| tr_1.png      | 0      |   FALSE |\n| tr_1.png      | 4      |    FALSE |\n\nIf we want to build a custom dataset that reads image locations form this csv file then we can do something like following.\n\n```python\nclass CustomDatasetFromImages(Dataset):\n    def __init__(self, csv_path):\n        \"\"\"\n        Args:\n            csv_path (string): path to csv file\n            img_path (string): path to the folder where images are\n            transform: pytorch transforms for transforms and tensor conversion\n        \"\"\"\n        # Transforms\n        self.to_tensor = transforms.ToTensor()\n        # Read the csv file\n        self.data_info = pd.read_csv(csv_path, header=None)\n        # First column contains the image paths\n        self.image_arr = np.asarray(self.data_info.iloc[:, 0])\n        # Second column is the labels\n        self.label_arr = np.asarray(self.data_info.iloc[:, 1])\n        # Third column is for an operation indicator\n        self.operation_arr = np.asarray(self.data_info.iloc[:, 2])\n        # Calculate len\n        self.data_len = len(self.data_info.index)\n\n    def __getitem__(self, index):\n        # Get image name from the pandas df\n        single_image_name = self.image_arr[index]\n        # Open image\n        img_as_img = Image.open(single_image_name)\n\n        # Check if there is an operation\n        some_operation = self.operation_arr[index]\n        # If there is an operation\n        if some_operation:\n            # Do some operation on image\n            # ...\n            # ...\n            pass\n        # Transform image to tensor\n        img_as_tensor = self.to_tensor(img_as_img)\n\n        # Get label(class) of the image based on the cropped pandas column\n        single_image_label = self.label_arr[index]\n\n        return (img_as_tensor, single_image_label)\n\n    def __len__(self):\n        return self.data_len\n\nif __name__ == \"__main__\":\n    # Call dataset\n    custom_mnist_from_images =  \\\n        CustomDatasetFromImages('..\u002Fdata\u002Fmnist_labels.csv')\n```\n\n## Incorporating Pandas with More Logic\n\nYet another example might be reading an image from CSV where the value of each pixel is listed in a column. (Sometimes MNIST is given this way). This just changes the logic in `__getitem__()`. In the end, you just return images as tensors and their labels. The data is divided into pixels like\n\nLabel     | pixel_1           | pixel_2  | ... |\n| ------------- |:-------------:| :-----:|  :-----:| \n| 1      | 50 | 99 | ... |\n| 0      | 21      |   223 | ... |\n| 9      | 44      |    112 | ... |\n\n\n\n```python\nclass CustomDatasetFromCSV(Dataset):\n    def __init__(self, csv_path, height, width, transforms=None):\n        \"\"\"\n        Args:\n            csv_path (string): path to csv file\n            height (int): image height\n            width (int): image width\n            transform: pytorch transforms for transforms and tensor conversion\n        \"\"\"\n        self.data = pd.read_csv(csv_path)\n        self.labels = np.asarray(self.data.iloc[:, 0])\n        self.height = height\n        self.width = width\n        self.transforms = transform\n\n    def __getitem__(self, index):\n        single_image_label = self.labels[index]\n        # Read each 784 pixels and reshape the 1D array ([784]) to 2D array ([28,28]) \n        img_as_np = np.asarray(self.data.iloc[index][1:]).reshape(28,28).astype('uint8')\n\t# Convert image from numpy array to PIL image, mode 'L' is for grayscale\n        img_as_img = Image.fromarray(img_as_np)\n        img_as_img = img_as_img.convert('L')\n        # Transform image to tensor\n        if self.transforms is not None:\n            img_as_tensor = self.transforms(img_as_img)\n        # Return image and the label\n        return (img_as_tensor, single_image_label)\n\n    def __len__(self):\n        return len(self.data.index)\n        \n\nif __name__ == \"__main__\":\n    transformations = transforms.Compose([transforms.ToTensor()])\n    custom_mnist_from_csv = \\\n        CustomDatasetFromCSV('..\u002Fdata\u002Fmnist_in_csv.csv', 28, 28, transformations)\n        \n```\n\n## A Custom-custom-custom Dataset \nIt has been a long time since I have updated this repository (huh 2 years) and during that time I have completely stopped using torchvision transforms and also csvs and whatnot (unless it is absolutely necessary). One reason I have stopped using torchivion transforms is because I have coded my own transforms but more importantly, I disliked the way transforms are often given as an argument in the dataset class when they are initialized in most of the best-practice examples, when it is not the best way of doing things. More often than not, for the training datasets I have coded over time, I had to use some form of preprocessing operation (flip, mirror, pad, add noise, saturate, crop, ...) randomly and wanted to have the freedom of choosing the degree I apply them, or not. So, my datasets often have a flow like below:\n\n```\n1- Get the location of the image\u002Fdata to read \n2- Read the image\u002Fdata\n3- Convert to numpy\n4- Do some processing on numpy array (randomly)\n5- Do some processing on numpy array (randomly)\n6- Do some processing on numpy array (randomly)\n...\n15- Convert data to tensor\nreturn location of data, name of data, data, and label\n```\nYou can obviously apply transforms just like I listed above too, in the end, it is a matter of taste. \n\nBelow, I'm sharing a barebone custom dataset that I'm using for most of my experiments. \n\n```python\nclass CustomDatasetFromFile(Dataset):\n    def __init__(self, folder_path):\n        \"\"\"\n        A dataset example where the class is embedded in the file names\n        This data example also does not use any torch transforms\n\n        Args:\n            folder_path (string): path to image folder\n        \"\"\"\n        # Get image list\n        self.image_list = glob.glob(folder_path+'*')\n        # Calculate len\n        self.data_len = len(self.image_list)\n\n    def __getitem__(self, index):\n        # Get image name from the pandas df\n        single_image_path = self.image_list[index]\n        # Open image\n        im_as_im = Image.open(single_image_path)\n\n        # Do some operations on image\n        # Convert to numpy, dim = 28x28\n        im_as_np = np.asarray(im_as_im)\u002F255\n        # Add channel dimension, dim = 1x28x28\n        # Note: You do not need to do this if you are reading RGB images\n        # or i there is already channel dimension\n        im_as_np = np.expand_dims(im_as_np, 0)\n        # Some preprocessing operations on numpy array\n        # ...\n        # ...\n        # ...\n\n        # Transform image to tensor, change data type\n        im_as_ten = torch.from_numpy(im_as_np).float()\n\n        # Get label(class) of the image based on the file name\n        class_indicator_location = single_image_path.rfind('_c')\n        label = int(single_image_path[class_indicator_location+2:class_indicator_location+3])\n        return (im_as_ten, label)\n\n    def __len__(self):\n        return self.data_len\n```\n\n## Using Data Loader\nPytorch DataLoaders just call `__getitem__()` and wrap them up to a batch. We can technically not use Data Loaders and call `__getitem__()` one at a time and feed data to the models (even though it is super convenient to use data loader). Continuing from the example above, if we assume there is a custom dataset called *CustomDatasetFromCSV* then we can call the data loader like:\n\n```python\n...\nif __name__ == \"__main__\":\n    # Define transforms\n    transformations = transforms.Compose([transforms.ToTensor()])\n    # Define custom dataset\n    custom_mnist_from_csv = \\\n        CustomDatasetFromCSV('..\u002Fdata\u002Fmnist_in_csv.csv',\n                             28, 28,\n                             transformations)\n    # Define data loader\n    mn_dataset_loader = torch.utils.data.DataLoader(dataset=custom_mnist_from_csv,\n                                                    batch_size=10,\n                                                    shuffle=False)\n    \n    for images, labels in mn_dataset_loader:\n        # Feed the data to the model\n```\n\nThe firsts argument of the dataloader is the dataset, from there  it calls `__getitem__()` of that dataset. *batch_size* determines how many individual data points will be wrapped with a single batch. If we assume a single image tensor is of size: 1x28x28 (D:1, H:28, W:28) then, with this dataloader the returned tensor will be 10x1x28x28 (Batch-Depth-Height-Width).\n\nA note on using **multi GPU**. The way that multi gpu is used with Pytorch data loaders is that, it tries to divide the batches evenly among all GPUs you have. So, if you use batch size that is less than amount of GPUs you have, it won't be able utilize all GPUs. \n\n## Future Updates\n\nI will continue updating this repository whenever I find spare time. Below, are some of the stuff I plan to include. Please let me know if you would like to see some other specific examples.\n\n- A working custom dataset for Imagenet with normalizations etc. *Note:* Huh, I have been wanting to do this for a long time. I will probably do this and maybe have another dataset for CIFAR too.\n- A custom dataset example for encoder-decoder networks like U-Net. *Note:* There is already a U-Net dataset my students coded for their project, have a look [here](https:\u002F\u002Fgithub.com\u002Fugent-korea\u002Fpytorch-unet-segmentation) if you are interested.\n","\u003Cp align=\"center\">\u003Cimg width=\"40%\" src=\"https:\u002F\u002Foss.gittoolsai.com\u002Fimages\u002Futkuozbulak_pytorch-custom-dataset-examples_readme_28a00358044f.png\" \u002F>\u003C\u002Fp>\n\n--------------------------------------------------------------------------------\n\n# PyTorch 自定义数据集示例\n\n*两年后的更新：* 我创建这个仓库来指导刚开始接触 PyTorch 的人（就像当年的我一样）已经过去很久了。然而，随着岁月的流逝和不同项目的开展，我构建数据集的方式也多次改变。我在 [这里](#a-custom-custom-custom-dataset) 添加了一个更基础的数据集，以展示我现在常用的实现方式。我想强调的是，自定义数据集之所以被称为“自定义”，正是因为它可以根据你的需求进行任意设计。因此，在你参与不同的项目后，自然而然会形成自己独特的数据集构建方式。本项目中提供的示例并不是创建自定义数据集的唯一正确方法，而是为了展示 PyTorch 数据集的灵活性和可能性。希望这个仓库能够帮助你更好地理解 PyTorch 数据集。\n\nPyTorch 官方仓库中也有一些自定义数据集的示例，比如 [这个](https:\u002F\u002Fgithub.com\u002Fpytorch\u002Ftutorials\u002Fblob\u002Fmaster\u002Fbeginner_source\u002Fdata_loading_tutorial.py)，但对于初学者来说（像当时的我一样），这些示例仍然有些晦涩难懂。我不得不花一些时间去弄清楚究竟需要做些什么才能构建一个完全自定义的数据集。为了避免大家翻阅大量资料的麻烦，我在这里总结了 PyTorch 数据集的基础知识。具体内容如下：\n\n# 主题\n- [自定义数据集基础](#custom-dataset-fundamentals)\n- [使用 torchvision transforms](#using-torchvision-transforms)\n- [另一种使用 torchvision transforms 的方式](#another-way-to-use-torchvision-transforms)\n- [集成 pandas（读取 CSV 文件）](#incorporating-pandas)\n- [在 `__getitem__()` 中加入更多逻辑的 pandas 集成](#incorporating-pandas-with-more-logic)\n- [将类别信息嵌入文件名](#a-custom-custom-custom-dataset)\n- [是否使用 torchvision transforms？](#a-custom-custom-custom-dataset)\n- [结合自定义数据集使用数据加载器](#using-data-loader)\n- [未来的更新（希望如此）](#future-updates)\n\n## 自定义数据集基础\n首先也是最重要的一点，就是创建一个数据集类。\n\n```python\nfrom torch.utils.data.dataset import Dataset\n\nclass MyCustomDataset(Dataset):\n    def __init__(self, ...):\n        # 初始化操作\n        \n    def __getitem__(self, index):\n        # 处理数据\n        return (img, label)\n\n    def __len__(self):\n        return count # 你拥有的样本数量\n```\n\n这就是你需要填充的基本框架，才能构建一个自定义数据集。数据集必须包含以下方法，以便后续被数据加载器使用：\n\n* `__init__()` 方法用于执行初始化逻辑，例如读取 CSV 文件、设置变换、筛选数据等。\n* `__getitem__()` 方法返回数据及其标签。该方法会在数据加载器中被调用，例如：\n```python\nimg, label = MyCustomDataset.__getitem__(99)  # 获取第 99 个样本\n```\n这里的 `MyCustomDataset` 返回了一张图像和一个标签，但这并不意味着 `__getitem__()` 只能返回这两样内容。根据你的应用场景，你可以返回多种类型的数据。\n\n需要注意的是，`__getitem__()` 必须为单个数据点返回特定类型的对象（如张量、NumPy 数组等），否则在数据加载器中会遇到类似以下的错误：\n\nTypeError: batch must contain tensors, numbers, dicts or lists; found \u003Cclass 'PIL.PngImagePlugin.PngImageFile'>\n\n* `__len__()` 方法返回数据集中样本的总数。\n\n## 使用 torchvision transforms\n在大多数示例中，`__init__()` 方法中的 `transforms` 参数通常被设置为 `None`。这是用来对你的数据或图像应用 torchvision 提供的变换操作。详细的变换列表可以参考 [这里](http:\u002F\u002Fpytorch.org\u002Fdocs\u002F0.2.0\u002Ftorchvision\u002Ftransforms.html) 和 [这里](https:\u002F\u002Fgithub.com\u002Fpytorch\u002Fvision\u002Fblob\u002Fmaster\u002Ftorchvision\u002Ftransforms\u002Ftransforms.py)。最常见的用法如下：\n\n```python\nfrom torch.utils.data.dataset import Dataset\nfrom torchvision import transforms\n\nclass MyCustomDataset(Dataset):\n    def __init__(self, ..., transforms=None):\n        # 初始化操作\n        ...\n        self.transforms = transforms\n        \n    def __getitem__(self, index):\n        # 处理数据\n        ...\n        data = # 从文件或图像中读取的数据\n        if self.transforms is not None:\n            data = self.transforms(data)\n        # 如果 transforms 不为空，则按照其创建顺序依次应用变换操作\n        return (img, label)\n\n    def __len__(self):\n        return count # 你拥有的数据（图像）数量\n        \nif __name__ == '__main__':\n    # 定义变换 (1)\n    transformations = transforms.Compose([transforms.CenterCrop(100), transforms.ToTensor()])\n    # 实例化数据集\n    custom_dataset = MyCustomDataset(..., transformations)\n    \n```\n\n## 另一种使用 torchvision transforms 的方式\n\n就我个人而言，我不太喜欢把数据集的变换逻辑放在数据集类之外（见上面的 (1)）。因此，除了上述示例中的用法外，你还可以这样使用：\n\n```python\nfrom torch.utils.data.dataset import Dataset\nfrom torchvision import transforms\n\nclass MyCustomDataset(Dataset):\n    def __init__(self, ...):\n        # 初始化操作\n        ...\n        # (2) 一种方法是单独定义每个变换\n        # 当你定义变换时，会调用该变换的 __init__() 方法\n        self.center_crop = transforms.CenterCrop(100)\n        self.to_tensor = transforms.ToTensor()\n        \n        # (3) 或者你也可以直接组合它们\n        self.transformations = \\\n            transforms.Compose([transforms.CenterCrop(100),\n                                transforms.ToTensor()])\n        \n    def __getitem__(self, index):\n        # 处理数据\n        ...\n        data = # 从文件或图像中读取的数据\n        \n        # 第二次调用变换时，会触发 __call__() 方法并应用变换\n        data = self.center_crop(data)  # (2)\n        data = self.to_tensor(data)  # (2)\n        \n        # 或者你可以直接使用组合好的变换\n        data = self.transformations(data)  # (3)\n        \n        # 注意，你只需要选择其中一种实现方式即可，(2) 或 (3)\n        return (img, label)\n\n    def __len__(self):\n        return count # 你拥有的数据（图像）数量\n        \nif __name__ == '__main__':\n    # 实例化数据集\n    custom_dataset = MyCustomDataset(...)\n    \n```\n\n## 结合 Pandas\n\n假设我们想用 Pandas 从一个 CSV 文件中读取数据。第一个示例的 CSV 文件如下（虽然没有表头，但这并不影响），它包含文件名、标签（类别）以及一个额外的操作指示符。根据这个操作标志，我们会对图像执行一些操作。\n\n| 文件名      | 标签           | 额外操作  |\n| ------------- |:-------------:| :-----:|\n| tr_0.png      | 5 | TRUE |\n| tr_1.png      | 0      |   FALSE |\n| tr_1.png      | 4      |    FALSE |\n\n如果我们想要构建一个自定义数据集，从这个 CSV 文件中读取图像路径，可以按照以下方式实现：\n\n```python\nclass CustomDatasetFromImages(Dataset):\n    def __init__(self, csv_path):\n        \"\"\"\n        参数:\n            csv_path (string): CSV 文件的路径\n            img_path (string): 图像所在文件夹的路径\n            transform: PyTorch 的变换工具，用于图像变换和转换为张量\n        \"\"\"\n        # 定义转换：将图像转换为张量\n        self.to_tensor = transforms.ToTensor()\n        # 读取 CSV 文件\n        self.data_info = pd.read_csv(csv_path, header=None)\n        # 第一列是图像路径\n        self.image_arr = np.asarray(self.data_info.iloc[:, 0])\n        # 第二列是标签\n        self.label_arr = np.asarray(self.data_info.iloc[:, 1])\n        # 第三列是操作指示符\n        self.operation_arr = np.asarray(self.data_info.iloc[:, 2])\n        # 计算数据长度\n        self.data_len = len(self.data_info.index)\n\n    def __getitem__(self, index):\n        # 从 Pandas DataFrame 中获取图像名称\n        single_image_name = self.image_arr[index]\n        # 打开图像\n        img_as_img = Image.open(single_image_name)\n\n        # 检查是否有操作\n        some_operation = self.operation_arr[index]\n        # 如果有操作\n        if some_operation:\n            # 对图像执行某些操作\n            # ...\n            # ...\n            pass\n        # 将图像转换为张量\n        img_as_tensor = self.to_tensor(img_as_img)\n\n        # 根据截取的 Pandas 列获取图像的标签（类别）\n        single_image_label = self.label_arr[index]\n\n        return (img_as_tensor, single_image_label)\n\n    def __len__(self):\n        return self.data_len\n\nif __name__ == \"__main__\":\n    # 调用数据集\n    custom_mnist_from_images =  \\\n        CustomDatasetFromImages('..\u002Fdata\u002Fmnist_labels.csv')\n```\n\n## 结合 Pandas 并加入更多逻辑\n\n另一个例子可能是从 CSV 文件中读取图像，其中每个像素的值都列在单独的一列中。（有时 MNIST 数据集就是以这种方式提供的）。这只需要修改 `__getitem__()` 方法中的逻辑即可。最终，你只需返回图像的张量及其标签。数据可能被组织成如下形式：\n\n| 标签     | pixel_1           | pixel_2  | ... |\n| ------------- |:-------------:| :-----:|  :-----:| \n| 1      | 50 | 99 | ... |\n| 0      | 21      |   223 | ... |\n| 9      | 44      |    112 | ... |\n\n\n\n```python\nclass CustomDatasetFromCSV(Dataset):\n    def __init__(self, csv_path, height, width, transforms=None):\n        \"\"\"\n        参数:\n            csv_path (string): CSV 文件的路径\n            height (int): 图像的高度\n            width (int): 图像的宽度\n            transform: PyTorch 的变换工具，用于图像变换和转换为张量\n        \"\"\"\n        self.data = pd.read_csv(csv_path)\n        self.labels = np.asarray(self.data.iloc[:, 0])\n        self.height = height\n        self.width = width\n        self.transforms = transform\n\n    def __getitem__(self, index):\n        single_image_label = self.labels[index]\n        # 读取每行的 784 个像素值，并将一维数组 ([784]) 重塑为二维数组 ([28,28])\n        img_as_np = np.asarray(self.data.iloc[index][1:]).reshape(28,28).astype('uint8')\n\t# 将 NumPy 数组转换为 PIL 图像，模式 'L' 表示灰度图\n        img_as_img = Image.fromarray(img_as_np)\n        img_as_img = img_as_img.convert('L')\n        # 将图像转换为张量\n        if self.transforms is not None:\n            img_as_tensor = self.transforms(img_as_img)\n        # 返回图像和标签\n        return (img_as_tensor, single_image_label)\n\n    def __len__(self):\n        return len(self.data.index)\n        \n\nif __name__ == \"__main__\":\n    transformations = transforms.Compose([transforms.ToTensor()])\n    custom_mnist_from_csv = \\\n        CustomDatasetFromCSV('..\u002Fdata\u002Fmnist_in_csv.csv', 28, 28, transformations)\n        \n```\n\n## 一个自定义的自定义的自定义数据集\n我已经很久没有更新这个仓库了（居然有两年了），在这段时间里，我几乎完全不再使用 torchvision 的变换以及 CSV 等文件格式（除非绝对必要）。我之所以不再使用 torchvision 的变换，一方面是因为我自己编写了相应的变换函数；但更重要的原因是，我不太喜欢在大多数最佳实践示例中，将变换作为参数传入数据集类并在初始化时设置的做法，因为这并不是一种最佳的方式。在我的训练数据集中，我经常需要对图像或数据进行一些随机的预处理操作（如翻转、镜像、填充、添加噪声、饱和度调整、裁剪等），并且希望能够自由地选择这些操作的程度，甚至决定是否应用它们。因此，我的数据集通常遵循如下流程：\n\n```\n1- 获取待读取图像\u002F数据的位置\n2- 读取图像\u002F数据\n3- 转换为 NumPy 数组\n4- 对 NumPy 数组进行一些处理（随机）\n5- 对 NumPy 数组进行一些处理（随机）\n6- 对 NumPy 数组进行一些处理（随机）\n...\n15- 将数据转换为张量\n返回数据位置、数据名称、数据和标签\n```\n当然，你也可以像我上面列出的那样直接应用变换，归根结底这只是个人偏好问题。\n\n下面我分享一个我在大多数实验中使用的极简自定义数据集。\n\n```python\nclass CustomDatasetFromFile(Dataset):\n    def __init__(self, folder_path):\n        \"\"\"\n        一个数据集示例，其中类别信息嵌入在文件名中。\n        此数据集也不使用任何 PyTorch 变换。\n\n        Args:\n            folder_path (string): 图像文件夹路径\n        \"\"\"\n        # 获取图像列表\n        self.image_list = glob.glob(folder_path+'*')\n        # 计算数据长度\n        self.data_len = len(self.image_list)\n\n    def __getitem__(self, index):\n        # 从文件路径中获取图像名称\n        single_image_path = self.image_list[index]\n        # 打开图像\n        im_as_im = Image.open(single_image_path)\n\n        # 对图像进行一些操作\n        # 转换为 NumPy 数组，尺寸为 28x28\n        im_as_np = np.asarray(im_as_im)\u002F255\n        # 增加通道维度，尺寸为 1x28x28\n        # 注意：如果你读取的是 RGB 图像，或者数据本身已经有通道维度，则无需执行此步骤。\n        im_as_np = np.expand_dims(im_as_np, 0)\n        # 对 NumPy 数组进行一些预处理操作\n        # ...\n        # ...\n        # ...\n\n        # 将图像转换为张量，并更改数据类型\n        im_as_ten = torch.from_numpy(im_as_np).float()\n\n        # 根据文件名获取图像的标签（类别）\n        class_indicator_location = single_image_path.rfind('_c')\n        label = int(single_image_path[class_indicator_location+2:class_indicator_location+3])\n        return (im_as_ten, label)\n\n    def __len__(self):\n        return self.data_len\n```\n\n## 使用数据加载器\nPyTorch 的数据加载器会调用 `__getitem__()` 方法，并将数据打包成批次。理论上，我们也可以不使用数据加载器，而是逐个调用 `__getitem__()` 方法并将数据输入模型（尽管使用数据加载器确实非常方便）。继续以上面的例子为例，假设我们有一个名为 *CustomDatasetFromCSV* 的自定义数据集，那么我们可以这样调用数据加载器：\n\n```python\n...\nif __name__ == \"__main__\":\n    # 定义变换\n    transformations = transforms.Compose([transforms.ToTensor()])\n    # 定义自定义数据集\n    custom_mnist_from_csv = \\\n        CustomDatasetFromCSV('..\u002Fdata\u002Fmnist_in_csv.csv',\n                             28, 28,\n                             transformations)\n    # 定义数据加载器\n    mn_dataset_loader = torch.utils.data.DataLoader(dataset=custom_mnist_from_csv,\n                                                    batch_size=10,\n                                                    shuffle=False)\n    \n    for images, labels in mn_dataset_loader:\n        # 将数据输入模型\n```\n\n数据加载器的第一个参数是数据集对象，它会调用该数据集的 `__getitem__()` 方法。`batch_size` 参数决定了每个批次包含多少个独立的数据点。如果假设单个图像张量的尺寸为 1x28x28（D:1, H:28, W:28），那么通过这个数据加载器返回的张量将是 10x1x28x28（批次-深度-高度-宽度）。\n\n关于 **多 GPU** 的使用说明。在 PyTorch 中，多 GPU 数据加载器的工作方式是尝试将批次均匀地分配到所有可用的 GPU 上。因此，如果使用的 `batch_size` 小于 GPU 的数量，就无法充分利用所有的 GPU。\n\n## 未来更新\n我会在有空的时候继续更新这个仓库。以下是我计划加入的一些内容。如果你希望看到其他特定的示例，请随时告诉我。\n\n- 一个可运行的 Imagenet 自定义数据集，包含归一化等操作。*注：* 嗯，我一直想做这件事很久了。我可能会同时再做一个 CIFAR 的自定义数据集。\n- 一个适用于编码器-解码器网络（如 U-Net）的自定义数据集示例。*注：* 我的学生们已经为他们的项目编写了一个 U-Net 数据集，感兴趣的话可以查看 [这里](https:\u002F\u002Fgithub.com\u002Fugent-korea\u002Fpytorch-unet-segmentation)。","# PyTorch 自定义数据集快速上手指南\n\n本指南基于 `pytorch-custom-dataset-examples` 项目，旨在帮助开发者快速掌握如何在 PyTorch 中构建灵活的自定义 Dataset 类。\n\n## 环境准备\n\n在开始之前，请确保您的开发环境满足以下要求：\n\n*   **操作系统**: Linux, macOS 或 Windows\n*   **Python**: 3.6 及以上版本\n*   **核心依赖**:\n    *   `torch`: PyTorch 深度学习框架\n    *   `torchvision`: 包含常用图像变换和数据集工具\n    *   `pandas`: 用于处理 CSV 数据文件（可选，视需求而定）\n    *   `Pillow` (PIL): 用于图像读取和处理\n\n**安装命令：**\n\n推荐使用国内镜像源（如清华源）以加速安装：\n\n```bash\npip install torch torchvision pandas pillow -i https:\u002F\u002Fpypi.tuna.tsinghua.edu.cn\u002Fsimple\n```\n\n## 基本使用\n\n构建自定义数据集的核心是继承 `torch.utils.data.Dataset` 类，并实现三个关键方法：`__init__`、`__getitem__` 和 `__len__`。\n\n### 1. 最简骨架代码\n\n这是构建任何自定义数据集的基础模板：\n\n```python\nfrom torch.utils.data.dataset import Dataset\n\nclass MyCustomDataset(Dataset):\n    def __init__(self, ...):\n        # 初始化逻辑：读取文件列表、设置变换等\n        pass\n        \n    def __getitem__(self, index):\n        # 根据索引加载单个样本（如图片）和标签\n        # 必须返回 Tensor、numpy array、dict 或 list 等类型\n        img = ... \n        label = ...\n        return (img, label)\n\n    def __len__(self):\n        # 返回数据集样本总数\n        return count \n```\n\n### 2. 集成 Torchvision 变换\n\n通常需要在数据加载时应用预处理（如裁剪、归一化）。推荐在 `__init__` 中接收变换对象，并在 `__getitem__` 中应用。\n\n```python\nfrom torch.utils.data.dataset import Dataset\nfrom torchvision import transforms\n\nclass MyCustomDataset(Dataset):\n    def __init__(self, ..., transforms=None):\n        self.transforms = transforms\n        \n    def __getitem__(self, index):\n        # 假设从文件加载了原始图像数据\n        data = load_image_from_file(index) \n        \n        # 应用变换\n        if self.transforms is not None:\n            data = self.transforms(data)\n            \n        label = get_label(index)\n        return (data, label)\n\n    def __len__(self):\n        return total_count\n        \nif __name__ == '__main__':\n    # 定义变换组合\n    transformations = transforms.Compose([\n        transforms.CenterCrop(100), \n        transforms.ToTensor()\n    ])\n    \n    # 实例化数据集\n    custom_dataset = MyCustomDataset(..., transformations)\n```\n\n### 3. 结合 Pandas 读取 CSV\n\n当数据路径和标签存储在 CSV 文件中时，可使用 `pandas` 进行解析。\n\n```python\nimport pandas as pd\nimport numpy as np\nfrom PIL import Image\nfrom torch.utils.data.dataset import Dataset\nfrom torchvision import transforms\n\nclass CustomDatasetFromImages(Dataset):\n    def __init__(self, csv_path, transforms=None):\n        self.transforms = transforms\n        # 读取 CSV (假设无表头：第 1 列为路径，第 2 列为标签)\n        self.data_info = pd.read_csv(csv_path, header=None)\n        self.image_arr = np.asarray(self.data_info.iloc[:, 0])\n        self.label_arr = np.asarray(self.data_info.iloc[:, 1])\n        self.data_len = len(self.data_info.index)\n\n    def __getitem__(self, index):\n        # 获取图片路径并打开\n        single_image_name = self.image_arr[index]\n        img_as_img = Image.open(single_image_name)\n\n        # 应用变换\n        if self.transforms is not None:\n            img_as_tensor = self.transforms(img_as_img)\n        else:\n            # 若无外部变换，至少需转为 Tensor\n            img_as_tensor = transforms.ToTensor()(img_as_img)\n\n        # 获取标签\n        single_image_label = self.label_arr[index]\n\n        return (img_as_tensor, single_image_label)\n\n    def __len__(self):\n        return self.data_len\n```\n\n### 4. 配合 DataLoader 使用\n\n定义好 Dataset 后，即可通过 `DataLoader` 进行批量加载和多进程读取：\n\n```python\nfrom torch.utils.data import DataLoader\n\n# 实例化数据集\ndataset = MyCustomDataset(..., transforms=transformations)\n\n# 创建 DataLoader\nloader = DataLoader(dataset, batch_size=32, shuffle=True, num_workers=4)\n\n# 遍历数据\nfor images, labels in loader:\n    # 训练逻辑\n    pass\n```\n\n> **注意**：`__getitem__` 返回的单个样本必须是可被自动堆叠的类型（如 Tensor、numpy 数组、数字、字典或列表），否则 `DataLoader` 会报错。","某计算机视觉工程师正在构建一个医疗影像分类模型，需要处理分散在多个 CSV 文件中的患者 X 光片及其对应的复杂诊断标签。\n\n### 没有 pytorch-custom-dataset-examples 时\n- 开发者需反复查阅晦涩的官方文档，难以理解如何正确继承 `Dataset` 基类并实现 `__getitem__` 方法。\n- 面对非标准存储结构（如图片路径与标签分离），不知如何在初始化阶段高效整合 Pandas 数据清洗逻辑。\n- 容易因返回数据类型错误（如直接返回 PIL 图像而非 Tensor）导致 DataLoader 报错，调试耗时极长。\n- 缺乏灵活的数据增强示例，难以将 torchvision 变换自然地嵌入自定义数据加载流程中。\n- 只能照搬固定模板，无法根据实际项目需求灵活调整数据集类的内部逻辑。\n\n### 使用 pytorch-custom-dataset-examples 后\n- 参考清晰的骨架代码，快速掌握 `__init__`、`__getitem__` 和 `__len__` 的核心写法，立即上手编写自定义类。\n- 直接复用集成 Pandas 读取 CSV 的成熟案例，轻松实现路径解析、标签映射及异常数据过滤的一站式处理。\n- 通过示例明确得知 `__getitem__` 必须返回张量或字典等特定类型，彻底避免批次加载时的类型错误。\n- 学习多种 torchvision 变换的接入模式，灵活选择在初始化时应用还是在线动态增强，提升模型泛化能力。\n- 领悟“自定义”的真谛，不再受限于固定范式，而是根据医疗数据的特殊性设计出最契合的数据管道。\n\npytorch-custom-dataset-examples 通过提供从基础到进阶的实战范例，将原本陡峭的自定义数据加载学习曲线转化为平滑的开发路径，显著提升了原型验证效率。","https:\u002F\u002Foss.gittoolsai.com\u002Fimages\u002Futkuozbulak_pytorch-custom-dataset-examples_f4b814ce.png","utkuozbulak","Utku Ozbulak","https:\u002F\u002Foss.gittoolsai.com\u002Favatars\u002Futkuozbulak_f5de4f68.jpg","Research Professor at Ghent University Global Campus.\r\n\r\nAdjunct Professor at George Mason Korea.\r\n\r\nWorking on Medical and Biomedical AI.","Ghent University","Incheon, South Korea","utku.ozbulak@gmail.com",null,"utkuozbulak.github.io","https:\u002F\u002Fgithub.com\u002Futkuozbulak",[84],{"name":85,"color":86,"percentage":87},"Python","#3572A5",100,873,114,"2026-04-02T23:23:17","MIT",1,"","未说明",{"notes":96,"python":94,"dependencies":97},"这是一个教学示例仓库，用于展示如何编写 PyTorch 自定义数据集类。文中未明确指定具体的操作系统、硬件配置或严格的版本要求。代码依赖标准的 PyTorch 生态（torch, torchvision）以及数据处理库（pandas, numpy, Pillow）。作者提到随着时间推移，其个人实践已不再强制依赖 torchvision transforms 或 CSV 文件，而是倾向于在数据集类内部实现更灵活的预处理逻辑。",[98,99,100,101,102],"torch","torchvision","pandas","numpy","Pillow",[14,16],"2026-03-27T02:49:30.150509","2026-04-19T15:26:26.396434",[107,112,117,122,127,132,137],{"id":108,"question_zh":109,"answer_zh":110,"source_url":111},42734,"如何修改自定义数据集以支持图像分割任务（即标签也是图像而非 CSV 中的类别）？","你只需要在 `__getitem__` 方法中返回两个张量即可。将原本返回 `(img_as_tensor, single_image_label)` 的代码修改为返回 `(img_as_tensor, target_as_tensor)`，其中 target_as_tensor 是作为地面真值（ground truth）的图像张量。对于 3D 图像分割，原理相同，只是返回的图像维度变为 (Slice x Height x Width)。","https:\u002F\u002Fgithub.com\u002Futkuozbulak\u002Fpytorch-custom-dataset-examples\u002Fissues\u002F4",{"id":113,"question_zh":114,"answer_zh":115,"source_url":116},42735,"如何优化从 CSV 加载 MNIST 数据时的性能，避免使用缓慢的 for 循环重塑图像？","当前的示例代码使用 for 循环逐个像素填充 numpy 数组，这会导致显著的性能下降（例如训练时间从 1 分钟增加到 30 分钟）。最佳实践是使用 `np.reshape` 进行向量化操作来替代循环，从而大幅提升数据加载速度。该问题已在后续更新中通过 PR #3 修复。","https:\u002F\u002Fgithub.com\u002Futkuozbulak\u002Fpytorch-custom-dataset-examples\u002Fissues\u002F2",{"id":118,"question_zh":119,"answer_zh":120,"source_url":121},42736,"PyTorch 和 TensorFlow 在输入数据的形状（Shape）要求上有什么区别？","在 PyTorch 中，输入数据的标准形状是 (Batch) x C x H x W（通道在前），而在 TensorFlow 中通常是 H x W x C（通道在后）。当使用 `torch.from_numpy` 转换数据时需注意这一点；如果使用 `torchvision.transforms.ToTensor`，它会自动处理这种转换并将 PIL Image 或 (H x W x C) 的 numpy 数组转换为 (C x H x W) 的 Tensor。","https:\u002F\u002Fgithub.com\u002Futkuozbulak\u002Fpytorch-custom-dataset-examples\u002Fissues\u002F8",{"id":123,"question_zh":124,"answer_zh":125,"source_url":126},42737,"在哪里可以找到示例中使用的 mnist_labels.csv 文件或生成代码？","维护者已上传包含完整训练\u002F测试数据集以及对应的 train\u002Ftest.csv 标签文件到 Dropbox。你可以直接下载该压缩包使用，链接为：https:\u002F\u002Fwww.dropbox.com\u002Fs\u002Fvc14b2cl9dz3kab\u002Fmnist_train_test.tar?dl=0","https:\u002F\u002Fgithub.com\u002Futkuozbulak\u002Fpytorch-custom-dataset-examples\u002Fissues\u002F5",{"id":128,"question_zh":129,"answer_zh":130,"source_url":131},42738,"如何在无监督学习中导入没有标签的数据集？","对于无监督学习或单类数据集，你可以在自定义数据集的 `__getitem__` 方法中只返回数据本身，或者返回数据和一个固定的占位符标签（例如全零或固定整数），具体取决于你的模型实现是否需要接收标签参数。","https:\u002F\u002Fgithub.com\u002Futkuozbulak\u002Fpytorch-custom-dataset-examples\u002Fissues\u002F9",{"id":133,"question_zh":134,"answer_zh":135,"source_url":136},42739,"自定义数据集的 `__getitem__` 方法可以返回超过两个变量吗？例如用于自编码器或需要返回文件名的情况。","是的，返回变量的数量不受限制，你可以返回任意数量的对象。除了图像和标签外，常见的做法是同时返回图像名称、图像路径或其他元数据。这对于构建用于去噪自编码器（输入图像和验证图像）或需要追踪文件来源的任务非常有用。","https:\u002F\u002Fgithub.com\u002Futkuozbulak\u002Fpytorch-custom-dataset-examples\u002Fissues\u002F1",{"id":138,"question_zh":139,"answer_zh":140,"source_url":141},42740,"为什么示例中没有包含更复杂的用例（如基于标签检索多张图片或文本增强）？","维护者故意保持示例简单，旨在帮助刚接触 PyTorch 的用户理解数据集的基本工作原理而不感到不知所措。一旦掌握了基础，实现多图像返回、文本数据随机增强等复杂案例并不困难，用户可以在此基础上自行扩展。","https:\u002F\u002Fgithub.com\u002Futkuozbulak\u002Fpytorch-custom-dataset-examples\u002Fissues\u002F6",[]]