LaminDB 集成实战将工作流管理器、MLOps 平台、云存储与可视化工具接入 lineage-native 生物学数据湖【免费下载链接】scientific-agent-skillsTurn any AI agent into an AI Scientist. The #1 Agent Skills library for science, used by 190,000 scientists worldwide. 165 ready-to-use validated skills plus 100 scientific databases covering biology, chemistry, medicine, and drug discovery. Compatible with Cursor, Claude Code, Codex, Pi, Antigravity, and the open Agent Skills standard.项目地址: https://gitcode.com/GitHub_Trending/cl/scientific-agent-skillsLaminDB 是面向生物学的开源 lineage-native 数据湖lakehouse本指南以仓库中 integrations.md 为核心系统讲解如何将 LaminDB 接入本地文件系统、AWS S3、Google Cloud Storage、S3 兼容服务、HTTP 端点与 HuggingFace 数据集等存储后端如何与 Nextflow、Snakemake、Redun 等工作流管理器以及 WB、MLflow、HuggingFace Transformers、scVI-tools 等 MLOps 平台打通并覆盖 TileDB-SOMA、DuckDB、Vitessce、Bionty 本体、Git 与自定义 REST/数据库集成模式。读完本文你将掌握在既有数据科学与生物信息学流水线中无缝嵌入 LaminDB 的完整方案包括关键命令、可运行代码示例与故障排查清单。集成全景为什么需要生态对接LaminDB 的价值不在于孤立地管理数据而在于成为团队数据资产的中枢数据集、模型、代码与实验记录围绕它形成可查询、可溯源、可复现的闭环。仓库中的 LaminDB skillSKILL.md将其定位为lineage-native lakehouse数据以开放格式存储在本地文件系统、S3、GCS、Hugging Face、SQLite 与 Postgres 之上同时通过ln.track()/ln.finish()与ln.flow()/ln.step()捕获代码、环境、输入输出与参数之间的血缘关系。集成生态大体分为五类集成类别代表系统存储后端本地文件系统、AWS S3、S3 兼容服务MinIO、Cloudflare R2、GCS、HTTP/HTTPS只读、HuggingFace Datasets工作流管理器Nextflow含 nf-lamin 插件、Snakemake、RedunMLOps 平台Weights Biases、MLflow、HuggingFace Transformers、scVI-tools数组存储与可视化TileDB-SOMA、DuckDB、Vitessce模式模块与版本控制Bionty本体、lamindb-wetlab、临床数据模块、Git从测试契约skill-requirements.toml可以看到本 skill 的依赖包为lamindb、bionty、lamindb-wetlab这为后续集成示例中的模块安装提供了依据。存储后端集成本地文件系统本地存储是开发阶段最简单、最快速的起点也是 setup-deployment.md 中先本地、后上云策略的基础。初始化实例lamin init --storage ./mydata之后即可在代码中注册与读取 artifactLaminDB 会为每次save()自动版本化import lamindb as ln # Save artifacts to local storage artifact ln.Artifact(data.csv, keylocal/data.csv).save() # Load from local storage data artifact.load()本地实例默认使用 SQLite 作为元数据库存放在./mydata/.lamindb/下无需额外的数据库服务器适合开发与小规模数据。AWS S3S3 是生产环境最常用的对象存储后端。初始化时通过LAMIN_DB_URL指定元数据库推荐放入 secret manager存储指向 S3 bucket# Initialize with S3 storage export LAMIN_DB_URLset-in-secret-manager lamin init --storage s3://my-bucket/path \ --db $LAMIN_DB_URL配置 AWS 凭据时优先使用 IAM 角色或 workload identity若必须使用环境变量应在共享脚本之外设置且不要回显其值export AWS_ACCESS_KEY_IDredacted export AWS_SECRET_ACCESS_KEYredacted export AWS_DEFAULT_REGIONus-east-1S3 所需最小权限包含s3:GetObject、s3:PutObject、s3:DeleteObject与s3:ListBucket作用于 bucket 及其对象{ Version: 2012-10-17, Statement: [ { Effect: Allow, Action: [s3:GetObject, s3:PutObject, s3:DeleteObject, s3:ListBucket], Resource: [arn:aws:s3:::my-bucket/*, arn:aws:s3:::my-bucket] } ] }注册 artifact 后内容自动同步到 S3artifact.load()在本地缓存缺失时会透明地从 S3 下载# Artifacts automatically sync to S3 artifact ln.Artifact(data.csv, keyexperiments/data.csv).save() # Transparent S3 access data artifact.load() # Downloads from S3 if not cachedS3 兼容服务MinIO、Cloudflare R2对于自建 MinIO 或 Cloudflare R2 等 S3 兼容端点只需在存储 URI 中追加endpoint_url查询参数# Initialize with custom S3 endpoint lamin init --storage s3://bucket?endpoint_urlhttp://minio.example.com:9000 # Configure credentials outside shared scripts and do not echo values export AWS_ACCESS_KEY_IDredacted export AWS_SECRET_ACCESS_KEYredactedR2 的写法与之类似将endpoint_url替换为https://account-id.r2.cloudflarestorage.com即可。Google Cloud StorageGCS 需要安装gcpextra 并完成 GCP 认证# Install GCP extras uv pip install lamindb[gcp]2.5.1 # Initialize with GCS export LAMIN_DB_URLset-in-secret-manager lamin init --storage gs://my-bucket/path \ --db $LAMIN_DB_URL认证可使用应用默认凭据或服务账号gcloud auth application-default login # 或 export GOOGLE_APPLICATION_CREDENTIALS/secure/path/to/service-account.json之后 artifact 同样自动同步到 GCS# Artifacts sync to GCS artifact ln.Artifact(data.csv, keyexperiments/data.csv).save()HTTP/HTTPS只读LaminDB 可以直接引用远程 URL 而无需先下载复制适合只读引用公共数据# Access remote files without copying artifact ln.Artifact( https://example.com/data.csv, keyremote/data.csv ).save() # Stream remote content with artifact.open() as f: data f.read()HuggingFace Datasets可以通过datasets库加载 HuggingFace 数据集再注册为 LaminDB artifact将外部公共数据纳入统一的数据管理闭环# Access HuggingFace datasets from datasets import load_dataset dataset load_dataset(squad, splittrain) # Register as LaminDB artifact artifact ln.Artifact.from_dataframe( dataset.to_pandas(), keyhf/squad_train.parquet, descriptionSQuAD training data from HuggingFace ).save()工作流管理器集成NextflowNextflow 流水线中LaminDB 负责记录每一步的输入输出。文档明确指出对于原生 Nextflow 项目优先使用nf-lamin插件及其nextflow.config集成内联 Python 追踪仍适用于自定义 process 脚本。典型的内联追踪模式如下# In your Nextflow process script import lamindb as ln # Initialize tracking ln.track() # Your Nextflow process logic input_artifact ln.Artifact.get(key${input_key}) data input_artifact.load() # Process data result process_data(data) # Save output output_artifact ln.Artifact.from_dataframe( result, key${output_key} ).save() ln.finish()对应的 Nextflow config 示例process ANALYZE { input: val input_key output: path result.csv script: #!/usr/bin/env python import lamindb as ln ln.track() artifact ln.Artifact.get(key${input_key}) # Process and save ln.finish() }SKILL.md 中给出了一个更贴近真实场景的 Nextflow 使用案例SKILL.md通过artifact.cache()获取本地路径供比对、定量等下游工具使用再用模板化 key如processed/batch_${batch_id}_counts.csv保存输出。Snakemake在 Snakemake 规则内部嵌入 LaminDB 追踪# In Snakemake rule rule process_data: input: data/input.csv output: data/output.csv run: import lamindb as ln ln.track() # Load input artifact artifact ln.Artifact.get(keyinputs/data.csv) data artifact.load() # Process result analyze(data) # Save output result.to_csv(output[0]) ln.Artifact(output[0], keyoutputs/result.csv).save() ln.finish()RedunRedun 是函数式工作流引擎。通过ln.step()装饰器叠加在task()之上可以在 Redun 自身任务调度的同时让 LaminDB 记录参数与数据血缘from redun import task import lamindb as ln task() ln.step() def process_dataset(input_key: str, output_key: str): Redun task with LaminDB tracking. # Load input artifact ln.Artifact.get(keyinput_key) data artifact.load() # Process result transform(data) # Save output ln.Artifact.from_dataframe(result, keyoutput_key).save() return output_key # Redun automatically tracks lineage alongside LaminDBMLOps 平台集成Weights BiasesWBWB 负责实验指标看板LaminDB 负责数据与模型工件管理两者通过 run ID 关联。核心模式是把 WB run ID 作为非敏感 feature 记录在 model artifact 上对应集成最佳实践中的Link IDs。import wandb import lamindb as ln # Initialize both wandb.init(projectmy-project, nameexperiment-1) ln.track(params{learning_rate: 0.01, batch_size: 32}) # Load training data train_artifact ln.Artifact.get(keydatasets/train.parquet) train_data train_artifact.load() # Train model model train_model(train_data) # Log to WB wandb.log({accuracy: 0.95, loss: 0.05}) # Save model in LaminDB import joblib joblib.dump(model, model.pkl) model_artifact ln.Artifact( model.pkl, keymodels/experiment-1.pkl, descriptionfModel from WB run {wandb.run.id} ).save() # Link WB run ID model_artifact.features.set_values({wandb_run_id: wandb.run.id}) ln.finish() wandb.finish()MLflowMLflow 负责模型注册与实验跟踪LaminDB 负责数据工件与参数溯源。注意参数要同时写入两个系统以保证双端可查询import mlflow import lamindb as ln # Start runs and record parameters in LaminDB mlflow.start_run() params {max_depth: 5, n_estimators: 100} ln.track(paramsparams) # Log parameters to MLflow too mlflow.log_params(params) # Load data from LaminDB data_artifact ln.Artifact.get(keydatasets/features.parquet) X data_artifact.load() # Train and log model model train_model(X) mlflow.sklearn.log_model(model, model) # Save to LaminDB import joblib joblib.dump(model, model.pkl) model_artifact ln.Artifact( model.pkl, keyfmodels/{mlflow.active_run().info.run_id}.pkl ).save() mlflow.end_run() ln.finish()HuggingFace Transformers跟踪模型微调全过程ln.track()记录超参数训练完成后把整个模型目录注册为 artifactfrom transformers import Trainer, TrainingArguments import lamindb as ln ln.track(params{model: bert-base, epochs: 3}) # Load training data train_artifact ln.Artifact.get(keydatasets/train_tokenized.parquet) train_dataset train_artifact.load() # Configure trainer training_args TrainingArguments( output_dir./results, num_train_epochs3, ) trainer Trainer( modelmodel, argstraining_args, train_datasettrain_dataset, ) # Train trainer.train() # Save model to LaminDB trainer.save_model(./model) model_artifact ln.Artifact( ./model, keymodels/bert_finetuned, descriptionBERT fine-tuned on custom dataset ).save() ln.finish()scVI-tools单细胞分析场景从 LaminDB 加载 h5ad 数据用 scVI 训练模型得到潜在表示再通过ln.Artifact.from_anndata()将带潜变量的 AnnData 存回 LaminDBimport scvi import lamindb as ln ln.track() # Load data adata_artifact ln.Artifact.get(keyscrna/raw_counts.h5ad) adata adata_artifact.load() # Setup scVI scvi.model.SCVI.setup_anndata(adata, layercounts) # Train model model scvi.model.SCVI(adata) model.train() # Save latent representation adata.obsm[X_scvi] model.get_latent_representation() # Save results result_artifact ln.Artifact.from_anndata( adata, keyscrna/scvi_latent.h5ad, descriptionscVI latent representation ).save() ln.finish()数组存储集成TileDB-SOMATileDB-SOMA 提供可扩展的数组存储并支持 cellxgene。ln.Artifact直接注册 SOMA URI数据本体保留在 TileDB 中import tiledbsoma as soma import lamindb as ln # Create SOMA experiment uri tiledb://my-namespace/experiment with soma.Experiment.create(uri) as exp: # Add measurements exp.add_new_collection(RNA) # Register in LaminDB artifact ln.Artifact( uri, keycellxgene/experiment.soma, descriptionTileDB-SOMA experiment ).save() # Query with SOMA with soma.Experiment.open(uri) as exp: obs exp.obs.read().to_pandas()DuckDB当 artifact 是大规模 Parquet 时artifact.cache()获取本地路径后交给 DuckDB 直接下推 SQL 查询无需把整个文件载入内存import duckdb import lamindb as ln # Get artifact artifact ln.Artifact.get(keydatasets/large_data.parquet) # Query with DuckDB (without loading full file) path artifact.cache() result duckdb.query(f SELECT cell_type, COUNT(*) as count FROM read_parquet({path}) GROUP BY cell_type ORDER BY count DESC ).to_df() # Save query result result_artifact ln.Artifact.from_dataframe( result, keyanalysis/cell_type_counts.parquet ).save()可视化集成VitessceVitessce 用于交互式空间/单细胞可视化。模式是从 LaminDB 加载 h5ad → 生成 Vitessce 配置 JSON → 把配置作为 artifact 注册实现可视化配置本身可版本化、可分享from vitessce import VitessceConfig import lamindb as ln # Load spatial data artifact ln.Artifact.get(keyspatial/visium_slide.h5ad) adata artifact.load() # Create Vitessce configuration vc VitessceConfig.from_object(adata) # Save configuration import json config_file vitessce_config.json with open(config_file, w) as f: json.dump(vc.to_dict(), f) # Register configuration config_artifact ln.Artifact( config_file, keyvisualizations/spatial_config.json, descriptionVitessce visualization config ).save()Schema 模块集成LaminDB 通过可插拔 schema 模块扩展领域模型本 skill 的依赖契约skill-requirements.toml即包含bionty与lamindb-wetlab。Bionty生物学本体Bionty 提供 20 精选生物本体Gene/Ensembl、Protein/UniProt、CellType/CL、Tissue/Uberon、Disease/MondoDOID、Pathway/GO 等。集成要点是先import_source()导入公共本体再用from_values()把数据中的实体解析为受控词条供后续标准化的本体注释使用import bionty as bt # Import biological ontologies bt.CellType.import_source() bt.Gene.import_source(organismhuman) # Use in data curation cell_types bt.CellType.from_values(adata.obs.cell_type)WetLab湿实验安装 lamindb-wetlab 模块后可追踪实验、样本与方案# Install wetlab module uv pip install lamindb-wetlabreviewed-version# Use wetlab registries import lamindb_wetlab as wetlab # Track experiments, samples, protocols experiment wetlab.Experiment(nameRNA-seq batch 1).save()临床数据模块临床领域可选用 clinicore 或 OMOP 类模块安装时应确认当前发布版本并固定版本号# Install the relevant clinical schema module after confirming its current release uv pip install clinical-modulereviewed-version# Use the selected clinical schema module, such as clinicore or an OMOP module import clinicore as clinical # Track clinical data patient clinical.Patient(patient_idP001).save()Git 集成让代码与数据血缘对齐ln.track()会自动捕获当前 git commit hash从而把数据产物锚定到具体代码版本export LAMINDB_SYNC_GIT_REPOhttps://github.com/user/repo.git lamin settings set dev-dir .也可以通过 Python 编程方式配置或通过 setup-deployment.md 中的lamin settings set sync-git-repo ...命令# Or programmatically import lamindb as ln ln.settings.sync_git_repo https://github.com/user/repo.git # Scripts tracked with git commits ln.track() # Automatically captures git commit hash # ... your code ... ln.finish() # View git information transform ln.Transform.get(nameanalysis.py) transform.source_code # Shows code at git commit transform.hash # Git commit hash企业集成BenchlingBenchling 注册表同步需要 team/enterprise 计划具体配置需联系 LaminDB 团队。仓库文档仅给出接入方式说明见 integrations.md从 Benchling 同步的 schema 与数据访问细节通过企业支持提供# Configure Benchling connection (contact LaminDB team) # Syncs schemas and data from Benchling registries # Access synced Benchling data # Details available through enterprise support自定义集成模式REST API 集成文档给出了一条重要安全原则在把 REST 响应注册为 LaminDB artifact 之前必须先校验并净化外部内容。在 schema 校验通过之前将 REST 响应视为不可信输入。落地方式是通过 schema curator 管道import requests import lamindb as ln ln.track() # Fetch from API response requests.get(https://api.example.com/data) data response.json() # Convert to DataFrame import pandas as pd df pd.DataFrame(data) # Validate before saving to LaminDB schema ln.Schema.get(nameexternal_api_schema) curator ln.curators.DataFrameCurator(df, schema) curator.validate() artifact curator.save_artifact( keyapi/fetched_data.parquet, descriptionData fetched from external API ) artifact.features.set_values({api_url: response.url}) ln.finish()DataFrameCurator的完整能力validate()、cat.standardize()、cat.add_ontology()等在 annotation-validation.md 中有详细展开这里用 curator 同时完成了结构校验与净化。数据库集成连接外部数据库时使用命名 secret如SOURCE_DB_URL绝不在代码或日志中打印连接串值查询出的行同样先经 schema 校验再入库import os import pandas as pd import sqlalchemy as sa import lamindb as ln ln.track() # Connect using a named secret; never paste or print the URL value engine sa.create_engine(os.environ[SOURCE_DB_URL]) # Query data query SELECT * FROM experiments WHERE date 2025-01-01 df pd.read_sql(query, engine) # Validate external rows before registration schema ln.Schema.get(nameexternal_experiments_schema) curator ln.curators.DataFrameCurator(df, schema) curator.validate() artifact curator.save_artifact( keyexternal_db/experiments_2025.parquet, descriptionExperiments from external database ) ln.finish()Croissant 元数据Croissant 是面向 ML 数据集发现与互操作的元数据格式。LaminDB artifact 以丰富元数据注册后可导出 Croissant 元数据以支持数据集的发现与互操作导出需要额外配置# Create artifact with rich metadata artifact ln.Artifact.from_dataframe( df, keydatasets/published_data.parquet, descriptionPublished dataset with Croissant metadata ).save() # Export Croissant metadata (requires additional configuration) # Enables dataset discovery and interoperability集成最佳实践原文档给出了十条经过实践检验的集成准则贯穿所有集成场景保持一致追踪在所有集成工作流中使用ln.track()。链接外部 ID把 WB run ID、MLflow experiment ID 等作为非敏感 feature 存储如artifact.features.set_values({wandb_run_id: ...})。数据集中化以 LaminDB 作为数据 artifact 的单一事实来源。参数双向同步同时向 LaminDB 与 ML 平台记录参数。版本整体对齐让代码git、数据LaminDB与实验ML 平台保持同步。缓存策略化为云存储配置合适的缓存位置详见lamin cache set与 setup-deployment.md 的缓存章节。使用本体背书注释通过模块专属管理器如artifact.cell_types.add(...)、schema 或类型化 feature 关联经过验证的 Bionty 记录。文档化集成为 artifact 添加说明其集成上下文的描述。增量测试先用小数据集验证集成是否工作。监控血缘用view_lineage()确保集成追踪确实生效。故障排查S3 凭据缺失test -n $AWS_ACCESS_KEY_ID echo AWS_ACCESS_KEY_ID is set test -n $AWS_SECRET_ACCESS_KEY echo AWS_SECRET_ACCESS_KEY is set export AWS_DEFAULT_REGIONus-east-1GCS 认证失败gcloud auth application-default login test -n $GOOGLE_APPLICATION_CREDENTIALS echo GOOGLE_APPLICATION_CREDENTIALS is setGit 同步失效# Ensure git repo is set lamin settings get sync-git-repo # Ensure youre in git repo git status # Commit changes before tracking git add . git commit -m Update analysis ln.track()MLflow artifact 未同步两个系统的 artifact 是独立的必须显式双写# Save explicitly to both systems mlflow.log_artifact(model.pkl) ln.Artifact(model.pkl, keymodels/model.pkl).save()小结LaminDB 的集成面覆盖了数据科学工作流的完整链路存储本地/S3/GCS/R2/MinIO/HTTP/HF、调度Nextflow/Snakemake/Redun、实验跟踪WB/MLflow/Transformers/scVI、数组计算TileDB-SOMA/DuckDB、可视化Vitessce、领域模式Bionty/wetlab/临床与版本控制Git。所有集成共享同一套核心原则——用ln.track()捕获血缘、用 curator schema 净化外部输入、用 feature 关联外部系统 ID、用view_lineage()验证可溯源性。实践这些模式时注意安全基线凭据与数据库 URL 一律走 secret manager 或命名环境变量切勿回显或提交明文结合 SKILL.md 的安全与安全默认值章节即可把 LaminDB 稳妥地嵌入任何既有流水线。【免费下载链接】scientific-agent-skillsTurn any AI agent into an AI Scientist. The #1 Agent Skills library for science, used by 190,000 scientists worldwide. 165 ready-to-use validated skills plus 100 scientific databases covering biology, chemistry, medicine, and drug discovery. Compatible with Cursor, Claude Code, Codex, Pi, Antigravity, and the open Agent Skills standard.项目地址: https://gitcode.com/GitHub_Trending/cl/scientific-agent-skills创作声明：本文部分内容由AI辅助生成（AIGC），仅供参考