Skip to content

bug: load study.pkl.gz / exp_args.pkl with a restricted unpickler (sfind-862fb47f) - #65

Open
devin-ai-integration[bot] wants to merge 2 commits into
mainfrom
devin/sfind-862fb47f-study-safe-unpickle
Open

devin-ai-integration[bot] wants to merge 2 commits into
mainfrom
devin/sfind-862fb47f-study-safe-unpickle

Conversation

@devin-ai-integration

@devin-ai-integration devin-ai-integration Bot commented Nov 5, 2026 •

Copy link
Copy Markdown

Resumo

Corrige o finding de code scan sfind-862fb47f (deserialização insegura): Study.load fazia pickle.load em <dir>/study.pkl.gz e, no ramo legado, ExpResult.exp_args fazia pickle.load em exp_args.pkl. Uma pasta de estudo plantada (drive compartilhado, arquivo enviado por colaborador) executava código ao ser carregada por archive_studies.search_for_reasons_to_archive, Study.load_most_recent(), relaunch do main.py etc.

Novo módulo agentlab.experiments.safe_pickle com RestrictedUnpickler.find_class:

if (module, name) in SAFE_GLOBALS:          # stdlib/numpy/pandas/pyarrow/omegaconf (lista explícita)
    return super().find_class(module, name)
if in_trusted_package(module) and "." not in name:   # agentlab, browsergym, bgym + $AGENTLAB_TRUSTED_PICKLE_MODULES
    obj = super().find_class(module, name)
    if is_dataclass/Enum/pydantic BaseModel(obj) and in_trusted_package(obj.__module__):
        return obj
raise UnsafePickleError(...)   # subclasse de pickle.UnpicklingError

Usado em Study.load (study.pkl.gz) e ExpResult.exp_args (exp_args.pkl — cobre o ramo legado de Study.load, load_result_df, find_incomplete). Funções, classes não-dados e gadgets via pacotes confiáveis (agentlab.experiments.study + os.system, make_study, ExpResult, …) são recusados antes de serem chamados. O formato em disco não muda (Study.save continua com pickle.dump), então estudos existentes continuam carregando.

Mudança de contrato (sinalizada): agent args definidos fora de agentlab/browsergym agora exigem AGENTLAB_TRUSTED_PICKLE_MODULES=meu_pacote para Study.load/exp_args funcionarem (documentado no README).

GAIA: o GaiaBenchmark guardava o DatasetDict do Hugging Face no pickle do study, o que o unpickler restrito recusa. Agora GaiaBenchmark.__getstate__ grava dataset=None (as tarefas já ficam em env_args_list), sem ampliar a allowlist para datasets.*.

tests/agents/test_agent.py define Cheat*Args no próprio módulo de teste; um fixture autouse adiciona esse módulo a AGENTLAB_TRUSTED_PICKLE_MODULES para o load_result_df (é exatamente a mudança de contrato acima).

Fora do escopo: step_*.pkl.gz / goal_object.pkl.gz (carregados pelo AgentXray) continuam com pickle puro — contêm objetos de resposta de provedores LLM e exigiriam uma allowlist maior; o README avisa para só abrir pastas de fontes confiáveis.

Como testar

  • pytest tests/experiments/test_safe_pickle.py — 18 passam. Os 3 testes com payload (study.pkl.gz malicioso, exp_args.pkl legado malicioso, search_for_reasons_to_archive em um results root plantado) falham em main (o arquivo marcador é criado) e passam nesta branch (marcador nunca criado, UnsafePickleError).
  • Round-trip real: make_study(...).save() + Study.load para miniwob_tiny_test, miniwob, workarena_l1, webarena/visualwebarena (SequentialStudies), assistantbench, weblinx, ParallelStudies com WebArenaInstanceVars, e os exp_args.pkl de 2024 em tests/data/test_study — 22/22 OK.
  • pytest tests/experiments tests/analyze -k "not miniwob and not launch_system and not ray" — 32 passed, 1 skipped.
  • GAIA: DatasetDict em memória → GaiaBenchmark → Study.save → Study.load: antes UnsafePickleError: datasets.arrow_dataset.Dataset, depois carrega as tarefas. Rodado com stubs de tapeagents/hydra (não instalados no dev/CI), por isso não virou teste no repo.
  • CI: tests/agents/test_agent.py (precisa do servidor MiniWoB) passa no job agentlab.
  • Não testado: entradas omegaconf da allowlist (TapeAgentArgs.config: DictConfig) — omegaconf/hydra não estão instalados no ambiente.

Devin-Org: engineering

Link to Devin session: https://devin-gtm.devinenterprise.com/sessions/7c6efa1d1d1d4d0f844f5a4bf0ce8f99
Open in Devin Desktop: https://devin-gtm.devinenterprise.com/desktop/session/7c6efa1d1d1d4d0f844f5a4bf0ce8f99?variant=devin


Devin Review

Status Commit
⚪ Not started —

Run Devin Review

Enable auto-review to review your pull requests automatically.

Devin Review (Staging)


Devin Review

Study.load and ExpResult.exp_args used pickle.load on files from any study
directory, so a planted study folder could run code on load (archive_studies,
load_most_recent, main.py relaunch). Add agentlab.experiments.safe_pickle,
which only resolves dataclasses/Enums/pydantic models from agentlab and
browsergym (plus AGENTLAB_TRUSTED_PICKLE_MODULES) and an allowlist of
stdlib/numpy/pandas/pyarrow data types.

Fixes code scan finding sfind-862fb47f130745099616ac639c00fc9c.
@devin-ai-integration

Copy link
Copy Markdown
Author

I'll fix CI failures and address comments from users with write access. I'll skip comments containing "(aside)".

  • Disable automatic comment, CI, and merge conflict monitoring

devin-ai-integration[bot]

This comment was marked as resolved.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

0 participants