Repository navigation
bug: never unpickle agent-writable files when loading experiment results (sfind-454ac387) - #66
Open
devin-ai-integration[bot] wants to merge 3 commits into
Open
devin-ai-integration[bot] wants to merge 3 commits into
devin-ai-integration[bot] wants to merge 3 commits into
Conversation
…lts (sfind-454ac387) - yield_all_exp_results walks with os.walk(followlinks=False), skips the shared_code sandbox and does not descend into experiment directories. - ExpResult loads exp_args.pkl / step_N.pkl.gz / goal_object.pkl.gz with a restricted unpickler (class allowlist, AGENTLAB_TRUSTED_PICKLE_MODULES).
Author
|
I'll fix CI failures and address comments from users with write access. I'll skip comments containing "(aside)".
|
…afe_pickle (sfind-454ac387)
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Resumo
Corrige o finding sfind-454ac387 (high): ao carregar resultados, o AgentLab fazia
pickle.loadem qualquerexp_args.pklencontrado porglob("**/exp_args.pkl")dentro do diretório do estudo — inclusive em<study_dir>/shared_code, que é o diretório de trabalho do sandbox de código do GAIA (gravável pelo agente). Um agente induzido por prompt injection podia gravarshared_code/x/exp_args.pkle executar código no host quandoStudy.runchamavaget_resultsao fim de cada trial.Duas camadas:
yield_all_exp_resultsemloop.py): o glob recursivo foi trocado por_find_exp_dirs, que usaos.walk(followlinks=False), nunca entra emCODE_SANDBOX_DIR_NAME("shared_code") e não desce dentro de um diretório de experimento (o AgentLab nunca aninha experimentos; isso cobre o symlinkexp_dir/code -> shared_code). Ordem agora determinística (ordenada).gaia.init_code_sandboxpassa a usar a mesma constante.ExpResult.exp_args,get_step_info,goal_object.pkl.gz): novoagentlab.experiments.safe_pickle.load, umpickle.Unpicklercomfind_classrestrito:exec,os.system,Path.write_text), exceto helpers de reconstrução (copyreg._reconstructor,numpy..._reconstruct/scalar/_frombuffer,_codecs.encode);agentlab,browsergym,bgym+ raízes extras emAGENTLAB_TRUSTED_PICKLE_MODULES(separadas por vírgula, p/ agentes em pacotes próprios);AgentArgs/BaseModelArgscustomizados no script de lançamento ou em testes) — nada é importado para resolvê-las;typing.Any(metadados deDictConfigdo OmegaConf/Hydra usados pelo GAIA);tapeagents/pydantic/omegaconf/openai/anthropicapenas dataclasses, enums, modelos pydantic e nós omegaconf (evita gadgets comoContainerExecutor);ndarray, dtypes, escalares;numpy.memmapbloqueado) ePIL.Image.Image.UnsafePickleError(subclasse deUnpicklingError) com mensagem indicando a env var.Compatibilidade: os pickles de
tests/data/test_study(era BrowserGym) carregam normalmente. Resultados que referenciem classes de pacotes de terceiros fora da lista passam a exigirAGENTLAB_TRUSTED_PICKLE_MODULES=<pacote>.study.pkl.gz(Study.load) continua compickle.load: fica na raiz do estudo, fora do alcance do sandbox, e contém DataFrames pandas — migrar para formato só-dados fica como follow-up.Como testar
test_safe_result_loading.pycobre: arquivos plantados emshared_code/x,shared_code/a/b, via symlinkexp_dir/codee aninhados num experimento não são listados poryield_all_exp_results/load_result_df/Study.load_exp_args_list; diretórios symlinkados não são seguidos; payloads emexp_args.pkl,step_0.pkl.gzegoal_object.pkl.gzlevantamUnsafePickleErrorsem criar o marker; round-trip deStepInforeal (numpy,Discussion,AgentInfo,StepTimestamps) e a env var.Reprodução do cenário do finding (estudo de teste +
shared_code/x/exp_args.pklque grava um marker), antes/depois:Sem mudança de contrato de API pública além da nova env var
AGENTLAB_TRUSTED_PICKLE_MODULESe da exceçãoUnsafePickleError. Testes de ponta a ponta com MiniWoB não foram rodados (servidor MiniWoB não disponível offline).Devin-Org: engineering
Link to Devin session: https://devin-gtm.devinenterprise.com/sessions/252d9207aa9545648a72ae60d8adfce9
Open in Devin Desktop: https://devin-gtm.devinenterprise.com/desktop/session/252d9207aa9545648a72ae60d8adfce9?variant=devin
Devin Review