Interacting with dbt inside docker container

[srpwnd]

My setup utilizes running dbt inside a docker container as a part of whole Meltano-based stack.

I was wondering if this plugin could interact with dbt executable inside such container. I tried replacing dbt executable with docker exec command accordingly:

path_to_dbt = "docker exec $(docker ps | grep meltano-airflow-scheduler | sed -e 's/ .*$//') meltano invoke dbt-duckdb",

but I'm receiving Executable not found error.

Is it possible to configure dbtpal to work with dbt inside docker container?

[PedramNavid]

hey @srpwnd, while I haven't tested this full, I have a theory and it's all about the difference between a command and an argument.

path_to_dbt must be a single command that can run, however you provided a full command with arguments.

In your case, docker is the command and the arguments are exec $(docker ps | grep meltano-airflow-scheduler | sed -e 's/ .*$//') meltano invoke dbt-duckdb

One option might be to set dbt_path to just docker.

Then set pre_cmd_args to: exec $(docker ps | grep meltano-airflow-scheduler | sed -e 's/ .*$//') meltano invoke dbt-duckdb"

By default, pre_cmd_args are set to

   pre_cmd_args = {
        "--use-colors",
        "--printer-width=10",
    },

I would override that whole thing and just set it to the docker args.

What that should do is change the way it builds the dbt command here:

dbtpal/lua/dbtpal/commands.lua

Line 15 in 46d6b7a

local pre_cmd_args = config.options.pre_cmd_args or {}

But again, very much untested so we might have to work on this one together and I've never used dbt within a docker container within meltano before.

[srpwnd]

It's a progress!

So I set up the path and pre-cmd-args like so

  path_to_dbt = "docker",
  pre_cmd_args = {
    "exec",
    "$(docker ps | grep meltano-airflow-scheduler | sed -e \'s/ .*$//\')",
    "meltano",
    "invoke",
    "dbt-duckdb"
  },

And now the telescope window opens up (although it doesn't do anything) but when running dbt run I get a following error which is something I'm not sure how to solve

[dbtpal] [WARN  08:45:11] ...e/nvim/site/pack/packer/start/dbtpal/lua/dbtpal/main.lua:73: dbt command encounted a handled error, see popup for details
Error executing vim.schedule lua callback: ...vim/site/pack/packer/start/dbtpal/lua/dbtpal/display.lua:29: invalid key: title
stack traceback:
        [C]: in function 'nvim_open_win'
        ...vim/site/pack/packer/start/dbtpal/lua/dbtpal/display.lua:29: in function 'popup'
        ...e/nvim/site/pack/packer/start/dbtpal/lua/dbtpal/main.lua:64: in function 'onexit'
        ...e/nvim/site/pack/packer/start/dbtpal/lua/dbtpal/main.lua:83: in function <...e/nvim/site/pack/packer/start/dbtpal/lua/dbtpal/main.lua:83>

The output of build command log message is following

[dbtpal] [DEBUG 09:12:51] ...im/site/pack/packer/start/dbtpal/lua/dbtpal/commands.lua:33: Building dbt command: docker exec $(docker ps | grep meltano-airflow-scheduler | sed -e 's/ .*$//') meltano invoke dbt-duckdb run --select stg_example_sample --profiles-dir /Users/srpwnd/repos/data-stack-ds/transform/profiles/duckdb --project-dir /Users/srpwnd/repos/data-stack-ds/transform

I tried running the generated command straight from my terminal and it throws following error:

08:14:44  Encountered an error:                                                                                                                                                                                       │
Runtime Error                                                                                                                                                                                                         │
  fatal: Invalid --project-dir flag. Not a dbt project. Missing dbt_project.yml file

Which after a bit of thinking made sense, because the path to dbt project and profiles is different on my local host and inside the docker container. Since all my dbt docker projects follow the same folder structure I tried manually setting the project and profiles dir

  path_to_dbt_project = "/project/transform",
  path_to_dbt_profiles_dir = "/project/transform/profiles/duckdb",

but it seems that somewhere along the code is accidentally overwriting the path_to_dbt_project with the auto detected value since the command build output after my change doesn't respect the project dir:

[dbtpal] [DEBUG 09:22:01] ...im/site/pack/packer/start/dbtpal/lua/dbtpal/commands.lua:33: Building dbt command: docker exec $(docker ps | grep meltano-airflow-scheduler | sed -e 's/ .*$//') meltano invoke dbt-duckdb run --select stg_example_sample --profiles-dir /project/transform/profiles/duckdb --project-dir /Users/srpwnd/repos/data-stack-ds/transform

I tried looking for where this might happen but my lack of knowledge of Lua didn't serve me very much and I was unable to find it.

I'm happy to debug and try this further if you can point me into the right directions!

[PedramNavid]

What version of neovim are you running?

Error executing vim.schedule lua callback: ...vim/site/pack/packer/start/dbtpal/lua/dbtpal/display.lua:29: invalid key: title

This is a little suspicious to me. It comes from this line:

    local win_opts = vim.tbl_deep_extend("force", {
        relative = "editor",
        style = "minimal",
        border = "double",
        width = width,
        height = height,
        col = left,
        row = top,
        title = title or "Job Results",
    }, opts or {})

    local buf = vim.api.nvim_create_buf(false, true)
    local win = vim.api.nvim_open_win(buf, true, win_opts)

I am wondering if title was introduced in a later version of neovim.

One thing that might help with the rest is increasing the log level to TRACE.
open neovim like this

DBTPAL_LOG_LEVEL=trace nvim

Once you open a dbt file, before running it, you can view the log messages and then paste the output of that after opening a SQL file, should help isolate the problem!

:messages

Here's a loom showing how it's done.
https://www.loom.com/share/24bf7d38fffb4c22909b549fe18d0356

[srpwnd]

As far as I'm aware it's the latest release build:

NVIM v0.8.3
Build type: Release
LuaJIT 2.1.0-beta3
Compiled by brew@Ventura-arm64.local

Features: +acl +iconv +tui
See ":help feature-compile"

   system vimrc file: "$VIM/sysinit.vim"
  fall-back for $VIM: "/opt/homebrew/Cellar/neovim/0.8.3/share/nvim"

Run :checkhealth for more info

Trace messages when opening up a dbt model:

[dbtpal] [TRACE 09:38:19] ...nvim/site/pack/packer/start/dbtpal/lua/dbtpal/config.lua:36: {
[dbtpal]   custom_dbt_syntax_enabled = true,
[dbtpal]   extended_path_search = true,
[dbtpal]   path_to_dbt = "dbt",
[dbtpal]   path_to_dbt_profiles_dir = "/Users/srpwnd/.dbt",
[dbtpal]   path_to_dbt_project = "",
[dbtpal]   pre_cmd_args = { "--use-colors", "--printer-width=10" },
[dbtpal]   protect_compiled_files = true
[dbtpal] }
[dbtpal] [TRACE 09:38:19] ...e/nvim/site/pack/packer/start/dbtpal/lua/dbtpal/init.lua:9: dbtpal initialized
[dbtpal] [DEBUG 09:38:19] ...im/site/pack/packer/start/dbtpal/lua/dbtpal/projects.lua:19: path_to_dbt is not set, attempting to autofind.
[dbtpal] [TRACE 09:38:19] ...im/site/pack/packer/start/dbtpal/lua/dbtpal/projects.lua:8: Searching for dbt project dir in stg_example_sample.sql

But then it outputs different config when I try to do dbt run:

[dbtpal] [TRACE 09:42:56] ...im/site/pack/packer/start/dbtpal/lua/dbtpal/commands.lua:7: dbtpal config: {
[dbtpal]   custom_dbt_syntax_enabled = true,
[dbtpal]   extended_path_search = true,
[dbtpal]   path_to_dbt = "docker",
[dbtpal]   path_to_dbt_profiles_dir = "/project/transform/profiles/duckdb",
[dbtpal]   path_to_dbt_project = "/Users/srpwnd/repos/data-stack-ds/transform",
[dbtpal]   pre_cmd_args = { "exec", "$(docker ps | grep meltano-airflow-scheduler | sed -e 's/ .*$//')", "meltano", "invoke", "dbt-duckdb" },
[dbtpal]   protect_compiled_files = true
[dbtpal] }
[dbtpal] [DEBUG 09:42:56] ...im/site/pack/packer/start/dbtpal/lua/dbtpal/commands.lua:33: Building dbt command: docker exec $(docker ps | grep meltano-airflow-scheduler | sed -e 's/ .*$//') meltano invoke dbt-duckd
b run --select stg_example_sample --profiles-dir /project/transform/profiles/duckdb --project-dir /Users/srpwnd/repos/data-stack-ds/transform
[dbtpal] [WARN  09:42:56] ...e/nvim/site/pack/packer/start/dbtpal/lua/dbtpal/main.lua:73: dbt command encounted a handled error, see popup for details
Error executing vim.schedule lua callback: ...vim/site/pack/packer/start/dbtpal/lua/dbtpal/display.lua:29: invalid key: title
stack traceback:
        [C]: in function 'nvim_open_win'
        ...vim/site/pack/packer/start/dbtpal/lua/dbtpal/display.lua:29: in function 'popup'
        ...e/nvim/site/pack/packer/start/dbtpal/lua/dbtpal/main.lua:64: in function 'onexit'
        ...e/nvim/site/pack/packer/start/dbtpal/lua/dbtpal/main.lua:83: in function <...e/nvim/site/pack/packer/start/dbtpal/lua/dbtpal/main.lua:83>

[PedramNavid]

Ah interesting, looks like title is only from the nightly version. I removed title for now, give this a shot: https://github.com/PedramNavid/dbtpal/releases/tag/v0.0.2

Still very odd that path_to_dbt_project = "" is not set. Where are you setting this in your config?

[srpwnd]

Ok, so the updated release helped with the invalid key: title error, but still doesn't fully work. I think now the incorrect path_to_dbt_project is the culprit.

I get the following messages:

[dbtpal] [TRACE 11:44:21] ...nvim/site/pack/packer/start/dbtpal/lua/dbtpal/config.lua:36: {
[dbtpal]   custom_dbt_syntax_enabled = true,
[dbtpal]   extended_path_search = true,
[dbtpal]   path_to_dbt = "dbt",
[dbtpal]   path_to_dbt_profiles_dir = "/Users/srpwnd/.dbt",
[dbtpal]   path_to_dbt_project = "",
[dbtpal]   pre_cmd_args = { "--use-colors", "--printer-width=10" },
[dbtpal]   protect_compiled_files = true
[dbtpal] }
[dbtpal] [TRACE 11:44:21] ...e/nvim/site/pack/packer/start/dbtpal/lua/dbtpal/init.lua:9: dbtpal initialized
[dbtpal] [DEBUG 11:44:21] ...im/site/pack/packer/start/dbtpal/lua/dbtpal/projects.lua:19: path_to_dbt is not set, attempting to autofind.
[dbtpal] [TRACE 11:44:21] ...im/site/pack/packer/start/dbtpal/lua/dbtpal/projects.lua:8: Searching for dbt project dir in stg_example_sample.sql
⚠️  No compatible snippets found, try updating `coq.artifacts`
[dbtpal] [TRACE 11:44:23] ...im/site/pack/packer/start/dbtpal/lua/dbtpal/commands.lua:7: dbtpal config: {
[dbtpal]   custom_dbt_syntax_enabled = true,
[dbtpal]   extended_path_search = true,
[dbtpal]   path_to_dbt = "docker",
[dbtpal]   path_to_dbt_profiles_dir = "/project/transform/profiles/duckdb",
[dbtpal]   path_to_dbt_project = "/Users/srpwnd/repos/data-stack-ds/transform",
[dbtpal]   pre_cmd_args = { "exec", "$(docker ps | grep meltano-airflow-scheduler | sed -e 's/ .*$//')", "meltano", "invoke", "dbt-duckdb" },
[dbtpal]   protect_compiled_files = true
[dbtpal] }
[dbtpal] [DEBUG 11:44:23] ...im/site/pack/packer/start/dbtpal/lua/dbtpal/commands.lua:33: Building dbt command: docker exec $(docker ps | grep meltano-airflow-scheduler | sed -e 's/ .*$//') meltano invoke dbt-duckd
b run --select stg_example_sample --profiles-dir /project/transform/profiles/duckdb --project-dir /Users/srpwnd/repos/data-stack-ds/transform
[dbtpal] [WARN  11:44:23] ...e/nvim/site/pack/packer/start/dbtpal/lua/dbtpal/main.lua:73: dbt command encounted a handled error, see popup for details

The pop-up shows nothing except for "Press q to close" message.

This is the full dbtpal config I have in my init.lua:

require("dbtpal").setup({
  path_to_dbt = "docker",
  pre_cmd_args = {
    "exec",
    "$(docker ps | grep meltano-airflow-scheduler | sed -e \'s/ .*$//\')",
    "meltano",
    "invoke",
    "dbt-duckdb"
  },
  path_to_dbt_project = "/project/transform",
  path_to_dbt_profiles_dir = "/project/transform/profiles/duckdb",
  extended_path_search = true,
  protect_compiled_files = true
})

vim.keymap.set('n', '<leader>drf', require("dbtpal").run)
vim.keymap.set('n', '<leader>drp', require("dbtpal").run_all)
vim.keymap.set('n', '<leader>dtf', require("dbtpal").test)
vim.keymap.set('n', '<leader>dm', require('dbtpal.telescope').dbt_picker)

require('telescope').load_extension('dbtpal')

In case you need the full picture here is a Packer config in full file and here is the above mentioned config of dbtpal.

[PedramNavid]

Looks like it does end up loading the dbt_project path, as seen here:

docker exec $(docker ps | grep meltano-airflow-scheduler | sed -e 's/ .*$//') meltano invoke dbt-duckdb run --select stg_example_sample --profiles-dir /project/transform/profiles/duckdb --project-dir /Users/srpwnd/repos/data-stack-ds/transform

if you run that command locally does it work?
it says dbt encountered a handled error..usually exit code 1 means something like your dbt run command failed. Very odd that it didn't capture any of the error messages in the popup though.

Try these two commands in neovim:

:lua require('dbtpal.main').run_command('ls')

:lua require('dbtpal.main').run_command('debug')

Do either of these work?

[srpwnd]

When running the docker exec I get the following error:

07:35:03  Encountered an error:
Runtime Error
  fatal: Invalid --project-dir flag. Not a dbt project. Missing dbt_project.yml file

and neither of the two commands in neovim work. Both of them give me the same error

[dbtpal] [WARN  11:26:29] ...e/nvim/site/pack/packer/start/dbtpal/lua/dbtpal/main.lua:73: dbt command encounted a handled error, see popup for details

and again the pop-up shows nothing.

Should I include any screenshots or more info?

[PedramNavid]

I'm not sure if that will help. At this point, I would have to go and say that running dbt inside a docker container isn't supported. I'd recommend installing dbt locally and using that when developing in neovim, until someone else can figure out how to get this working.

[srpwnd]

I've done some digging around and from my point of view it seems there might be some issue in plenary.job but that code is too advanced for my understanding and I found nothing relevant in the repo's issues.

Hopefully someone will be able to help.

Unfortunatelly using dbt locally is not possible in my workflow.

[kylejbrk]

@srpwnd did you ever find a solution to this?

[srpwnd]

@srpwnd did you ever find a solution to this?

Unfortunately not. But I've since moved to a different workflow anyway.

[kylejbrk]

@srpwnd did you ever find a solution to this?

Unfortunately not. But I've since moved to a different workflow anyway.

what workflow did you move to? or did you mean you dont use dbt anymore?