Parallel execution

[diit]

What would be a proof of concept for executing an array of chromeless "agents" in parallel?

ie (currently)

for(var item in parsedList) {
        const page = await chromeless
            .goto(parsedList[item].href)
            .evaluate(() => {
            })
}

Current Theory

Promise.all(arrayOfAgents).then(result => ...)

However, it returns an array of undefined.

[td0m]

I think this is a great idea, currently I am pulling a lot of data in parallel by running multiple terminals executing node scraper.js but I agree a native parallel execution would be useful. +1

[notatestuser]

This seems to work if multiple instances of Chromeless are created. Am I missing something?

[LeMoussel]

Does it work with multiple instances of 'Chromeles' and one instance of Chrome on AWS ?

[td0m]

I don't know about AWS, but it works on my local machine with multiple chromeless instances and one chrome headless running.

[notatestuser]

@LeMoussel yes.

[ldenoue]

So would it work if I use chromeless from a NodeJS server that handles requests from several users?
I mean, does chromeless span several chrome processes or uses several tabs? And if tabs, are they closed automatically?

[adieuadieu]

Each instance of new Chromeless will create it's own tab and then close it when you call chromeless.end(). When passing the remote: true option, each instance will connect to it's own Proxy (e.g. one instance of chromeless, one lambda invocation.) In this case, calling chromeless.end() will effectively end the Lambda function invocation.

I wrote some more in #24 (comment) about running multiple Chromeless instances at once.

[sul4bh]

Is it possible to run parallel tests locally?

I have an example based on #24 (comment) which does the following:

• Opens up 3 Chrome windows for 3 tests in total.
• Throws the exception:

 Unhandled Rejection
  Error: unexpected server response (500)
    ClientRequest._req.on (node_modules/ws/lib/WebSocket.js:651:26)

I am hoping the 3 windows that I get do not share session. That will allow me to write atomic tests.

I am on Chrome Canary 62.0.3182.0 (64-bit)

[adieuadieu]

@sul4bh make sure each instance has it's own debugger port with --remote-debugging-port=XXXX

Typically Chrome doesn't like to run more than one instance of itself. To run multiple, separate instances of Chrome locally, you might need to run each within a Docker container. From our work-in-progress CircleCI config:

docker run \
  -d \
  --rm \
  --name chrome \
  --shm-size 1024m \
  -p 9222:9222 \
  --cap-add SYS_ADMIN \
  yukinying/chrome-headless-browser

[sul4bh]

@adieuadieu
I am trying to make it super easy for product engineers to write integration tests. Not sure if introducing docker based headless chrome would help with that. I want to keep the whole process as simple as possible. But the CI can use docker based headless setup. Thanks for the pointer to your config file.

I was thinking, we can add an option key to ChromelessOptions that will make the Chrome instances to start with a randomized debug port number (https://github.com/graphcool/chromeless/blob/master/src/chrome/local.ts#L40) What do you think about it? I can work on a PR if you think such option can be useful.