No files are copied if i run the dump command

[phatair]

Hello,
i want to migrate the uploads from our RocketChat Installation from GridFS to FileSystem

i have installed phyton3 and phyton-pip.
I have also installed phyton-pip pymongo and boto3
i set filesystem in the RocketChat Upload config. If i upload a file in Rocketchat it is placed in /var/snap/rocketchat-server/common/uploads/

After that i made the py file executable and run the script with the follwoing syntax
./migrate.py -c dump -r rocketchat -t FileSystem -d /var/snap/rocketchat-server/common/uploads/

a csv.log in /var/snap/rocketchat-server/common/uploads/ but its empty and no files are copied.
What did i wrong?

it would be great if someone can help me.
thanks!

[phatair]

Problem solved - we used the snap version and the DB is called parties and not rocketchat

[rongten]

Sorry to disturb, but beside using -r parties instead of -r rocketchat did you have to change anything else? I get a message like 'pymongo.errors.OperationFailure: Authentication failed.' when I try.

[rongten]

So, the problem that for the snap mongo is without authentication. So in my case I removed the username and password parameters form the MongoClient call at line 91 and it is currently dumping the files.

[Alexdc-12]

IM having the same issue as Rongten, i have removed the calls for user/pass in the mongo.py file but now receive a further error regarding RAM, DB is aroungd 5.5GB is this the problem?
pymongo.errors.OperationFailure: Executor error during find command :: caused by :: Sort operation used more than the maximum 33554432 bytes of RAM. Add an index, or specify a smaller limit., full error: {'operationTime': Timestamp(1635796276, 1), 'ok': 0.0, 'errmsg': 'Executor error during find command :: caused by :: Sort operation used more than the maximum 33554432 bytes of RAM. Add an index, or specify a smaller limit.', 'code': 96, 'codeName': 'OperationFailed', '$clusterTime': {'clusterTime': Timestamp(1635796276, 1), 'signature': {'hash': b'\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00', 'keyId': 0}}}

[aquaspy]

Unforntunately, same result here:

raise OperationFailure(errmsg, code, response, max_wire_version) pymongo.errors.OperationFailure: Executor error during find command :: caused by :: Sort operation used more than the maximum 33554432 bytes of RAM. Add an index, or specify a smaller limit., full error: {'operationTime': Timestamp(1657847213, 3), 'ok': 0.0, 'errmsg': 'Executor error during find command :: caused by :: Sort operation used more than the maximum 33554432 bytes of RAM. Add an index, or specify a smaller limit.', 'code': 96, 'codeName': 'OperationFailed', '$clusterTime': {'clusterTime': Timestamp(1657847213, 3), 'signature': {'hash': b'\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00', 'keyId': 0}}}

I edited the .py script to do not use user and pass, it was working until dumping 769, and then crashed :(

765. Dumping WJshd43CbzoR48i54 Clipboard - 15 de Fevereiro de 2022 às 15:56.png
766. Dumping Y6J5YH6foPgF9TRxD thumb-Clipboard - 15 de Fevereiro de 2022 às 15:56.png
767. Dumping 2WE7oTjQwbPCKduyo Clipboard - 15 de Fevereiro de 2022 às 16:02.png
768. Dumping BTmsMh3nGsn97Wudt thumb-Clipboard - 15 de Fevereiro de 2022 às 16:02.png
769. Dumping XNvuNvEJXvHZEtouy 1644954289716.m4a

At this point, I think I will start a clean install with filesystem since the beginning.

[Sx3]

Same issue any help ?

[Sx3]

@aquaspy @Alexdc-12

pymongo.errors.OperationFailure: Executor error during find command :: caused by :: Sort operation used more than the maximum 33554432 bytes of RAM. Add an index, or specify a smaller limit.,
This is due to the default memory allocation of mongo
so here is the fix...

login into the mongo db
in terminal mongo or rocketchat-server.mong for rocketchat snap
then change the memory value from 32MB to 320MB or anything larger

db.adminCommand({setParameter: 1, internalQueryExecMaxBlockingSortBytes: **335544320**})

also if you are running on a VM increase the No of CPUs and RAM so that the script will run faster :)

[MaisumAbbas]

In my case, for loop gets skipped in dumpfiles function even though it should run. Not sure why.

Look into it and my cursor object was getting skipped due to being zero for some reason. I had to use rewind() => uploads.rewind() to reset the cursor position to make it work.