Upload error to S3 (DigitalOcean Spaces)

[Robert-turbo]

Hi,

after uploading 1535 files to S3 (DigitalOcean Spaces) an error occurs:

Traceback (most recent call last):
  File "/dumpfiles/gridfsmigrate/./migrate.py", line 236, in <module>
    obj.dumpfiles("rocketchat_uploads", store)
  File "/dumpfiles/gridfsmigrate/./migrate.py", line 121, in dumpfiles
    key = store.put(filename, data, upload)
  File "/dumpfiles/gridfsmigrate/./migrate.py", line 71, in put
    key = self.uniqueID + "/uploads/" + entry['rid'] + "/" + entry[
KeyError: 'userId'

does anyone know what can be core of this problem?

[yanehi]

same error here with S3 uploads

[stefanandres]

For what it's worth, I've came across the same problem. I think the mongodb just had some files with deleted users or something.
To skip them I put a ugly workaround in the script like this:

     def put(self, filename, data, entry):
-        key = self.uniqueID + "/uploads/" + entry['rid'] + "/" + entry[
-            'userId'] + "/" + entry['_id']
-        if 'type' in entry.keys():
-            self.s3.Object(self.bucket, key).put(
-                Body=data,
-                ContentType=entry['type'],
-                ContentDisposition='inline; filename="' + self.encodeURI(entry['name']) + '"')
-        else:
-            self.s3.Object(self.bucket, key).put(
-                Body=data,
-                ContentDisposition='inline; filename="' + self.encodeURI(entry['name']) + '"')
-        return key
-
+        try:
+          key = self.uniqueID + "/uploads/" + entry['rid'] + "/" + entry[
+              'userId'] + "/" + entry['_id']
+          if 'type' in entry.keys():
+              self.s3.Object(self.bucket, key).put(
+                  Body=data,
+                  ContentType=entry['type'],
+                  ContentDisposition='inline; filename="' + self.encodeURI(entry['name']) + '"')
+          else:
+              self.s3.Object(self.bucket, key).put(
+                  Body=data,
+                  ContentDisposition='inline; filename="' + self.encodeURI(entry['name']) + '"')
+          return key
+        except Exception as error:
+          print("Skipping: ", repr(entry))
+          print("An exception occurred:", error)
+

 class Migrator():
     def __init__(self,
@@ -139,10 +144,11 @@ class Migrator():
     def writelog(self):
         file = open(self.logfile, "a")
         for entry in self.log:
-            line = entry["id"] + "," + entry["file"] + "," + entry[
-                "collection"] + ",log" + "," + entry[
-                    "key"] + "\n"
-            file.write(line)
+            if id in entry:
+              line = entry["id"] + "," + entry["file"] + "," + entry[
+                  "collection"] + ",log" + "," + entry[
+                      "key"] + "\n"
+              file.write(line)
         file.close()