Amazon数据集处理问题

[ZhuYiZ-1205]

作者您好，在预处理Amazon Fashion数据集的时候，采用5-core或者3-core过滤都没有达到论文中表格里的值，请问您是怎么处理的？或者说有您当时使用的原始数据集么？可以传于我一份么？非常感谢！！！

[ninghaolu]

While following the data processing pipeline for the Fashion dataset, I encountered a discrepancy. In your processed data, the llm_embedding.npy file has a shape of (4722, 1536), and the inter.txt file includes 9094 users and 4722 items. However, no matter how I adjust or debug the user_core and item_core parameters in data_process.py, I have not been able to reproduce exactly 4722 unique items in my processed results. Similar problem happened in the Beauty dataset.

I was wondering if any additional filtering or truncation on users or items was applied that might not be explicitly mentioned in the paper or the codebase. If possible, could you also share the exact values of user_core and item_core used when processing the Fashion and Beauty dataset?

Your help and response are highly appreciated!

[lollipopmark]

I'm not the author, but I find somthing useful. The Beauty dadaset has 394908 inters, which is the same length with the baseline MELT, you can refer to MELT code and use their preprocess.ipynb.

The Fashion dataset has asin "B0002Z1JNK", see data/fashion/get_item_embedding.ipynb, and this "B0002Z1JNK" occurs in Fashion 2018 version.

[23fu-max]

你好，我想问问针对yelp数据集的inter.txt文件的数据量15720条是通过何种方式处理得来的，我利用项目的原始处理代码进行过滤处理，无法得到与论文相同的预处理数据规模

[lollipopmark]

在此，我替作者给出复现代码：

对于Beauty数据集，请在2014版本网页下载Per-category files中的Beautyreviews ，然后执行下面的代码，得到的beauty_inter.txt与LLMESR作者给出的zip包中的inter完全相同：

import gzip
import json
from collections import defaultdict

raw_path = "reviews_Beauty.json.gz"
out_path = "beauty_inter.txt"

countU = defaultdict(int)
countP = defaultdict(int)
with gzip.open(raw_path, "rt", encoding="utf-8") as f:
    for line in f:
        r = json.loads(line)
        countU[r["reviewerID"]] += 1
        countP[r["asin"]] += 1

usermap = {}
User = {}
usernum = 0
with gzip.open(raw_path, "rt", encoding="utf-8") as f:
    for line in f:
        r = json.loads(line)
        user = r["reviewerID"]
        item = r["asin"]
        timestamp = int(r["unixReviewTime"])
        if countU[user] < 5 or countP[item] < 5:
            continue
        if user not in usermap:
            usernum += 1
            usermap[user] = usernum
            User[usernum] = []
        User[usermap[user]].append([timestamp, item])

itemmap = {}
itemnum = 0
interactions = 0
with open(out_path, "w", encoding="utf-8") as out:
    for user in User:
        User[user].sort(key=lambda x: x[0])
        for _, item in User[user]:
            if item not in itemmap:
                itemnum += 1
                itemmap[item] = itemnum
            out.write(f"{user} {itemmap[item]}\n")
            interactions += 1

print(f"wrote {out_path}")
print(f"users={usernum}, items={itemnum}, interactions={interactions}")

对于Fashion数据集，我说了The Fashion dataset has asin "B0002Z1JNK", see data/fashion/get_item_embedding.ipynb, and this "B0002Z1JNK" occurs only in Fashion 2018 version. 请在amazon2018找到Amazon Fashionreviews下载，然后执行下面的代码，得到的fashion_inter.txt与LLMESR作者给出的zip包中的inter完全相同：

import gzip
import json
from collections import defaultdict

raw_path = "AMAZON_FASHION.json.gz"
out_path = "fashion_inter.txt"

user_items = {}
with gzip.open(raw_path, "rt", encoding="utf-8") as f:
    for line in f:
        r = json.loads(line)
        if float(r["overall"]) <= 0.0:
            continue
        user = r["reviewerID"]
        item = r["asin"]
        timestamp = int(r["unixReviewTime"])
        if user not in user_items:
            user_items[user] = []
        user_items[user].append((item, timestamp))

for user in user_items:
    user_items[user].sort(key=lambda x: x[1])
    user_items[user] = [item for item, _ in user_items[user]]

user_count = defaultdict(int)
item_count = defaultdict(int)
for user, items in user_items.items():
    for item in items:
        user_count[user] += 1
        item_count[item] += 1

ok = False
while not ok:
    ok = True
    for num in user_count.values():
        if num < 3:
            ok = False
            break
    if ok:
        for num in item_count.values():
            if num < 2:
                ok = False
                break
    if ok:
        break

    for user in user_count:
        if user_count[user] < 3:
            user_items.pop(user)
        else:
            for item in user_items[user]:
                if item_count[item] < 2:
                    user_items[user].remove(item)

    user_count = defaultdict(int)
    item_count = defaultdict(int)
    for user, items in user_items.items():
        for item in items:
            user_count[user] += 1
            item_count[item] += 1

user2id = {}
item2id = {}
next_user_id = 1
next_item_id = 1

with open(out_path, "w", encoding="utf-8") as out:
    for user, items in user_items.items():
        if user not in user2id:
            user2id[user] = next_user_id
            next_user_id += 1
        uid = user2id[user]
        for item in items:
            if item not in item2id:
                item2id[item] = next_item_id
                next_item_id += 1
            out.write(f"{uid} {item2id[item]}\n")

print(f"wrote {out_path}")
print(f"users={len(user2id)}, items={len(item2id)}, interactions={sum(user_count.values())}")

[lollipopmark]

你好，我想问问针对yelp数据集的inter.txt文件的数据量15720条是通过何种方式处理得来的，我利用项目的原始处理代码进行过滤处理，无法得到与论文相同的预处理数据规模

你好，我不是作者，但我复现了，方法是：
这个yelp是2013年的RecSys2013: Yelp Business Rating Prediction RecSys Challenge 2013: Yelp business rating prediction中给出的数据集（当然，也可能有其他出处，但我这个能完全复现作者的数据）。请在kaggle下载，然后找到yelp-recsys-2013/yelp_training_set/yelp_training_set/yelp_training_set_review.json这个文件，接着执行下面的代码得到的yelp_inter.txt与LLMESR作者给出的zip包中的inter完全相同：

import json
from collections import defaultdict

raw_path = "yelp-recsys-2013/yelp_training_set/yelp_training_set/yelp_training_set_review.json"
out_path = "yelp_inter.txt"

rows = []
with open(raw_path, "r", encoding="utf-8") as f:
    for line in f:
        r = json.loads(line)
        date = r["date"]
        if date < "2000-01-01 00:00:00" or date > "2019-12-31 00:00:00":
            continue
        if float(r["stars"]) <= 0.0:
            continue
        time = int(date.replace("-", "").replace(":", "").replace(" ", ""))
        rows.append((r["user_id"], r["business_id"], time))

countU = defaultdict(int)
countP = defaultdict(int)
for user, item, _ in rows:
    countU[user] += 1
    countP[item] += 1

User = {}
for user, item, time in rows:
    if countU[user] < 3 or countP[item] < 3:
        continue
    if user not in User:
        User[user] = []
    User[user].append((item, time))

user2id = {}
item2id = {}
user_id = 1
item_id = 1
interactions = 0

with open(out_path, "w", encoding="utf-8") as out:
    for user, item_time in User.items():
        item_time.sort(key=lambda x: x[1])
        items = [item for item, _ in item_time]
        if user not in user2id:
            user2id[user] = user_id
            user_id += 1
        uid = user2id[user]
        mapped_items = []
        for item in items:
            if item not in item2id:
                item2id[item] = item_id
                item_id += 1
            mapped_items.append(item2id[item])
        if len(mapped_items) < 3:
            continue
        for item in mapped_items:
            out.write(f"{uid} {item}\n")
            interactions += 1

print(f"wrote {out_path}")
print(f"users={len(user2id)}, items={len(item2id)}, interactions={interactions}")

[lollipopmark]

@ZhuYiZ-1205 @ninghaolu @23fu-max 虽已过去很久，但欢迎你们按我所说的试试。

[lollipopmark]

@liuqidong07 也请作者确认下我给出的代码，若无误，建议Close this issue.

[liuqidong07]

Dear @lollipopmark , I'd like to express my sincere thanks for your efforts in reproducing preprocessing the dataset. First, I'm sorry for missing this part of the code, as it's left on servers at Tencent, which I cannot access now. Second, I'm sorry to hear about the difficulties you had with your paper submission to 华东师范大学学报. If possible, you could rebuttal for your paper and cc me (E-mail: liuqidong@xjtu.edu.cn). I'm very willing to make a defense for you. Finally, thank you very much, and I'll attach your code to the README of the repo.