Wip: CSV import #650
There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,83 @@ | ||
| import { FC } from 'react'; | ||
| import { ImportFormFilePage } from '../ImportFromFilePage'; | ||
| import { Trans } from '@lingui/macro'; | ||
| import { MainTitle } from '../../components/MainTitle'; | ||
|
|
||
| export const ImportFromCsvPage: FC = () => { | ||
| return ( | ||
| <> | ||
| <MainTitle | ||
| elements={[<Trans>Import</Trans>, <Trans>from CSV</Trans>]} | ||
| /> | ||
|
|
||
| <ImportFormFilePage | ||
| source="CSV" | ||
| fileType="text/csv" | ||
| itemTypes={['movie', 'tv']} | ||
| instructions={<div>{ | ||
| <CsvInstructions /> | ||
| }</div>} | ||
| /> | ||
| </> | ||
| ); | ||
| }; | ||
|
|
||
| const CsvInstructions: FC = () => { | ||
|
|
||
| //workaround because react and the ligui lib process escapes differently | ||
| const newline = '\\n'; | ||
|
|
||
| return ( | ||
| <details className="instructions"> | ||
| <summary className="instructions-summary"> | ||
| <Trans>Click to show CSV file requirements</Trans> | ||
| </summary> | ||
| <div className="instructions-content"> | ||
| <ul className="ml-8 list-decimal"> | ||
| <li><Trans>Comma delimited, plain UTF-8 files only</Trans></li> | ||
| <li><Trans>Quote and escape characters are optional but must use double quotes: "</Trans></li> | ||
| <li><Trans>Windows (\r{newline}), Linux ({newline}) and old macOS (\r) record delimiters are auto-detected</Trans></li> | ||
| <li><Trans>The first line MUST be column headers</Trans></li> | ||
| <li> | ||
| <Trans>Allowed column headers are, in any order and case-insensitive:</Trans><br/> | ||
| <code>type, imdbId, tmdbId, tvdbId, listId, rating, seen, season, episode</code> | ||
| </li> | ||
| <li><Trans>The only mandatory column is</Trans> <code>type</code></li> | ||
| <li> | ||
| <Trans>Valid values for <code>type</code> are:</Trans><br/> | ||
| <code>tv, movie</code> | ||
| </li> | ||
| <li> | ||
| <Trans>Other columns are optional, but you must include at least one of:</Trans><br/> | ||
| <code>tmdbId, imdbId, tvdbId</code> | ||
| </li> | ||
| <li><Trans>Leading and trailing whitespaces are stripped</Trans></li> | ||
| <li><Trans>Any record that cannot be parsed or contains errors will be skipped</Trans></li> | ||
| <li><Trans>Any record with missing fields compared to header will be skipped</Trans></li> | ||
| <li><Trans>List IDs must exist and be owned by the user</Trans></li> | ||
| <li><Trans>Items with invalid or other users list IDs are discarded</Trans></li> | ||
| <li><Trans>The watchlist list ID is found on the Lists page</Trans></li> | ||
| <li><Trans>Seen is a Y/N column only</Trans></li> | ||
| <li> | ||
| <Trans>Movies will be looked up in this order:</Trans><br /> | ||
| <code>tmdbId, imdbId</code> | ||
| </li> | ||
| <li> | ||
| <Trans>TV shows will be looked up in this order:</Trans><br /> | ||
| <code>tmdbId, imdbId, tvdbId</code> | ||
| </li> | ||
| <li><Trans>TV shows must only use the show's main ID from tvdb, tmdb, or imdb</Trans></li> | ||
| <li> | ||
| <Trans>To set episodes of a TV show as Seen, must provide a record for each:</Trans><br /> | ||
| <code>season</code> <Trans>and</Trans> <code>episode</code> | ||
| </li> | ||
| <li><Trans>Valid ratings are decimal values between 0.1 and 10.0</Trans></li> | ||
| <li> | ||
| <Trans>If you use "out of 5" ratings, multiply the value by 2</Trans><br/> | ||
| <Trans>Eg: for a rating of 4 out of 5, provide a value of 8</Trans> | ||
| </li> | ||
| </ul> | ||
| </div> | ||
| </details> | ||
| ); | ||
| }; |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,110 @@ | ||
| import { parse } from 'csv-parse/sync'; | ||
| import _ from 'lodash'; | ||
| import { record, z } from 'zod'; | ||
|
|
||
| import { ImportDataType, ImportListItem, ImportRatingItem, ImportSeenHistoryItem, ImportWatchlistItem } from '../repository/importRepository.js'; | ||
| import { listRepository } from '../repository/listRepository.js'; | ||
| import { mediaTypeSchema } from '../entity/mediaItemModel.js'; | ||
|
|
||
| export const csvImport = { | ||
| async map(user: number, csvData: string): Promise<ImportDataType> { | ||
| const data = csvImportSchema.parse( | ||
| parse(csvData, { | ||
| bom: true, //detects and removes any utf-8 byte order marks | ||
| delimiter: ',', //did you know the "C" in CSV stands for "comma"? :) | ||
| columns: header => header.map((column: string) => column.toLowerCase()), | ||
| skip_empty_lines: true, //blank lines are ignored | ||
| skip_records_with_error: true, //if any field parsing errors (eg: invalid numbers), skip entire record | ||
| trim: true //strip leading and trailing whitespace in fields | ||
| }) | ||
| ) | ||
| .filter(item => | ||
| (Object.values(mediaTypeSchema.Values).includes(item.type)) // sanity check | ||
| && ((item.type === 'tv' && (item.tmdbid || item.imdbid || item.tvdbid)) | ||
| ||(item.type === 'movie' && (item.tmdbid || item.imdbid))) | ||
| ); | ||
|
|
||
| const dateNow = new Date(); | ||
| const userLists = await listRepository.getLists({userId: user}); | ||
| const watchListId = userLists.find(userList => userList.isWatchlist)?.id; | ||
|
|
||
| const importLists = _(data) | ||
| .map((item) => item.listid) | ||
| .uniq() | ||
| .value(); | ||
|
ramebd marked this conversation as resolved.
|
||
|
|
||
| const importData: ImportDataType = { | ||
| ratings: data | ||
| .filter((item) => item.rating && item.rating > 0) | ||
| .map((item) => (<ImportRatingItem>{ | ||
| itemType: item.type, | ||
| tmdbId: item.tmdbid ? item.tmdbid : undefined, | ||
|
Owner
There was a problem hiding this comment.
Author
There was a problem hiding this comment. I had considered this, but a tmdbId value of 0 is invalid anyway, so ending up with undefined ensures no bodgy data is passed any further down the line to be queried against the TMDB API or stored in the database. In that case, it might even be better to explicitly fail the whole import on parsing errors where no (valid) ids are found for a given row, but that might be overkill 🙂 Querying tmdb api for a movie id of "0" returns a 404 with: |
||
| imdbId: item.imdbid ? item.imdbid : undefined, | ||
| tvdbId: item.tvdbid ? item.tvdbid : undefined, | ||
| rating: item.rating, | ||
| ratedAt: dateNow, | ||
|
Owner
There was a problem hiding this comment. either use the date provided by the user, or do not set date at all |
||
| episode: item.type === 'tv' ? { | ||
| seasonNumber: item.season ? item.season : undefined, | ||
| episodeNumber: item.episode ? item.episode : undefined | ||
| } : undefined | ||
| })), | ||
| watchlist: data | ||
| .filter((item) => item.listid && watchListId && item.listid == watchListId) | ||
| .map((item) => (<ImportWatchlistItem>{ | ||
| itemType: item.type, | ||
| tmdbId: item.tmdbid ? item.tmdbid : undefined, | ||
| imdbId: item.imdbid ? item.imdbid : undefined, | ||
| tvdbId: item.tvdbid ? item.tvdbid : undefined, | ||
| addedAt: dateNow | ||
| })), | ||
| seenHistory: data | ||
| .filter((item) => item.seen == 'Y') | ||
| .map((item) => (<ImportSeenHistoryItem>{ | ||
| itemType: item.type === 'tv' ? 'episode' : item.type, | ||
| tmdbId: item.tmdbid ? item.tmdbid : undefined, | ||
| imdbId: item.imdbid ? item.imdbid : undefined, | ||
| tvdbId: item.tvdbid ? item.tvdbid : undefined, | ||
| seenAt: dateNow, | ||
|
Owner
There was a problem hiding this comment. either use the date provided by the user, or do not set date at all |
||
| episode: item.type === 'tv' ? { | ||
| seasonNumber: item.season ? item.season : undefined, | ||
|
Owner
There was a problem hiding this comment. if
Author
There was a problem hiding this comment. Ahh yep, dang it, I missed that one 🤦♂️ |
||
| episodeNumber: item.episode ? item.episode : undefined | ||
| } : undefined | ||
| })), | ||
| lists: userLists | ||
| .filter((userList) => importLists.find(importList => | ||
| !userList.isWatchlist && importList == userList.id)) | ||
| .map((list) => (<ImportListItem>{ | ||
| name: list.name, | ||
| description: list.description, | ||
| traktId: list.traktId, | ||
|
ramebd marked this conversation as resolved.
|
||
| createdAt: dateNow, | ||
|
Owner
There was a problem hiding this comment. either use the date provided by the user, or do not set date at all |
||
| items: data | ||
| .filter((item) => item.listid == list.id) | ||
| .map((item) => ({ | ||
| itemType: item.type, | ||
| tmdbId: item.tmdbid ? item.tmdbid : undefined, | ||
| imdbId: item.imdbid ? item.imdbid : undefined, | ||
| tvdbId: item.tvdbid ? item.tvdbid : undefined, | ||
| addedAt: dateNow | ||
| })) | ||
| })) | ||
| } | ||
|
|
||
| return importData; | ||
| }, | ||
| }; | ||
|
|
||
| const csvImportSchema = z.array( | ||
| z.object({ | ||
| //lowercase every column name to allow case insensitive parsing | ||
| type: z.enum(['tv', 'movie']), | ||
| tmdbid: z.coerce.number().optional(), | ||
| imdbid: z.string().optional(), | ||
| tvdbid: z.coerce.number().optional(), | ||
| listid: z.coerce.number().optional(), | ||
| rating: z.coerce.number().optional(), | ||
| seen: z.string().optional(), | ||
| season: z.coerce.number().optional(), | ||
| episode: z.coerce.number().optional() | ||
| }) | ||
| ); | ||
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Uh oh!
There was an error while loading. Please reload this page.