Implementation

Implementation

Table of contents

File system structure
Backend storage
    data object
    session.json
Frontend structure
    Adding a new view
Subcorpora runs
    currRun object
    Creating a run
    Uploading metadata files
    Run report
Data management
    Collections and keyword lists
    Uploading a new corpus
    Past runs reports

File system structure

The file directory is organized as follows:

• build/: The folder /build/static will contain all of the JavaScript chunks that are compiled when you run npm run build. None of the files in this folder need to be edited.
• data/: We store all of our data here, uploaded through the web application or manually added. The folder data/corpus-files contains all the uploaded corpus files. The file session.json contains the current session and can be shared with others to instantiate the current session on another version of the application. The folder data/runs contains the files from the runs that we've done. The file run.json is a temporary file that contains the data for our current run (probably best to remove this intermediary data storing later).
• node_modules/: This folder contains all the code for our Node modules and packages. This folder automatically appears when you run npm install when you first download the codebase. You do not need to edit anything in this folder manually.
• public/: What the web application defaults to if the web application breaks. We honestly don’t use this folder too much.
• src/: This folder contains all the front-end React code for the application. All the components are in the containers folder.
• .gitignore: This file contains all the files that Git should ignore when pushing to the GitHub.
• index.js: The entry point for the web application. This contains the backend server code (such as GET and POST urls). All editing of backend functionality will be done in this file.
• package.json and package-lock.json: This doesn't need to be modified--just do npm install --save when installing dependencies and packages.
• README.md: This file contains our README for our Git repository.

Backend storage

We store the current state of our application into a data object, located in index.js. All updates to the collections, metadata files, keyword lists, and new and old runs are stored within data. The data object is then saved to data/session.json. We ensure that they are synced after each time the data object is updated using the function saveToSessionFile() in index.js.

session.json

When the application starts up, we read in the data/session.json file if one exists; otherwise, we create a blank one.

data object

We initialize the data object by loading whatever is in data/session.json into the object. The basic data object that we store in the backend contains keywords-lists, collections, and runs, which are self-explanatory. We will populate these objects as the user performs actions on the tool.

We also read through the collections folder, printing warning messages for collections that are in the session file but don't exist in the folder. We exclude these. See the function initializeData() in index.js.

Frontend structure

The basic layout of our web application is specified in src/containers/DefaultLayout/DefaultLayout.js.

Figure 1: Layout of the application.

The <AppBar></AppBar> is the bar across the top of our page. The <Drawer></Drawer> is the sidebar that the user can open and close (controlled by the functions handleDrawerOpen and handleDrawerClose. Each button in the sidebar navigation is contained within a <ListItem></ListItem>.

Our main view is within <main></main>, which has a bunch of different <Route></Route>s. We use exact paths to specify the URL.

Adding a new view

To add a new view, create a new component in src/containers, making sure that you have a package.json file within the component's folder. For example, the collections page is located in src/containers/Collections, which contains:

• Collections.js: the actual React.js code that specifies the view
• package.json: information about our component

Then, within src/DefaultLayout/DefaultLayout.js, add in a loadable at the top of the file. For the collections page, for example:

const Collections = loadable(() => import('../Collections'));

Then, add a route:

<Route exact path='/collections' component={Collections} />

This route would go to https://localhost:5000/collections, for example.

Subcorpora runs

When the user starts to run the subcorpora tool, they create a name and ID for it, select the collections and keyword lists they want to include, and select a metadata file for the run.

currRun object

All of this information is stored within the object currRun in index.js so we can keep track of the current run. A new currRun object is generated whenever the user creates a new run.

Creating a run

We see below the process of creating a run.

Figure 2: Communication between user, frontend, backend, and data objects.

Each time the user enters in their selection, it is passed from the frontend to the backend, and the data is then stored into the currRun object. Once all the information for the run is complete, the src/tool_script.py Python script is called, which then runs the tool, sending progress updates to the backend as it runs. As the backend receives these updates, it sends them over to the frontend, which notifies the user what's going on through a progress bar. Lastly, the currRun object is written into data/session.json, effectively saving the run. The user is then brought over to the run report.

Uploading metadata files

The user is given an option to upload a new metadata file. Once they click upload on the frontend, the POST request /upload_metadata is called in index.js. It creates a unique uploadFunction which uploads to the folder data/metadata-files using multer.

Run report

We have two different reports that are shown: the summary report and individual reports for each combination of keyword list and collection. Below, we see how the frontend view is structured. All of the view is stored in src/containers/Report/Report.js, and each child component is stored in a separate folder.

Figure 3: Structure of how the parent component communicates with switching between the individual and summary reports.

The Loading component is the only part of this view to appear until the progress is 100. Once it reaches 100, we know the Python script has finished running and that we can display the data. The Navigation component and the SummaryReport components appear, populated using data passed from the Report parent component. As the user clicks through the navigation, the view is switched accordingly.

Data management

Collections and keyword lists

The collections and keyword lists are stored within the backend data object. When the frontend displays the tables of data, they are pulled using a GET request in index.js. The user can then add, edit, and delete from this table, which updates data in the backend and sync the frontend to match the new data object.

Uploading a new corpus

The user uploads corpus files when they add a new collection. Once they click upload on the frontend, the POST request /upload_collection is called in index.js. It creates a unique uploadFunction which creates a new folder in data/corpus-files based on the collection ID given and uploads the files using multer.

Past runs reports

Similarly, the past runs are stored within a table and the delete functionality works the same way. Upon clicking on the report link, the currRun object in index.js is updated with the id of the clicked report. currRun.afterRun is updated to false (since we're going to the report directly from the "Past Runs" page and not after a run). We then use a redirect onto the report, a view located in src/containers/Report/PastRunsReport/PastRunsReport.js, which is very similar to the layout of src/containers/Report/Report.js.