129 lines
5.7 KiB
Markdown
129 lines
5.7 KiB
Markdown
# C3DB Download Tool
|
|
|
|
The C3DB Download Tool allows for easy scraping to a local JSON database and downloading of files from the C3
|
|
(Customs Creators Collective) database, a collection of custom songs for Guitar Hero, Rock Band, and similar clone
|
|
games.
|
|
|
|
This tool exists because the C3DB is very hard to mass download from: each song must be found in the extensive
|
|
list, selected manually, and a second link clicked through, before a random file name is obtained. This tool
|
|
simplifies the process by first collecting information about all available songs of a particular type, and then is
|
|
able to download songs based on customizable filters (e.g. by genre, artist, author, etc.) and output them in a
|
|
standardized format.
|
|
|
|
To use the tool, first use the "database" command to build or modify your local JSON database, then use the
|
|
"download" command to download songs.
|
|
|
|
To avoid overloading or abusing the C3DB website, this tool operates exclusively in sequential mode by design; at
|
|
most one page is scraped (for "database build") or song downloaded (for "download") at once. Additionally, the tool
|
|
design ensures that the JSON database of songs is stored locally, so it only needs to be built once and then is
|
|
reused to perform actual downloads without putting further load on the website.
|
|
|
|
## Installation
|
|
|
|
1. Install the Python3 requirements from `requirements.txt`.
|
|
|
|
1. Copy the script to a virtualenv, somewhere in your $PATH or execute directly from this folder (see Usage below).
|
|
|
|
## Usage
|
|
|
|
Before running a command, use the build-in help via the `-h`/`--help` option to view the available option(s) of
|
|
the command.
|
|
|
|
The general process of using `c3dbdl` is as follows:
|
|
|
|
1. Select a download location, and either specify it with the `-d`/`--download-directory` option or via the
|
|
environment variable `C3DBDL_DOWNLOAD_DIRECTORY`.
|
|
|
|
1. Select a base URL. Use this to determine what game(s) you want to want to limit to, or use the default to
|
|
fetch all avilable songs for all games, and either specify it with the `-u`/`--base-url` option or via the
|
|
environment variable `C3DBDL_BASE_URL`.
|
|
|
|
1. Initialize your C3DB JSON database with `c3dbdl [options] database build`. This will take a fair amount
|
|
of time to complete as all pages of the chosen base URL are scanned. Note that if you cancel this process, no
|
|
data will be saved, so let it complete!
|
|
|
|
1. Download any song(s) you want with `c3dbdl [options] download [options]`.
|
|
|
|
## Filtering
|
|
|
|
Filtering out the songs in the database is a key part of this tool. You might want to be able to grab only select
|
|
genres, artists, authors, etc. to make your custom song packs.
|
|
|
|
`c3dbdl` is able to filter by several key categories:
|
|
|
|
* `genre`: The genre of the song.
|
|
* `artist`: The artist of the song.
|
|
* `album`: The album of the song.
|
|
* `title`: The title of the song.
|
|
* `year`: The year of the album/song.
|
|
* `author`: The author of the file on C3DB.
|
|
|
|
Note that we *cannot* filter - mostly for parsing difficulty reasons - by intrument type or difficulty, by song
|
|
length, or by any other information not mentioned above.
|
|
|
|
Filtering is always done during the download stage; the JSON database will always contain all possible entries.
|
|
|
|
To use filters, append one or more `--filter` options to your `c3dbdl download` command. A filter option begins
|
|
with the literal `--filter`, followed by the category (e.g. `genre` or `artist`), then finally the text to filter
|
|
on, for instance `Rock` or `Santana` or `2012`. The text must be quoted if it contains whitespace.
|
|
|
|
If more that one filter is specified, they are treated as a logical AND, i.e. all the listed filters must apply to
|
|
a given song for it to be downloaded in that run.
|
|
|
|
Filters allow powerfully specific download selections to be run. For example, let's look for all songs by Rush
|
|
from the album Vapor Trails (the remixed version) authored by ejthedj:
|
|
|
|
```
|
|
c3dbdl download --filter artist Rush --filter album "Vapor Trails [Remixed]" --author ejthedj
|
|
```
|
|
|
|
This shouldfind , as of 2023-04-02, exactly one song, "Sweet Miracle":
|
|
|
|
```
|
|
Found 28942 songs from JSON database file 'Downloads/c3db.json'
|
|
Downloading 1 song files...
|
|
Downloading song "Rush - Sweet Miracle" by ejthedj...
|
|
Downloading from https://dl.c3universe.com/s/ejthedj/sweetMiracle...
|
|
```
|
|
|
|
Feel free to experiment.
|
|
|
|
## Output Format
|
|
|
|
When downloading files, it may be advantageous to customize the output directory and filename structure to better
|
|
match what you plan to do with the files. For instance, for pure organiation you might want nicely laid out
|
|
files with clear directory structures and names, while for Onyx packaging you might want everything in a flat
|
|
directory.
|
|
|
|
`c3dbdl` provides complete flexibility in the output file format. When downloading, use the `--file-structure`
|
|
option to set the file structure. This value is an interpolated string containing one or more field variables,
|
|
which are mapped at download file. The available fields are:
|
|
|
|
* `genre`: The genre of the song.
|
|
* `artist`: The artist of the song.
|
|
* `album`: The album of the song.
|
|
* `title`: The title of the song.
|
|
* `year`: The year of the album/song.
|
|
* `author`: The author of the file on C3DB.
|
|
* `orig_file`: The original filename that would be downloaded by e.g. a browser.
|
|
|
|
The default structure leverages all of these options to create an archive-ready structure as follows:
|
|
|
|
```
|
|
{genre}/{artist}/{album}/{title} [{year}] ({author}).{orig_file}
|
|
```
|
|
|
|
As an example:
|
|
|
|
```
|
|
Prog/Rush/Vapor Trails [Remixed]/Sweet Miracle [2002] (ejthedj).sweetMiracle
|
|
```
|
|
|
|
Note that any parent director(ies) will be automatically created down the whole tree until the final filename.
|
|
|
|
## Help
|
|
|
|
This is a quick and dirty tool I wrote to quickly grab collections of songs. I provide no guarantee of success
|
|
when using this tool. If you have issues, please open an issue on this repository and provide *full details*
|
|
of your problem.
|