136 lines
5.8 KiB
Markdown
136 lines
5.8 KiB
Markdown
# Customs Creators Collective archive tool
|
|
|
|
The Customs Creators Collective archive tool allows for easy scraping to a local JSON database and downloading of
|
|
files from the C3 (Customs Creators Collective) database, a collection of custom songs for Rock Band and similar
|
|
clone games.
|
|
|
|
This tool exists because the C3DB is very hard to mass download from: each song must be found in the extensive
|
|
list, selected manually, and a second link clicked through, before a random file name is obtained. This tool
|
|
simplifies the process by first collecting information about all available songs of a particular type, and then
|
|
is able to download songs based on customizable filters (e.g. by genre, artist, author, etc.) and output them in
|
|
a standardized format.
|
|
|
|
To use the tool, first use the "database" command to build or modify your local JSON database, then use the
|
|
"download" command to download songs.
|
|
|
|
## Installation
|
|
|
|
### `pip`
|
|
|
|
1. Use `pip3 install .` to install the package to a virtualenv or your system Python. The tool will be available
|
|
as `c3dbdl` in your shell.
|
|
|
|
### Manual
|
|
|
|
1. Install the Python3 requirements from `requirements.txt`.
|
|
|
|
1. Copy the file `c3dbdl/c3dbdl.py` to somewhere in your `$PATH`. You can optionally remove the `.py` if you
|
|
with for command compatibility with a `pip` installation.
|
|
|
|
## Usage
|
|
|
|
Before running a command, use the build-in help via the `-h`/`--help` option to view the available option(s) of
|
|
the command.
|
|
|
|
The general process of using `c3dbdl` is as follows:
|
|
|
|
1. Select a download location, and either specify it with the `-d`/`--download-directory` option or via the
|
|
environment variable `C3DBDL_DOWNLOAD_DIRECTORY`.
|
|
|
|
1. Select a base URL. Use this to determine what game(s) you want to want to limit to, or use the default to
|
|
fetch all avilable songs for all games, and either specify it with the `-u`/`--base-url` option or via the
|
|
environment variable `C3DBDL_BASE_URL`.
|
|
|
|
1. Initialize your C3DB JSON database with `c3dbdl [options] database build`. This will take a fair amount
|
|
of time to complete as all pages of the chosen base URL, and all song pages (30,000+) are scanned. Note that if
|
|
you cancel this process, no data will be saved, so let it complete! The default concurrency setting should make
|
|
this relatively quick but YMMV.
|
|
|
|
1. Download any song(s) you want with `c3dbdl [options] download [options]`.
|
|
|
|
## Filtering
|
|
|
|
Filtering out the songs in the database is a key part of this tool. You might want to be able to grab only select
|
|
genres, artists, authors, etc. to make your custom song packs.
|
|
|
|
`c3dbdl` is able to filter by several key categories:
|
|
|
|
* `genre`: The genre of the song.
|
|
* `artist`: The artist of the song.
|
|
* `album`: The album of the song.
|
|
* `title`: The title of the song.
|
|
* `year`: The year of the album/song.
|
|
* `author`: The author of the file on C3DB.
|
|
|
|
Note that we *cannot* filter - mostly for parsing difficulty reasons - by intrument type or difficulty, by song
|
|
length, or by any other information not mentioned above.
|
|
|
|
Filtering is always done during the download stage; the JSON database will always contain all possible entries.
|
|
|
|
To use filters, append one or more `--filter` options to your `c3dbdl download` command. A filter option begins
|
|
with the literal `--filter`, followed by the category (e.g. `genre` or `artist`), then finally the text to filter
|
|
on, for instance `Rock` or `Santana` or `2012`. The text must be quoted if it contains whitespace.
|
|
|
|
If more that one filter is specified, they are treated as a logical AND, i.e. all the listed filters must apply to
|
|
a given song for it to be downloaded in that run.
|
|
|
|
Filters allow powerfully specific download selections to be run. For example, let's look for all songs by Rush
|
|
from the album Vapor Trails (the remixed version) authored by ejthedj:
|
|
|
|
```
|
|
c3dbdl download --filter artist Rush --filter album "Vapor Trails [Remixed]" --author ejthedj
|
|
```
|
|
|
|
This shouldfind , as of 2023-04-02, exactly one song, "Sweet Miracle":
|
|
|
|
```
|
|
Found 28942 songs from JSON database file 'Downloads/c3db.json'
|
|
Downloading 1 song files...
|
|
Downloading song "Rush - Sweet Miracle" by ejthedj...
|
|
Downloading from https://dl.c3universe.com/s/ejthedj/sweetMiracle...
|
|
```
|
|
|
|
In addition to the above filters, within each song may be more than one download link. To filter these links,
|
|
use the "-i"/"--download-id" and "-d"/"--download-descr" (see the help for details).
|
|
|
|
Feel free to experiment.
|
|
|
|
## Output Format
|
|
|
|
When downloading files, it may be advantageous to customize the output directory and filename structure to better
|
|
match what you plan to do with the files. For instance, for pure organiation you might want nicely laid out
|
|
files with clear directory structures and names, while for Onyx packaging you might want everything in a flat
|
|
directory.
|
|
|
|
`c3dbdl` provides complete flexibility in the output file format. When downloading, use the `--file-structure`
|
|
option to set the file structure. This value is an interpolated string containing one or more field variables,
|
|
which are mapped at download file. The available fields are:
|
|
|
|
* `genre`: The genre of the song.
|
|
* `artist`: The artist of the song.
|
|
* `album`: The album of the song.
|
|
* `title`: The title of the song.
|
|
* `year`: The year of the album/song.
|
|
* `author`: The author of the file on C3DB.
|
|
* `orig_file`: The original filename that would be downloaded by e.g. a browser.
|
|
|
|
The default structure leverages all of these options to create an archive-ready structure as follows:
|
|
|
|
```
|
|
{genre}/{author}/{artist}/{album}/{title} [{year}].{orig_file}
|
|
```
|
|
|
|
As an example:
|
|
|
|
```
|
|
Prog/Rush/Vapor Trails [Remixed]/Sweet Miracle [2002] (ejthedj).sweetMiracle
|
|
```
|
|
|
|
Note that any parent director(ies) will be automatically created down the whole tree until the final filename.
|
|
|
|
## Help
|
|
|
|
This is a quick and dirty tool I wrote to quickly grab collections of songs. I provide no guarantee of success
|
|
when using this tool. If you have issues, please open an issue on this repository and provide *full details*
|
|
of your problem.
|