# Customs Creators Collective archive tool

The Customs Creators Collective archive tool allows for easy scraping to a local JSON database and downloading of
files from the C3 (Customs Creators Collective) database, a collection of custom songs for Rock Band and similar
clone games.

This tool exists because the C3DB is very hard to mass download from: each song must be found in the extensive
list, selected manually, and a second link clicked through, before a random file name is obtained. This tool
simplifies the process by first collecting information about all available songs of a particular type, and then
is able to download songs based on customizable filters (e.g. by genre, artist, author, etc.) and output them in
a standardized format.

To use the tool, first use the "database" command to build or modify your local JSON database, then use the
"download" command to download songs.

## Installation

### `pip`

1. Use `pip3 install .` to install the package to a virtualenv or your system Python. The tool will be available
as `c3dbdl` in your shell.

### Manual

1. Install the Python3 requirements from `requirements.txt`.

1. Copy the file `c3dbdl/c3dbdl.py` to somewhere in your `$PATH`. You can optionally remove the `.py` if you
with for command compatibility with a `pip` installation.

## Usage

Before running a command, use the build-in help via the `-h`/`--help` option to view the available option(s) of
the command.

The general process of using `c3dbdl` is as follows:

1. Select a download location, and either specify it with the `-d`/`--download-directory` option or via the
environment variable `C3DBDL_DOWNLOAD_DIRECTORY`.

1. Select a base URL. Use this to determine what game(s) you want to want to limit to, or use the default to
fetch all avilable songs for all games, and either specify it with the `-u`/`--base-url` option or via the
environment variable `C3DBDL_BASE_URL`.

1. Initialize your C3DB JSON database with `c3dbdl [options] database build`. This will take a fair amount
of time to complete as all pages of the chosen base URL, and all song pages (30,000+) are scanned. Note that if
you cancel this process, no data will be saved, so let it complete! The default concurrency setting should make
this relatively quick but YMMV.

1. Download any song(s) you want with `c3dbdl [options] download [options]`.

## Filtering

Filtering out the songs in the database is a key part of this tool. You might want to be able to grab only select
genres, artists, authors, etc. to make your custom song packs.

`c3dbdl` is able to filter by several key categories:

* `genre`: The genre of the song.
* `artist`: The artist of the song.
* `album`: The album of the song.
* `title`: The title of the song.
* `year`: The year of the album/song.
* `author`: The author of the file on C3DB.

Note that we *cannot* filter - mostly for parsing difficulty reasons - by intrument type or difficulty, by song
length, or by any other information not mentioned above.

Filtering is always done during the download stage; the JSON database will always contain all possible entries.

To use filters, append one or more `--filter` options to your `c3dbdl download` command. A filter option begins
with the literal `--filter`, followed by the category (e.g. `genre` or `artist`), then finally the text to filter
on, for instance `Rock` or `Santana` or `2012`. The text must be quoted if it contains whitespace.

If more that one filter is specified, they are treated as a logical AND, i.e. all the listed filters must apply to
a given song for it to be downloaded in that run.

Filters allow powerfully specific download selections to be run. For example, let's look for all songs by Rush
from the album Vapor Trails (the remixed version) authored by ejthedj:

```
c3dbdl download --filter artist Rush --filter album "Vapor Trails [Remixed]" --author ejthedj
```

This shouldfind , as of 2023-04-02, exactly one song, "Sweet Miracle":

```
Found 28942 songs from JSON database file 'Downloads/c3db.json'
Downloading 1 song files...
Downloading song "Rush - Sweet Miracle" by ejthedj...
Downloading from https://dl.c3universe.com/s/ejthedj/sweetMiracle...
```

In addition to the above filters, within each song may be more than one download link. To filter these links,
use the "-i"/"--download-id" and "-d"/"--download-descr" (see the help for details).

Feel free to experiment.

## Output Format

When downloading files, it may be advantageous to customize the output directory and filename structure to better
match what you plan to do with the files. For instance, for pure organiation you might want nicely laid out
files with clear directory structures and names, while for Onyx packaging you might want everything in a flat
directory.

`c3dbdl` provides complete flexibility in the output file format. When downloading, use the `--file-structure`
option to set the file structure. This value is an interpolated string containing one or more field variables,
which are mapped at download file. The available fields are:

* `genre`: The genre of the song.
* `artist`: The artist of the song.
* `album`: The album of the song.
* `title`: The title of the song.
* `year`: The year of the album/song.
* `author`: The author of the file on C3DB.
* `orig_file`: The original filename that would be downloaded by e.g. a browser.

The default structure leverages all of these options to create an archive-ready structure as follows:

```
{genre}/{author}/{artist}/{album}/{title} [{year}].{orig_file}
```

As an example:

```
Prog/Rush/Vapor Trails [Remixed]/Sweet Miracle [2002] (ejthedj).sweetMiracle
```

Note that any parent director(ies) will be automatically created down the whole tree until the final filename.

## Help

This is a quick and dirty tool I wrote to quickly grab collections of songs. I provide no guarantee of success
when using this tool. If you have issues, please open an issue on this repository and provide *full details*
of your problem.