joshuaboniface/c3dbdl
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity

Add support for filtering on instrument parts

Browse Source
This commit is contained in:
Joshua Boniface
2023-04-07 05:22:57 -04:00
parent cb1a5c5d58
commit 50db6daf55
2 changed files with 235 additions and 78 deletions
Download Patch File Download Diff File Expand all files Collapse all files

81
README.md
View File

@ -109,7 +109,7 @@ Once a database has been built, you can start searching for downloading songs.
To search for songs, use the `search` command. This command takes `--filter` arguments in order to show what
song(s) would be downloaded by a given filter, without actually triggering the download. Once you have a valid
filter from a search, you can use it to download.
filter from a search, you can use it to `download` precisely the song(s) you want.
To download songs, use the `download` command. See the following sections for more details on the specifics of
the filters and output formatting of the `download` command.
@ -124,9 +124,17 @@ not overwrite it and will simply skip downloading the file.
### Filtering
Filtering out the songs in the database is a key part of this tool. You might want to be able to grab only select
genres, artists, authors, etc. to make your custom song packs.
genres, artists, authors, etc. to make your custom song packs, or songs with particular instruments.
`c3dbdl` is able to filter by several key categories:
If multiple filters, of either type, are specified, they are treated as a logical AND, i.e. all the listed
filters, both information and instrument, must apply to a given song for it to be matched.
Filtering is always done during the search/download stage; the JSON database will always contain all possible
entries from the C3DB.
#### Information Filters
`c3dbdl` is able to filter songs by their general information in several key categories:
* `genre`: The genre of the song.
* `artist`: The artist of the song.
@ -135,35 +143,58 @@ genres, artists, authors, etc. to make your custom song packs.
* `year`: The year of the album/song.
* `author`: The author of the file on C3DB.
Note that we *cannot* filter - mostly for parsing difficulty reasons - by intrument type or difficulty, by song
length, or by any other information not mentioned above.
To use information filters, append one or more `--filter` options to your `c3dbdl search` or `download` command. An
information filter option begins with the literal `--filter`, followed by the field (e.g. `genre` or `artist`), then
finally the text value to filter on, for instance `Rock` or `Santana` or `2012`. The text must be quoted if it
contains any whitespace.
Filtering is always done during the search/download stage; the JSON database will always contain all possible entries.
Information filter values are fuzzy. They are case insensitive, and use the `in` construct. So, for example, the
filter string `--filter song "edmund fitzgerald"` would match the song title "The Wreck of the Edmund Fitzgerald".
To use filters, append one or more `--filter` options to your `c3dbdl download` or `search` command. A filter option
begins with the literal `--filter`, followed by the category (e.g. `genre` or `artist`), then finally the text to
filter on, for instance `Rock` or `Santana` or `2012`. The text must be quoted if it contains whitespace.
If more that one filter is specified, they are treated as a logical AND, i.e. all the listed filters must apply to
a given song for it to be downloaded in that run.
Filter values are fuzzy. They are case insensitive, and use the `in` construct. So, for example, the filter string
`--filter song "edmund fitzgerald"` would match the song title "The Wreck of the Edmund Fitzgerald".
Filters allow very specific download selections to be run. For example, let's look for all songs by Rush
from the album Vapor Trails (the remixed version) authored by ejthedj:
For example, to find all songs by Rush from the album Vapor Trails (the remixed version) authored by ejthedj:
```
c3dbdl download --filter artist Rush --filter album "Vapor Trails [Remixed]" --author ejthedj
c3dbdl search --filter artist Rush --filter album "Vapor Trails [Remixed]" --filter author ejthedj
Found 19563 songs from JSON database file 'Downloads/c3db.json'
Downloading 1 song files...
> Downloading song "Rush - Sweet Miracle" by ejthedj...
Downloading file "Rock Band 3 Xbox 360" from https://dl.c3universe.com/s/ejthedj/sweetMiracle...
Successfully downloaded to ../Prog/ejthedj/Rush/Vapor Trails [Remixed]/Sweet Miracle [2002].sweetMiracle
Found 1 matching songs:
> Song: "Rush - Sweet Miracle" from "Vapor Trails [Remixed] (2002)" by ejthedj
Instruments: guitar [2], bass [3], drums [4], vocals [4], keys [None]
Available downloads:
* Rock Band 3 Xbox 360
```
In this case, one song matched and was downloaded. Feel free to experiment with the various filters to find
exactly what you're looking for.
In this case, one song matched; applying the same filter to a `download` would thus download only the single song.
#### Instrument Filters
In addition to the information filters, `c3dbdl` can also filter by available instrument parts. There are 5 valid
instruments that can be filtered on:
* `guitar`
* `bass`
* `drums`
* `vocals`
* `keys`
To use instrument filters, append one or more `--instrument-filter` options to your `c3dbdl search` or `download`
command. An instrument filter option begins with the literal `--instrument-filter`, followed by the instrument
you wish to filter on.
If a part contains the instrument at any difficulty (from 0-6), it will match the filter; if the instrument part
is missing, it will not match.
You can also invert the match by adding `no-` to the instrument name. So `--instrument-filter no-keys` would
only match songs *without* a keys part.
For example, to find all songs by Rush which have a keys part:
```
```
If this case, X songs matched (only one shown for simplicity); applying the same filter to a `download` would
thus download all X songs.
### Output Format

202
c3dbdl/c3dbdl.py
View File

@ -170,7 +170,6 @@ def fetchSongData(entries):
song_entry["dl_links"] = dl_links
# Return messages and song entry
print(song_entry)
return messages, song_entry
@ -500,6 +499,16 @@ def database():
nargs=2,
help="Add a filter option.",
)
@click.option(
"-s",
"--instrument-filter",
"_instrument_filters",
envvar="C3DBDL_DL_INSTFILTERS",
default=[],
multiple=True,
nargs=1,
help="Add an instrument filter."
)
@click.option(
"-l",
"--limit",
@ -524,10 +533,24 @@ def database():
default=None,
help='Download only "dl_links" entries with this in their description (fuzzy).',
)
def download(_filters, _id, _desc, _limit, _file_structure):
def download(_filters, _instrument_filters, _id, _desc, _limit, _file_structure):
"""
Download song(s) from the C3DB webpage.
Filters allow granular selection of the song(s) to download. Multiple filters can be
specified, and a song is selected only if ALL filters match (logical AND). Filters are
specified in the form "--filter <database_key> <value>" for information filters, or
"--instrument-filter [no-]<instrument>" for instrument filters.
For a full list of and explanation for filters, see the "search" command help
(command "c3dbdl search --help").
In addition to filters, each song may have more than one download link, to provide
multiple versions of the same song (for example, normal and multitracks, or alternate
charts). For each song, the "-i"/"--download-id" and "-d"/"--download-descr" options
can help filter these out, or both can be left blank to download all possible files
for a given song.
\b
The output file structure can be specified as a path format with any of the following
fields included, surrounded by curly braces:
@ -543,28 +566,6 @@ def download(_filters, _id, _desc, _limit, _file_structure):
The default output file structure is:
"{artist}/{album}/{title}.{author}.{orig_name}"
Filters allow granular selection of the song(s) to download. Multiple filters can be
specified, and a song is selected only if ALL filters match (logical AND). Each filter
is in the form "--filter [database_key] [value]".
The valid "database_key" values are identical to the output file fields above, except
for "orig_name".
\b
For example, to download all songs in the genre "Rock":
--filter genre Rock
\b
Or to download all songs by the artist "Rush" and the author "MyName":
--filter artist Rush --filter author MyName
In addition to filters, each song may have more than one download link, to provide
multiple versions of the same song (for example, normal and multitracks, or alternate
charts). For each song, the "-i"/"--download-id" and "-d"/"--download-descr" options
can help filter these out, or both can be left blank to download all possible files
for a given song. Mostly useful when being extremely restrictive with filters, less
so when downloading many songs at once.
\b
The following environment variables can be used for scripting purposes:
* C3DBDL_DL_FILE_STRUCTURE: equivalent to "--file-structure"
@ -581,18 +582,47 @@ def download(_filters, _id, _desc, _limit, _file_structure):
pending_songs = list()
for song in all_songs:
if len(_filters) < 1:
if len(_filters) < 1 and len(_instrument_filters) < 1:
add_to_pending = True
else:
# Parse the information filters
if len(_filters) > 0:
try:
pending_filters = [
pending_information_filters = [
_filter[1].lower() in song[_filter[0]].lower()
for _filter in _filters
]
add_to_pending = all(pending_filters)
information_add_to_pending = all(pending_information_filters)
except KeyError as e:
click.echo(f"Invalid filter field {e}")
exit(1)
else:
information_add_to_pending = True
# Parse the instrument filters
if len(_instrument_filters) > 0:
try:
pending_instrument_filters = list()
for instrument_filter in _instrument_filters:
if re.match("^no-", instrument_filter):
instrument_filter = instrument_filter.replace('no-', '')
if song["instruments"][instrument_filter] is None:
pending_instrument_filters.append(True)
else:
pending_instrument_filters.append(False)
else:
if song["instruments"][instrument_filter] is not None:
pending_instrument_filters.append(True)
else:
pending_instrument_filters.append(False)
instrument_add_to_pending = all(pending_instrument_filters)
except KeyError as e:
click.echo(f"Invalid instrument value {e}")
exit(1)
else:
instrument_add_to_pending = True
add_to_pending = all([information_add_to_pending, instrument_add_to_pending])
if add_to_pending:
pending_songs.append(song)
@ -600,7 +630,7 @@ def download(_filters, _id, _desc, _limit, _file_structure):
if _limit is not None:
pending_songs = pending_songs[0:_limit]
click.echo(f"Downloading {len(pending_songs)} song files...")
click.echo(f"Downloading {len(pending_songs)} songs...")
for song in pending_songs:
downloadSong(config["download_directory"], _file_structure, song, _id, _desc)
@ -615,22 +645,78 @@ def download(_filters, _id, _desc, _limit, _file_structure):
default=[],
multiple=True,
nargs=2,
help="Add a filter option.",
help="Add an information filter.",
)
def search(_filters):
@click.option(
"-s",
"--instrument-filter",
"_instrument_filters",
envvar="C3DBDL_DL_INSTFILTERS",
default=[],
multiple=True,
nargs=1,
help="Add an instrument filter."
)
def search(_filters, _instrument_filters):
"""
Search for song(s) from the C3DB local database.
Filters allow granular selection of the song(s) to download. Multiple filters can be
specified, and a song is selected only if ALL filters match (logical AND). Each filter
is in the form "--filter [database_key] [value]".
specified, and a song is selected only if ALL filters match (logical AND). Filters are
specified in the form "--filter <database_key> <value>" for information filters, or
"--instrument-filter [no-]<instrument>" for instrument filters.
For a full list of and explanation for filters, see the "download" command help
(command "c3dbdl download --help").
Information filters match against the basic information of a song, for example finding
songs by a given artist, from a given album, by a given chart author, etc.
Filter values are fuzzy and case insensitive, so for example "word" would match
against a song titled "In The Word".
\b
The valid fields for the "<database_key>" value are:
* genre: The genre of the song.
* artist: The artist of the song.
* album: The album of the song.
* title: The title of the song.
* year: The year of the album/song.
* author: The author of the file on C3DB.
\b
For example, to download all songs in the genre "Rock":
--filter genre Rock
\b
Or to download all songs by the artist "Rush" and the author "MyName":
--filter artist Rush --filter author MyName
Instrument filters allow selection of the presence of instruments. If an instrument
fitler is given, only songs which contain parts for the given instrument(s) will be
shown.
\b
The valid instruments (case-sensitive, i.e. must be lowercase) are:
* guitar
* bass
* drums
* vocals
* keys
To negate an instrument filter, append "no-" to the instrument name.
\b
For example, to download only songs that have a keys part but no vocal part:
--instrument-filter keys --instrument-filter no-vocals
Note that while instrument difficulties are displayed in the output of this command,
they can not be filtered on; this is up to the user to do manually. The purpose of
instrument filters is to ensure that songs contain or don't contain given parts, not
to granularly select the difficulty of said parts (that's for the players of the game
to do, not us).
\b
The following environment variables can be used for scripting purposes:
* C3DBDL_DL_FILTERS: equivalent to "--filter"; limited to one instance
* C3DBDL_DL_INSTFILTERS: equivalent to "--instrument-filter"; limited to one instance
"""
with open(config["database_filename"], "r") as fh:
@ -642,18 +728,47 @@ def search(_filters):
pending_songs = list()
for song in all_songs:
if len(_filters) < 1:
if len(_filters) < 1 and len(_instrument_filters) < 1:
add_to_pending = True
else:
# Parse the information filters
if len(_filters) > 0:
try:
pending_filters = [
pending_information_filters = [
_filter[1].lower() in song[_filter[0]].lower()
for _filter in _filters
]
add_to_pending = all(pending_filters)
information_add_to_pending = all(pending_information_filters)
except KeyError as e:
click.echo(f"Invalid filter field {e}")
exit(1)
else:
information_add_to_pending = True
# Parse the instrument filters
if len(_instrument_filters) > 0:
try:
pending_instrument_filters = list()
for instrument_filter in _instrument_filters:
if re.match("^no-", instrument_filter):
instrument_filter = instrument_filter.replace('no-', '')
if song["instruments"][instrument_filter] is None:
pending_instrument_filters.append(True)
else:
pending_instrument_filters.append(False)
else:
if song["instruments"][instrument_filter] is not None:
pending_instrument_filters.append(True)
else:
pending_instrument_filters.append(False)
instrument_add_to_pending = all(pending_instrument_filters)
except KeyError as e:
click.echo(f"Invalid instrument value {e}")
exit(1)
else:
instrument_add_to_pending = True
add_to_pending = all([information_add_to_pending, instrument_add_to_pending])
if add_to_pending:
pending_songs.append(song)
@ -662,7 +777,18 @@ def search(_filters):
click.echo()
for entry in pending_songs:
click.echo(
f"""> "{entry['artist']} - {entry['title']}" from "{entry['album']} ({entry['year']})" by {entry['author']}"""
f"""> Song: "{entry['artist']} - {entry['title']}" from "{entry['album']} ({entry['year']})" by {entry['author']}"""
)
instrument_list = list()
for instrument in entry["instruments"]:
instrument_list.append(f"{instrument} [{entry['instruments'][instrument]}]")
click.echo(
f""" Instruments: {', '.join(instrument_list)}""",
)
click.echo(
f""" Available downloads:"""
)
for link in entry["dl_links"]:
click.echo(f""" * {link['description']}""")