Add support for filtering on instrument parts

.flake8

@@ -5,5 +5,6 @@
 #   * E203 (whitespace before ':'): Black recommends this as disabled
 ignore = W503, E501
 extend-ignore = E203
+exclude = build/
 # Set the max line length to 88 for Black
 max-line-length = 88

README.md

@@ -109,7 +109,7 @@ Once a database has been built, you can start searching for downloading songs.
 
 To search for songs, use the `search` command. This command takes `--filter` arguments in order to show what
 song(s) would be downloaded by a given filter, without actually triggering the download. Once you have a valid
-filter from a search, you can use it to download.
+filter from a search, you can use it to `download` precisely the song(s) you want.
 
 To download songs, use the `download` command. See the following sections for more details on the specifics of
 the filters and output formatting of the `download` command.
@@ -124,9 +124,17 @@ not overwrite it and will simply skip downloading the file.
 ### Filtering
 
 Filtering out the songs in the database is a key part of this tool. You might want to be able to grab only select
-genres, artists, authors, etc. to make your custom song packs.
+genres, artists, authors, etc. to make your custom song packs, or songs with particular instruments.
 
-`c3dbdl` is able to filter by several key categories:
+If multiple filters, of either type, are specified, they are treated as a logical AND, i.e. all the listed
+filters, both information and instrument, must apply to a given song for it to be matched.
+
+Filtering is always done during the search/download stage; the JSON database will always contain all possible
+entries from the C3DB.
+
+#### Information Filters
+
+`c3dbdl` is able to filter songs by their general information in several key categories:
 
 * `genre`: The genre of the song.
 * `artist`: The artist of the song.
@@ -135,35 +143,58 @@ genres, artists, authors, etc. to make your custom song packs.
 * `year`: The year of the album/song.
 * `author`: The author of the file on C3DB.
 
-Note that we *cannot* filter - mostly for parsing difficulty reasons - by intrument type or difficulty, by song
-length, or by any other information not mentioned above.
+To use information filters, append one or more `--filter` options to your `c3dbdl search` or `download` command. An
+information filter option begins with the literal `--filter`, followed by the field (e.g. `genre` or `artist`), then
+finally the text value to filter on, for instance `Rock` or `Santana` or `2012`. The text must be quoted if it
+contains any whitespace.
 
-Filtering is always done during the search/download stage; the JSON database will always contain all possible entries.
+Information filter values are fuzzy. They are case insensitive, and use the `in` construct. So, for example, the
+filter string `--filter song "edmund fitzgerald"` would match the song title "The Wreck of the Edmund Fitzgerald".
 
-To use filters, append one or more `--filter` options to your `c3dbdl download` or `search` command. A filter option
-begins with the literal `--filter`, followed by the category (e.g. `genre` or `artist`), then finally the text to
-filter on, for instance `Rock` or `Santana` or `2012`. The text must be quoted if it contains whitespace.
-
-If more that one filter is specified, they are treated as a logical AND, i.e. all the listed filters must apply to
-a given song for it to be downloaded in that run.
-
-Filter values are fuzzy. They are case insensitive, and use the `in` construct. So, for example, the filter string
-`--filter song "edmund fitzgerald"` would match the song title "The Wreck of the Edmund Fitzgerald".
-
-Filters allow very specific download selections to be run. For example, let's look for all songs by Rush
-from the album Vapor Trails (the remixed version) authored by ejthedj:
+For example, to find all songs by Rush from the album Vapor Trails (the remixed version) authored by ejthedj:
 
 ```
-c3dbdl download --filter artist Rush --filter album "Vapor Trails [Remixed]" --author ejthedj
+c3dbdl search --filter artist Rush --filter album "Vapor Trails [Remixed]" --filter author ejthedj
 Found 19563 songs from JSON database file 'Downloads/c3db.json'
-Downloading 1 song files...
-> Downloading song "Rush - Sweet Miracle" by ejthedj...
-Downloading file "Rock Band 3 Xbox 360" from https://dl.c3universe.com/s/ejthedj/sweetMiracle...
-Successfully downloaded to ../Prog/ejthedj/Rush/Vapor Trails [Remixed]/Sweet Miracle [2002].sweetMiracle
+Found 1 matching songs:
+
+> Song: "Rush - Sweet Miracle" from "Vapor Trails [Remixed] (2002)" by ejthedj
+  Instruments: guitar [2], bass [3], drums [4], vocals [4], keys [None]
+    Available downloads:
+        * Rock Band 3 Xbox 360
 ```
 
-In this case, one song matched and was downloaded. Feel free to experiment with the various filters to find
-exactly what you're looking for.
+In this case, one song matched; applying the same filter to a `download` would thus download only the single song.
+
+#### Instrument Filters
+
+In addition to the information filters, `c3dbdl` can also filter by available instrument parts. There are 5 valid
+instruments that can be filtered on:
+
+* `guitar`
+* `bass`
+* `drums`
+* `vocals`
+* `keys`
+
+To use instrument filters, append one or more `--instrument-filter` options to your `c3dbdl search` or `download`
+command. An instrument filter option begins with the literal `--instrument-filter`, followed by the instrument
+you wish to filter on.
+
+If a part contains the instrument at any difficulty (from 0-6), it will match the filter; if the instrument part
+is missing, it will not match.
+
+You can also invert the match by adding `no-` to the instrument name. So `--instrument-filter no-keys` would
+only match songs *without* a keys part.
+
+For example, to find all songs by Rush which have a keys part:
+
+```
+
+```
+
+If this case, X songs matched (only one shown for simplicity); applying the same filter to a `download` would
+thus download all X songs.
 
 ### Output Format
 

c3dbdl/c3dbdl.py

@@ -33,11 +33,24 @@ from concurrent.futures import ThreadPoolExecutor, as_completed
 CONTEXT_SETTINGS = dict(help_option_names=["-h", "--help"], max_content_width=120)
 
 
-def fetchSongData(entry):
-    song_entry = dict()
+def fetchSongData(entries):
+    song_entry = {
+        "artist": None,
+        "title": None,
+        "album": None,
+        "song_link": None,
+        "genre": None,
+        "year": None,
+        "length": None,
+        "author": None,
+        "instruments": dict(),
+        "dl_links": list(),
+    }
     messages = list()
+    found_instruments = False
 
-    for idx, td in enumerate(entry.find_all("td")):
+    # Find song details
+    for idx, td in enumerate(entries[0].find_all("td")):
         if idx == 2:
             # Artist
             song_entry["artist"] = td.find("a").get_text().strip().replace("/", "+")
@@ -74,6 +87,35 @@ def fetchSongData(entry):
             # Author (of chart)
             song_entry["author"] = td.find("a").get_text().strip().replace("/", "+")
 
+    # Find song instruments and difficulties
+    for idx, td in enumerate(entries[1].find_all("td")):
+        if (
+            len(list(td.find_all("div", attrs={"style": "width:110px;float:left"}))) > 0
+            and not found_instruments
+        ):
+            for instrument in td.find_all(
+                "div", attrs={"style": "width:110px;float:left"}
+            ):
+                difficulty_link = (
+                    instrument.find_all(
+                        "a", attrs={"style": "text-decoration: none;color:#000"}
+                    )[1]
+                    .get("href")
+                    .split("/")
+                )
+                instrument_name = (
+                    difficulty_link[-2].split("_")[-1].replace("prokeys", "keys")
+                )
+                instrument_diff = int(difficulty_link[-1])
+                if instrument_diff < 1:
+                    # No part
+                    instrument_difficulty = None
+                else:
+                    # Link difficulty - 1
+                    instrument_difficulty = instrument_diff - 1
+                song_entry["instruments"][instrument_name] = instrument_difficulty
+            found_instruments = True
+
     if (
         song_entry
         and song_entry["author"]
@@ -83,9 +125,6 @@ def fetchSongData(entry):
         messages.append(
             f"> Found song entry for {song_entry['artist']} - {song_entry['title']} by {song_entry['author']}"
         )
-        for entry_type in ["artist", "album", "genre", "year", "length"]:
-            if not song_entry[entry_type]:
-                song_entry[entry_type] = "None"
 
         # Get download links from the actual song page
         attempts = 1
@@ -130,7 +169,7 @@ def fetchSongData(entry):
             return None
         song_entry["dl_links"] = dl_links
 
-        # Append to the database
+        # Return messages and song entry
         return messages, song_entry
 
 
@@ -178,11 +217,22 @@ def buildDatabase(pages, concurrency):
             "tbody"
         )
 
+        # This is weird, but because of the table layout, there are two table rows for
+        # each song: the first is the song info, the second is the instruments
+        # So we must make a single "entry" that is a list of the two elements, then
+        # handle that later in fetchSongData.
         entries = list()
+        entry_idx = 0
+        entry_data = list()
         for entry in table_html.find_all("tr", attrs={"class": "odd"}):
             if len(entry) < 1:
                 break
-            entries.append(entry)
+            entry_data.append(entry)
+            entry_idx += 1
+            if entry_idx == 2:
+                entries.append(entry_data)
+                entry_idx = 0
+                entry_data = list()
 
         click.echo("Fetching and parsing song pages...")
         with ThreadPoolExecutor(max_workers=concurrency) as executor:
@@ -449,6 +499,16 @@ def database():
     nargs=2,
     help="Add a filter option.",
 )
+@click.option(
+    "-s",
+    "--instrument-filter",
+    "_instrument_filters",
+    envvar="C3DBDL_DL_INSTFILTERS",
+    default=[],
+    multiple=True,
+    nargs=1,
+    help="Add an instrument filter."
+)
 @click.option(
     "-l",
     "--limit",
@@ -473,10 +533,24 @@ def database():
     default=None,
     help='Download only "dl_links" entries with this in their description (fuzzy).',
 )
-def download(_filters, _id, _desc, _limit, _file_structure):
+def download(_filters, _instrument_filters, _id, _desc, _limit, _file_structure):
     """
     Download song(s) from the C3DB webpage.
 
+    Filters allow granular selection of the song(s) to download. Multiple filters can be
+    specified, and a song is selected only if ALL filters match (logical AND). Filters are
+    specified in the form "--filter <database_key> <value>" for information filters, or
+    "--instrument-filter [no-]<instrument>" for instrument filters.
+
+    For a full list of and explanation for filters, see the "search" command help
+    (command "c3dbdl search --help").
+
+    In addition to filters, each song may have more than one download link, to provide
+    multiple versions of the same song (for example, normal and multitracks, or alternate
+    charts). For each song, the "-i"/"--download-id" and "-d"/"--download-descr" options
+    can help filter these out, or both can be left blank to download all possible files
+    for a given song.
+
     \b
     The output file structure can be specified as a path format with any of the following
     fields included, surrounded by curly braces:
@@ -492,28 +566,6 @@ def download(_filters, _id, _desc, _limit, _file_structure):
     The default output file structure is:
         "{artist}/{album}/{title}.{author}.{orig_name}"
 
-    Filters allow granular selection of the song(s) to download. Multiple filters can be
-    specified, and a song is selected only if ALL filters match (logical AND). Each filter
-    is in the form "--filter [database_key] [value]".
-
-    The valid "database_key" values are identical to the output file fields above, except
-    for "orig_name".
-
-    \b
-    For example, to download all songs in the genre "Rock":
-      --filter genre Rock
-
-    \b
-    Or to download all songs by the artist "Rush" and the author "MyName":
-      --filter artist Rush --filter author MyName
-
-    In addition to filters, each song may have more than one download link, to provide
-    multiple versions of the same song (for example, normal and multitracks, or alternate
-    charts). For each song, the "-i"/"--download-id" and "-d"/"--download-descr" options
-    can help filter these out, or both can be left blank to download all possible files
-    for a given song. Mostly useful when being extremely restrictive with filters, less
-    so when downloading many songs at once.
-
     \b
     The following environment variables can be used for scripting purposes:
       * C3DBDL_DL_FILE_STRUCTURE: equivalent to "--file-structure"
@@ -530,14 +582,47 @@ def download(_filters, _id, _desc, _limit, _file_structure):
     pending_songs = list()
 
     for song in all_songs:
-        if len(_filters) < 1:
+        if len(_filters) < 1 and len(_instrument_filters) < 1:
             add_to_pending = True
         else:
+            # Parse the information filters
+            if len(_filters) > 0:
                 try:
-                add_to_pending = all(_filter[1].lower() in song[_filter[0]].lower() for _filter in _filters)
-            except KeyError:
-                click.echo(f"Invalid filter field {_filter[0]}")
+                    pending_information_filters = [
+                        _filter[1].lower() in song[_filter[0]].lower()
+                        for _filter in _filters
+                    ]
+                    information_add_to_pending = all(pending_information_filters)
+                except KeyError as e:
+                    click.echo(f"Invalid filter field {e}")
                     exit(1)
+            else:
+                information_add_to_pending = True
+
+            # Parse the instrument filters
+            if len(_instrument_filters) > 0:
+                try:
+                    pending_instrument_filters = list()
+                    for instrument_filter in _instrument_filters:
+                        if re.match("^no-", instrument_filter):
+                            instrument_filter = instrument_filter.replace('no-', '')
+                            if song["instruments"][instrument_filter] is None:
+                                pending_instrument_filters.append(True)
+                            else:
+                                pending_instrument_filters.append(False)
+                        else:
+                            if song["instruments"][instrument_filter] is not None:
+                                pending_instrument_filters.append(True)
+                            else:
+                                pending_instrument_filters.append(False)
+                    instrument_add_to_pending = all(pending_instrument_filters)
+                except KeyError as e:
+                    click.echo(f"Invalid instrument value {e}")
+                    exit(1)
+            else:
+                instrument_add_to_pending = True
+
+            add_to_pending = all([information_add_to_pending, instrument_add_to_pending])
 
         if add_to_pending:
             pending_songs.append(song)
@@ -545,7 +630,7 @@ def download(_filters, _id, _desc, _limit, _file_structure):
     if _limit is not None:
         pending_songs = pending_songs[0:_limit]
 
-    click.echo(f"Downloading {len(pending_songs)} song files...")
+    click.echo(f"Downloading {len(pending_songs)} songs...")
 
     for song in pending_songs:
         downloadSong(config["download_directory"], _file_structure, song, _id, _desc)
@@ -560,22 +645,78 @@ def download(_filters, _id, _desc, _limit, _file_structure):
     default=[],
     multiple=True,
     nargs=2,
-    help="Add a filter option.",
+    help="Add an information filter.",
 )
-def search(_filters):
+@click.option(
+    "-s",
+    "--instrument-filter",
+    "_instrument_filters",
+    envvar="C3DBDL_DL_INSTFILTERS",
+    default=[],
+    multiple=True,
+    nargs=1,
+    help="Add an instrument filter."
+)
+def search(_filters, _instrument_filters):
     """
     Search for song(s) from the C3DB local database.
 
     Filters allow granular selection of the song(s) to download. Multiple filters can be
-    specified, and a song is selected only if ALL filters match (logical AND). Each filter
-    is in the form "--filter [database_key] [value]".
+    specified, and a song is selected only if ALL filters match (logical AND). Filters are
+    specified in the form "--filter <database_key> <value>" for information filters, or
+    "--instrument-filter [no-]<instrument>" for instrument filters.
 
-    For a full list of and explanation for filters, see the "download" command help
-    (command "c3dbdl download --help").
+    Information filters match against the basic information of a song, for example finding
+    songs by a given artist, from a given album, by a given chart author, etc.
+
+    Filter values are fuzzy and case insensitive, so for example "word" would match
+    against a song titled "In The Word".
+
+    \b
+    The valid fields for the "<database_key>" value are:
+      * genre: The genre of the song.
+      * artist: The artist of the song.
+      * album: The album of the song.
+      * title: The title of the song.
+      * year: The year of the album/song.
+      * author: The author of the file on C3DB.
+
+    \b
+    For example, to download all songs in the genre "Rock":
+      --filter genre Rock
+
+    \b
+    Or to download all songs by the artist "Rush" and the author "MyName":
+      --filter artist Rush --filter author MyName
+
+    Instrument filters allow selection of the presence of instruments. If an instrument
+    fitler is given, only songs which contain parts for the given instrument(s) will be
+    shown.
+
+    \b
+    The valid instruments (case-sensitive, i.e. must be lowercase) are:
+      * guitar
+      * bass
+      * drums
+      * vocals
+      * keys
+
+    To negate an instrument filter, append "no-" to the instrument name.
+
+    \b
+    For example, to download only songs that have a keys part but no vocal part:
+      --instrument-filter keys --instrument-filter no-vocals
+
+    Note that while instrument difficulties are displayed in the output of this command,
+    they can not be filtered on; this is up to the user to do manually. The purpose of
+    instrument filters is to ensure that songs contain or don't contain given parts, not
+    to granularly select the difficulty of said parts (that's for the players of the game
+    to do, not us).
 
     \b
     The following environment variables can be used for scripting purposes:
       * C3DBDL_DL_FILTERS: equivalent to "--filter"; limited to one instance
+      * C3DBDL_DL_INSTFILTERS: equivalent to "--instrument-filter"; limited to one instance
     """
 
     with open(config["database_filename"], "r") as fh:
@@ -587,24 +728,72 @@ def search(_filters):
     pending_songs = list()
 
     for song in all_songs:
-        if len(_filters) < 1:
+        if len(_filters) < 1 and len(_instrument_filters) < 1:
             add_to_pending = True
         else:
+            # Parse the information filters
+            if len(_filters) > 0:
                 try:
-                add_to_pending = all(_filter[1].lower() in song[_filter[0]].lower() for _filter in _filters)
-            except KeyError:
-                click.echo(f"Invalid filter field {_filter[0]}")
+                    pending_information_filters = [
+                        _filter[1].lower() in song[_filter[0]].lower()
+                        for _filter in _filters
+                    ]
+                    information_add_to_pending = all(pending_information_filters)
+                except KeyError as e:
+                    click.echo(f"Invalid filter field {e}")
                     exit(1)
+            else:
+                information_add_to_pending = True
+
+            # Parse the instrument filters
+            if len(_instrument_filters) > 0:
+                try:
+                    pending_instrument_filters = list()
+                    for instrument_filter in _instrument_filters:
+                        if re.match("^no-", instrument_filter):
+                            instrument_filter = instrument_filter.replace('no-', '')
+                            if song["instruments"][instrument_filter] is None:
+                                pending_instrument_filters.append(True)
+                            else:
+                                pending_instrument_filters.append(False)
+                        else:
+                            if song["instruments"][instrument_filter] is not None:
+                                pending_instrument_filters.append(True)
+                            else:
+                                pending_instrument_filters.append(False)
+                    instrument_add_to_pending = all(pending_instrument_filters)
+                except KeyError as e:
+                    click.echo(f"Invalid instrument value {e}")
+                    exit(1)
+            else:
+                instrument_add_to_pending = True
+
+            add_to_pending = all([information_add_to_pending, instrument_add_to_pending])
 
         if add_to_pending:
             pending_songs.append(song)
 
-    click.echo(f"Found {len(pending_songs)} matchin song files:")
+    click.echo(f"Found {len(pending_songs)} matching songs:")
+    click.echo()
     for entry in pending_songs:
         click.echo(
-            f"""> "{entry['artist']} - {entry['title']}" by {entry['author']}..."""
+            f"""> Song: "{entry['artist']} - {entry['title']}" from "{entry['album']} ({entry['year']})" by {entry['author']}"""
         )
 
+        instrument_list = list()
+        for instrument in entry["instruments"]:
+            instrument_list.append(f"{instrument} [{entry['instruments'][instrument]}]")
+        click.echo(
+            f"""  Instruments: {', '.join(instrument_list)}""",
+        )
+            
+        click.echo(
+            f"""  Available downloads:"""
+        )
+        for link in entry["dl_links"]:
+            click.echo(f"""    * {link['description']}""")
+        click.echo()
+
 
 @click.group(context_settings=CONTEXT_SETTINGS)
 @click.option(