Compare commits
7 Commits
083ff1884b
...
master
Author | SHA1 | Date | |
---|---|---|---|
7a2154db69 | |||
0a0fb144c0 | |||
6ecb62431e | |||
447eb4120a | |||
aa680dadc2 | |||
61b4d0e429 | |||
cb1a5c5d58 |
1
.flake8
1
.flake8
@ -5,5 +5,6 @@
|
|||||||
# * E203 (whitespace before ':'): Black recommends this as disabled
|
# * E203 (whitespace before ':'): Black recommends this as disabled
|
||||||
ignore = W503, E501
|
ignore = W503, E501
|
||||||
extend-ignore = E203
|
extend-ignore = E203
|
||||||
|
exclude = build/
|
||||||
# Set the max line length to 88 for Black
|
# Set the max line length to 88 for Black
|
||||||
max-line-length = 88
|
max-line-length = 88
|
||||||
|
104
README.md
104
README.md
@ -105,28 +105,37 @@ guaranteed to be smaller than the total number listed on the C3DB website.
|
|||||||
|
|
||||||
## Searching & Downloading
|
## Searching & Downloading
|
||||||
|
|
||||||
Once a database has been built, you can start searching for downloading songs.
|
Once a database has been built, you can start searching for and downloading songs.
|
||||||
|
|
||||||
To search for songs, use the `search` command. This command takes `--filter` arguments in order to show what
|
To search for songs, use the `search` command. This command takes `--filter` arguments in order to show what
|
||||||
song(s) would be downloaded by a given filter, without actually triggering the download. Once you have a valid
|
song(s) would be downloaded by a given filter, along with their basic information, without actually triggering
|
||||||
filter from a search, you can use it to download.
|
a download. Once you have a valid filter from a search, you can use it to `download` precisely the song(s) you
|
||||||
|
want.
|
||||||
|
|
||||||
To download songs, use the `download` command. See the following sections for more details on the specifics of
|
See the following sections for more details on the specifics of the filters and output formatting of the
|
||||||
the filters and output formatting of the `download` command.
|
`search` and `download` commands.
|
||||||
|
|
||||||
By default, when downloading a given song, all possible download links (`dl_links`) will be downloaded; this
|
By default, when downloading a given song, all possible download links (`dl_links`) will be downloaded; this
|
||||||
can be limited by using the `-i`/`--download-id` and `-d`/`--download-descr` options to pick and choose specific
|
can be limited by using the `-i`/`--download-id` and `-d`/`--download-descr` options to pick and choose specific
|
||||||
files.
|
files. A specific example usecase would be to specify `--download-descr 360` to only download Xbox 360 RBCONs.
|
||||||
|
|
||||||
Once a song has been downloaded, assuming that the file structure doesn't change, subsequent `download` runs will
|
Once a song has been downloaded, assuming that the file structure doesn't change, subsequent `download` runs will
|
||||||
not overwrite it and will simply skip downloading the file.
|
not overwrite it and will simply skip downloading the file.
|
||||||
|
|
||||||
### Filtering
|
### Filtering
|
||||||
|
|
||||||
Filtering out the songs in the database is a key part of this tool. You might want to be able to grab only select
|
Filtering out the songs in the database is a key part of this tool. You might want to be able to grab only songs
|
||||||
genres, artists, authors, etc. to make your custom song packs.
|
with certain genres, artists, instruments, etc. or by certain authors, to make your custom song packs.
|
||||||
|
|
||||||
`c3dbdl` is able to filter by several key categories:
|
If multiple filters are specified, they are treated as a logical AND, i.e. *all* of the give filters must apply
|
||||||
|
to a song for it to be matched.
|
||||||
|
|
||||||
|
Filtering is always done during the search/download stage; the JSON database will always contain all possible
|
||||||
|
entries from the C3DB.
|
||||||
|
|
||||||
|
#### Information Filters
|
||||||
|
|
||||||
|
`c3dbdl` is able to filter songs by their general information in several key categories:
|
||||||
|
|
||||||
* `genre`: The genre of the song.
|
* `genre`: The genre of the song.
|
||||||
* `artist`: The artist of the song.
|
* `artist`: The artist of the song.
|
||||||
@ -135,35 +144,68 @@ genres, artists, authors, etc. to make your custom song packs.
|
|||||||
* `year`: The year of the album/song.
|
* `year`: The year of the album/song.
|
||||||
* `author`: The author of the file on C3DB.
|
* `author`: The author of the file on C3DB.
|
||||||
|
|
||||||
Note that we *cannot* filter - mostly for parsing difficulty reasons - by intrument type or difficulty, by song
|
To use information filters, append one or more `--filter` options to your `c3dbdl search` or `download` command. An
|
||||||
length, or by any other information not mentioned above.
|
information filter option begins with the literal `--filter`, followed by the field (e.g. `genre` or `artist`), then
|
||||||
|
finally the text value to filter on, for instance `Rock` or `Santana` or `2012`. The text must be quoted if it
|
||||||
|
contains any whitespace.
|
||||||
|
|
||||||
Filtering is always done during the search/download stage; the JSON database will always contain all possible entries.
|
Information filter values are fuzzy. They are case insensitive, and use the `in` construct. So, for example, the
|
||||||
|
filter string `--filter song "edmund fitzgerald"` would match the song title "The Wreck of the Edmund Fitzgerald".
|
||||||
|
|
||||||
To use filters, append one or more `--filter` options to your `c3dbdl download` or `search` command. A filter option
|
For example, to find all songs by Rush from the album Vapor Trails (the remixed version) authored by ejthedj:
|
||||||
begins with the literal `--filter`, followed by the category (e.g. `genre` or `artist`), then finally the text to
|
|
||||||
filter on, for instance `Rock` or `Santana` or `2012`. The text must be quoted if it contains whitespace.
|
|
||||||
|
|
||||||
If more that one filter is specified, they are treated as a logical AND, i.e. all the listed filters must apply to
|
|
||||||
a given song for it to be downloaded in that run.
|
|
||||||
|
|
||||||
Filter values are fuzzy. They are case insensitive, and use the `in` construct. So, for example, the filter string
|
|
||||||
`--filter song "edmund fitzgerald"` would match the song title "The Wreck of the Edmund Fitzgerald".
|
|
||||||
|
|
||||||
Filters allow very specific download selections to be run. For example, let's look for all songs by Rush
|
|
||||||
from the album Vapor Trails (the remixed version) authored by ejthedj:
|
|
||||||
|
|
||||||
```
|
```
|
||||||
c3dbdl download --filter artist Rush --filter album "Vapor Trails [Remixed]" --author ejthedj
|
c3dbdl search --filter artist Rush --filter album "Vapor Trails [Remixed]" --filter author ejthedj
|
||||||
Found 19563 songs from JSON database file 'Downloads/c3db.json'
|
Found 19563 songs from JSON database file 'Downloads/c3db.json'
|
||||||
Downloading 1 song files...
|
Found 1 matching songs:
|
||||||
> Downloading song "Rush - Sweet Miracle" by ejthedj...
|
|
||||||
Downloading file "Rock Band 3 Xbox 360" from https://dl.c3universe.com/s/ejthedj/sweetMiracle...
|
> Song: "Rush - Sweet Miracle" from "Vapor Trails [Remixed] (2002)" by ejthedj
|
||||||
Successfully downloaded to ../Prog/ejthedj/Rush/Vapor Trails [Remixed]/Sweet Miracle [2002].sweetMiracle
|
Instruments: guitar [2], bass [3], drums [4], vocals [4], keys [None]
|
||||||
|
Available downloads:
|
||||||
|
* Rock Band 3 Xbox 360
|
||||||
```
|
```
|
||||||
|
|
||||||
In this case, one song matched and was downloaded. Feel free to experiment with the various filters to find
|
In this case, one song matched; applying the same filter to a `download` would thus download only the single song.
|
||||||
exactly what you're looking for.
|
|
||||||
|
#### Instrument Filters
|
||||||
|
|
||||||
|
In addition to the information filters, `c3dbdl` can also filter by available instrument parts. There are 5 valid
|
||||||
|
instruments that can be filtered on:
|
||||||
|
|
||||||
|
* `guitar`
|
||||||
|
* `bass`
|
||||||
|
* `drums`
|
||||||
|
* `vocals`
|
||||||
|
* `keys`
|
||||||
|
|
||||||
|
To use instrument filters, append one or more `--filter instrument <instrument>` options to your `c3dbdl search` or
|
||||||
|
`download` command. An instrument filter option begins with the literal `--filter instrument`, followed by the
|
||||||
|
instrument you wish to filter on.
|
||||||
|
|
||||||
|
If a part contains the instrument at any difficulty (from 0-6), it will match the filter; if the instrument part
|
||||||
|
is missing, it will not match.
|
||||||
|
|
||||||
|
You can also invert the match by adding `no-` to the instrument name. So `--filter instrument no-keys` would
|
||||||
|
only match songs *without* a keys part.
|
||||||
|
|
||||||
|
For example, to find all songs by Rush which have a keys part but no vocal part:
|
||||||
|
|
||||||
|
```
|
||||||
|
c3dbdl search --filter artist Rush --filter instrument keys --filter instrument no-vocals
|
||||||
|
Found 19562 songs from JSON database file 'Downloads/c3db.json'
|
||||||
|
Found 1 matching songs:
|
||||||
|
|
||||||
|
> Song: "Rush - La Villa Strangiato" from "Hemispheres (1978)" by DoNotPassGo
|
||||||
|
Instruments: guitar [6], bass [5], drums [6], vocals [None], keys [1]
|
||||||
|
Available downloads:
|
||||||
|
* Rock Band 3 Xbox 360
|
||||||
|
* Rock Band 3 Wii
|
||||||
|
* Rock Band 3 PS3
|
||||||
|
* Phase Shift
|
||||||
|
* Rock Band 3 Xbox 360 (Alternate Version)
|
||||||
|
```
|
||||||
|
|
||||||
|
In this case, one song matched; applying the same filter to a `download` would thus download only the single song.
|
||||||
|
|
||||||
### Output Format
|
### Output Format
|
||||||
|
|
||||||
|
474
c3dbdl/c3dbdl.py
474
c3dbdl/c3dbdl.py
@ -33,11 +33,24 @@ from concurrent.futures import ThreadPoolExecutor, as_completed
|
|||||||
CONTEXT_SETTINGS = dict(help_option_names=["-h", "--help"], max_content_width=120)
|
CONTEXT_SETTINGS = dict(help_option_names=["-h", "--help"], max_content_width=120)
|
||||||
|
|
||||||
|
|
||||||
def fetchSongData(entry):
|
def fetchSongData(entries):
|
||||||
song_entry = dict()
|
song_entry = {
|
||||||
|
"artist": None,
|
||||||
|
"title": None,
|
||||||
|
"album": None,
|
||||||
|
"song_link": None,
|
||||||
|
"genre": None,
|
||||||
|
"year": None,
|
||||||
|
"length": None,
|
||||||
|
"author": None,
|
||||||
|
"instruments": dict(),
|
||||||
|
"dl_links": list(),
|
||||||
|
}
|
||||||
messages = list()
|
messages = list()
|
||||||
|
found_instruments = False
|
||||||
|
|
||||||
for idx, td in enumerate(entry.find_all("td")):
|
# Find song details
|
||||||
|
for idx, td in enumerate(entries[0].find_all("td")):
|
||||||
if idx == 2:
|
if idx == 2:
|
||||||
# Artist
|
# Artist
|
||||||
song_entry["artist"] = td.find("a").get_text().strip().replace("/", "+")
|
song_entry["artist"] = td.find("a").get_text().strip().replace("/", "+")
|
||||||
@ -74,6 +87,35 @@ def fetchSongData(entry):
|
|||||||
# Author (of chart)
|
# Author (of chart)
|
||||||
song_entry["author"] = td.find("a").get_text().strip().replace("/", "+")
|
song_entry["author"] = td.find("a").get_text().strip().replace("/", "+")
|
||||||
|
|
||||||
|
# Find song instruments and difficulties
|
||||||
|
for idx, td in enumerate(entries[1].find_all("td")):
|
||||||
|
if (
|
||||||
|
len(list(td.find_all("div", attrs={"style": "width:110px;float:left"}))) > 0
|
||||||
|
and not found_instruments
|
||||||
|
):
|
||||||
|
for instrument in td.find_all(
|
||||||
|
"div", attrs={"style": "width:110px;float:left"}
|
||||||
|
):
|
||||||
|
difficulty_link = (
|
||||||
|
instrument.find_all(
|
||||||
|
"a", attrs={"style": "text-decoration: none;color:#000"}
|
||||||
|
)[1]
|
||||||
|
.get("href")
|
||||||
|
.split("/")
|
||||||
|
)
|
||||||
|
instrument_name = (
|
||||||
|
difficulty_link[-2].split("_")[-1].replace("prokeys", "keys")
|
||||||
|
)
|
||||||
|
instrument_diff = int(difficulty_link[-1])
|
||||||
|
if instrument_diff < 1:
|
||||||
|
# No part
|
||||||
|
instrument_difficulty = None
|
||||||
|
else:
|
||||||
|
# Link difficulty - 1
|
||||||
|
instrument_difficulty = instrument_diff - 1
|
||||||
|
song_entry["instruments"][instrument_name] = instrument_difficulty
|
||||||
|
found_instruments = True
|
||||||
|
|
||||||
if (
|
if (
|
||||||
song_entry
|
song_entry
|
||||||
and song_entry["author"]
|
and song_entry["author"]
|
||||||
@ -83,9 +125,6 @@ def fetchSongData(entry):
|
|||||||
messages.append(
|
messages.append(
|
||||||
f"> Found song entry for {song_entry['artist']} - {song_entry['title']} by {song_entry['author']}"
|
f"> Found song entry for {song_entry['artist']} - {song_entry['title']} by {song_entry['author']}"
|
||||||
)
|
)
|
||||||
for entry_type in ["artist", "album", "genre", "year", "length"]:
|
|
||||||
if not song_entry[entry_type]:
|
|
||||||
song_entry[entry_type] = "None"
|
|
||||||
|
|
||||||
# Get download links from the actual song page
|
# Get download links from the actual song page
|
||||||
attempts = 1
|
attempts = 1
|
||||||
@ -114,8 +153,6 @@ def fetchSongData(entry):
|
|||||||
for link_entry in download_links:
|
for link_entry in download_links:
|
||||||
link = link_entry.get("href")
|
link = link_entry.get("href")
|
||||||
description = link_entry.get_text().strip()
|
description = link_entry.get_text().strip()
|
||||||
if "c3universe.com" not in link:
|
|
||||||
continue
|
|
||||||
messages.append(f"Found download link: {link} ({description})")
|
messages.append(f"Found download link: {link} ({description})")
|
||||||
dl_links.append(
|
dl_links.append(
|
||||||
{
|
{
|
||||||
@ -130,7 +167,7 @@ def fetchSongData(entry):
|
|||||||
return None
|
return None
|
||||||
song_entry["dl_links"] = dl_links
|
song_entry["dl_links"] = dl_links
|
||||||
|
|
||||||
# Append to the database
|
# Return messages and song entry
|
||||||
return messages, song_entry
|
return messages, song_entry
|
||||||
|
|
||||||
|
|
||||||
@ -178,11 +215,22 @@ def buildDatabase(pages, concurrency):
|
|||||||
"tbody"
|
"tbody"
|
||||||
)
|
)
|
||||||
|
|
||||||
|
# This is weird, but because of the table layout, there are two table rows for
|
||||||
|
# each song: the first is the song info, the second is the instruments
|
||||||
|
# So we must make a single "entry" that is a list of the two elements, then
|
||||||
|
# handle that later in fetchSongData.
|
||||||
entries = list()
|
entries = list()
|
||||||
|
entry_idx = 0
|
||||||
|
entry_data = list()
|
||||||
for entry in table_html.find_all("tr", attrs={"class": "odd"}):
|
for entry in table_html.find_all("tr", attrs={"class": "odd"}):
|
||||||
if len(entry) < 1:
|
if len(entry) < 1:
|
||||||
break
|
break
|
||||||
entries.append(entry)
|
entry_data.append(entry)
|
||||||
|
entry_idx += 1
|
||||||
|
if entry_idx == 2:
|
||||||
|
entries.append(entry_data)
|
||||||
|
entry_idx = 0
|
||||||
|
entry_data = list()
|
||||||
|
|
||||||
click.echo("Fetching and parsing song pages...")
|
click.echo("Fetching and parsing song pages...")
|
||||||
with ThreadPoolExecutor(max_workers=concurrency) as executor:
|
with ThreadPoolExecutor(max_workers=concurrency) as executor:
|
||||||
@ -202,6 +250,70 @@ def buildDatabase(pages, concurrency):
|
|||||||
return found_songs
|
return found_songs
|
||||||
|
|
||||||
|
|
||||||
|
def downloadFile(download_url, download_path, download_filename):
|
||||||
|
attempts = 1
|
||||||
|
p = None
|
||||||
|
try:
|
||||||
|
with requests.get(download_url, stream=True) as r:
|
||||||
|
while attempts <= 3:
|
||||||
|
try:
|
||||||
|
r.raise_for_status()
|
||||||
|
break
|
||||||
|
except Exception:
|
||||||
|
click.echo(
|
||||||
|
f"Download attempt failed: HTTP {r.status_code}; retrying {attempts}/3"
|
||||||
|
)
|
||||||
|
sleep(attempts)
|
||||||
|
attempts += 1
|
||||||
|
if r is None or r.status_code != 200:
|
||||||
|
if r:
|
||||||
|
code = r.status_code
|
||||||
|
else:
|
||||||
|
code = "-1"
|
||||||
|
raise HTTPError(download_url, code, "", None, None)
|
||||||
|
|
||||||
|
if not os.path.exists(download_path):
|
||||||
|
os.makedirs(download_path)
|
||||||
|
|
||||||
|
with open(download_filename, "wb") as f:
|
||||||
|
for chunk in r.iter_content(chunk_size=8192):
|
||||||
|
f.write(chunk)
|
||||||
|
click.echo(f"Successfully downloaded to {download_filename}")
|
||||||
|
except Exception as e:
|
||||||
|
click.echo(f"Download attempt failed: {e}")
|
||||||
|
return None
|
||||||
|
|
||||||
|
def parseC3Universe(dl_link):
|
||||||
|
try:
|
||||||
|
p = requests.get(dl_link)
|
||||||
|
parsed_html = BeautifulSoup(p.text, "html.parser")
|
||||||
|
download_element = (
|
||||||
|
parsed_html.body.find("div", attrs={"class": "lock-head"})
|
||||||
|
.find("a")
|
||||||
|
)
|
||||||
|
download_url = download_element.get("href")
|
||||||
|
return download_url
|
||||||
|
except Exception as e:
|
||||||
|
click.echo(f"Failed parsing or retrieving file URL from link {dl_link}: {e}")
|
||||||
|
return None
|
||||||
|
|
||||||
|
|
||||||
|
def parseMediafire(dl_link):
|
||||||
|
try:
|
||||||
|
p = requests.get(dl_link)
|
||||||
|
parsed_html = BeautifulSoup(p.text, "html.parser")
|
||||||
|
download_element = parsed_html.find(
|
||||||
|
"a", attrs={"aria-label": "Download file"}
|
||||||
|
)
|
||||||
|
if download_element is not None:
|
||||||
|
download_url = download_element.get("href")
|
||||||
|
else:
|
||||||
|
download_url = re.search(r"'(http[s]*://download[0-9]+.mediafire.com/.*)';", p.text).group(1)
|
||||||
|
return download_url
|
||||||
|
except Exception as e:
|
||||||
|
click.echo(f"Failed parsing or retrieving file URL from link {dl_link}: {e}")
|
||||||
|
return None
|
||||||
|
|
||||||
def downloadSong(destination, filename, entry, dlid, dldesc):
|
def downloadSong(destination, filename, entry, dlid, dldesc):
|
||||||
click.echo(
|
click.echo(
|
||||||
f"""> Downloading song "{entry['artist']} - {entry['title']}" by {entry['author']}..."""
|
f"""> Downloading song "{entry['artist']} - {entry['title']}" by {entry['author']}..."""
|
||||||
@ -228,19 +340,17 @@ def downloadSong(destination, filename, entry, dlid, dldesc):
|
|||||||
return
|
return
|
||||||
|
|
||||||
for dl_link in dl_links:
|
for dl_link in dl_links:
|
||||||
try:
|
if 'dl.c3universe.com' in dl_link['link']:
|
||||||
p = requests.get(dl_link["link"])
|
download_url = parseC3Universe(dl_link["link"])
|
||||||
if p.status_code != 200:
|
elif 'www.mediafire.com' in dl_link["link"]:
|
||||||
raise HTTPError(dl_link["link"], p.status_code, "", None, None)
|
download_url = parseMediafire(dl_link["link"])
|
||||||
|
else:
|
||||||
|
click.echo("Download URL is not valid for CLI download; skipping...")
|
||||||
|
click.echo(f"URL: {dl_link['link']}")
|
||||||
|
continue
|
||||||
|
|
||||||
parsed_html = BeautifulSoup(p.text, "html.parser")
|
if download_url is None:
|
||||||
download_url = (
|
click.echo(f"No valid download URL found, skipping...")
|
||||||
parsed_html.body.find("div", attrs={"class": "lock-head"})
|
|
||||||
.find("a")
|
|
||||||
.get("href")
|
|
||||||
)
|
|
||||||
except Exception as e:
|
|
||||||
click.echo(f"Failed parsing or retrieving HTML link: {e}")
|
|
||||||
continue
|
continue
|
||||||
|
|
||||||
download_filename = filename.format(
|
download_filename = filename.format(
|
||||||
@ -262,38 +372,7 @@ def downloadSong(destination, filename, entry, dlid, dldesc):
|
|||||||
click.echo(f"File exists at {download_filename}")
|
click.echo(f"File exists at {download_filename}")
|
||||||
continue
|
continue
|
||||||
|
|
||||||
attempts = 1
|
downloadFile(download_url, download_path, download_filename)
|
||||||
p = None
|
|
||||||
try:
|
|
||||||
with requests.get(download_url, stream=True) as r:
|
|
||||||
while attempts <= 3:
|
|
||||||
try:
|
|
||||||
r.raise_for_status()
|
|
||||||
break
|
|
||||||
except Exception:
|
|
||||||
click.echo(
|
|
||||||
f"Download attempt failed: HTTP {r.status_code}; retrying {attempts}/3"
|
|
||||||
)
|
|
||||||
sleep(attempts)
|
|
||||||
attempts += 1
|
|
||||||
if r is None or r.status_code != 200:
|
|
||||||
if r:
|
|
||||||
code = r.status_code
|
|
||||||
else:
|
|
||||||
code = "-1"
|
|
||||||
raise HTTPError(download_url, code, "", None, None)
|
|
||||||
|
|
||||||
if not os.path.exists(download_path):
|
|
||||||
os.makedirs(download_path)
|
|
||||||
|
|
||||||
with open(download_filename, "wb") as f:
|
|
||||||
for chunk in r.iter_content(chunk_size=8192):
|
|
||||||
f.write(chunk)
|
|
||||||
click.echo(f"Successfully downloaded to {download_filename}")
|
|
||||||
except Exception as e:
|
|
||||||
click.echo(f"Download attempt failed: {e}")
|
|
||||||
continue
|
|
||||||
|
|
||||||
|
|
||||||
@click.command(name="build", short_help="Build the local database.")
|
@click.command(name="build", short_help="Build the local database.")
|
||||||
@click.option(
|
@click.option(
|
||||||
@ -443,11 +522,10 @@ def database():
|
|||||||
"-f",
|
"-f",
|
||||||
"--filter",
|
"--filter",
|
||||||
"_filters",
|
"_filters",
|
||||||
envvar="C3DBDL_DL_FILTERS",
|
|
||||||
default=[],
|
default=[],
|
||||||
multiple=True,
|
multiple=True,
|
||||||
nargs=2,
|
nargs=2,
|
||||||
help="Add a filter option.",
|
help="Add a search filter.",
|
||||||
)
|
)
|
||||||
@click.option(
|
@click.option(
|
||||||
"-l",
|
"-l",
|
||||||
@ -462,6 +540,7 @@ def database():
|
|||||||
"-i",
|
"-i",
|
||||||
"--download-id",
|
"--download-id",
|
||||||
"_id",
|
"_id",
|
||||||
|
envvar="C3DBDL_DL_ID",
|
||||||
default=None,
|
default=None,
|
||||||
type=int,
|
type=int,
|
||||||
help='Download only "dl_links" entry N (1 is first, etc.), or all if unspecified.',
|
help='Download only "dl_links" entry N (1 is first, etc.), or all if unspecified.',
|
||||||
@ -470,6 +549,7 @@ def database():
|
|||||||
"-d",
|
"-d",
|
||||||
"--download-descr",
|
"--download-descr",
|
||||||
"_desc",
|
"_desc",
|
||||||
|
envvar="C3DBDL_DL_DESCR",
|
||||||
default=None,
|
default=None,
|
||||||
help='Download only "dl_links" entries with this in their description (fuzzy).',
|
help='Download only "dl_links" entries with this in their description (fuzzy).',
|
||||||
)
|
)
|
||||||
@ -477,6 +557,19 @@ def download(_filters, _id, _desc, _limit, _file_structure):
|
|||||||
"""
|
"""
|
||||||
Download song(s) from the C3DB webpage.
|
Download song(s) from the C3DB webpage.
|
||||||
|
|
||||||
|
Filters allow granular selection of the song(s) to download. Multiple filters can be
|
||||||
|
specified, and a song is selected only if ALL filters match (logical AND). Filters are
|
||||||
|
specified in the form "--filter <field> <value>".
|
||||||
|
|
||||||
|
For a full list of and explanation for filters, see the help output for the "search"
|
||||||
|
command ("c3dbdl search --help").
|
||||||
|
|
||||||
|
In addition to filters, each song may have more than one download link, to provide
|
||||||
|
multiple versions of the same song (for example, normal and multitracks, or alternate
|
||||||
|
charts). For each song, the "-i"/"--download-id" and "-d"/"--download-descr" options
|
||||||
|
can help filter these out, or both can be left blank to download all possible files
|
||||||
|
for a given song.
|
||||||
|
|
||||||
\b
|
\b
|
||||||
The output file structure can be specified as a path format with any of the following
|
The output file structure can be specified as a path format with any of the following
|
||||||
fields included, surrounded by curly braces:
|
fields included, surrounded by curly braces:
|
||||||
@ -492,33 +585,12 @@ def download(_filters, _id, _desc, _limit, _file_structure):
|
|||||||
The default output file structure is:
|
The default output file structure is:
|
||||||
"{artist}/{album}/{title}.{author}.{orig_name}"
|
"{artist}/{album}/{title}.{author}.{orig_name}"
|
||||||
|
|
||||||
Filters allow granular selection of the song(s) to download. Multiple filters can be
|
|
||||||
specified, and a song is selected only if ALL filters match (logical AND). Each filter
|
|
||||||
is in the form "--filter [database_key] [value]".
|
|
||||||
|
|
||||||
The valid "database_key" values are identical to the output file fields above, except
|
|
||||||
for "orig_name".
|
|
||||||
|
|
||||||
\b
|
|
||||||
For example, to download all songs in the genre "Rock":
|
|
||||||
--filter genre Rock
|
|
||||||
|
|
||||||
\b
|
|
||||||
Or to download all songs by the artist "Rush" and the author "MyName":
|
|
||||||
--filter artist Rush --filter author MyName
|
|
||||||
|
|
||||||
In addition to filters, each song may have more than one download link, to provide
|
|
||||||
multiple versions of the same song (for example, normal and multitracks, or alternate
|
|
||||||
charts). For each song, the "-i"/"--download-id" and "-d"/"--download-descr" options
|
|
||||||
can help filter these out, or both can be left blank to download all possible files
|
|
||||||
for a given song. Mostly useful when being extremely restrictive with filters, less
|
|
||||||
so when downloading many songs at once.
|
|
||||||
|
|
||||||
\b
|
\b
|
||||||
The following environment variables can be used for scripting purposes:
|
The following environment variables can be used for scripting purposes:
|
||||||
* C3DBDL_DL_FILE_STRUCTURE: equivalent to "--file-structure"
|
* C3DBDL_DL_FILE_STRUCTURE: equivalent to "--file-structure"
|
||||||
* C3DBDL_DL_FILTERS: equivalent to "--filter"; limited to one instance
|
|
||||||
* C3DBDL_DL_LIMIT: equivalent to "--limit"
|
* C3DBDL_DL_LIMIT: equivalent to "--limit"
|
||||||
|
* C3DBDL_DL_ID: equivalent to "--download-id"
|
||||||
|
* C3DBDL_DL_DESCR: equivalent to "--download-descr"
|
||||||
"""
|
"""
|
||||||
|
|
||||||
with open(config["database_filename"], "r") as fh:
|
with open(config["database_filename"], "r") as fh:
|
||||||
@ -530,14 +602,77 @@ def download(_filters, _id, _desc, _limit, _file_structure):
|
|||||||
pending_songs = list()
|
pending_songs = list()
|
||||||
|
|
||||||
for song in all_songs:
|
for song in all_songs:
|
||||||
if len(_filters) < 1:
|
add_to_pending = True
|
||||||
add_to_pending = True
|
song_filters = _filters
|
||||||
else:
|
song_information_filters = list()
|
||||||
try:
|
song_instrument_filters = list()
|
||||||
add_to_pending = all(_filter[1].lower() in song[_filter[0]].lower() for _filter in _filters)
|
|
||||||
except KeyError:
|
if len(_filters) > 0:
|
||||||
click.echo(f"Invalid filter field {_filter[0]}")
|
# Extract the instrument filters
|
||||||
exit(1)
|
for _filter in song_filters:
|
||||||
|
if _filter[0] == "instrument":
|
||||||
|
song_instrument_filters.append(_filter[1].lower())
|
||||||
|
else:
|
||||||
|
song_information_filters.append(_filter)
|
||||||
|
|
||||||
|
if len(song_information_filters) > 0 or len(song_instrument_filters) > 0:
|
||||||
|
# Parse the information filters
|
||||||
|
if len(song_information_filters) > 0:
|
||||||
|
try:
|
||||||
|
pending_information_filters = list()
|
||||||
|
for information_filter in song_information_filters:
|
||||||
|
filter_field = information_filter[0].lower()
|
||||||
|
filter_value = information_filter[1].lower()
|
||||||
|
if re.match("^!", filter_value):
|
||||||
|
filter_value = filter_value.replace("!", "")
|
||||||
|
if filter_value in song[filter_field].lower():
|
||||||
|
pending_information_filters.append(False)
|
||||||
|
else:
|
||||||
|
pending_information_filters.append(True)
|
||||||
|
elif re.match("^~", filter_value):
|
||||||
|
filter_value = filter_value.replace("~", "")
|
||||||
|
if filter_value in song[filter_field].lower():
|
||||||
|
pending_information_filters.append(True)
|
||||||
|
else:
|
||||||
|
pending_information_filters.append(False)
|
||||||
|
else:
|
||||||
|
if filter_value == song[filter_field].lower():
|
||||||
|
pending_information_filters.append(True)
|
||||||
|
else:
|
||||||
|
pending_information_filters.append(False)
|
||||||
|
information_add_to_pending = all(pending_information_filters)
|
||||||
|
except KeyError as e:
|
||||||
|
click.echo(f"Invalid filter field {e}")
|
||||||
|
exit(1)
|
||||||
|
else:
|
||||||
|
information_add_to_pending = True
|
||||||
|
|
||||||
|
# Parse the instrument filters
|
||||||
|
if len(song_instrument_filters) > 0:
|
||||||
|
try:
|
||||||
|
pending_instrument_filters = list()
|
||||||
|
for instrument_filter in song_instrument_filters:
|
||||||
|
if re.match("^no-", instrument_filter):
|
||||||
|
instrument_filter = instrument_filter.replace("no-", "")
|
||||||
|
if song["instruments"][instrument_filter] is None:
|
||||||
|
pending_instrument_filters.append(True)
|
||||||
|
else:
|
||||||
|
pending_instrument_filters.append(False)
|
||||||
|
else:
|
||||||
|
if song["instruments"][instrument_filter] is not None:
|
||||||
|
pending_instrument_filters.append(True)
|
||||||
|
else:
|
||||||
|
pending_instrument_filters.append(False)
|
||||||
|
instrument_add_to_pending = all(pending_instrument_filters)
|
||||||
|
except KeyError as e:
|
||||||
|
click.echo(f"Invalid instrument value {e}")
|
||||||
|
exit(1)
|
||||||
|
else:
|
||||||
|
instrument_add_to_pending = True
|
||||||
|
|
||||||
|
add_to_pending = all(
|
||||||
|
[information_add_to_pending, instrument_add_to_pending]
|
||||||
|
)
|
||||||
|
|
||||||
if add_to_pending:
|
if add_to_pending:
|
||||||
pending_songs.append(song)
|
pending_songs.append(song)
|
||||||
@ -545,7 +680,7 @@ def download(_filters, _id, _desc, _limit, _file_structure):
|
|||||||
if _limit is not None:
|
if _limit is not None:
|
||||||
pending_songs = pending_songs[0:_limit]
|
pending_songs = pending_songs[0:_limit]
|
||||||
|
|
||||||
click.echo(f"Downloading {len(pending_songs)} song files...")
|
click.echo(f"Downloading {len(pending_songs)} songs...")
|
||||||
|
|
||||||
for song in pending_songs:
|
for song in pending_songs:
|
||||||
downloadSong(config["download_directory"], _file_structure, song, _id, _desc)
|
downloadSong(config["download_directory"], _file_structure, song, _id, _desc)
|
||||||
@ -556,26 +691,75 @@ def download(_filters, _id, _desc, _limit, _file_structure):
|
|||||||
"-f",
|
"-f",
|
||||||
"--filter",
|
"--filter",
|
||||||
"_filters",
|
"_filters",
|
||||||
envvar="C3DBDL_DL_FILTERS",
|
|
||||||
default=[],
|
default=[],
|
||||||
multiple=True,
|
multiple=True,
|
||||||
nargs=2,
|
nargs=2,
|
||||||
help="Add a filter option.",
|
help="Add a search filter.",
|
||||||
)
|
)
|
||||||
def search(_filters):
|
def search(_filters):
|
||||||
"""
|
"""
|
||||||
Search for song(s) from the C3DB local database.
|
Search for song(s) from the C3DB local database.
|
||||||
|
|
||||||
Filters allow granular selection of the song(s) to download. Multiple filters can be
|
Filters allow granular selection of the song(s) to download. Multiple filters can be
|
||||||
specified, and a song is selected only if ALL filters match (logical AND). Each filter
|
specified, and a song is selected only if ALL filters match (logical AND). Filters are
|
||||||
is in the form "--filter [database_key] [value]".
|
specified in the form "--filter <field> <value>".
|
||||||
|
|
||||||
For a full list of and explanation for filters, see the "download" command help
|
|
||||||
(command "c3dbdl download --help").
|
|
||||||
|
|
||||||
\b
|
\b
|
||||||
The following environment variables can be used for scripting purposes:
|
The valid fields for the "<field>" value are:
|
||||||
* C3DBDL_DL_FILTERS: equivalent to "--filter"; limited to one instance
|
* genre: The genre of the song.
|
||||||
|
* artist: The artist of the song.
|
||||||
|
* album: The album of the song.
|
||||||
|
* title: The title of the song.
|
||||||
|
* year: The year of the album/song.
|
||||||
|
* author: The author of the file on C3DB.
|
||||||
|
* instrument: An instrument chart for the song.
|
||||||
|
|
||||||
|
\b
|
||||||
|
For example, to download all songs in the genre "Rock":
|
||||||
|
--filter genre Rock
|
||||||
|
|
||||||
|
\b
|
||||||
|
Or to download all songs by the artist "Rush" and the author "MyName":
|
||||||
|
--filter artist Rush --filter author MyName
|
||||||
|
|
||||||
|
Filter values are case insensitive, and non-instrument filters can be made fuzzy by
|
||||||
|
adding a tilde ("~") to the beginning of the "<value>".
|
||||||
|
|
||||||
|
\b
|
||||||
|
For example, to match all songs with "Word" in their titles:
|
||||||
|
--filter title ~word
|
||||||
|
|
||||||
|
A filter can be negated by adding an exclamation mark ("!") to the beginning of the
|
||||||
|
"<value>". Note that "!" must be escaped or single-quoted under BASH.
|
||||||
|
|
||||||
|
\b
|
||||||
|
For example, to match all songs except those by Yes as their artist:
|
||||||
|
--filter artist '!Yes'
|
||||||
|
|
||||||
|
Instrument filters allow selection of the presence of instruments. If an instrument
|
||||||
|
fitler is given, only songs which contain parts for the given instrument(s) will be
|
||||||
|
shown.
|
||||||
|
|
||||||
|
\b
|
||||||
|
The valid instruments are:
|
||||||
|
* guitar
|
||||||
|
* bass
|
||||||
|
* drums
|
||||||
|
* vocals
|
||||||
|
* keys
|
||||||
|
|
||||||
|
To negate an instrument filter and find only entires without the specified
|
||||||
|
instrument, append "no-" to the instrument name.
|
||||||
|
|
||||||
|
\b
|
||||||
|
For example, to download only songs that have a keys part but no vocal part:
|
||||||
|
--filter instrument keys --filter instrument no-vocals
|
||||||
|
|
||||||
|
Note that while instrument difficulties are displayed in the output of this command,
|
||||||
|
they can not be filtered on; this is up to the user to do manually. The purpose of
|
||||||
|
instrument filters is to ensure that songs contain or don't contain given parts, not
|
||||||
|
to granularly select the difficulty of said parts (that's for the players of the game
|
||||||
|
to do, not us).
|
||||||
"""
|
"""
|
||||||
|
|
||||||
with open(config["database_filename"], "r") as fh:
|
with open(config["database_filename"], "r") as fh:
|
||||||
@ -587,24 +771,100 @@ def search(_filters):
|
|||||||
pending_songs = list()
|
pending_songs = list()
|
||||||
|
|
||||||
for song in all_songs:
|
for song in all_songs:
|
||||||
if len(_filters) < 1:
|
add_to_pending = True
|
||||||
add_to_pending = True
|
song_filters = _filters
|
||||||
else:
|
song_information_filters = list()
|
||||||
try:
|
song_instrument_filters = list()
|
||||||
add_to_pending = all(_filter[1].lower() in song[_filter[0]].lower() for _filter in _filters)
|
|
||||||
except KeyError:
|
if len(_filters) > 0:
|
||||||
click.echo(f"Invalid filter field {_filter[0]}")
|
# Extract the instrument filters
|
||||||
exit(1)
|
for _filter in song_filters:
|
||||||
|
if _filter[0] == "instrument":
|
||||||
|
song_instrument_filters.append(_filter[1].lower())
|
||||||
|
else:
|
||||||
|
song_information_filters.append(_filter)
|
||||||
|
|
||||||
|
if len(song_information_filters) > 0 or len(song_instrument_filters) > 0:
|
||||||
|
# Parse the information filters
|
||||||
|
if len(song_information_filters) > 0:
|
||||||
|
try:
|
||||||
|
pending_information_filters = list()
|
||||||
|
for information_filter in song_information_filters:
|
||||||
|
filter_field = information_filter[0].lower()
|
||||||
|
filter_value = information_filter[1].lower()
|
||||||
|
if re.match("^!", filter_value):
|
||||||
|
filter_value = filter_value.replace("!", "")
|
||||||
|
if filter_value in song[filter_field].lower():
|
||||||
|
pending_information_filters.append(False)
|
||||||
|
else:
|
||||||
|
pending_information_filters.append(True)
|
||||||
|
elif re.match("^~", filter_value):
|
||||||
|
filter_value = filter_value.replace("~", "")
|
||||||
|
if filter_value in song[filter_field].lower():
|
||||||
|
pending_information_filters.append(True)
|
||||||
|
else:
|
||||||
|
pending_information_filters.append(False)
|
||||||
|
else:
|
||||||
|
if filter_value == song[filter_field].lower():
|
||||||
|
pending_information_filters.append(True)
|
||||||
|
else:
|
||||||
|
pending_information_filters.append(False)
|
||||||
|
information_add_to_pending = all(pending_information_filters)
|
||||||
|
except KeyError as e:
|
||||||
|
click.echo(f"Invalid filter field {e}")
|
||||||
|
exit(1)
|
||||||
|
else:
|
||||||
|
information_add_to_pending = True
|
||||||
|
|
||||||
|
# Parse the instrument filters
|
||||||
|
if len(song_instrument_filters) > 0:
|
||||||
|
try:
|
||||||
|
pending_instrument_filters = list()
|
||||||
|
for instrument_filter in song_instrument_filters:
|
||||||
|
if re.match("^no-", instrument_filter):
|
||||||
|
instrument_filter = instrument_filter.replace("no-", "")
|
||||||
|
if song["instruments"][instrument_filter] is None:
|
||||||
|
pending_instrument_filters.append(True)
|
||||||
|
else:
|
||||||
|
pending_instrument_filters.append(False)
|
||||||
|
else:
|
||||||
|
if song["instruments"][instrument_filter] is not None:
|
||||||
|
pending_instrument_filters.append(True)
|
||||||
|
else:
|
||||||
|
pending_instrument_filters.append(False)
|
||||||
|
instrument_add_to_pending = all(pending_instrument_filters)
|
||||||
|
except KeyError as e:
|
||||||
|
click.echo(f"Invalid instrument value {e}")
|
||||||
|
exit(1)
|
||||||
|
else:
|
||||||
|
instrument_add_to_pending = True
|
||||||
|
|
||||||
|
add_to_pending = all(
|
||||||
|
[information_add_to_pending, instrument_add_to_pending]
|
||||||
|
)
|
||||||
|
|
||||||
if add_to_pending:
|
if add_to_pending:
|
||||||
pending_songs.append(song)
|
pending_songs.append(song)
|
||||||
|
|
||||||
click.echo(f"Found {len(pending_songs)} matchin song files:")
|
click.echo(f"Found {len(pending_songs)} matching songs:")
|
||||||
|
click.echo()
|
||||||
for entry in pending_songs:
|
for entry in pending_songs:
|
||||||
click.echo(
|
click.echo(
|
||||||
f"""> "{entry['artist']} - {entry['title']}" by {entry['author']}..."""
|
f"""> Song: "{entry['artist']} - {entry['title']}" ({entry['length']}, {entry['genre']}) from "{entry['album']} ({entry['year']})" by {entry['author']}"""
|
||||||
)
|
)
|
||||||
|
|
||||||
|
instrument_list = list()
|
||||||
|
for instrument in entry["instruments"]:
|
||||||
|
instrument_list.append(f"{instrument} [{entry['instruments'][instrument]}]")
|
||||||
|
click.echo(
|
||||||
|
f""" Instruments: {', '.join(instrument_list)}""",
|
||||||
|
)
|
||||||
|
|
||||||
|
click.echo(""" Available downloads:""")
|
||||||
|
for link in entry["dl_links"]:
|
||||||
|
click.echo(f""" * {link['description']}""")
|
||||||
|
click.echo()
|
||||||
|
|
||||||
|
|
||||||
@click.group(context_settings=CONTEXT_SETTINGS)
|
@click.group(context_settings=CONTEXT_SETTINGS)
|
||||||
@click.option(
|
@click.option(
|
||||||
|
Reference in New Issue
Block a user