Adjust default structure again

README.md

@@ -29,8 +29,9 @@ with for command compatibility with a `pip` installation.
 
 ## Usage
 
-Before running a command, use the build-in help via the `-h`/`--help` option to view the available option(s) of
+Before running any command, use the built-in help via the `-h`/`--help` option to view the available option(s)
-the command.
+of the command. This option is available everywhere by virtue of the Click tool, so use it frequently to get
+a comprehensive understanding of all available options and how they work.
 
 The general process of using `c3dbdl` is as follows:
 
@@ -48,7 +49,72 @@ this relatively quick but YMMV.
 
 1. Download any song(s) you want with `c3dbdl [options] download [options]`.
 
-## Filtering
+## Database & Included Data
+
+The database is contained in a JSON document which lists all possible songs which were scraped from the C3DB
+pages during the `database build` step.
+
+To obtain the database, first the specified base URL is downloaded to get a list of pages, and then each page
+is iterated through. Within each page, all "song" table entries are extracted for information, and the song
+page itself visited to obtain a full list of download links. The song iteration is performed in parallel with
+a default of 10 simultaneous jobs (configurable with `-c`/`--concurrency`) to speed up downloading.
+
+Once all pages and songs have been scanned, the results are saved into the database file specified, which can
+then be reused for future downloads. Note that cancelling a `database build` before it is finished will result
+in an empty database and the process will have to be started again from the beginning.
+
+A database file cannot be updated; it must be replaced wholesale. You can however interactively edit your local
+database with the `database edit` command should you choose to do so (for instance, to normalize album names
+or similar).
+
+The contents of the database includes all information required for filtering and downloading as described below.
+An example entry (first entry on the first page) is:
+
+```
+{
+  "artist": "Heatwave",
+  "title": "Boogie Nights",
+  "album": "Too Hot to Handle",
+  "song_link": "https://db.c3universe.com/song/-34018",
+  "genre": "Pop-Rock",
+  "year": "1976",
+  "length": "0:05",
+  "author": "D97", 
+  "dl_links": [
+    {
+      "link": "https://dl.c3universe.com/642d6ab2aa5b87.10964554",
+      "description": "Rock Band 3 Xbox 360"
+    }
+  ]
+}
+```
+
+### Download Links
+
+The `c3dbdl` tool is very picky about the download links (`dl_links`) it selects. Specifically, it will *only*
+include links from `c3universe.com`, and not any other external "download sites" such as Mega.nz, Angelfire,
+etc.
+
+This is done because the non-iteractive, command-based download method is not compatible with those sites, and
+we want this tool to be as automated as possible. Requiring some manual clickthrough of a web page would defeat
+the purpose here, and thus, we simply exclude them and require you download any such songs manually.
+
+If a song ends up with no `dl_links` during scanning, for instance because they all pointed to such external
+"download sites", it will not be included in the database. Thus, the final number of songs in your database is
+guaranteed to be smaller than the total number listed on the C3DB website.
+
+## Downloading
+
+Once a database has been built, you can start downloading songs.
+
+By default, when downloading a given song, all possible download links (`dl_links`) will be downloaded; this
+can be limited by using the `-i`/`--download-id` and `-d`/`--download-descr` options to pick and choose specific
+files.
+
+Once a song has been downloaded, assuming that the file structure doesn't change, subsequent `download` runs will
+not overwrite it and will simply skip downloading the file.
+
+### Filtering
 
 Filtering out the songs in the database is a key part of this tool. You might want to be able to grab only select
 genres, artists, authors, etc. to make your custom song packs.
@@ -74,28 +140,22 @@ on, for instance `Rock` or `Santana` or `2012`. The text must be quoted if it co
 If more that one filter is specified, they are treated as a logical AND, i.e. all the listed filters must apply to
 a given song for it to be downloaded in that run.
 
-Filters allow powerfully specific download selections to be run. For example, let's look for all songs by Rush
+Filters allow very specific download selections to be run. For example, let's look for all songs by Rush
 from the album Vapor Trails (the remixed version) authored by ejthedj:
 
 ```
 c3dbdl download --filter artist Rush --filter album "Vapor Trails [Remixed]" --author ejthedj
-```
+Found 19563 songs from JSON database file 'Downloads/c3db.json'
-
-This shouldfind , as of 2023-04-02, exactly one song, "Sweet Miracle":
-
-```
-Found 28942 songs from JSON database file 'Downloads/c3db.json'
 Downloading 1 song files...
-Downloading song "Rush - Sweet Miracle" by ejthedj...
+> Downloading song "Rush - Sweet Miracle" by ejthedj...
-Downloading from https://dl.c3universe.com/s/ejthedj/sweetMiracle...
+Downloading file "Rock Band 3 Xbox 360" from https://dl.c3universe.com/s/ejthedj/sweetMiracle...
+Successfully downloaded to ../Prog/ejthedj/Rush/Vapor Trails [Remixed]/Sweet Miracle [2002].sweetMiracle
 ```
 
-In addition to the above filters, within each song may be more than one download link. To filter these links,
+In this case, one song matched and was downloaded. Feel free to experiment with the various filters to find
-use the "-i"/"--download-id" and "-d"/"--download-descr" (see the help for details).
+exactly what you're looking for.
 
-Feel free to experiment.
+### Output Format
-
-## Output Format
 
 When downloading files, it may be advantageous to customize the output directory and filename structure to better
 match what you plan to do with the files. For instance, for pure organiation you might want nicely laid out
@@ -112,20 +172,26 @@ which are mapped at download file. The available fields are:
 * `title`: The title of the song.
 * `year`: The year of the album/song.
 * `author`: The author of the file on C3DB.
-* `orig_file`: The original filename that would be downloaded by e.g. a browser.
+* `orig_name`: The original filename that would be downloaded by e.g. a browser.
 
-The default structure leverages all of these options to create an archive-ready structure as follows:
+The default structure leverages most of these options to create an archive-ready structure as follows:
 
 ```
-{genre}/{author}/{artist}/{album}/{title} [{year}].{orig_file}
+{artist}/{album}/{title}.{author}.{orig_name}
 ```
 
-As an example:
+As an example, as shown in the previous section:
 
 ```
-Prog/Rush/Vapor Trails [Remixed]/Sweet Miracle [2002] (ejthedj).sweetMiracle
+Rush/Vapor Trails [Remixed]/Sweet Miracle.ejthedj.sweetMiracle
 ```
 
+The genre is excluded because in my experience it is a fairly useless metric and is often incorrectly set,
+so it gets in the way more often than not. You are free of course to add it in to your own custom structure.
+The date is excluded for similar reasons and because if you know the album, you know the date.
+
+If any field is missing during download, it is replaced with "None".
+
 Note that any parent director(ies) will be automatically created down the whole tree until the final filename.
 
 ## Help

c3dbdl/c3dbdl.py

@@ -436,7 +436,7 @@ def database():
     "--file-structure",
     "_file_structure",
     envvar="C3DBDL_DL_FILE_STRUCTURE",
-    default="{genre}/{author}/{artist}/{album}/{title} [{year}].{orig_name}",
+    default="{artist}/{album}/{title}.{author}.{orig_name}",
     help="Specify the output file/directory stucture.",
 )
 @click.option(
@@ -490,7 +490,7 @@ def download(_filters, _id, _desc, _limit, _file_structure):
 
     \b
     The default output file structure is:
-        "{genre}/{author}/{artist}/{album}/{title} [{year}].{orig_name}"
+        "{artist}/{album}/{title}.{author}.{orig_name}"
 
     Filters allow granular selection of the song(s) to download. Multiple filters can be
     specified, and a song is selected only if ALL filters match (logical AND). Each filter