Danbooru Image tagger/renamer << NEED HELP!!!

[Kyrdua]

wee, i've tagged 70% of my gallery now. this thing works like a charm!



i haven't tried tagging with XP though.

UPDATE:  tagged with XP now, works perfectly!

[Jorin]

That looks awesome. I've been looking for something like this for a while, especially since I want to keep artist metadata in the files. I gave it a try, but I got this output:

Code: [Select]

D:\Download>python sheska.py 85c0fc90513b9a2ff6ccbd4f314eb329.jpg
Processing 85c0fc90513b9a2ff6ccbd4f314eb329.jpg
Checking for original hash...  hashing... Warning: Tag 'md5sum' does not exist
Nothing to do.
 ok
Fetching tags from the internets...
Retrying with hash implied from filename...
Warning: Tag 'md5sum' does not exist
Nothing to do.
FILE_NOT_FOUND

The weird thing is I downloaded this image right off Danbooru, so the md5 should be ok. Other images I've tested have pulled maybe 1 or 2 tags, but not all of them. I used the updated script with the tag and Exiftool is in my path, so I'm not sure what's missing.

[Kyrdua]

sheska uses sankaku to search for tags by default. you have to change the json url to this:

unparsedJson = urlopen('http://danbooru.donmai.us/post/index.json?tags=md5%3A' + image.hash)

it got "file not found" because some pics that are in sankaku are not in danbooru and vice versa.

[ph4zr]

Hm... it's not necro if I post after someone else bumped it, is it? Also, I apologize in advance for this huge friggin' wall of text.

Re: alt-text as file name (old): The problem with that is there seems to be a 255* character limit on file names. The total character length of all tags, combined with spacing, can easily reach well beyond that on over well tagged images. If you're going to go that route, it's also a pain to do it for each individual image if you're more than just a casual browser (i.e. if you batch download), so you'd want to automate the process somehow. At that point, you might as well just put the tags into the image itself, if it's supported, or store them locally in some kind of custom database.

(click to show/hide)

The exact number I hear varies, anywhere from 249(?) to 260, as well as there being an artificial X character limit on full path name using most programs to name files. The latter can be circumvented, although it's a pain if you're moving things using the default Windows copy handler, but I'm not sure about file names specifically. I also don't know if it's -just- a Windows limitation or a file system limitation.

and i think a photo organizer for imageboards is a great idea, definitely would be great if you'll make one.

I'll post here if I start working on it.

If you do that (or if you open a new thread for the project), please do make a note of it in this post so I see it in my updated topics. ...'twould be most enlightening (and useful). XD

(click to show/hide)

Also, if you ever happen to implement an open-source, full, standalone booru browser that supports downloading and tagging images, as well as filtering off images you already own, that'd be awesome! -hint hint- For boorus that support the API, it'd probably mostly be an issue of designing a functional GUI, since any code you need for pulling images and tag data could just be ported into a module and re-used willy-nilly. Boorus conveniently have a unique post-id for every image uploaded, so once you match those against your own images, existence is easy enough to verify for a given collection. That's just me being greedy (and lazy), though.

Heck, if you felt really ambitious, you could even have the browser pull the images from your local store rather than retrieving them remotely, for an ever so slight performance boost. But there are a lot of features I'd love to see in such a browser. That's just the tip of the iceberg.

The farthest I've gotten is extracting tag data for the files as saved by DD, and extracting tag data from Picasa .ini files, cross-checking file names between what I have and what I originally downloaded, looking for a match (as well as unmatched files). At some point I toyed with a perl script for parsing a file and adding tags to an image, or going in reverse and checking for existing tags, but I have since lost that script, so it's as if I never did it to begin with. Matching the pieces together is mostly an issue of modularizing the code and committing to a contract for each piece of functionality.

What I haven't actually tried is importing tag data into the images from the DB, or parsing file names for the original md5 hash (where it exists). The former because I've been using PHP for everything, the latter because my naming conventions have changed over time, and finding the hash is thus non-trivial at this point. In any case, I lost my previous image collection, so for now I've just been trying to rebuild it rather than (re-)tag everything or make sure I don't download duplicates.

Besides that, any GUI I touch turns into utter trash, so the best I could ever do is a PHP based web interface, or a really clunky and bloated Java one. PHP obviously requires the user has it installed (if you want to manage your own images), and clunky and bloated is... obviously worthless. The infrastructure is another story, and I already have PHP and a server installed for development, so for me that's a non-issue (beyond actually creating it, oy). Such a project would obviously never be fit to go live for personal use, though.

As I got sidetracked into mentioning in the previous spoiler... I've been toying with the idea of a photo retagger or tag manager to work w/ the Danbooru Downloader's** SQLite data. I didn't want to start the project during the semester b/c it'd suck me into its world and I'd never do my HW, but this summer I'd like to do something about my ~10GB collection of (untagged) images. If you actually started such a project, the code could potentially be re-purposed or modified to both scan over images for updated tags and check for updated posts to specific tag sets. Useful for finally finding out who that "character-request" actually belonged to, for example.

My project would mostly be focused on applying data I already have, whereas yours would be/is/seems to be mostly about getting data the user didn't grab in the first place (i.e., what you want to do is more complicated than just parsing text and importing modules)*. XD
*Outsourcing, FTW!

(click to show/hide)

If you haven't used it before, Danbooru Downloader is a Firefox addon for downloading from boorus. It stores all of the tagging information it recognizes from the site in a SQLite DB, making it a perfect resource for tag harvesting (well tagged images, anyway). So far the only problem I've had with it is that it doesn't detect ratings properly on gelbooru, making browsing images in mixed company potentially... interesting, to say the least. Well, that and having to remember to reset the "always show original image" flag on gelbooru after clearing my cache/cookies, as the "download full-size/original image" functionality apparently doesn't work properly there either. Instead it downloads the samples until I reload the page w/ the flag properly set.

Related to the "storing tags in file names" idea, it can also be configured to attempt to name the file with as many tags as will fit. I mostly find it useful for keeping tabs on characters and artists, and sorting into folders by copyright. Actually saving the tags can get rather messy, with only a partial match on anything too far down the tag list. Configuration for (limited) whitelisting and blacklisting, as well as special naming conventions for whitelisted tags, is also available.

One other "quirk" it has is that it stores tags in a single field, separated by commas. I'm not sure what the official specification for tag data is, but preserving any tags with commas in them requires you pay attention when you extract them. So far I've never found a tag starting or ending with a space, so I usually just look for that.

But I'm not so sure about pulling tag data directly from boorus for existing images that aren't already associated with a post (on the client side)—i.e. any file a user downloaded manually and renamed. Without the exact hash value you'd have a hell of a time figuring out what's what on a one-by-one basis. You could use iqdb to find the original image pretty easily for an individual, unmodified image—preferably sized down a bit for upload—but does iqdb have an API you can use to do it via a script? Managing any more than a few dozen images this way would be painstakingly slow.

Python, hm? Never got past "Hello world" with it, although you hear great things. I was planning on using perl for the exiftool package, and php for dealing with the SQLite DBs, dumping tag data into a more friendly format for parsing with perl, or just perl if I export to a csv format first. How do you handle tagging with python? It looks like a system call to the standalone binary version of exiftool, but I'm not familiar enough w/ python to rule out something else.

If you actually want -all- of the tags, and not just ones you whitelist (or don't blacklist), it probably wouldn't be too difficult to just have perl connect to said downloader DB and tag any supported images you downloaded via DD using the exiftool module. I'm a bit pickier about which tags I accept, though. Some I have no use for.

(click to show/hide)

PNG and GIF files are problematic, but Picasa's .ini files are simply formatted enough that adding -known- tag information is a fairly trivial task, provided you have a way of locating the relevant file. All the data is stored without any pathing information, so it's easy enough to find the relevant entry once you're in the right directory. If you want to keep any tags you already manually entered, it might be slightly more complicated, but not significantly so.

Mostly the issue is maintaining any special information Picasa put in there itself, but you can probably safely ignore those lines and copy them verbatim into the new .ini file. Doing it efficiently for crowded directories might be less straightforward. Going image by image could potentially result in thousands of reparses and edits to the same file if you don't group images by directory when you update tagging information.

Also (IMO) non-trivial is merging directories or moving files to new directories automatically. Due to the way Picasa stores information in .ini files, you would you to parse the relevant .ini files, updating them to add or remove entries from the list as you add or remove files from the directory, respectively. So far Picasa doesn't seem smart enough to detect that a file has moved and update the .ini files, and for some reason I've never had luck telling it to put all images from one directory into another without jumping through various hoops.

For other programs I'm not sure, since it would largely depend on how they handle tag data that can't be imported into the image itself.

(click to show/hide)

On a related note, I remember coming across a batch downloader for booru sites on one of the imageboard forums. It was written in perl, but horribly commented (i.e., not commented at all). The code itself looked simple enough that if you were familiar w/ HTTP requests in perl you could probably make a bit of sense of it. I've never used them, though (or OOP in perl, for that matter), so I got lost fairly quickly trying to add comments for eventual modification.

If I find it again I'll link to it if there's any desire. If I can't I probably have the code here somewhere, with whatever comments I added to it. The poster did note that it was broken for some sites, although I'm not sure which ones in particular. Probably gelbooru, for one... =/

[Jorin]

Hm... it's not necro if I post after someone else bumped it, is it?

Sorry if I was off base for bumping the thread. It was only on the second page and seemed reasonably new, so I figured I'd give it a shot.

Kyrdua, replacing that URL like you showed fixed the problem. It looks like I had confused something in the script previously, because there were two json url's. I also needed to replace instances of xmp:TagsList with xmp:Subject in order to get Picasa to properly read the tags. It works awesome now (except Picasa displays the tags in reverse-alphabetical order for some reason...).

Thanks a lot for the help! This will make it so much easier to manage my thousands of images. Any suggestions on how to feed multiple filenames into the command line aside from typing them manually? * wildcards seem to cause a problem.

[ph4zr]

Sorry if I was off base for bumping the thread. It was only on the second page and seemed reasonably new, so I figured I'd give it a shot.

Heh. No... I'm mostly just covering my own rear. Different forums probably have different guidelines for what constitutes necro, and I don't know where this one stands. On a faster moving board, someone would probably yell at you for bumping a thread from as recently as yesterday. On others, month old threads are still fair game. I've not discovered a hard and fast rule. /off-topic, but I thought I'd clarify

Edit: Plus, it isn't like you didn't have anything relevant to say, so it wasn't a bump for bumping's sake. Better to revive an old(er) topic than start a new one that's exactly the same, and all.

[Jorin]

Cool, good to know. I'm going to try out that Danbooru Downloader add-on because I've been interested in doing something like this with music files too.

If a stand-alone browser does end up being in the works, I agree about allowing it to pull data locally, not only for the performance increase but also simply so you can get your existing collection up to speed. It would be feasible for it to access any or all of the main databases (Sankaku, Danbooru, etc., just switch them on or off when scanning) to see which files you've downloaded come from where.

Another feature that would suit this project really well would be including the tag types information in the browser. This is something normal image browsers like Picasa can't do, but it's really useful in this kind of image collection. You want to know what's an artist tag, and what's a copyright tag for example.

[ph4zr]

see above

(click to show/hide)

Just from looking at the tagging guidelines on booru sites, I'm led to believe that those "tags" are actually stored with their type prefixes. The JPG tag metadata equivalent would simply be to add a pre-determined prefix to special tag types, such as "copy:", "char:". Pools (~Picasa Albums) I'm not sure about, but they would likely either have their own table-space or also use a specially formatted tag, such as "pool:###" or "pool:{name}". The latter seems more likely, since tags are already being queried for the image, and looking up the post ID in a different (likely highly populated) table that isn't even guaranteed to have a hit seems like a serious waste of time.

Prefixes are certainly the most straightforward way to approach it in Picasa, and they're easily searchable—you could even search for all copyright tags by using "copy:" or "copy:*" as a search string, if the software supported tag listing like that. The only immediately obvious downside is that it's unsightly. Actually querying for specific tag types from the booru might be slightly more difficult, as I'm not sure if the API will tell you what's what. If you go the route of HTML parsing, the tag type would be indicated by the class of the element wrapping the tag in question, just because styling requires some form of identifier for application to appropriate elements.

DD does not appear to save tag types in its DB, although it still applies them to templates properly. Looking at its source, it seems to be parsing HTML directly rather than using the API—which makes sense given its nature as a browser extension. Someone with sufficient time and motivation to deal with unfamiliar code could probably alter DD's functionality enough to hide images with known post IDs, though I'm not sure if this can be done early enough to prevent the browser from fetching the image previews. However, this would still require you to have downloaded the image using DD previously, and would not work with existing image collections. Still, a thought for anyone familiar w/ browser extensions. The code is modular enough that the rating problem w/ gelbooru could be fixed independently without breaking other functionality.

(click to show/hide)

About checking different sites: if they use the exact same image—i.e., if tag data hasn't been modified and the images haven't been edited—you could use the md5 hash to identify them. Or if the site hashes the image data independently of file data, if that's even possible, the md5 sum used by one site might still be identical on other sites—I honestly don't know how JPEGs and other image formats are laid out, so maybe it's not possible or is just too much of a hassle. In any case, the post ID is pretty much guaranteed to be different on each site and you'd have to use an identification service if the images are different in any detectable way.

Anything more advanced than a simple hash check, name/ID check, or bit by bit comparison I wouldn't know anything about. Image identification is best left to the realm of the graphically inclined and AI, IMO. I couldn't even begin to guess how such services detect image similarity if they aren't using that or tag comparisons. If someone -had- such a module it could probably be incorporated, but you'd need to fetch the image for comparison at that point, anyway.

Personally I don't mind having duplicates from different sites, just because it's too much work to go through 10k+ images looking for them, and it's easy enough to just ignore them or delete them as I see them. But I admit it would still be -nice- to be able to find duplicates and prune them or set them off to the side.

Danbooru Downloader fix for Gelbooru ratings
||NOTE: If you downloaded any of these files before the 2011.04.27 or the last edit time, whichever is older, you need to get the updated versions, because I was an idiot and packed the wrong folder. If I did it again, let me know. And hit me over the head with ... something soft, please.

(click to show/hide)

Edit: Hm... I tried re-installing the downloader, apparently it no longer de-compresses itself to its own extension folder. The only difference if it does NOT de-compress, is to unpack the "danbooru_downloader@cuberocks.net.xpi" archive to its own folder, and then perform the same edits. If you don't feel like repacking it (zip format, compression optional) and renaming it to an ".xpi" file, you can probably just delete the .xpi and Firefox should detect the addon in its folder. BUT I have to verify that last bit. Unpacking/repacking the .xpi is the same procedure as for the ".jar" file. In the meantime, I wouldn't modify the .xpi files unless you've done it before, since I haven't tried that method yet.

Edit^2: Just remember to delete either the unpacked folder OR the XPI file before you try to launch Firefox again. It doesn't like it when two addons have the same name... as I can attest after spending about 10 minutes trying to figure out why I didn't seem to have any. -facepalms-

Disclaimer: Back up the DD extension before you do any of this. I might have forgotten a step somewhere, or it might not work -quite- the same in Firefox versions less than 4. I.e. I'm not sure what "find updates" will do to the addon in older versions. This edit applies to version 1.7.1 that I found at http://www.brothersoft.com/danbooru-downloader-397279.html, which is the only version I've come across. If you find a newer one, for heaven's sakes LET ME KNOW! XD

Oh, right, another disclaimer: I haven't gone through all of the code, and I don't know anything about XPI installers. If it destroys the world or taints your soul gem (or both), I take no responsibility!

Short Version
|| if you have at least as much clue what you're doing as I did at the time

Edit "danbooru_downloader _tab.js" in "danbooru_downloader .jar\chrome\content\" in the extensions folder. Be sure to repack properly. Compression doesn't seem to matter, but if you accidentally repack with an additional folder level the addon won't function at all. After doing all that, force Firefox to re-validate the addon by checking for an update and restart.

If your editor doesn't support line numbering, you can just search for the contents, there should only be one hit.
Original code: danbooru_downloader _tab.js || LINE 329

Code: [Select]

 			var details = this.tabDocument.getElementById('stats').childNodes[3].childNodes;

Replacement code: danbooru_downloader _tab.js

Code: [Select]

		//MODIFICATION|ph4zr BEG
			var stat_section = this.tabDocument.getElementById('stats').childNodes;
			var child_x = -1;
			for(var x=0; x<stat_section.length; x++)
			{
				if(stat_section[x].tagName == "UL") {
					child_x = x;
					break;
				}
			}
		//MODIFICATION|ph4zr END

		//MODIFICATION|ph4zr LINE
		//REPLACEMENT
			var details = this.tabDocument.getElementById('stats').childNodes[child_x].childNodes;
		//ORIGINAL
			// var details = this.tabDocument.getElementById('stats').childNodes[3].childNodes;

There is probably a better way to do all of that, but I was trying to stick as close to the original coder's style as possible, with as few edits as possible. If I had my way I'd probably change some of it to regexps and go through commenting all the code. Most people don't look at the source, though, so comments would likely just bloat the addon—mine would literally almost double its unpacked size if I took the time to do it.

Long Version
|| if you aren't really sure where to find stuff, or aren't very comfortable with archives

If you want gelbooru ratings to be identified properly, you can modify the extension according to these instructions. You will need 7-zip or an equally functionally archive/ZIP program, a text editor for saving in -plain- text (I prefer Notepad++ on Windows), and the ability to find DD's extension folder. You -should- be able to find it on Windows at:

%appdata%/Mozilla/Firefox/Profiles/__USERSPECIFIC_DIRECTORY__/extensions/danbooru_downloader@cuberocks.net

The %appdata% string is an environment variable that should de-reference properly if you just paste it into an explorer window. Obviously "__USERSPECIFIC_DIRE CTORY__" is a directory that changes per person, so you'll have to browse to it yourself. If that doesn't work, the -default- user %appdata% value is "C:\Users\__USERNAME__\AppData\Roaming\", keep in mind that "AppData" is a hidden folder by default.

Now that you're here, browse to the "chrome"* folder. Take a second to back up "danbooru_downloader .jar"**, just as a precaution in case something goes wrong. Depending on how comfortable you are with zip files (or extensions in general), you can either unpack the entire zip file or just extract "danbooru_downloader _tab.js" from "danbooru_downloader .jar\chrome\content\" for editing.

*I don't know why it's called that either.
**Assuming you have file extension display enabled. If you don't, you can enable it in Windows via "Explorer Menu/Tools/Folder Options | Tab:View", by unchecking "Hide extensions for known file types". The menus might be slightly different depending on your OS version.

/branch

If you did NOT unpack the entire thing, make your edits to the file, save it somewhere, and add it back to the right folder in the archive (I just drag it to the window). 7-zip will probably compress this, but it didn't seem to have any effect.

/alternate branch

If you unpacked the entire thing, browse to the file and perform the edits at the top. After you do this, add the unpacked "chrome" folder to a 7-zip archive in ZIP format with the "store" option for compression settings. You can probably compress it, but the original isn't compressed, so I figure "why tempt fate?"

If you did this properly, by default 7-zip will have created a "chrome.zip" file. Take a second to verify that "chrome" is the top level folder in the archive, if it isn't you need to make sure to repack the actual "chrome" folder, not its contents or the folder it's in. If you are repacking, make sure you're repacking the one you created when you decompressed the archive. Now rename "chrome.zip" to "danbooru_downloader .jar" and place it in location of the original "danbooru_downloader .jar" file. At this point you can delete the unpacked files.

/branch end

Having done that, -restart- Firefox and open up the addons panel. In Firefox 4, scroll all the way down to the bottom, you'll notice that "Danbooru Downloader" is disabled as "incompatible". Just right-click and click "find updates". This won't actually do anything, but Firefox will do whatever it does and ask you to restart. Once you do that, the addon should work properly.

Lazy Version
|| for those who are just lazy bastards (like me) and/or have a high level of trust (not like me)

Assuming you're fine with not knowing what you're putting into your addons folder, you can just download "danbooru_downloader .jar" directly from the following link and place it in the appropriate folder (detailed above in big bold letters) the XPI file and install that to Firefox the same as any other extension. ... after you back up the other one in case things go horribly, horribly wrong. Alternatively, you can download the modified source file and replace it in the appropriate archives.

XPI file: Install this directly
http://www.megaupload.com/?d=YPH7FW9N
JAR file: Replace the original JAR file in the DD extension directory
http://www.megaupload.com/?d=296A4WOV
ZIP archive of modified source file: ...if you're downloading this, surely you know what to do with it.
http://www.megaupload.com/?d=61PTB8KR

[Jorin]

(click to show/hide)

I didn't actually know there was a standard way to store tag types/prefixes. This is very cool. There would be advantages to using both internal (prefix:tag) and external (A large table such as http://danbooru.donmai.us/tag) methods for this. Internal metadata is obviously more portable and easier to migrate because it can move with the image and be scanned into a database. That's helpful with well-established fields or prefixes that will be useful in a lot of different settings. An external metadatabase would be more helpful for specific uses and niche information that you may not want to store directly in the file.

Exiftool's readout of the subject field in images I downloaded from Danbooru and tagged with Sheska didn't show any prefixes among the tags. They were just simply the tags by themselves. I wonder if this actually means Danbooru reads from a table to provide the information for tag types/prefixes rather than reading prefixes from the image files themselves.

For duplicate detection, unless there's a simple and effective (and free!) multi-format algorithm for edge detection and RGB values I'd be more inclined to leave it up to the experts. I'm with you in that I don't really need to worry about duplicates that much. I just delete them when I see them.

I noticed that DD didn't pull the tags when I tried to use it. The subject field is empty when I read it with Exiftool. Should it be tagging the images, or just only downloading them?

It would be incredibly useful to link the functionality of DD and the Sheska script (or just Exiftool if that's possible) if the latter situation is the case, that way you'd get images downloaded into organized folders, and also searchable with the internal tags in an image viewer.

(click to show/hide)

I noticed DD has an option to control which sites trigger it, and that information ultimately directs images to different local folders. If it's possible to run Sheska or a Sheska-ish thing on an image automatically after it downloads with DD, it should also be possible to slipstream the correct json url for whichever site the file came from, that way the tags could be included automatically from their source as well.

If a standalone *booru viewer is in the cards, colour-coding the prefixes and making sure they're searchable would be great. Blacklisting and whitelisting is already accounted for within DD, so various tags could be filtered. Also, image viewers like Picasa should allow you to detect unwanted tags and mass-remove them from your collection as well.

It looks like almost all the ingredients are here to make a useful app. What's missing still is a process to mass-tag a bunch (like, hundreds or thousands) of files already downloaded, provided they're not renamed and/or they have their original md5 hash.

[Tiffanys]

I use Tagbooru to download whole gobs of junk: http://code.google.com/p/tagbooru/

[ph4zr]

I use Tagbooru to download whole gobs of junk: http://code.google.com/p/tagbooru/

I tend to shy away from batch downloaders, just because I like to see what I'm putting in my library. There are some images I'd just as soon not add, that would be incredibly hard to find if I batched all of Touhou... as I'm sure I'd do first thing.

(click to show/hide)

I didn't actually know there was a standard way to store tag types/prefixes.

Well, what I was talking about is just the convention the boorus seem to use to determine tag types. "copy:*" is just how they tell you to tag images when you upload or edit images. If the booru just has a separate table where it stores tags of a certain type, they might not be flagged or prefixed in any special way when you request them via API.

One reason for separate tables would be that it's likely more efficient to search copyrights if they're all in one table than it is if you're doing pattern matching on literally thousands of tags.

I noticed that DD didn't pull the tags when I tried to use it. The subject field is empty when I read it with Exiftool. Should it be tagging the images, or just only downloading them?

DD doesn't do image tagging. I'm not sure if Firefox extensions can be given file modification privileges, so it might not even be possible to tag them as you download them. Then there's the issue of finding a Javascript module that understands image tagging in the first place.

It would be incredibly useful to link the functionality of DD and the Sheska script (or just Exiftool if that's possible) if the latter situation is the case, that way you'd get images downloaded into organized folders, and also searchable with the internal tags in an image viewer.

You could probably write a script to grab the tags from DD's SQLite DB and pass them to Exiftool. If you're saving the md5 hash in the filename, Sheska should be able to successfully look up the file information. Alternatively, DD stores hash and site information in its SQLite DB, so you could look it up there, too. At that point you're already interacting with the DB, though, so it'd probably be easier to use Exiftool.

Unless of course you need updated tags. Poorly tagged or less recently downloaded images would be good candidates for a re-lookup.

I noticed DD has an option to control which sites trigger it, and that information ultimately directs images to different local folders. If it's possible to run Sheska or a Sheska-ish thing on an image automatically after it downloads with DD, it should also be possible to slipstream the correct json url for whichever site the file came from, that way the tags could be included automatically from their source as well.

You'd have to be monitoring the folders for changes to detect new files. I've never used it myself, but with logging enabled you might be able to pull new file names from there, which is probably easier than monitoring entire directories. Well, I think it is. I've never actually tried monitoring a folder for changes. Picasa seems to do okay detecting new images, though.

Sorting into folders based on site is also just a default setting in DD. You can disable it so all images get lumped together.

If a standalone *booru viewer is in the cards, colour-coding the prefixes and making sure they're searchable would be great. Blacklisting and whitelisting is already accounted for within DD, so various tags could be filtered. Also, image viewers like Picasa should allow you to detect unwanted tags and mass-remove them from your collection as well.

I didn't think my original white- and black-list comment through entirely when I phrased it as such. I actually meant I'd like it if such a tagger would ignore tags for metadata purposes, i.e., keep the image, but don't tag it with certain tags. Mainly I'd want it because some tags I don't have many images for, and I'm not a big fan of orphans. On the topic of DDs white- and black-listing, the only problem I have with that is it limits how many tags you can list.

/rest is down at bottom under "/general dev"

It looks like almost all the ingredients are here to make a useful app. What's missing still is a process to mass-tag a bunch (like, hundreds or thousands) of files already downloaded, provided they're not renamed and/or they have their original md5 hash.

I haven't looked at Sheska yet. If a filename has the md5 hash (and no really random 32 character A-Za-z0-9 sequences), it should be easy enough to go through a list of files and extract it from the name. DD seems to rely entirely on text parsing and the DOM, which is useful, but I think using the APIs (when available) would allow for faster and more efficient downloading/re-tagging. If a site doesn't expose its API that's another story, but I don't know if that's true of any of the booru sites at the moment.

Then of course there's the ethical issues surrounding code re-use. If it's all open source it's no big deal, but I'm not sure if DD is released under such a license. The gel-rating's fix was minor enough I didn't see an issue, but technically I probably should have asked for permission. I haven't seen any newer versions, though, so I'm guessing it's either been abandoned or deemed "final"*.

*IMO, no project is ever actually "final" or "done". At most they reach the end of their development life-cycle and are no longer worked on.

/general dev
The main issue I have with such a thing is that I'm horrible with front ends. Well, actually, I'm not that slick in general to begin with, but my front ends tend to fall noticeable lower on the quality scale. The other issue is my actual follow through is terrible. I remember posting months ago that I'd like to take a look at a piece of code for generating OST offers... I think I opened it once, commented two lines, and haven't looked at it since. x.x I probably should take a look at it again, while text parsing and HTML generation is still "fresh" from this semester's coursework.

Even assuming I got as far as starting it, once I had a working script or makeshift API for my script to interact with, that can be tweaked as I need it, I'd probably start to lose focus and go play with something else. =/

School is getting out soon, though, so I dunno. I still have to get around to tagging my own library, so at the very least I'll be trying to use the DD SQLite DB, and probably interface with and/or re-purpose Xiong's Sheska a bit. At this moment I have Xiong's Sheska, tagbooru, and the old DLer I found in the danbooru forums, as well as DD itself and some kind of JS... thingy, to look at. There's no lack of code inspiration, I'm sure.

[Jorin]

Just a quick reply because I'm busy applying for a job (in user experience design!), I'm interested in user interfaces and I'd have fun working on this. It wouldn't have to be extremely powerful, just slick and functional enough to do the job well. New features could be added over time anyway.

[ph4zr]

Hmm...? What kind of languages would work best? The only ones I'd be comfortable drafting it in are Java, PHP, perl, or possibly python if I can pick it up easily enough. C++ for an eventual port, maybe, but I'm not sure I'd want to start it there b/c there's so much more I can screw up. I haven't dealt with pointers or OOP in C++ in literally years, nor have I used it for any kind of non-trivial application or network communication. Then there's the fact it's easier to port the others to other platforms than it is a C++ project. I doubt most would want to compile the source themselves, after all.

(click to show/hide)

Java is probably the most portable, but also has the most overhead. C++ is probably the fastest, but its compiled code is also the least portable. PHP, perl, and Python are somewhere in the middle, with each having their own strong suits. AFAIK all of them have the necessary modules for doing most of the tasks, although PHP and Python may have to use system calls to exiftool to actually tag images. There probably is an app module to do it directly, but I haven't come across one.

What would be best for GUIs I have absolutely no idea. Since you do that kind of thing, your perspective will probably differ. For me at least, all of my Java interfaces have been godawful ugly, but I'm not a very graphic-y person, either.

Honestly I'm not very dynamic with languages. I stick to what I've used for the most part. Still, if there's another language you had in mind, it never hurts to learn something new, either. And there are almost certainly users on the forum who have the necessary expertise to do it in some language of choice, though AFAIK xiong is the only one who has expressed even potential interest in a similar project, with fairly low priority at that.

If a project did get started, it might be appropriate to either create a dedicated thread for it or move the discussions/planning to a different environment. If it doesn't violate site rules, maybe they wouldn't really care, but there's still the fact that trading code via DL links isn't very efficient. I've never attempted a collaborative project, so I wouldn't know how to go about it, but it can't be too too hard to setup github or svn. /famous last words?

In any case, I couldn't even think about doing anything at all for at least another week. Posting in forums is a tad less time consuming than coding. ...the latter of which I should already be doing for my final... =/

[Jorin]

I figure it's good to be as lightweight and portable as possible, and it would be cool if it could work on mobile devices as well (kind of like the Pixiv app). I think I'd rule out Java and C++ to start at least. I'm not sure whether PHP could do this on its own. Perl and Python might be the best for compatibility. Whichever can do this without needing Exiftool would be a big help, since exif metadata isn't present in anything besides .jpg files. There are a lot of .png and .gif files in this collection and it would be necessary to write metadata for them too.

My problem is I'm good on the research and design side, but I have a gaping hole in my skillset when it comes to actually programming. I could prototype it and design the overall architecture and interface down to the mouse-click, but I wouldn't know where to start for actually building it. I'm starting to learn how to program, but it's a bit of a learning curve for me.

But for now it's always possible to brainstorm. If and when it's possible to make this a project (perhaps if anyone else is interested too) it would be fun to get involved. It would be possible to at least get started with a wiki or something. Those are good for version control and note taking and are especially helpful for multi-person projects. No joke: wikis got me through grad school.

[ph4zr]

/formats
I don't really know enough about image formats to be familiar with the metadata PNG and GIF files can store. Exiftool is convenient because it provides a library for writing metadata to JPEGs. Even Picasa doesn't store any metadata in PNGs or GIFs, so my guess is there isn't an accepted tagging standard. A more general approach would be to just store the metadata externally, but then obviously it's not stored with the images themselves, making re-organization somewhat of a hassle.

/mobile|porting
Your comment about mobile devices perplexes me slightly. Is a smartphone or tablet type device appropriate for hosting a user's image library? If you are just browsing that's one thing, but if you are also downloading it seems you'd want a desktop environment of some sort.

I've never actually programmed for a mobile device before, but my understanding is they tend to differ a fair bit in terms of what your available options are for languages and design. If you were designing a mobile app I suspect you'd have to build it with a specific mobile device in mind, although the underlying logic could probably remain untouched.

Honestly I'd be happy just to get it working on Windows in whatever language was most convenient. Porting could come later, although designing with portability in mind would definitely simplify it. As long as you don't rely too heavily on system calls or language/platform specific libraries and functionality, everything else is just translation.
Which is obviously no small feat even when you understand both languages, for which fansub TLs provide more than sufficient evidence.

/experience
Experience probably wouldn't really matter, at least not to me. I'd approach it more as I would a hobby or learning experience than a job. Besides that, even just designing the interface would be a step up. What was or wasn't possible, or was simply too much of a hassle to actually implement, could likely be dealt with on a case by case basis.

I tend to start with a top down approach for design and prototyping and then implement it from the bottom up, so knowing what functionality the GUI requires would make it easier to design the back-end. I.e., knowing what you want to do makes it easier to figure out what you need to do to make it happen.

Heck, they actually pay people to design pretty looking interfaces, so it's not like "just" designing an interface is necessarily a trivial task.

[Kyrdua]

um... should i rename the thread title? while the stuff you guys are talking about aren't really off from the actual topic the current name really doesn't suit it. so... any thoughts?

/obviousy has nothing to contribute *shrinks in shame*

[ph4zr]

Not really sure what you'd name it, to be honest. So far it's still all hypothetical/speculative, so it isn't as if there's a particularly accurate way to describe the (as yet non-existent) project. Sorry if we hijacked your thread, though. =/

If a project actually gets started, a new thread probably will be created for the purpose. I probably should've since the first wall, but it was purely speculative and I didn't want to generate any expectations. << fairly poor follow through, as noted previously
I'd probably be the last to know, though, since I don't really watch the tech forum too too closely. Mostly I just auto-refresh recent posts and updated.

(click to show/hide)

'sok. I mostly just get carried away in the thinking process. My "skill" in programming is roughly comparable to my skill in image "manipulation" as evidenced by SSs, avatars, and sigs*. With images it's cut, paste, resize. With code it's... er, take code snippets of stuff I've already done, swap out irrelevant bits for relevant ones, and basically smash them together until they work the way I want.

Well, more or less. You have to create something to begin with, but everything I've done is mostly just a modification of "Hello World" and whatever you find in tutorials/forum posts, or a modification of a modification, et cetera. I.e., my "contributions" aren't really differentiable from those of someone who has no idea what he's talking about.

*ahahahaa, what sig!? << the point

[Kyrdua]

^ lol, i don't mind the hijack. the stuff you're planning sounds quite useful anyway.
but yeah, it could use it's own topic once you've made something that works.

[Jorin]

In terms of the thread name, the idea for the project literally is a Danbooru image tagger, though Image Board Browser/Tagger Program (Ideas) might be a better fit. Whichever works for you. It's true this is only still hypothetical, so it would probably only need to be renamed if anything got started. For now we're just brainstorming.

As for the project, I figure since we're at this stage, why not keep going with the top-down approach and figuring out which features would be best. As for the mobile thing, I perplexed myself as well upon reading it again! It would be incredible for browsing, but not at all useful for handling image collections. Thinking too far ahead here.

[qoute=kyrdua]/obviousy has nothing to contribute *shrinks in shame*[/quote]
Any ideas at all for how this project could be useful are welcome!

[Kyrdua]

▲ then in that case, it deserves a new thread, since it's still in brainstorming stage.