Just want to say thanks for all the work you did and keeping it going as long as you did.
Posted under General
Just want to say thanks for all the work you did and keeping it going as long as you did.
I do have a server to host this with a lot of free space and bandwidth.
I am not willing to support or upgrade current engine, though. @poweryoga if no better proposals, you can contact me, I guess.
EasyDeibu said:
I do have a server to host this with a lot of free space and bandwidth.
I am not willing to support or upgrade current engine, though. @poweryoga if no better proposals, you can contact me, I guess.
The space is not the problem, but having an up-to-date system is.
The version of danbooru used here is super old, to the point that the version of ruby required isn't even available for download anymore (not really, but we're getting there.)
Needless to say, as time goes on the probability of the software to be incompatible with the rest of the system (read: nothing will work) increases.
As Dosu said, either there is someone with enough know-how to keep track of the existing workarounds and somehow manage to keep the site on life support, or there is someone with enough know-how and time to attempt a migration to a more recent version of the whole software stack.
The ownership and the economic side of the thing seem to be taken care of, from what I understood.
EasyV said:
The space is not the problem, but having an up-to-date system is.
The version of danbooru used here is super old, to the point that the version of ruby required isn't even available for download anymore (not really, but we're getting there.)
Needless to say, as time goes on the probability of the software to be incompatible with the rest of the system (read: nothing will work) increases.
As Dosu said, either there is someone with enough know-how to keep track of the existing workarounds and somehow manage to keep the site on life support, or there is someone with enough know-how and time to attempt a migration to a more recent version of the whole software stack.
The ownership and the economic side of the thing seem to be taken care of, from what I understood.
I do have my own booru engine and it worked well in the past. Pools, links, translations, but no forum and no wiki.
If we migrate, it will be easy to support for me. But while you can migrate images and tags, user accounts are probably not possible.
May as well throw my two cents in here since it came up on the Discord too.
I've been trying to get the latest version of Danbooru to work and while it kinda-sorta does, I have a sinking feeling that upgrading just isn't going to work out. I count at least seven external services required so far; if I'm reading it right a bunch of stuff OYP's version of Danbooru handles internally seems to have been offloaded to various cloud APIs/CDNs. I guess that makes sense for Danbooru proper where they have the traffic and content to justify the extra work but for OYP it'd be a nightmare. If any one of the services breaks, or even just breaks compatibility, the site goes down and stays down until someone upgrades the software.
So, uh, migration to a different booru engine sounds like a good plan if we're gonna do this thing. IIRC OYP used to use Gelbooru before switching to Danbooru waaaaay back, but it was kind of shit back then. What's it like now? What other options are there?
I'm also working on something else but it's not far enough along to get anyone's hopes up yet.
If we migrate to another engine, we can probably create dummy accounts for contributors, and they can get access by resetting the password using same email. This way they will still be linked to their contributions.
OYP was on shimmie. Never on gelbooru. We went from shimmie -> Danbooru -> Danbooru 2.
And I haven't looked at the current version of Danbooru so I don't know how many services are now offloaded. If that's the case it'll be more complicated still to manage. Still the biggest problem is... well, updating the site. I'don't remember if I've mentioned this before but: If anyone can get a current version of danbooru up and running, migration should be less painful provided most of the databases haven't changed all that much. (For accounts it'd hopefully just be a "forgot your password" reset if we ever get it up and running so it'll update it in the DB.
Thanks for all the offers of space, bandwidth and hosting from everyone. It's truly appreciated.
poweryoga said:
OYP was on shimmie. Never on gelbooru.
S... strongest...
So, I know that migrating to latest Danbooru is the best option.
But I kinda have no interest in supporting ruby-based engine. If no one will be able to do it, my offer to migrating to my engine still stands. It will be easy to support for me going forward.
I would need to translate interface into English, and probably tweak css to feel more like home, but those are easy parts. There's no wiki and no forum, but they can be handled separately.
What if we were to use a newer-but-not-too-new version of danbooru?
I mean, since at some point it started offloading some stuff to third-party services, we could use a version released before it started doing that.
It's likely to be quite old, but it should still be newer (and hopefully more stable) than the current version we're on.
EasyDeibu said:
But I kinda have no interest in supporting ruby-based engine.
Yeah, that's my problem too.
@EasyV
I'd hesitate to do any sort of upgrade at all if it's not the latest. Reintroducing old bugs that have been fixed years ago is really.... not a good idea. Danbooru isn't a great repo to begin with and I have no interest in figuring out what versions of various libraries we need to support older versions of stuff. The version of danbooru2 we use is super old.
Fair enough. I was just stating possible solutions.
It's probably harder to do than a fresh upgrade since now you have to dig through supported versions of ruby, apache, psql, etc. At least with the latest repo you can just grab the latest stuff and it'll most likely work.
^ FWIW I can confirm that getting the latest (as of a month ago) Danbooru running on Debian Buster (or Devuan with the Buster repo enabled, if you know why you'd want to) is fairly straightforward. You don't even need to use the extra repos/build stuff from source like the example install script, Buster has new enough everything out of the box. You do need to symlink the yarn binary since Debian renamed it to avoid a conflict and everything from Albert's GitHub is mandatory, even the stuff with "test" in the name, but that's about it.
Proper setup + maintenence is way the hell outta my league though. All three (and that's not counting apt or git!) of the dependency managers broke at some point and they're not smart/documented enough for a non-programmer to un-break without a full reinstall (which I did a lot). Then there's the GitHub/AWS/Google API/CDN crap I mentioned above, plus I can't Ruby or SQL to save my life...
So yeah. If by some miracle there's someone reading this who thinks that sounds like fun, come say hi on the Discord or the Google Group. I'd love to make myself useful but right now I can just barely set up a static web server before I'm in over my head.
(Okay, I really gotta stop spewing my stupidity all over this thread. If anyone actually wants my help with anything OYP related just holler/@ me.)
Today, I spent the whole day trying to install Danbooru on a Debian 10 system installed in a virtual machine. The following is a short recap of what I did and discovered:
To begin with, the current install script is broken. Some package names are wrong so apt-get is never executed. The problem is that the problematic line in the script names a lot of packages so pretty much all the important packages are never installed. Ultimately, I ended up manually executing the steps that the script is supposed to automatize.
It appears that 3rd party services are actually optional. By default the configuration does list some endpoints, but doesn't specify any particular key or shared secret, so even if the system tries to set them up during deploy they are not actually used.
The only "real" dependency is Github, but that's only because deploying requires a git repository. It's 100% possible to keep said repository locally inside the same server as the actual thing. In fact, in our case that's probably the better thing to do if we proceed with this plan. Being just a configured danbooru installation, there's no need to depend on Github for our deploy.
While checking the configuration, I discovered a possible reason as to why the current incarnation of OYP can't switch from debug mode to production: the config/deploy/production.rb file lists a number of servers and their role in the system. For the actual Danbooru website it makes sense to have multiple machines, given that they have to serve a lot of users at the same time. Given that OYP is fairly small, setting just one of these servers as "localhost" should suffice. Thanks to poweryoga giving me the current source some time ago, I discovered that the OYP's production.rb file lists two machines called "sonohara.donmai.us" and "hijiribe.donmai.us" which most definitely do not exist. Not to mention the user to access those machines as is called "albert", which might or might not be an existing user in OYP's server.
Speaking of configurations, while I was looking the web for the purpose of some files, I ended up in a thread in Danbooru's forum. That thread was opened by poweryoga 9 years ago. It wasn't particularily helpful, but that's because they were talking about something specific that I didn't need at that moment.
Eventually I managed to reach the time to actually deploy it, but I had to stop due to some errors with the code dealing with the database. Unfortunately, I couldn't troubleshoot it at all and had to give up after a whole hour attempting things. For what is worth, it might've been caused by my previous failed attempts, so I decided to delete the virtual machine completely and to try again in a few days, after resting. The database error in particular was very frustrating and I'm not in the mood to have any more of that.
Once and if I successfully install the thing I will note anything that might be of interest.
The problem with the database was caused by a misconfiguration. Once that was solved that error disappeared.
Eventually I managed to deploy it, adding a couple of configurations that might or might not be needed if and when we do it for this place.
I had to place a few lines of code to disable New Relic, because otherwise the connection would've been refused.
The first problem I met with the new deploy is that new users are given an advice to updgrade their account to gold or platinum for some dollars. We don't really need this, but I don't know where to look to disable it.
Another one was caused by a lack of a certificate for HTTPS, but that's only because I was working with localhost.
I briefly used the new istance to upload an image and create a few things. Most basic features work as they are now, but on the other hand there are a lot of new UI elements for things that don't concern us.
Unfortunately, a number of old features that work currently require an AWS instance. I wonder if we can change something in the code to make it use the local database rather than AWS. I'll look into it. Other 3rd party services aren't really useful in my opinion given our scope.
EasyV said:
To begin with, the current install script is broken. Some package names are wrong so apt-get is never executed. The problem is that the problematic line in the script names a lot of packages so pretty much all the important packages are never installed. Ultimately, I ended up manually executing the steps that the script is supposed to automatize.
Uh, whoops, thought I mentioned that. Sorry. I ended up doing the same, that's why I could say Buster is up to date.
EasyV said:
Unfortunately, a number of old features that work currently require an AWS instance. I wonder if we can change something in the code to make it use the local database rather than AWS. I'll look into it. Other 3rd party services aren't really useful in my opinion given our scope.
And that's what I meant about things being offloaded. Hope you can find a fix.
I feel really bad for saying this but don't really have anything useful to add... you've made more progress in a day than I did in a week. I really am way out of my depth here. Was there anything in particular you wanted me for?
BaronMind said:
Was there anything in particular you wanted me for?
Maybe you have some more insights?
I only installed danbooru ex-novo, but I don't really know anything OYP-specific besides what I can gather from the source code poweryoga sent me.
I'm not really sure what could be useful, though... If you have anything I suppose right now it's better than nothing at all.
Uh, okay... OYP specifics that I can remember, some of which may not have survived the switch to Danbooru 2:
...sure there was more but all I can remember is cosmetic stuff like shii's avatar and the 404 koReimu. (BTW, does anyone still have the old site data? Might help here, plus I know some stuff like post versions didn't survive the move; nice to restore it if possible but not expecting miracles.)
Other than that, if you have the sauce and can read it you're way ahead of me. Hell, I think you probably know more about the site w.r.t. Danbooru than anyone else here. All I got is is the stuff I scraped off the site for my mirror project, and all THAT tells me is that OYP's database is a horrible mess (negative tag counts, invalid tag names, some private data publically accessible, some public data not publically accessible...).
BaronMind said:
- No ads/like buttons/upgrade nag messages/etc.
This is probably the hardest part to change, at least for upgrade messages.
For the ads I think without the necessary informations it simply won't work or something.
BaronMind said:
- Non-members can search more than two tags (don't remember the actual limit).
This is a limitation that exists even now, and I don't know if it can be changed. I don't think it can be changed, though it's very likely that registered users can be configured to have unlimited tags without being gold/platinum.
BaronMind said:
- No hiding downvoted comments (was being abused, IIRC the old site removed the voting buttons entirely?).
Again, downvoted comments are hidden by default for non-members right now.
The same as above apply.
BaronMind said:
- Comment bump limit raised (to what IDK but I remember being annoyed about the reset after the update).
If it's for registered users then it should be configurable just fine.
BaronMind said:
...sure there was more but all I can remember is cosmetic stuff like shii's avatar and the 404 koReimu. (BTW, does anyone still have the old site data? Might help here, plus I know some stuff like post versions didn't survive the move; nice to restore it if possible but not expecting miracles.)
Not sure about avatars, but the 404 (and other error pages) can be configured freely as long as we set up the server (current versions of danbooru use nginx) to display the appropriate page.
BaronMind said:
Other than that, if you have the sauce and can read it you're way ahead of me. Hell, I think you probably know more about the site w.r.t. Danbooru than anyone else here. All I got is is the stuff I scraped off the site for my mirror project, and all THAT tells me is that OYP's database is a horrible mess (negative tag counts, invalid tag names, some private data publically accessible, some public data not publically accessible...).
I see, thanks. Unfortunately the database is the most critical component here (obviously) so if there are some wacky data it might cause problems.
Ideally, we would need a couple of mirrors of the state the database is in when (and if) the migration happens, so that if things break we don't lose anything. However, in theory, danbooru has a "migrate" task that should "fix" or at least bring up to date the whole thing.