And it failed spectacularly.
We only needed a simple form, but we wanted to be fancy, so we used “nextcloud forms”.
The docker image automatically updated the install to nextcloud 30, but the forms app requires nextcloud 29 or lower. No warning whatsoever. It’s an official app, couldn’t they wait that it was ready for NC 30 before launching it? The newsletter boasts “NC hub 9 is the best thing after sliced bread” yet i don’t see any difference both in visual or performance compared to NC hub 2
Conclusion: we made our business to rely on nextcloud forms as a signup form, but the only reason we were using it was disabled who knows how many weeks ago.
You must log in or # to comment.
Wait, you update productions systems without running a staging environment? Or even checking the update notes and your installed apps? Also no backups? What kind of business are you running over there?
To be fair a certain security company was in global news for exactly that same send it behavior. Why waste precious resources on multiple instances? Investors hate waste. 😅
If I understand correctly, nextcloud automatically updated … which I didn’t think it would, normally. Maybe it’s a “feature” of the AIO docker image?
Yes no staging because it’s something used at most by 2 concurrent users, we were ok with 95% reliability (we discovered it was disabled after at least two weeks lol)
Otherwise we would just have signed up to one of the many cloud forms sites at $100/year
Backups daily but it’s unthinkable to revert something like nextcloud to a months old one
Subscribed to both newsletter and RSS feed to know about issues (the command to update the docker images isn’t automated but manually issued). The maintainer of the forms app is nextcloud itself so any incompatibility should have been written in red bold characters in the blog posts and newsletter.
The docker image automatically updated the install to nextcloud 30, but the forms app requires nextcloud 29 or lower.
Lol. Do not blame others for your incompetence. If you have automatically updates enabled then that is your fault when it breaks things. Just pin the major version with a tag like nextcloud:29 or something. Upgrading major versions automatically in production is a terrible decision.
They’re releasing a new version every two month or so and dropping them rapidly from support, pinning it with a tag means that in 12 months the install would be exploitable.
Now, I did directly to production because this is low priority stuff, but it would have happened even with a testing stage. I would have never noticed that the forms apps was disabled, the system disabled it without any notification.
You would expect that an official app supports the latest release, no?
This wasn’t an app released by a nobody in their free time, this is a main feature heavily advertised in their blog. Look by yourself:
https://nextcloud.com/blog/nextcloud-forms-to-keep-your-surveys-private/
It’s not unreasonable to get pissed when 6 months after that blog post it doesn’t support the latest release anymore.
They’re releasing a new version every two month or so and dropping them rapidly from support, pinning it with a tag means that in 12 months the install would be exploitable.
The lifecycle can be found with a single online search. Here https://github.com/nextcloud/server/wiki/Maintenance-and-Release-Schedule
Releases are maintained for roughly a year.
Set yourself a notification if you forget it otherwise.
Exactly, they have a release schedule, why their own plugin, that they’re heavily promoting as a feature, isn’t following that? If for some reason the forms app isn’t ready for that date, why not postponing the launch instead of having it broken for who know how many months?
It’s not a plugin made by someone else in their free time. They knew that by updating to NC 30 that feature that was marketed just 6 months ago would be disabled, so at least have the decency to write it in the release notes. I subscribe to the newsletter and the RSS for what, just enjoy the marketing buzzwords?
It’s like if Microsoft releases an operating system with a buggy and broken taskbar because of a rushed self imposed deadline and fixes it one year later.
Okay, let’s be angry at the company and frown a lot at what happened. Gurr, bad company, evil.
And now think of what you’d rather have - a working system, or a reason to be angry? If you have something that integrated with something else, lock it down at a specific version so you control the upgrade and know those versions work 100% of the time together. “Latest” is just asking for trouble - be it in a docker image, in dependencies or elsewhere. It’s absolutely not a “best practice” if it isn’t even a code smell or an outright bug. You could’ve had a slightly outdated version, which won’t be “exploitable” - you wouldn’t have enough time to exploit anything in that time, especially with smaller companies and obscure exploits.
Instead of putting out the fire, you could’ve been now looking into the upgrade, seeing on UAT or Test or whatever that forms aren’t supported, chilling till they are supported or complaining that they aren’t.
Upgrades breaking shit is like programming / devops 101, and a huge reason for technical debt in very old projects. Leaving all that to chance is just irressponsible.
the upgrade command is sent manually, it’s not automated and it’s not unattended. It would have made ZERO difference if i tagged 29 and then in test tagged 30. The upgrade would have not failed, it would have given ZERO warnings, I would have seen that everything worked as expected (because who tests the useless survey that is filled once a semester?) and I would have pushed the update to production.
Who tests the useless survey? Everyone with regression tests. Like dude, everything you talk about has been written “in blood” from years of hosting production systems. If the useless survey is needed, then write a test for it, or a testcase to manually try it. Don’t just upgrade, see that the app is up and push to prod, that’s not testing, that’s asking for trouble.
I’m not arguing that my procedure is right (it is not), I’m arguing that nextcloud rushed the release in an incredibly unprofessional way.
When it’s the last time that Canonical, in the hurry to release Ubuntu xx.yy broke snapd without?
When it’s the last time that Apple, in the hurry to release iOS xx, made it totally incompatible with iMovie and had no timeline on when it would be fixed?
When it’s the last time that Microsoft pushed a Windows update that disabled Microsoft office and left it unfixed for months?
Nextcloud decided to take care of the forms app. They decided to promote it as a selling point. They decided when to release the update. They decided to still push the update even if their own form app didn’t get any new release in 4 months and isn’t compatible.
They aren’t contractually obligated to release a new, indistinguishable version (except some new bits that make it incompatible) every quarter.
Not ready? Delay one week. Still not ready? Delay an additional week. It came out that it needs a complete rewrite and it will take months? Write a note in the blog saying that feature is no more a selling point but now it is deprecated/unmaintained/unsupported and it will be automatically disabled without confirmation.
As someone said, a good app is only late until it ships, a bad app is bad until it’s patched. Why being perpetually bad with updates that nobody is asking?
Backups and rollbacks should be your next endeavor.
I have daily Borg backups held for at least one year but the problem is that the issue came out at least two weeks ago and nobody noticed. It’s better to have nothing (customer gets error page when viewing useless survey that nobody is watching) rather to restore such a old backup (everyone loses 2-4 weeks of data)
Specify a Version Tag in docker compose and update nextcloud deliberately through the webapp, that way it doesn’t update automatically on a pull
Docker automatically upgrades if you tell it to by specifying “latest” or not specifying a version number. But it only upgrades if you issue the pull command or the compose up command. There are ways to start without a pull like using start or restart. So yes, there was warning and something you did actively told it to upgrade.
And it’s really bad practice to update any software without testing, especially between breaking/major version numbers.
Finally, it’s not uncommon for a platform to release its update and then the plugins or addons to follow. Especially with major updates that require lots of testing before release. This allows plugin/add-on makers to fully test their software with the release version of the platform rather than all of the plugin makers having to wait for one that may be lagging behind.
You again commented on the docker upgrade comment that I said it’s irrelevant. It’s really like saying “if you wore pants that day, they wouldn’t have done that, it’s your fault”. It doesn’t make sense to spend $1000 in operating costs to host a useless survey that gets 3 responses a year. If it breaks for a week nobody dies.
Focus on how they’re moving fast and breaking things, ok? It’s not normal that official plugins don’t support the latest stable release. It’s not an alpha, it’s not a beta, it’s stable. Stable means everything needs to work. Official plugins need to support the latest stable release. It’s acceptable only if this was a third party plugin made by a hobbyist in their free time
WordPress updates also break many plugins but it never happened that a stable release blocked official ones like woocommerce.
By the way, now I have learned that the latest version of nextcloud is a public beta and it’s better to always stay one version behind. So why don’t they call it public beta?
Never upgrade to the latest and greatest of … anything really, especially in production. Let others test it first, or as suggested already, have a staging environment where you test the upgrade first. I guess you can still downgrade nextcloud though, especially if you have a backup.
Are you using the AIO image? I don’t know how well that works, but yeah, I absolutely hate automatic updates like that. I tried it once and I decided to use the plain “official but not supported” docker image instead, where I manage things myself. Never had an issue, and I can control which version I’m running, I can backup to wherever I want, using whichever system I want, etc.
AIO has a updater but it is manual by default You need to enable automatic updates yourself, which… Is done through a bash script you need to add yourself into the system crontab
And not only that, the instructions do say things could break and even suggests setting up backups for such
There was a recent related discussion on Hacker News and the top comment discusses why this sort of solution is not likely to be the best fit for smaller organizations. In short, doing it well requires time and effort from someone technically sophisticated, who must do more than the bare minimum for good results, as you just learned.
Even then, it’s likely to be less reliable than solutions hosted by big corporations and when there’s a problem, it’s your problem. I don’t want to discourage you, but understand what you’re committing to and make sure you have adequate buy-in in your organization.
That reminds me of work. I’m old, young me has been through the mistakes and the pain of wanting to control and self-host everything.
Now I manage a team of young idealists who have not yet been burned sufficiently hard by reality and I feel like I spend half of my time denying them permission to add new self-hosted services to our stack.
Just last month a young padawan was pissed at the spent on an external auth service and had been pushing hard for a self hosted OSS solution which he was convinced he could handle by himself (which was most likely true, from a purely technical standpoint).
Since he wouldn’t let it go, I “punished” him by having him spend one day in excel and powerpoint to prepare a cost benefit analysis to present to the architecture review board, including server cost, backups, redundancy, security, monitoring, pen-testing, auditing, his time and all the bells and whistles we needed to be compliant with all the ISO-x we have to be. (we’re in a banking related field).
Our estimated internal cost ended up about 6x the one of the SASS solutions and still wasn’t as reliable.
Most people don’t understand the amount of effort it requires to run a secure & reliable system and if I had a dollar for everytime I heard it’s as simple as “docker run”, I could retire early.
The forms app is useless. It’s basically for surveys. I can’t see how you’d use it for signups.
I wrote signups but I meant survey (in another comment I wrote “I would have never checked the useless survey”)
Docker is kind of a giant mess in my experience. The trick to it is creating backup plans to recover your data when it fails. As such, I don’t really recommend it to anyone at all.
I wouldn’t recommend Docker for a production environment either, but there are plenty of container-based solutions that use OCI compatible images just fine and they are very widely used in production. Having said that, plenty of people run docker images in a homelab setting and they work fine. I don’t like running rootful containers under a system daemon, but calling it a giant mess doesn’t seem fair in my experience.
Docker is kind of a giant mess in my experience. The trick to it is creating backup plans to recover your data when it fails.
Thats the trick for any production service. Especially when you do an update.
