The problem we are facing at my company is: we have a federated Jenkins setup where each team got their own instance and my team managing Jenkins updates, etc.

You don't know the pain we suffer from this plugin ecosystem. Installing a bunch of plugins which depends on a bunch of other stuff, there's no real dependency management for this and it's thoroughly painful when one of the teams decide to upgrade one of their plugins and break their whole setup because of it.

It has happened before even with basic plugins like "Git", one upgrade completely broke more than 50% of our instances (we are running about 50 instances right now) because a lot of other plugins relied on it and then needed to be upgraded, these upgrades then broke other plugins which depended on them and this started a domino effect.

Also, some plugins are just released and never taken care of, we have to maintain about 10 forks of plugins because there's no curation about them.

We came to the point where Jenkins flexibility by itself is a liability.

I know that Jenkins is wonderful for smaller teams but there's a point where the lack of governance really hurts and it's the main burden about it.

But of course, that's a given for any flexibility, too much of it and you have to design boundaries to contain problems... I'm just ranting because after this stint I'm burned about Jenkins, to the point where I don't want ever in my life to have to work with or use it.

The core plugin system is painfully broken.

It's like kernel panics: if an application causes the kernel to panic, then the more fundamental bug is in the kernel, not the application. If upgrading a plugin through the UI can cause other plugins to break because of mismanaged dependencies, then that's not really a bug in the plugin, but in the plugin management scheme.

That Jenkins does not forbid administrators from upgrading through the UI plugins which are fixed dependencies of other plugins without at least a warning message about the possible damage to the instance, is a critical defect, one that has gone unfixed practically since the creation of the tool. And so it is a defect which should immediately disqualify the tool from consideration in any large environment where the need to maintain critical updates (security etc.) while keeping uptime high is crucial.

You can manage plugins by dropping jar files in the plugins folder. It sounds like you have 50 teams with their own Jenkins which they have administrative control over, leading to chaos. This doesn't surprise me. Have you considered letting your team also manage the plugins and locking them all to a specific version?

Exactly this, we let they have full administrative power before because each team had their own little collection of plugins they needed and we couldn't manage the migration out of the "big" old master while also having to curate their plugin collections.

Our current discussion internally is about where to set their boundaries, remove administrative access and curate a set of plugins we are comfortable with maintaining (and that makes sense for most of the teams).

> You don't know the pain we suffer from this plugin ecosystem. Installing a bunch of plugins which depends on a bunch of other stuff, there's no real dependency management for this and it's thoroughly painful when one of the teams decide to upgrade one of their plugins and break their whole setup because of it.

That problem is called Java package management, Maven just plainly sucks compared to composer and npm.

At my company we solved the problem relatively easy though: a Docker image that compiles in all the plugin jars (it downloads them via mvn). Upgrading is usually easy (you just write the new version numbers from the "Available updates" screen into the pom.xml), but sometimes does takes one or two hours of wrestling when you have plugins that introduce new dependencies (which mvn cannot detect).

Why mavens sucks? What are its weakness?

Hey piva00, my name is Tracy and I'm James' colleague at CloudBees. I think the project my team is working on may help you with the pain you're describing here.

We've made a curated distribution of key integrations and plugins for Jenkins and this distribution protected our users from the breaking upgrades related to SCM API 2.0, which includes the Git plugin upgrade you mentioned here.

If you're interested, send me an email (tkennedy@cloudbees.com) and I'd be happy to talk to you more about the problems you're experiencing with the plugin ecosystem.

What is the reason for choosing this model?

I assume that the teams are still using the pre-pipeline jenkins jobs where they have to make the changes on the Jenkins server directly right? If you could switch to Jenkins pipelines with a single master, you could give that same freedom to your dev teams to manage their own build without having to manage a dedicated Jenkins master for each one.

No way! We switched to this model exactly because it's TOO painful to run a single master with thousands of jobs, you don't scale that at all, it's impossible to do any kind of upgrade because there's always someone using Jenkins for something quite urgent or important.

We have more than 500 engineers, it's seriously impossible to run a single master in this environment. We run both pipeline jobs and JobDSL (and some teams still rely on JJB).

There's nothing in Jenkinsfile/Pipeline that would help to mitigate any of the problems of scaling a master to hundreds of developers and tens of teams.

The dev teams all have control of their jobs through source control, and already had before the federation of instances.

I think the issue with a plugin ecosystem is that far too many 3rd party plugins do not respect the workflow of the core system that users expect to encounter - the UI appearance and functionality can easily confuse users, or even completely destroy the experience. Plugin integrations are often so poor that they ruin the product as a whole; particularly whenever a "top 10" plugin has serious bugs or flaws that prevent it from providing what a user really needs. Half-baked plugins that users depend on as a necessary evil are rarely a good thing. There is a reason that important functionality must be officially integrated into the platform: the core system must be presented as a unified and consistent interface.

Community plugins can be a godsend, but they can also be the exact reason why a software service falls apart.

There are many problems with the plugin ecosystem.

The big ones: * Many don't work with pipelines (scm sync config plugin for example). And you don't know this until you try * Upgrading a plugin can and often does have unintended, and hard to detect problems (only way to know is to run every job and find the failures) * It's hard to automate plugin installation