...in the community, I work at gcoop, which is a Free Software Cooperative,
meaning that we work exclusively with Free Software
and horizontally, as an organization.
We have no bosses, we have no employees.
Five of our partners from gcoop came to the event,
it's our first time at DebConf for everyone
We sponsored this event and give thanks to Debian
because we have 20 years working with Free Software
and it wouldn't have been possible without Debian.
I'm going to show an update of a talk I gave before in 2022,
which is about the GNU/Linux migration project of Filiales
from Banco Credicoop
Cooperativo Limitado, it's a cooperative bank.
And I'll try to quickly review the migration process
and then we'll look at the differences in recent years.
This project...
Basically it's a project of Infrastructure as Code
using Ansible and AWX.
AWX is a web platform, which is a graphical interface for Ansible,
but on the other hand it allows orchestrating and managing and administering
all the Ansible playbooks.
and this was done for Banco Credicoop.
from gcoop.
So, at the beginning of the project, what we had to do was, over
6 months,
was trying to discover what tools in the Free Software ecosystem
would allow us to carry out the entire infrastructure migration
of a bank
throughout all of Argentina.
And what we found was that the automation tool was
Ansible
because we were already using it.
That we would use GitLab.
That for ActiveDirectory authentication within Linux,
we would use IPA.
AWX, obviously all the servers would use Proxmox.
We had already been working for years with Proxmox.
The VMs inside the Proxmox servers would be Debian, obviously.
And so, for workstations, due to the fact that there were
3,000 workstations
and because of support reasons, we chose a Debian-based distribution
um...
A slightly more updated version.
And what we had there, it's not very visible, but what's moving is that
...underneath
all the components, it's a timeline of all the components we use
for dependencies
that are being updated worldwide, all the
time.
Meaning, at that point in a given month, a new version of a component
came out, along with another component and another.
And this evolves as you're thinking about how to approach things, the entire ecosystem updates.
the entire ecosystem keeps updating.
So it's a problem to solve.
But what we achieved is that in Development we managed to orchestrate an idea
of deployment,
which is basically: we have an AWX, let me point it out like this,
this is an AWX, and its source of truth is a GitLab,
so AWX reads from GitLab
everything that would be...
Meaning playbooks, and the first step is deploying the iDRAC.
The iDRAC is the computer of the computer, inside the servers
inside the Dell servers we use
before the server even has anything; there's a computer called iDRAC,
using a protocol called Redfish, we wrote playbooks that AWX uses to
connect to iDRAC
and what it does is partition the disk, configure BIOS and reboot into PXE.
That is, a new computer, fresh out of the box.
And immediately that will end up generating from a Proxmox,
it will ultimately generate another Proxmox.
Inside that Proxmox We have several VMs.
One of those VMs is a PXE server, I don't see anything here, but it must be this one.
We have a server, well we have a CDN, which is a processing server,
an Nginx that has all the resources, which in turn is a proxy for another,
an Apache,
and a server that replicates this data.
This is all done in an automated, unattended way, without
operator intervention,
meaning a server just comes out of the factory, the box is opened,
Power and Network are installed,
and from AWX a script is launched,
this is the AWX infrastructure, sorry, the Proxmox one, we were going to see,
you can see it here,
how it's launched, and this is just for monitoring, let's say,
everything is done in an unattended manner and the operator does not need to see
anything running
AWX deploys Proxmox via PXE; it's actually several stages,
first it deploys a Debian NetInstall
completely unattended, meaning there's nothing to touch
at all,
it goes directly into OEM mode, configuring everything needed for that
server to be operational,
and when it finishes, in a hook at the end of the Debian NetInstaller,
we wrote a script that takes the MAC address of that host, which
is unique,
and registers it in the AWX inventory.
So that host, which just received a random IP,
we now know what its IP will be, what its iDRAC is,
permanently, for all the years this server will have,
referenced to that MAC and to a serial number from the
iDRAC itself.
So, in this way it's easy to identify it.
On the left, all you see is the debug of what's happening on the
PXE server,
and as you can see, the server is on the PXE server,
the server installs itself with no manual intervention and finishes installed,
it's a Debian.
The next playbook launched on AWX converts that Debian into a
Proxmox,
also in an unattended way.
Meaning we don't install Proxmox directly; we install Debian and
convert it to Proxmox.
That's an advantage of Proxmox being a Debian-based distro,
where the only thing that changes is adding a repo.
And done, all the necessary packages are configured.
And when configuration finishes,
on a Proxmox, we launch another playbook that creates the VMs,
and another playbook that creates the services inside each VM.
All the VMs we use are KVM.
This is a bank, it needs greater robustness in security,
so all the hosts are up to date.
On the left you can see the creation of all the different VMs.
There are 10 VMs per server, and roughly rounding off, about 300 physical
servers
That gives us a total of 3,000 Debian hosts.
Installed in an automated way across the entire country.
And similar to the Proxmox, we have the workstations.
For the workstations we created an image.
Here we're seeing a test virtual workstation from Development, inside
a Proxmox.
And what we do is generate a base image, also a NetInstall,
in this case based on a Debian-based distro.
And then we configure it, also in the same way, with a script.
It takes the hostname of the host, which can be configured later or reconfigured
at the destination location.
It's registered in the inventory and also becomes available with a manufacturer
serial number.
They're all HP workstations.
We can uniquely identify them, even if the MAC address changes at
some point.
It's always the same device.
The AWX web interface is basically a webpage built
with Django.
And there are many operators who work launching different playbooks
that we create, to keep everything operational.
And the PXE server also takes workstation images.
Meaning you can take a workstation on-site, if it got misconfigured or something happened,
doesn't matter.
The full workstation image is rebuilt via PXE.
No time is lost.
It has time to see what's wrong with it.
Nobody manually accesses the host.
It's rebuilt from scratch.
Why?
Because there are no local data inside the workstation.
The data is on an NFS server with Kerberos in the same branch network
of the branch where we are.
For deployment at the branch, it's a bit more complex because for branch deployment,
as we said, we have 3,000 workstations.
Of these 3,000 workstations, there are no users
on the workstations.
Meaning, if you look at /etc/passwd on those workstations, there are
no local users
beyond what comes with the system.
What we use is a FreeIPA client that connects to IPA, and from IPA
it connects to the 4 ADs we have.
And they pull those domain users directly on the fly.
And for this reason each workstation also needs to be enrolled in the domain.
But what happens is they enroll to the FreeIPA domain.
And FreeIPA connects to the FreeIPA domain.
And FreeIPA has a trust relationship with the ActiveDirectory
domain
Meaning it's like a subdomain.
And what this allows is for all those users to have their password in the AD,
which is the Bank's infrastructure, without changing anything for them.
For them it's transparent.
And from all the GNU/Linux hosts they use the same user directly.
What's in the middle that enables this is called FreeIPA.
And to speed up the deployment process, what we have new is a cache
of cache of cache.
We have several proxies.
Initially we had Debian Apt-Cacher.
Now there are also some Debian mirrors.
And then we have Nginx and Squid, all the way down.
So this works throughout the entire country.
Here's a bit of the hardware it started with.
3,000 HP ProDesk.
3,000 Dell servers.
Sorry, 300 Dell servers.
3,000 virtual Debian hosts.
Then it gradually changed a bit.
But then we also found 3,500 different peripherals.
Weird things like a check scanner or a ticket printer.
This is what the original infrastructure was that the project started
with in 2018.
It's been evolving.
Some VMs we've also changed.
And quickly what gave us problems.
Scaling problems.
This is a graph of login attempts during the day.
And what you'll see is there's a small problem here at 10 in the morning.
Meaning, 3,000 people wanted to log in at 10 AM.
You see it too when they drop off after 4 PM.
The logins start going up again.
And so, there's a problem with delay and caching here.
And other problems, imagine with 3,000 users,
with the robust security policies of a bank,
every day many passwords expire and they have to be changed and all that.
And this change has to be done from the login screen of
our distro
Meaning directly from there.
They're told the password has expired and at that moment it asks for the old one, twice
the new one.
In the morning early, someone gets confused.
That generates another wrong password.
But in this...
Now I'll tell you what was done to improve it.
And so, to handle this too, well,
part of the automation we achieved is,
instead of going to AWX, which is a graphical interface,
and clicking to create playbooks, workflows, inventories,
and granting permissions manually,
what we did is a git repo called awx,
a repo called inventory,
and they're all JSON or YAML files,
that GitLab CI directly verifies and deploys
into a Development AWX
immediately as we do the git push.
So in this way we can have all stages verified
and automatically deployed on a Development AWX for testing.
On the Production AWX,
this deploy is triggered manually, let's say, but it creates the same
instances.
This guarantees having all infrastructure as versioned code.
And to give you an idea of the project scale,
this is a view of the Bank's productive infrastructure.
I said, about 300 branches,
distributed across almost all provinces of the country.
And now, if we zoom in on this, we'll understand a bit more,
all these are hosts.
And all these hosts are controlled by one over here,
which I'll find soon, there it is.
This one here is the AWX, a single VM,
not even on a physical host,
which controls all the others and deploys to all the others.
Obviously they don't deploy them all at once; they deploy in stages.
And what we have is that each line, for example here,
we'll see, this is the province of...
the province of Santa Fe.
I'll try to brighten it a bit.
All of this is the hosts from all the different branches
in the entire province of Santa Fe.
And if we zoom in on one, here, for example, it's f0372.
And so, within the concept of that branch,
we have a local cache, which is a CDN,
the PVE, which is the Proxmox server,
the REP, which is the File Server, a Kerberized NFS,
a log host, which receives logs from all the others
and forwards them.
Initially we did this with rsyslog.
There's a VPN node that we don't intervene on,
but we did automatic installation,
which is basically what the Bank staff configures.
We have an Apt-Cacher, here, local, within the branch.
What else do we have?
Well, this git one, in the end we dropped it.
The print server, which is a CUPS,
where all the printers at the location are configured.
And then, here is the representation of the different printers at that branch.
And then we'll have the different workstations.
And finally the rest of the equipment there.
Ticket printers and such.
And this infrastructure,
this infrastructure is repeated throughout the entire bank.
Meaning that if you don't do it in an automated way,
it's impossible to maintain.
And what we achieved with this is that this AWX infrastructure
allows us so that there's no longer a need to log into a host manually
and see what happens.
There's an AWX playbook, from Headquarters,
where there's already a template to solve each problem.
And you run that template and it records everything that happens,
a traceable log.
Obviously there are different permission levels.
Who can do that? Who can't?
Schedules, for example.
I don't know, if we want all the hosts to shut down at a certain time,
well, there's a template that sets a poweroff schedule at certain time
for each host.
For example, and you launch it.
You can do things like SNMP sweeps
to find out if certain hosts are alive or dead.
Meaning, all of that ends up in a single,
finally a PostgreSQL database, managed from AWX.
And if we wanted to see a bit of what it's like,
this is, as a view of all the playbooks there are in AWX.
Here's our AWX.
Here I tried to organize them a bit.
Here we have iDRAC/Redfish.
rsyslog, CDN.
Well, here we have the inventory.
I don't know if you can see anything there?
Can't see it?
Not even I can see it here!
But, let's see, over here.
Here, for example, we have a role that clones a KVM VM in Proxmox.
So there's a role just for that.
Over here we'll see more.
This role creates a Proxmox KVM VM from an ISO
directly.
And it's an unattended ISO.
Meaning, there's nothing to do.
That's for the VPN.
A Proxmox role to configure cloud-init for each of the VMs.
We use the OpenStack image.
Let's say cloud, but without cloud.
Local.
To do a qm restore.
Meaning, we can restore a VM from a backup and it boots up.
And so, there's a playbook for everything.
Well, this is roughly what the migration project was.
I'll try to summarize, and move on to the new stuff.
A bit about the difference from 2022 to 2026 of what's been worked on.
Well, there are over 200 git repositories to control all this.
Now what's being done, part of what was actually done,
we worked a lot on cybersecurity.
We changed from rsyslog to auditd.
We integrated NUT for UPSs, for UPS monitoring.
We're working on updating the Debian versions of those VMs from 10 to 13.
This, remember, started in 2018.
The FreeIPA automation, now a new version was done...
Meaning, what we had currently in Production was only one large
FreeIPA VM.
And since the bank has OpenShift,
we're doing a migration of FreeIPA to OpenShift.
Also all with automated playbooks.
Here as a summary of everything done for cybersecurity.
It's a role that connects to each host and says what's right, what's wrong.
Some things it can fix and others it simply says
this cannot go to Production like this.
We also worked on kernel booting,
so that some server models would appear
and to identify network cards and have everything work automatically
and disk repartitioning, and there are different server levels.
Regeneration of all these images.
Everything related to the lifecycle of VMs that need to be provisioned.
Several that were taken down.
Others that changed because they didn't need a secondary disk.
The order in which they boot up.
A report of the status of all this.
All those VMs and updates.
Well, here's a bit about what I mentioned regarding server versions.
And we'll move on since we're short on time.
For workstations is where most of the work has been done,
because originally we worked with Workstation 18.04
and now we've moved to 24.04 if I'm not mistaken.
And then there were some problems they had.
Remember this original infrastructure wasn't GNU and worked with Firefox 9.
We took it to v68 and now we're taking it to v120, and tests are being done with v140.
pruebas con v140.
The problem isn't really the environment, but the ecosystem
of all the internal applications that aren't updated and don't work.
Basically it's not an easy task.
So for a while more than one browser version coexisted.
Eh...
And that was another challenge.
And well, also issues around updating the kernel version.
We had to patch some difficult CVEs that came out
recently.
Well, at some point we also tested Chrome as an alternative for some
sites.
We do everything with a playbook policy configuration
Firefox settings.
The user basically can't do anything.
Meaning, like...
They can't even change the desktop wallpaper.
This...
But well, it's the way to manage 3,000 users.
Well then...
Nobody goes into a host to configure it!
From AWX a playbook is launched directly.
And, uh...
Well, the WakeOnLAN topic, integration, kernel updates,
sftp fixes.
Meaning, everything that would lead to the new IPA version.
We worked quite a bit with AppArmor to restrict...
um...
Some things that are important.
Login handling.
Also handling the ability to change passwords.
um...
Well, backups and all that.
And, uh...
Let's see...
A bit of AWX here.
um...
Well, there have been 198 releases since we finished migrating it.
Because like any large infrastructure, when you finish migrating it
you have to start migrating again.
Basically.
It's in constant change.
And, well, all the time new needs come up...
um...
Well, we have CloneZilla to boot a host.
with a pre-built image.
um...
And solving problems.
um...
Here, uh...
Isolated profiles.
um...
Isolated from each other due to configuration issues.
um...
A CCTV component was added.
Equipment integrated with the security camera DVRs.
um...
So, well, that's a new inventory that was added.
um...
HP Linux Tools, for example, this...
A detail is that at one point, uh...
A host would freeze and hang in strange situations.
It took us a while to investigate what was happening.
It had to do with a BIOS power saving configuration.
And so the solution was simple.
It was to go into 3,000 hosts, change the BIOS config and reboot them.
Done.
It's very simple.
Do you need 3,000 technicians?
or
um...
As many as there are branches across the country.
It's impossible.
And, well, there, uh...
I started investigating HP's FTP.
And I found a guy in Linux who had a tool that allows you
to write to the UEFI.
um...
With the BIOS configuration it will take on the next reboot.
This requires compiling a kernel module and such.
And...
um...
We told them to ask HP for permission and guarantee they wouldn't
turn into 3,000 bricks.
um...
They said...
Yes!
And, well, we did several tests.
It never happened to us.
So they all booted up fine.
um...
And that allows, well, centralized administration again,
you could even change the BIOS password on all hosts
centrally and remotely.
So it's super useful.
um...
Well.
Here, for example, uh...
Modification of Git tokens and...
There are a lot of things.
This...
At the URL shown below, https://filiales-gnu-linux.g.coop.ar
All of this will be available.
So if you want to see it in detail.
um...
And since we have five minutes left, uh...
If you agree, I'll wrap up here and you can ask any questions...
Because we won't get to see everything.
Hello.
um...
Are the ActiveDirectories you use Windows or do you also use Windows Server?
Yes, it's Bank infrastructure.
um...
That was already preexisting.
And they weren't going to change it.
The challenge was to get all those users to work on the new
workstations.
um...
And it works!
First of all, impressive.
Thanks to the Free Software community! Uh...
We put the pieces together.
They got the most out of Ansible, but not I thought so much could really
be done
I had a small doubt about something I saw.
You said you changed from rsyslog to auditd.
Yes.
It was a cybersecurity requirement.
But the intention...
Let's say, rsyslog logs at an application level, maybe a
system level.
And auditd focuses more on the syscalls of the system itself.
Yes, but it allows more detail.
You can specify exactly which parts you want.
No, no, no.
I see.
My doubt is, are you logging syscalls from 3,000 clients?
That is...
Not from all the workstations.
Actually it is...
Eh...
mainly from VMs, and some things.
It's selective, not everything.
Ah, ah.
No, because otherwise there's no way.
No, no, it's fine.
That was just it.
Exactly to avoid sending all the logs.
Meaning, sending selectively.
All this information goes to a SIEM at Headquarters.
Just one?
Yes, well, it could be more than one node.
Ah, no.
But, let's say, conceptually it goes to a SIEM that everything is then displayed
there what happens.
And all of AWX deploy logs also
go to the SIEM...
So also...
As a SIEM?
No, another one I don't remember now, but it's well-known.
Now, if I tell you, I'd be lying.
Thank you very much.
I agree with the colleague on the hard work, tremendous effort.
And all done with Free Software.
My question is basically based on the experience of the entire project.
What went really well?
What are you going to start doing, besides what you've mentioned?
And what things went one way and you said, no, this isn't the right approach,
we pivoted?
The biggest challenge overall isn't technical per se, but rather dealing with
the priorities of everything that needs to be done.
And with things where at the user level, many free applications aren't designed
for such a large scale.
And a problem where you want users to be unable to touch anything.
For example, there are some PDFs that are smart forms you can fill
fill out.
And then you have to print them.
And that was really complex to solve.
At first we had to install Adobe Acrobat with, I don't know, Wine or
something horrible like that.
Because it was the only thing that supported it.
Evince displayed them, but didn't allow us to fill them out.
Later we had an option I don't remember which one that allowed
filling them out.
But it didn't let you hide the comments on each form,
in each textbox.
And that would print out and wouldn't work.
Now there's been work on Okular, which has all of that.
But it has no per-file configuration option to
disable all that.
Meaning the user can disable it at the moment.
Well, so we worked on it.
We're touching Okular's code to make that work.
The same thing happened with some binaries that have no translation and no translation
support.
And well, what we could we edited with a hexadecimal editor.
And embedded our binary.
Things like that, let's say, with respect to the Asians, we call them Chinese.
Meaning, it's a hack that sometimes you have to dig into and resolve.
Or complex bugs where, I don't know, you enter a screen and for various reasons,
if you move the mouse a bit more down to the right,
it crashes GNOME.
And so you have to put a script that prevents you from going down there.
And also things that a user who owns their environment doesn't have a problem with
because they can customize it.
And an end user can't touch anything.
And you, as administrator of this infrastructure, also don't want them to
touch it.
But you have to give a solution.
So, a thousand things.
What we didn't get involved in and outsourced was the check scanner
integration part.
I tried for a while and couldn't make it work.
That was outsourced.
But it works now.
Yes, we did manage to get the ticket printer to work properly.
And that implied a challenge because the banking core at the time used
Java applets.
And that, nothing, was impossible.
We had to put a chroot in the middle with something old for that to work.
Well, in the end we managed to avoid all of that.
Because now that core already detects when it's GNU/Linux.
And so, it outputs code and we put in a local backend that does
everything needed and it works.
But yes, it's a challenge involving many people.
I'm currently not on the project.
I was in the initial development and migration phase.
There are three people now...
who work full-time on this.
And every day something new appears.
Great.
I can stay here and chat.
Well.
OSiRiS, first of all thank you.
For those of us who came here to learn.
The admiration you generate with everything you've shared.
My question isn't about...
It's gcoop's work too, not mine...
Everyone's.
My question goes more...
I'm selling it.
It goes more that way.
Not the technical side.
But rather the team or human group.
Not the human side.
What's at Filiales GNU/Linux
that if I don't understand is the organization behind all
this project.
How many people are needed to do this?
And how do you manage it?
Well, initially it was a six-month project.
Of two people.
A functional analyst and me.
To see if the project was viable.
Then it started...
I think we started with three people.
At the peak we reached three plus five.
To five?
Three plus five.
Right, yes.
Always with a PM.
No, but at the peak it was five, not that way.
Meaning, I think there...
I don't count the PM, poor guy.
Because he always talks about...
The PM always talks about how we're not people.
That we are developers!
That we are weird things.
So, well.
It's revenge.
No, at the peak it was five people.
Today there are three.
DevOps people, let's say.
But you have to...
Let's say, it's full stack.
So to speak.
But then you have to touch or redesign an application.
Meaning, we had to design applications to work around existing issues.
Meaning, it's the fun and complex part at the same time.
I thought you were going to say three hundred, I don't know.
Thank you very much.
No, no, sorry.
We designed the automation for all of this.
The bank has its own arsenal.
An army of people who use all of this daily.
They did the migration in the middle of the 2020 pandemic.
In less than a year!
But it was a physical army of people who resolved all of that.
AWX operators, I think there are about sixty.
Meaning, like...
Well, there's a lot of people.
The bank's infrastructure is very large.
And it's all On-premise.
It's all local.
Meaning, and all with Free Software.
Thank you very much.
There, Alejandro.
I'm putting a question in the chat for you.
If you can answer it there later.
Yes.
And later I have here colleagues who speak English.
And if there's anyone who doesn't speak Spanish.
Go ahead.
And Alejandro will be around today.
So you can keep asking him questions.
Thank you very much.
OSiUX too!