WEBVTT 00:10.770 --> 00:16.390 ...in the community, I work at gcoop, which is a Free Software Cooperative, 00:16.630 --> 00:19.450 meaning that we work exclusively with Free Software 00:19.450 --> 00:22.470 and horizontally, as an organization. 00:22.490 --> 00:24.130 We have no bosses, we have no employees. 00:26.370 --> 00:30.030 Five of our partners from gcoop came to the event, 00:30.210 --> 00:32.270 it's our first time at DebConf for everyone 00:32.950 --> 00:36.590 We sponsored this event and give thanks to Debian 00:36.590 --> 00:39.050 because we have 20 years working with Free Software 00:39.050 --> 00:40.870 and it wouldn't have been possible without Debian. 00:41.850 --> 00:47.430 I'm going to show an update of a talk I gave before in 2022, 00:47.770 --> 00:52.310 which is about the GNU/Linux migration project of Filiales 00:52.310 --> 00:52.850 from Banco Credicoop 00:53.870 --> 00:56.050 Cooperativo Limitado, it's a cooperative bank. 00:56.570 --> 01:03.050 And I'll try to quickly review the migration process 01:03.630 --> 01:06.370 and then we'll look at the differences in recent years. 01:07.490 --> 01:08.690 This project... 01:14.650 --> 01:17.590 Basically it's a project of Infrastructure as Code 01:17.590 --> 01:21.050 using Ansible and AWX. 01:21.170 --> 01:28.430 AWX is a web platform, which is a graphical interface for Ansible, 01:28.550 --> 01:33.650 but on the other hand it allows orchestrating and managing and administering 01:33.650 --> 01:35.190 all the Ansible playbooks. 01:35.390 --> 01:38.670 and this was done for Banco Credicoop. 01:39.050 --> 01:39.570 from gcoop. 01:43.130 --> 01:48.550 So, at the beginning of the project, what we had to do was, over 01:48.550 --> 01:48.810 6 months, 01:49.350 --> 01:53.290 was trying to discover what tools in the Free Software ecosystem 01:53.290 --> 01:58.190 would allow us to carry out the entire infrastructure migration 01:58.190 --> 01:58.550 of a bank 01:58.550 --> 02:01.210 throughout all of Argentina. 02:01.690 --> 02:05.250 And what we found was that the automation tool was 02:05.250 --> 02:05.530 Ansible 02:05.530 --> 02:06.950 because we were already using it. 02:07.430 --> 02:09.230 That we would use GitLab. 02:10.710 --> 02:16.270 That for ActiveDirectory authentication within Linux, 02:16.270 --> 02:16.890 we would use IPA. 02:18.450 --> 02:21.290 AWX, obviously all the servers would use Proxmox. 02:21.390 --> 02:23.670 We had already been working for years with Proxmox. 02:24.070 --> 02:27.210 The VMs inside the Proxmox servers would be Debian, obviously. 02:27.970 --> 02:31.730 And so, for workstations, due to the fact that there were 02:31.730 --> 02:32.330 3,000 workstations 02:33.010 --> 02:38.070 and because of support reasons, we chose a Debian-based distribution 02:38.970 --> 02:39.450 um... 02:39.450 --> 02:40.650 A slightly more updated version. 02:41.230 --> 02:45.270 And what we had there, it's not very visible, but what's moving is that 02:45.270 --> 02:45.890 ...underneath 02:45.890 --> 02:48.890 all the components, it's a timeline of all the components we use 02:48.890 --> 02:49.370 for dependencies 02:49.370 --> 02:54.030 that are being updated worldwide, all the 02:54.030 --> 02:54.330 time. 02:54.630 --> 02:59.550 Meaning, at that point in a given month, a new version of a component 02:59.550 --> 03:01.510 came out, along with another component and another. 03:01.630 --> 03:04.350 And this evolves as you're thinking about how to approach things, the entire ecosystem updates. 03:04.350 --> 03:05.470 the entire ecosystem keeps updating. 03:05.950 --> 03:08.850 So it's a problem to solve. 03:10.710 --> 03:18.490 But what we achieved is that in Development we managed to orchestrate an idea 03:18.490 --> 03:19.030 of deployment, 03:19.290 --> 03:26.110 which is basically: we have an AWX, let me point it out like this, 03:26.910 --> 03:36.210 this is an AWX, and its source of truth is a GitLab, 03:36.270 --> 03:38.610 so AWX reads from GitLab 03:38.610 --> 03:40.030 everything that would be... 03:40.030 --> 03:45.830 Meaning playbooks, and the first step is deploying the iDRAC. 03:45.950 --> 03:48.730 The iDRAC is the computer of the computer, inside the servers 03:48.730 --> 03:49.610 inside the Dell servers we use 03:49.610 --> 03:53.750 before the server even has anything; there's a computer called iDRAC, 03:53.770 --> 03:58.030 using a protocol called Redfish, we wrote playbooks that AWX uses to 03:58.030 --> 03:59.570 connect to iDRAC 03:59.570 --> 04:06.190 and what it does is partition the disk, configure BIOS and reboot into PXE. 04:06.830 --> 04:10.010 That is, a new computer, fresh out of the box. 04:10.030 --> 04:19.530 And immediately that will end up generating from a Proxmox, 04:19.530 --> 04:21.110 it will ultimately generate another Proxmox. 04:21.310 --> 04:23.610 Inside that Proxmox We have several VMs. 04:23.810 --> 04:32.750 One of those VMs is a PXE server, I don't see anything here, but it must be this one. 04:34.650 --> 04:40.010 We have a server, well we have a CDN, which is a processing server, 04:40.030 --> 04:45.950 an Nginx that has all the resources, which in turn is a proxy for another, 04:45.950 --> 04:46.370 an Apache, 04:48.170 --> 04:51.790 and a server that replicates this data. 04:52.350 --> 04:57.230 This is all done in an automated, unattended way, without 04:57.230 --> 04:58.230 operator intervention, 04:58.370 --> 05:03.930 meaning a server just comes out of the factory, the box is opened, 05:03.930 --> 05:05.870 Power and Network are installed, 05:05.870 --> 05:11.330 and from AWX a script is launched, 05:12.950 --> 05:16.890 this is the AWX infrastructure, sorry, the Proxmox one, we were going to see, 05:16.950 --> 05:17.550 you can see it here, 05:17.650 --> 05:23.170 how it's launched, and this is just for monitoring, let's say, 05:23.290 --> 05:27.090 everything is done in an unattended manner and the operator does not need to see 05:27.090 --> 05:28.090 anything running 05:28.190 --> 05:33.390 AWX deploys Proxmox via PXE; it's actually several stages, 05:33.390 --> 05:35.730 first it deploys a Debian NetInstall 05:35.870 --> 05:40.050 completely unattended, meaning there's nothing to touch 05:40.050 --> 05:40.370 at all, 05:40.830 --> 05:45.230 it goes directly into OEM mode, configuring everything needed for that 05:45.230 --> 05:47.370 server to be operational, 05:47.450 --> 05:52.830 and when it finishes, in a hook at the end of the Debian NetInstaller, 05:54.230 --> 05:58.890 we wrote a script that takes the MAC address of that host, which 05:58.890 --> 05:59.150 is unique, 05:59.410 --> 06:02.610 and registers it in the AWX inventory. 06:03.090 --> 06:05.590 So that host, which just received a random IP, 06:05.870 --> 06:09.730 we now know what its IP will be, what its iDRAC is, 06:09.750 --> 06:14.370 permanently, for all the years this server will have, 06:15.690 --> 06:22.250 referenced to that MAC and to a serial number from the 06:22.250 --> 06:22.510 iDRAC itself. 06:22.650 --> 06:24.830 So, in this way it's easy to identify it. 06:25.990 --> 06:30.950 On the left, all you see is the debug of what's happening on the 06:30.950 --> 06:33.370 PXE server, 06:34.150 --> 06:35.850 and as you can see, the server is on the PXE server, 06:35.850 --> 06:40.430 the server installs itself with no manual intervention and finishes installed, 06:40.470 --> 06:41.290 it's a Debian. 06:41.570 --> 06:47.050 The next playbook launched on AWX converts that Debian into a 06:47.050 --> 06:47.450 Proxmox, 06:47.590 --> 06:49.470 also in an unattended way. 06:49.530 --> 06:52.670 Meaning we don't install Proxmox directly; we install Debian and 06:52.670 --> 06:53.590 convert it to Proxmox. 06:54.030 --> 06:59.530 That's an advantage of Proxmox being a Debian-based distro, 06:59.670 --> 07:01.490 where the only thing that changes is adding a repo. 07:01.670 --> 07:04.310 And done, all the necessary packages are configured. 07:04.430 --> 07:05.830 And when configuration finishes, 07:05.830 --> 07:10.230 on a Proxmox, we launch another playbook that creates the VMs, 07:10.830 --> 07:15.510 and another playbook that creates the services inside each VM. 07:15.710 --> 07:17.410 All the VMs we use are KVM. 07:18.510 --> 07:21.570 This is a bank, it needs greater robustness in security, 07:21.810 --> 07:23.830 so all the hosts are up to date. 07:24.250 --> 07:27.290 On the left you can see the creation of all the different VMs. 07:27.550 --> 07:33.390 There are 10 VMs per server, and roughly rounding off, about 300 physical 07:33.390 --> 07:33.990 servers 07:34.070 --> 07:35.810 That gives us a total of 3,000 Debian hosts. 07:35.830 --> 07:39.630 Installed in an automated way across the entire country. 07:40.990 --> 07:45.790 And similar to the Proxmox, we have the workstations. 07:45.850 --> 07:47.870 For the workstations we created an image. 07:47.910 --> 07:53.490 Here we're seeing a test virtual workstation from Development, inside 07:53.490 --> 07:54.210 a Proxmox. 07:54.590 --> 08:01.870 And what we do is generate a base image, also a NetInstall, 08:02.090 --> 08:05.230 in this case based on a Debian-based distro. 08:05.830 --> 08:11.230 And then we configure it, also in the same way, with a script. 08:11.890 --> 08:18.630 It takes the hostname of the host, which can be configured later or reconfigured 08:18.630 --> 08:19.930 at the destination location. 08:20.070 --> 08:24.630 It's registered in the inventory and also becomes available with a manufacturer 08:24.630 --> 08:26.050 serial number. 08:26.070 --> 08:28.410 They're all HP workstations. 08:28.690 --> 08:32.570 We can uniquely identify them, even if the MAC address changes at 08:32.570 --> 08:32.930 some point. 08:32.930 --> 08:34.710 It's always the same device. 08:35.830 --> 08:39.490 The AWX web interface is basically a webpage built 08:39.490 --> 08:40.030 with Django. 08:40.490 --> 08:46.350 And there are many operators who work launching different playbooks 08:46.350 --> 08:49.850 that we create, to keep everything operational. 08:50.450 --> 08:55.290 And the PXE server also takes workstation images. 08:55.370 --> 08:58.950 Meaning you can take a workstation on-site, if it got misconfigured or something happened, 08:58.950 --> 09:00.690 doesn't matter. 09:00.850 --> 09:05.050 The full workstation image is rebuilt via PXE. 09:05.130 --> 09:05.810 No time is lost. 09:05.810 --> 09:06.590 It has time to see what's wrong with it. 09:06.950 --> 09:09.330 Nobody manually accesses the host. 09:09.470 --> 09:11.350 It's rebuilt from scratch. 09:11.450 --> 09:12.010 Why? 09:12.050 --> 09:14.830 Because there are no local data inside the workstation. 09:15.070 --> 09:22.950 The data is on an NFS server with Kerberos in the same branch network 09:22.950 --> 09:23.630 of the branch where we are. 09:24.990 --> 09:30.890 For deployment at the branch, it's a bit more complex because for branch deployment, 09:30.890 --> 09:32.730 as we said, we have 3,000 workstations. 09:32.830 --> 09:34.790 Of these 3,000 workstations, there are no users 09:34.790 --> 09:36.290 on the workstations. 09:36.450 --> 09:40.690 Meaning, if you look at /etc/passwd on those workstations, there are 09:40.690 --> 09:41.130 no local users 09:42.310 --> 09:44.150 beyond what comes with the system. 09:44.630 --> 09:50.050 What we use is a FreeIPA client that connects to IPA, and from IPA 09:50.050 --> 09:52.410 it connects to the 4 ADs we have. 09:52.610 --> 09:56.090 And they pull those domain users directly on the fly. 09:56.670 --> 10:00.730 And for this reason each workstation also needs to be enrolled in the domain. 10:00.830 --> 10:03.910 But what happens is they enroll to the FreeIPA domain. 10:03.910 --> 10:04.770 And FreeIPA connects to the FreeIPA domain. 10:04.770 --> 10:07.870 And FreeIPA has a trust relationship with the ActiveDirectory 10:07.870 --> 10:08.170 domain 10:08.550 --> 10:09.790 Meaning it's like a subdomain. 10:10.290 --> 10:14.910 And what this allows is for all those users to have their password in the AD, 10:14.910 --> 10:17.390 which is the Bank's infrastructure, without changing anything for them. 10:17.830 --> 10:19.210 For them it's transparent. 10:19.370 --> 10:25.190 And from all the GNU/Linux hosts they use the same user directly. 10:25.430 --> 10:27.790 What's in the middle that enables this is called FreeIPA. 10:28.070 --> 10:31.810 And to speed up the deployment process, what we have new is a cache 10:31.810 --> 10:32.350 of cache of cache. 10:32.470 --> 10:33.710 We have several proxies. 10:34.950 --> 10:38.870 Initially we had Debian Apt-Cacher. 10:39.090 --> 10:41.330 Now there are also some Debian mirrors. 10:42.390 --> 10:46.150 And then we have Nginx and Squid, all the way down. 10:46.150 --> 10:50.150 So this works throughout the entire country. 10:53.070 --> 10:56.110 Here's a bit of the hardware it started with. 10:56.190 --> 10:58.530 3,000 HP ProDesk. 10:59.250 --> 11:00.730 3,000 Dell servers. 11:00.910 --> 11:02.250 Sorry, 300 Dell servers. 11:02.250 --> 11:04.070 3,000 virtual Debian hosts. 11:04.090 --> 11:05.310 Then it gradually changed a bit. 11:05.590 --> 11:09.750 But then we also found 3,500 different peripherals. 11:10.270 --> 11:13.870 Weird things like a check scanner or a ticket printer. 11:14.230 --> 11:19.190 This is what the original infrastructure was that the project started 11:19.190 --> 11:20.350 with in 2018. 11:21.650 --> 11:24.010 It's been evolving. 11:24.810 --> 11:26.550 Some VMs we've also changed. 11:27.990 --> 11:30.650 And quickly what gave us problems. 11:30.650 --> 11:32.930 Scaling problems. 11:33.090 --> 11:39.850 This is a graph of login attempts during the day. 11:40.050 --> 11:44.150 And what you'll see is there's a small problem here at 10 in the morning. 11:44.390 --> 11:48.170 Meaning, 3,000 people wanted to log in at 10 AM. 11:49.450 --> 11:52.130 You see it too when they drop off after 4 PM. 11:52.170 --> 11:54.090 The logins start going up again. 11:55.030 --> 12:00.110 And so, there's a problem with delay and caching here. 12:00.110 --> 12:03.570 And other problems, imagine with 3,000 users, 12:03.710 --> 12:07.570 with the robust security policies of a bank, 12:07.810 --> 12:11.110 every day many passwords expire and they have to be changed and all that. 12:11.310 --> 12:15.870 And this change has to be done from the login screen of 12:15.870 --> 12:16.070 our distro 12:16.410 --> 12:19.150 Meaning directly from there. 12:19.350 --> 12:23.010 They're told the password has expired and at that moment it asks for the old one, twice 12:23.010 --> 12:23.410 the new one. 12:23.970 --> 12:25.750 In the morning early, someone gets confused. 12:25.810 --> 12:28.410 That generates another wrong password. 12:28.490 --> 12:30.090 But in this... 12:30.090 --> 12:33.070 Now I'll tell you what was done to improve it. 12:33.810 --> 12:36.710 And so, to handle this too, well, 12:37.370 --> 12:40.210 part of the automation we achieved is, 12:41.050 --> 12:44.490 instead of going to AWX, which is a graphical interface, 12:44.490 --> 12:48.870 and clicking to create playbooks, workflows, inventories, 12:49.630 --> 12:52.370 and granting permissions manually, 12:52.450 --> 12:56.630 what we did is a git repo called awx, 12:56.710 --> 12:58.390 a repo called inventory, 12:58.390 --> 13:02.370 and they're all JSON or YAML files, 13:02.450 --> 13:10.270 that GitLab CI directly verifies and deploys 13:10.270 --> 13:10.570 into a Development AWX 13:10.570 --> 13:14.330 immediately as we do the git push. 13:14.890 --> 13:18.990 So in this way we can have all stages verified 13:18.990 --> 13:25.070 and automatically deployed on a Development AWX for testing. 13:25.230 --> 13:26.970 On the Production AWX, 13:26.970 --> 13:31.510 this deploy is triggered manually, let's say, but it creates the same 13:31.510 --> 13:31.990 instances. 13:32.210 --> 13:36.330 This guarantees having all infrastructure as versioned code. 13:37.270 --> 13:39.830 And to give you an idea of the project scale, 13:40.610 --> 13:44.650 this is a view of the Bank's productive infrastructure. 13:45.430 --> 13:48.190 I said, about 300 branches, 13:48.190 --> 13:50.870 distributed across almost all provinces of the country. 13:51.330 --> 13:54.410 And now, if we zoom in on this, we'll understand a bit more, 13:54.710 --> 13:55.490 all these are hosts. 13:56.130 --> 14:01.190 And all these hosts are controlled by one over here, 14:02.670 --> 14:04.050 which I'll find soon, there it is. 14:05.630 --> 14:09.470 This one here is the AWX, a single VM, 14:09.750 --> 14:11.250 not even on a physical host, 14:11.350 --> 14:13.890 which controls all the others and deploys to all the others. 14:14.410 --> 14:17.390 Obviously they don't deploy them all at once; they deploy in stages. 14:17.690 --> 14:20.930 And what we have is that each line, for example here, 14:22.110 --> 14:25.450 we'll see, this is the province of... 14:25.490 --> 14:25.630 the province of Santa Fe. 14:26.430 --> 14:29.370 I'll try to brighten it a bit. 14:33.400 --> 14:37.280 All of this is the hosts from all the different branches 14:37.280 --> 14:38.660 in the entire province of Santa Fe. 14:39.280 --> 14:47.800 And if we zoom in on one, here, for example, it's f0372. 14:48.160 --> 14:51.420 And so, within the concept of that branch, 14:51.460 --> 14:54.820 we have a local cache, which is a CDN, 14:54.980 --> 14:56.420 the PVE, which is the Proxmox server, 14:56.420 --> 15:00.460 the REP, which is the File Server, a Kerberized NFS, 15:02.140 --> 15:06.120 a log host, which receives logs from all the others 15:06.120 --> 15:06.480 and forwards them. 15:06.600 --> 15:09.480 Initially we did this with rsyslog. 15:09.700 --> 15:12.260 There's a VPN node that we don't intervene on, 15:12.300 --> 15:14.300 but we did automatic installation, 15:14.400 --> 15:20.000 which is basically what the Bank staff configures. 15:21.680 --> 15:25.260 We have an Apt-Cacher, here, local, within the branch. 15:26.300 --> 15:27.100 What else do we have? 15:27.400 --> 15:29.020 Well, this git one, in the end we dropped it. 15:30.340 --> 15:32.760 The print server, which is a CUPS, 15:33.280 --> 15:35.440 where all the printers at the location are configured. 15:35.720 --> 15:40.760 And then, here is the representation of the different printers at that branch. 15:41.000 --> 15:44.880 And then we'll have the different workstations. 15:46.240 --> 15:50.500 And finally the rest of the equipment there. 15:51.460 --> 15:53.120 Ticket printers and such. 15:53.540 --> 15:55.000 And this infrastructure, 15:55.000 --> 15:56.860 this infrastructure is repeated throughout the entire bank. 15:57.160 --> 15:59.800 Meaning that if you don't do it in an automated way, 16:00.040 --> 16:01.460 it's impossible to maintain. 16:01.800 --> 16:07.260 And what we achieved with this is that this AWX infrastructure 16:07.260 --> 16:10.940 allows us so that there's no longer a need to log into a host manually 16:10.940 --> 16:11.760 and see what happens. 16:12.320 --> 16:15.460 There's an AWX playbook, from Headquarters, 16:16.100 --> 16:19.460 where there's already a template to solve each problem. 16:19.680 --> 16:22.880 And you run that template and it records everything that happens, 16:23.080 --> 16:24.020 a traceable log. 16:25.120 --> 16:27.180 Obviously there are different permission levels. 16:27.220 --> 16:29.100 Who can do that? Who can't? 16:30.360 --> 16:31.160 Schedules, for example. 16:31.260 --> 16:34.220 I don't know, if we want all the hosts to shut down at a certain time, 16:34.300 --> 16:38.600 well, there's a template that sets a poweroff schedule at certain time 16:38.600 --> 16:38.960 for each host. 16:39.120 --> 16:40.420 For example, and you launch it. 16:40.440 --> 16:43.380 You can do things like SNMP sweeps 16:43.380 --> 16:46.580 to find out if certain hosts are alive or dead. 16:46.740 --> 16:49.200 Meaning, all of that ends up in a single, 16:49.260 --> 16:52.380 finally a PostgreSQL database, managed from AWX. 16:52.380 --> 16:57.000 And if we wanted to see a bit of what it's like, 16:57.080 --> 17:00.580 this is, as a view of all the playbooks there are in AWX. 17:01.360 --> 17:03.200 Here's our AWX. 17:03.680 --> 17:06.480 Here I tried to organize them a bit. 17:09.480 --> 17:12.180 Here we have iDRAC/Redfish. 17:15.960 --> 17:17.160 rsyslog, CDN. 17:18.580 --> 17:20.300 Well, here we have the inventory. 17:20.300 --> 17:22.360 I don't know if you can see anything there? 17:22.380 --> 17:23.600 Can't see it? 17:23.680 --> 17:24.860 Not even I can see it here! 17:26.080 --> 17:27.520 But, let's see, over here. 17:28.540 --> 17:34.040 Here, for example, we have a role that clones a KVM VM in Proxmox. 17:34.260 --> 17:36.280 So there's a role just for that. 17:36.900 --> 17:38.200 Over here we'll see more. 17:39.880 --> 17:45.420 This role creates a Proxmox KVM VM from an ISO 17:45.420 --> 17:46.660 directly. 17:47.200 --> 17:50.260 And it's an unattended ISO. 17:50.340 --> 17:51.060 Meaning, there's nothing to do. 17:51.060 --> 17:52.900 That's for the VPN. 17:53.260 --> 17:57.600 A Proxmox role to configure cloud-init for each of the VMs. 17:57.680 --> 18:00.100 We use the OpenStack image. 18:00.260 --> 18:02.640 Let's say cloud, but without cloud. 18:02.860 --> 18:03.380 Local. 18:05.480 --> 18:07.020 To do a qm restore. 18:07.240 --> 18:12.120 Meaning, we can restore a VM from a backup and it boots up. 18:12.300 --> 18:14.620 And so, there's a playbook for everything. 18:15.360 --> 18:18.400 Well, this is roughly what the migration project was. 18:19.480 --> 18:23.580 I'll try to summarize, and move on to the new stuff. 18:23.680 --> 18:28.840 A bit about the difference from 2022 to 2026 of what's been worked on. 18:29.660 --> 18:33.740 Well, there are over 200 git repositories to control all this. 18:36.740 --> 18:40.020 Now what's being done, part of what was actually done, 18:40.160 --> 18:42.760 we worked a lot on cybersecurity. 18:42.820 --> 18:46.120 We changed from rsyslog to auditd. 18:46.120 --> 18:52.560 We integrated NUT for UPSs, for UPS monitoring. 18:54.580 --> 18:59.880 We're working on updating the Debian versions of those VMs from 10 to 13. 19:00.840 --> 19:02.100 This, remember, started in 2018. 19:03.980 --> 19:06.860 The FreeIPA automation, now a new version was done... 19:07.000 --> 19:12.080 Meaning, what we had currently in Production was only one large 19:12.080 --> 19:13.140 FreeIPA VM. 19:13.240 --> 19:15.140 And since the bank has OpenShift, 19:15.140 --> 19:19.140 we're doing a migration of FreeIPA to OpenShift. 19:19.560 --> 19:21.940 Also all with automated playbooks. 19:22.700 --> 19:27.820 Here as a summary of everything done for cybersecurity. 19:28.740 --> 19:32.480 It's a role that connects to each host and says what's right, what's wrong. 19:33.000 --> 19:35.440 Some things it can fix and others it simply says 19:35.440 --> 19:37.480 this cannot go to Production like this. 19:38.480 --> 19:41.500 We also worked on kernel booting, 19:41.500 --> 19:43.920 so that some server models would appear 19:43.920 --> 19:47.240 and to identify network cards and have everything work automatically 19:47.240 --> 19:51.860 and disk repartitioning, and there are different server levels. 19:55.780 --> 19:58.280 Regeneration of all these images. 19:59.140 --> 20:03.420 Everything related to the lifecycle of VMs that need to be provisioned. 20:04.320 --> 20:05.320 Several that were taken down. 20:05.540 --> 20:08.000 Others that changed because they didn't need a secondary disk. 20:08.720 --> 20:10.920 The order in which they boot up. 20:11.940 --> 20:13.900 A report of the status of all this. 20:13.920 --> 20:17.160 All those VMs and updates. 20:17.660 --> 20:21.600 Well, here's a bit about what I mentioned regarding server versions. 20:23.540 --> 20:26.480 And we'll move on since we're short on time. 20:26.600 --> 20:29.080 For workstations is where most of the work has been done, 20:29.080 --> 20:34.920 because originally we worked with Workstation 18.04 20:34.920 --> 20:37.980 and now we've moved to 24.04 if I'm not mistaken. 20:38.520 --> 20:42.460 And then there were some problems they had. 20:43.240 --> 20:49.380 Remember this original infrastructure wasn't GNU and worked with Firefox 9. 20:50.240 --> 20:55.640 We took it to v68 and now we're taking it to v120, and tests are being done with v140. 20:55.640 --> 20:56.360 pruebas con v140. 20:57.040 --> 21:01.340 The problem isn't really the environment, but the ecosystem 21:01.340 --> 21:05.280 of all the internal applications that aren't updated and don't work. 21:05.340 --> 21:07.260 Basically it's not an easy task. 21:07.540 --> 21:10.540 So for a while more than one browser version coexisted. 21:11.560 --> 21:12.080 Eh... 21:12.460 --> 21:14.580 And that was another challenge. 21:15.120 --> 21:19.000 And well, also issues around updating the kernel version. 21:19.520 --> 21:24.800 We had to patch some difficult CVEs that came out 21:24.800 --> 21:26.040 recently. 21:29.340 --> 21:33.320 Well, at some point we also tested Chrome as an alternative for some 21:33.320 --> 21:33.760 sites. 21:34.260 --> 21:37.700 We do everything with a playbook policy configuration 21:37.700 --> 21:39.280 Firefox settings. 21:39.640 --> 21:41.260 The user basically can't do anything. 21:41.260 --> 21:41.860 Meaning, like... 21:41.860 --> 21:43.380 They can't even change the desktop wallpaper. 21:44.220 --> 21:44.740 This... 21:44.740 --> 21:47.260 But well, it's the way to manage 3,000 users. 21:47.560 --> 21:48.500 Well then... 21:48.500 --> 21:50.500 Nobody goes into a host to configure it! 21:50.620 --> 21:52.620 From AWX a playbook is launched directly. 21:54.080 --> 21:55.020 And, uh... 21:55.020 --> 21:59.480 Well, the WakeOnLAN topic, integration, kernel updates, 21:59.740 --> 22:01.700 sftp fixes. 22:02.100 --> 22:06.240 Meaning, everything that would lead to the new IPA version. 22:06.760 --> 22:10.940 We worked quite a bit with AppArmor to restrict... 22:10.940 --> 22:11.500 um... 22:11.500 --> 22:13.160 Some things that are important. 22:13.760 --> 22:15.240 Login handling. 22:16.300 --> 22:19.300 Also handling the ability to change passwords. 22:19.480 --> 22:19.880 um... 22:19.880 --> 22:21.960 Well, backups and all that. 22:22.960 --> 22:24.100 And, uh... 22:24.100 --> 22:25.540 Let's see... 22:26.740 --> 22:28.220 A bit of AWX here. 22:28.740 --> 22:29.340 um... 22:30.040 --> 22:34.240 Well, there have been 198 releases since we finished migrating it. 22:34.260 --> 22:36.640 Because like any large infrastructure, when you finish migrating it 22:36.640 --> 22:37.340 you have to start migrating again. 22:37.480 --> 22:38.000 Basically. 22:38.360 --> 22:39.920 It's in constant change. 22:40.940 --> 22:44.640 And, well, all the time new needs come up... 22:45.380 --> 22:45.920 um... 22:46.940 --> 22:51.520 Well, we have CloneZilla to boot a host. 22:51.520 --> 22:52.600 with a pre-built image. 22:52.620 --> 22:53.140 um... 22:53.140 --> 22:54.220 And solving problems. 22:55.180 --> 22:55.720 um... 22:55.720 --> 22:56.400 Here, uh... 22:56.750 --> 22:57.680 Isolated profiles. 22:57.740 --> 22:58.280 um... 22:58.280 --> 23:00.560 Isolated from each other due to configuration issues. 23:01.200 --> 23:01.740 um... 23:01.740 --> 23:03.260 A CCTV component was added. 23:03.300 --> 23:07.000 Equipment integrated with the security camera DVRs. 23:07.960 --> 23:08.500 um... 23:08.500 --> 23:11.920 So, well, that's a new inventory that was added. 23:12.600 --> 23:13.020 um... 23:13.390 --> 23:15.460 HP Linux Tools, for example, this... 23:15.460 --> 23:18.120 A detail is that at one point, uh... 23:18.120 --> 23:22.540 A host would freeze and hang in strange situations. 23:22.940 --> 23:25.180 It took us a while to investigate what was happening. 23:25.500 --> 23:28.180 It had to do with a BIOS power saving configuration. 23:28.420 --> 23:30.480 And so the solution was simple. 23:30.640 --> 23:34.620 It was to go into 3,000 hosts, change the BIOS config and reboot them. 23:34.660 --> 23:34.840 Done. 23:34.960 --> 23:35.460 It's very simple. 23:36.140 --> 23:37.340 Do you need 3,000 technicians? 23:37.560 --> 23:37.920 or 23:39.440 --> 23:39.840 um... 23:39.840 --> 23:41.740 As many as there are branches across the country. 23:41.960 --> 23:42.800 It's impossible. 23:43.160 --> 23:44.480 And, well, there, uh... 23:44.480 --> 23:46.720 I started investigating HP's FTP. 23:46.940 --> 23:51.240 And I found a guy in Linux who had a tool that allows you 23:51.240 --> 23:53.460 to write to the UEFI. 23:54.380 --> 23:54.780 um... 23:54.780 --> 23:57.560 With the BIOS configuration it will take on the next reboot. 23:57.800 --> 23:59.540 This requires compiling a kernel module and such. 23:59.940 --> 24:00.340 And... 24:01.240 --> 24:01.640 um... 24:01.640 --> 24:05.440 We told them to ask HP for permission and guarantee they wouldn't 24:05.460 --> 24:06.440 turn into 3,000 bricks. 24:06.520 --> 24:06.940 um... 24:06.940 --> 24:07.460 They said... 24:07.460 --> 24:07.740 Yes! 24:08.440 --> 24:10.420 And, well, we did several tests. 24:10.540 --> 24:11.400 It never happened to us. 24:11.520 --> 24:13.220 So they all booted up fine. 24:13.880 --> 24:14.400 um... 24:14.400 --> 24:18.080 And that allows, well, centralized administration again, 24:18.080 --> 24:21.920 you could even change the BIOS password on all hosts 24:21.920 --> 24:22.900 centrally and remotely. 24:22.960 --> 24:24.220 So it's super useful. 24:25.280 --> 24:25.800 um... 24:27.040 --> 24:27.560 Well. 24:27.760 --> 24:29.420 Here, for example, uh... 24:30.300 --> 24:32.980 Modification of Git tokens and... 24:32.980 --> 24:34.040 There are a lot of things. 24:34.040 --> 24:34.720 This... 24:34.720 --> 24:40.840 At the URL shown below, https://filiales-gnu-linux.g.coop.ar 24:40.840 --> 24:41.280 All of this will be available. 24:41.340 --> 24:42.780 So if you want to see it in detail. 24:43.960 --> 24:44.440 um... 24:44.440 --> 24:46.380 And since we have five minutes left, uh... 24:46.380 --> 24:51.220 If you agree, I'll wrap up here and you can ask any questions... 24:51.340 --> 24:52.920 Because we won't get to see everything. 25:07.720 --> 25:08.200 Hello. 25:17.260 --> 25:17.740 um... 25:17.740 --> 25:22.180 Are the ActiveDirectories you use Windows or do you also use Windows Server? 25:22.200 --> 25:23.900 Yes, it's Bank infrastructure. 25:24.280 --> 25:24.760 um... 25:24.760 --> 25:26.040 That was already preexisting. 25:26.040 --> 25:27.100 And they weren't going to change it. 25:29.640 --> 25:33.600 The challenge was to get all those users to work on the new 25:33.600 --> 25:33.880 workstations. 25:35.240 --> 25:35.680 um... 25:35.680 --> 25:36.500 And it works! 25:47.640 --> 25:49.620 First of all, impressive. 25:50.620 --> 25:52.460 Thanks to the Free Software community! Uh... 25:52.480 --> 25:54.020 We put the pieces together. 25:54.240 --> 25:57.660 They got the most out of Ansible, but not I thought so much could really 25:57.660 --> 25:57.980 be done 25:58.580 --> 26:01.660 I had a small doubt about something I saw. 26:02.140 --> 26:05.020 You said you changed from rsyslog to auditd. 26:05.180 --> 26:05.580 Yes. 26:06.420 --> 26:08.060 It was a cybersecurity requirement. 26:08.780 --> 26:09.840 But the intention... 26:09.840 --> 26:14.980 Let's say, rsyslog logs at an application level, maybe a 26:14.980 --> 26:15.260 system level. 26:15.380 --> 26:18.280 And auditd focuses more on the syscalls of the system itself. 26:18.640 --> 26:20.500 Yes, but it allows more detail. 26:21.460 --> 26:24.160 You can specify exactly which parts you want. 26:24.720 --> 26:25.500 No, no, no. 26:25.520 --> 26:25.900 I see. 26:26.140 --> 26:33.420 My doubt is, are you logging syscalls from 3,000 clients? 26:33.420 --> 26:34.360 That is... 26:34.360 --> 26:35.900 Not from all the workstations. 26:35.900 --> 26:37.100 Actually it is... 26:38.200 --> 26:38.760 Eh... 26:38.760 --> 26:40.540 mainly from VMs, and some things. 26:40.720 --> 26:41.760 It's selective, not everything. 26:42.000 --> 26:42.580 Ah, ah. 26:42.620 --> 26:43.480 No, because otherwise there's no way. 26:43.880 --> 26:44.640 No, no, it's fine. 26:44.760 --> 26:45.320 That was just it. 26:45.540 --> 26:47.260 Exactly to avoid sending all the logs. 26:47.460 --> 26:49.640 Meaning, sending selectively. 26:49.980 --> 26:53.760 All this information goes to a SIEM at Headquarters. 26:54.040 --> 26:54.460 Just one? 26:55.120 --> 26:57.660 Yes, well, it could be more than one node. 26:58.040 --> 26:58.480 Ah, no. 26:58.500 --> 27:02.940 But, let's say, conceptually it goes to a SIEM that everything is then displayed 27:02.940 --> 27:03.520 there what happens. 27:04.200 --> 27:06.080 And all of AWX deploy logs also 27:06.080 --> 27:09.020 go to the SIEM... 27:09.220 --> 27:10.260 So also... 27:10.260 --> 27:10.620 As a SIEM? 27:11.040 --> 27:15.620 No, another one I don't remember now, but it's well-known. 27:17.000 --> 27:18.680 Now, if I tell you, I'd be lying. 27:18.700 --> 27:18.760 Thank you very much. 27:23.580 --> 27:28.660 I agree with the colleague on the hard work, tremendous effort. 27:29.240 --> 27:31.440 And all done with Free Software. 27:31.960 --> 27:35.900 My question is basically based on the experience of the entire project. 27:37.940 --> 27:39.340 What went really well? 27:39.420 --> 27:39.780 What went really well? 27:40.600 --> 27:43.280 What are you going to start doing, besides what you've mentioned? 27:43.360 --> 27:47.200 And what things went one way and you said, no, this isn't the right approach, 27:47.340 --> 27:48.760 we pivoted? 27:48.820 --> 27:55.860 The biggest challenge overall isn't technical per se, but rather dealing with 27:55.860 --> 27:58.100 the priorities of everything that needs to be done. 27:58.280 --> 28:05.320 And with things where at the user level, many free applications aren't designed 28:05.320 --> 28:08.380 for such a large scale. 28:08.380 --> 28:12.420 And a problem where you want users to be unable to touch anything. 28:12.560 --> 28:18.500 For example, there are some PDFs that are smart forms you can fill 28:18.500 --> 28:19.480 fill out. 28:19.760 --> 28:21.580 And then you have to print them. 28:21.800 --> 28:24.180 And that was really complex to solve. 28:24.300 --> 28:28.300 At first we had to install Adobe Acrobat with, I don't know, Wine or 28:28.300 --> 28:29.100 something horrible like that. 28:29.760 --> 28:31.720 Because it was the only thing that supported it. 28:33.300 --> 28:36.620 Evince displayed them, but didn't allow us to fill them out. 28:38.780 --> 28:42.240 Later we had an option I don't remember which one that allowed 28:42.240 --> 28:42.760 filling them out. 28:42.760 --> 28:48.880 But it didn't let you hide the comments on each form, 28:48.880 --> 28:50.320 in each textbox. 28:50.580 --> 28:52.420 And that would print out and wouldn't work. 28:52.920 --> 28:56.840 Now there's been work on Okular, which has all of that. 28:57.020 --> 29:01.100 But it has no per-file configuration option to 29:01.100 --> 29:02.220 disable all that. 29:02.280 --> 29:04.060 Meaning the user can disable it at the moment. 29:05.060 --> 29:06.440 Well, so we worked on it. 29:06.620 --> 29:09.640 We're touching Okular's code to make that work. 29:10.300 --> 29:13.560 The same thing happened with some binaries that have no translation and no translation 29:13.560 --> 29:13.840 support. 29:13.960 --> 29:16.480 And well, what we could we edited with a hexadecimal editor. 29:16.660 --> 29:18.360 And embedded our binary. 29:19.320 --> 29:25.860 Things like that, let's say, with respect to the Asians, we call them Chinese. 29:25.980 --> 29:29.620 Meaning, it's a hack that sometimes you have to dig into and resolve. 29:29.700 --> 29:34.640 Or complex bugs where, I don't know, you enter a screen and for various reasons, 29:34.640 --> 29:36.380 if you move the mouse a bit more down to the right, 29:36.380 --> 29:37.640 it crashes GNOME. 29:37.880 --> 29:41.880 And so you have to put a script that prevents you from going down there. 29:43.120 --> 29:51.720 And also things that a user who owns their environment doesn't have a problem with 29:51.720 --> 29:52.600 because they can customize it. 29:53.000 --> 29:56.660 And an end user can't touch anything. 29:56.880 --> 30:00.900 And you, as administrator of this infrastructure, also don't want them to 30:00.900 --> 30:01.240 touch it. 30:01.280 --> 30:02.520 But you have to give a solution. 30:05.220 --> 30:06.420 So, a thousand things. 30:07.400 --> 30:11.240 What we didn't get involved in and outsourced was the check scanner 30:11.240 --> 30:11.640 integration part. 30:11.760 --> 30:14.160 I tried for a while and couldn't make it work. 30:14.200 --> 30:15.380 That was outsourced. 30:16.900 --> 30:17.640 But it works now. 30:18.480 --> 30:20.700 Yes, we did manage to get the ticket printer to work properly. 30:21.460 --> 30:28.660 And that implied a challenge because the banking core at the time used 30:28.660 --> 30:29.420 Java applets. 30:30.320 --> 30:32.500 And that, nothing, was impossible. 30:32.520 --> 30:36.520 We had to put a chroot in the middle with something old for that to work. 30:36.560 --> 30:38.220 Well, in the end we managed to avoid all of that. 30:38.640 --> 30:44.160 Because now that core already detects when it's GNU/Linux. 30:44.360 --> 30:51.600 And so, it outputs code and we put in a local backend that does 30:51.600 --> 30:52.900 everything needed and it works. 30:53.020 --> 30:54.940 But yes, it's a challenge involving many people. 30:55.040 --> 30:56.580 I'm currently not on the project. 30:57.240 --> 31:00.600 I was in the initial development and migration phase. 31:01.200 --> 31:02.500 There are three people now... 31:02.520 --> 31:04.180 who work full-time on this. 31:04.420 --> 31:06.260 And every day something new appears. 31:08.440 --> 31:08.880 Great. 31:15.540 --> 31:17.680 I can stay here and chat. 31:21.620 --> 31:22.060 Well. 31:23.080 --> 31:24.280 OSiRiS, first of all thank you. 31:25.620 --> 31:27.520 For those of us who came here to learn. 31:29.240 --> 31:32.240 The admiration you generate with everything you've shared. 31:32.380 --> 31:33.140 My question isn't about... 31:33.140 --> 31:34.200 It's gcoop's work too, not mine... 31:34.520 --> 31:35.220 Everyone's. 31:35.400 --> 31:36.620 My question goes more... 31:36.620 --> 31:37.240 I'm selling it. 31:37.600 --> 31:38.520 It goes more that way. 31:38.620 --> 31:39.620 Not the technical side. 31:39.700 --> 31:43.220 But rather the team or human group. 31:43.240 --> 31:43.520 Not the human side. 31:43.560 --> 31:45.120 What's at Filiales GNU/Linux 31:45.120 --> 31:49.480 that if I don't understand is the organization behind all 31:49.480 --> 31:50.540 this project. 31:51.240 --> 31:53.520 How many people are needed to do this? 31:53.680 --> 31:55.540 And how do you manage it? 31:55.920 --> 31:59.480 Well, initially it was a six-month project. 31:59.540 --> 32:00.660 Of two people. 32:01.440 --> 32:03.280 A functional analyst and me. 32:04.100 --> 32:06.300 To see if the project was viable. 32:08.300 --> 32:09.500 Then it started... 32:09.500 --> 32:11.060 I think we started with three people. 32:11.720 --> 32:13.220 At the peak we reached three plus five. 32:13.240 --> 32:13.380 To five? 32:13.780 --> 32:14.360 Three plus five. 32:16.330 --> 32:16.890 Right, yes. 32:17.510 --> 32:18.450 Always with a PM. 32:18.630 --> 32:20.370 No, but at the peak it was five, not that way. 32:20.570 --> 32:21.750 Meaning, I think there... 32:21.750 --> 32:23.450 I don't count the PM, poor guy. 32:23.970 --> 32:25.690 Because he always talks about... 32:25.690 --> 32:27.850 The PM always talks about how we're not people. 32:27.890 --> 32:28.990 That we are developers! 32:29.510 --> 32:30.690 That we are weird things. 32:30.790 --> 32:31.210 So, well. 32:31.450 --> 32:32.330 It's revenge. 32:32.830 --> 32:34.070 No, at the peak it was five people. 32:35.630 --> 32:36.590 Today there are three. 32:37.370 --> 32:38.630 DevOps people, let's say. 32:38.850 --> 32:40.010 But you have to... 32:40.750 --> 32:42.150 Let's say, it's full stack. 32:42.550 --> 32:43.750 So to speak. 32:43.750 --> 32:47.530 But then you have to touch or redesign an application. 32:47.770 --> 32:51.630 Meaning, we had to design applications to work around existing issues. 32:52.370 --> 32:55.550 Meaning, it's the fun and complex part at the same time. 32:56.130 --> 32:58.030 I thought you were going to say three hundred, I don't know. 32:58.170 --> 32:58.830 Thank you very much. 32:58.830 --> 32:59.430 No, no, sorry. 32:59.430 --> 33:03.950 We designed the automation for all of this. 33:04.030 --> 33:06.870 The bank has its own arsenal. 33:06.890 --> 33:10.430 An army of people who use all of this daily. 33:10.670 --> 33:13.470 They did the migration in the middle of the 2020 pandemic. 33:13.750 --> 33:15.050 In less than a year! 33:16.190 --> 33:19.910 But it was a physical army of people who resolved all of that. 33:20.250 --> 33:22.530 AWX operators, I think there are about sixty. 33:22.710 --> 33:23.210 Meaning, like... 33:23.770 --> 33:25.030 Well, there's a lot of people. 33:25.910 --> 33:27.270 The bank's infrastructure is very large. 33:27.410 --> 33:28.890 And it's all On-premise. 33:28.950 --> 33:29.530 It's all local. 33:29.770 --> 33:31.690 Meaning, and all with Free Software. 33:32.810 --> 33:33.530 Thank you very much. 33:34.670 --> 33:35.470 There, Alejandro. 33:35.550 --> 33:36.590 I'm putting a question in the chat for you. 33:36.730 --> 33:38.090 If you can answer it there later. 33:38.110 --> 33:38.370 Yes. 33:38.370 --> 33:40.370 And later I have here colleagues who speak English. 33:40.490 --> 33:42.590 And if there's anyone who doesn't speak Spanish. 33:43.750 --> 33:43.950 Go ahead. 33:43.970 --> 33:45.970 And Alejandro will be around today. 33:45.990 --> 33:46.890 So you can keep asking him questions. 33:47.050 --> 33:47.590 Thank you very much. 33:47.590 --> 33:47.910 OSiUX too!