1 00:00:10,770 --> 00:00:16,390 ...in the community, I work at gcoop, which is a Free Software Cooperative, 2 00:00:16,630 --> 00:00:19,450 meaning that we work exclusively with Free Software 3 00:00:19,450 --> 00:00:22,470 and horizontally, as an organization. 4 00:00:22,490 --> 00:00:24,130 We have no bosses, we have no employees. 5 00:00:26,370 --> 00:00:30,030 Five of our partners from gcoop came to the event, 6 00:00:30,210 --> 00:00:32,270 it's our first time at DebConf for everyone 7 00:00:32,950 --> 00:00:36,590 We sponsored this event and give thanks to Debian 8 00:00:36,590 --> 00:00:39,050 because we have 20 years working with Free Software 9 00:00:39,050 --> 00:00:40,870 and it wouldn't have been possible without Debian. 10 00:00:41,850 --> 00:00:47,430 I'm going to show an update of a talk I gave before in 2022, 11 00:00:47,770 --> 00:00:52,310 which is about the GNU/Linux migration project of Filiales 12 00:00:52,310 --> 00:00:52,850 from Banco Credicoop 13 00:00:53,870 --> 00:00:56,050 Cooperativo Limitado, it's a cooperative bank. 14 00:00:56,570 --> 00:01:03,050 And I'll try to quickly review the migration process 15 00:01:03,630 --> 00:01:06,370 and then we'll look at the differences in recent years. 16 00:01:07,490 --> 00:01:08,690 This project... 17 00:01:14,650 --> 00:01:17,590 Basically it's a project of Infrastructure as Code 18 00:01:17,590 --> 00:01:21,050 using Ansible and AWX. 19 00:01:21,170 --> 00:01:28,430 AWX is a web platform, which is a graphical interface for Ansible, 20 00:01:28,550 --> 00:01:33,650 but on the other hand it allows orchestrating and managing and administering 21 00:01:33,650 --> 00:01:35,190 all the Ansible playbooks. 22 00:01:35,390 --> 00:01:38,670 and this was done for Banco Credicoop. 23 00:01:39,050 --> 00:01:39,570 from gcoop. 24 00:01:43,130 --> 00:01:48,550 So, at the beginning of the project, what we had to do was, over 25 00:01:48,550 --> 00:01:48,810 6 months, 26 00:01:49,350 --> 00:01:53,290 was trying to discover what tools in the Free Software ecosystem 27 00:01:53,290 --> 00:01:58,190 would allow us to carry out the entire infrastructure migration 28 00:01:58,190 --> 00:01:58,550 of a bank 29 00:01:58,550 --> 00:02:01,210 throughout all of Argentina. 30 00:02:01,690 --> 00:02:05,250 And what we found was that the automation tool was 31 00:02:05,250 --> 00:02:05,530 Ansible 32 00:02:05,530 --> 00:02:06,950 because we were already using it. 33 00:02:07,430 --> 00:02:09,230 That we would use GitLab. 34 00:02:10,710 --> 00:02:16,270 That for ActiveDirectory authentication within Linux, 35 00:02:16,270 --> 00:02:16,890 we would use IPA. 36 00:02:18,450 --> 00:02:21,290 AWX, obviously all the servers would use Proxmox. 37 00:02:21,390 --> 00:02:23,670 We had already been working for years with Proxmox. 38 00:02:24,070 --> 00:02:27,210 The VMs inside the Proxmox servers would be Debian, obviously. 39 00:02:27,970 --> 00:02:31,730 And so, for workstations, due to the fact that there were 40 00:02:31,730 --> 00:02:32,330 3,000 workstations 41 00:02:33,010 --> 00:02:38,070 and because of support reasons, we chose a Debian-based distribution 42 00:02:38,970 --> 00:02:39,450 um... 43 00:02:39,450 --> 00:02:40,650 A slightly more updated version. 44 00:02:41,230 --> 00:02:45,270 And what we had there, it's not very visible, but what's moving is that 45 00:02:45,270 --> 00:02:45,890 ...underneath 46 00:02:45,890 --> 00:02:48,890 all the components, it's a timeline of all the components we use 47 00:02:48,890 --> 00:02:49,370 for dependencies 48 00:02:49,370 --> 00:02:54,030 that are being updated worldwide, all the 49 00:02:54,030 --> 00:02:54,330 time. 50 00:02:54,630 --> 00:02:59,550 Meaning, at that point in a given month, a new version of a component 51 00:02:59,550 --> 00:03:01,510 came out, along with another component and another. 52 00:03:01,630 --> 00:03:04,350 And this evolves as you're thinking about how to approach things, the entire ecosystem updates. 53 00:03:04,350 --> 00:03:05,470 the entire ecosystem keeps updating. 54 00:03:05,950 --> 00:03:08,850 So it's a problem to solve. 55 00:03:10,710 --> 00:03:18,490 But what we achieved is that in Development we managed to orchestrate an idea 56 00:03:18,490 --> 00:03:19,030 of deployment, 57 00:03:19,290 --> 00:03:26,110 which is basically: we have an AWX, let me point it out like this, 58 00:03:26,910 --> 00:03:36,210 this is an AWX, and its source of truth is a GitLab, 59 00:03:36,270 --> 00:03:38,610 so AWX reads from GitLab 60 00:03:38,610 --> 00:03:40,030 everything that would be... 61 00:03:40,030 --> 00:03:45,830 Meaning playbooks, and the first step is deploying the iDRAC. 62 00:03:45,950 --> 00:03:48,730 The iDRAC is the computer of the computer, inside the servers 63 00:03:48,730 --> 00:03:49,610 inside the Dell servers we use 64 00:03:49,610 --> 00:03:53,750 before the server even has anything; there's a computer called iDRAC, 65 00:03:53,770 --> 00:03:58,030 using a protocol called Redfish, we wrote playbooks that AWX uses to 66 00:03:58,030 --> 00:03:59,570 connect to iDRAC 67 00:03:59,570 --> 00:04:06,190 and what it does is partition the disk, configure BIOS and reboot into PXE. 68 00:04:06,830 --> 00:04:10,010 That is, a new computer, fresh out of the box. 69 00:04:10,030 --> 00:04:19,530 And immediately that will end up generating from a Proxmox, 70 00:04:19,530 --> 00:04:21,110 it will ultimately generate another Proxmox. 71 00:04:21,310 --> 00:04:23,610 Inside that Proxmox We have several VMs. 72 00:04:23,810 --> 00:04:32,750 One of those VMs is a PXE server, I don't see anything here, but it must be this one. 73 00:04:34,650 --> 00:04:40,010 We have a server, well we have a CDN, which is a processing server, 74 00:04:40,030 --> 00:04:45,950 an Nginx that has all the resources, which in turn is a proxy for another, 75 00:04:45,950 --> 00:04:46,370 an Apache, 76 00:04:48,170 --> 00:04:51,790 and a server that replicates this data. 77 00:04:52,350 --> 00:04:57,230 This is all done in an automated, unattended way, without 78 00:04:57,230 --> 00:04:58,230 operator intervention, 79 00:04:58,370 --> 00:05:03,930 meaning a server just comes out of the factory, the box is opened, 80 00:05:03,930 --> 00:05:05,870 Power and Network are installed, 81 00:05:05,870 --> 00:05:11,330 and from AWX a script is launched, 82 00:05:12,950 --> 00:05:16,890 this is the AWX infrastructure, sorry, the Proxmox one, we were going to see, 83 00:05:16,950 --> 00:05:17,550 you can see it here, 84 00:05:17,650 --> 00:05:23,170 how it's launched, and this is just for monitoring, let's say, 85 00:05:23,290 --> 00:05:27,090 everything is done in an unattended manner and the operator does not need to see 86 00:05:27,090 --> 00:05:28,090 anything running 87 00:05:28,190 --> 00:05:33,390 AWX deploys Proxmox via PXE; it's actually several stages, 88 00:05:33,390 --> 00:05:35,730 first it deploys a Debian NetInstall 89 00:05:35,870 --> 00:05:40,050 completely unattended, meaning there's nothing to touch 90 00:05:40,050 --> 00:05:40,370 at all, 91 00:05:40,830 --> 00:05:45,230 it goes directly into OEM mode, configuring everything needed for that 92 00:05:45,230 --> 00:05:47,370 server to be operational, 93 00:05:47,450 --> 00:05:52,830 and when it finishes, in a hook at the end of the Debian NetInstaller, 94 00:05:54,230 --> 00:05:58,890 we wrote a script that takes the MAC address of that host, which 95 00:05:58,890 --> 00:05:59,150 is unique, 96 00:05:59,410 --> 00:06:02,610 and registers it in the AWX inventory. 97 00:06:03,090 --> 00:06:05,590 So that host, which just received a random IP, 98 00:06:05,870 --> 00:06:09,730 we now know what its IP will be, what its iDRAC is, 99 00:06:09,750 --> 00:06:14,370 permanently, for all the years this server will have, 100 00:06:15,690 --> 00:06:22,250 referenced to that MAC and to a serial number from the 101 00:06:22,250 --> 00:06:22,510 iDRAC itself. 102 00:06:22,650 --> 00:06:24,830 So, in this way it's easy to identify it. 103 00:06:25,990 --> 00:06:30,950 On the left, all you see is the debug of what's happening on the 104 00:06:30,950 --> 00:06:33,370 PXE server, 105 00:06:34,150 --> 00:06:35,850 and as you can see, the server is on the PXE server, 106 00:06:35,850 --> 00:06:40,430 the server installs itself with no manual intervention and finishes installed, 107 00:06:40,470 --> 00:06:41,290 it's a Debian. 108 00:06:41,570 --> 00:06:47,050 The next playbook launched on AWX converts that Debian into a 109 00:06:47,050 --> 00:06:47,450 Proxmox, 110 00:06:47,590 --> 00:06:49,470 also in an unattended way. 111 00:06:49,530 --> 00:06:52,670 Meaning we don't install Proxmox directly; we install Debian and 112 00:06:52,670 --> 00:06:53,590 convert it to Proxmox. 113 00:06:54,030 --> 00:06:59,530 That's an advantage of Proxmox being a Debian-based distro, 114 00:06:59,670 --> 00:07:01,490 where the only thing that changes is adding a repo. 115 00:07:01,670 --> 00:07:04,310 And done, all the necessary packages are configured. 116 00:07:04,430 --> 00:07:05,830 And when configuration finishes, 117 00:07:05,830 --> 00:07:10,230 on a Proxmox, we launch another playbook that creates the VMs, 118 00:07:10,830 --> 00:07:15,510 and another playbook that creates the services inside each VM. 119 00:07:15,710 --> 00:07:17,410 All the VMs we use are KVM. 120 00:07:18,510 --> 00:07:21,570 This is a bank, it needs greater robustness in security, 121 00:07:21,810 --> 00:07:23,830 so all the hosts are up to date. 122 00:07:24,250 --> 00:07:27,290 On the left you can see the creation of all the different VMs. 123 00:07:27,550 --> 00:07:33,390 There are 10 VMs per server, and roughly rounding off, about 300 physical 124 00:07:33,390 --> 00:07:33,990 servers 125 00:07:34,070 --> 00:07:35,810 That gives us a total of 3,000 Debian hosts. 126 00:07:35,830 --> 00:07:39,630 Installed in an automated way across the entire country. 127 00:07:40,990 --> 00:07:45,790 And similar to the Proxmox, we have the workstations. 128 00:07:45,850 --> 00:07:47,870 For the workstations we created an image. 129 00:07:47,910 --> 00:07:53,490 Here we're seeing a test virtual workstation from Development, inside 130 00:07:53,490 --> 00:07:54,210 a Proxmox. 131 00:07:54,590 --> 00:08:01,870 And what we do is generate a base image, also a NetInstall, 132 00:08:02,090 --> 00:08:05,230 in this case based on a Debian-based distro. 133 00:08:05,830 --> 00:08:11,230 And then we configure it, also in the same way, with a script. 134 00:08:11,890 --> 00:08:18,630 It takes the hostname of the host, which can be configured later or reconfigured 135 00:08:18,630 --> 00:08:19,930 at the destination location. 136 00:08:20,070 --> 00:08:24,630 It's registered in the inventory and also becomes available with a manufacturer 137 00:08:24,630 --> 00:08:26,050 serial number. 138 00:08:26,070 --> 00:08:28,410 They're all HP workstations. 139 00:08:28,690 --> 00:08:32,570 We can uniquely identify them, even if the MAC address changes at 140 00:08:32,570 --> 00:08:32,930 some point. 141 00:08:32,930 --> 00:08:34,710 It's always the same device. 142 00:08:35,830 --> 00:08:39,490 The AWX web interface is basically a webpage built 143 00:08:39,490 --> 00:08:40,030 with Django. 144 00:08:40,490 --> 00:08:46,350 And there are many operators who work launching different playbooks 145 00:08:46,350 --> 00:08:49,850 that we create, to keep everything operational. 146 00:08:50,450 --> 00:08:55,290 And the PXE server also takes workstation images. 147 00:08:55,370 --> 00:08:58,950 Meaning you can take a workstation on-site, if it got misconfigured or something happened, 148 00:08:58,950 --> 00:09:00,690 doesn't matter. 149 00:09:00,850 --> 00:09:05,050 The full workstation image is rebuilt via PXE. 150 00:09:05,130 --> 00:09:05,810 No time is lost. 151 00:09:05,810 --> 00:09:06,590 It has time to see what's wrong with it. 152 00:09:06,950 --> 00:09:09,330 Nobody manually accesses the host. 153 00:09:09,470 --> 00:09:11,350 It's rebuilt from scratch. 154 00:09:11,450 --> 00:09:12,010 Why? 155 00:09:12,050 --> 00:09:14,830 Because there are no local data inside the workstation. 156 00:09:15,070 --> 00:09:22,950 The data is on an NFS server with Kerberos in the same branch network 157 00:09:22,950 --> 00:09:23,630 of the branch where we are. 158 00:09:24,990 --> 00:09:30,890 For deployment at the branch, it's a bit more complex because for branch deployment, 159 00:09:30,890 --> 00:09:32,730 as we said, we have 3,000 workstations. 160 00:09:32,830 --> 00:09:34,790 Of these 3,000 workstations, there are no users 161 00:09:34,790 --> 00:09:36,290 on the workstations. 162 00:09:36,450 --> 00:09:40,690 Meaning, if you look at /etc/passwd on those workstations, there are 163 00:09:40,690 --> 00:09:41,130 no local users 164 00:09:42,310 --> 00:09:44,150 beyond what comes with the system. 165 00:09:44,630 --> 00:09:50,050 What we use is a FreeIPA client that connects to IPA, and from IPA 166 00:09:50,050 --> 00:09:52,410 it connects to the 4 ADs we have. 167 00:09:52,610 --> 00:09:56,090 And they pull those domain users directly on the fly. 168 00:09:56,670 --> 00:10:00,730 And for this reason each workstation also needs to be enrolled in the domain. 169 00:10:00,830 --> 00:10:03,910 But what happens is they enroll to the FreeIPA domain. 170 00:10:03,910 --> 00:10:04,770 And FreeIPA connects to the FreeIPA domain. 171 00:10:04,770 --> 00:10:07,870 And FreeIPA has a trust relationship with the ActiveDirectory 172 00:10:07,870 --> 00:10:08,170 domain 173 00:10:08,550 --> 00:10:09,790 Meaning it's like a subdomain. 174 00:10:10,290 --> 00:10:14,910 And what this allows is for all those users to have their password in the AD, 175 00:10:14,910 --> 00:10:17,390 which is the Bank's infrastructure, without changing anything for them. 176 00:10:17,830 --> 00:10:19,210 For them it's transparent. 177 00:10:19,370 --> 00:10:25,190 And from all the GNU/Linux hosts they use the same user directly. 178 00:10:25,430 --> 00:10:27,790 What's in the middle that enables this is called FreeIPA. 179 00:10:28,070 --> 00:10:31,810 And to speed up the deployment process, what we have new is a cache 180 00:10:31,810 --> 00:10:32,350 of cache of cache. 181 00:10:32,470 --> 00:10:33,710 We have several proxies. 182 00:10:34,950 --> 00:10:38,870 Initially we had Debian Apt-Cacher. 183 00:10:39,090 --> 00:10:41,330 Now there are also some Debian mirrors. 184 00:10:42,390 --> 00:10:46,150 And then we have Nginx and Squid, all the way down. 185 00:10:46,150 --> 00:10:50,150 So this works throughout the entire country. 186 00:10:53,070 --> 00:10:56,110 Here's a bit of the hardware it started with. 187 00:10:56,190 --> 00:10:58,530 3,000 HP ProDesk. 188 00:10:59,250 --> 00:11:00,730 3,000 Dell servers. 189 00:11:00,910 --> 00:11:02,250 Sorry, 300 Dell servers. 190 00:11:02,250 --> 00:11:04,070 3,000 virtual Debian hosts. 191 00:11:04,090 --> 00:11:05,310 Then it gradually changed a bit. 192 00:11:05,590 --> 00:11:09,750 But then we also found 3,500 different peripherals. 193 00:11:10,270 --> 00:11:13,870 Weird things like a check scanner or a ticket printer. 194 00:11:14,230 --> 00:11:19,190 This is what the original infrastructure was that the project started 195 00:11:19,190 --> 00:11:20,350 with in 2018. 196 00:11:21,650 --> 00:11:24,010 It's been evolving. 197 00:11:24,810 --> 00:11:26,550 Some VMs we've also changed. 198 00:11:27,990 --> 00:11:30,650 And quickly what gave us problems. 199 00:11:30,650 --> 00:11:32,930 Scaling problems. 200 00:11:33,090 --> 00:11:39,850 This is a graph of login attempts during the day. 201 00:11:40,050 --> 00:11:44,150 And what you'll see is there's a small problem here at 10 in the morning. 202 00:11:44,390 --> 00:11:48,170 Meaning, 3,000 people wanted to log in at 10 AM. 203 00:11:49,450 --> 00:11:52,130 You see it too when they drop off after 4 PM. 204 00:11:52,170 --> 00:11:54,090 The logins start going up again. 205 00:11:55,030 --> 00:12:00,110 And so, there's a problem with delay and caching here. 206 00:12:00,110 --> 00:12:03,570 And other problems, imagine with 3,000 users, 207 00:12:03,710 --> 00:12:07,570 with the robust security policies of a bank, 208 00:12:07,810 --> 00:12:11,110 every day many passwords expire and they have to be changed and all that. 209 00:12:11,310 --> 00:12:15,870 And this change has to be done from the login screen of 210 00:12:15,870 --> 00:12:16,070 our distro 211 00:12:16,410 --> 00:12:19,150 Meaning directly from there. 212 00:12:19,350 --> 00:12:23,010 They're told the password has expired and at that moment it asks for the old one, twice 213 00:12:23,010 --> 00:12:23,410 the new one. 214 00:12:23,970 --> 00:12:25,750 In the morning early, someone gets confused. 215 00:12:25,810 --> 00:12:28,410 That generates another wrong password. 216 00:12:28,490 --> 00:12:30,090 But in this... 217 00:12:30,090 --> 00:12:33,070 Now I'll tell you what was done to improve it. 218 00:12:33,810 --> 00:12:36,710 And so, to handle this too, well, 219 00:12:37,370 --> 00:12:40,210 part of the automation we achieved is, 220 00:12:41,050 --> 00:12:44,490 instead of going to AWX, which is a graphical interface, 221 00:12:44,490 --> 00:12:48,870 and clicking to create playbooks, workflows, inventories, 222 00:12:49,630 --> 00:12:52,370 and granting permissions manually, 223 00:12:52,450 --> 00:12:56,630 what we did is a git repo called awx, 224 00:12:56,710 --> 00:12:58,390 a repo called inventory, 225 00:12:58,390 --> 00:13:02,370 and they're all JSON or YAML files, 226 00:13:02,450 --> 00:13:10,270 that GitLab CI directly verifies and deploys 227 00:13:10,270 --> 00:13:10,570 into a Development AWX 228 00:13:10,570 --> 00:13:14,330 immediately as we do the git push. 229 00:13:14,890 --> 00:13:18,990 So in this way we can have all stages verified 230 00:13:18,990 --> 00:13:25,070 and automatically deployed on a Development AWX for testing. 231 00:13:25,230 --> 00:13:26,970 On the Production AWX, 232 00:13:26,970 --> 00:13:31,510 this deploy is triggered manually, let's say, but it creates the same 233 00:13:31,510 --> 00:13:31,990 instances. 234 00:13:32,210 --> 00:13:36,330 This guarantees having all infrastructure as versioned code. 235 00:13:37,270 --> 00:13:39,830 And to give you an idea of the project scale, 236 00:13:40,610 --> 00:13:44,650 this is a view of the Bank's productive infrastructure. 237 00:13:45,430 --> 00:13:48,190 I said, about 300 branches, 238 00:13:48,190 --> 00:13:50,870 distributed across almost all provinces of the country. 239 00:13:51,330 --> 00:13:54,410 And now, if we zoom in on this, we'll understand a bit more, 240 00:13:54,710 --> 00:13:55,490 all these are hosts. 241 00:13:56,130 --> 00:14:01,190 And all these hosts are controlled by one over here, 242 00:14:02,670 --> 00:14:04,050 which I'll find soon, there it is. 243 00:14:05,630 --> 00:14:09,470 This one here is the AWX, a single VM, 244 00:14:09,750 --> 00:14:11,250 not even on a physical host, 245 00:14:11,350 --> 00:14:13,890 which controls all the others and deploys to all the others. 246 00:14:14,410 --> 00:14:17,390 Obviously they don't deploy them all at once; they deploy in stages. 247 00:14:17,690 --> 00:14:20,930 And what we have is that each line, for example here, 248 00:14:22,110 --> 00:14:25,450 we'll see, this is the province of... 249 00:14:25,490 --> 00:14:25,630 the province of Santa Fe. 250 00:14:26,430 --> 00:14:29,370 I'll try to brighten it a bit. 251 00:14:33,400 --> 00:14:37,280 All of this is the hosts from all the different branches 252 00:14:37,280 --> 00:14:38,660 in the entire province of Santa Fe. 253 00:14:39,280 --> 00:14:47,800 And if we zoom in on one, here, for example, it's f0372. 254 00:14:48,160 --> 00:14:51,420 And so, within the concept of that branch, 255 00:14:51,460 --> 00:14:54,820 we have a local cache, which is a CDN, 256 00:14:54,980 --> 00:14:56,420 the PVE, which is the Proxmox server, 257 00:14:56,420 --> 00:15:00,460 the REP, which is the File Server, a Kerberized NFS, 258 00:15:02,140 --> 00:15:06,120 a log host, which receives logs from all the others 259 00:15:06,120 --> 00:15:06,480 and forwards them. 260 00:15:06,600 --> 00:15:09,480 Initially we did this with rsyslog. 261 00:15:09,700 --> 00:15:12,260 There's a VPN node that we don't intervene on, 262 00:15:12,300 --> 00:15:14,300 but we did automatic installation, 263 00:15:14,400 --> 00:15:20,000 which is basically what the Bank staff configures. 264 00:15:21,680 --> 00:15:25,260 We have an Apt-Cacher, here, local, within the branch. 265 00:15:26,300 --> 00:15:27,100 What else do we have? 266 00:15:27,400 --> 00:15:29,020 Well, this git one, in the end we dropped it. 267 00:15:30,340 --> 00:15:32,760 The print server, which is a CUPS, 268 00:15:33,280 --> 00:15:35,440 where all the printers at the location are configured. 269 00:15:35,720 --> 00:15:40,760 And then, here is the representation of the different printers at that branch. 270 00:15:41,000 --> 00:15:44,880 And then we'll have the different workstations. 271 00:15:46,240 --> 00:15:50,500 And finally the rest of the equipment there. 272 00:15:51,460 --> 00:15:53,120 Ticket printers and such. 273 00:15:53,540 --> 00:15:55,000 And this infrastructure, 274 00:15:55,000 --> 00:15:56,860 this infrastructure is repeated throughout the entire bank. 275 00:15:57,160 --> 00:15:59,800 Meaning that if you don't do it in an automated way, 276 00:16:00,040 --> 00:16:01,460 it's impossible to maintain. 277 00:16:01,800 --> 00:16:07,260 And what we achieved with this is that this AWX infrastructure 278 00:16:07,260 --> 00:16:10,940 allows us so that there's no longer a need to log into a host manually 279 00:16:10,940 --> 00:16:11,760 and see what happens. 280 00:16:12,320 --> 00:16:15,460 There's an AWX playbook, from Headquarters, 281 00:16:16,100 --> 00:16:19,460 where there's already a template to solve each problem. 282 00:16:19,680 --> 00:16:22,880 And you run that template and it records everything that happens, 283 00:16:23,080 --> 00:16:24,020 a traceable log. 284 00:16:25,120 --> 00:16:27,180 Obviously there are different permission levels. 285 00:16:27,220 --> 00:16:29,100 Who can do that? Who can't? 286 00:16:30,360 --> 00:16:31,160 Schedules, for example. 287 00:16:31,260 --> 00:16:34,220 I don't know, if we want all the hosts to shut down at a certain time, 288 00:16:34,300 --> 00:16:38,600 well, there's a template that sets a poweroff schedule at certain time 289 00:16:38,600 --> 00:16:38,960 for each host. 290 00:16:39,120 --> 00:16:40,420 For example, and you launch it. 291 00:16:40,440 --> 00:16:43,380 You can do things like SNMP sweeps 292 00:16:43,380 --> 00:16:46,580 to find out if certain hosts are alive or dead. 293 00:16:46,740 --> 00:16:49,200 Meaning, all of that ends up in a single, 294 00:16:49,260 --> 00:16:52,380 finally a PostgreSQL database, managed from AWX. 295 00:16:52,380 --> 00:16:57,000 And if we wanted to see a bit of what it's like, 296 00:16:57,080 --> 00:17:00,580 this is, as a view of all the playbooks there are in AWX. 297 00:17:01,360 --> 00:17:03,200 Here's our AWX. 298 00:17:03,680 --> 00:17:06,480 Here I tried to organize them a bit. 299 00:17:09,480 --> 00:17:12,180 Here we have iDRAC/Redfish. 300 00:17:15,960 --> 00:17:17,160 rsyslog, CDN. 301 00:17:18,580 --> 00:17:20,300 Well, here we have the inventory. 302 00:17:20,300 --> 00:17:22,360 I don't know if you can see anything there? 303 00:17:22,380 --> 00:17:23,600 Can't see it? 304 00:17:23,680 --> 00:17:24,860 Not even I can see it here! 305 00:17:26,080 --> 00:17:27,520 But, let's see, over here. 306 00:17:28,540 --> 00:17:34,040 Here, for example, we have a role that clones a KVM VM in Proxmox. 307 00:17:34,260 --> 00:17:36,280 So there's a role just for that. 308 00:17:36,900 --> 00:17:38,200 Over here we'll see more. 309 00:17:39,880 --> 00:17:45,420 This role creates a Proxmox KVM VM from an ISO 310 00:17:45,420 --> 00:17:46,660 directly. 311 00:17:47,200 --> 00:17:50,260 And it's an unattended ISO. 312 00:17:50,340 --> 00:17:51,060 Meaning, there's nothing to do. 313 00:17:51,060 --> 00:17:52,900 That's for the VPN. 314 00:17:53,260 --> 00:17:57,600 A Proxmox role to configure cloud-init for each of the VMs. 315 00:17:57,680 --> 00:18:00,100 We use the OpenStack image. 316 00:18:00,260 --> 00:18:02,640 Let's say cloud, but without cloud. 317 00:18:02,860 --> 00:18:03,380 Local. 318 00:18:05,480 --> 00:18:07,020 To do a qm restore. 319 00:18:07,240 --> 00:18:12,120 Meaning, we can restore a VM from a backup and it boots up. 320 00:18:12,300 --> 00:18:14,620 And so, there's a playbook for everything. 321 00:18:15,360 --> 00:18:18,400 Well, this is roughly what the migration project was. 322 00:18:19,480 --> 00:18:23,580 I'll try to summarize, and move on to the new stuff. 323 00:18:23,680 --> 00:18:28,840 A bit about the difference from 2022 to 2026 of what's been worked on. 324 00:18:29,660 --> 00:18:33,740 Well, there are over 200 git repositories to control all this. 325 00:18:36,740 --> 00:18:40,020 Now what's being done, part of what was actually done, 326 00:18:40,160 --> 00:18:42,760 we worked a lot on cybersecurity. 327 00:18:42,820 --> 00:18:46,120 We changed from rsyslog to auditd. 328 00:18:46,120 --> 00:18:52,560 We integrated NUT for UPSs, for UPS monitoring. 329 00:18:54,580 --> 00:18:59,880 We're working on updating the Debian versions of those VMs from 10 to 13. 330 00:19:00,840 --> 00:19:02,100 This, remember, started in 2018. 331 00:19:03,980 --> 00:19:06,860 The FreeIPA automation, now a new version was done... 332 00:19:07,000 --> 00:19:12,080 Meaning, what we had currently in Production was only one large 333 00:19:12,080 --> 00:19:13,140 FreeIPA VM. 334 00:19:13,240 --> 00:19:15,140 And since the bank has OpenShift, 335 00:19:15,140 --> 00:19:19,140 we're doing a migration of FreeIPA to OpenShift. 336 00:19:19,560 --> 00:19:21,940 Also all with automated playbooks. 337 00:19:22,700 --> 00:19:27,820 Here as a summary of everything done for cybersecurity. 338 00:19:28,740 --> 00:19:32,480 It's a role that connects to each host and says what's right, what's wrong. 339 00:19:33,000 --> 00:19:35,440 Some things it can fix and others it simply says 340 00:19:35,440 --> 00:19:37,480 this cannot go to Production like this. 341 00:19:38,480 --> 00:19:41,500 We also worked on kernel booting, 342 00:19:41,500 --> 00:19:43,920 so that some server models would appear 343 00:19:43,920 --> 00:19:47,240 and to identify network cards and have everything work automatically 344 00:19:47,240 --> 00:19:51,860 and disk repartitioning, and there are different server levels. 345 00:19:55,780 --> 00:19:58,280 Regeneration of all these images. 346 00:19:59,140 --> 00:20:03,420 Everything related to the lifecycle of VMs that need to be provisioned. 347 00:20:04,320 --> 00:20:05,320 Several that were taken down. 348 00:20:05,540 --> 00:20:08,000 Others that changed because they didn't need a secondary disk. 349 00:20:08,720 --> 00:20:10,920 The order in which they boot up. 350 00:20:11,940 --> 00:20:13,900 A report of the status of all this. 351 00:20:13,920 --> 00:20:17,160 All those VMs and updates. 352 00:20:17,660 --> 00:20:21,600 Well, here's a bit about what I mentioned regarding server versions. 353 00:20:23,540 --> 00:20:26,480 And we'll move on since we're short on time. 354 00:20:26,600 --> 00:20:29,080 For workstations is where most of the work has been done, 355 00:20:29,080 --> 00:20:34,920 because originally we worked with Workstation 18.04 356 00:20:34,920 --> 00:20:37,980 and now we've moved to 24.04 if I'm not mistaken. 357 00:20:38,520 --> 00:20:42,460 And then there were some problems they had. 358 00:20:43,240 --> 00:20:49,380 Remember this original infrastructure wasn't GNU and worked with Firefox 9. 359 00:20:50,240 --> 00:20:55,640 We took it to v68 and now we're taking it to v120, and tests are being done with v140. 360 00:20:55,640 --> 00:20:56,360 pruebas con v140. 361 00:20:57,040 --> 00:21:01,340 The problem isn't really the environment, but the ecosystem 362 00:21:01,340 --> 00:21:05,280 of all the internal applications that aren't updated and don't work. 363 00:21:05,340 --> 00:21:07,260 Basically it's not an easy task. 364 00:21:07,540 --> 00:21:10,540 So for a while more than one browser version coexisted. 365 00:21:11,560 --> 00:21:12,080 Eh... 366 00:21:12,460 --> 00:21:14,580 And that was another challenge. 367 00:21:15,120 --> 00:21:19,000 And well, also issues around updating the kernel version. 368 00:21:19,520 --> 00:21:24,800 We had to patch some difficult CVEs that came out 369 00:21:24,800 --> 00:21:26,040 recently. 370 00:21:29,340 --> 00:21:33,320 Well, at some point we also tested Chrome as an alternative for some 371 00:21:33,320 --> 00:21:33,760 sites. 372 00:21:34,260 --> 00:21:37,700 We do everything with a playbook policy configuration 373 00:21:37,700 --> 00:21:39,280 Firefox settings. 374 00:21:39,640 --> 00:21:41,260 The user basically can't do anything. 375 00:21:41,260 --> 00:21:41,860 Meaning, like... 376 00:21:41,860 --> 00:21:43,380 They can't even change the desktop wallpaper. 377 00:21:44,220 --> 00:21:44,740 This... 378 00:21:44,740 --> 00:21:47,260 But well, it's the way to manage 3,000 users. 379 00:21:47,560 --> 00:21:48,500 Well then... 380 00:21:48,500 --> 00:21:50,500 Nobody goes into a host to configure it! 381 00:21:50,620 --> 00:21:52,620 From AWX a playbook is launched directly. 382 00:21:54,080 --> 00:21:55,020 And, uh... 383 00:21:55,020 --> 00:21:59,480 Well, the WakeOnLAN topic, integration, kernel updates, 384 00:21:59,740 --> 00:22:01,700 sftp fixes. 385 00:22:02,100 --> 00:22:06,240 Meaning, everything that would lead to the new IPA version. 386 00:22:06,760 --> 00:22:10,940 We worked quite a bit with AppArmor to restrict... 387 00:22:10,940 --> 00:22:11,500 um... 388 00:22:11,500 --> 00:22:13,160 Some things that are important. 389 00:22:13,760 --> 00:22:15,240 Login handling. 390 00:22:16,300 --> 00:22:19,300 Also handling the ability to change passwords. 391 00:22:19,480 --> 00:22:19,880 um... 392 00:22:19,880 --> 00:22:21,960 Well, backups and all that. 393 00:22:22,960 --> 00:22:24,100 And, uh... 394 00:22:24,100 --> 00:22:25,540 Let's see... 395 00:22:26,740 --> 00:22:28,220 A bit of AWX here. 396 00:22:28,740 --> 00:22:29,340 um... 397 00:22:30,040 --> 00:22:34,240 Well, there have been 198 releases since we finished migrating it. 398 00:22:34,260 --> 00:22:36,640 Because like any large infrastructure, when you finish migrating it 399 00:22:36,640 --> 00:22:37,340 you have to start migrating again. 400 00:22:37,480 --> 00:22:38,000 Basically. 401 00:22:38,360 --> 00:22:39,920 It's in constant change. 402 00:22:40,940 --> 00:22:44,640 And, well, all the time new needs come up... 403 00:22:45,380 --> 00:22:45,920 um... 404 00:22:46,940 --> 00:22:51,520 Well, we have CloneZilla to boot a host. 405 00:22:51,520 --> 00:22:52,600 with a pre-built image. 406 00:22:52,620 --> 00:22:53,140 um... 407 00:22:53,140 --> 00:22:54,220 And solving problems. 408 00:22:55,180 --> 00:22:55,720 um... 409 00:22:55,720 --> 00:22:56,400 Here, uh... 410 00:22:56,750 --> 00:22:57,680 Isolated profiles. 411 00:22:57,740 --> 00:22:58,280 um... 412 00:22:58,280 --> 00:23:00,560 Isolated from each other due to configuration issues. 413 00:23:01,200 --> 00:23:01,740 um... 414 00:23:01,740 --> 00:23:03,260 A CCTV component was added. 415 00:23:03,300 --> 00:23:07,000 Equipment integrated with the security camera DVRs. 416 00:23:07,960 --> 00:23:08,500 um... 417 00:23:08,500 --> 00:23:11,920 So, well, that's a new inventory that was added. 418 00:23:12,600 --> 00:23:13,020 um... 419 00:23:13,390 --> 00:23:15,460 HP Linux Tools, for example, this... 420 00:23:15,460 --> 00:23:18,120 A detail is that at one point, uh... 421 00:23:18,120 --> 00:23:22,540 A host would freeze and hang in strange situations. 422 00:23:22,940 --> 00:23:25,180 It took us a while to investigate what was happening. 423 00:23:25,500 --> 00:23:28,180 It had to do with a BIOS power saving configuration. 424 00:23:28,420 --> 00:23:30,480 And so the solution was simple. 425 00:23:30,640 --> 00:23:34,620 It was to go into 3,000 hosts, change the BIOS config and reboot them. 426 00:23:34,660 --> 00:23:34,840 Done. 427 00:23:34,960 --> 00:23:35,460 It's very simple. 428 00:23:36,140 --> 00:23:37,340 Do you need 3,000 technicians? 429 00:23:37,560 --> 00:23:37,920 or 430 00:23:39,440 --> 00:23:39,840 um... 431 00:23:39,840 --> 00:23:41,740 As many as there are branches across the country. 432 00:23:41,960 --> 00:23:42,800 It's impossible. 433 00:23:43,160 --> 00:23:44,480 And, well, there, uh... 434 00:23:44,480 --> 00:23:46,720 I started investigating HP's FTP. 435 00:23:46,940 --> 00:23:51,240 And I found a guy in Linux who had a tool that allows you 436 00:23:51,240 --> 00:23:53,460 to write to the UEFI. 437 00:23:54,380 --> 00:23:54,780 um... 438 00:23:54,780 --> 00:23:57,560 With the BIOS configuration it will take on the next reboot. 439 00:23:57,800 --> 00:23:59,540 This requires compiling a kernel module and such. 440 00:23:59,940 --> 00:24:00,340 And... 441 00:24:01,240 --> 00:24:01,640 um... 442 00:24:01,640 --> 00:24:05,440 We told them to ask HP for permission and guarantee they wouldn't 443 00:24:05,460 --> 00:24:06,440 turn into 3,000 bricks. 444 00:24:06,520 --> 00:24:06,940 um... 445 00:24:06,940 --> 00:24:07,460 They said... 446 00:24:07,460 --> 00:24:07,740 Yes! 447 00:24:08,440 --> 00:24:10,420 And, well, we did several tests. 448 00:24:10,540 --> 00:24:11,400 It never happened to us. 449 00:24:11,520 --> 00:24:13,220 So they all booted up fine. 450 00:24:13,880 --> 00:24:14,400 um... 451 00:24:14,400 --> 00:24:18,080 And that allows, well, centralized administration again, 452 00:24:18,080 --> 00:24:21,920 you could even change the BIOS password on all hosts 453 00:24:21,920 --> 00:24:22,900 centrally and remotely. 454 00:24:22,960 --> 00:24:24,220 So it's super useful. 455 00:24:25,280 --> 00:24:25,800 um... 456 00:24:27,040 --> 00:24:27,560 Well. 457 00:24:27,760 --> 00:24:29,420 Here, for example, uh... 458 00:24:30,300 --> 00:24:32,980 Modification of Git tokens and... 459 00:24:32,980 --> 00:24:34,040 There are a lot of things. 460 00:24:34,040 --> 00:24:34,720 This... 461 00:24:34,720 --> 00:24:40,840 At the URL shown below, https://filiales-gnu-linux.g.coop.ar 462 00:24:40,840 --> 00:24:41,280 All of this will be available. 463 00:24:41,340 --> 00:24:42,780 So if you want to see it in detail. 464 00:24:43,960 --> 00:24:44,440 um... 465 00:24:44,440 --> 00:24:46,380 And since we have five minutes left, uh... 466 00:24:46,380 --> 00:24:51,220 If you agree, I'll wrap up here and you can ask any questions... 467 00:24:51,340 --> 00:24:52,920 Because we won't get to see everything. 468 00:25:07,720 --> 00:25:08,200 Hello. 469 00:25:17,260 --> 00:25:17,740 um... 470 00:25:17,740 --> 00:25:22,180 Are the ActiveDirectories you use Windows or do you also use Windows Server? 471 00:25:22,200 --> 00:25:23,900 Yes, it's Bank infrastructure. 472 00:25:24,280 --> 00:25:24,760 um... 473 00:25:24,760 --> 00:25:26,040 That was already preexisting. 474 00:25:26,040 --> 00:25:27,100 And they weren't going to change it. 475 00:25:29,640 --> 00:25:33,600 The challenge was to get all those users to work on the new 476 00:25:33,600 --> 00:25:33,880 workstations. 477 00:25:35,240 --> 00:25:35,680 um... 478 00:25:35,680 --> 00:25:36,500 And it works! 479 00:25:47,640 --> 00:25:49,620 First of all, impressive. 480 00:25:50,620 --> 00:25:52,460 Thanks to the Free Software community! Uh... 481 00:25:52,480 --> 00:25:54,020 We put the pieces together. 482 00:25:54,240 --> 00:25:57,660 They got the most out of Ansible, but not I thought so much could really 483 00:25:57,660 --> 00:25:57,980 be done 484 00:25:58,580 --> 00:26:01,660 I had a small doubt about something I saw. 485 00:26:02,140 --> 00:26:05,020 You said you changed from rsyslog to auditd. 486 00:26:05,180 --> 00:26:05,580 Yes. 487 00:26:06,420 --> 00:26:08,060 It was a cybersecurity requirement. 488 00:26:08,780 --> 00:26:09,840 But the intention... 489 00:26:09,840 --> 00:26:14,980 Let's say, rsyslog logs at an application level, maybe a 490 00:26:14,980 --> 00:26:15,260 system level. 491 00:26:15,380 --> 00:26:18,280 And auditd focuses more on the syscalls of the system itself. 492 00:26:18,640 --> 00:26:20,500 Yes, but it allows more detail. 493 00:26:21,460 --> 00:26:24,160 You can specify exactly which parts you want. 494 00:26:24,720 --> 00:26:25,500 No, no, no. 495 00:26:25,520 --> 00:26:25,900 I see. 496 00:26:26,140 --> 00:26:33,420 My doubt is, are you logging syscalls from 3,000 clients? 497 00:26:33,420 --> 00:26:34,360 That is... 498 00:26:34,360 --> 00:26:35,900 Not from all the workstations. 499 00:26:35,900 --> 00:26:37,100 Actually it is... 500 00:26:38,200 --> 00:26:38,760 Eh... 501 00:26:38,760 --> 00:26:40,540 mainly from VMs, and some things. 502 00:26:40,720 --> 00:26:41,760 It's selective, not everything. 503 00:26:42,000 --> 00:26:42,580 Ah, ah. 504 00:26:42,620 --> 00:26:43,480 No, because otherwise there's no way. 505 00:26:43,880 --> 00:26:44,640 No, no, it's fine. 506 00:26:44,760 --> 00:26:45,320 That was just it. 507 00:26:45,540 --> 00:26:47,260 Exactly to avoid sending all the logs. 508 00:26:47,460 --> 00:26:49,640 Meaning, sending selectively. 509 00:26:49,980 --> 00:26:53,760 All this information goes to a SIEM at Headquarters. 510 00:26:54,040 --> 00:26:54,460 Just one? 511 00:26:55,120 --> 00:26:57,660 Yes, well, it could be more than one node. 512 00:26:58,040 --> 00:26:58,480 Ah, no. 513 00:26:58,500 --> 00:27:02,940 But, let's say, conceptually it goes to a SIEM that everything is then displayed 514 00:27:02,940 --> 00:27:03,520 there what happens. 515 00:27:04,200 --> 00:27:06,080 And all of AWX deploy logs also 516 00:27:06,080 --> 00:27:09,020 go to the SIEM... 517 00:27:09,220 --> 00:27:10,260 So also... 518 00:27:10,260 --> 00:27:10,620 As a SIEM? 519 00:27:11,040 --> 00:27:15,620 No, another one I don't remember now, but it's well-known. 520 00:27:17,000 --> 00:27:18,680 Now, if I tell you, I'd be lying. 521 00:27:18,700 --> 00:27:18,760 Thank you very much. 522 00:27:23,580 --> 00:27:28,660 I agree with the colleague on the hard work, tremendous effort. 523 00:27:29,240 --> 00:27:31,440 And all done with Free Software. 524 00:27:31,960 --> 00:27:35,900 My question is basically based on the experience of the entire project. 525 00:27:37,940 --> 00:27:39,340 What went really well? 526 00:27:39,420 --> 00:27:39,780 What went really well? 527 00:27:40,600 --> 00:27:43,280 What are you going to start doing, besides what you've mentioned? 528 00:27:43,360 --> 00:27:47,200 And what things went one way and you said, no, this isn't the right approach, 529 00:27:47,340 --> 00:27:48,760 we pivoted? 530 00:27:48,820 --> 00:27:55,860 The biggest challenge overall isn't technical per se, but rather dealing with 531 00:27:55,860 --> 00:27:58,100 the priorities of everything that needs to be done. 532 00:27:58,280 --> 00:28:05,320 And with things where at the user level, many free applications aren't designed 533 00:28:05,320 --> 00:28:08,380 for such a large scale. 534 00:28:08,380 --> 00:28:12,420 And a problem where you want users to be unable to touch anything. 535 00:28:12,560 --> 00:28:18,500 For example, there are some PDFs that are smart forms you can fill 536 00:28:18,500 --> 00:28:19,480 fill out. 537 00:28:19,760 --> 00:28:21,580 And then you have to print them. 538 00:28:21,800 --> 00:28:24,180 And that was really complex to solve. 539 00:28:24,300 --> 00:28:28,300 At first we had to install Adobe Acrobat with, I don't know, Wine or 540 00:28:28,300 --> 00:28:29,100 something horrible like that. 541 00:28:29,760 --> 00:28:31,720 Because it was the only thing that supported it. 542 00:28:33,300 --> 00:28:36,620 Evince displayed them, but didn't allow us to fill them out. 543 00:28:38,780 --> 00:28:42,240 Later we had an option I don't remember which one that allowed 544 00:28:42,240 --> 00:28:42,760 filling them out. 545 00:28:42,760 --> 00:28:48,880 But it didn't let you hide the comments on each form, 546 00:28:48,880 --> 00:28:50,320 in each textbox. 547 00:28:50,580 --> 00:28:52,420 And that would print out and wouldn't work. 548 00:28:52,920 --> 00:28:56,840 Now there's been work on Okular, which has all of that. 549 00:28:57,020 --> 00:29:01,100 But it has no per-file configuration option to 550 00:29:01,100 --> 00:29:02,220 disable all that. 551 00:29:02,280 --> 00:29:04,060 Meaning the user can disable it at the moment. 552 00:29:05,060 --> 00:29:06,440 Well, so we worked on it. 553 00:29:06,620 --> 00:29:09,640 We're touching Okular's code to make that work. 554 00:29:10,300 --> 00:29:13,560 The same thing happened with some binaries that have no translation and no translation 555 00:29:13,560 --> 00:29:13,840 support. 556 00:29:13,960 --> 00:29:16,480 And well, what we could we edited with a hexadecimal editor. 557 00:29:16,660 --> 00:29:18,360 And embedded our binary. 558 00:29:19,320 --> 00:29:25,860 Things like that, let's say, with respect to the Asians, we call them Chinese. 559 00:29:25,980 --> 00:29:29,620 Meaning, it's a hack that sometimes you have to dig into and resolve. 560 00:29:29,700 --> 00:29:34,640 Or complex bugs where, I don't know, you enter a screen and for various reasons, 561 00:29:34,640 --> 00:29:36,380 if you move the mouse a bit more down to the right, 562 00:29:36,380 --> 00:29:37,640 it crashes GNOME. 563 00:29:37,880 --> 00:29:41,880 And so you have to put a script that prevents you from going down there. 564 00:29:43,120 --> 00:29:51,720 And also things that a user who owns their environment doesn't have a problem with 565 00:29:51,720 --> 00:29:52,600 because they can customize it. 566 00:29:53,000 --> 00:29:56,660 And an end user can't touch anything. 567 00:29:56,880 --> 00:30:00,900 And you, as administrator of this infrastructure, also don't want them to 568 00:30:00,900 --> 00:30:01,240 touch it. 569 00:30:01,280 --> 00:30:02,520 But you have to give a solution. 570 00:30:05,220 --> 00:30:06,420 So, a thousand things. 571 00:30:07,400 --> 00:30:11,240 What we didn't get involved in and outsourced was the check scanner 572 00:30:11,240 --> 00:30:11,640 integration part. 573 00:30:11,760 --> 00:30:14,160 I tried for a while and couldn't make it work. 574 00:30:14,200 --> 00:30:15,380 That was outsourced. 575 00:30:16,900 --> 00:30:17,640 But it works now. 576 00:30:18,480 --> 00:30:20,700 Yes, we did manage to get the ticket printer to work properly. 577 00:30:21,460 --> 00:30:28,660 And that implied a challenge because the banking core at the time used 578 00:30:28,660 --> 00:30:29,420 Java applets. 579 00:30:30,320 --> 00:30:32,500 And that, nothing, was impossible. 580 00:30:32,520 --> 00:30:36,520 We had to put a chroot in the middle with something old for that to work. 581 00:30:36,560 --> 00:30:38,220 Well, in the end we managed to avoid all of that. 582 00:30:38,640 --> 00:30:44,160 Because now that core already detects when it's GNU/Linux. 583 00:30:44,360 --> 00:30:51,600 And so, it outputs code and we put in a local backend that does 584 00:30:51,600 --> 00:30:52,900 everything needed and it works. 585 00:30:53,020 --> 00:30:54,940 But yes, it's a challenge involving many people. 586 00:30:55,040 --> 00:30:56,580 I'm currently not on the project. 587 00:30:57,240 --> 00:31:00,600 I was in the initial development and migration phase. 588 00:31:01,200 --> 00:31:02,500 There are three people now... 589 00:31:02,520 --> 00:31:04,180 who work full-time on this. 590 00:31:04,420 --> 00:31:06,260 And every day something new appears. 591 00:31:08,440 --> 00:31:08,880 Great. 592 00:31:15,540 --> 00:31:17,680 I can stay here and chat. 593 00:31:21,620 --> 00:31:22,060 Well. 594 00:31:23,080 --> 00:31:24,280 OSiRiS, first of all thank you. 595 00:31:25,620 --> 00:31:27,520 For those of us who came here to learn. 596 00:31:29,240 --> 00:31:32,240 The admiration you generate with everything you've shared. 597 00:31:32,380 --> 00:31:33,140 My question isn't about... 598 00:31:33,140 --> 00:31:34,200 It's gcoop's work too, not mine... 599 00:31:34,520 --> 00:31:35,220 Everyone's. 600 00:31:35,400 --> 00:31:36,620 My question goes more... 601 00:31:36,620 --> 00:31:37,240 I'm selling it. 602 00:31:37,600 --> 00:31:38,520 It goes more that way. 603 00:31:38,620 --> 00:31:39,620 Not the technical side. 604 00:31:39,700 --> 00:31:43,220 But rather the team or human group. 605 00:31:43,240 --> 00:31:43,520 Not the human side. 606 00:31:43,560 --> 00:31:45,120 What's at Filiales GNU/Linux 607 00:31:45,120 --> 00:31:49,480 that if I don't understand is the organization behind all 608 00:31:49,480 --> 00:31:50,540 this project. 609 00:31:51,240 --> 00:31:53,520 How many people are needed to do this? 610 00:31:53,680 --> 00:31:55,540 And how do you manage it? 611 00:31:55,920 --> 00:31:59,480 Well, initially it was a six-month project. 612 00:31:59,540 --> 00:32:00,660 Of two people. 613 00:32:01,440 --> 00:32:03,280 A functional analyst and me. 614 00:32:04,100 --> 00:32:06,300 To see if the project was viable. 615 00:32:08,300 --> 00:32:09,500 Then it started... 616 00:32:09,500 --> 00:32:11,060 I think we started with three people. 617 00:32:11,720 --> 00:32:13,220 At the peak we reached three plus five. 618 00:32:13,240 --> 00:32:13,380 To five? 619 00:32:13,780 --> 00:32:14,360 Three plus five. 620 00:32:16,330 --> 00:32:16,890 Right, yes. 621 00:32:17,510 --> 00:32:18,450 Always with a PM. 622 00:32:18,630 --> 00:32:20,370 No, but at the peak it was five, not that way. 623 00:32:20,570 --> 00:32:21,750 Meaning, I think there... 624 00:32:21,750 --> 00:32:23,450 I don't count the PM, poor guy. 625 00:32:23,970 --> 00:32:25,690 Because he always talks about... 626 00:32:25,690 --> 00:32:27,850 The PM always talks about how we're not people. 627 00:32:27,890 --> 00:32:28,990 That we are developers! 628 00:32:29,510 --> 00:32:30,690 That we are weird things. 629 00:32:30,790 --> 00:32:31,210 So, well. 630 00:32:31,450 --> 00:32:32,330 It's revenge. 631 00:32:32,830 --> 00:32:34,070 No, at the peak it was five people. 632 00:32:35,630 --> 00:32:36,590 Today there are three. 633 00:32:37,370 --> 00:32:38,630 DevOps people, let's say. 634 00:32:38,850 --> 00:32:40,010 But you have to... 635 00:32:40,750 --> 00:32:42,150 Let's say, it's full stack. 636 00:32:42,550 --> 00:32:43,750 So to speak. 637 00:32:43,750 --> 00:32:47,530 But then you have to touch or redesign an application. 638 00:32:47,770 --> 00:32:51,630 Meaning, we had to design applications to work around existing issues. 639 00:32:52,370 --> 00:32:55,550 Meaning, it's the fun and complex part at the same time. 640 00:32:56,130 --> 00:32:58,030 I thought you were going to say three hundred, I don't know. 641 00:32:58,170 --> 00:32:58,830 Thank you very much. 642 00:32:58,830 --> 00:32:59,430 No, no, sorry. 643 00:32:59,430 --> 00:33:03,950 We designed the automation for all of this. 644 00:33:04,030 --> 00:33:06,870 The bank has its own arsenal. 645 00:33:06,890 --> 00:33:10,430 An army of people who use all of this daily. 646 00:33:10,670 --> 00:33:13,470 They did the migration in the middle of the 2020 pandemic. 647 00:33:13,750 --> 00:33:15,050 In less than a year! 648 00:33:16,190 --> 00:33:19,910 But it was a physical army of people who resolved all of that. 649 00:33:20,250 --> 00:33:22,530 AWX operators, I think there are about sixty. 650 00:33:22,710 --> 00:33:23,210 Meaning, like... 651 00:33:23,770 --> 00:33:25,030 Well, there's a lot of people. 652 00:33:25,910 --> 00:33:27,270 The bank's infrastructure is very large. 653 00:33:27,410 --> 00:33:28,890 And it's all On-premise. 654 00:33:28,950 --> 00:33:29,530 It's all local. 655 00:33:29,770 --> 00:33:31,690 Meaning, and all with Free Software. 656 00:33:32,810 --> 00:33:33,530 Thank you very much. 657 00:33:34,670 --> 00:33:35,470 There, Alejandro. 658 00:33:35,550 --> 00:33:36,590 I'm putting a question in the chat for you. 659 00:33:36,730 --> 00:33:38,090 If you can answer it there later. 660 00:33:38,110 --> 00:33:38,370 Yes. 661 00:33:38,370 --> 00:33:40,370 And later I have here colleagues who speak English. 662 00:33:40,490 --> 00:33:42,590 And if there's anyone who doesn't speak Spanish. 663 00:33:43,750 --> 00:33:43,950 Go ahead. 664 00:33:43,970 --> 00:33:45,970 And Alejandro will be around today. 665 00:33:45,990 --> 00:33:46,890 So you can keep asking him questions. 666 00:33:47,050 --> 00:33:47,590 Thank you very much. 667 00:33:47,590 --> 00:33:47,910 OSiUX too!