[elrepo] nvidia plugin slow

[Phil Perry]

On 01/06/2019 00:18, Steve Cleveland wrote:
> 
> On 5/31/19 1:18 PM, Phil Perry wrote:

<Snip>

>>
>> Searching 20 packages
>> searching package kmod-nvidia-410.93-1.el7_6.elrepo.x86_64
>> searching in provides entries
>> searching package kmod-nvidia-418.43-1.el7_6.elrepo.x86_64
>> searching in provides entries
>> searching package kmod-nvidia-418.56-1.el7_6.elrepo.x86_64
>> searching in provides entries
>> searching package kmod-nvidia-418.74-1.el7_6.elrepo.x86_64
>> searching in provides entries
>> searching package kmod-nvidia-430.14-1.el7_6.elrepo.x86_64
>> searching in provides entries
>> searching package nvidia-x11-drv-410.93-1.el7_6.elrepo.x86_64
>> searching in provides entries
>> searching package nvidia-x11-drv-418.43-1.el7_6.elrepo.x86_64
>> searching in provides entries
>> searching package nvidia-x11-drv-418.56-1.el7_6.elrepo.x86_64
>> searching in provides entries
>> searching package nvidia-x11-drv-418.74-1.el7_6.elrepo.x86_64
>> searching in provides entries
>> searching package nvidia-x11-drv-430.14-1.el7_6.elrepo.x86_64
>> searching in provides entries
>> searching package nvidia-x11-drv-libs-410.93-1.el7_6.elrepo.i686
>> searching in provides entries
>> searching package nvidia-x11-drv-libs-410.93-1.el7_6.elrepo.x86_64
>> searching in provides entries
>> searching package nvidia-x11-drv-libs-418.43-1.el7_6.elrepo.i686
>> searching in provides entries
>> searching package nvidia-x11-drv-libs-418.43-1.el7_6.elrepo.x86_64
>> searching in provides entries
>> searching package nvidia-x11-drv-libs-418.56-1.el7_6.elrepo.i686
>> searching in provides entries
>> searching package nvidia-x11-drv-libs-418.56-1.el7_6.elrepo.x86_64
>> searching in provides entries
>> searching package nvidia-x11-drv-libs-418.74-1.el7_6.elrepo.i686
>> searching in provides entries
>> searching package nvidia-x11-drv-libs-418.74-1.el7_6.elrepo.x86_64
>> searching in provides entries
>> searching package nvidia-x11-drv-libs-430.14-1.el7_6.elrepo.i686
>> searching in provides entries
>> searching package nvidia-x11-drv-libs-430.14-1.el7_6.elrepo.x86_64
>> searching in provides entries
> 
> 
> debug 5 was definitely the most useful.  The 'searching in provides 
> entries' is taking almost two seconds for each entry.
> 
> pkgsack time is between 40 and 60 seconds.
> 
> rpmdb time: 0.000
> pkgsack time: 42.327
> tags time: 0.000
> 

For me, to scroll through the above 20 entries takes around 2-3 seconds 
and the reported pkgsack time is 5-6 seconds. My best guess is it's 
taking around 2-3secs to search all available packages to get a list of 
those providing blacklist provides, and then another 2-3secs to loop 
through the 20 packages identified.

> That is using the official elrepo repository.  I have a local mirror 
> that includes some additional older versions. So the problem is much 
> more  severe in our production systems.  But I have cleaned up a bunch 
> of older versions.  And now that I know what's going on, I can find ways 
> to work around it.
> 

The additional packages could be contributing.

I've trimmed the repositories to remove older releases, just leaving the 
last two. I'll wait for the mirrors to update and then test again to see 
the impact, but if I'm right above, then I'd expect to see it shave 
around 1-1.5secs off my pkgsack time.

> I will take a look at your code repository to see I can come up with any 
> suggestions, but I am certainly not a python coder myself.  Looping 
> through an array shouldn't take that long though
> 

You'd be surprised. When I was writing the yum-plugin-elrepo plugin to 
check kmod compatibility with available kernels, I wrote a section of 
code to loop through each kernel() symbol requires for each kmod package 
to compare and match against those kernel() symbols provided by each 
available kernel and that took an infeasibly long amount of time and was 
simply not practical. For example, a single kmod-nvidia package has 
around 400 kernel() symbol requires and each kernel provides around 
17,000 kernel symbols, so matching just one kmod package against one 
kernel package could require 6 million iterations of the loop! Then 
multiply that up a couple hundred available kmod packages in the repo 
and a dozen plus kernels and you soon see how looping through an array 
can take a relatively long time in a slow language like python. In the 
end I cheated and ended up matching against kernel version rather than 
trying to resolve each individual package dependency.

I really need to figure out how yum does dep resolution as it must be 
using a C library or something. There is no way it's being done in python.

>>
>> I could probably prune that back to the last 2 driver releases which 
>> should save considerable time if this is the cause.
>>
>> I'll leave it alone for now so you can have a play at different 
>> debuglevels and share any observations, and then we can try trimming 
>> the number of packages to see the results.
>>
>> Other than that, we would need to look at optimising the python code. 
>> I'm not much of a python expert, but I don't see much that can be 
>> optimised. The only thing that springs to mind is that we are 
>> searching all packages for 'blacklist' provides, when we could maybe 
>> limit that to only searching the elrepo repository. I've no idea if 
>> this is relevant - do you have a huge number of additional packages 
>> available to yum through any large repositories? In theory it should 
>> be easy enough to test with some debug code to show us what's going on 
>> at each stage so we can see exactly what's adding to the time.
> 
> 
> We don't use any repos out of the ordinary.  EPEL, elrepo and a few 
> other small ones.
> 
> If I'm reading things right, the search for blacklist provides (where 
> you seach all packages) only happens the one time?  Then the 'loop' 
> happens on the packages with the 'blacklist' provides, correct?
> 

Yes, the code searches once for all packages with blacklist provides, 
and loops through those looking for a match. To get an idea of numbers, 
each 'nvidia' package contains around 600 blacklist provides, and there 
are 4 packages for each driver release. Assuming your hardware is 
supported and not blacklisted, that's a full 2,400 iterations of the 
loop for each driver release, or 12,000 iterations for the 5 driver 
releases we currently have in the repo.

Phil